First hMailServer 5.4 alpha available

Use this forum if you want to discuss a problem or ask a question related to a hMailServer beta release.
User avatar
martin
Developer
Developer
Posts: 6835
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

First hMailServer 5.4 alpha available

Post by martin » 2010-10-24 22:10

A hMailServer 5.4 alpha is now available:
http://www.hmailserver.com/?page=download_latest

Please notice that this is the first alpha and as such has not received much testing. Please do not install this in any production environment where you can't risk losing data.

Half a ton of changes have been made in version 5.4; a complete list will be provided at a later time. A few example of changes follows:
  • A built-in ClamAV integration has been added.
  • hMailServer no longer stores the full path to message files in the database.
  • It's possible to test the anti-virus integration from hMailServer administrator.
Thanks to everyone who has helped with this!

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: First hMailServer 5.4 alpha available

Post by Bill48105 » 2010-10-24 23:05

WOO thanks martin! I upgraded a test server here & so far so good.

I'll note some new *optional* 'hidden' changes that are in testing for now that require adding keys to the hMailServer.INI file in the hmailserver/bin folder if anyone wants to try them out:
(The comment lines that start with ; are not needed but help describe each key)
(Only EXAMPLE values are shown. Although reasonable you should decide what is best for your needs)
(You must restart the hmailserver service for the changes to take effect.)

Code: Select all

[Settings]
SepSvcLogs=1
; This key tells hmailserver to split SMTP/IMAP/POP into their own log files such as
; hmailserver_SMTP_2010-10-24.log hmailserver_IMAP_2010-10-24.log hmailserver_POP3_2010-10-24.log
; Default is the old method if key doesn't exist or set to 0.

LogLevel=9
; This key puts a little more control over how much is logged where 1=lowest/least 9=highest/most. 
; At this point if set to 1 it filters out the POP UIDL/LIST SENT results and some IMAP lines that aren't
; normally needed otherwise doesn't do anything but could be useful for other things later. 
; This defaults to old method if key doesn't exist or if debug logging is enabled

MaxLogLineLen=300
; This key causes log lines to be neatly truncated at specified # of characters to reduce log sizes.
; Example: "POP3D"	3952	2	"2010-10-24 15:18:59.924"	"127.0.0.1"	"SENT: 1 756[nl]2 ... 350[nl]1973 371[nl]."
; This defaults to disabled if key doesn't exist or LogLevel > 1 or if debug logging is enabled

QuickRetries=3
; This key tells hmail to over-ride default wait/retry schedule for 1st defined # of tries which should
; allow one to set retry delay higher while not causing greylisted messages to be stuck too long.
; This defaults to 0 or disabled if not defined

QueueRandomnessMinutes=7
; This key defines how many minutes to use for 'QuickRetries' that over-rides the default retry schedule.
; This does not apply to NEW messages as they are still queued for immediate delivery as before.
; This defaults to 0 or disabled

MXTriesFactor=5
; This key defines a multiplier to determine how many MX/A server results to try per retry.
; For example: If set to 2 & the recipient has 11 servers, on 1st try, 1 x MXTriesFactor or 2 are tried.
; On 2nd try, 2 x MXTriesFactor or 4 are tried. On 3rd 6 are tried & so on. Eventually all 11
; would be tried if the message was not successively delivered enough times.
; This defaults to disabled if not defined

QuickRetriesMinutes=7
; Defines # of minutes to delay for each retry if QuickRetries is enabled otherwise normal delivery schedule is used
; This defaults to 0 or disabled
I realize INI settings are not ideal but for testing at this point they should be OK. Eventually if people find these functions useful they will likely be added to the Administrator and have database settings instead.

Hopefully people find some of these changes useful. :)
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

Melibokus73
Normal user
Normal user
Posts: 87
Joined: 2009-06-26 20:13

Re: First hMailServer 5.4 alpha available

Post by Melibokus73 » 2010-10-25 17:02

Hi Martin,

Great, this Alhpa seems to run fine.


Did also make quick look at Issue 248 fixed in this Alpha:

http://www.hmailserver.com/issues/?248

seems to run: now there is a adequat Delivered-To: line:

But only one strange thing:

Mail to nosuchuser@mydomain.com went to Mirror and Catchall => OK

Mail to myself@mydomain.com (from myselfexternal@mymailprovider.com external account) went to Mirror and my mailaccount => OK

Mail to myself@mydomain.com (send by myself@mydomain.com directly local by mailclient to hmailserver ) went to my mailaccount and?? nothing ?!?! could you reproduce this ?

This did work with last Release of HMS (with wrong Delivered-To: Mirror line)

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: First hMailServer 5.4 alpha available

Post by Bill48105 » 2010-10-25 17:26

Hey Melibokus73,
Are you saying mirror doesn't work local to local? It works for me..
"SMTPC" 2916 0 "2010-10-25 11:22:31.329" "APP" "SMTPDeliverer - Message 7587: Delivering message from bill@domain1.com to bill@domain1.com. File: C:\hMailServer\Dat ... AD89-CD5709CA63CA}.eml"
"SMTPC" 2268 0 "2010-10-25 11:22:31.510" "APP" "SMTPDeliverer - Message 7588: Delivering message from bill@domain1.com to test@mydomain2.com. File: C:\hMailServer\Data\ ... 9709-8B4AB08A4679}.eml"
The 2nd line is the mirror copy.. Are you sure it's not working for you or possibly overlooked somehow?
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

User avatar
Slug
Moderator
Moderator
Posts: 1369
Joined: 2005-03-13 05:42
Location: Sydney Australia
Contact:

Re: First hMailServer 5.4 alpha available

Post by Slug » 2010-10-25 17:32

Downloading now, will give it a run ..

Michael
Missing Hmailserver ... Now running Debian servers

^DooM^
Site Admin
Posts: 13862
Joined: 2005-07-29 16:18
Location: UK

Re: First hMailServer 5.4 alpha available

Post by ^DooM^ » 2010-10-25 17:40

Worked fine for me so far for the past 24 hours.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: First hMailServer 5.4 alpha available

Post by Bill48105 » 2010-10-25 17:44

OOPS. I see I forgot one of the INI settings:

Code: Select all

QuickRetriesMinutes=7
; Defines # of minutes to delay for each retry if QuickRetries is enabled otherwise normal delivery schedule is used
; Defaults to 6 if QuickRetries is enabled otherwise ignored
Also QueueRandomnessFactor is NOT related to QuickRetries as stated originally so the line:

Code: Select all

; This defaults to 6 minutes if key doesn't exist but QuickRetries is defined otherwise normal retry is used
Should instead be:

Code: Select all

; This defaults to 0 or disabled
Perhaps a mod could please update my earlier post to help keep em in 1 place & avoid confusion? :D
Thx
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

ObiWan
Senior user
Senior user
Posts: 278
Joined: 2010-07-21 14:30
Location: Halfway between Germany and Egypt

Re: First hMailServer 5.4 alpha available

Post by ObiWan » 2010-10-25 18:35

martin wrote: A hMailServer 5.4 alpha is now available:
http://www.hmailserver.com/?page=download_latest
COOL !!

Just a quick question; does updating an existing 5.x install with
this new version need some additional steps (e.g. to change the
DB message paths or the like) or it will just work "out of the box" ?

Just asking since I'm thinking to try it on a "test" system but
before screwing everything up and ending in a "SNAFU" I'd like
to know if there's something to try to avoid such a situation :)

Anyways, keep up the good work, way to go !!

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: First hMailServer 5.4 alpha available

Post by Bill48105 » 2010-10-25 18:54

Hey ObiWan,
I am 99% sure nothing is done with existing messages during upgrade (or if there is it's not done with the SQL upgrade files in the DBScripts folder I've looked at..) BUT diags will give you a 'notice' that old messages with full path exist like:
ERROR: Full paths are stored in the database.
I am pretty sure that is just a friendly reminder you have some with full path still & not an indication something is broken. I've upgraded a few 5.3 test machines but don't recall if there were messages in boxes still or not but if you want you could setup 5.3 on a test machine, send a few messages but leave them in the user box, upgrade to 5.4 & view the hm_messages table to see.. Or maybe someone else who knows for sure can confirm.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

User avatar
mattg
Moderator
Moderator
Posts: 19897
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: First hMailServer 5.4 alpha available

Post by mattg » 2010-10-26 03:06

ObiWan wrote:Just a quick question; does updating an existing 5.x install with
this new version need some additional steps (e.g. to change the
DB message paths or the like) or it will just work "out of the box" ?
Here is an earlier thread in which Martin discussed this issue...

http://www.hmailserver.com/forum/viewto ... 13&t=16128
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: First hMailServer 5.4 alpha available

Post by Bill48105 » 2010-10-26 03:36

It's not confirmed yet but there is possibly an issue with the 5.4 alpha downloading POP3 messages which seems to occur with a lot of messages. (At least 100's from what I've seen but it might just be more likely to happen with more messages..)

If anyone has run into unexplained timeouts or errors while downloading messages (in this 5.4 alpha or recent 5.3 releases in case the problem existed before) please post up a log snippet around the time of the problem for comparison.

Here is one of many I've seen but still investigating:
"POP3D" 2744 24625 "2010-10-25 16:06:53.562" "192.168.2.3" "SENT: ."
"POP3D" 5384 24625 "2010-10-25 16:06:58.390" "192.168.2.3" "RECEIVED: RETR 768"
"POP3D" 448 24625 "2010-10-25 16:06:58.421" "192.168.2.3" "SENT: ."
"POP3D" 5384 24625 "2010-10-25 16:07:04.343" "192.168.2.3" "RECEIVED: RETR 769"
"POP3D" 5904 24625 "2010-10-25 16:07:04.343" "192.168.2.3" "RECEIVED: "
"POP3D" 5904 24625 "2010-10-25 16:07:04.343" "192.168.2.3" "SENT: -ERR Invalid command in current state."
"POP3D" 2744 24625 "2010-10-25 16:07:04.343" "192.168.2.3" "SENT: ."
"POP3D" 5384 24625 "2010-10-25 16:07:04.343" "192.168.2.3" "SENT: ."
"POP3D" 3712 24634 "2010-10-25 16:07:51.000" "192.168.2.3" "SENT: +OK mydomain.net POP3"
With Outlook it resulted in some messages getting downloaded over & over causing duplicates of those messages in the Inbox. With Outlook express I get a popup error such as this:
Message number 357 could not be retrieved. Account: '127.0.0.1 (1)', Server: '127.0.0.1', Protocol: POP3, Server Response: '.', Port: 110, Secure(SSL): No, Server Error: 0x800CCC90, Error Number: 0x800420CD
Hopefully this is just a false alarm but more testing by others should help confirm it or not.
Thx
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

Melibokus73
Normal user
Normal user
Posts: 87
Joined: 2009-06-26 20:13

Re: First hMailServer 5.4 alpha available

Post by Melibokus73 » 2010-10-26 11:54

Bill48105 wrote:Hey Melibokus73,
Are you saying mirror doesn't work local to local? It works for me..
"SMTPC" 2916 0 "2010-10-25 11:22:31.329" "APP" "SMTPDeliverer - Message 7587: Delivering message from bill@domain1.com to bill@domain1.com. File: C:\hMailServer\Dat ... AD89-CD5709CA63CA}.eml"
"SMTPC" 2268 0 "2010-10-25 11:22:31.510" "APP" "SMTPDeliverer - Message 7588: Delivering message from bill@domain1.com to test@mydomain2.com. File: C:\hMailServer\Data\ ... 9709-8B4AB08A4679}.eml"
The 2nd line is the mirror copy.. Are you sure it's not working for you or possibly overlooked somehow?
Bill
Hi Bill,

I did some more tests,

from external I got still no problems .

but local to local it got better but it does only work "most times" ... first I did think, I had to do with use of Default or Alias domainname oder username at the emailadresse I send the mail to.

External I could only use alias domain for incoming mails on my testing maschine. Local I could use Default domain and Aliasdomain.
But I still could not 100% catch or reproduce this Problem. I try one....


Once also a mail to "nosuchuser@mydomain" only got into catchall, not to Mirror


BTW:

I also saw strange behaviour of the from: line of only some mails : about 20-30% got changed to Mirror@mydomain.com or Catchall@mydomain.com
these happend in all 3 accounts: catchall, Mirror or my user account

User avatar
Slug
Moderator
Moderator
Posts: 1369
Joined: 2005-03-13 05:42
Location: Sydney Australia
Contact:

Re: First hMailServer 5.4 alpha available

Post by Slug » 2010-10-26 12:50

Bill48105 wrote:
Perhaps a mod could please update my earlier post to help keep em in 1 place & avoid confusion? :D
Thx
Bill
Done, but check to see that its right ..

Michael
Missing Hmailserver ... Now running Debian servers

Melibokus73
Normal user
Normal user
Posts: 87
Joined: 2009-06-26 20:13

Re: First hMailServer 5.4 alpha available

Post by Melibokus73 » 2010-10-26 12:53

PS:

here are some logs:

all mails are local send from myself@mydomain.com


this Mail did work.It was send to myalias@mydomain.com

Code: Select all

"SMTPC"	2708	0	"2010-10-26 11:56:17.250"	"APP"	"SMTPDeliverer - Message 1213545: Delivering message from myself@mydomain.com to myself@mydomain.com. File: C:\Program Files (x86)\hMailServer\Data\{B80F79D8-2790-40DF-809E-E3213E58AD57}.eml"
"SMTPC"	2712	0	"2010-10-26 11:56:18.500"	"APP"	"SMTPDeliverer - Message 1213547: Delivering message from myself@mydomain.com to Mirror@mydomain.com. File: C:\Program Files (x86)\hMailServer\Data\{9822193D-BB10-4CAA-8302-F12E3EB77B0D}.eml"
"SMTPC"	2712	0	"2010-10-26 11:56:20.203"	"APP"	"SMTPDeliverer - Message 1213547: Message delivery thread completed."
"SMTPC"	2708	0	"2010-10-26 11:56:20.437"	"APP"	"SMTPDeliverer - Message 1213545: Message delivery thread completed."
this 2nd one was send to myself@myALIASdomain.com

Code: Select all

"SMTPC"	2708	0	"2010-10-26 11:56:59.187"	"APP"	"SMTPDeliverer - Message 1213548: Delivering message from Mirror@mydomain.com to myself@mydomain.com. File: C:\Program Files (x86)\hMailServer\Data\{F84D3D75-381D-4A3D-84A3-A41D11C040CF}.eml"
"SMTPC"	2708	0	"2010-10-26 11:57:01.890"	"APP"	"SMTPDeliverer - Message 1213548: Message delivery thread completed."
as you could see I only devliverred once and with a changed from line (Mirror@mydomain.de)


the strange thing is: this 3rd mail is exact the same as the 2nd, only send 3 minutes laster => everything fine ?!?!?!

Code: Select all

"SMTPC"	2708	0	"2010-10-26 12:00:30.250"	"APP"	"SMTPDeliverer - Message 1213550: Delivering message from myself@mydomain.com to myself@mydomain.com. File: C:\Program Files (x86)\hMailServer\Data\{8A206604-815F-4FD3-8F22-2EE1D1C49C4A}.eml"
"SMTPC"	2712	0	"2010-10-26 12:00:31.515"	"APP"	"SMTPDeliverer - Message 1213552: Delivering message from myself@mydomain.com to Mirror@mydomain.com. File: C:\Program Files (x86)\hMailServer\Data\{2742D8B5-F3AE-4FAE-8F08-F003369E2A3E}.eml"
"SMTPC"	2708	0	"2010-10-26 12:00:34.890"	"APP"	"SMTPDeliverer - Message 1213550: Message delivery thread completed."
"SMTPC"	2712	0	"2010-10-26 12:00:35.734"	"APP"	"SMTPDeliverer - Message 1213552: Message delivery thread completed."

I did try it 4-5 times again => all OK


but now I did try a mail to fsdfsdafsdf@mydomain.com

Code: Select all

"SMTPC"	2708	0	"2010-10-26 12:44:51.562"	"APP"	"SMTPDeliverer - Message 1213568: Delivering message from Mirror@mydomain.com to Catchall@mydomain.com. File: C:\Program Files (x86)\hMailServer\Data\{4C66FFED-AB2C-4CD7-A7CD-91AC0B2820FB}.eml"
"SMTPC"	2708	0	"2010-10-26 12:44:52.859"	"APP"	"SMTPDeliverer - Message 1213568: Message delivery thread completed."
same Problem as above: only got to catchall and From line changed to "Mirror@mydomain.com"

Could it be that this is a strange kind of race condition Issue?

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: First hMailServer 5.4 alpha available

Post by Bill48105 » 2010-10-28 07:44

Hewdy,
I have a working fix for the 5.4 alpha POP3 issue that I've been testing here but waiting to hear back from martin on if he has a better solution. The issue was definitely timing related so depending on many factors one might never see it or might be plagued by it.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

ITSyncForce
Normal user
Normal user
Posts: 32
Joined: 2009-02-11 12:30

Re: First hMailServer 5.4 alpha available

Post by ITSyncForce » 2010-10-28 09:45

I found a bug in the SMTP Routes.

I use a secundairy MX server in case my primairy mailserver is down. I know it is production but as this is my secundairy server that only gets spam (my primairy mailserver has a downtime of less then 5 minutes a month) I upgraded it to see the changes.

I just wanted to add an extra route for a new domain. I added the domain and pressed Save.

At first I could not find the domain as the list of domains is affabetically I would have expected it the be at G as the domain started with a G (lets call it G.com).
After taking a closer look I found the domainname in the list between some domains with a S (S.com). Opening the settings of the domain in the tab General it still showed S.com but in the list on the left it showed G.com. Also all other settings where of G.com.

I changed the settings and S.com into G.com and now the domain settings where correct after a save. I had to recreate a new S.com with its settings. And as far as I can tell this time it did not overwrite any other domain.

However both s.com and G.com now appear at the bottom of the list of domains that is no longer alfabatically.

polloda
New user
New user
Posts: 3
Joined: 2010-09-30 07:02

Re: First hMailServer 5.4 alpha available

Post by polloda » 2010-11-04 23:29

Can I use it for my production server ?

^DooM^
Site Admin
Posts: 13862
Joined: 2005-07-29 16:18
Location: UK

Re: First hMailServer 5.4 alpha available

Post by ^DooM^ » 2010-11-04 23:42

polloda wrote:Can I use it for my production server ?
No
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

ObiWan
Senior user
Senior user
Posts: 278
Joined: 2010-07-21 14:30
Location: Halfway between Germany and Egypt

Re: First hMailServer 5.4 alpha available

Post by ObiWan » 2010-11-05 09:44

^DooM^ wrote:
polloda wrote:Can I use it for my production server ?
No
Well "at your own risk" may be more appropriate :D

^DooM^
Site Admin
Posts: 13862
Joined: 2005-07-29 16:18
Location: UK

Re: First hMailServer 5.4 alpha available

Post by ^DooM^ » 2010-11-05 09:58

They mean the same thing. If he installs it after I have said No, it's at his own risk if something breaks ;)
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

User avatar
mattg
Moderator
Moderator
Posts: 19897
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: First hMailServer 5.4 alpha available

Post by mattg » 2010-11-05 15:14

I reckon if the question is actually asked, then the answer must be 'no'. :shock:

As an aside, what's with the location in the profile, people?
'Somewhere near Orion's Belt' is interesting. Although I'd bet that it only looks like Orion's belt from about this planet, anywhere else in the universe and it may look like a crescent rather than a straight line...and so may be called 'Orion's beer gut' perhaps...

and -21.10516,46.057649 = in the middle of Madagascar? Surely not ^Doom^... Or are you just wishing that you were there. :)
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: First hMailServer 5.4 alpha available

Post by tBB » 2010-11-05 15:45

mattg wrote:I reckon if the question is actually asked, then the answer must be 'no'. :shock:
I reckon the question was already answered before it was posted 8)
martin wrote:Please do not install this in any production environment

^DooM^
Site Admin
Posts: 13862
Joined: 2005-07-29 16:18
Location: UK

Re: First hMailServer 5.4 alpha available

Post by ^DooM^ » 2010-11-05 17:55

Matt, my location was just me being bored :)
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

User avatar
mattg
Moderator
Moderator
Posts: 19897
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: First hMailServer 5.4 alpha available

Post by mattg » 2010-11-06 09:53

^DooM^ wrote:Matt, my location was just me being bored :)
:mrgreen: :mrgreen:

Tomorrow I'm off to a local pub (170 kms away) for lunch - http://nindigully.com/
Going for a road train burger I reckon
1 KG meat pattie, a side of chips and Onion rings.

Reckon I'll get my cholesterol checked the following week. Perhaps I should do before and after comparisons.
Image
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

^DooM^
Site Admin
Posts: 13862
Joined: 2005-07-29 16:18
Location: UK

Re: First hMailServer 5.4 alpha available

Post by ^DooM^ » 2010-11-06 10:17

wow how the hell can anyone eat that much?
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: First hMailServer 5.4 alpha available

Post by tBB » 2010-11-06 10:58

mattg wrote:Tomorrow I'm off to a local pub (170 kms away) for lunch
I love Australia :lol:

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: First hMailServer 5.4 alpha available

Post by Bill48105 » 2010-11-06 17:35

Holy crap is that a big-arse burger! Why on earth would anyone want to subject themselves to eating that? Just go out & gnaw directly on the cow's rear quarter. lol

Back OT (although I'm now STARVING thanks matt :P lol) I've been working on the 5.4 alpha some more. I believe I have a reasonable fix in place for the thread race condition that was causing the pop3 errors I was getting during download. It took me hours & hours over DAYS to track that down. So far the fix has been working great for me so it's a matter of martin looking it over and perhaps suggesting improvements. I've been running it up on 2 servers here for a week now without problems but definitely needs some testing before going towards beta let alone production but I've been quite happy with the results of the testing I've done..

I got ETRN in. Client & server side although client-side makes use of External account section for now to ease efforts in getting it in & working. Held messages show up in the GUI admin's Status-Queue area as "Route HOLD" for next try and where one can force a retry or remove from queue like other messages in queue if desired. Note that Empty button does not remove HOLD messages unless they are currently being tried for delivery. They would need to be forced retry then use Empty if that was desired.

I got Archiving in which makes a copy of every in/out email in defined folder utilizing hardlinks if enabled which GREATLY reduces load creating the copies if there are multiple recipients. (It sort of follows the hmail Data structure such as C:\Archive\domain1.com\user1 etc. A copy is put in for local sender, a copy in for each local recipient & an 'Incoming' folder when sender is non-local). It is highly recommended the archive folder be managed (such as script zip'ing up & deleting messages from the archive folder & storing off-site for example). For now there are a couple INI settings to enable/control this which will be posted with the update.

I added 3rd return result to OnAcceptMessage script event that allow 4xx response with custom line to allow for soft-rejecting messages if needed.

I added OnSMTPData script event that is called after sender sends DATA which follows OnAcceptMessage result options (there is no message at that time so no access to it obviously..) and allows for rejecting a message BEFORE it is accepted unlike OnAcceptMessage. As with the other script calls care should be taken to be reasonable with its use.. (I originally wanted to have event for MAIL FROM and RCPT TO as well but I am not sure of issues that could arise from such a thing especially if message is to say 100 people & the event is called that many times in succession. I suppose if one was VERY careful to keep the script code in the function reasonable it should be OK but for now the DATA one should suffice as a 3-fer.)

I swear there was more but hey that's all I can recall ATM.

FOOD CALLS! :D
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

SkipFitz
New user
New user
Posts: 27
Joined: 2006-08-03 18:30

Re: First hMailServer 5.4 alpha available

Post by SkipFitz » 2010-11-11 03:31

Even though I can't now find where I read it on this message board, I thought I saw a recent post about how 5.4 will allow for the backup and restore of a single domain.

Was it the LSD? The mushrooms? The case of wine? Or, am I simply missing that option in my 5.4 install?

^DooM^
Site Admin
Posts: 13862
Joined: 2005-07-29 16:18
Location: UK

Re: First hMailServer 5.4 alpha available

Post by ^DooM^ » 2010-11-11 03:33

I remember that too so it's not just you :)
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

SkipFitz
New user
New user
Posts: 27
Joined: 2006-08-03 18:30

Re: First hMailServer 5.4 alpha available

Post by SkipFitz » 2010-11-11 03:50

Oh, good. :D

Off-and-on throughout the day I've been searching for the thread where I read that, but I just haven't had any luck. It's good to know that I'm not entirely crazy. Well, of course, at least not for this reason.

^DooM^
Site Admin
Posts: 13862
Joined: 2005-07-29 16:18
Location: UK

Re: First hMailServer 5.4 alpha available

Post by ^DooM^ » 2010-11-11 04:07

I can't find it either although I have many references to it being added at some point. Ask Bill nicely he may add it for you :)
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

SkipFitz
New user
New user
Posts: 27
Joined: 2006-08-03 18:30

Re: First hMailServer 5.4 alpha available

Post by SkipFitz » 2010-11-11 04:34

Dear Bill48105,

You're a handsome man. And powerful. The ladies? They swoon at the site of your handsometicity and power. Also, if it helps, I''ll re-name my child after you.

:mrgreen:

Seriously, though, I'm not sure if this feature's still on the roadmap for this version, but if it is, I'd love to see it implemented. Mucho appreciation for all the hard work you and martin put into the coding.

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: First hMailServer 5.4 alpha available

Post by Bill48105 » 2010-11-12 14:57

LOL SkipFitz you slay me. :D Thx for the sweet nothings but not sure it'll help. :P My brain is still in another state (as in I was traveling not on shrooms lol) and I don't recall such a thing as backup/restore per domain.. The dbsync tool allows per-domain but think that's been that way awhile, not sure. Is there a feature request for it explaining the benefits? Seems it would be something that could be done in scripting unless you are most concerned with it being in the gui.. Something you likely found already is if you backup (full) using GUI , edit the xml in the zip (such as removing all but one domain) then restore it then you get just that one domain restored *BUT* (and this is a HUGE but - lol you said huge butt!) hmail will erase everything (even data folder meaning all messages) so you lose all the existing domains! Worse there is no warning (or at least was this way as I've not tested recently) which seems could easily cause major grief if not expected or thought out..

Anyhow are you having to restore one domain enough for something like this to be implemented?
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

SkipFitz
New user
New user
Posts: 27
Joined: 2006-08-03 18:30

Re: First hMailServer 5.4 alpha available

Post by SkipFitz » 2010-11-12 17:38

Aha! I blame ^DooM^ :D . In response to a question about a domain-specific backup, he replied with the following:
Not currently. The best thing you can do is copy the whole lot then remove the domains you do not want.

There should be domain specific backup in 5.4
From here. That's where I initially read about the option; sorry it took me such a long time to find it.

Anyway, I'm in a situation where I need to move several domains from an existing production box to another existing production box; however, the target box already has existing domains. This is not an option that'll be needed on an ongoing basis, although I suspect a good case could be made for needing to restore a single domain you accidentally deleted. (Haven't done that. Yet. :D)

It's not a huge deal, and I can do a lot of it through a manual process; I was just excited about the possibility of a single domain backup. If it's not planned, then that's okay. After all, it's not like you don't have enough on your plate as it is.

^DooM^
Site Admin
Posts: 13862
Joined: 2005-07-29 16:18
Location: UK

Re: First hMailServer 5.4 alpha available

Post by ^DooM^ » 2010-11-12 18:05

Yeah I cant see anywhere where it says 5.4 it will be added though, maybe it was in IRC or over PM.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: First hMailServer 5.4 alpha available

Post by Bill48105 » 2010-11-12 18:22

Ahh OK yeah I don't recall ever seeing that thread & I think that was before I even joined the forum. :D If there isn't a feature request for it feel free to add but I have a feeling there isn't huge demand for it although it could prove handy in certain circumstances. If I was merging 2 servers odds are I'd backup both & merge the XML files or some other method like copying the Data tree & export/import of the DB *BUT* I dealing with index counters & UID's could be fun.. But yeah if you're using backup/restore (or any method for that matter) please use tons of care because of the 'clear all' that happens..
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

ObiWan
Senior user
Senior user
Posts: 278
Joined: 2010-07-21 14:30
Location: Halfway between Germany and Egypt

Re: First hMailServer 5.4 alpha available

Post by ObiWan » 2010-12-02 16:06

Since we're at betas (ok, alpha) and the like

Any chance the new 5.4 will implement SRS
(referring to http://www.openspf.org/SRS) ?

I'm asking this since imVHo having SRS would
help solving a number of issues in case one
has mailboxes forwarding to external addresses
or distribution lists containing external addresses

Also, since hMailServer already uses SPF I think
that adding SRS to it would be a logical (and btw
a good) thing

since we're at it; infos/code/stuff about SRS

http://www.libsrs2.org/
http://www.libsrs2.org/srs/srs.pdf
http://ostatic.com/libsrs

[edit]

Notice that here http://www.libsrs2.org/download.html
is available the source code for the "libsrs" library, the
code builds under win32 as well and may be linked as
a static library to whatever C program

spyd3r0x
New user
New user
Posts: 1
Joined: 2010-12-02 21:11

Re: First hMailServer 5.4 alpha available

Post by spyd3r0x » 2010-12-02 21:14

Bill48105 wrote:

Code: Select all

QueueRandomnessMinutes=7
; This key defines how many minutes to use for 'QuickRetries' that over-rides the default retry schedule.
; This does not apply to NEW messages as they are still queued for immediate delivery as before.
; This defaults to 0 or disabled

QuickRetriesMinutes=7
; Defines # of minutes to delay for each retry if QuickRetries is enabled otherwise normal delivery schedule is used
; This defaults to 0 or disabled
I'm a little confused by these two ini settings. The descriptions seems to indicate that they do the same thing (define how long to wait until a 'QuickRetry" happens). Is the description for QueueRandomnessMinutes incorrect, or am I understanding them incorrectly?

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: First hMailServer 5.4 alpha available

Post by Bill48105 » 2010-12-04 07:59

ObiWan wrote:Since we're at betas (ok, alpha) and the like

Any chance the new 5.4 will implement SRS
(referring to http://www.openspf.org/SRS) ?

I'm asking this since imVHo having SRS would
help solving a number of issues in case one
has mailboxes forwarding to external addresses
or distribution lists containing external addresses

Also, since hMailServer already uses SPF I think
that adding SRS to it would be a logical (and btw
a good) thing

since we're at it; infos/code/stuff about SRS

http://www.libsrs2.org/
http://www.libsrs2.org/srs/srs.pdf
http://ostatic.com/libsrs

[edit]

Notice that here http://www.libsrs2.org/download.html
is available the source code for the "libsrs" library, the
code builds under win32 as well and may be linked as
a static library to whatever C program
Hey ObiWan,
SRS is super high on my wish list but besides tracking down the buffer/iocp errors and some cleanup martin really wants to freeze the code to get 5.4 beta then production asap. Once that happens we'll likely discuss things to work on next for 5.4.x or such. Thx for the info though, handy to have it all in one place.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: First hMailServer 5.4 alpha available

Post by Bill48105 » 2010-12-04 08:19

spyd3r0x wrote:
Bill48105 wrote:

Code: Select all

QueueRandomnessMinutes=7
; This key defines how many minutes to use for 'QuickRetries' that over-rides the default retry schedule.
; This does not apply to NEW messages as they are still queued for immediate delivery as before.
; This defaults to 0 or disabled

QuickRetriesMinutes=7
; Defines # of minutes to delay for each retry if QuickRetries is enabled otherwise normal delivery schedule is used
; This defaults to 0 or disabled
I'm a little confused by these two ini settings. The descriptions seems to indicate that they do the same thing (define how long to wait until a 'QuickRetry" happens). Is the description for QueueRandomnessMinutes incorrect, or am I understanding them incorrectly?
Hey spyd3r0x,
Yeah unfortunately I forgot the one then could no longer edit it & I wasn't too clear to the mods on what to put so I'll clarify here.

Probably something like:

Code: Select all

QueueRandomnessMinutes=7
; This value defines random # of minutes to be added to the # of minutes to wait before the next retry
; between 0 & the defined value. Applies to all retries if enabled, normal & "Quick".  Defaults to 0 or disabled
Don't get too attached to the INI settings.. Not sure how long they'll be there or if those features will even stay in the official builds. That being said I will likely have them in my builds no matter what because the whole reason I added them was to fill my needs & figuring others could benefit from them as well. So we'll see, hopefully the features stay but just get added to the GUI/admin perhaps with different names & better descriptions. :D

To answer your question though those settings are similar but definitely different. Examples would likely help the best:

Let's say you have normal retries in the admin set to 10 tries with delay of 60 minutes. If you send an email to a server that delays/greylists your message it will be stuck there for an hour before it is tried again.. The only workaround was to set the default delay to like 10 minutes but then the problem was ALL messages were retried that often causing a LOT more traffic & tries of messages where the server was clearly down or having issues. (If the emails were going to a known domain you could also get around by setting up a Route & defining retries/minutes in there but not ideal way to deal with it..) In comes QuickRetries.. If that is set to 3 for example, for the 1st 3 tries the delay is not the system-wide default (60 minutes in my example) but instead is # you set for QuickRetriesMinutes.. So if QuickRetriesMinutes=7 hmail will try (possibly get greylisted), wait 7 min, try again, wait 7 min, try again, wait 7 min, try again, wait 60 min, try again, wait 60 min, etc until max tries is reached. In essence it makes for a 2 tiered retry delay where the 1st X tries are at the lower delay then the normal delay is resumed for the rest.

QueueRandomnessMinutes extends onto ALL retries, not just the "Quick" retries.. What happens is whenever any email fails & is put back in the queue a random # of minutes between 0 & the value of QueueRandomnessMinutes is chosen by hmail. So if QueueRandomnessMinutes=10 then the above QuickRetries example might have instead looked like:
Try (greylisted), wait 8 minutes, try, wait 17 minutes, try, wait 12 minutes, try, wait 9 minutes, try, wait 61 minutes, try, wait 69 minutes, etc until max tries is reached. (Image if 10 emails to yahoo were sent at once & yahoo greylisted them all or was having issues so they all had to be retried. Without the randomness factor added they would all essentially be tried at the same time every time or nearly the same time but with the randomness added they are less likely to occur at the same time spreading the workload and creating "gaps" in the queue for new/other messages to be placed & hopefully delivered more quickly if the queue threads aren't all tied up trying to deliver those 10 stuck yahoo messages at the same time..)

As you can see QuickRetries helps with greylisting while allowing you to set your default delay higher reducing your server load yet retrying messages quicker the 1st few times where QueueRandomnessMinutes helps stagger the messages out further spreading out the server workload.. I have been using an updated 5.4 on my server and those settings help a lot and hopefully others agree they are useful. (They are advanced settings many people may not even need to worry about & that is why they default to being disabled but anyone with a busy server should appreciate the extra control it gives.)
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

teco
Normal user
Normal user
Posts: 35
Joined: 2010-12-20 14:26

Re: First hMailServer 5.4 alpha available

Post by teco » 2010-12-20 17:23

Just a short question:
Alpha Version is available since a few weeks. Any estimation when Beta 1 can be await?

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: First hMailServer 5.4 alpha available

Post by Bill48105 » 2010-12-20 21:06

Hoping soon teco. Think it was partly waiting to see response to 5.3.4 beta since that had the big buffer/iocp code changes in it that are also in 5.4. martin has been cleaning up some stuff I added/changed to his liking as well as updating some libraries etc.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

snico
New user
New user
Posts: 27
Joined: 2010-01-04 12:18

Re: First hMailServer 5.4 alpha available

Post by snico » 2010-12-28 15:58

martin wrote: [*] hMailServer no longer stores the full path to message files in the database.
Hello Martin

I have my mail on SSD disk 80Gb in D:\MAILS
Each night, i move some big mail >10Mo on SATA E:\MAILS

If hMailServer no longer stores the full path to message files in the database, this will not work anymore?


Thank you.
Nicolas Cavaré
hMailServer 5.4.1 B1951 external MSSQL 2008 r2
Windows 2008 r2 | 2 x Intel Xeon E5-2650v2 | 64 Go DDR3 | 2 x Intel SSD 600go

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: First hMailServer 5.4 alpha available

Post by Bill48105 » 2010-12-28 18:39

Hi Nicolas.
So you move the files then edit the path in the database so the mail is available?

Since 5.4 now rebuilds the path on the fly based on the base Data folder & JUST the filename stoed in the database vs having entire path stored in the database I'd say that would no longer be possible..
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

snico
New user
New user
Posts: 27
Joined: 2010-01-04 12:18

Re: First hMailServer 5.4 alpha available

Post by snico » 2010-12-28 19:26

Hi!

yes, i edit the path in the database!

It is possible to allow to have relative and abolute path in the same database in version 5.4 ?
Nicolas Cavaré
hMailServer 5.4.1 B1951 external MSSQL 2008 r2
Windows 2008 r2 | 2 x Intel Xeon E5-2650v2 | 64 Go DDR3 | 2 x Intel SSD 600go

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: First hMailServer 5.4 alpha available

Post by Bill48105 » 2010-12-28 22:29

Nicolas,
Based on what I've seen in 5.4 it seems very unlikely.. A lot was changed for the relative paths to work & although I've not seen 5.3.x source I suspect there is no turning back.. Maybe martin will pop in at some point & confirm/deny that but I have a feeling you'll be stuck in 5.3.x land or will need to keep all mail in the same Data folder..
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

snico
New user
New user
Posts: 27
Joined: 2010-01-04 12:18

Re: First hMailServer 5.4 alpha available

Post by snico » 2010-12-28 23:21

Ok i will kepp all mail in the same folder.

I have a second solution.

I will try to make a script to replace attachement by a link to a webserver.

So i can kepp all my Mail on a SSD disk 80GB. :D
Nicolas Cavaré
hMailServer 5.4.1 B1951 external MSSQL 2008 r2
Windows 2008 r2 | 2 x Intel Xeon E5-2650v2 | 64 Go DDR3 | 2 x Intel SSD 600go

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: First hMailServer 5.4 alpha available

Post by Bill48105 » 2010-12-28 23:50

snico wrote:Ok i will kepp all mail in the same folder.

I have a second solution.

I will try to make a script to replace attachement by a link to a webserver.

So i can kepp all my Mail on a SSD disk 80GB. :D
Are you using SSD for speed or ?? It seems hard to believe hmailserver serves up mail any faster from SSD vs normal hard drive.. I mean most everything is in database until point email is sent to client & that would be most likely limited by upload speed which would be WAYYY slower than any modern hard drive.. Granted some email access where message data is read to send to the client separate from the message itself (like webmail doing server-side sort for example although that can be helped by enabling indexing in hmail) might benefit but I'm guessing most normal email access wouldn't benefit.

Have you done performance testing to compare or just assuming hmailserver benefits from Data folder being on SSD? Just wondering if it is worth all the trouble moving mail or attachments to another drive vs just having Data folder on normal HDD & perhaps just database on SSD.. Benchmarks showing differences might be interesting.

Anyway, suppose we're off topic at this point & perhaps best discussed in a new thread. :)
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

rjk
Normal user
Normal user
Posts: 248
Joined: 2010-03-30 19:30
Location: uʍop ǝpısdn

Re: First hMailServer 5.4 alpha available

Post by rjk » 2011-01-04 16:52

I unfortunately made the stupid mistake of upgrading a production server to 5.4 - and as far as I know there isn't any going back. I could restore a backup, but I thought I ought to post the errors I am getting to see if it could be beneficial to the devs first.

The hMailServer service appears to hang periodically, and eat up 100% of one CPU, (25% of my quad core server). The following errors have appeared in my logs at various points:

Code: Select all

"ERROR"	5592	"2011-01-04 08:35:59.915"	"Severity: 2 (High), Code: HM5339, Source: ByteBuffer::~ByteBuffer, Description: Error when emptying buffer. This: 176340016"
"ERROR"	6176	"2011-01-04 08:35:59.915"	"Severity: 2 (High), Code: HM5339, Source: ByteBuffer::~ByteBuffer, Description: Error when emptying buffer. This: 169162064"
"ERROR"	5592	"2011-01-04 08:35:59.915"	"Severity: 2 (High), Code: HM5339, Source: TCPConnection::HandleWrite, Description: An unknown error occurred while handling buffer write and notifying protocol parser that data was sent."
"ERROR"	6176	"2011-01-04 08:35:59.915"	"Severity: 3 (Medium), Code: HM5136, Source: TCPConnection::HandleRead, Description: An error occured while parsing data. Data length: 12, Data: 354 go ahead. Remote IP: XXX.XXX.XXX.XXX"
"ERROR"	5592	"2011-01-04 08:35:59.915"	"Severity: 2 (High), Code: HM5339, Source: TCPConnection::HandleWrite, Description: An unknown error occurred while handling buffer write. Session ID: 5689, Transferred bytes: 120002"
"ERROR"	5592	"2011-01-04 08:35:59.915"	"Severity: 2 (High), Code: HM4208, Source: IOCPQueueWorkerTask::DoWork, Description: An unknown error occured while handling asynchronous requests."
"ERROR"	3352	"2011-01-04 08:36:01.257"	"Severity: 2 (High), Code: HM5339, Source: ByteBuffer::~ByteBuffer, Description: Error when emptying buffer. This: 176775568"
"ERROR"	3352	"2011-01-04 08:36:01.257"	"Severity: 2 (High), Code: HM5339, Source: TCPConnection::HandleWrite, Description: An unknown error occurred while handling buffer write and popping queue."
"ERROR"	3352	"2011-01-04 08:36:01.257"	"Severity: 2 (High), Code: HM5339, Source: TCPConnection::HandleWrite, Description: An unknown error occurred while handling buffer write. Session ID: 5689, Transferred bytes: 59982"

Code: Select all

"ERROR"	5048	"2011-01-03 19:21:25.552"	"Severity: 3 (Medium), Code: HM5136, Source: TCPConnection::HandleRead, Description: An error occured while parsing data. Data length: 40, Data: 354 continue.  finished with "\r\n.\r\n". Remote IP: XXX.XXX.XXX.XXX"
Not to mention these too:

Code: Select all

"ERROR"	4180	"2011-01-03 08:42:57.959"	"Severity: 2 (High), Code: HM4208, Source: IOCPQueueWorkerTask::DoWork, Description: An unknown error occured while handling asynchronous requests."
"ERROR"	6168	"2011-01-03 08:42:58.117"	"Severity: 2 (High), Code: HM5339, Source: ByteBuffer::~ByteBuffer, Description: Error when emptying buffer. This: 11775920"
"ERROR"	6168	"2011-01-03 08:42:58.117"	"Severity: 2 (High), Code: HM5339, Source: TCPConnection::HandleWrite, Description: An unknown error occurred while handling buffer write and popping queue."
"ERROR"	6168	"2011-01-03 08:42:58.118"	"Severity: 2 (High), Code: HM5339, Source: TCPConnection::HandleWrite, Description: An unknown error occurred while handling buffer write. Session ID: 2, Transferred bytes: 59967"
"ERROR"	6168	"2011-01-03 08:42:58.118"	"Severity: 2 (High), Code: HM5339, Source: ByteBuffer::~ByteBuffer, Description: Error when emptying buffer. This: 11778776"
"APPLICATION"	3952	"2011-01-03 08:42:58.118"	"SMTPDeliverer - Message 11387: Message could not be delivered. Scheduling it for later delivery in 30 minutes.nn"
"ERROR"	6168	"2011-01-03 08:42:58.119"	"Severity: 2 (High), Code: HM5339, Source: ByteBuffer::~ByteBuffer, Description: Error when emptying buffer. This: 134127912"
"ERROR"	6168	"2011-01-03 08:42:58.120"	"Severity: 2 (High), Code: HM4208, Source: IOCPQueueWorkerTask::DoWork, Description: An unknown error occured while handling asynchronous requests."
"APPLICATION"	3952	"2011-01-03 08:42:58.122"	"SMTPDeliverer - Message 11387: Message delivery thread completed."
"APPLICATION"	6032	"2011-01-03 08:43:05.790"	"OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"APPLICATION"	5736	"2011-01-03 08:43:13.401"	"OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"APPLICATION"	5736	"2011-01-03 08:43:13.402"	"OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"APPLICATION"	5736	"2011-01-03 08:43:13.402"	"OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"APPLICATION"	5736	"2011-01-03 08:43:13.402"	"OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"APPLICATION"	5736	"2011-01-03 08:43:22.808"	"OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"APPLICATION"	5736	"2011-01-03 08:43:22.809"	"OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"APPLICATION"	5736	"2011-01-03 08:43:22.809"	"OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"APPLICATION"	5736	"2011-01-03 08:43:22.809"	"OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"APPLICATION"	6468	"2011-01-03 08:43:38.942"	"OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"APPLICATION"	6032	"2011-01-03 08:43:41.372"	"OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"APPLICATION"	5736	"2011-01-03 08:43:55.333"	"OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"APPLICATION"	5736	"2011-01-03 08:43:55.333"	"OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"APPLICATION"	5736	"2011-01-03 08:43:55.334"	"OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"APPLICATION"	5736	"2011-01-03 08:43:55.334"	"OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"APPLICATION"	6812	"2011-01-03 08:44:06.634"	"OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"APPLICATION"	6812	"2011-01-03 08:44:20.683"	"OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
Hope this helps you guys. Killing and starting the service again seems to work. If you need debug logs, let me know and I will enable them.

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: First hMailServer 5.4 alpha available

Post by Bill48105 » 2011-01-04 20:36

rjk,
Is that using the old 5.4 alpha or the updated one that martin posted recently? (Don't be confused by the fact the build # dropped. As noted that was a mistake so go by the date of the build for that one not the build #)
Thx
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

rjk
Normal user
Normal user
Posts: 248
Joined: 2010-03-30 19:30
Location: uʍop ǝpısdn

Re: First hMailServer 5.4 alpha available

Post by rjk » 2011-01-04 21:03

Bill48105 wrote:rjk,
Is that using the old 5.4 alpha or the updated one that martin posted recently? (Don't be confused by the fact the build # dropped. As noted that was a mistake so go by the date of the build for that one not the build #)
Thx
Bill
Installed B1915.
I assume that the furor over messed up build numbers was only related to B1913 and B1914.

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: First hMailServer 5.4 alpha available

Post by Bill48105 » 2011-01-04 22:05

rjk wrote:Installed B1915.
I assume that the furor over messed up build numbers was only related to B1913 and B1914.
Oh OK sorry I hadn't realized martin had posted up 1915 which should be latest & likely was just build # update by martin to avoid confusion..

I've been running a custom built version of 5.4 here for 2 months now on a server that is quite busy & have yet to run into that or any other problems/crashes.. Granted it doesn't have the iocp/buffer code changes that martin put in recently but I never had the issues to begin with & honestly I've not been in a real hurry to updated my main server that is working so well. :D

So how long does hmailserver need to be running before that happens? Could you please post up your performance settings & give us an idea of how busy your server is? (how many pop/imap clients & about how many messages it handles for smtp/pop/imap, cpu/memory usage normally etc) The bummer is we thought that issue was licked because we could no longer reproduce but maybe the timing was just shifted or something. Hopefully the additional logging that martin added will help narrow it down further..
Thx
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

rjk
Normal user
Normal user
Posts: 248
Joined: 2010-03-30 19:30
Location: uʍop ǝpısdn

Re: First hMailServer 5.4 alpha available

Post by rjk » 2011-01-04 22:30

Bill48105 wrote:
rjk wrote:Installed B1915.
I assume that the furor over messed up build numbers was only related to B1913 and B1914.
Oh OK sorry I hadn't realized martin had posted up 1915 which should be latest & likely was just build # update by martin to avoid confusion..

I've been running a custom built version of 5.4 here for 2 months now on a server that is quite busy & have yet to run into that or any other problems/crashes.. Granted it doesn't have the iocp/buffer code changes that martin put in recently but I never had the issues to begin with & honestly I've not been in a real hurry to updated my main server that is working so well. :D

So how long does hmailserver need to be running before that happens? Could you please post up your performance settings & give us an idea of how busy your server is? (how many pop/imap clients & about how many messages it handles for smtp/pop/imap, cpu/memory usage normally etc) The bummer is we thought that issue was licked because we could no longer reproduce but maybe the timing was just shifted or something. Hopefully the additional logging that martin added will help narrow it down further..
Thx
Bill
OK I will enable debug logs etc.
Users: about 20 POP users no IMAP. So far it has not lasted more than about 24 hours on the new build, seems to leak memory ever so slightly then hang. Volume, about 50-100 mails per day. Memory usage, not sure - probably around 8,000k normally. CPU usage never more than a few %, however now it seems to use a little more - however that might be because of ClamAV which I implemented not too long ago (but before upgrading to 5.4 though) I haven't monitored too closely previous to this. If its any help, hitting STOP from the Admin console hangs the whole works.

Hmm odd now that I look it appears that someone is using IMAP, wasn't aware of setting any of my users up with it. I have kept it enabled for webmail, but perhaps I should disable external access. Appears to be a blackberry - they support POP as far as I know though.

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: First hMailServer 5.4 alpha available

Post by Bill48105 » 2011-01-04 22:43

rjk wrote:OK I will enable debug logs etc.
Users: about 20 POP users no IMAP. So far it has not lasted more than about 24 hours on the new build, seems to leak memory ever so slightly then hang. Volume, about 50-100 mails per day. Memory usage, not sure - probably around 8,000k normally. CPU usage never more than a few %, however now it seems to use a little more - however that might be because of ClamAV which I implemented not too long ago (but before upgrading to 5.4 though) I haven't monitored too closely previous to this. If its any help, hitting STOP from the Admin console hangs the whole works.

Hmm odd now that I look it appears that someone is using IMAP, wasn't aware of setting any of my users up with it. I have kept it enabled for webmail, but perhaps I should disable external access. Appears to be a blackberry - they support POP as far as I know though.
Wow your usage seems very low, especially compared to mine here without issues.. (My server handles about 10,000 messages per day on average to give you an idea)

The 1st thing you might try is disabling clam.. I realize that is not ideal but worth trying just to see. I use 5.4's clamd client (not the command-line av or clamwin options in hmail) connecting to another box running clamd on centos. So not sure if you mean you are doing the new 5.4 clamd client stuff or running clam client exe but either way it is worth trying with that disabled & we can go from there.

Once hmailserver service is hanging I would not try using anything in the GUI/admin.. Just close it then restart the service.

I have about 20 users or so who use Roundcube webmail a lot via IMAP. (for many it is their only email client & for others it is just when away from their desk). Other than RoundCube I am the only one who uses IMAP (via IP ranges with IMAP disabled for internet range but enabled for rangers for web server running webmail & my IP) but just 1 blackberry user on imap should not cause you grief.

Besides disabling clam av I'd focus on your performance settings to start with.. In my tests setting cpu priority to lowest helped a lot but again it is a bummer as I thought this was licked.

Btw if you are familiar with SQL (which db you use btw?) it isn't too difficult to change db version back & install 5.3.x again if needed.. Obviously care needs to be taken (including backups & recommend disabling access to the server while working on it to be safe) but should be possible at least if that is the route you care to go.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

rjk
Normal user
Normal user
Posts: 248
Joined: 2010-03-30 19:30
Location: uʍop ǝpısdn

Re: First hMailServer 5.4 alpha available

Post by rjk » 2011-01-04 23:08

Bill48105 wrote:Wow your usage seems very low, especially compared to mine here without issues.. (My server handles about 10,000 messages per day on average to give you an idea)

The 1st thing you might try is disabling clam.. I realize that is not ideal but worth trying just to see. I use 5.4's clamd client (not the command-line av or clamwin options in hmail) connecting to another box running clamd on centos. So not sure if you mean you are doing the new 5.4 clamd client stuff or running clam client exe but either way it is worth trying with that disabled & we can go from there.

Once hmailserver service is hanging I would not try using anything in the GUI/admin.. Just close it then restart the service.

I have about 20 users or so who use Roundcube webmail a lot via IMAP. (for many it is their only email client & for others it is just when away from their desk). Other than RoundCube I am the only one who uses IMAP (via IP ranges with IMAP disabled for internet range but enabled for rangers for web server running webmail & my IP) but just 1 blackberry user on imap should not cause you grief.

Besides disabling clam av I'd focus on your performance settings to start with.. In my tests setting cpu priority to lowest helped a lot but again it is a bummer as I thought this was licked.

Btw if you are familiar with SQL (which db you use btw?) it isn't too difficult to change db version back & install 5.3.x again if needed.. Obviously care needs to be taken (including backups & recommend disabling access to the server while working on it to be safe) but should be possible at least if that is the route you care to go.
Bill
I use Clam via command line, and it has worked for a few weeks on hmail v5.3. In the GUI/Admin console, hitting stop causes a hang, regardless of whether it was hung previously. (In 5.4 not 5.3)
My Priority setting has been at Above Normal ever since I installed hmail several months ago. The server is dedicated to mail, RDP and web (IIS), and as you noted is extremely low traffic. Command threads is at 30, Delivery Threads is at 25, and async task threads is at 30. Indexing enabled. Cache hit rates for domain says 75, Account 9, others 0. Since my last post, the memory usage has crept slowly up from 9900k to 11000k, after being at about 8000k for most of the morning after a service restart from a hang.
I am using MySQL, but I really am not familiar with SQL enough to make a query. Can run a script though. Everything is running on Server 2008 R2 on two dual core procs and 4 GB RAM. Old Dell PE 2850 machine. Data directory 700-800 MB. Memory usage usually 2gb to spare.

rjk
Normal user
Normal user
Posts: 248
Joined: 2010-03-30 19:30
Location: uʍop ǝpısdn

Re: First hMailServer 5.4 alpha available

Post by rjk » 2011-01-05 00:06

The memory usage has doubled to 16000k+ now, and the logs are producing all sorts of interesting things. Hasn't crashed, but when it does, shall I PM the logs to you Bill, or Martin or both? I dunno if there is an automated scrubber anywhere, but I presume you aren't evil :)

Hmm, it hasn't hung like it did previously, but it is refusing connections - or should I say, connections are timing out with no response not even a reset. But no high CPU usage.

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: First hMailServer 5.4 alpha available

Post by Bill48105 » 2011-01-05 00:46

rjk,
Yeah well that is good & all but at this point I assume you want it to work then worry about optimizing right? ;) IOW what worked before may not work now so unless you can go back to what you had before you need to worry about getting what you have now working..

If I was to make a suggestion on how to try & get a working set at this point I'd disable clam (keep attachment stripping & end-user client AV updated & warn users to be extra careful if needed), set priority to below normal, threads to 15, 10, 5. Turn off indexing (i realize webmail will suffer but again you want a working base then try adjusting/re-enabling), set max connections for SMTP/POP/IMAP to 20 (I realize you will likely have blocked connections but again trying to establish working set & then work your settings back up). ** MAKE NOTE OF ALL CURRENT SETTINGS BEFORE CHANGING THOUGH! Heck do a backup even to be safe..

With those settings see if the problem still happen or happen less frequently. If it seems fine then you can try enabling/increasing things one at a time & waiting/testing in between.

Not sure why stop locks up hmailserver.. I use it all the time with 5.4 here.. Either you have some very strange setup/config or martin has introduce some very bad bug.

My $ on mem & lockup issue is you running clam exe.. Is hmailserver.exe using all the mem or some other exe's like clam? My second guess for memory issue would be indexing as that is a MAJOR resource beast. IMAP is too and add the 2 together & you'll see 100's of MB's of memory fluctuations..

You are welcome to PM me logs (I won't use for evil purposes MUWAHHHH!!) but normally martin asks they be sent to martin@hmailserver.com with link to relevant thread.

Btw, in the event you want to try & downgrade we can try helping you with that but odds are it'd be easiest if you dropped into IRC so we could talk about it real-time.

Also I'm happy to share my build of 5.4 that I'm using live on my server but keep in mind it is from November so it does not have all the recent updates martin put in, particularly the iocp/buffer changes but then again it works on mine so perhaps it'd work on yours & get you by until martin is able to work out if there are issues in current build..
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

rjk
Normal user
Normal user
Posts: 248
Joined: 2010-03-30 19:30
Location: uʍop ǝpısdn

Re: First hMailServer 5.4 alpha available

Post by rjk » 2011-01-05 01:10

Bill48105 wrote:rjk,
Yeah well that is good & all but at this point I assume you want it to work then worry about optimizing right? ;) IOW what worked before may not work now so unless you can go back to what you had before you need to worry about getting what you have now working..

If I was to make a suggestion on how to try & get a working set at this point I'd disable clam (keep attachment stripping & end-user client AV updated & warn users to be extra careful if needed), set priority to below normal, threads to 15, 10, 5. Turn off indexing (i realize webmail will suffer but again you want a working base then try adjusting/re-enabling), set max connections for SMTP/POP/IMAP to 20 (I realize you will likely have blocked connections but again trying to establish working set & then work your settings back up). ** MAKE NOTE OF ALL CURRENT SETTINGS BEFORE CHANGING THOUGH! Heck do a backup even to be safe..

With those settings see if the problem still happen or happen less frequently. If it seems fine then you can try enabling/increasing things one at a time & waiting/testing in between.

Not sure why stop locks up hmailserver.. I use it all the time with 5.4 here.. Either you have some very strange setup/config or martin has introduce some very bad bug.

My $ on mem & lockup issue is you running clam exe.. Is hmailserver.exe using all the mem or some other exe's like clam? My second guess for memory issue would be indexing as that is a MAJOR resource beast. IMAP is too and add the 2 together & you'll see 100's of MB's of memory fluctuations..

You are welcome to PM me logs (I won't use for evil purposes MUWAHHHH!!) but normally martin asks they be sent to martin@hmailserver.com with link to relevant thread.

Btw, in the event you want to try & downgrade we can try helping you with that but odds are it'd be easiest if you dropped into IRC so we could talk about it real-time.

Also I'm happy to share my build of 5.4 that I'm using live on my server but keep in mind it is from November so it does not have all the recent updates martin put in, particularly the iocp/buffer changes but then again it works on mine so perhaps it'd work on yours & get you by until martin is able to work out if there are issues in current build..
Bill
Not an enormous issue because it is a family/personal setup just trying to help you guys iron out bugs.
Clam only runs long enough to scan the files it needs to, then exits. hMailServer is using hardly any memory at all, the only reason I mentioned it was because it is higher than it normally is. (16 MB instead of 8 MB)
I have made all suggested changes and will see how it goes. will leave debug logging enabled but disable other logs except application.

User avatar
mattg
Moderator
Moderator
Posts: 19897
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: First hMailServer 5.4 alpha available

Post by mattg » 2011-01-05 01:25

rjk wrote:...Users: about 20 POP users no IMAP. So far it has not lasted more than about 24 hours on the new build, seems to leak memory ever so slightly then hang. Volume, about 50-100 mails per day. Memory usage, not sure - probably around 8,000k normally. CPU usage never more than a few %, however now it seems to use a little more - however that might be because of ClamAV which I implemented not too long ago (but before upgrading to 5.4 though) I haven't monitored too closely previous to this. If its any help, hitting STOP from the Admin console hangs the whole works.
Just to be clear, rjk you are using ClamAV and not ClamWIN is that correct?
Which variant of ClamAV?
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

rjk
Normal user
Normal user
Posts: 248
Joined: 2010-03-30 19:30
Location: uʍop ǝpısdn

Re: First hMailServer 5.4 alpha available

Post by rjk » 2011-01-05 01:37

mattg wrote:Just to be clear, rjk you are using ClamAV and not ClamWIN is that correct?
Which variant of ClamAV?
"ClamAV 0.96.5 - Win32 [tBB]"
I see freshclam.exe running all the time, but clamdscan.exe appears only occasionally - when scanning an attachment. Configured according to a (sticky?) thread somewhere around here.

Post Reply