hmailserver hang

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
User avatar
Conzi
Normal user
Normal user
Posts: 147
Joined: 2006-02-24 14:44
Location: Varese, Italy

hmailserver hang

Post by Conzi » 2010-10-12 19:52

today my hmailserver crashed (1 month from the last time)
the server stopped to answer to imap, pop and smtp
But today I have a full debug log.

These are the errors:

"ERROR" 4604 "2010-10-12 17:08:23.750" "Severity: 4 (Low), Code: HM5137, Source: TCPConnection::PostTimeout, Description: An error occured while sending a timeout message to the client. Remote IP: , Error code: 10009, Message: Handle di file fornito non valido"
"ERROR" 7704 "2010-10-12 17:54:13.265" "Severity: 2 (High), Code: HM5339, Source: TCPConnection::HandleWrite, Description: An unknown error occurred while handling buffer write."
"ERROR" 7960 "2010-10-12 17:54:13.281" "Severity: 2 (High), Code: HM5339, Source: TCPConnection::HandleWrite, Description: An unknown error occurred while handling buffer write."
"ERROR" 8052 "2010-10-12 17:54:13.281" "Severity: 2 (High), Code: HM5339, Source: TCPConnection::HandleWrite, Description: An unknown error occurred while handling buffer write."
"ERROR" 1780 "2010-10-12 17:54:13.296" "Severity: 2 (High), Code: HM4208, Source: IOCPQueueWorkerTask::DoWork, Description: An unknown error occured while handling asynchronous requests."
"ERROR" 7960 "2010-10-12 17:54:13.296" "Severity: 2 (High), Code: HM4208, Source: IOCPQueueWorkerTask::DoWork, Description: An unknown error occured while handling asynchronous requests."
"ERROR" 7704 "2010-10-12 17:54:13.296" "Severity: 2 (High), Code: HM4208, Source: IOCPQueueWorkerTask::DoWork, Description: An unknown error occured while handling asynchronous requests."
"ERROR" 8052 "2010-10-12 17:54:13.296" "Severity: 2 (High), Code: HM4208, Source: IOCPQueueWorkerTask::DoWork, Description: An unknown error occured while handling asynchronous requests."
"ERROR" 5056 "2010-10-12 17:54:14.156" "Severity: 2 (High), Code: HM4208, Source: IOCPQueueWorkerTask::DoWork, Description: An unknown error occured while handling asynchronous requests."
"ERROR" 7036 "2010-10-12 17:54:14.156" "Severity: 2 (High), Code: HM4208, Source: IOCPQueueWorkerTask::DoWork, Description: An unknown error occured while handling asynchronous requests."
"ERROR" 5188 "2010-10-12 17:54:14.203" "Severity: 2 (High), Code: HM4208, Source: IOCPQueueWorkerTask::DoWork, Description: An unknown error occured while handling asynchronous requests."
"ERROR" 6152 "2010-10-12 17:54:14.250" "Severity: 2 (High), Code: HM4208, Source: IOCPQueueWorkerTask::DoWork, Description: An unknown error occured while handling asynchronous requests."
"ERROR" 6252 "2010-10-12 17:54:15.187" "Severity: 2 (High), Code: HM4208, Source: IOCPQueueWorkerTask::DoWork, Description: An unknown error occured while handling asynchronous requests."
"ERROR" 6252 "2010-10-12 17:54:17.328" "Severity: 2 (High), Code: HM4208, Source: IOCPQueueWorkerTask::DoWork, Description: An unknown error occured while handling asynchronous requests."
"ERROR" 6252 "2010-10-12 17:54:27.156" "Severity: 2 (High), Code: HM4208, Source: IOCPQueueWorkerTask::DoWork, Description: An unknown error occured while handling asynchronous requests."
"ERROR" 6252 "2010-10-12 17:54:29.359" "Severity: 2 (High), Code: HM4208, Source: IOCPQueueWorkerTask::DoWork, Description: An unknown error occured while handling asynchronous requests."
"ERROR" 204 "2010-10-12 18:24:41.046" "Severity: 4 (Low), Code: HM5137, Source: TCPConnection::PostTimeout, Description: An error occured while sending a timeout message to the client. Remote IP: , Error code: 10009, Message: Handle di file fornito non valido"
"ERROR" 8172 "2010-10-12 18:25:11.796" "Severity: 4 (Low), Code: HM5137, Source: TCPConnection::PostTimeout, Description: An error occured while sending a timeout message to the client. Remote IP: , Error code: 10009, Message: Handle di file fornito non valido"
"ERROR" 4044 "2010-10-12 18:37:46.609" "Severity: 3 (Medium), Code: HM5125, Source: IMAPFolders::Refresh(), Description: Unable to retrieve folder list for account. Folder recursion not parseable. Account: 94, Parent: -1"


restart server (restar hmailserver service don't work, I have killed the process) at 18:36

some other info:
56 domain and ~185 user
22 imap connection at crash time
Mysql community 5.1.33 (xampp)
~60MB ram usage at crash time (hmailserver.exe)
server: xeon E5405 with 2GB ram
windows server 2003R2 sp2 (xen vps)
hmail versione: 5.3.3B1846
no spamassassin
no antivirus (only for these days)

Here the full log.
http://www.zonaweb.it/hmail_log_12102010.7z
(the file is password protected, if someone wants to view it I can send to him the password in PM)

thanks to those who want to help me....

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: hmailserver hang

Post by ^DooM^ » 2010-10-13 00:34

Have you tried repairing the SQL database tables?
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

User avatar
Conzi
Normal user
Normal user
Posts: 147
Joined: 2006-02-24 14:44
Location: Varese, Italy

Re: hmailserver hang

Post by Conzi » 2010-10-13 00:51

Hi ^DooM^
SQL tables are not corrupt .... after restart the server works properly ...

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: hmailserver hang

Post by Bill48105 » 2010-10-13 01:34

Hey Conzi,
Unfortunately we've seen that a few times & not sure anyone has figured out exactly what is going on.. Looking at the code with your logs it looks like there is a TCP timeout that occurs (remote end drop? Router/ISP failure? Firewall/AV block? etc) and then all hell breaks loose with errors throwing left & right until seems it stops accepting connections. Did you happen to do a netstat -a at the time to see if there were too many connections & all your ports were used up? I've seen that happen with TIME_WAIT etc as if it was DoS attack effect. Any event log entries?

I'll download your logs & see if I can figure anything out from it otherwise martin might have questions/comments to try & track it down. Might need some more debug code added to track it or better log what happened to figure out the fix. For now you might use a monitor system like monit or such so you are at least aware of the issue & can act on it if needed. Would love to know what netstat -a showed at the time so if it happens again please be sure to do that.
Thx
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: hmailserver hang

Post by Bill48105 » 2010-10-13 02:50

Checked out your logs. Wow you have a busy server, especially IMAP. It was an SMTP session that was going when the error you posted happened although as I said a lot was going on no doubt. It looks like all TCP communication was stopped at the time of the event and eventually all connections timed out. Question is what might cause that.. First guess would be firewall but let me guess you have no firewall software on the server? Second guess is you have so many connections open that Windows IP stack has crashed. netstat might help but you should have event log entries..

I'll spend some time comparing your logs to the source code & see if I can come up with anything. In the meantime please check out netstat -a and see how many connections you have, especially TIME_WAIT ones.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

User avatar
Conzi
Normal user
Normal user
Posts: 147
Joined: 2006-02-24 14:44
Location: Varese, Italy

Re: hmailserver hang

Post by Conzi » 2010-10-13 11:44

Nothing in the windows event log...
hmailserver isn't crashed, simply has stopped to work.
Today I have changed the smtp max simultaneous connections from 0 to 20.

User avatar
katip
Senior user
Senior user
Posts: 779
Joined: 2006-12-22 07:58
Location: Istanbul

Re: hmailserver hang

Post by katip » 2010-10-13 21:17

5.3.3 b.1879, W2K3, MySQL, 2 GB RAM, ~50 users 10K mails/day
same thing happened to me yesterday, first time (once again) since 3 months or so. HMS runs behind ASSP which works perfectly as itself, so SMTP session limits are under full control and antispam + AV is done at ASSP level, all on the same box. there was nothing extraordinary in both ASSP and HMS logs as it happened:

"ERROR" 3228 "2010-10-12 17:04:26.765" "Severity: 3 (Medium), Code: HM5136, Source: TCPConnection::HandleRead, Description: An error occured while parsing data. Data length: 7, Data: RETR 45. Remote IP: xx.xx.xx.xx"
"ERROR" 3268 "2010-10-12 17:04:26.765" "Severity: 2 (High), Code: HM5339, Source: TCPConnection::HandleWrite, Description: An unknown error occurred while handling buffer write."
"ERROR" 3268 "2010-10-12 17:04:26.781" "Severity: 2 (High), Code: HM4208, Source: IOCPQueueWorkerTask::DoWork, Description: An unknown error occured while handling asynchronous requests."
"ERROR" 3204 "2010-10-12 17:04:26.937" "Severity: 2 (High), Code: HM4208, Source: IOCPQueueWorkerTask::DoWork, Description: An unknown error occured while handling asynchronous requests."
"ERROR" 1576 "2010-10-12 17:04:27.062" "Severity: 2 (High), Code: HM4208, Source: IOCPQueueWorkerTask::DoWork, Description: An unknown error occured while handling asynchronous requests."
"ERROR" 780 "2010-10-12 17:04:31.390" "Severity: 2 (High), Code: HM4208, Source: IOCPQueueWorkerTask::DoWork, Description: An unknown error occured while handling asynchronous requests."
"ERROR" 3628 "2010-10-12 17:04:51.812" "Severity: 3 (Medium), Code: HM5158, Source: MessageIndexer::DoWork, Description: An error occured while indexing messages. The indexing was aborted."
"ERROR" 4032 "2010-10-12 17:05:20.312" "Severity: 2 (High), Code: HM4223, Source: DeliveryTask::DoWork, Description: SMTPDeliverer::DeliverMessage() failed"
"ERROR" 4032 "2010-10-12 17:05:20.343" "Severity: 2 (High), Code: HM4223, Source: DeliveryTask::DoWork, Description: SMTPDeliverer::DeliverMessage() failed"
"ERROR" 4032 "2010-10-12 17:05:20.359" "Severity: 2 (High), Code: HM4223, Source: DeliveryTask::DoWork, Description: SMTPDeliverer::DeliverMessage() failed"
"ERROR" 3228 "2010-10-12 17:05:25.015" "Severity: 2 (High), Code: HM4208, Source: IOCPQueueWorkerTask::DoWork, Description: An unknown error occured while handling asynchronous requests."
"ERROR" 3028 "2010-10-12 17:10:35.796" "Severity: 2 (High), Code: HM4208, Source: IOCPQueueWorkerTask::DoWork, Description: An unknown error occured while handling asynchronous requests."

...kill & restart the service.
BTW,
Remote IP nr. above is known by me, it means that this is an authenticated connection. But this is also strange to me because HMS' SMTP port is 125 which listens to ASSP on 127.0.0.1 only.
this may or may not be related with the above though...
Katip
--
HMS 5.7.0 x64, MariaDB 10.4.10 x64, SA 3.4.2, ClamAV 0.101.2 + SaneS

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: hmailserver hang

Post by Bill48105 » 2010-10-13 22:21

Hi Katip,
Not sure if it is related but sure looks similar. Did that remote user happen to suffer internet outage around that time, get out of wireless range etc? Did you happen to do netstat -a before you restarted hmail?

When I get time I am going to do some further testing but it looks like there is a TCP/IP transport failure and hmail isn't handling it very gracefully.. My plan is to wait until transfers are going on my test server & pull the plug mid way & see what happens. :D if I can duplicate the problem we should be able to track it down & hopefully fix it.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

rob2k2
New user
New user
Posts: 28
Joined: 2013-12-12 03:07

Re: hmailserver hang

Post by rob2k2 » 2013-12-12 03:31

Any updates to this..

I'm running hmail on server 2012 64 bit standard vmhost and same as guest o/s. And this happens at least once a month.. i have a monitor on the smtp service.. but it's really annoying.. and leads to a bit of downtime.. when i can't fix it right away..

I'm runing 5.3.3-B1879 - i was thinking of upgrading.. but i have tons of custom scripts.. and i poured all my energy into this version.. inc. running zpanel etc.. i'm afraid to upgrade it..

i tried a stress test.. I started to copy a 6 GB file.. to the host with shared nic card on guest.. over my local network.. but not on the same volume as the host resides.. and got this error "hm5339, Source: TCP Connection::HandleWrite, Descriptoin: an unknown error occurred.... shortly into the copy..

Then it just sat there.. didn't respond to any connections.. - i had to actually restart the entire server.. i tried restarting individual services.. but no luck..

Followed by a HM4208, Source: IOCPQueueWorkerTask:: DoWork, Description: An unknown error occurred.. etc..... i've tried re-delegating memory and resources.. from the host - but doesn't seem to make any difference.. it happens at random.. but i seem to be able to reproduce the error.. if i really tax the host machine.. but i haven't isolated it really...

I use to run 4.4.. never had an issue.. wish i didn't upgrade.. but only reason i upgraded was for better spam scoring... I completed the upgrade in july.. and hmail has probably hung about 7-8 times randomly.. since then..

I'm using a shared nic.. between vm's and host - i'm going to try and add a new nic.. to the host.. and dedicate it to that guest vm.. and i'm also going to put it on it's own drive array,.. at this point i have no idea if it's a i/o issue or a network issue? or something else entirely.. Im also running an exchange guest.. with no issues yet.. in the same setup with the same host..

i think it has something to do with the Nic.. is this a coding issue??? not handling dropped connections or poor network performance?? or could this be an io issue.. I'm totally dumbfounded.. i have ordered the new drive array.. and i will put in a new nic.. and post my results.. but just curious if anyone else has had this issue and solved it?

I'm about the same 20 - 30 imap connections.. 40 - 50 domains..

User avatar
mattg
Moderator
Moderator
Posts: 21114
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: hmailserver hang

Post by mattg » 2013-12-12 03:42

Please check this thread >> http://www.hmailserver.com/forum/viewto ... 10&t=21420

ALSO, please start a a new thread as per forum rules. (This thread was for an earlier version, and the handling of that error has changed in the three years since this thread. the above link is specifically for a special build for the hMailserver version that you (and most of us) are using.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

Post Reply