A lot of errors: HM5062, HM4208, HM4227,etc

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
bambam
New user
New user
Posts: 24
Joined: 2013-05-28 13:06

A lot of errors: HM5062, HM4208, HM4227,etc

Post by bambam » 2014-11-26 18:09

Hi!

I see a lot of these (more than 10000) messages per day:
ERROR" 11244 "2014-11-26 15:13:19.400" "Severity: 3 (Medium), Code: HM4227, Source: File::ReadFile, Description: An unknown error occurred while reading file from disk."
"ERROR" 11244 "2014-11-26 15:13:19.400" "Severity: 3 (Medium), Code: HM5042, Source: File::ReadTextFile, Description: An unknown error occurred while reading a text file from disk."
"ERROR" 11244 "2014-11-26 15:13:19.400" "Severity: 1 (Critical), Code: HM5062, Source: IMAPFetch::ReportCriticalError_, Description: ERROR HM10001 - IMAP FETCH: Error when loading MIME message. ******
ERROR" 11244 "2014-11-26 15:13:19.431" "Severity: 3 (Medium), Code: HM5136, Source: TCPConnection::AsyncReadCompleted, Description: An error occured while parsing data. Data length: 75, Data: bcba UID FETCH 49393:49399 (UID FLAGS RFC822.SIZE BODY.PEEK[] INTERNALDATE). Remote IP: *********
ERROR" 11244 "2014-11-26 15:13:19.463" "Severity: 2 (High), Code: HM4208, Source: ExceptionHandler::Run, Description: An error occured while executing 'IOCPQueueWorkerTask', Message: bad allocation"
"ERROR" 10532 "2014-11-26 15:13:21.682" "Severity: 3 (Medium), Code: HM4227, Source: File::ReadFile, Description: An unknown error occurred while reading file from disk."
"ERROR" 10532 "2014-11-26 15:13:21.682" "Severity: 3 (Medium), Code: HM5042, Source: File::ReadTextFile, Description: An unknown error occurred while reading a text file from disk."
"ERROR" 10532 "2014-11-26 15:13:21.682" "Severity: 1 (Critical), Code: HM5062, Source: IMAPFetch::ReportCriticalError_, Description: ERROR HM10001 - IMAP FETCH: Error when loading MIME message. - ********
"ERROR" 10532 "2014-11-26 15:13:21.729" "Severity: 2 (High), Code: HM4208, Source: ExceptionHandler::Run, Description: An error occured while executing 'IOCPQueueWorkerTask', Message: bad allocation"
ERROR" 10888 "2014-11-26 15:13:31.058" "Severity: 3 (Medium), Code: HM5136, Source: TCPConnection::AsyncReadCompleted, Description: An error occured while parsing data. Data length: 164, Data: 0ga1 UID FETCH

"ERROR" 2736 "2014-11-26 17:42:26.342" "Severity: 3 (Medium), Code: HM4227, Source: File::ReadFile, Description: An unknown error occurred while reading file from disk."

I've replaced sensitive information like message patch (which includes domain name) and IP addresses with ****.

I noticed that errors with message path are related to messages larger than 10MB.

I've set the SMTP to 60480KB and restart hmail. The problems persist. Previously it was 20480KB.

Also i get a lot of .dmp, like 2-3 per day.

Imap sync stops and i need to restart hmail to resume. SMTP is not affected.

I run the latest build of Hmailserver on a Windows 2012 R2 machine, with 24 GB of ram, 2 SSD in RAID 1 and a new E3-1270 v2 CPU. CPU load is 20-50%.
During working hours there are 25-30 IMAP sessions. User mailbox ranges from 15-40 GB.

I've started a Data Colector Set in pefrmon and i can see these:
1. Disk Transfers /sec (IOPS) is most of time less than 200 with rare peaks at 500.
2. Memory usage (Private bytes and Working set) are less 1 GB during the day.
3. Seldom times disk IOPS are queued. The queue is less than 2 IOPS.

It doesn't matter how many IMAP sessions are connected. It happens at 01:00 am as it happens at 04:00 pm.

I run several instances of hmail. These problems are specific to this one. Also this is the biggest deployment. The Data folder size is more than 400 GB.


Any help would be highly apreciated.

Regards,
Bam

percepts
Senior user
Senior user
Posts: 5282
Joined: 2009-10-20 16:33
Location: Sceptred Isle

Re: A lot of errors: HM5062, HM4208, HM4227,etc

Post by percepts » 2014-11-26 18:58

which version of hmail and when did you upgrade to that version.

Which SQL server are you using and which version.

When did these errors start happening? Didi anything change on your system at that time? Other software installed or windows update etc.

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: A lot of errors: HM5062, HM4208, HM4227,etc

Post by martin » 2014-11-26 18:59

Can you send me one of the dump files? And also let me know what version you're using?

I suspect it's due to an out-of-memory issue some users have been seeing. That specific one will be fixed in 5.7, but a dump file could tell me a bit more.

My address is martin@hmailserver.com.
Martin Knafve
martin@hmailserver.com
https://twitter.com/knafve

bambam
New user
New user
Posts: 24
Joined: 2013-05-28 13:06

Re: A lot of errors: HM5062, HM4208, HM4227,etc

Post by bambam » 2014-11-26 21:01

Hello Martin and Percepts!

Thank you both for your prompt replies!

I've started this server with 5.5.2-B2129 on the 30th of October 2014 and upgraded to B2145 on the 20th of November. Errors started to appear in the logs on day 1.

Background info: I've migrated the email service to this machine from an old Linux machine. The migration was done with some imap migration tool.
Nothing changed on the machine except for regular windows updates and IIS with PHP installation for phpwebadmin and roundcube.

MySQL version is 5.6.2..1 x64.

I would not bet on the memory shortage, as my perfmon graphs don't back up this theory. I'll attach those too, along logs and dmp files. Or maybe i can't read them the right way...

I'll be sending the files to Martin in a couple of minutes.

Regards,

Bam

bambam
New user
New user
Posts: 24
Joined: 2013-05-28 13:06

Re: A lot of errors: HM5062, HM4208, HM4227,etc

Post by bambam » 2014-11-26 21:15

Martin,

Regarding out of memory errors, would you recommend I set up the flag that allows the app to address more than 1.8 GB of RAM as per this thread viewtopic.php?f=7&t=26317 ?

Regards,

Bam

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: A lot of errors: HM5062, HM4208, HM4227,etc

Post by martin » 2014-11-26 21:18

I would at least give it a try if I were you, if the issue is very recurring.
Martin Knafve
martin@hmailserver.com
https://twitter.com/knafve

bambam
New user
New user
Posts: 24
Joined: 2013-05-28 13:06

Re: A lot of errors: HM5062, HM4208, HM4227,etc

Post by bambam » 2014-11-26 21:23

Thanks for the advice. I'll do that and get back to you.

Regards,

Bam

bambam
New user
New user
Posts: 24
Joined: 2013-05-28 13:06

Re: A lot of errors: HM5062, HM4208, HM4227,etc

Post by bambam » 2014-11-26 21:42

Hi Martin!

Done that. We'll see tomorrow some results as during the night the load is minimal (less than 40 MB of memory used).

Regards,

Bam

bambam
New user
New user
Posts: 24
Joined: 2013-05-28 13:06

Re: A lot of errors: HM5062, HM4208, HM4227,etc

Post by bambam » 2014-11-27 10:07

Hi!

Just updating my thread. For the moment the error pattern changed. I can only see 2 errors like: "ERROR" 10868 "2014-11-27 09:31:10.158" "Severity: 2 (High), Code: HM4208, Source: ExceptionHandler::Run, Description: An error occured while executing 'IOCPQueueWorkerTask', Message: bad allocation"
and less than 30 like

"ERROR" 3004 "2014-11-27 09:44:06.579" "Severity: 3 (Medium), Code: HM5026, Source: PersistentMessage::_WriteDataToMessageFile, Description: Message retrieval failed because message file PATH TO .eml FILE REMOVED BY ME FOR PRIVACY REASONS did not exist."

Also two minidumps were created.

I've sent them all to Martin.

However hms memory usage did not go over 1.2 GB of RAM yet.

Chnages I've made so far:
1. increased smtp max eml limit to about 60 MB (from 20)
2. aplied the "app can use more than 2 gb" flag
3. decreased the values of the parameters in performance.

Changes 1 and 3 did not have any immediate effect.

I'll wait and see.

Regards,

BAM

User avatar
mattg
Moderator
Moderator
Posts: 20216
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: A lot of errors: HM5062, HM4208, HM4227,etc

Post by mattg » 2014-11-27 12:56

bambam wrote:"ERROR" 3004 "2014-11-27 09:44:06.579" "Severity: 3 (Medium), Code: HM5026, Source: PersistentMessage::_WriteDataToMessageFile, Description: Message retrieval failed because message file PATH TO .eml FILE REMOVED BY ME FOR PRIVACY REASONS did not exist."
This one is normally caused by AntiVirus software on the server removing the message.
You should exclude the hmailserver data directory from AV scanning
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: A lot of errors: HM5062, HM4208, HM4227,etc

Post by martin » 2014-11-27 15:10

Roughly how many users are there on the server? And how many messages?

The messages you originally get are related to memory allocation, which is why I suggest a memory usage issue.
Martin Knafve
martin@hmailserver.com
https://twitter.com/knafve

bambam
New user
New user
Posts: 24
Joined: 2013-05-28 13:06

Re: A lot of errors: HM5062, HM4208, HM4227,etc

Post by bambam » 2014-11-27 15:14

Hi Mattg!

Thanks for your reply! I've opened a new thread for this issue here: viewtopic.php?f=7&t=27426.
Since I've sorted the issues I had last night this is the only persistent and recurring type error in my logs right now.

Regarding AV. I'm using Eset File Security for Windows Server. The hmailserver folder is excluded from scanning and real time protection.
Anyway, nothing shows up in the AV's logs so I would rule this possibility out for the moment.

Regards,

BAM

percepts
Senior user
Senior user
Posts: 5282
Joined: 2009-10-20 16:33
Location: Sceptred Isle

Re: A lot of errors: HM5062, HM4208, HM4227,etc

Post by percepts » 2014-11-27 15:22

you must also exclude hmailserver/temp
Last edited by percepts on 2014-11-27 15:25, edited 1 time in total.

bambam
New user
New user
Posts: 24
Joined: 2013-05-28 13:06

Re: A lot of errors: HM5062, HM4208, HM4227,etc

Post by bambam » 2014-11-27 15:25

martin wrote:Roughly how many users are there on the server? And how many messages?

The messages you originally get are related to memory allocation, which is why I suggest a memory usage issue.
Hi!

You were right about memory issues! As it turns out hmail is climbing up to 3.2 GB of memory since this morning and no errors logged except for some 300 errors for which I opened a new thread: viewtopic.php?f=7&t=27426.

If you want I can send the new logs, dmp and perfmon log to you email.

There are around 30 users (some with laptop+mobile phone). In the database, the hm_messages table has roughtly 1.95 million records. The file count on the server seems to be consistent with that number.

Thanks,

BAM

bambam
New user
New user
Posts: 24
Joined: 2013-05-28 13:06

Re: A lot of errors: HM5062, HM4208, HM4227,etc

Post by bambam » 2014-11-27 15:29

percepts wrote:you must also exclude hmailserver/temp
Thanks for this!
I've excluded the whole hmailserver folder and added .eml to the exclusion list generally.
Regards

bambam
New user
New user
Posts: 24
Joined: 2013-05-28 13:06

Re: A lot of errors: HM5062, HM4208, HM4227,etc

Post by bambam » 2014-11-27 23:43

Hi!

Here are the updates to this situation.
Today there were no problems regarding "Bad allocation". Also HMS didn't crash. However it is concerning that memory usage reached 3.7 GB. I'm afraid it will run out of memory if I add more users.

Is there any way to decrease memory usage by tweaking the settings? Maybe from the performance tab?
Are there plans to release a 64 bit version of HMS?

Regards,

BAM

User avatar
mattg
Moderator
Moderator
Posts: 20216
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: A lot of errors: HM5062, HM4208, HM4227,etc

Post by mattg » 2014-11-28 00:35

bambam wrote:Regarding AV. I'm using Eset File Security for Windows Server. The hmailserver folder is excluded from scanning and real time protection.
See incompatible software here >> https://www.hmailserver.com/documentati ... quirements

Are you using an inline AV scanner?
bambam wrote:However it is concerning that memory usage reached 3.7 GB.
How many users?
How are they connecting (IMAP / POP3)?
How much mail does this server handle per day?
martin wrote:Roughly how many users are there on the server? And how many messages?
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

bambam
New user
New user
Posts: 24
Joined: 2013-05-28 13:06

Re: A lot of errors: HM5062, HM4208, HM4227,etc

Post by bambam » 2014-12-03 12:39

Hi guys!

Thanks so much for your support as it has been really helpful. I think I've nailed it down. Hmail seems OK now. No more errors for the last 3 days :D ! I'll share with you my findings. I'm curious if any of you can confirm these:
- I've managed to sensibly lower the memory and CPU usage (1.5 GB and an average of 35% CPU usage) by increasing the performance > advanced >max number of command threads to 30. Increasing this to 60 reduced it further: 1.1 GB mem and 15% CPU. We have arount 60 IMAP session simultaneously.
- Lots of errors can be caused by lowering the SMTP max message size while you use IMAP if you already have in your Data folder messages larger than the limit you are enforcing. This issue also affects scenarios where you are migrating user mailboxes from another server. You should check that the limit you have on hmail accommodates existing messages. All existing messages that are larger than this limit will cause errors whenever accessed and I have a feeling that this will also increase CPU and memory usage.
- Memory usage increases highly if you have migrated user mailboxes form another mail server and users are syncing large amounts of messages through IMAP. Allowing the app to use more than 2 GB of memory will help through those days.

Regards,

BAM

percepts
Senior user
Senior user
Posts: 5282
Joined: 2009-10-20 16:33
Location: Sceptred Isle

Re: A lot of errors: HM5062, HM4208, HM4227,etc

Post by percepts » 2014-12-03 12:49

If you don't have Performance / Message Indexing switched on then I would enable it. It helps greatly with IMAP performance on large mailboxes.

Aslo disuade users with large mailboxes from doing searches on mail body content. Date, sender, subject are fine but not body content.

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: A lot of errors: HM5062, HM4208, HM4227,etc

Post by martin » 2014-12-04 09:03

Just so you know, I'm still working on addressing this. Reducing memory usage and adding 64-bit support.
Martin Knafve
martin@hmailserver.com
https://twitter.com/knafve

Post Reply