OutOfMemoryHandler - hMailServer has run out of memory

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
nahosting
New user
New user
Posts: 12
Joined: 2013-02-14 13:32

OutOfMemoryHandler - hMailServer has run out of memory

Post by nahosting » 2020-06-29 12:13

Hi All,

hMailServer 5.6.7-B2425

After years of stability, in the last few weeks we have start to get a lot of random "OutOfMemoryHandler - hMailServer has run out of memory, clearing caches" issues. Here is a typical one from this morning

"APPLICATION" 12168 "2020-06-28 15:17:57.260" "OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"APPLICATION" 12168 "2020-06-28 15:17:57.260" "OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"APPLICATION" 12168 "2020-06-28 15:17:57.260" "OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"ERROR" 12168 "2020-06-28 15:17:57.260" "Severity: 2 (High), Code: HM4208, Source: File::ReadFile(), Description: An error occured when reading file., Message: bad allocation"
"ERROR" 12168 "2020-06-28 15:17:57.260" "Severity: 3 (Medium), Code: HM4227, Source: File::ReadFile, Description: An unknown error occurred while reading file from disk."
"ERROR" 12168 "2020-06-28 15:17:57.260" "Severity: 3 (Medium), Code: HM5042, Source: File::ReadTextFile, Description: An unknown error occurred while reading a text file from disk."
"ERROR" 12168 "2020-06-28 15:17:57.260" "Severity: 1 (Critical), Code: HM5062, Source: IMAPFetch::ReportCriticalError_, Description: ERROR HM10001 - IMAP FETCH: Error when loading MIME message. - D:\ProgramData\hMailServer\Data\[REMOVED DOMAIN]\[REMOVED EMAIL]\FF\{FF24A71F-F059-4EBA-B35B-AF97FE6534F9}.eml"
"APPLICATION" 12168 "2020-06-28 15:17:57.260" "OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"APPLICATION" 12168 "2020-06-28 15:17:57.260" "OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"ERROR" 12168 "2020-06-28 15:17:57.260" "Severity: 3 (Medium), Code: HM5136, Source: TCPConnection::AsyncReadCompleted, Description: An error occured while parsing data. Data length: 185, Data: v3tn UID FETCH 242:244,405:406,427:428,510:511,525:526,685:686,692:694,823:824,859:860,888:889,908:912,927:928,945:947,993:995,1014:1015 (UID FLAGS RFC822.SIZE BODY.PEEK[] INTERNALDATE). Remote IP: [REMOVED]"
"ERROR" 12168 "2020-06-28 15:17:57.260" "Severity: 2 (High), Code: HM4208, Source: ExceptionHandler::Run, Description: An error occured while executing 'IOCPQueueWorkerTask', Message: bad allocation"

hMailServer then has a number of emails stuck in the queue with 0 delivery attempts. These emails are generally, but no always, fairly large i.e. 10mb upwards.
The only way of stopping/resolving these errors is to restart the service.

We have searched the forums and done the following

1. Checked all the hardware, everything is working fine.
2. Removed our own script, which has been working for years - not resolved the issue.
3. Checked that hMailServer doesn't actually run out of memory for a 32 bit app by having a script that monitors it - https://www.hmailserver.com/forum/viewtopic.php?t=26317
4. Checked the AV setup and even switched it off - no difference.
5. Cleaned up the Data directory (moved old files)
6. Run script from here - https://www.hmailserver.com/forum/viewtopic.php?t=27646
7. Run DDS
8. Checked mailbox limits (set in smtp at 25mb, all domains are 0)

Gone through this thread https://www.hmailserver.com/forum/viewtopic.php?t=27391

It just seems to be with larger files, now got a scenario where a clients mailbox can't be downloaded due to constant errors as soon as it hits a problem email file.

Any help appreciated as service is starting to suffer. Thanks.

User avatar
johang
Normal user
Normal user
Posts: 168
Joined: 2008-09-01 09:20

Re: OutOfMemoryHandler - hMailServer has run out of memory

Post by johang » 2020-06-29 14:01

nahosting wrote:
2020-06-29 12:13
Hi All,

hMailServer 5.6.7-B2425

After years of stability, in the last few weeks we have start to get a lot of random "OutOfMemoryHandler - hMailServer has run out of memory, clearing caches" issues. Here is a typical one from this morning

"APPLICATION" 12168 "2020-06-28 15:17:57.260" "OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"APPLICATION" 12168 "2020-06-28 15:17:57.260" "OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"APPLICATION" 12168 "2020-06-28 15:17:57.260" "OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"ERROR" 12168 "2020-06-28 15:17:57.260" "Severity: 2 (High), Code: HM4208, Source: File::ReadFile(), Description: An error occured when reading file., Message: bad allocation"
"ERROR" 12168 "2020-06-28 15:17:57.260" "Severity: 3 (Medium), Code: HM4227, Source: File::ReadFile, Description: An unknown error occurred while reading file from disk."
"ERROR" 12168 "2020-06-28 15:17:57.260" "Severity: 3 (Medium), Code: HM5042, Source: File::ReadTextFile, Description: An unknown error occurred while reading a text file from disk."
"ERROR" 12168 "2020-06-28 15:17:57.260" "Severity: 1 (Critical), Code: HM5062, Source: IMAPFetch::ReportCriticalError_, Description: ERROR HM10001 - IMAP FETCH: Error when loading MIME message. - D:\ProgramData\hMailServer\Data\[REMOVED DOMAIN]\[REMOVED EMAIL]\FF\{FF24A71F-F059-4EBA-B35B-AF97FE6534F9}.eml"
"APPLICATION" 12168 "2020-06-28 15:17:57.260" "OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"APPLICATION" 12168 "2020-06-28 15:17:57.260" "OutOfMemoryHandler - hMailServer has run out of memory, clearing caches."
"ERROR" 12168 "2020-06-28 15:17:57.260" "Severity: 3 (Medium), Code: HM5136, Source: TCPConnection::AsyncReadCompleted, Description: An error occured while parsing data. Data length: 185, Data: v3tn UID FETCH 242:244,405:406,427:428,510:511,525:526,685:686,692:694,823:824,859:860,888:889,908:912,927:928,945:947,993:995,1014:1015 (UID FLAGS RFC822.SIZE BODY.PEEK[] INTERNALDATE). Remote IP: [REMOVED]"
"ERROR" 12168 "2020-06-28 15:17:57.260" "Severity: 2 (High), Code: HM4208, Source: ExceptionHandler::Run, Description: An error occured while executing 'IOCPQueueWorkerTask', Message: bad allocation"

hMailServer then has a number of emails stuck in the queue with 0 delivery attempts. These emails are generally, but no always, fairly large i.e. 10mb upwards.
The only way of stopping/resolving these errors is to restart the service.

We have searched the forums and done the following

1. Checked all the hardware, everything is working fine.
2. Removed our own script, which has been working for years - not resolved the issue.
3. Checked that hMailServer doesn't actually run out of memory for a 32 bit app by having a script that monitors it - https://www.hmailserver.com/forum/viewtopic.php?t=26317
4. Checked the AV setup and even switched it off - no difference.
5. Cleaned up the Data directory (moved old files)
6. Run script from here - https://www.hmailserver.com/forum/viewtopic.php?t=27646
7. Run DDS
8. Checked mailbox limits (set in smtp at 25mb, all domains are 0)

Gone through this thread https://www.hmailserver.com/forum/viewtopic.php?t=27391

It just seems to be with larger files, now got a scenario where a clients mailbox can't be downloaded due to constant errors as soon as it hits a problem email file.

Any help appreciated as service is starting to suffer. Thanks.

Marc H had very similar problem
https://www.hmailserver.com/forum/viewtopic.php?t=33562
however there is no way to tell from the post if he fixed it or not
___________________________________________________________end of the line
spam filter appliance gateway: www.mailcleaner.org

nahosting
New user
New user
Posts: 12
Joined: 2013-02-14 13:32

Re: OutOfMemoryHandler - hMailServer has run out of memory

Post by nahosting » 2020-06-29 16:25

We just tried the diagnostics from HMSSettingsDiagnostics.zip
The noticeable thing was 147K whitelist entries.
We currently have loads more of the same errors.
Currently 256 IMAP connections.

User avatar
johang
Normal user
Normal user
Posts: 168
Joined: 2008-09-01 09:20

Re: OutOfMemoryHandler - hMailServer has run out of memory

Post by johang » 2020-06-29 17:03

___________________________________________________________end of the line
spam filter appliance gateway: www.mailcleaner.org

nahosting
New user
New user
Posts: 12
Joined: 2013-02-14 13:32

Re: OutOfMemoryHandler - hMailServer has run out of memory

Post by nahosting » 2020-06-29 20:07

Based on that thread and looking at other similar ones we have updated the threading from 15,10,15 to 4,2,4 based on the number cpus for the Xeon E3-1220 v5 it is running on (although the server never has any cpu issues). Restarted the service.

nahosting
New user
New user
Posts: 12
Joined: 2013-02-14 13:32

Re: OutOfMemoryHandler - hMailServer has run out of memory

Post by nahosting » 2020-06-29 21:30

nahosting wrote:
2020-06-29 20:07
Based on that thread and looking at other similar ones we have updated the threading from 15,10,15 to 4,2,4
Changing the threading made no difference.

User avatar
johang
Normal user
Normal user
Posts: 168
Joined: 2008-09-01 09:20

Re: OutOfMemoryHandler - hMailServer has run out of memory

Post by johang » 2020-06-29 22:17

nahosting wrote:
2020-06-29 12:13
It just seems to be with larger files, now got a scenario where a clients mailbox can't be downloaded due to constant errors as soon as it hits a problem email file.
My thoughts...

disk I/O ?
timeout ?
read/write ?
are those files possibly locked by any other process somehow.. ?

have you tested different imap clients ? apple, thunderbird, outlook, webmail (do they all behave the same.. getting stuck..)


in a previous life running mail services on *nix, we split DATA directorys over "separate" disk sets to get rid of problems like the ones you describe
read somewhere in the hmailserver forum about splitting on seprate servers moving domains to new hosts to minimize load


even though you say you check for mem usage ..
at what point of RAM usage does it happen for you ?
i would try what snico did .. https://www.hmailserver.com/forum/viewt ... 17#p160610



and i would sneak peak over on 5.7 and consider it ..
https://github.com/hmailserver/hmailserver/issues/258
___________________________________________________________end of the line
spam filter appliance gateway: www.mailcleaner.org

User avatar
jim.bus
Senior user
Senior user
Posts: 351
Joined: 2011-05-28 11:49
Location: US

Re: OutOfMemoryHandler - hMailServer has run out of memory

Post by jim.bus » 2020-06-30 09:36

I found this thread https://www.hmailserver.com/forum/viewtopic.php?t=29099.

The error messages listed in it looked very similar to the errors reported with your problem. There was no resolution in this thread as far as I could tell but the person who had this problem noted the problem seemingly only happened when fetching large email messages with IMAP and from your error messages and statements you are processing IMAP messages. And you were getting errors on IMAP. I can't tell by the way the person 'wrote big mails using IMAP' if he met the IMAP Mails (individual messages) were large or if he meant there were a lot of IMAP fetches or connections.

nahosting
New user
New user
Posts: 12
Joined: 2013-02-14 13:32

Re: OutOfMemoryHandler - hMailServer has run out of memory

Post by nahosting » 2020-06-30 10:07

Thanks jim.bus that does seem very similar and looking at the logs it does appear to be an issue with IMAP clients getting large files. Going to try various clients today and see if we can get to the bottom of this.

User avatar
jim.bus
Senior user
Senior user
Posts: 351
Joined: 2011-05-28 11:49
Location: US

Re: OutOfMemoryHandler - hMailServer has run out of memory

Post by jim.bus » 2020-06-30 12:43

I am getting out of my element with Out Of Memory errors. My installation never has to worry about such things as my installation is extremely small. But I was originally looking at this as a possible area of concern and on looking at this User Help file entry, Cache is related to memory usage.

See this discussion of how Cache works: https://www.hmailserver.com/documentati ... ails_cache

I do not know if this is even relevant but depending on how many Database objects you might be using at any given time, could you have so many objects that you might use all available Cache memory. If there is any correlation to this Out Of Memory error one way to confirm this is the area where problem might be would be to try Disabling Cache using hMailAdmin in the Performance Settings. This would most likely increase hMailServer Processing time but if the Out Of Memory error messages went away this would show you were running out of Cache.

However, from reading the link above and other things, I'm not expecting that to be the problem but Cache is in memory and your Error Messages indicates you may be running out of Cache as hMailServer is clearing the Cache. I would think you would need a very large number of Database objects to fill up all the available memory.

Disabling Cache is simple to do as it is simply unchecking the Cache option in the Performance section of hMailAdmin.

Another possibility related to the Cache is this thread which I had also considered this issue as a possibility and it appears Martin (Developer) also considered this in this thread. So my thoughts may very well be relevant also:

https://www.hmailserver.com/forum/viewtopic.php?t=27391

In this thread the person discovered when large messages were being processed the errors occurred. The person was not letting Windows manage the Virtual Memory Page File. His Page File was only manually set for 512 MB which is extremely small. However, it is not clear to me when he increased the Page File size if it was the solution that fixed the problem because he stated other errors cropped up. But this link is a very old link and there have been upgrades to hMailServer such that these additional errors might have been fixed by now. I let Windows System Manage my Page File and my Page File Allocation is 4864 MB. So I would check this out as well.

User avatar
jim.bus
Senior user
Senior user
Posts: 351
Joined: 2011-05-28 11:49
Location: US

Re: OutOfMemoryHandler - hMailServer has run out of memory

Post by jim.bus » 2020-06-30 13:29

One additional comment about my second alternate issue to look at in my prior post, If the data being read into memory such as maybe in the IMAP Fetch was larger than the Page File size, this would probably create a Page Fault in Windows which possibly hMailServer detects before it becomes a Page Fault. But if the Page File size too small to accommodate the size of the Page File this would probably cause the Out Of Memory Errors as a Page Fault error is addressing memory which is outside of your memory allocation as best I understand what a Page Fault is.

If you look at your error messages you get a Critical Error when trying to Fetch an IMAP Message. The error occuse when loading the MIME message. This could be because the message is too large to fit in the memory allocated which would be directly related to what the person found in the first link. That person found when the messages were too large they got Out Of Memory Errors. So, I would definitely check to see if your Windows Page File is being System Managed or the Page File size looks like it may be too small.

palinka
Senior user
Senior user
Posts: 1942
Joined: 2017-09-12 17:57

Re: OutOfMemoryHandler - hMailServer has run out of memory

Post by palinka » 2020-06-30 13:49

nahosting wrote:
2020-06-29 16:25
We just tried the diagnostics from HMSSettingsDiagnostics.zip
The noticeable thing was 147K whitelist entries.
We currently have loads more of the same errors.
Currently 256 IMAP connections.
Good Lord! That has to be your problem. Why don't you offload that to a separate database?

https://hmailserver.com/forum/viewtopic ... 20&t=33602

Post Reply