New server, hMailServer and IMAP performance

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
mrhappy
New user
New user
Posts: 9
Joined: 2010-08-12 13:54

New server, hMailServer and IMAP performance

Post by mrhappy » 2019-07-08 12:37

Hi folks,

First post ever so please bear with me and sorry for the long one. Looking for advice, please.

I work for a small company. We have a few email addresses, but one which is hosted by hMailServer is a shared address for most of our workforce, for 90% of our business communications with our clients etc. We do not have individual email addresses and the shared account can be accessed by our workforce on 3 different machines. The machines are of the same spec, core i5-7500 with SSD running Windows 10 and Thunderbird (60.7.2) as the mail client.

We have recently upgraded our server to an HP Proliant ML110 Gen10 (Xeon Silver 4110, 32GB, Win Server 2016) and successfully migrated our hMailServer database from our old server. hMailServer is version 5.6.4-B2283 using MSSQLExpress

The new server has 2x SSD in RAID 1 for OS, 2x 2TB SATA HDD in RAID 1 for data, which is where the hMailServer database resides.

Overall, performance on the clients is much improved, however we are now experiencing short delays (up to ~30 seconds) when navigating the IMAP subfolders in Thunderbird. Thunderbird is configured to cache or store NO emails locally.

New incoming messages can usually be read instantaneously, but when accessing older messages I think the delay is proportional to message size. When an email is sent, switching to and updating the Sent mailbox causes a delay each time. Then, again navigating between emails here delay is proportional to message size.

The main point of this post is, what do you think is the root cause for this delay?

When specifying the server, I wanted to use SSDs for the data and hMailServer database storage location, but our IT consultant - who ordered, built, supplied and configured the server to us - insisted that a SATA HDD array would be more than adequate, based on his experience with managing MS Exchange servers.

Please let me know if and what other information might be useful. We us Avast Business CloudCare on server and workstations for AV/security.

Many thanks in advance. James

mrhappy
New user
New user
Posts: 9
Joined: 2010-08-12 13:54

Re: New server, hMailServer and IMAP performance

Post by mrhappy » 2019-07-08 12:41

The size of the database for this mail account is 175810.266MB (172GB) as reported by hMailServer.

User avatar
mattg
Moderator
Moderator
Posts: 20144
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: New server, hMailServer and IMAP performance

Post by mattg » 2019-07-08 13:17

mrhappy wrote:
2019-07-08 12:37
The main point of this post is, what do you think is the root cause for this delay?
mrhappy wrote:
2019-07-08 12:41
The size of the database for this mail account is 175810.266MB (172GB) as reported by hMailServer.
Related events I'd say
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
Dravion
Senior user
Senior user
Posts: 1435
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: New server, hMailServer and IMAP performance

Post by Dravion » 2019-07-08 14:46

Youre right, SSD would be way faster than a HDD Array.
Of cause there are older SSDs which are slow to, but modern SAMSUNG SSDs for example doing verry well.
I5 CPU are a little dated and you dhould upgrade to latest Xenon or I9 CPUs and fastest possible ECC RAM Modules.
Mainboard or NIC Adapter can slow down stuff.

PS: Is your SQLServer Express realy 172 GBytes big
or do you mean your DATA Directory is 172 GBytes big or is the Problematic IMAP Accounts Data in the Data Folder 172 GBytes big?

Regarding loading speed in older Emails.
Lookups and search speed for older Emails via IMAP
are not done via SQL Database lookups.
EML files are opened by hMailServer and scanned
line by line (like most IMAP Servers are doing, which is not the best way to do it).Only MS-Exhange has a diffrent approach usinga LDAP Style (non sql) Hierachical DB to store all data in big EDB Database files, which means your Consultant is comparing Apple and Oranges. New IMAP Messages are cached in RAM for a time while others are not.
If you want bigger caches you need a 64-Bit build of
hMailServer (32-Bit hMailServer builds can only access 2 GBytes, even if your 64-Bit HP Proliant Server has 32 GBytes RAM avaiable).

mrhappy
New user
New user
Posts: 9
Joined: 2010-08-12 13:54

Re: New server, hMailServer and IMAP performance

Post by mrhappy » 2019-07-08 15:34

Hi Dravion, thanks for your reply.
Dravion wrote:
2019-07-08 14:46
I5 CPU are a little dated and you dhould upgrade to latest Xenon or I9 CPUs and fastest possible ECC RAM Modules.
Mainboard or NIC Adapter can slow down stuff.
Just to clarify, the core i5 CPUs are in the 3 workstations we are using for email clients. They are Dell OptiPlex 3050 micro and just do email and internet for about a dozen users.
Dravion wrote:
2019-07-08 14:46
PS: Is your SQLServer Express realy 172 GBytes big
or do you mean your DATA Directory is 172 GBytes big or is the Problematic IMAP Accounts Data in the Data Folder 172 GBytes big?
Apologies, the data directory for that account is 172GB. The SQL database is not that large.

Is this a large amount of email data for one IMAP account on hMailServer? If yes, what's the best course of action for archiving this data?
Currently I use the archive function of Thunderbird to move old emails, e.g. imap://[OUR_MAIL_ACCOUNT]@[OUR_HMAILSERV]/Archives/2018/2018-06/Sent%20Items (default Tbird behaviour), but acknowledge this doesn't actually move the email data files.
Dravion wrote:
2019-07-08 14:46
If you want bigger caches you need a 64-Bit build of
hMailServer (32-Bit hMailServer builds can only access 2 GBytes, even if your 64-Bit HP Proliant Server has 32 GBytes RAM avaiable).
Would you recommend we should use a 64-bit build of hMailServer? How easy is it to switch from 32-bit to 64-bit builds? How do I do it?(heading off-topic now!)

Is this the source of the problem? Or just a contributing factor?
Dravion wrote:
2019-07-08 14:46
Youre right, SSD would be way faster than a HDD Array...

...Regarding loading speed in older Emails. Lookups and search speed for older Emails via IMAP are not done via SQL Database lookups. EML files are opened by hMailServer and scanned line by line (like most IMAP Servers are doing, which is not the best way to do it).
So, moving the data to an SSD array would be a good option?

mrhappy
New user
New user
Posts: 9
Joined: 2010-08-12 13:54

Re: New server, hMailServer and IMAP performance

Post by mrhappy » 2019-07-08 15:35

mattg wrote:
2019-07-08 13:17
mrhappy wrote:
2019-07-08 12:37
The main point of this post is, what do you think is the root cause for this delay?
mrhappy wrote:
2019-07-08 12:41
The size of the database for this mail account is 175810.266MB (172GB) as reported by hMailServer.
Related events I'd say
So, as per my reply to Dravion, it's the data directory that's 172GB, not the SQL database. Does this change your answer?

Is 172GB big for an account in hMailServer?

Thanks :)

User avatar
Dravion
Senior user
Senior user
Posts: 1435
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: New server, hMailServer and IMAP performance

Post by Dravion » 2019-07-08 16:33

You have to be more specific.

Is your entire hMailServer DATA Folder 172 GBytes big (which includes the Emails of all hMailServer Accounts)
or are we talking about only 1 IMAP Account and which has 172 GBytes of data?)

If the entire DATA Folder is 172 GBytes big (for all Users)
its not that big.

PS: SSD is allways a good decision and even more
for Multithreaded Servers (like hMailServer) because diffrent threads trying to read diffrent Data from diffrent positions which can be dramatically slowed down by moving Readheads.

Switching from 32 to 64-Bit (and back) is easy:
1) Backup your DATA Folder (just in case)
2) Install hMailServer 64-Bit over your existing installation

You can use the LTS 64-Bit built
https://www.hmailserver.com/forum/viewt ... 10&t=34039

PS2: Allways install the latest Windows updates and update your Motherboard Chipset Drivers because if you running on poor Windows Default Drivers it can harm your Performance extremely.

mrhappy
New user
New user
Posts: 9
Joined: 2010-08-12 13:54

Re: New server, hMailServer and IMAP performance

Post by mrhappy » 2019-07-08 22:38

Dravion wrote:
2019-07-08 16:33
You have to be more specific.

Is your entire hMailServer DATA Folder 172 GBytes big (which includes the Emails of all hMailServer Accounts)
or are we talking about only 1 IMAP Account and which has 172 GBytes of data?)

If the entire DATA Folder is 172 GBytes big (for all Users)
its not that big.
The data folder for just this one account is 172GB.

There are other 3 other accounts hosted by hMailServer, but they are really small in comparison.

If this is big for a single account, what would you recommend I should do about it?

I'm convinced we should move the data to an SSD array. And if switching to a 64-bit build is that easy, we'll do that in the process.

Does SQL need to changed at all, or will this be OK? I'm assuming (though I don't know off the top of my head) that we installed a 64-it version of this. I can check tomorrow.

Many thanks

User avatar
Dravion
Senior user
Senior user
Posts: 1435
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: New server, hMailServer and IMAP performance

Post by Dravion » 2019-07-09 01:20

Ok, if a specific Account is 172 GBytes big, loading and search speed for older Messages will increase because
the C++ lookup code is not that good for realy large Accounts (it simply scans the whole bunch of any single file EML File linked to the Account line by line).

We should take care of this in the future, but for now your
best bet is to increase your computing power by raw hardware upgrades and use of a recent 64-Bit build of hMailServer

Tips:
1)
Make sure you use a SSD for hMailServers DATA Folder
(the faster the Data for search is loaded, the better)

2)
Buy fast SSDs, not slow SSD ones, speed can differ
dramatically even in terms of SSDs.

3)
Make sure you have a top notch CPU like Intel core i9
or a decent Xenon CPU with large caches (increases raw searchspeed the most). Make sure cooling is ok, or your CPU will slowdown automatically to prevent damage

4)
Make sure you have fast RAM Modules installed (for example DDR4) with high transferrates to make sure your
CPU has plenty of data chunks to search in and does not go idle and drink a coffee meanwhile.

5)
Allways install your Mainboard Chipset Drivers and other Hardware Drivers and avoid using Windows Default drivers.

6)
Use System Performance Test tools so you can see what works out best for you and optimize the weak spots even more if possible.
Last edited by Dravion on 2019-07-09 01:29, edited 5 times in total.

User avatar
mattg
Moderator
Moderator
Posts: 20144
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: New server, hMailServer and IMAP performance

Post by mattg » 2019-07-09 01:21

mrhappy wrote:
2019-07-08 22:38
I'm assuming (though I don't know off the top of my head) that we installed a 64-it version of this.
I wouldn't assume that - the 64 bit driver is also tricky (read near impossible)to implement with standard HMailserver builds
mrhappy wrote:
2019-07-08 22:38
The data folder for just this one account is 172GB.
That is big

You say you have a single IMAP account that is used on three devices for this singularly large IMAP account. That is a lot of server load, effectively each of the clients is competing with each other, and manipulating the entire account every microsecond.

Have you looked at public folders? These were designed for multiple user access
Alternately, move many of the emails to another lesser used account (ie archive all mail over a year old or such)
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
Dravion
Senior user
Senior user
Posts: 1435
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: New server, hMailServer and IMAP performance

Post by Dravion » 2019-07-09 01:35

Does SQL need to changed at all, or will this be OK? I'm assuming (though I don't know off the top of my head) that we installed a 64-it version of this. I can check tomorrow.
No, you shouldnt touch your DB because thats not your performance issue . In terms of DB Upgrades you can do a lot wrong, so without need i would change your DB at this time.

mrhappy
New user
New user
Posts: 9
Joined: 2010-08-12 13:54

Re: New server, hMailServer and IMAP performance

Post by mrhappy » 2019-07-09 10:36

Thank you both for your helpful replies.
mattg wrote:
2019-07-09 01:21
That is big

You say you have a single IMAP account that is used on three devices for this singularly large IMAP account. That is a lot of server load, effectively each of the clients is competing with each other, and manipulating the entire account every microsecond.

Have you looked at public folders? These were designed for multiple user access
Alternately, move many of the emails to another lesser used account (ie archive all mail over a year old or such)
Please could you explain a bit about Public folders. If this is an hMailServer feature, how do I got about implementing it?

I think archiving old emails is a good way forward. We currently have emails back to 2010, albeit store in not the Inbox but an Archives folder on the IMAP account. Does this make any difference to the speed of searching the Inbox, when the bulk of the account's data is stored in other IMAP folders?
Dravion wrote:
2019-07-09 01:20
2)
Buy fast SSDs, not slow SSD ones, speed can differ dramatically even in terms of SSDs.

3)
Make sure you have a top notch CPU like Intel core i9 or a decent Xenon CPU with large caches (increases raw searchspeed the most). Make sure cooling is ok, or your CPU will slowdown automatically to prevent damage
Can you recommend any particular SSDs? Samsung EVO or Crucial, for example, or do we need to look at enterprise grade drives?

CPU in our ProLiant ML110 is Xeon Silver 4110 8-core 2.1GHz. RAM is DDR4. Apart from this issue we have no other complaints about the server. I will ensure we have the latest drivers.

Many thanks

User avatar
SorenR
Senior user
Senior user
Posts: 3192
Joined: 2006-08-21 15:38
Location: Denmark

Re: New server, hMailServer and IMAP performance

Post by SorenR » 2019-07-09 11:57

mrhappy wrote:
2019-07-09 10:36
Thank you both for your helpful replies.

Please could you explain a bit about Public folders. If this is an hMailServer feature, how do I got about implementing it?

I think archiving old emails is a good way forward. We currently have emails back to 2010, albeit store in not the Inbox but an Archives folder on the IMAP account. Does this make any difference to the speed of searching the Inbox, when the bulk of the account's data is stored in other IMAP folders?
- Public folders are similar to Outlook Shared folders: https://www.hmailserver.com/documentati ... otocolimap

- If you have configured "Performance" with the hMailAdmin GUI then "date", "from, "to", "cc" and "subject" are stored as metadata in the DB to allow faster serverside sorting.

hMailAdmin -> Settings -> Advanced -> Performance

If you search for anything else in your emails it is like doing a full text search in Explorer WITHOUT indexing. The more files (emails) the longer it takes.

@Dravion, @RvdH ...
If we create a Windows index on .\Data\* can we modify hMail core code to use it?
It's an easy way (I think) to improve data search and we do not have to "re-invent the wheel".
SørenR.

“With age comes wisdom, but sometimes age comes alone.”
- Oscar Wilde

palinka
Senior user
Senior user
Posts: 1114
Joined: 2017-09-12 17:57

Re: New server, hMailServer and IMAP performance

Post by palinka » 2019-07-09 12:01

SorenR wrote:
2019-07-09 11:57
@Dravion, @RvdH ...
If we create a Windows index on .\Data\* can we modify hMail core code to use it?
It's an easy way (I think) to improve data search and we do not have to "re-invent the wheel".
Totally agree. This would be a YUUUGE boost in performance.

mrhappy
New user
New user
Posts: 9
Joined: 2010-08-12 13:54

Re: New server, hMailServer and IMAP performance

Post by mrhappy » 2019-07-09 12:20

SorenR wrote:
2019-07-09 11:57
- If you have configured "Performance" with the hMailAdmin GUI then "date", "from, "to", "cc" and "subject" are stored as metadata in the DB to allow faster serverside sorting.

hMailAdmin -> Settings -> Advanced -> Performance

If you search for anything else in your emails it is like doing a full text search in Explorer WITHOUT indexing. The more files (emails) the longer it takes.
Thanks for the tip, I will look into this now.
SorenR wrote:
2019-07-09 11:57
@Dravion, @RvdH ...
If we create a Windows index on .\Data\* can we modify hMail core code to use it?
It's an easy way (I think) to improve data search and we do not have to "re-invent the wheel".
Sounds like a neat plan!

A further observation to note is that the delay is typically for emails with attachments, the greater the attachment size the longer the delay. It will often bring up the message text before loading the attachment.

mrhappy
New user
New user
Posts: 9
Joined: 2010-08-12 13:54

Re: New server, hMailServer and IMAP performance

Post by mrhappy » 2019-07-09 12:28

I've enabled message indexing within performance and it's currently processing 258700 messages.

Based on the attached screenshots, would you recommend changing any of the other settings from their defaults?
Attachments
performance c.png
performance b.png
performance a.png

User avatar
SorenR
Senior user
Senior user
Posts: 3192
Joined: 2006-08-21 15:38
Location: Denmark

Re: New server, hMailServer and IMAP performance

Post by SorenR » 2019-07-09 13:08

mrhappy wrote:
2019-07-09 12:28
I've enabled message indexing within performance and it's currently processing 258700 messages.

Based on the attached screenshots, would you recommend changing any of the other settings from their defaults?
I don't have experience with that volume of emails... My server is running on an old dual core Windows Server 2003 with 5 users, built and installed 12/12-2007 IIRC ... so it's 100% idle about 99% of the time :mrgreen:
SørenR.

“With age comes wisdom, but sometimes age comes alone.”
- Oscar Wilde

User avatar
mattg
Moderator
Moderator
Posts: 20144
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: New server, hMailServer and IMAP performance

Post by mattg » 2019-07-09 16:44

mrhappy wrote:
2019-07-09 12:28
would you recommend changing any of the other settings from their defaults?
I run threading at 250,25,10 but I played with the settings for a while to arrive at those numbers, and it will depend on your hardware in comparison to mine and also the size of most emails

I have over 450000 messages, but only about 50 GB across my entire data folder
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
SorenR
Senior user
Senior user
Posts: 3192
Joined: 2006-08-21 15:38
Location: Denmark

Re: New server, hMailServer and IMAP performance

Post by SorenR » 2019-07-10 18:41

Building new 64 bit server... :mrgreen:
Attachments
IMG_20190710_183221.jpg
SørenR.

“With age comes wisdom, but sometimes age comes alone.”
- Oscar Wilde

palinka
Senior user
Senior user
Posts: 1114
Joined: 2017-09-12 17:57

Re: New server, hMailServer and IMAP performance

Post by palinka » 2019-07-11 01:37

SorenR wrote:
2019-07-10 18:41
Building new 64 bit server... :mrgreen:
Most people start with the case. :mrgreen:

User avatar
SorenR
Senior user
Senior user
Posts: 3192
Joined: 2006-08-21 15:38
Location: Denmark

Re: New server, hMailServer and IMAP performance

Post by SorenR » 2019-07-11 03:01

palinka wrote:
2019-07-11 01:37
SorenR wrote:
2019-07-10 18:41
Building new 64 bit server... :mrgreen:
Most people start with the case. :mrgreen:
MB = Asus P5Q TURBO
CPU = Intel E8400 Core2 Duo 3GHz
VGA = 3dfx Vodoo3 3000
RAM = 6 GB
HD1 = Maxstor 8 - 80 GB (old Asterisk on Centos 5.5, 1 FXO card and 4 port FXS card by Digium)
HD2 = Maxstor 10 - 300 GB (old backup drive)
HD3 = Samsung 500GB/7200RPM/16M <== This will be the one !
PSU = NorthQ 400W

That is what I managed to put together from salvaged parts from the past 15 years - so until now - IT'S FREE! :mrgreen:

It is going to be interesting to see the difference, my current server is Intel E6550 Core2 Duo 2.33GHz with 2 GB RAM and a Samsung 500GB/7200RPM/16M running 32 bit Windows Server 2003 R2.

I will be downloading a 180 day demo of Windows Server 2019 64 bit for the other one - to start with.
SørenR.

“With age comes wisdom, but sometimes age comes alone.”
- Oscar Wilde

User avatar
mattg
Moderator
Moderator
Posts: 20144
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: New server, hMailServer and IMAP performance

Post by mattg » 2019-07-11 07:36

Try a HyperV Core build

They are free, with no time limits. No windows GUI, but hmailserver GUI works fine
HyperV Core is designed to host VMs, but you can run (non windows) stuff on the bare metal
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

Post Reply