IMAP performance on LARGE (10,000+ messages) folders...

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
JHMvP
New user
New user
Posts: 4
Joined: 2012-12-05 13:44

IMAP performance on LARGE (10,000+ messages) folders...

Post by JHMvP » 2012-12-05 15:09

Coming from Linux-environments, I've installed hMailserver on a Windows-box and actually I'm surprised with the simplicity of it; both file structure and installation! And now I've got a very good/fast/stable running hMailserver for one certain environment;
- 4 users working on 1 and the same mail-account/-box :!:
- Mail-account/-box now contains 25,000 messages
- Whereof 2 (archive) folders with 10,000+ messages

hMailserver runs with 5% CPU utilisation (with regular spikes up to 35%) and 300Mb RAM usage (steady).

There's only one catch; when one user opens one of the archive-folders, it first takes 5 sec. to open de folder and show the contents and then it takes approx. 2 minutes to open the message. During this time the hMailserver runs at 95-100% CPU but RAM stays at 300Mb and disk I/O is next to none.

Again, this happens only with these two 10,000+ archive-folders and also when only one user is logged on. I've tested folders up to 5,000 messages and these are eye-blink-fast!

hMailserver is installed with all default options (no spamfilter, no virusscanner, default performance-settings, indexing turned on, internal MSSQL CE database) on a;
- Pentium 4 531 (3Ghz, 1 core, 2 threads)
- 512 Mb RAM
- 500Gb PATA harddisk
- 1Gbps NIC
- Windows XP Pro, SP3, fully updated until 01-12-2012
- Nothing else installed but hMailserver

Clients are on local LAN with;
- Intel i5 laptops, 2Gb RAM
- 1Gbps NIC
- Windows 7 with Windows Live Mail client
- Windows Live Mail does only IMAP to the server with sync'ing truned off

In the email-client IMAP-sync is turned off for all folders because the smaller folders are extremily fast and sync'ing of the two large folder takes forever! And regular checking for new messages during working time, also. And, I've got another 20 folders with archived email (also 5,000 to 10,000+ emails each) that should be moved to the hMailserver!!! :shock:

So, now my questions;
1. Should I alter any of the Performance-settings?
2. Should I move from MSSQL CE to MySQL?
3. Should I upgrade only the CPU to avoid problems with 10,000+ message folders? If yes, would an Intel Core2 Duo (3Ghz, 2 core, 2 threads) processor do?
4. Should I upgrade RAM, although it seems that no more than 300Mb is being used?
5. Should I move to SATA or RAID although there's next to no disk I/O?
6. Any other suggestions?

Thank you all in advance!

User avatar
Caspar
Senior user
Senior user
Posts: 378
Joined: 2008-09-08 11:47
Contact:

Re: IMAP performance on LARGE (10,000+ messages) folders...

Post by Caspar » 2012-12-05 15:24

JHMvP wrote:So, now my questions;
1. Should I alter any of the Performance-settings?
2. Should I move from MSSQL CE to MySQL?
3. Should I upgrade only the CPU to avoid problems with 10,000+ message folders? If yes, would an Intel Core2 Duo (3Ghz, 2 core, 2 threads) processor do?
4. Should I upgrade RAM, although it seems that no more than 300Mb is being used?
5. Should I move to SATA or RAID although there's next to no disk I/O?
6. Any other suggestions?

Thank you all in advance!
1. Not as far as I know, but I don't generly tweak those settings, maybe someone else can answer this
2. YES! with these amounts don't use MSSQL CE, use either MSSQL (express) or MySQL. Since most people here have better experience with MySQL I would recommend that
3. upgrading the CPU might also help alot especialy since the Pentium 4 are realy old processors, but this is only a *possibility*
4. it should be enough, although for the server I would recommend using 2 GB minimum.
5. I would recommend using either a RAID 1 or a RAID 5 depending on if you want to upgrade disk size with the same type of disks. Especially for server I always would recommend using either one of those, because in case of hdd failure, you can still keep going.
6. not at this moment!
If you have strange problems or errors use the log analyzer! http://log.damnation.org.uk
Join us on IRC! http://hmailserver.com/irc_fullscreen.php

JHMvP
New user
New user
Posts: 4
Joined: 2012-12-05 13:44

Re: IMAP performance on LARGE (10,000+ messages) folders...

Post by JHMvP » 2012-12-05 15:33

Caspar wrote:1. Not as far as I know, but I don't generly tweak those settings, maybe someone else can answer this
2. YES! with these amounts don't use MSSQL CE, use either MSSQL (express) or MySQL. Since most people here have better experience with MySQL I would recommend that
3. upgrading the CPU might also help alot especialy since the Pentium 4 are realy old processors, but this is only a *possibility*
4. it should be enough, although for the server I would recommend using 2 GB minimum.
5. I would recommend using either a RAID 1 or a RAID 5 depending on if you want to upgrade disk size with the same type of disks. Especially for server I always would recommend using either one of those, because in case of hdd failure, you can still keep going.
6. not at this moment!
Thanks for your FAST reply!

But I did mention that RAM usage is 300Mb steady all over the day and that disk I/O is next to none. Regarding my large folder issue, upgrading these parts won't make any difference then, I suppose? And what increase in performance is to be expected from changing databases (according to my environment)?

^DooM^
Site Admin
Posts: 13862
Joined: 2005-07-29 16:18
Location: UK

Re: IMAP performance on LARGE (10,000+ messages) folders...

Post by ^DooM^ » 2012-12-05 16:06

MSSQLCE is meant for small home setups with a few emails a day, it's not powerful enough to be used high capacity. I would highly recommend using MySQL or MSSQL dedicated if you have the capacity. I would also enable Message Indexing in hMail as well (Not while using mssqlce).
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

JHMvP
New user
New user
Posts: 4
Joined: 2012-12-05 13:44

Re: IMAP performance on LARGE (10,000+ messages) folders...

Post by JHMvP » 2012-12-05 17:01

^DooM^ wrote:MSSQLCE is meant for small home setups with a few emails a day, it's not powerful enough to be used high capacity. I would highly recommend using MySQL or MSSQL dedicated if you have the capacity. I would also enable Message Indexing in hMail as well (Not while using mssqlce).
Eh...so MSSQL CE doesn't support Message Indexing? But hMailserver says all email is indexed! It that screen referring to something else? If yes, does Message Indexing benefit my environment?

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: IMAP performance on LARGE (10,000+ messages) folders...

Post by Bill48105 » 2012-12-05 18:28

mssql ce is meant for testing & personal use. It is not designed for or licensed to be used any other way.

indexing can be done with mssql ce but if your bottleneck is mssql ce you're just adding more load by enabling indexing so it's not recommended you do it. use a real database.

The huge cpu load is likely hmail accessing the hm_messages table but could be lack of memory to cache the EML files in the Data tree. Odds are a real database & more memory will help especially if you enable hmail's message indexing (which caches file headers in the database) but you can't really test that theory until you migrate to a real database & get some more memory available to Windows.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

User avatar
dzekas
Senior user
Senior user
Posts: 2486
Joined: 2005-10-13 21:28
Location: Lithuania

Re: IMAP performance on LARGE (10,000+ messages) folders...

Post by dzekas » 2012-12-05 20:15

JHMvP wrote:
^DooM^ wrote:MSSQLCE is meant for small home setups with a few emails a day, it's not powerful enough to be used high capacity. I would highly recommend using MySQL or MSSQL dedicated if you have the capacity. I would also enable Message Indexing in hMail as well (Not while using mssqlce).
Eh...so MSSQL CE doesn't support Message Indexing? But hMailserver says all email is indexed! It that screen referring to something else? If yes, does Message Indexing benefit my environment?
It supports them. You just would not want to click on that indexing checkbox in some hmailserver versions. :) Maybe newer versions fixed it.

Please note that MSSQL CE is not database server. It is one database file and functions to use it.

JHMvP
New user
New user
Posts: 4
Joined: 2012-12-05 13:44

Re: IMAP performance on LARGE (10,000+ messages) folders...

Post by JHMvP » 2012-12-06 12:47

Thank you for your suggestions!

Just a last question; what is the most straightforward way to change a hMailserver (server or just dataset) from MSSQLCE to MySQL?

Thanks in advance!

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: IMAP performance on LARGE (10,000+ messages) folders...

Post by Bill48105 » 2012-12-06 17:08

JHMvP wrote:Thank you for your suggestions!

Just a last question; what is the most straightforward way to change a hMailserver (server or just dataset) from MSSQLCE to MySQL?

Thanks in advance!
If your Data tree is not too large (15GB on newer 5.4 & 1.5G on previous versions) then you can do a backup then restore, switching the database type in between.

On newer 5.4 build I added ini settings to enable restore of JUST the database but it's not complete & has caveats. (Main thing is you rename your Data folder BEFORE restoring then restore back. Not ideal but it works.) The reason is hmail wipes the Data tree & user mailboxes during restore so obviously you need to have complete backup and be extremely careful what you are doing. But the special ini settings at least give a method to restore just database that wasn't possible before.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

Post Reply