Performance issues sending/receiving email

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
martijng
New user
New user
Posts: 4
Joined: 2020-06-09 15:28

Performance issues sending/receiving email

Post by martijng » 2020-06-09 15:51

We use hMailserver for our company for a few years now. We only use it for sending email and signing it with DKIM. In general this works perfect.

We only face some performance issues. The server can only handing around 4.000 messages per hour. Most of the time this is not a problem, but once in a while we have a big mailing (like >50.000) where it takes really too long.

As far as I can tell the bottleneck is not in sending, but for the server to receive/accept the email. The software that sends it (with and c# smtp client) takes as long as until the last one is delivered by hMailserver.
Before we used IIS smtp, which took about a second to receive 50.000 emails with the same server.

If I look in the delivery queue there are normally max 4 emails with "as soon as possible" waiting.

The server itself is also totally not busy at all. <1% cpu <1% hdd activity <1% network "1% memory by hMailServer".

There is no virusscanner, anti spam or anything else configured. Only DKIM signing.

Performance settings:
Cache Enabled
TTL eveything 60 seconds. Hit rate at Domain is 98.

Threading
command: 15
delivery threads: 10
Max task threads: 15
Priority: normal

Indexing: off

Can somebody help me finding out how to improve the performance?
And is there some kind of trick how to debug it? As this is our production server I would like to test it on a test server, but is there a way to test performance without actually sending email?

User avatar
RvdH
Senior user
Senior user
Posts: 1105
Joined: 2008-06-27 14:42
Location: Netherlands

Re: Performance issues sending/receiving email

Post by RvdH » 2020-06-09 16:47

Tried reading documentation on that matter? Doesn't seem so as you're using defaults...those settings are pretty well documented
https://www.hmailserver.com/documentati ... erformance
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
mattg
Moderator
Moderator
Posts: 21036
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Performance issues sending/receiving email

Post by mattg » 2020-06-10 02:05

So your sending software sends 50 000 individual emails, and not BCC?
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
johang
Senior user
Senior user
Posts: 320
Joined: 2008-09-01 09:20

Re: Performance issues sending/receiving email

Post by johang » 2020-06-10 08:30

martijng wrote:
2020-06-09 15:51
We use hMailserver for our company for a few years now. We only use it for sending email and signing it with DKIM. In general this works perfect.

We only face some performance issues. The server can only handing around 4.000 messages per hour. Most of the time this is not a problem, but once in a while we have a big mailing (like >50.000) where it takes really too long.

As far as I can tell the bottleneck is not in sending, but for the server to receive/accept the email. The software that sends it (with and c# smtp client) takes as long as until the last one is delivered by hMailserver.
split workload into more sessions at c#client ( of course you have to adjust acordingly at hmailserver side )
Before we used IIS smtp, which took about a second to receive 50.000 emails with the same server.
with same server ? ???
Before = c#client --> IISsmtp --> hmailserver
now = c#client -----------------> hmailserver

put back the IISsmtp funtionality until you rsolved what you want to resolve .. ( or is IIS the solution ? )
If I look in the delivery queue there are normally max 4 emails with "as soon as possible" waiting.

The server itself is also totally not busy at all. <1% cpu <1% hdd activity <1% network "1% memory by hMailServer".

There is no virusscanner, anti spam or anything else configured. Only DKIM signing.

Performance settings:
Cache Enabled
TTL eveything 60 seconds. Hit rate at Domain is 98.

Threading
command: 15
delivery threads: 10
Max task threads: 15
Priority: normal

Indexing: off

Can somebody help me finding out how to improve the performance?
And is there some kind of trick how to debug it? As this is our production server I would like to test it on a test server, but is there a way to test performance without actually sending email?
there is no trick to debugging.. it is old fashioned leg work, takes time and testing, well you decide if you should use a test system (recomended of course) or if you want to work on your production system.. to test performance ( i would say behaviour ) you need to do what is supposed to happen to and in all of the links in the chain ... so i would say you need to actually send something through ..
___________________________________________________________end of the line
spam filter appliance gateway: www.mailcleaner.org

User avatar
RvdH
Senior user
Senior user
Posts: 1105
Joined: 2008-06-27 14:42
Location: Netherlands

Re: Performance issues sending/receiving email

Post by RvdH » 2020-06-10 09:44

As said, he only uses the hmailserver for sending out (and DKIM sign outgoing) e-mails, what this tells me he can basically use every (or at least plenty more) of the available system resources, I would suggest you increase Delivery threads to 100 or even more, 50000 mail are quite some mails, but you have to find the sweetspot for this yourself
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
mattg
Moderator
Moderator
Posts: 21036
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Performance issues sending/receiving email

Post by mattg » 2020-06-10 11:26

Also with the IIS solution, I'm guessing that a drop folder was used, not a SMTP connection
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
RvdH
Senior user
Senior user
Posts: 1105
Joined: 2008-06-27 14:42
Location: Netherlands

Re: Performance issues sending/receiving email

Post by RvdH » 2020-06-10 11:59

mattg wrote:
2020-06-10 11:26
Also with the IIS solution, I'm guessing that a drop folder was used, not a SMTP connection
+1

@martijng, do you authenticate the user that sends the mails or did you exclude that ip(range) from virus/spamchecks?
if not. Unauthenticated mail is always checked for spam if not disabled by ip(range) so you possibly could gain some performance excluding spam checks
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

martijng
New user
New user
Posts: 4
Joined: 2020-06-09 15:28

Re: Performance issues sending/receiving email

Post by martijng » 2020-06-10 13:55

Whow I never expected such great response. Thank you so far already.
So your sending software sends 50 000 individual emails, and not BCC?
Correct. Those are all personalized emails.
with same server ? ???
Before = c#client --> IISsmtp --> hmailserver
now = c#client -----------------> hmailserver
With the same server yes.
Before = c#client --> IISsmtp
Because we wanted to improve the delivery rate we needed DKIM support. So we turned off IIS SMTP and switched to hMailserver
Now: c#client --> hMailserver
IIS Smtp is not a solution at all, just the old way of sending without DKIM support.

Both used an SMTP client. Therefore the client settings were not adjusted, just the server software itself.
do you authenticate the user that sends the mails or did you exclude that ip(range) from virus/spamchecks?
if not. Unauthenticated mail is always checked for spam if not disabled by ip(range) so you possibly could gain some performance excluding spam checks
No I don't authenticate anything. Our servers, including localhost, are included in a whitelist (IP Ranges tab).

ClamAV and CalmWin are turned off.
Unauthenticated mail is always checked for spam if not disabled by ip(range) so you possibly could gain some performance excluding spam checks
Anti virus and anti spam were turned on in all the ip ranges. I turned them off.
I would suggest you increase Delivery threads to 100 or even more, 50000 mail are quite some mails, but you have to find the sweetspot for this yourself
I set the delivery threads to 100 and see what happens.

Also I will try to setup a test environment, including 'fake' smtp receive servers.

It is not a high urgent problem, as for now we just know some mailings can take long and take that into account for the planning. A small problem is that we don't have high mailings that often, so it is hard to test.

Thank you again for your support so far. I will keep you informed if I have done some tests.

mikedibella
Normal user
Normal user
Posts: 227
Joined: 2016-12-08 02:21

Re: Performance issues sending/receiving email

Post by mikedibella » 2020-06-10 17:07

My hypothesis is that your system is I/O bound. The large number of messages is saturating disk queues and the whole system slows to a crawl as virtual memory operations compete with data reads and write for the I/O channel . Check the relevant Windows performance counters to verify. To fix, consider using seperate disks for system and data, and if they are virtual disks, put them on seperate physical stores. If your server is on bare metal, consider putting the HMS program directories on an SSD.

martijng
New user
New user
Posts: 4
Joined: 2020-06-09 15:28

Re: Performance issues sending/receiving email

Post by martijng » 2020-06-11 20:09

After a whole day of research I kind of found the problem. I never had benchmarked the email server before, because I couldn't think off a way without sending them to the open world.

To start with. In this topic I only tested sending email from our Test server to the production hMailserver, as lately we had a large mailing (50.000) so I had the time to monitor it.
For large amounts of emails we always run our application from the test server (and sends through the prod hMailserver) as this normally used to give best performance (our software also communicates with a database in the same physical location).

TL;DR;
It's not hMailserver, it's probably Windows Firewall on the sending server. Why? I don't know yet.

Long version how I tested and the results (quite interesting results if you ask me)
To be able to do some tests I wrote two applications:
1. BulkEmailSender (short BS): An application that sends dummy emails, multipart (plain and html), with 1000 random lorum ipsum words, to email addresses: test{no}@localhost.
2. EmailToEmlReceiver (short ER): An application that acts as a SMTP server, listens on port 9999 and stores all incomming emails to eml files.

I tested with 3 computers/servers:
Test server (in a datacenter)
Prod server (in a datacenter)
Workstation (local office, 18ms away from the servers)
The servers are connected to the same switch.

I added a route on Test (under Protocols > SMTP > Routes) named localhost and added the target SMTP Host localhost on port 9999 (so ER will receive those emails).

Now the results (all using BS for sending):

I started testing the test server
(so all emails go first to hMailserver and will directly be delivered to ER)
Workstation --> test = 250 sec
test --> test = 73 sec
test --> test directly to ER, skipping hMailserver = 7 sec
A lot faster then I expected as the hardware from Test is nothing compared to Prod. Here I don't see the slow results where my topic is about.


Workstation --> prod = 83 sec (not bad at all. where is the problem?)
prod --> prod = 9.2 sec (whow!)
test --> prod = 844 sec (what is happening here? At least able to reproduce the problem I described)

then let's test the other way around:
prod --> test = 72 sec (so test is a lot faster?)
prod --> test BR = 8,0 sec
maybe something strange in the switch or something. So let's test more.
I now did the same configuration on Prod and ran ER there, just in case.
test --> prod BR = 3,3 sec (why this fast, doesn't make sense, same source, same destination, but instead of hMailserver ER)

This is the point where I did something I shouldn't: switch off both Windows Firewalls. (the servers are also behind a hardware firewall, so not that bad)
test --> prod = 11,8
Windows Firewall Prod on:
test --> prod = 12,4
Test firewall on again:
test --> prod = around 350 sec.

So the conclusion, sending from test to prod to hMailserver is extremely slow with the Windows Firewall on Test on. Sending instead to ER is super fast using the same Windows Firewall.

I still don't have any clue what causes this behavior. But it is unlikely that hMailserver is the cause, or it does something that the Windows Firewall from Test doesn't like. I will look into it another time.
Thank you all for your help! If I will find the root cause I will post it here for who is interested.

martijng
New user
New user
Posts: 4
Joined: 2020-06-09 15:28

Re: Performance issues sending/receiving email

Post by martijng » 2020-06-17 15:06

I tracked down the issue further and solved the performance problem.

TL;DR;
Problem related to: https://www.hmailserver.com/forum/viewt ... p?p=221287. LLMNR was the main problem in combination with hMailServer (Server 2012 as sender, Server 2012 R2 as receiver), both mostly default Windows server settings.

I used WireShark to examine the total flow.

With the firewall off, there was a LLMNR package received at the sender (multicast), just before the QUEUED command and replied by the sender.
With the firewall on, there was a LLMNR package received at the sender, then a timeout (I think of 400ms), then another request, another timeout, then QUEUED.
If I did the same to my own ER (SMTP dummy server), there is no LLMNR package at all, so hMailServer directly or indirectly uses LLMNR if used in a local netwerk.

With disabeling LLMNR (and Netbios over TCP/IP as both can can be abused) the package is gone and everything is fast (7,8sec for 1000 emails).

Thank you all for your help :)

User avatar
johang
Senior user
Senior user
Posts: 320
Joined: 2008-09-01 09:20

Re: Performance issues sending/receiving email

Post by johang » 2020-06-17 17:35

martijng wrote:
2020-06-17 15:06
I tracked down the issue further and solved the performance problem.

TL;DR;
Problem related to: https://www.hmailserver.com/forum/viewt ... p?p=221287. LLMNR was the main problem in combination with hMailServer (Server 2012 as sender, Server 2012 R2 as receiver), both mostly default Windows server settings.

I used WireShark to examine the total flow.

With the firewall off, there was a LLMNR package received at the sender (multicast), just before the QUEUED command and replied by the sender.
With the firewall on, there was a LLMNR package received at the sender, then a timeout (I think of 400ms), then another request, another timeout, then QUEUED.
If I did the same to my own ER (SMTP dummy server), there is no LLMNR package at all, so hMailServer directly or indirectly uses LLMNR if used in a local netwerk.

With disabeling LLMNR (and Netbios over TCP/IP as both can can be abused) the package is gone and everything is fast (7,8sec for 1000 emails).

Thank you all for your help :)

Thank you for reporting the cause of problem being LLMNR
___________________________________________________________end of the line
spam filter appliance gateway: www.mailcleaner.org

Post Reply