Attachment upload speed

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
polarunion
Normal user
Normal user
Posts: 245
Joined: 2004-04-05 20:21
Location: Ottawa, Canada
Contact:

Attachment upload speed

Post by polarunion » 2004-06-25 06:33

I've been recieving some complaints from my users stating that it takes a really long time to emails with attachments. Upon further testing I would have to agree.

I've only once attached a large file 20+MB and it worked great back in version 3.0 i think it was, but it seems that it's slowed to a crawl - only about 5kbps!

I just tried to send a 500Kb file and it took almost 5 minutes to process. It never was slow like this. attachments over 1 mb now time out in squirrelmail. if sent with outlook they take well over 10 minutes..

I can transfer files to the same server via FTP at 1.5MB/s so why the big difference?

I thought it might have had something to do with php - but i tried an upload script and it too performed at the 1.5MB/s. I also attempted send emails with large attachments with outlook. It too worked at a crawl. Seems that there's something that's causing a bottleneck.

Is there any way to speed it up? There's nothing in the way of the pipe. It's clear and it can handle much much quicker speeds, but hmail seems to want to process such attachments at a mere 5kb/s.

Any ideas?

calvi
Normal user
Normal user
Posts: 65
Joined: 2004-03-17 23:34
Location: Melbourne, Australia

Post by calvi » 2004-06-25 06:44

Yes you are right,

I tried a 20m/b attachment direct to the hMailserver (outlook express -pop3) and its VERY slow.
Interestingly if you check the details while it is sending and hit stop it then sends it really quickly.
When you recieve the message same thing. Really slow, hit cancel then it comes in real quick.

Something fishy here.

PS. Make sure you test with a direct connection, ie not through ASSP or some other gateway. ASSP can be slow if it does not have enough horsepower.

polarunion
Normal user
Normal user
Posts: 245
Joined: 2004-04-05 20:21
Location: Ottawa, Canada
Contact:

Post by polarunion » 2004-06-25 07:20

wow.

I looked into this some more. Working with two computers in hand - the server and a laptop for test use, I noticed that the computer had completely exhausted itself in resources.

No wonder it wasn't performing. It was practically running itself into submission..


test pic one Image

This is the taskmangager as I clicked on the 35MB email. It took about a minute to open the message. Once the message was openned, I clicked on one of 4 mp3's at an average size of 6 megs.

Hmailserver was operating between 70-99% of cpu resources.

test pic two Image

This was the taskmanager list as I was downloading one of the mp3 attachments.

test pic three Image

This is the taskmanager just after it had downloaded.

I noticed PHP running at a rather high level. Have any other users been experiencing this? This worries me. It would take nothing for a couple of emails to be sent and to have this server completely derailed...

For the time being, i've returned squirrelmail attachments down to 2 MB.
Unfortunately this did not solve the slow upload times for message attachments.

polarunion
Normal user
Normal user
Posts: 245
Joined: 2004-04-05 20:21
Location: Ottawa, Canada
Contact:

Post by polarunion » 2004-06-25 07:23

Stats on my system...

WinXP Pro
Apache 2.49
PHP - latest 4.##
Activestate perl
hmailserver 3.2 beta3
assp 1.0.10 w/ clamav


all on pentium III 667Mhz 133FSB
512 Ram
80 Gig 7200RPM

Connection is a 3Mb down 1.5Mb up cable connection.

I now have about 40 users. most of them have gone to bed for the night so all stats are on a completely open server with no connections.
Last edited by polarunion on 2004-06-25 08:24, edited 1 time in total.

polarunion
Normal user
Normal user
Posts: 245
Joined: 2004-04-05 20:21
Location: Ottawa, Canada
Contact:

Post by polarunion » 2004-06-25 07:35

Ok well first of all, ASSP is FAR more CPU intensive than I once thought. It was actually doing something to those messages as they were being transferred to the server - maybe checking for viruses as it was downloading? I don't know but it caused a real bottleneck.

In fact - removing ASSP improved performance 10 000x. hmailserver is still using far more resources than necessary when sending or recieving the attachment, however it was not what was causing it to upload so slowly.

At first glance I'm going to attribute this speed leak to the ClamAV integration with ASSP - which is what I was using instead of the hmailserver clamav integration. I figured it would save more CPU power just by blocking it at the SMTP. I was very wrong on that.

I'll have to take that up with ASSP.

Martin, is there no way to just upload the attachment into the server as a whole part and allow the message to link to it? Wouldn't that be faster than having hmail encode it into a .hma file? I'm sure that takes a lot of CPU power along and in addition to that it must cause a bottleneck when the person is uploading the file. there's just no reason why i can't upload about as fast as i'm able to download. It takes me about 5 minutes for a 2mb attachment and have it uploaded to the server just from the home network! No wonder users are having troubles with attachment slowness.

That's why php_file_upload script that i have is able to accept large files at around 1.5MB/s - cause it simply copies the incoming data to a folder and does not rearange its structure.

Anyway BE WARNED anyone using ASSP + hmailserver.
If somebody sends your server a string of messages with large attachments, you can expect it to derail ASSP, bring your server to a standstill, and who knows, maybe even eat your computer whole. It's a monster when it deals with attachments.

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2004-06-25 12:18

polarunion wrote:Martin, is there no way to just upload the attachment into the server as a whole part and allow the message to link to it
It's the mail client that does this encoding. The mail client sends _exactly_ the data you see in the .hma file. hMailServer never encodes the data in the .hma files. It's Outlook Express, Squirrelmail and other mail clients that does the encoding. hMailServer simply receives the data and saves it in the data folder. So hMailServer should receive the data fairly fast. Doesn't it?

But when you click on an email in Squirrelmail with many large attachments, hmailserver must parse the data you see in the .hma file to determine which attachments exists. This is a very CPU-intensive task and the main bottleneck with the IMAP protocol.
calvi wrote: I tried a 20m/b attachment direct to the hMailserver (outlook express -pop3) and its VERY slow.
If you send a large 20 mb attachment using Outlook Express, OE must first encode the attachment, create a mime message (exactly the data found in the .hma file). On my computer, I get the following times with OE:

Time for OE to compose the data to send: 20 seconds
Time for OE to really send the data to hmailserver: 2-3 seconds
Time for OE to move the message to the Sent items folder: 5 seconds.

You can't measure the time from you click "Start" til the message is delivered since this time includes the time it takes for Outlook to encode the data.

Anyway, I'll try to change a few numbers in the TCP/IP code today to see if I can increase the speed...

calvi
Normal user
Normal user
Posts: 65
Joined: 2004-03-17 23:34
Location: Melbourne, Australia

Post by calvi » 2004-06-25 14:38

Martin,

I will try this out myself. You may be right, perhaps I hit cancel just as my computer finished encoding the message. Won't be able to try it until monday though.

My apologies if this is the case.

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2004-06-25 15:54

Hehe no probs. :)
Let me know how it goes.

polarunion
Normal user
Normal user
Posts: 245
Joined: 2004-04-05 20:21
Location: Ottawa, Canada
Contact:

Post by polarunion » 2004-06-25 16:10

heh, interresting. learning something every day.


but still when you compare the speeds of uploading a file onto the server via ftp and sending an already endoded message with Outlook, outlook takes much much longer.

For example.

Test: 25mb data

upload with ftp: 2 minutes

upload with outlook to hmailserver (after message is encoded): 15 minutes.

Any ideas where is the difference coming from?

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2004-06-25 16:36

I did some tests with Outlook Express. One 50 mb file took about 10 seconds to send to the mail server. Could you try sending your email using Outlook Express instead of Outlook?

How do you measure what time it takes? The easiest way is to turn on SMTP logging and check the time difference between "SENT: 354 ok send! end with <crlf>.<crlf>" and "SENT: 250 message queued". Then you'll se exactly how long time the data transfer takes.

Are you using some firewall or gateway or anti virus scanner between Outlook and hMailServer?

polarunion
Normal user
Normal user
Posts: 245
Joined: 2004-04-05 20:21
Location: Ottawa, Canada
Contact:

Post by polarunion » 2004-06-25 16:55

ok, will do. I'll test again with the logs to give you exact times.

So far my times were given just by looking at my watch. Not accurate, but I just wanted to point out the discrepency.

Right now I've removed ASSP entirely because it chokes on large base 64 attachments. So far I've found no solution to this and it seems this is a problem that has not been addressed by ASSP.

http://sourceforge.net/forum/message.php?msg_id=2216708

I currently only have hmail listening on 25 with clamav virus scanning. I've tested it already with anti virus on and off. there was no difference. clam av scans the attachments it seems, after its been uploaded to the server.

Also there is no firewall. All ports 25 143 110 are NAT to the server.

So for the test, I'm sending all large emails from a computer on the same network as the server. I am actually using outlook express. I should have been more accurate there.

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2004-06-25 17:58

To do what it's supposed to do, ASSP must decode large MIME message containing attachemnts encoded with base64 and other encodings. This decoding is quite cpu intensive and requires a lot of memory. And ASSP is written in Perl if I'm not wrong. And perl is not the fastest language in the world...

Let me know how long it takes with Outlook Express and without ASSP activated.

polarunion
Normal user
Normal user
Posts: 245
Joined: 2004-04-05 20:21
Location: Ottawa, Canada
Contact:

Post by polarunion » 2004-06-25 20:37

so far this is what I've come up with..

These are completely accurate tests - no bull.

email being sent to user on hmailserver from user on the same server.. sender is on home network with server - but is sending it to outside which is being routed back in.

ASSP is disabled. Using hmailserver 3.2 beta 3 with clamav plugin.

email has empty body, but includes a single mp3 with a size of 21.4 MB


First test:

Squirrelmail 1.4.3a

START > "SMTPD" 5044 "2004-06-25 14:06:49"
END > "SMTPD" 5044 "2004-06-25 14:07:10"

Total time = 1:21

Outlook Express

START > "SMTPD" 4592 "2004-06-25 14:14:14"
END > "SMTPD" 4592 "2004-06-25 14:26:54"

Total Time = 12:30

I just downloaded thunderbird. going to try that as well..

Things to note: While sending the message from my laptop to the server, php ran exhaustively while sending with SM. hMailserver also ran very high as this seemed to be a very intense project for it.

When sending with OE, hmail barely broke a sweat - though worked at about 12 times slower than SM. When the message was being queued at the end of the send, hmail spiked to 100% CPU load for about 10-15 seconds before delclining again to normal levels.

Ram used by hmailserver during the process - ~91.5 Mb and declined to ~85.5.

That's a lot of ram!

Hope this helps a bit.

********update

Thunderbird Trial

START > "SMTPD" 4172 "2004-06-25 14:43:09"
END > "SMTPD" 4172 "2004-06-25 14:44:22"

Total Time = 1:11

Fastest time yet, however I also observed hmailserver running at 145mb ram. it later droped to 85.

It seems outlook is doing something funky. I don't like this one bit.

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2004-06-25 21:56

Don't understand what you mean with this:
"Email being sent to user on hmailserver from user on the same server.. sender is on home network with server - but is sending it to outside which is being routed back in."

What CPU speed?
Is there any anti virus software running on the computer?

polarunion
Normal user
Normal user
Posts: 245
Joined: 2004-04-05 20:21
Location: Ottawa, Canada
Contact:

Post by polarunion » 2004-06-25 22:21

sorry to confuse you... the details weren't really important but here's another go at what i tried to say.

what i was trying to say there was - user1@mydomain was sending to user2@mydomain > both hosted on the same server.

User1@mydomain is sending mail from the same network as the server. The message is still being sent over the internet via that address - so this is not an intranet email server.

Sorry, I tend to make things more complicated for you when I try to be extremely precise.

Read up - my server info is there. Pentium III MMX Coppermine 667Mhz 133FSB

lastly - only norton is running. it uses less than 2% system resources

User avatar
Jason Weir
Normal user
Normal user
Posts: 58
Joined: 2004-02-02 23:41
Location: Chichester, NH
Contact:

Post by Jason Weir » 2004-06-25 23:48

I'm not having any large attachment problems with ASSP. I started a new thread in the off topic area to see if we can get to the bottom of this. ASSP works so good, lets see if we can get any bugs worked out

http://www.hmailserver.com/forum/viewtopic.php?t=422

Thanks
Jason

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2004-06-26 01:51

On my computer it sends a 25 mb attachment in a few seconds. However, I will create a new version this weekend which will have better performance in the SMTP part of the server.

calvi
Normal user
Normal user
Posts: 65
Joined: 2004-03-17 23:34
Location: Melbourne, Australia

Post by calvi » 2004-06-26 02:33

Sorry Martin,

you can remove tracker 149&150 I ran some proper tests...

http://www.hmailserver.com/forum/viewtopic.php?t=422

summary...

1. Sending through ASSP(no Clam)>Worldmail running on ol' pentium = 17 minutes 25s. (don't even bother with clam on, she can't take it captain).
2. Sending direct to Worldmail running on ol' pentium = 40s
3. Receiving from worldmail running on ol' pentium = 25s.
4. Sending through ASSP(no Clam)>hMailserver on Sledgehammer = 21s
5. Sending through ASSP(Clam)>hMailserver on Sledgehammer = 2min 44s.
6. Sending direct to hMailserver on Sledgehammer = 10s.
7. Recieving from hMailserver on Sledgehammer = 3s.

Now bear in mind that all the send times include encoding in outlook and moving the email their respective folders, this was consistent at around 6-7s.


but speed increases are of course always welcome :).

nwkit
Normal user
Normal user
Posts: 133
Joined: 2004-04-19 03:57
Location: Canada

Post by nwkit » 2004-06-26 03:32

i sent 3 files that was 20mb to my hmailserver from a different network..so it was using the internet the whole way...my upload speed was checked to be at my max...around 67kb/s...and i checked my server containing hmailserver..and it was still running smoothly...no big jump of ram usage from hmailserver...it just took a while to send cuz the file was sooo big

the only thing i noticed was that perl was using 35mb ram and apache using like 24mb....but i just checked again..like after several hours i sent the large test email, and the ram usage is at the same place...so not sure if that is relevant or not...

polarunion
Normal user
Normal user
Posts: 245
Joined: 2004-04-05 20:21
Location: Ottawa, Canada
Contact:

Post by polarunion » 2004-06-26 07:31

yes, i too experienced elevated ram useage which did not decrease after use.

Checking server resources tonight I noticed hmail at 143Mb. After restarting the service it returned to 4Mb.

Has anyone else looked at the three tests I've performed and were they able to recreate this?

I'm worried of the 12 minutes it takes for OE to handle a message, when it only takes 1.3 for SM and 1.1 for thunderbird. Something isn't right.

Just empty your log for the day and enable smtp filtering. Send a 20 meg file and check it after it completes it send with SM, OE and another mail ap such as thunderbird. Compare the times of the first connection to the 'message queued' line. That will give you a precise difference in time. People should be getting the same ratio as what I've been getting - with deviations dependant on your network speed and server power.

Lastly, compare these with simple file transfer speed capabilities.

Everything up to this point has been relative. If anyone has any other suggestions on what I might try to improve upload speeds, I'm all ears. Thanks for your efforts all. I'm sure we can fix this.

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2004-06-26 12:40

I have dome some changes that will decrease memory usage. A new version will be put up in the weekend or on monday.

In the next version, the "Message queued" line includes the time in milliseconds the data transfer took. Will be easier to get exact measurments of the time with this.

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2004-06-27 17:13

The new version is up. Would be interesting to compare some numbers.
On my computer, a delivery of a 25 mb attachment takes about 8 seconds.

There's a new setting in hmailserver.ini:

Code: Select all

[Settings]
SocketBufferSize=40000
This settings says how many bytes the internal buffer should be when sending and receiving data. If it's not set, the default value is 40 000. If I increase this value from 40 000 to 100 000, the above delivery takes 5 seconds instead of 8. Changing the value requires server restart.

(I'm not sure what the best default value for this setting is yet. The higher the default value, the more memory will the server use. )

The new version also includes a small feature in the logs. The message queued now looks like this:

Code: Select all

250 Message queued (6.159 seconds)
(The time is the time it took for the data to be delivered from the client to the server.)

I suggest you try the new version and try to send the message with OE again. And check the Message queued line for the exact delivery time.

polarunion
Normal user
Normal user
Posts: 245
Joined: 2004-04-05 20:21
Location: Ottawa, Canada
Contact:

Post by polarunion » 2004-06-28 18:44

wow, that's pretty cool that you've done that. I'll try it later tonight.. Thanks for the upgrade and for taking the time to look into this. I'll keep you up to date on how it's working for me.

polarunion
Normal user
Normal user
Posts: 245
Joined: 2004-04-05 20:21
Location: Ottawa, Canada
Contact:

Post by polarunion » 2004-06-30 21:28

UNBELIEVABLE!


Update:::


I've just tested the times for the latest hmailserver release and received great times and great results..

To get these results I installed Perl 5.6 instead of the 5.8 - couldn't tell you if this makes a difference because I also upgraded ASSP to Calvi's modded ASSP and I'm now using the HMail 25 build.

using the same attachment as the tests before along with the same Squirrelmail and Outlook E. Ii was able to attain the following results.

Hmailserver build 25 - ClamAv Scan
No ASSP

OE - 14.481 Seconds - down from 10 minutes!!!!
SM - 19.358 Seconds - down from a minute and a half.

John Calvi's ASSP Mod SMTP Proxy - No AV scan
Hmailserver build 25 - ClamAv Scan

OE - 87.806 seconds
SM - 90.600 seconds



Memory usage was constant at about 4.5Mb-7.5Mb

I can't tell you how pleased I am with these results. I can't thank Martin/John enough.

We just have to work on virus protection on hmail's side because I think it would handle it more efficiently than ASSP's integration with ClamAv which as we all know is notoriously slow.

Cheers - Pat

Post Reply