hMailServer 4.3, build 227

Post by **martin** » 2006-08-05 17:03

This build tries to address the problem with randomly stopping SMTP servers. Since I have not been able to reproduce it myself thought, it's hard to say whether it really helps.

Some things are strange about this issue. First, hMailServer checks the return values for all Windows API calls that has to do with TCP/IP and Windows doesn't return any errors. Secondly, it only happens for SMTP, even though it's the same code that handles SMTP as POP3 and IMAP.

Since it only happens with SMTP connections, I think it has something to do with how the connections are closed. POP3/IMAP connections are almost always disconnected properly by the client, while SMTP connections are often disconnected improperly, for example by spammers.

I've done the following changes in this build:

- I've simplified the code that disconnects timeouted clients. hMailServer no longer says "Goodbye" to clients before disconnectnig them. It's likely that a client which has timed out has lost to the connection to hMailServer, and then it doesn't matter whether hMailServer sends a goodbye or not.

- When a socket is being closed, hMailServer now explicitly tell Windows how it should be done. Previous builds relied on Windows to close the socket the "default way".

So I appreciate if you upgrade to this build, turn on logging and send the logs to me at martin@hmailserver.com if the problem occur.

Build available here:
http://dev.hmailserver.com/hMailServer-4.3-B227.exe

mega · Post by **mega** » 2006-08-07 02:53

last 2 versions, when I try to turn on greylisting for my domain I get this error

Run-time error '-2147220503 (800403e9)'

Method '~' of object '~' failed

clicking ok, exits the admin tool

Post by **martin** » 2006-08-07 12:19

Just tested but it works for me. Are you using build 227?
Do you get the error when you click in the grey glisting field, or when you click Save in the dialog?
Is it possible for you to change other properties in the domain?

mega · Post by **mega** » 2006-08-07 14:04

build 227

when I click save, if I do nothing, but click the domain and click save I also get the same error

I have two domains, and both cause the error

I thought maybe the update to 227 failed, but all the tables look ok

mega · Post by **mega** » 2006-08-07 16:16

update on this, I cant update any user's either, same error

koncept · Post by **koncept** » 2006-08-07 17:07

I am not having any problems with these. The only thing I am wondering is if we can expand the plus addressing character list....but at the same time the list here is also the least likely to be used in a real address.

Post by **martin** » 2006-08-07 17:12

mega, I've tried this but don't get the error. What database server are you using? The two last builds has not been teste din SQL Server, so perhaps it's that? Do you get any error in the hMailServer error log?

Edit: Just read in the other thread (you appear to have posted to the wrong thread) that you managed to solve it by re-running some of the update statements. What database server are you using?

koncept:
Well, the idéa was to limit the list to characters that are unlikely to be used in real email addresses. I felt that being able to enter anything, such as the character "a", could cause an enormous amount of confusion.

Which character(s) did you think about?

koncept · Post by **koncept** » 2006-08-07 17:16

i was thinking the underscore and period but those seem like they may be used too frequently in an email address. i completely agree with not allowing any character to be used. I am currently using 227 on ms sql with out those issues mentioned, however its not a very active server...

Post by **martin** » 2006-08-07 17:17

I think I'll skip underscore and period as valid plusaddressing characters While it may be nice for some users, I think it would be confusing for the remaining 95%

Mega posted about those issues in another thread. Apparantly the upgrade from the earlier version seems to have failed which caused the problem. (I've updated my previous post for some follow up questions on this).

mega · Post by **mega** » 2006-08-07 21:20

I coulda swore I replied to this

I ran the updates by hand, from the update sql log

now everything works, no errors, using mysql 5.0.22

Post by **martin** » 2006-08-07 21:22

Yes, you answered to this, but you answered in another thread, not in this one.

Mega, what database server and version are you using? I want to figure out why your upgrade failed in the first place. And if you remember what version you upgraded from, that would be great as well.

mega · Post by **mega** » 2006-08-07 21:24

had huge logs, didnt realize they were there

<log snippet removed by martin...>

TONS more

Post by **martin** » 2006-08-07 21:30

Yeah, those are a side effect of the failing upgrade. What version of MySQL are you using? Are you using the internal or an external?

mega · Post by **mega** » 2006-08-07 21:49

external, 5.0.22

Post by **martin** » 2006-08-07 21:54

Do you remember what hMailServer version you used before the upgrade?

mega · Post by **mega** » 2006-08-08 04:30

I do not, it was 3 months or so old

cveillon · Post by **cveillon** » 2006-08-08 19:43

Martin, good news. Haven't had a SMTP lockup in three days. However, during those three days I was restarting the hmailSQL and hmail server every day at 6AM local. I've disabled my cron batch file and will keep you abreast of my progress.

with best regards,
Chuck

Post by **martin** » 2006-08-08 19:45

Ok.. well, it's a bit cheating to automatically restart the server every day - Hard to spot this problem then.

GlenC · Post by **GlenC** » 2006-08-08 21:31

Well... I can say 3 days and no problems... without cheating

Post by **martin** » 2006-08-08 21:40

Hehe, congratulations..

But 3 days isn't very much... Do you have any idéa how often the SMTP stopped in the previous builds?

If you have turned on logging, I wouldn't mind having a copy of it sent to martin@hmailserver.com. Then I can take a look at it and see whether the connection timeouts behaves as I wanted them to. (In previous builds, the log created on my server didn't look exactly the same as the log I received from you, so would be nice to confrim that it now looks correct in your log as well).

(One minor issue I know of is that whenever you restart the server, 3 messages regarding failing socket shutdown will be added to the error log. These aren't errors so they can be discarded. I've removed them from the log in the source.)

cveillon · Post by **cveillon** » 2006-08-08 23:03

martin wrote:Ok.. well, it's a bit cheating to automatically restart the server every day - Hard to spot this problem then.

Hard? not really. I've quickly scanned each log file searching for gaps of 'SMTPD' entries. Like most, my server processes these continuously. If I find a gap of more than a minute or two from the time of restart, then I would know if there was a problem.

All three days show that SMTP was processing right up to the time of restart. Sorry I didn't make that more clear.

martin wrote: (One minor issue I know of is that whenever you restart the server, 3 messages regarding failing socket shutdown will be added to the error log. These aren't errors so they can be discarded. I've removed them from the log in the source.)

Thanks for this tidbit. I was wondering if it may have been a problem.

with best regards,
Chuck

Post by **martin** » 2006-08-08 23:05

Ahaaa now I get it. I didn't understand that you actually scanned the log files. Well that sounds good.

GlenC · Post by **GlenC** » 2006-08-08 23:22

Martin... I know 3 days isn´t much, I was just jumping on the bandwagon

Anyway, you are welcome to my logs but it might be a few hours before I can send them. I´m at work now and apparently my connection at home is down for some reason (not hmail but the connection itself). Regarding how often the problem occurs, all I can say is "not much". Since you released the alpha it has only occurred on my machine no more than 5 times... total. So I may have to wait more than a week to prove anything, and even then I won´t be sure.

Post by **martin** » 2006-08-08 23:30

Alright. Well there's nothing wrong with being positive.

GlenC · Post by **GlenC** » 2006-08-08 23:43

heh... I know you´re tired of hearing about this problem. Actually I trust it´s probably ok now or you would have heard something already from one of the other testers. Anyway... my connection at home is back up so the last 3 days logs are on their way.

Post by **martin** » 2006-08-08 23:46

Actually, I'm not tired about it yet. I find it quite interesting. But I wouldn't mind resolving this now and move forward. =)

Edit: Thanks for the log. The new "password masking" functionality in 4.3 is nice. Now I no longer have to sit and try to forget peoples passwords.

Slug · Post by **Slug** » 2006-08-09 03:41

martin wrote: Edit: Thanks for the log. The new "password masking" functionality in 4.3 is nice. Now I no longer have to sit and try to forget peoples passwords.

"News flash"

Man from Helsingborg Sweden arrested for posing as a software developer, whilst really stealing peoples email passwords and selling them to the mafia..

Now on a serious note, I said it before and I'll say it again. I have never had the SMTP issue with any build. ***BUT*** I now have IMAP timing out with this build happens about 1 or 2 times a day. When it stops, if I retry it works straight away and I won’t have any issues for hours. what I mean it only does it once never twice in a row.

Michael

Post by **martin** » 2006-08-09 08:53

Slug, if you send me such a log I might be able to say what's causing it.

Slug · Post by **Slug** » 2006-08-09 09:35

martin wrote:Slug, if you send me such a log I might be able to say what's causing it.

I was going to send it later tonight...

Thanks
Michael

cveillon · Post by **cveillon** » 2006-08-09 21:18

Martin,
It happened again. SMTP lockup. I've emailed the logs to you. Hope it helps!

with best regards,
Chuck

Post by **martin** » 2006-08-09 21:26

Crap.

No messages in the windows event viewer at this time?
What network card (manufacturer) are you using?
What virus scanner are you using in Windows? (I'm not talking about the one you've configured in hMailServer). Are you using "live" scanning?
Have you modified the number of TCP/IP threads in the performance settings?
Can you backup your settings (not the domains, messages, just the settings) and send the zip file to me so that I can configure my server to run just as yours?

cveillon · Post by **cveillon** » 2006-08-09 21:49

martin wrote:Crap.

No messages in the windows event viewer at this time?
What network card (manufacturer) are you using?
What virus scanner are you using in Windows? (I'm not talking about the one you've configured in hMailServer). Are you using "live" scanning?
Have you modified the number of TCP/IP threads in the performance settings?
Can you backup your settings (not the domains, messages, just the settings) and send the zip file to me so that I can configure my server to run just as yours?

1. No, like before. No entries in any of the event logs.
2. Intel PRO 100 VE
3. Only one virus scanner installed on this server, Clamwin. No scheduled scans. No others, period.
4. I have performed the TCPIP.SYS mod on all my XPSP2 servers. It replaces the SP2 tcpip.sys file to a SP1 version. This should have removed the 10 per second limit that SP2 introduces. I believe the SP1 version allows 50 per second. If that limit is reached, it should show a error in the event log, event ID 4226. No other changes or enhancements have been made. This would include the registry as well.
5. I'll email you the settings soon.

with best regards,
Chuck

cveillon · Post by **cveillon** » 2006-08-09 22:05

Went ahead and installed the latest drivers from Intel. Old drivers dated 2003. Now running 2006 driver. Hope it helps.

Chuck

Post by **martin** » 2006-08-09 22:07

I doubt it will help. The problem disappeared in my network after having patched the router. I don't think it really had anything to do with that though. Can't see how it would possible effect this.. (Was using a Dlink DI-624).

Post by **martin** » 2006-08-09 22:11

If the problem occurs again, and you notice it, could you please do the following:

Run the following command from a prompt
netstat -a -b -n
to display what software is listening on what ports on your computer and then send the output to me. hMailServer should be listening on 25, 110 and 143.

And after that try to restart the hMailServer server from within hMailServer Administrator and see if that makes it possible to connect to port 25.

Post by **martin** » 2006-08-09 22:14

Even more suggestions:

Can you try to run LSPFix.exe and see what Winsock LSP you've installed? LSP's are "Winsock addins" that intercepts all the traffic between hMailServer and Winsock. Some badly written LSP's can sometimes cause strange problems.

http://cexx.org/lspfix.htm

Don't remove any of the LSP's... can cause many problems.

cveillon · Post by **cveillon** » 2006-08-09 22:25

martin wrote:Even more suggestions:

Can you try to run LSPFix.exe and see what Winsock LSP you've installed? LSP's are "Winsock addins" that intercepts all the traffic between hMailServer and Winsock. Some badly written LSP's can sometimes cause strange problems.

http://cexx.org/lspfix.htm

Don't remove any of the LSP's... can cause many problems.

Yes, quite.

Pretty familar with the tool lspfix, and the problems of rogue/spyware stacks...
Doesn't seem to be any problems there. This is what's listed:
mswsock.dll (TCPIP)
winrnr.dll (NTDS)
rsvpsp.dll (Protocol handler)

Post by **martin** » 2006-08-09 22:28

Yupp, those are the default LSP's..

cveillon · Post by **cveillon** » 2006-08-09 22:34

martin wrote:If the problem occurs again, and you notice it, could you please do the following:

Run the following command from a prompt
netstat -a -b -n
to display what software is listening on what ports on your computer and then send the output to me. hMailServer should be listening on 25, 110 and 143.

And after that try to restart the hMailServer server from within hMailServer Administrator and see if that makes it possible to connect to port 25.

Be happy to. I use TCPView from sysinternals.com and hope that it meets your approval.

In case you're not already familar with it, It's a freeware product that does the same thing but gives you a windows interface with easy 'save as' text capability.
http://www.sysinternals.com/Utilities/TcpView.html

with best regards,
Chuck

Post by **martin** » 2006-08-09 22:37

Yupp, that should work fine. =)

Run it myself from time to time. Sysinternals was bought by Microsoft some week ago. Not sure if that's good..

Post by **martin** » 2006-08-09 23:01

I imported your backup to my server so now I'm running with the same configuration. I've created a small test application which runs two threads and connects, tries to send an email and then disconnects improperly, the same way as the last connection did before your server stopped. I will run this over the night and see if something happens.

Are you familiar with Ethereal? Perhaps it would be possible to install this on the server and then run it the next time it happens. In Ethereal, you specify that you want to log tcp.port == 25, and then try to connect to the server (from another computer). Then every single TCP/IP packet will be logged. However, Ethereal is a bit tricky to get started with and skip it unless you're already familiar with it.. =)

cveillon · Post by **cveillon** » 2006-08-09 23:31

martin wrote: Are you familiar with Ethereal? Perhaps it would be possible to install this on the server and then run it the next time it happens. In Ethereal, you specify that you want to log tcp.port == 25, and then try to connect to the server (from another computer). Then every single TCP/IP packet will be logged. However, Ethereal is a bit tricky to get started with and skip it unless you're already familiar with it.. =)

Yes, I've been using it a few years now.

Will do.

cveillon · Post by **cveillon** » 2006-08-09 23:37

FYI
I was just wondering if it could be the 'windows firewall' flaking out on port 25, but checked and I had already disabled it.

Post by **martin** » 2006-08-09 23:54

I think you can skip that, I have some good news again.

Using the C# app I wrote to simulate incorrectly behaving SMTP-clients I have been able to reproduce the problem four times now. After the problem has occured, I can not open any new connections to the SMTP port, but I can still connect to the POP3 and IMAP port of the server so it feels like it's the same problem you have been having. Or to be more correct - I can connect to the SMTP server, but no data is being transmitted (telnet window is just black), and nothing is noted in the hMailServer log.

The problem occurs after having run the C# app for about 5-10 minutes, after about 700-1300 SMTP connections.

Have to go to bed now but will continue with this tomorrow. At the moment it feels like this should be pretty easily resolved when I can reproduce it "on demand". (That's why I'm going to bed now, before I discover that it's harder than I think)

(I'll make sure to add this application to the standard hMailServer tests to prevent this issue from reappearing)

(Edit 1: At the moment I've run 3000 SMTP connections without getting the error again. But I still think that since I've been able to reproduce it 4-5 times in an hour, it's at least quite easily reproduced
Edit 2: Yupp, now it happened again)

cveillon · Post by **cveillon** » 2006-08-10 01:06

Fantastic!

Hope you have a great nights sleep. If you're like me, you'll think about in your dreams.

Post by **martin** » 2006-08-10 20:28

This is going slowly forward... Apparantly, the issue is only reproducible for me when I connect via port forwarding in my wireless router. If I connect directly to hMailServer, the problem doesn't seem to be occur (tried with over 100 000 connections earlier today and got a bit frustrated when the problem suddenly did not occur). That may explain why the problem was reduced when I updated the firmware of my router. Or I'm just being lucky at the moment.

cveillon · Post by **cveillon** » 2006-08-10 21:40

Sorry it's moving so slow.

Would you still like me to grab a packet capture the next time it happens??

Post by **martin** » 2006-08-10 21:55

Nope, I don't need any more feedback. Should be able to solve it with the info I got, since I can reproduce it. (I've already captured output from Ethereal, TCPview myself)

I think I will have to look at tomorrow and in the weekend. At least it's going forward.. It's just that I have to do some test, wait between 5 minutes up to 2 hours to see the result, then do another test and so on. But I'm sure I'll solve it, unless it suddenly stops being reproducible. it's just a bit tricky. What would life be without these issues.

^DooM^ · Post by **^DooM^** » 2006-08-10 22:25

martin wrote:What would life be without these issues.

Boring

Slug · Post by **Slug** » 2006-08-11 03:18

martin wrote:Apparantly, the issue is only reproducible for me when I connect via port forwarding in my wireless router.

This might explain why I have never seen this problem. What brand of router are you using Martin ??

Michael

cveillon · Post by **cveillon** » 2006-08-11 04:54

I've only started seeing this problem after installing 4.3. I've been running hmail since 3.x, with the same switch and router. I've processed thousands of messages for multiple domains and never had this problem before. It could be a coincidence, but I doesn't seem likely.

Post by **martin** » 2006-08-11 11:50

I didn't mean that the problem was directly related to my D-link router. The problem is somewhere in hMailServer 4.3/Windows. But different hardware is probably why it only happens for some of us. Certain hardware might be slower or faster and make the problem appear.

cveillon · Post by **cveillon** » 2006-08-11 16:50

martin wrote:I didn't mean that the problem was directly related to my D-link router. The problem is somewhere in hMailServer 4.3/Windows. But different hardware is probably why it only happens for some of us. Certain hardware might be slower or faster and make the problem appear.

With the new Intel driver I'm using, I'm able to enable packet coalescing. This reduces performance, but will force all transmitting packets fragments into a single buffer before sending them to the network. Who knows, maybe it will solve this problem.

Will let you know how that works out.

Post by **martin** » 2006-08-11 17:17

I doubt it will do any difference for you. You're probably forced to restart the server regulary until I've found the solution, which I hope I'll do during this weekend..

cveillon · Post by **cveillon** » 2006-08-11 17:46

Yes, I've had to restart it three times in the last 48 hours. Thanks and best of luck fixing it this weekend!

with best regards,
Chuck

Post by **martin** » 2006-08-12 00:55

At least it feels like I'm getting closer and closer now. Appears to be some undocumented behaviour in Winsock but it's a little to early to say.

hMailServer creates a set of client sockets before any clients has connected. So whenever a client wants to connect, there's a socket ready for it. This set of pre-created sockets should be refilled but it appears this is where it goes wrong. The set of socket is refilled, but it appears Winsock isn't aware of it, so when a client wants to connect, Winsock doesn't know what to do with it.

Well, the struggle continues.

Post by **martin** » 2006-08-12 14:31

I think I've figured out what's causing it now. Cveillon, could you install this build and confirm that you get errors in your hMailServer log when the problem occurs?

http://dev.hmailserver.com/dev/hMailServer-4.3-B228.exe

cveillon · Post by **cveillon** » 2006-08-13 06:09

Ok, installed B228. Will let you know when something turns up.

with best regards,
Chuck

Post by **martin** » 2006-08-13 10:16

OK. I will put up a new build later that should solve the fixes. I've run a test over the night opening 280 000 connections without getting the error now so it looks good so far.

PedroMartins · Post by **PedroMartins** » 2006-08-13 12:24

I also had this problema (pop3 hangs) and with this version B228 it seems that the problem is solved

Thanks for the great work

Pedro Miguel Martins