Strange outgoing connection issue..

Use this forum if you want to discuss a problem or ask a question related to a hMailServer beta release.
Post Reply
foxmichaels
New user
New user
Posts: 8
Joined: 2017-11-29 20:09

Strange outgoing connection issue..

Post by foxmichaels » 2018-08-09 17:17

Hey all -
Thought I would post here and see what else to check, as I'm experiencing an odd out-going mail issue.
Running 5.6.8-B2431 as an out-going server only. (One account that is sending the mail, one rule to delete all incoming mail to that account, including and failure-to-send emails.)
Delivery settings: Retry count: 2, Time between retries: 30mins.

At some point during the night, one receiving email address server starts to timeout, giving the error in hMail logs:

Code: Select all

A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond"
So, their outbound emails start queuing up. Then, other SMTP servers start giving this error, and the queue gets larger, and larger.
Even emails to *our own receiving server* on another IP start to fail with this error message.

Here's the odd thing, though - pausing/starting the service does not change anything. Stopping and restarting the service (Service Manager) does not change anything.
However, if Clear the Queue, Stop/Restart service, and then try another email to our own server, the message goes through without issue.

Now here are two things I haven't done yet, but plan on:
(1) Clear the queue without stopping/starting server.
(2) Watch incoming connections to our other server, and see if a connection attempt is even made.

Other than those, though, any one have anything to check?
Cheers -
Mike

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Strange outgoing connection issue..

Post by mattg » 2018-08-09 23:11

Almost seems like you have a loop that involves another server, say a backup MX, or an SMTP relayer

WHat are your thread settings in hMailserver
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

foxmichaels
New user
New user
Posts: 8
Joined: 2017-11-29 20:09

Re: Strange outgoing connection issue..

Post by foxmichaels » 2018-08-10 18:01

mattg wrote:
2018-08-09 23:11
Almost seems like you have a loop that involves another server, say a backup MX, or an SMTP relayer

WHat are your thread settings in hMailserver
This instance is running on a web server that hosts our SaaS applications - for our clients. So no inbound is accepted, the server sits and accepts outgoing emails from all of the SaaS instances, and then connects outbound.

Right now, 'Performance' settings are:
Cache:
- Domain: 120sec (hit rate: 91), Account: 120sec (hit rate: 84), Alias: 60sec (hit rate 0), Distribution List: 60 (hit rate 0)
Threading:
- Max Number Command Threads: 15
- Delivery Threads: 15
- Max Number Async Task Threads: 15
- Worker Thread Priority: Normal
Message Indexing:
- Enabled (I realize this doesn't affect outgoing emails, so to speak.)


The interesting thing is that when I do a netstat, I see open ports to the machines that are timing out as SYN_SENT. Which tells me it's a firewall issue. However - this does not change even if I *completely* disable the firewall. (Which isn't good, obviously - but I'm testing here! :) ) And again - it continues if I stop and restart the server.

Now this morning, I'm going to run a test - I'm going to try clearing the queue manually, NOT stop and restart the service, then send another to our own server and see what happens. This will address the thing (1) I mentioned in my opening post.

Edit: Okay, test run: Just cleaning the queue, and sending out another email does not work - it still reports the error about not receiving a response. I then cleared the queue again, and paused, then resumed the service and tried sending one out. No change. I then cleared the queue, *stopped* the service (via services.msc, though I'm sure the command-line would have the same effect), restarted the service, and tried a test mail, and that went through. Strange. :(

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Strange outgoing connection issue..

Post by mattg » 2018-08-11 01:16

Ok...

Is there some other service dependent on the hMailserver service on your machine?
Are you playing with just the windows software firewall, or is there a hardware firewall or UTM device somewhere in the mix? what brand of router / modem is in use (model numbers too if possible please)
Any Antivirus on your server? On your network that may be inspecting all network traffic?
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
Dravion
Senior user
Senior user
Posts: 2071
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: Strange outgoing connection issue..

Post by Dravion » 2018-08-11 12:51

I remember some Microsoft Nuts integrated a so called Telemetry gathering function in Visual C/C++ 2013 which affected automatically every program compiled with the Compiler.Independent Researcher found out about it and Microsoft came up with some silly excuses. It can now de disabled by some surpression link switch durning build time.

I will file a issue on Github about it.
Attachments
lqBLS.jpg

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Strange outgoing connection issue..

Post by RvdH » 2018-08-11 13:47

Why are you posting something completely irrelevant?

I can't find any traces of 'telemetry_main_invoke_trigger' in any of the hmailserver executables....besides that hmailserver is build with VS 2013 and not with VS 2015 Update 2 that introduced the 'telemetry_main_invoke_trigger'
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
Dravion
Senior user
Senior user
Posts: 2071
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: Strange outgoing connection issue..

Post by Dravion » 2018-08-11 15:23

The phone home functions in Visual C/C++ was "officially" added in VS 2015 Update 2 and it can be manually overriden by

// source wise
extern "C"
{
void _cdecl __vcrt_initialize_telemetry_provider() {}
void _cdecl __telemetry_main_invoke_trigger() {}
void _cdecl __telemetry_main_return_trigger() {}
void _cdecl __vcrt_uninitialize_telemetry_provider() {}
};

// linker wise
or just by adding "notelemetry.obj" as linker switch.

But any default C/C++ App will phone home to MS without the knowlege of the Programmer until a removal patch was installed or the
above shown, manual actions will be performed to avoid it.

However:
The Visual C/C++ Runtime Libs msvcp120.dll and msvcr120.dll are closed source.We cannot check the sourcecode, we only can see the exported functions and hMailServer needs it. Regarding the 2015 CRT case: Microsoft didnt say anything until 3dparty security Researchers found strange remote connections in some firewall logs. Only after the proof was allready out and undenieable, Microsoft came up with some silly excuses like "its for improving the customer experience".

However 2:
I created a little C/C++ Console App and added "notelemetry.obj" to the linker tab and hit compile.

Message:
Error 2 error LNK1104: cannot open file 'notelemetry.obj' c:\Users\Dravion\documents\visual studio 2013\Projects\testtele\testtele\LINK

It looks like the VS2015 Telemetry stuff isnt embedded in VS2013 as described right now, but i take it with a grain of salt (just my 2 cents)

User avatar
Dravion
Senior user
Senior user
Posts: 2071
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: Strange outgoing connection issue..

Post by Dravion » 2018-08-11 15:42

foxmichaels wrote:
2018-08-09 17:17
At some point during the night, one receiving email address server starts to timeout, giving the error in hMail logs:

Code: Select all

A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond"
This indicates that the remote IP address is reachable.
For Winsock using programs (like hMailServer.exe) the default timeout is max 30 secs.

1) Make sure the Destination host is reachable (try to ping, to telnet)
2) Try diffrent DSL/Mobile Internet connections ways in the case your ip is blacklisted
3) Check if you are on some DNS or IP Blacklist with https://mxtoolbox.com/

foxmichaels
New user
New user
Posts: 8
Joined: 2017-11-29 20:09

Re: Strange outgoing connection issue..

Post by foxmichaels » 2018-08-13 17:13

mattg wrote:
2018-08-11 01:16
Ok...

Is there some other service dependent on the hMailserver service on your machine?
Are you playing with just the windows software firewall, or is there a hardware firewall or UTM device somewhere in the mix? what brand of router / modem is in use (model numbers too if possible please)
Any Antivirus on your server? On your network that may be inspecting all network traffic?
This is running on an Amazon virtual machine. Windows firewall turned off. Amazon's firewall turned on to allow all outgoing connections. No anti-virus at the moment. (Waiting to install it until after i figure this odd issue out.)
The hMail service DOES depend on the hmailServerMySQL service, which runs the database.

Dravion wrote:
2018-08-11 15:42
This indicates that the remote IP address is reachable.
For Winsock using programs (like hMailServer.exe) the default timeout is max 30 secs.

1) Make sure the Destination host is reachable (try to ping, to telnet)
2) Try diffrent DSL/Mobile Internet connections ways in the case your ip is blacklisted
3) Check if you are on some DNS or IP Blacklist with https://mxtoolbox.com/
Apparently none of the rest of my message was read.

To summarize:
When this happens, ALL outbound connections (made by a specific tread? All threads?) report back with this error - even a known good mail server - our own company's. Pausing/Restarting hMail service does not cause any change.
Stopping/Restarting hMail service does not cause any change.
Clearing the outbound queue does not cause any change.
Clearing queue and stopping/restarting the hMail server causes change - all mails start going out after that. Including ones to our own company's.

I am waiting to hear back from my IT to see if our mail server (referred to as 'own company's' up above) is even getting a ping/connection attempt from this virtual host.

User avatar
Dravion
Senior user
Senior user
Posts: 2071
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: Strange outgoing connection issue..

Post by Dravion » 2018-08-13 20:35

Ok, this looks odd.

Can you download Wireshark and capture 2-5 of Traffic from you Amazon Server and attach a pcap log to your next reply?

https://www.wireshark.org/#download

foxmichaels
New user
New user
Posts: 8
Joined: 2017-11-29 20:09

Re: Strange outgoing connection issue..

Post by foxmichaels » 2018-08-13 21:33

Dravion wrote:
2018-08-13 20:35
Ok, this looks odd.

Can you download Wireshark and capture 2-5 of Traffic from you Amazon Server and attach a pcap log to your next reply?

https://www.wireshark.org/#download
Will do.
Will have to report back tomorrow, as new development:
I did NOT clear out the queue manually today. (This means a lot of our clients didn't get notifications, but.. they'll live. 8)) I let it clear as retry count was reached, and outgoing queue was emptied.

I then forced an email to out company server - and it went through this time.
Despite manually clearing the queue did NOT work, this time, letting it clear the queue on it's own did.

This has GOT to be a red-herring, unless it has to do with threading.

I have to wait until tomorrow now (this server handles EU traffic!) to get the queue to back up and cause the issue again.

Thanks!

foxmichaels
New user
New user
Posts: 8
Joined: 2017-11-29 20:09

Re: Strange outgoing connection issue..

Post by foxmichaels » 2018-08-16 17:30

Hey all -
Okay, I got Wireshark logs. One is for about 5 minus worth of traffic, and is 67Mb in size. The other is filtered ONLY on an email attempt to my server. My server never saw an incoming attempt on it.
The two logs are zipped together, here:
http://tsqmadness.com/wireshark.zip (12.3Mb)

I'm not a network specialist, so I can't decode the error messages in the smaller file - trying to connect to my server. Input is greatly appreciated. :)

foxmichaels
New user
New user
Posts: 8
Joined: 2017-11-29 20:09

Re: Strange outgoing connection issue..

Post by foxmichaels » 2018-08-16 18:16

As a followup -
I realized that with the mixed results I'm getting, there had to be something else.

I knew about this:
https://aws.amazon.com/premiumsupport/k ... -throttle/
But had ruled it out because our account (business, SaaS machines) was supposed to have it automatically disabled.

I have opened a ticket with them - **to verify it has actually been deactivated**. Because I am going to eat my shorts if they come back and say 'ooops, we forgot'. For wasting *MY* time, as well as everyone elses' here. :evil:

User avatar
Dravion
Senior user
Senior user
Posts: 2071
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: Strange outgoing connection issue..

Post by Dravion » 2018-08-16 18:29

Yeah, that would be cool.

Sometimes we get some strange support cases in from Amazon EC2/AWS Users.
Would be interresting to compile a little list of Problems regarding Amazon Virtual Servers in terms of hMailServer because its only a matter
of time we get the next Amazon specific case in.

foxmichaels
New user
New user
Posts: 8
Joined: 2017-11-29 20:09

Re: Strange outgoing connection issue..

Post by foxmichaels » 2018-08-16 20:26

Dravion wrote:
2018-08-16 18:29
Yeah, that would be cool.

Sometimes we get some strange support cases in from Amazon EC2/AWS Users.
Would be interresting to compile a little list of Problems regarding Amazon Virtual Servers in terms of hMailServer because its only a matter
of time we get the next Amazon specific case in.

Yup, well, shoot me out of a canon.
Hopefully this will be able to be searched in case anyone in the future has an issue like this.

Amazon support replied back stating 'Oh, we're sorry - all instances made after <date> had the throttling enabled.' They were disabled, and as soon as they did, the outgoing emails flew out.

My company's gonna make a donation to the project to thank everyone for puttin' up with this shit. :mrgreen:

User avatar
jimimaseye
Moderator
Moderator
Posts: 10053
Joined: 2011-09-08 17:48

Re: Strange outgoing connection issue..

Post by jimimaseye » 2018-08-16 21:27

Unbelievable.
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
Dravion
Senior user
Senior user
Posts: 2071
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: Strange outgoing connection issue..

Post by Dravion » 2018-08-16 23:17

Excellent. Nice Job. We need document such Problems because a diffrent User will have the same Problems soon i guess.

Post Reply