Sub OnHELO(oClient) progress?

Use this forum if you want to discuss a problem or ask a question related to a hMailServer beta release.
Post Reply
User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Sub OnHELO(oClient) progress?

Post by SorenR » 2016-08-10 18:11

EDIT:

This thread includes now recognised builds containing new functionality. The latest RvdH inclusive release: viewtopic.php?p=252325#p252325

Mod.
----------------------------------------------------------------



Well... Not sure how this is progressing so...

Moved my personal changes forward from 5.4.2-B1964 to 5.6.5-B2367. Those of you with programming genes know how to work this file. Not a Git-Geek - did SVN back in the day, worked mostly on 'nix, thus the "diff" ;-)

Pitfalls (YES, there is...) If you assign ANY value to "Return.Value" in "Sub OnHELO(oClient)" the server WILL crash. Here, I've said it, be warned!

Code: Select all

diff -bur Z:\hMailServer-5.6.5\B2367/ScriptServer.cpp Z:\hMailServer-5.6.5\B2367.1/ScriptServer.cpp
--- Z:\hMailServer-5.6.5\B2367/ScriptServer.cpp	2016-05-24 13:12:26.000000000 +0200
+++ Z:\hMailServer-5.6.5\B2367.1/ScriptServer.cpp	2016-08-10 13:38:00.999999900 +0200
@@ -29,7 +29,8 @@
       has_on_error_(false),
       has_on_delivery_failed_(false),
       has_on_external_account_download_(false),
-      has_on_smtpdata_(false)
+      has_on_smtpdata_(false),
+      has_on_helo_(false)
    {
       
    }
@@ -97,6 +98,7 @@
          has_on_delivery_failed_ = DoesFunctionExist_("OnDeliveryFailed");
          has_on_external_account_download_ = DoesFunctionExist_("OnExternalAccountDownload");
          has_on_smtpdata_ = DoesFunctionExist_("OnSMTPData");
+         has_on_helo_ = DoesFunctionExist_("OnHELO");
 
       }
       catch (...)
@@ -251,6 +253,12 @@
             return;
          break;
 
+      case EventOnHELO:
+            event_name_ = _T("OnHELO");
+            if (!has_on_helo_)
+            return;
+         break;
+
       case EventCustom:
          break;
       default:
diff -bur Z:\hMailServer-5.6.5\B2367/ScriptServer.h Z:\hMailServer-5.6.5\B2367.1/ScriptServer.h
--- Z:\hMailServer-5.6.5\B2367/ScriptServer.h	2016-05-24 13:12:26.000000000 +0200
+++ Z:\hMailServer-5.6.5\B2367.1/ScriptServer.h	2016-08-10 13:02:02.999999900 +0200
@@ -28,6 +28,7 @@
          
          EventOnExternalAccountDownload = 1011,
          EventOnSMTPData = 1012,
+         EventOnHELO = 1013,
       };
 
       ScriptServer(void);
@@ -64,6 +65,7 @@
       bool has_on_delivery_failed_;
       bool has_on_external_account_download_;
       bool has_on_smtpdata_;
+      bool has_on_helo_;
 
       String script_contents_;
       String script_extension_;
diff -bur Z:\hMailServer-5.6.5\B2367/SMTPConnection.cpp Z:\hMailServer-5.6.5\B2367.1/SMTPConnection.cpp
--- Z:\hMailServer-5.6.5\B2367/SMTPConnection.cpp	2016-05-24 13:12:26.000000000 +0200
+++ Z:\hMailServer-5.6.5\B2367.1/SMTPConnection.cpp	2016-08-10 13:30:54.999999900 +0200
@@ -1513,6 +1513,51 @@
          return;
       }
 
+      //
+      // Event OnHELO
+      //
+      if (Configuration::Instance()->GetUseScriptServer())
+      {
+         std::shared_ptr<ScriptObjectContainer> pContainer = std::shared_ptr<ScriptObjectContainer>(new ScriptObjectContainer);
+         std::shared_ptr<Result> pResult = std::shared_ptr<Result>(new Result);
+         std::shared_ptr<ClientInfo> pClientInfo = std::shared_ptr<ClientInfo>(new ClientInfo);
+
+         pClientInfo->SetIPAddress(GetIPAddressString());
+         pClientInfo->SetPort(GetLocalEndpointPort());
+         pClientInfo->SetHELO(helo_host_);
+
+         pContainer->AddObject("HMAILSERVER_CLIENT", pClientInfo, ScriptObject::OTClient);
+         pContainer->AddObject("Result", pResult, ScriptObject::OTResult);
+
+         String sEventCaller = "OnHELO(HMAILSERVER_CLIENT)";
+         ScriptServer::Instance()->FireEvent(ScriptServer::EventOnHELO, sEventCaller, pContainer);
+
+         switch (pResult->GetValue())
+         {
+            case 1:
+            {
+               String sErrorMessage = "554 Rejected";
+               EnqueueWrite_(sErrorMessage);
+               LogAwstatsMessageRejected_();
+               return;
+            }
+            case 2:
+            {
+               String sErrorMessage = "554 " + pResult->GetMessage();
+               EnqueueWrite_(sErrorMessage);
+               LogAwstatsMessageRejected_();
+               return;
+            }
+            case 3:
+            {
+               String sErrorMessage = "453 " + pResult->GetMessage();
+               EnqueueWrite_(sErrorMessage);
+               LogAwstatsMessageRejected_();
+               return;
+            }
+         }
+      }
+
       SendEHLOKeywords_();
 
       if (current_state_ == INITIAL)
@@ -1531,6 +1576,51 @@
          return;
       }
 
+      //
+      // Event OnHELO
+      //
+      if (Configuration::Instance()->GetUseScriptServer())
+      {
+         std::shared_ptr<ScriptObjectContainer> pContainer = std::shared_ptr<ScriptObjectContainer>(new ScriptObjectContainer);
+         std::shared_ptr<Result> pResult = std::shared_ptr<Result>(new Result);
+         std::shared_ptr<ClientInfo> pClientInfo = std::shared_ptr<ClientInfo>(new ClientInfo);
+
+         pClientInfo->SetIPAddress(GetIPAddressString());
+         pClientInfo->SetPort(GetLocalEndpointPort());
+         pClientInfo->SetHELO(helo_host_);
+
+         pContainer->AddObject("HMAILSERVER_CLIENT", pClientInfo, ScriptObject::OTClient);
+         pContainer->AddObject("Result", pResult, ScriptObject::OTResult);
+
+         String sEventCaller = "OnHELO(HMAILSERVER_CLIENT)";
+         ScriptServer::Instance()->FireEvent(ScriptServer::EventOnHELO, sEventCaller, pContainer);
+
+         switch (pResult->GetValue())
+         {
+            case 1:
+            {
+               String sErrorMessage = "554 Rejected";
+               EnqueueWrite_(sErrorMessage);
+               LogAwstatsMessageRejected_();
+               return;
+            }
+            case 2:
+            {
+               String sErrorMessage = "554 " + pResult->GetMessage();
+               EnqueueWrite_(sErrorMessage);
+               LogAwstatsMessageRejected_();
+               return;
+            }
+            case 3:
+            {
+               String sErrorMessage = "453 " + pResult->GetMessage();
+               EnqueueWrite_(sErrorMessage);
+               LogAwstatsMessageRejected_();
+               return;
+            }
+         }
+      }
+
       EnqueueWrite_("250 Hello.");
 
       if (current_state_ == INITIAL)
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2016-08-10 19:29

Thanks, now i get it...last changes should have been inside SMTPConnection.cpp and not in ScriptServer.cpp as documented here
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Sub OnHELO(oClient) progress?

Post by SorenR » 2016-08-10 19:54

RvdH wrote:Thanks, now i get it...last changes should have been inside SMTPConnection.cpp and not in ScriptServer.cpp as documented here
Oops... :oops: :mrgreen:
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Sub OnHELO(oClient) progress?

Post by SorenR » 2016-08-11 17:47

Hmm... There has been a development... Something works when it really should not :mrgreen:

When I initially made this for my 5.4.2 I could not use "Result.Value" and "Result.Message" as the server would crash ... :roll:

When I ported my changes to a fresh 5.6.5-B2367 I assumed (I know... Assumption Is The Mother Of All Fuckups!) it would behave the same way but I left the code in there. Well, just played a bit with it - no errors!

Code: Select all

"DEBUG"	868	"2016-08-11 17:30:02.649"	"Executing event OnHELO"
"DEBUG"	868	"2016-08-11 17:30:02.649"	"Event completed"
"SMTPD"	868	166	"2016-08-11 17:30:02.649"	"127.0.0.1"	"SENT: 554 Whooa... Whaz' up?"

Code: Select all

   Sub OnHELO(oClient)
      Result.Message = "Whooa... Whaz' up?"
      Result.Value = 2
   End Sub
If someone wants to try it out I have a fresh compiled 5.6.5-B2367.1 "hMailServer.exe" on my webserver...
http://www.lolle.org/images/hmailserver/hmailserver.rar
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
jimimaseye
Moderator
Moderator
Posts: 10053
Joined: 2011-09-08 17:48

Re: Sub OnHELO(oClient) progress?

Post by jimimaseye » 2016-08-11 21:51

Thanks soren. I'm currently away on hols but when I get back to it I will be more than happy to test/use it. Will you keep the link available for some weeks?
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Sub OnHELO(oClient) progress?

Post by SorenR » 2016-08-11 23:03

jimimaseye wrote:Thanks soren. I'm currently away on hols but when I get back to it I will be more than happy to test/use it. Will you keep the link available for some weeks?
If not, you have my email :wink:
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
jimimaseye
Moderator
Moderator
Posts: 10053
Joined: 2011-09-08 17:48

Re: Sub OnHELO(oClient) progress?

Post by jimimaseye » 2016-08-12 00:01

Out of interest is it an Install package or will it be a straight manual hmailserver.exe service program swapout (to replace the official one currently installed in the program directory)?
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Sub OnHELO(oClient) progress?

Post by SorenR » 2016-08-12 10:30

Manual swap.
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2016-08-13 15:00

Works like a charm!
"TCPIP" 7048 "2016-08-13 14:56:39.506" "TCP - 213.131.38.246 connected to *.*.*.*:25."
"SMTPD" 7048 55166 "2016-08-13 14:56:39.506" "213.131.38.246" "SENT: 220 mail.domain.com ESMTP"
"SMTPD" 7748 55166 "2016-08-13 14:56:39.584" "213.131.38.246" "RECEIVED: EHLO ylmf-pc"
"SMTPD" 7748 55166 "2016-08-13 14:56:39.599" "213.131.38.246" "SENT: 554 Back off!"
"TCPIP" 7048 "2016-08-13 14:56:48.803" "TCP - 213.131.38.246 connected to *.*.*.*:25."
"SMTPD" 7048 55167 "2016-08-13 14:56:48.803" "213.131.38.246" "SENT: 220 mail.domain.com ESMTP"
"SMTPD" 6420 55167 "2016-08-13 14:56:48.897" "213.131.38.246" "RECEIVED: EHLO ylmf-pc"
"SMTPD" 6420 55167 "2016-08-13 14:56:48.913" "213.131.38.246" "SENT: 554 Back off!"

CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Sub OnHELO(oClient) progress?

Post by SorenR » 2016-08-13 17:54

He he ;—)
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2016-08-13 18:06

I have been fiddling around with your AddGreyList Sub inside OnHELO(oClient) event in EventHandlers.vbs

With below changes it will verify the ip against valid domain spf ip ranges en should block faked HELO/EHLO headers to be whitelisted

Code: Select all

   Sub OnHELO(oClient)
      If (Left(oClient.IPAddress, 10) = "192.168.0.") Then Exit Sub
      If (Left(oClient.IPAddress, 10) = "80.160.77.") Then Exit Sub
      If (oClient.Port = 25) Then Wait(20)

      Dim oRegEx
      Set oRegEx = CreateObject("VBScript.RegExp")
      oRegEx.IgnoreCase = True
      oRegEx.Global = False

		oRegEx.Pattern= "^([a-z]+[0-9]{2}\-[a-z]{2}[0-9]\-)(obe\.outbound\.protection\.outlook\.com)$|" &_
						"^(mail\-[a-z]{2}[0-9]\-f[0-9]{1,3})(\.google\.com)$|" &_
						"^(mail[a-z]\-[a-z]{2})(\.linkedin\.com)$|" &_
						"^(mx\-out\.facebook\.com)$|" &_
						"^(mail[a-z]{1}\-[a-z]{2})(\.linkedin\.com)$|" &_
						"^(spruce\-goose\-[a-z]{2}|spring\-chicken\-[a-z]{2})(\.twitter\.com)$|" &_
						"^([a-z]{3}[\d]{3}\-[a-z]{2,3}[\d]{1}[a-z]{1}[\d]{1,2})(\.hotmail\.com)$"
						
      If oRegEx.Test(oClient.HELO) Then Call AddGreyList(oClient.IPAddress, oClient.HELO)

      ...

      Set oRegEx = Nothing.
   End Sub

Code: Select all

Function getDomainName(byVal strHELO)
	dim aryDomain, str2ndLevel, strTopLevel
	getDomainName = ""
	If Len(strHELO) > 0 Then  	
		aryDomain = Split(strHELO,".")
		If uBound(aryDomain) >= 1 Then
			str2ndLevel = aryDomain(uBound(aryDomain)-1)
			strTopLevel = aryDomain(uBound(aryDomain))			
			getDomainName = str2ndLevel & "." & strTopLevel
		End If
	End If
End Function

Code: Select all

Sub AddGreyList(ByVal strIP, ByVal strHELO)
  
	Dim oRegEx
	Set oRegEx = CreateObject("VBScript.RegExp")
	oRegEx.IgnoreCase = True
	oRegEx.Global = False
	
	Select Case getDomainName(strHELO)
	
	case "hotmail.com"
		' https://mail.live.com/mail/ipspace.aspx
		oRegEx.Pattern= "^65\.54\.190\.([0-9]|[1-5][0-9]|6[0-3])$|" &_
						"^65\.54\.190\.(6[4-9]|[7-9][0-9]|1([0-1][0-9]|2[0-7]))$|" &_
						"^65\.54\.190\.(1(2[8-9]|[3-8][0-9]|9[0-1]))$|" &_
						"^65\.54\.190\.(1(9[2-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^65\.55\.116\.([0-9]|[1-5][0-9]|6[0-3])$|" &_
						"^65\.55\.111\.(6[4-9]|[7-9][0-9]|1([0-1][0-9]|2[0-7]))$|" &_
						"^65\.55\.116\.(6[4-9]|[7-9][0-9]|1([0-1][0-9]|2[0-7]))$|" &_
						"^65\.55\.111\.(1(2[8-9]|[3-8][0-9]|9[0-1]))$|" &_
						"^65\.55\.34\.([0-9]|[1-5][0-9]|6[0-3])$|" &_
						"^65\.55\.34\.(6[4-9]|[7-9][0-9]|1([0-1][0-9]|2[0-7]))$|" &_
						"^65\.55\.34\.(1(2[8-9]|[3-8][0-9]|9[0-1]))$|" &_
						"^65\.55\.34\.(1(9[2-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^65\.55\.90\.([0-9]|[1-5][0-9]|6[0-3])$|" &_
						"^65\.55\.90\.(6[4-9]|[7-9][0-9]|1([0-1][0-9]|2[0-7]))$|" &_
						"^65\.55\.90\.(1(2[8-9]|[3-8][0-9]|9[0-1]))$|" &_
						"^65\.55\.90\.(1(9[2-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^65\.54\.51\.(6[4-9]|[7-9][0-9]|1([0-1][0-9]|2[0-7]))$|" &_
						"^65\.54\.61\.(6[4-9]|[7-9][0-9]|1([0-1][0-9]|2[0-7]))$|" &_
						"^207\.46\.66\.([0-9]|1[0-5])$|" &_
						"^157\.55\.0\.(1(9[2-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^157\.55\.1\.(1(2[8-9]|[3-8][0-9]|9[0-1]))$|" &_
						"^157\.55\.2\.([0-9]|[1-5][0-9]|6[0-3])$|" &_
						"^157\.55\.2\.(6[4-9]|[7-9][0-9]|1([0-1][0-9]|2[0-7]))$"
	case "outlook.com"	
		' https://technet.microsoft.com/en-us/library/dn163583(v=exchg.150).aspx (2016-06-24)
		oRegEx.Pattern= "^23\.103\.(1(3[2-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^23\.103\.(1(3[6-9]|4[0-3]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^23\.103\.(1(4[4-9]|5[0-9]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^23\.103\.(1(9[8-9]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^23\.103\.(2(0[0-7]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^40\.(9[2-5])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^40\.107\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^65\.55\.88\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^65\.55\.169\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^94\.245\.120\.(6[4-9]|[7-9][0-9]|1([0-1][0-9]|2[0-7]))$|" &_
						"^104\.47\.([0-9]|[1-9][0-9]|1([0-1][0-9]|2[0-7]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^134\.170\.101\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^134\.170\.140\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^134\.170\.171\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^157\.55\.133\.([0-9]|[1-9][0-9]|1([0-1][0-9]|2[0-7]))$|" &_
						"^157\.56\.87\.(1(9[2-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^157\.56\.(1(1[0-1]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^157\.56\.112\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^157\.56\.116\.([0-9]|[1-9][0-9]|1([0-1][0-9]|2[0-7]))$|" &_
						"^157\.56\.120\.([0-9]|[1-9][0-9]|1([0-1][0-9]|2[0-7]))$|" &_
						"^207\.46\.51\.(6[4-9]|[7-9][0-9]|1([0-1][0-9]|2[0-7]))$|" &_
						"^207\.46\.100\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^207\.46\.108\.([0-9]|[1-9][0-9]|1([0-1][0-9]|2[0-7]))$|" &_
						"^207\.46\.163\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^213\.199\.154\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^213\.199\.180\.(1(2[8-9]|[3-8][0-9]|9[0-1]))$|" &_
						"^216\.32\.(1(8[0-1]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$"
	case "twitter.com"	
		oRegEx.Pattern= "^199\.16\.(1(5[6-9]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^199\.59\.(1(4[8-9]|5[0-1]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^8\.25\.(1(9[4-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^8\.25\.(1(9[6-7]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^204\.92\.114\.203$|^204\.92\.114\.(2(0[4-5]))$|^23\.21\.83\.90$"
	case "facebook.com"	
		oRegEx.Pattern= "^69\.63\.179\.25$|^66\.220\.159\.18$|" &_
						"^69\.63\.178\.(1(2[8-9]|[3-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^69\.63\.184\.([0-9]|[1-9][0-9]|1([0-1][0-9]|2[0-7]))$|" &_
						"^66\.220\.144\.(1(2[8-9]|[3-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^66\.220\.155\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^69\.171\.232\.([0-9]|[1-9][0-9]|1([0-1][0-9]|2[0-7]))$|" &_
						"^69\.171\.232\.(1(2[8-9]|[3-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^66\.220\.157\.([0-9]|[1-9][0-9]|1([0-1][0-9]|2[0-7]))$|" &_
						"^69\.171\.244\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$"			
	case "linkedin.com"	
		oRegEx.Pattern= "^199\.101\.162\.([0-9]|[1-9][0-9]|1([0-1][0-9]|2[0-7]))$|" &_
						"^108\.174\.3\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^108\.174\.6\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^216\.136\.162\.65$|^199\.101\.161\.130$"
	case "google.com"	
		' https://support.google.com/a/answer/60764?hl=en
		oRegEx.Pattern= "^64\.18\.([0-9]|1[0-5])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^64\.233\.(1([6-8][0-9]|9[0-1]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^66\.102\.([0-9]|1[0-5])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^66\.249\.(8[0-9]|9[0-5])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^72\.14\.(1(9[2-9])|2([0-4][0-9]|5[0-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^74\.125\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^108\.177\.([8-9]|1[0-5])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^173\.194\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^207\.126\.(1(4[4-9]|5[0-9]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^209\.85\.(1(2[8-9]|[3-9][0-9])|2([0-4][0-9]|5[0-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^216\.58\.(1(9[2-9])|2([0-1][0-9]|2[0-3]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^216\.239\.(3[2-9]|[4-5][0-9]|6[0-3])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^172\.217\.([0-9]|[1-2][0-9]|3[0-1])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$"
	Case Else
		oRegEx.Pattern= "^(?:[0-9]{1,3}\.){3}[0-9]{1,3}$"
	End select

	If oRegEx.Test(strIP) Then
	....
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2016-09-13 15:59

Is it safe to block anything that comes from direct IP, eg: 'HELO 172.111.198.131' and/or 'HELO [172.111.198.131]'?

I been running it for a while like below (and BAN them accordingly) and all results in ipranges ban entries contain entries from questionable origin, eg: India, Pakistan, Vietnam, China etc. etc.
I have had zero false entries this far

Code: Select all

		oRegEx.Pattern = "^(\[?" & oClient.IPAddress & "\]?)$|^(\.[a-z]+)$|^(.+\.\.[a-z]+)$|([\!\@\#\$\%\^\&\*\(\)\{\}])"
		If oRegEx.Test(oClient.HELO) Then
			Call AutoBan(oClient.IPAddress, oClient.HELO, 1, "h")
			Result.Message = "Rejected - HELO message should contain FQDN"
			Result.Value = 2
			Exit Sub
		End If
		
		oRegEx.Pattern = "^(\[?(?:[0-9]{1,3}\.){3}[0-9]{1,3}\]?)$"
		If oRegEx.Test(oClient.HELO) Then
			REM local lan IP adresses
			' oRegEx.Pattern = "^(\[?1((0)|(92\.168)|(72\.((1[6-9])|(2[0-9])|(3[0-1])))|(27))\..*\]?)$"
			' If oRegEx.Test(oClient.HELO) Then
				' Result.Value = 0
				' Exit Sub
			' Else	
				Call AutoBan(oClient.IPAddress, oClient.HELO, 1, "h")
				Result.Message = "Rejected - HELO message should contain FQDN"
				Result.Value = 2
				Exit Sub
			' End If
		End If
		
http://www.linuxmagic.com/best_practice ... omain.html
http://faculty.cs.niu.edu/~rickert/cf/bad-ehlo.html
https://github.com/Exim/exim/wiki/AclHeloTricks
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Sub OnHELO(oClient) progress?

Post by SorenR » 2016-09-13 16:19

RFC 2821 (SMTP) says:
"In situations in which the SMTP client system does not have a meaningful domain name (e.g., when its address is dynamically allocated and no reverse mapping record is available), the client SHOULD send an address literal"

So... It's up to you if you want to go "above and beyond" :mrgreen:

Spammers break RFC rules all the time and if they can, so can we ... 8)
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2016-09-13 16:24

Yep i noticed that line as well :wink:
in my results it are likely all spammers, considering their questionable origin...guess i'll stick with it!

BTW, the above rules only filter HELO entries on port 25, our clients all use port 587 as SMTP port (example thunderbird uses: HELO [127.0.0.1])
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Sub OnHELO(oClient) progress?

Post by SorenR » 2016-09-13 16:44

RvdH wrote:Yep i noticed that line as well :wink:
in my results it are likely all spammers, considering their questionable origin...guess i'll stick with it!

BTW, the above rules only filter HELO entries on port 25, our clients all use port 587 as SMTP port (example thunderbird uses: HELO [127.0.0.1])
Thunderbird should resolve it's own address with an rDNS lookup, if it fails to do so and there IS a FQDN for the client it must be an old bug in Thunderbird from 2009 that has resurfaced...

https://bugzilla.mozilla.org/show_bug.cgi?id=279525
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2016-09-14 15:03

This all got me thinking and makes me wonder if something like a OnAUTH(oClient) event handler could be useful for scripting, eg:

Return String Username
Return Boolean IsAuthenticated
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Sub OnHELO(oClient) progress?

Post by SorenR » 2016-09-14 17:34

RvdH wrote:This all got me thinking and makes me wonder if something like a OnAUTH(oClient) event handler could be useful for scripting, eg:

Return String Username
Return Boolean IsAuthenticated
Hmm... oClient only have 4 members; HELO, IPAddress, Port and Username...

Perhaps two events?

OnAUTH(oClient) and OnAUTHFailed(oClient, sReason)
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2016-09-14 17:41

SorenR wrote:
RvdH wrote:This all got me thinking and makes me wonder if something like a OnAUTH(oClient) event handler could be useful for scripting, eg:

Return String Username
Return Boolean IsAuthenticated
Hmm... oClient only have 4 members; HELO, IPAddress, Port and Username...

Perhaps two events?

OnAUTH(oClient) and OnAUTHFailed(oClient, sReason)

Add a member to oClient? :)

Anyway, i'm trying to build something on my dev box...without much luck until now :)
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2016-09-16 16:01

Anyone can give me some pointers where to add oClient methods?
So far i have found these items in: InterfaceClient.h, InterfaceClient,cpp, ClientInfo.h, ClientInfo.cpp

Still trying to get something like oClient.Authenticated Boolean value and/or oClient.STARTTLS Boolean value as requested here
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2016-10-21 14:05

For the enthusiasts...

hMaiiServer 5.6.6-B2383.3

It contains these 3 fixes
  • Supports Sub OnHELO(oClient) event, issue #153
  • Fixed Incorrect DEBUG logging for event 'OnDeliverMessage', issue #181
  • Include HTMLBody into IMAP TEXT search, pull #193
5.6.6-B2383.3.7z
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2016-12-13 18:11

5.6.6-B2383.6

it contains the previous fixes from 5.6.6-B2383.3, plus:
  • Fixed implicit conversion: "int" to "unsigned char" pull #204
  • Faulty: SMTP 'Disconnect client after too many invalid commands' pull issue #160
  • SMTP server error "550 Unsupported ESMTP extension" on MAIL FROM:... AUTH=<> [with fix] issue #164
5.6.6-B2383.6.7z
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2017-01-18 16:49

5.6.6-B2383.7

it contains the previous fixes from 5.6.6-B2383.6, plus:
  • Removed warning if backup was more than 1,5GB and 15GB limit. There's no longer a recommended max-size - the time will vary with the installation size. issue #69
5.6.6-B2383.7.7z


Beta builds:

5.6.7-B2405.6
  • Supports Sub OnHELO(oClient) event, issue #153
  • Fixed Incorrect DEBUG logging for event 'OnDeliverMessage', issue #181
  • Include HTMLBody into IMAP TEXT search, pull #193
  • Fixed implicit conversion: "int" to "unsigned char" pull #204
  • Faulty: SMTP 'Disconnect client after too many invalid commands' pull issue #160
  • SMTP server error "550 Unsupported ESMTP extension" on MAIL FROM:... AUTH=<> [with fix] issue #164
5.6.7-B2405.6.7z

5.6.7-B2405.7

it contains the previous fixes from 5.6.7-B2405.6, plus:
  • Removed warning if backup was more than 1,5GB and 15GB limit. There's no longer a recommended max-size - the time will vary with the installation size. issue #69
5.6.7-B2405.7.7z
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
jimimaseye
Moderator
Moderator
Posts: 10053
Joined: 2011-09-08 17:48

Re: Sub OnHELO(oClient) progress?

Post by jimimaseye » 2017-01-18 17:19

Can you clarify something: what is the difference between 5.6.6-b2383.7 and beta (5.6.7-b2405.7) ? Because the changelog mods being shown against both seem to be the same (from 5.6.6-B2383.3). What makes them different/what am I not seeing?
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2017-01-18 17:30

Read the beta changelog?

https://www.hmailserver.com/changelog/?version=5.6.7

But in short...

5.6.6-b2383.7 = OpenSSL 1.0.1u
5.6.7-b2405.7 = OpenSSL upgraded from 1.0.1u to 1.0.2j, Upgraded BOOST from 1.56.0 to 1.63.0, https://github.com/hmailserver/hmailserver/issues/208
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
jimimaseye
Moderator
Moderator
Posts: 10053
Joined: 2011-09-08 17:48

Re: Sub OnHELO(oClient) progress?

Post by jimimaseye » 2017-01-18 17:53

Gotcha. Now I understand.

So the BETA versions are Martins genuine betas with your additions (OnHELO etc).
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2017-01-18 17:59

Exactly
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
jimimaseye
Moderator
Moderator
Posts: 10053
Joined: 2011-09-08 17:48

Re: Sub OnHELO(oClient) progress?

Post by jimimaseye » 2017-01-18 18:11

Effectively you have brought forward many of the 5.7 fixes ('issues' fixed) into existing 5.6.6 (with your OnHELO addition) for people to use . So people do not have to wait for martin to release 5.7 for these. (Just as well as he seems to be off the boil regarding moving forward with this project, it doesnt seem to be coming forward at any speed - too busy with work etc I presume).
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
jimimaseye
Moderator
Moderator
Posts: 10053
Joined: 2011-09-08 17:48

Re: Sub OnHELO(oClient) progress?

Post by jimimaseye » 2017-02-02 20:42

RvdH wrote:5.6.6-B2383.7

it contains the previous fixes from 5.6.6-B2383.6, plus:
.
.
.
So, Ruud, with your new found Hmailserver coding skills, whats the chances of you making this mod: https://github.com/hmailserver/hmailserver/issues/178 ?

Adding an Autoban is already in the source somewhere, DisableAUTHList is already in the source somewhere, maybe you can work out how to add the autobanning to the DisableAUTHList function?
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2017-02-08 22:25

jimimaseye wrote:
RvdH wrote:5.6.6-B2383.7

it contains the previous fixes from 5.6.6-B2383.6, plus:
.
.
.
So, Ruud, with your new found Hmailserver coding skills, whats the chances of you making this mod: https://github.com/hmailserver/hmailserver/issues/178 ?

Adding an Autoban is already in the source somewhere, DisableAUTHList is already in the source somewhere, maybe you can work out how to add the autobanning to the DisableAUTHList function?
I doubt with my "skills" this can be accomplished :wink:

At the time you posted that topic i agreed this could be useful, but now few months later i have to say it seems unnecessary (at least for me) i hardly see login attempts on port 25 anymore...so it seems the abusers/attackers do learn after a while and simply give up trying
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2017-02-08 22:58

btw, with the fixes of issue #160 the 'Authentication not enabled' error contributes to the Disconnect client after too many invalid commands counter

Code: Select all

      if (!GetAuthIsEnabled_())
      {
         SendErrorResponse_(504, "Authentication not enabled.");
         return;
      }

Code: Select all

   void 
   SMTPConnection::SendErrorResponse_(int iErrorCode, const String &sResponse)
   {
		if (iErrorCode >= 500 && iErrorCode <= 599)
		{
			cur_no_of_invalid_commands_++;

			if (Configuration::Instance()->GetDisconnectInvalidClients() &&
				cur_no_of_invalid_commands_ > Configuration::Instance()->GetMaximumIncorrectCommands())
			{
				// Disconnect
				EnqueueWrite_("Too many invalid commands. Bye!");
				pending_disconnect_ = true;
				EnqueueDisconnect();
				return;
			}
		}

		String sData;
		sData.Format(_T("%d %s"), iErrorCode, sResponse.c_str());

		EnqueueWrite_(sData);
   }

Maybe it could be worth to issue a immediate disconnect after the 'Authentication not enabled' error , eg:

Code: Select all

      if (!GetAuthIsEnabled_())
      {
			SendErrorResponse_(504, "Authentication not enabled.");
			pending_disconnect_ = true;
			EnqueueDisconnect();
			return;
      }
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2017-07-01 09:20

5.6.7-B2407.9

This is the latest beta Version 5.6.7 - Build 2407 (2017-01-26)
It contains the previous fixes from 5.6.7-B2405.7, plus:
  • Speed up 'update hm_messages set messageflags' #221
  • Treat authenticated users as localsender if the sender is authenticated and AuthUserIsLocal=1 INI setting Office 2016 Bug

5.6.7-B2407.9.7z
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
Dravion
Senior user
Senior user
Posts: 2071
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: Sub OnHELO(oClient) progress?

Post by Dravion » 2017-07-01 11:27

What is your plan for this fork in the Future?
Does Martin accept any Pullrequests so this can be merged or backported into the officia release branches on Github?

This was my Problem working on the server core because Martin said verry straight "he gives a fuck what other people are doing, its just its square time project"

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Sub OnHELO(oClient) progress?

Post by SorenR » 2017-07-01 16:01

Dravion wrote:What is your plan for this fork in the Future?
Does Martin accept any Pullrequests so this can be merged or backported into the officia release branches on Github?

This was my Problem working on the server core because Martin said verry straight "he gives a fuck what other people are doing, its just its square time project"
Well... Ubunto began it's life as a fork from Debian and ...

Jomla is a fork from Mambo
Webkit (by Apple) is a fork from KHTML
WordPress is a fork from b2/CafeLog
Microsoft SQL Server is a fork from Sybase SQL Server
Apache HTTP Server is a fork from NCSA HTTPd

8)

"If you don't like it, fork it."

https://www.theregister.co.uk/2016/07/2 ... e_forkery/
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
Dravion
Senior user
Senior user
Posts: 2071
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: Sub OnHELO(oClient) progress?

Post by Dravion » 2017-07-01 18:43

SorenR wrote:
Dravion wrote:What is your plan for this fork in the Future?
Does Martin accept any Pullrequests so this can be merged or backported into the officia release branches on Github?
"If you don't like it, fork it."
I forked it allready in the past. However...
If its not possible collaborating with Martin, it makes no sense for me trying to improve his work and nothing will improve or help the Project. Its fighting against Windmills i think.

For hMailServer i think we can do some extensions and utilities which can be used with hMailServer to. Currently i work with Declan on a Remote API TCP/IP Remote Windows Service. This new Windows Service will add a new TCP/IP Remote Access API Server to hMailServer (for TLS we use LibreSSL). The Remote Access Server for hMailServer can be used to remotely control hMailServer without the need of a Webserver, COM/DCOM and remote clients are not forced running Windows or using a Webbrowser.This Detail is important because Declan wants remotely connect and admistre hMailServer from its own Android Java Smartphone App. So, Java and C/C++ support will be avaiable from the start, but we plan to use LUA as integrated builtin scripting language so non C++ Programmers can remotely script some tasks with LUA which is allmost as easy to learn as VBScript.

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2017-07-03 10:08

I have no idea really...the builds i make only contain fixes that are worth something for myself or fixes that in my opinion already should have been merged into the current 5.6.7 branch
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

AndreL
Normal user
Normal user
Posts: 31
Joined: 2016-06-07 15:42

Re: Sub OnHELO(oClient) progress?

Post by AndreL » 2017-08-03 08:39

A big chunk of my users will migrate to Outlook 2016 (mandatory for integration with another tool).

So I installed the version 5.6.7-B2415.9 from RvdH (previous one was 5.6.7-B2407). So far totally stable.
Annoying bugs like the “office 2016 bug” are solved. Thanks !

Still the missing feature is the support of the Outlook Imap folders auto discovery.
Any advice or guidelines are welcome.

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2017-08-05 21:33

5.6.7-B2415.10

This is the latest beta Version 5.6.7 - Build 2415 (2017-07-09)
It contains the previous fixes from 5.6.7-B2405.9, plus:
  • Add Return-Path header as topmost header before sending the message to SA (+ delete Return-Path header after the SA check completes) #116

5.6.7-B2415.10.7z
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
Dravion
Senior user
Senior user
Posts: 2071
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: Sub OnHELO(oClient) progress?

Post by Dravion » 2017-08-05 23:26

I downloaded your archive but i cannot find the source.
Do you have a Github Repo?

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2017-08-06 10:31

Dravion wrote:I downloaded your archive but i cannot find the source.
Do you have a Github Repo?
Now i do... :lol:
https://github.com/RvdHout/hmailserver

i committed all my changes at once...so you have to look carefully what is actually changed
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
Dravion
Senior user
Senior user
Posts: 2071
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: Sub OnHELO(oClient) progress?

Post by Dravion » 2017-08-06 10:44

Cool. :D
I will take a look at it.

ps:
As far as can see you made changes in this
section of the code only, is this correct?

https://github.com/RvdHout/hmailserver/ ... erver/SMTP

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2017-08-06 11:04

  • hmailserver/source/Server/Common/AntiSpam/SpamTestSpamAssassin.cpp
  • hmailserver/source/Server/Common/Application/BackupExecuter.cpp
  • hmailserver/source/Server/Common/Application/IniFileSettings.cpp
  • hmailserver/source/Server/Common/Application/IniFileSettings.h
  • hmailserver/source/Server/Common/BO/Messages.cpp
  • hmailserver/source/Server/Common/Persistence/PersistentMessage.cpp
  • hmailserver/source/Server/Common/Persistence/PersistentMessage.h
  • hmailserver/source/Server/Common/Scripting/Events.cpp
  • hmailserver/source/Server/Common/Scripting/ScriptServer.cpp
  • hmailserver/source/Server/Common/Scripting/ScriptServer.h
  • hmailserver/source/Server/Common/TCPIP/IPAddress.cpp
  • hmailserver/source/Server/Common/Util/File.cpp
  • hmailserver/source/Server/IMAP/IMAPCommandSearch.cpp
  • hmailserver/source/Server/SMTP/RecipientParser.cpp
  • hmailserver/source/Server/SMTP/SMTPConnection.cpp
And 'Version.h' to reflect the current build nr.
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2017-09-19 09:33

5.6.7-B2415.11

This is the latest beta Version 5.6.7 - Build 2415 (2017-07-09)
It contains the previous fixes/additions from 5.6.7-B2405.10, plus:
  • Experimental eventhandler OnClientLogon(oClient), New ClientInfo property oClient.Authenticated (Boolean)
Note:
  • In the OnClientLogon(oClient) event, oClient.Username always holds the value passed when authenticating the user, in later events like OnSmtpData, OnAcceptMessage the oClient.Username is empty when authentication has failed (to be compatible with current behavior/scripts)
  • OnSmtpData, OnAcceptMessage events can also make use of the value oClient.Authenticated (Boolean)

Code: Select all

Sub OnClientLogon(oClient)
	If oClient.Authenticated then
		EventLog.Write("Successful login for " & oClient.Username & " from " & oClient.IpAddress & " on port " & oClient.Port & "")
	Else
		EventLog.Write("Failed login for " & oClient.Username & " from " & oClient.IpAddress & " on port " & oClient.Port & "")
	End if
End Sub
Source:
github.com/RvdHout/hmailserver

Download:
5.6.7-B2415.11.7z
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Sub OnHELO(oClient) progress?

Post by mattg » 2017-09-25 11:23

I've been using this for over a week - looks good

however...
I have thunderbird open all day with some 14 mail accounts connected to my server.

My errors@example.com gets sent automated errors from various systems and scripts etc, and today there is a device that is sending that address an email every five minutes or so.
No big deal, just need to sort this device out

In my event log I get

Code: Select all

2380	"2017-09-25 16:49:07.533"	"Successful login for errors@example.com from 192.168.0.1 on port 143"
2380	"2017-09-25 16:49:07.689"	"Successful login for errors@example.com from 192.168.0.1 on port 143"
8040	"2017-09-25 16:49:07.845"	"Successful login for errors@example.com from 192.168.0.1 on port 143"
7360	"2017-09-25 16:49:08.017"	"Successful login for errors@example.com from 192.168.0.1 on port 143"
7360	"2017-09-25 16:49:08.173"	"Successful login for errors@example.com from 192.168.0.1 on port 143"
2992	"2017-09-25 16:49:08.349"	"Successful login for errors@example.com from 192.168.0.1 on port 143"
8040	"2017-09-25 16:49:08.517"	"Successful login for errors@example.com from 192.168.0.1 on port 143"
8996	"2017-09-25 16:49:08.879"	"Successful login for errors@example.com from 192.168.0.1 on port 143"
2380	"2017-09-25 16:49:09.048"	"Successful login for errors@example.com from 192.168.0.1 on port 143"
5052	"2017-09-25 16:49:09.580"	"Successful login for errors@example.com from 192.168.0.1 on port 143"
8996	"2017-09-25 16:49:10.298"	"Successful login for errors@example.com from 192.168.0.1 on port 143"
None of MY other accounts get logged
This account is the only account that has new mail
There is only 11 lines, not 14
This repeats every five minutes as the new email to example.com arrives
All accounts are set for IMAP IDLE

None of this is really important, just thought you'd like to know

Every time I send an email, it registers as logging on, and everytime I receive an email I get multiple entries as above.
Someone logging on via POP3 gets logged as expected, all outgoing mail is logged, I guess as expected, just the IMAP accounts when receiving new mail.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2017-09-27 12:10

I think this behaviour is correct...
If you enable debug logging you'lll notice the IMAP client sends the 'A0002 LOGIN' command when fetching new messages, for example roundcube does this with every IMAP command send to the server
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Sub OnHELO(oClient) progress?

Post by mattg » 2017-10-08 02:30

Because I have been using this build I also now have access to Sub OnHELO

This is heaps of fun.
swap 'XXX.XXX.XXX.XXX' for your public IP address

I use the random rejection messages in a few other places in my eventhandlers.vbs now. Thanks for the idea SorenR.
Not that it matters, because I don't think that anyone is listening to my witty auto-replies.

Code: Select all

Sub OnHELO(oClient)
	If oClient.helo = "XXX.XXX.XXX.XXX" Or oClient.helo = "[XXX.XXX.XXX.XXX]"  Or oClient.helo = "ylmf-pc" Or oClient.helo = "User" Then
		Result.Message = RandomRejection()
		EventLog.Write("Random Rejection sent to IP - " & oClient.IPAddress & "  Rejection Message was '" & Result.Message & "'")
		Result.Value = 2
	End If    
End Sub

Function RandomRejection()
	Dim i, RejectionStrings(10)
	RejectionStrings(0) = "I'm sorry Jim, I can't do that"
	RejectionStrings(1) = "Hasta la vista, Baby"
	RejectionStrings(2) = "Have you tried switching it off and then on again"
	RejectionStrings(3) = "It's hard to have a battle of wits with someone who is half armed"
	RejectionStrings(4) = "Computer says 'No'"
	RejectionStrings(5) = "Artificial intelligence is no match for natural stupidity"
	RejectionStrings(6) = "Oooops LOL"
	RejectionStrings(7) = "Here comes Karma..."
	RejectionStrings(8) = "How did that happen?"
	RejectionStrings(9) = "Now go away or I shall taunt you again"
	RejectionStrings(10) = "The last thing I want to do is insult you. But it IS on the list."
	Randomize
	i = int(Rnd()*11)
	RandomRejection = RejectionStrings(i)
End Function
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2017-10-08 17:29

Nice one :lol:

Btw, have you checked IMAP logging (i wrote debug before i see now) and do you agree the behavior resulting in the excessive 'Successful login' logging is caused by client sending LOGIN commands?
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Sub OnHELO(oClient) progress?

Post by mattg » 2017-10-25 05:26

I've noticed that a few times my hMailserver has simply stopped accepting new connections, and I need to restart via Windows Services.

This seems to stop after something happens in my OnHelo sub

"SMTPD" 6692 2922 "2017-10-25 11:34:08.364" "203.113.204.51" "RECEIVED: EHLO User"
"DEBUG" 6692 "2017-10-25 11:34:08.364" "Executing event OnHELO"
"DEBUG" 3756 "2017-10-25 11:34:34.118" "No messages to index."
"DEBUG" 3756 "2017-10-25 11:35:36.489" "No messages to index."
"DEBUG" 3756 "2017-10-25 11:36:38.849" "No messages to index."
"DEBUG" 3756 "2017-10-25 11:37:41.208" "No messages to index."
"DEBUG" 3756 "2017-10-25 11:38:43.568" "No messages to index."
"DEBUG" 3756 "2017-10-25 11:39:45.958" "No messages to index."
"DEBUG" 3756 "2017-10-25 11:40:48.333" "No messages to index."
"DEBUG" 3756 "2017-10-25 11:41:50.701" "No messages to index."

The only thing that continues to work is External account download, nothing else...until I restart the service

Code: Select all

Sub OnHELO(oClient)
	Dim tempString
	EventLog.Write("Details for OnHELO - IP Address = " & oClient.IPAddress & "  Helo sent was '" & CStr(oClient.helo) & "'")
	If oClient.helo = "ylmf-pc" Or oClient.helo = "User" Then
 		TempString = "    Helo = '" & oClient.helo & "' was Autobanned for using known spammer EHLO"
		Call CustomMonthlyLog(TempString, "Port" & oClient.port)
		Call AutobanIP(oClient.ipaddress, 14, "Using My IP as EHLO")
		Result.Message = RandomRejection()
		EventLog.Write("Random Rejection sent to IP - " & oClient.IPAddress & "  Rejection Message was '" & Result.Message & "'")
    	Wait(14)
		Result.Value = 2
	End If    
End Sub
This was written to my events log
6692 "2017-10-25 11:34:08.364" "Details for OnHELO - IP Address = 203.113.204.51 Helo sent was 'User'"

This was written to my port 25 log (this connection was port 25)
2017-10-25 11:34:08.394 Helo = 'User' was Autobanned for using known spammer EHLO

But then no Autoban happened.

Obviously ends unexpectedly somewhere in my AutoBanIP sub.
I've just added some eventlog lines to my AutoBanIP sub to try and find where it stops

Here is what it currently looks like, and it has been banning ~15/25 per day from elsewhere in my eventhandlers.vbs, mostly overseas connections to IMAP or POP3 ports

Code: Select all

Sub AutobanIP(IPAddress, NumberOfDays, ReasonForBan)
	EventLog.Write("Autoban IP Address started for IP = " & IPAddress & " For " & NumberofDays & " Number of days for reason " & Reasonforban)
	Dim oApp
	Set oApp = CreateObject("hMailServer.Application")

' 		Give this script permission to access all
' 		hMailServer settings.
	Call oApp.Authenticate("Administrator", g_sAdminPassword)

	Dim oSecurityRange, i
	For i = 0 To oApp.Settings.SecurityRanges.Count -1
		If IPAddress = oApp.Settings.SecurityRanges.Item(i).LowerIP Then Exit sub
	Next
	Set oSecurityRange = oApp.Settings.SecurityRanges.Add()
	EventLog.Write("Autoban IP range being set for IP Address " & IPAddress)
	With oSecurityRange
		.lowerip = ipaddress
		.upperip = ipaddress
		.priority = 20
		.allowdeliveryfromlocaltolocal = False
		.allowdeliveryfromlocaltoremote = False
		.allowdeliveryfromremotetolocal = False
		.allowdeliveryfromremotetoremote = False
		.allowimapconnections = False
		.allowsmtpconnections = False
		.allowpop3connections = False
		.expires = True
		.ExpiresTime = DateAdd("d", NumberOfDays, Now())
		.name = ReasonForBan & " - banned for " & NumberOfDays & " days" 
'			.name = "script " & IPAddress & " - banned for " & NumberOfDays & " days - Reason is " & ReasonForBan
		i = False
		While i = False
			On Error Resume Next
			.save
			If Err.Number = 0 Then
				EventLog.Write("Autoban IP range saved for IP Address " & IPAddress)
				On Error Goto 0
				i = True
				Exit Sub
			End If
			Wait(1)
		Wend
	End With
End Sub
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Sub OnHELO(oClient) progress?

Post by mattg » 2017-10-25 05:44

Just had another (whilst typing this)

"DEBUG" 7516 "2017-10-25 13:21:41.112" "TCP connection started for session 79"
"SMTPD" 7516 79 "2017-10-25 13:21:41.112" "203.113.204.51" "SENT: 220 example.com ESMTP"
"SMTPD" 2588 79 "2017-10-25 13:21:41.144" "203.113.204.51" "RECEIVED: EHLO User"
"DEBUG" 2588 "2017-10-25 13:21:41.144" "Executing event OnHELO"
"DEBUG" 7512 "2017-10-25 13:22:13.472" "No messages to index."
"DEBUG" 7512 "2017-10-25 13:23:15.836" "No messages to index."
"DEBUG" 7512 "2017-10-25 13:24:18.206" "No messages to index."

"DEBUG" 8044 "2017-10-25 13:26:41.145" "The client has timed out. Session: 79"
"SMTPD" 8044 79 "2017-10-25 13:26:41.145" "203.113.204.51" "SENT: 421 Connection timeout."
"DEBUG" 8772 "2017-10-25 13:26:46.159" "The client has timed out. Session: 79"


My eventlog says
2588 "2017-10-25 13:21:41.191" "Autoban IP Address started for IP = 203.113.204.51 For 14 Number of days for reason Using My IP as EHLO"
2588 "2017-10-25 13:21:41.300" "Autoban IP range being set for IP Address 203.113.204.51"


The save didn't happen...?

But no Autoban created (I've done it manually now)
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Sub OnHELO(oClient) progress?

Post by RvdH » 2017-10-25 10:05

Can't say i have experienced something like this...or at least i did not notice it


I use SorenR's scripts to autoban

Code: Select all

   Function Wait(sec)
      With CreateObject("WScript.Shell")
         .Run "timeout /T " & Int(sec), 0, True
'        .Run "sleep -m " & Int(sec * 1000), 0, True
'        .Run "powershell Start-Sleep -Milliseconds " & Int(sec * 1000), 0, True
      End With
   End Function

   Function LockFile(strPath)
      Const Append = 8
      Const Unicode = -1
      With CreateObject("Scripting.FileSystemObject")
         Dim oFile, i
         For i = 0 To 30
            On Error Resume Next
            Set oFile = .OpenTextFile(strPath, Append, True, Unicode)
            If (Not Err.Number = 70) Then
               Set LockFile = oFile
               On Error Goto 0
               Exit For
            End If
            On Error Goto 0
            Wait(1)
         Next
      End With
      Set oFile = Nothing
      If (Err.Number = 70) Then
         EventLog.Write("ERROR: EventHandlers.vbs")
         EventLog.Write("File " & strPath & " is locked and timeout was exceeded.")
         Err.Clear
      ElseIf (Err.Number <> 0) Then
         EventLog.Write("ERROR: EventHandlers.vbs : Function LockFile")
         EventLog.Write("Error       : " & Err.Number)
         EventLog.Write("Error (hex) : 0x" & Hex(Err.Number))
         EventLog.Write("Source      : " & Err.Source)
         EventLog.Write("Description : " & Err.Description)
         Err.Clear
      End If
   End Function

   Function Lookup(strRegEx, strMatch)
      With CreateObject("VBScript.RegExp")
         .Global = False
         .Pattern = strRegEx
         .IgnoreCase = True
         If .Test(strMatch) Then
            Lookup = True
         Else
            Lookup = False
         End If
      End With
   End Function

   '
   ' sType can be one of the following;
   ' "yyyy" Year, "m" Month, "d" Day, "h" Hour, "n" Minute, "s" Second
   '
   Sub AutoBan(sIPAddress, sReason, iDuration, sType)
      Dim oApp : Set oApp = CreateObject("hMailServer.Application")
      Call oApp.Authenticate(ADMIN, PASSWORD)
      With LockFile("c:\hmailserver\temp\autoban.lck")
         On Error Resume Next
         oApp.Settings.SecurityRanges.Refresh
         If (oApp.Settings.SecurityRanges.ItemByName("(" & sReason & ") " & sIPAddress) Is Nothing) Then
            With oApp.Settings.SecurityRanges.Add
               .Name = "(" & sReason & ") " & IPAddress
               .LowerIP = sIPAddress
               .UpperIP = sIPAddress
               .Priority = 20
               .Expires = True
               .ExpiresTime = DateAdd(sType, iDuration, Now())
               .Save
            End With
         End If
         On Error Goto 0
         .Close
      End With
   End Sub
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Sub OnHELO(oClient) progress?

Post by SorenR » 2017-10-26 01:15

mattg wrote:Just had another (whilst typing this)

"DEBUG" 7516 "2017-10-25 13:21:41.112" "TCP connection started for session 79"
"SMTPD" 7516 79 "2017-10-25 13:21:41.112" "203.113.204.51" "SENT: 220 example.com ESMTP"
"SMTPD" 2588 79 "2017-10-25 13:21:41.144" "203.113.204.51" "RECEIVED: EHLO User"
"DEBUG" 2588 "2017-10-25 13:21:41.144" "Executing event OnHELO"
"DEBUG" 7512 "2017-10-25 13:22:13.472" "No messages to index."
"DEBUG" 7512 "2017-10-25 13:23:15.836" "No messages to index."
"DEBUG" 7512 "2017-10-25 13:24:18.206" "No messages to index."

"DEBUG" 8044 "2017-10-25 13:26:41.145" "The client has timed out. Session: 79"
"SMTPD" 8044 79 "2017-10-25 13:26:41.145" "203.113.204.51" "SENT: 421 Connection timeout."
"DEBUG" 8772 "2017-10-25 13:26:46.159" "The client has timed out. Session: 79"


My eventlog says
2588 "2017-10-25 13:21:41.191" "Autoban IP Address started for IP = 203.113.204.51 For 14 Number of days for reason Using My IP as EHLO"
2588 "2017-10-25 13:21:41.300" "Autoban IP range being set for IP Address 203.113.204.51"


The save didn't happen...?

But no Autoban created (I've done it manually now)
If you receive concurrent connections (winthin milliseconds) from the same IP and you try to ban that IP then "i" will never become "true" because "Err.Number" can never become "0" ... The IP was banned from another session during the while...wend loop.

I think... :mrgreen:
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
jimimaseye
Moderator
Moderator
Posts: 10053
Joined: 2011-09-08 17:48

Re: Sub OnHELO(oClient) progress?

Post by jimimaseye » 2017-10-26 08:35

SorenR wrote: If you receive concurrent connections (winthin milliseconds) from the same IP and you try to ban that IP then "i" will never become "true" because "Err.Number" can never become "0" ... The IP was banned from another session during the while...wend loop.

I think... :mrgreen:
So how do we stop this situation? Having the server freeze and require restarting is not very practical (to say the least).
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Sub OnHELO(oClient) progress?

Post by mattg » 2017-10-26 09:30

I dim the i in the sub, it should only have scope for the sub, in the current iteration (run) of the sub.

I've changed to use the lockfile like SorenR does, so we will see if that does better...
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: RE: Re: Sub OnHELO(oClient) progress?

Post by SorenR » 2017-10-26 13:03

jimimaseye wrote:
SorenR wrote: If you receive concurrent connections (winthin milliseconds) from the same IP and you try to ban that IP then "i" will never become "true" because "Err.Number" can never become "0" ... The IP was banned from another session during the while...wend loop.

I think... :mrgreen:
So how do we stop this situation? Having the server freeze and require restarting is not very practical (to say the least).
That's why I did the filelocking thingy, to control when what is done. A form of forced preemptive multitasking. ;-)
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
jimimaseye
Moderator
Moderator
Posts: 10053
Joined: 2011-09-08 17:48

Re: RE: Re: Sub OnHELO(oClient) progress?

Post by jimimaseye » 2017-10-26 13:45

SorenR wrote: So how do we stop this situation? Having the server freeze and require restarting is not very practical (to say the least).
That's why I did the filelocking thingy, to control when what is done. A form of forced preemptive multitasking. ;-)
Right. I didnt realise matt had not included the filelock-check function.
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Sub OnHELO(oClient) progress?

Post by mattg » 2017-10-26 23:20

Stopped again on me overnight
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

mikedibella
Senior user
Senior user
Posts: 837
Joined: 2016-12-08 02:21

Re: Sub OnHELO(oClient) progress?

Post by mikedibella » 2017-10-26 23:53

If you assume you are entering the race condition because the .Save fails due to the presence of a duplicate Autoban entry created on a different thread, wouldn't you see that entry in the database when you restart the service?

If the .Save is failing because a duplicate exists, you need to catch the particular error state in your code and exit from the loop.

I'm not sure I even understand why you think you need to retry Saving indefinitely. Why not just try once and log a failure on error?

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Sub OnHELO(oClient) progress?

Post by mattg » 2017-10-26 23:58

NO Autoban entry exists for that IP, before or after server restart, so it simply isn't created

I will play again with my script and try and catch better errors I think
When my hMailsevrer stopped last night it wasn't due to this OnHELO sub stopping part way through, so I need to investigate what exactly is happening.

I never had the stop issue on the hMailsevrer official Beta build, only since I moved to RVHD's special build. I may need to move back to the official builds too.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

mikedibella
Senior user
Senior user
Posts: 837
Joined: 2016-12-08 02:21

Re: Sub OnHELO(oClient) progress?

Post by mikedibella » 2017-10-27 00:59

I think you will need to find some usable terminal condition to cause an exit from the loop, because if you get into the loop in a state where the .Save fails, you you will loop endlessly and that is probably what is cause the behavior you are seeing.

insomniac2k2
Normal user
Normal user
Posts: 87
Joined: 2016-08-09 19:47

Re: Sub OnHELO(oClient) progress?

Post by insomniac2k2 » 2017-10-29 19:56

While testing some code that i just implemented for GreyWhitelisting, i ran into an interesting find. This issue is consistent and can be replicated on demand:

Using SorenR's regex logic OnHELO I get a proper match from google and successfully call for a Greylist Whitelist addition. But the interesting part is that I seem to get 2 matches and call for an addition for the same IP within milliseconds of each other. This causes an access violation and throws about 30 errors in the log.

Here's what i can confirm:

It's not my executable that is causing the issue, as i am logging the matches from the EventHandler and writing to the event log before the call. See below:

Code: Select all

   Sub OnHELO(oClient)
   '
   ' BEGIN
   '

   Dim strRegEx
   strRegEx = "^[a-z]+[0-9]{2}(-)[a-z]{2}[0-9](-obe\.outbound\.protection\.outlook\.com)$|"     &_
              "^(a)[0-9]{1,2}(-)[0-9]{1,3}(\.smtp-out\.amazonses\.com)$|"                       &_
              "^(mail-)[a-z]{2}[0-9](-f)[0-9]{1,3}(\.google\.com)$|"                            &_
              "^(spring-chicken-)[a-z]{2}(\.twitter\.com)$|"                                    &_
              "^(mail)[a-z](-)[a-z]{2}(\.linkedin\.com)$|"                                      &_
              "^(mx)[0-9](\.)[a-z]{3}(\.paypal\.com)$|"                                         &_
              "^(mx-out\.facebook\.com)$|"                                                      &_
              "^(relay99\.mysmtp\.com)$|"                                                       &_
              "(\.lifewavemembers\.com)$|"                                                      &_
              "(\.companymobile\.dk)$|"                                                         &_
              "(\.gratisdns\.dk)$|"                                                             &_
              "(\.electric\.net)$|"                                                             &_
              "(\.jobindex\.dk)$|"                                                              &_
              "(\.anpdm\.com)$|"                                                                &_
              "(\.exigo\.com)$|"                                                                &_
              "(\.post\.dk)$"
   If Lookup(strRegEx, oClient.HELO) Then
	EventLog.Write("Message from: " & oClient.IPAddress & " " & oClient.HELO & " Added as to Greylist Whitelist do to match in approved list") 
        Set oShell = CreateObject("WScript.Shell")
	oShell.Run """C:\Program Files (x86)\hMailServer\Bin\ProxyAuth.exe"" -secret Mangos -greywhitelist " & oClient.IPAddress & " " & oClient.HELO
      Exit Sub
   End If

   '
   ' END
   '
   End Sub
The google ip address that is being added is this. Notice the redundant entries that happen within milliseconds of each other.:
6060 "2017-10-29 10:39:15.920" "Message from: 74.125.82.50 mail-wm0-f50.google.com Added as to Greylist Whitelist do to match in approved list"
6560 "2017-10-29 10:39:16.639" "Message from: 74.125.82.50 mail-wm0-f50.google.com Added as to Greylist Whitelist do to match in approved list"

This throws the DB error:

"ERROR" 1820 "2017-10-29 10:20:14.588" "Severity: 2 (High), Code: HM5032, Source: DALConnection::Execute, Description: Source: SQLCEConnection::Execute(), Code: HM10044, Description: Error while executing SQL statement:
INSERT INTO hm_greylisting_whiteaddresses (whiteipaddress, whiteipdescription) VALUES (@whiteipaddress2, @whiteipdescription3)
Microsoft SQL Server Compact OLE DB Provider
Value violated the integrity constraints for a column or table."
"ERROR" 5048 "2017-10-29 10:23:29.850" "Severity: 2 (High), Code: HM5032, Source: DALConnection::Execute, Description: Source: SQLCEConnection::Execute(), Code: HM10044, Description: Error while executing SQL statement:
INSERT INTO hm_greylisting_whiteaddresses (whiteipaddress, whiteipdescription) VALUES (@whiteipaddress2, @whiteipdescription3)
Microsoft SQL Server Compact OLE DB Provider
Value violated the integrity constraints for a column or table."

So here is my first guess to what could cause this behavior without cracking the source open:

Is a .tmp file created before OnHELO? If so, it may be possible that this is causing a redundant check against the host, thus causing duplicate matches and duplicate calls? I still have to debug more, but wanted to post my findings in case someone has seen this behavior with OnHELO already.


Actually, reading SorenR's comment above, the issue looks VERY similar.

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: RE: Re: Sub OnHELO(oClient) progress?

Post by SorenR » 2017-10-29 23:15

insomniac2k2 wrote:While testing some code that i just implemented for GreyWhitelisting, i ran into an interesting find. This issue is consistent and can be replicated on demand:

Using SorenR's regex logic OnHELO I get a proper match from google and successfully call for a Greylist Whitelist addition. But the interesting part is that I seem to get 2 matches and call for an addition for the same IP within milliseconds of each other. This causes an access violation and throws about 30 errors in the log.

Here's what i can confirm:

It's not my executable that is causing the issue, as i am logging the matches from the EventHandler and writing to the event log before the call. See below:

Code: Select all

   Sub OnHELO(oClient)
   '
   ' BEGIN
   '

   Dim strRegEx
   strRegEx = "^[a-z]+[0-9]{2}(-)[a-z]{2}[0-9](-obe\.outbound\.protection\.outlook\.com)$|"     &_
              "^(a)[0-9]{1,2}(-)[0-9]{1,3}(\.smtp-out\.amazonses\.com)$|"                       &_
              "^(mail-)[a-z]{2}[0-9](-f)[0-9]{1,3}(\.google\.com)$|"                            &_
              "^(spring-chicken-)[a-z]{2}(\.twitter\.com)$|"                                    &_
              "^(mail)[a-z](-)[a-z]{2}(\.linkedin\.com)$|"                                      &_
              "^(mx)[0-9](\.)[a-z]{3}(\.paypal\.com)$|"                                         &_
              "^(mx-out\.facebook\.com)$|"                                                      &_
              "^(relay99\.mysmtp\.com)$|"                                                       &_
              "(\.lifewavemembers\.com)$|"                                                      &_
              "(\.companymobile\.dk)$|"                                                         &_
              "(\.gratisdns\.dk)$|"                                                             &_
              "(\.electric\.net)$|"                                                             &_
              "(\.jobindex\.dk)$|"                                                              &_
              "(\.anpdm\.com)$|"                                                                &_
              "(\.exigo\.com)$|"                                                                &_
              "(\.post\.dk)$"
   If Lookup(strRegEx, oClient.HELO) Then
	EventLog.Write("Message from: " & oClient.IPAddress & " " & oClient.HELO & " Added as to Greylist Whitelist do to match in approved list") 
        Set oShell = CreateObject("WScript.Shell")
	oShell.Run """C:\Program Files (x86)\hMailServer\Bin\ProxyAuth.exe"" -secret Mangos -greywhitelist " & oClient.IPAddress & " " & oClient.HELO
      Exit Sub
   End If

   '
   ' END
   '
   End Sub
The google ip address that is being added is this. Notice the redundant entries that happen within milliseconds of each other.:
6060 "2017-10-29 10:39:15.920" "Message from: 74.125.82.50 mail-wm0-f50.google.com Added as to Greylist Whitelist do to match in approved list"
6560 "2017-10-29 10:39:16.639" "Message from: 74.125.82.50 mail-wm0-f50.google.com Added as to Greylist Whitelist do to match in approved list"

This throws the DB error:

"ERROR" 1820 "2017-10-29 10:20:14.588" "Severity: 2 (High), Code: HM5032, Source: DALConnection::Execute, Description: Source: SQLCEConnection::Execute(), Code: HM10044, Description: Error while executing SQL statement:
INSERT INTO hm_greylisting_whiteaddresses (whiteipaddress, whiteipdescription) VALUES (@whiteipaddress2, @whiteipdescription3)
Microsoft SQL Server Compact OLE DB Provider
Value violated the integrity constraints for a column or table."
"ERROR" 5048 "2017-10-29 10:23:29.850" "Severity: 2 (High), Code: HM5032, Source: DALConnection::Execute, Description: Source: SQLCEConnection::Execute(), Code: HM10044, Description: Error while executing SQL statement:
INSERT INTO hm_greylisting_whiteaddresses (whiteipaddress, whiteipdescription) VALUES (@whiteipaddress2, @whiteipdescription3)
Microsoft SQL Server Compact OLE DB Provider
Value violated the integrity constraints for a column or table."

So here is my first guess to what could cause this behavior without cracking the source open:

Is a .tmp file created before OnHELO? If so, it may be possible that this is causing a redundant check against the host, thus causing duplicate matches and duplicate calls? I still have to debug more, but wanted to post my findings in case someone has seen this behavior with OnHELO already.


Actually, reading SorenR's comment above, the issue looks VERY similar.
If you look back in your other thread you'll see in my GreyWhiteList code i am using my eventlocking to avoid violating the database constraint. ;-)
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

Post Reply