Greylisting by subnetwork

Use this forum if you want to suggest a new feature to hMailServer. Before posting, please search the forum to confirm that it has not already been suggested.
gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Greylisting by subnetwork

Post by gelbukh » 2018-06-13 22:16

Please store in the greylist triplet only the subnet part of the IP, such as 123.123.123.* or even 123.123.*.*.

Reason: Many services nowadays use multiple (often hundreds if not thousands) IPs for sending mail. Such mail never passes greylisting because each connection attempt is made from a different IP.

Bypassing greylisting by SPF may alleviate the problem for large services that have their SPF correctly configured, but I have seen too many legitimate servers, such as university department servers, that do not have SPF correctly configured, so mail from them never passes greylisting.

This renders greylisting unusable since the damage from missing an important business message is much greater than the annoyance from spam.

However, such servers usually use addresses from the same subnetwork, such as 123.123.123.11, 123.123.123.12, 123.123.123.13, etc. Identifying the sender by the subnet, such as 123.123.123.*, would GREATLY reduce missed messages and GREATY improve delivery time from such servers, while almost not increase chances for a spam message to pass.

User avatar
Dravion
Senior user
Senior user
Posts: 2071
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: Greylisting by subnetwork

Post by Dravion » 2018-06-14 00:17

hMailServers internal design isnt ready for such massive tasks. If you want to block thousands of IPs with a fast rate its a job for the Windows Firewall or a DMZ Gatekeeping Firewall. The other mitigation technique for such scenarios is blocking entire DNS-Zones at DNS-Server level (for example: Bind DNS-Server RPZ DNS-Firewall feature allows something like this).

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Greylisting by subnetwork

Post by mattg » 2018-06-14 01:09

Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

BramV
New user
New user
Posts: 7
Joined: 2014-01-30 11:28

Re: Greylisting by subnetwork

Post by BramV » 2018-07-11 17:53

Yes, greylist whitelisting IPs by CIDR notation could be really helpful!
Another nice option would be by domain like in postgrey. Then with one rule we could for example whitelist Outlook/Office365: *outbound.protection.outlook.com

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-07-11 22:16

BramV wrote:
2018-07-11 17:53
Another nice option would be by domain like in postgrey. Then with one rule we could for example whitelist Outlook/Office365: *outbound.protection.outlook.com
Well... ;-)

Code: Select all

Option Explicit

'  ******************************************************************************************************************************
'  ********** Settings                                                                                                 **********
'  ******************************************************************************************************************************

   ' COM authentication
   Private Const ADMIN = "Administrator"
   Private Const PASSWORD = "You'd like that, eh?"

'  ******************************************************************************************************************************
'  ********** Functions                                                                                                **********
'  ******************************************************************************************************************************

   Function Wait(sec)
      With CreateObject("WScript.Shell")
         .Run "timeout /T " & Int(sec), 0, True
'        .Run "sleep -m " & Int(sec * 1000), 0, True
'        .Run "powershell Start-Sleep -Milliseconds " & Int(sec * 1000), 0, True
      End With
   End Function

   Function LockFile(strPath)
      Const Append = 8
      Const Unicode = -1
      With CreateObject("Scripting.FileSystemObject")
         Dim oFile, i
         For i = 0 To 30
            On Error Resume Next
            Set oFile = .OpenTextFile(strPath, Append, True, Unicode)
            If Not (Err.Number = 70) Then
               Set LockFile = oFile
               On Error Goto 0
               Exit For
            End If
            On Error Goto 0
            Wait(1)
         Next
      End With
      Set oFile = Nothing
      If (Err.Number = 70) Then
         EventLog.Write("ERROR: EventHandlers.vbs")
         EventLog.Write("File " & strPath & " is locked and timeout was exceeded.")
         Err.Clear
      ElseIf (Err.Number <> 0) Then
         EventLog.Write("ERROR: EventHandlers.vbs : Function LockFile")
         EventLog.Write("Error       : " & Err.Number)
         EventLog.Write("Error (hex) : 0x" & Hex(Err.Number))
         EventLog.Write("Source      : " & Err.Source)
         EventLog.Write("Description : " & Err.Description)
         Err.Clear
      End If
   End Function

   Function Lookup(strRegEx, strMatch) : Lookup = False
      With CreateObject("VBScript.RegExp")
         .Global = False
         .Pattern = strRegEx
         .IgnoreCase = True
         If .Test(strMatch) Then Lookup = True
      End With
   End Function

'  ******************************************************************************************************************************
'  ********** Subroutines                                                                                              **********
'  ******************************************************************************************************************************

   Sub GreyWhiteList(strIP, strDesc)
      Dim oApp : Set oApp = CreateObject("hMailServer.Application")
      Call oApp.Authenticate(ADMIN, PASSWORD)
      With LockFile("c:\hmailserver\temp\greywhitelist.lck")
         On Error Resume Next
         oApp.Settings.AntiSpam.GreyListingWhiteAddresses.Refresh
         If (oApp.Settings.AntiSpam.GreyListingWhiteAddresses.ItemByName(strIP) Is Nothing) Then
            With oApp.Settings.AntiSpam.GreyListingWhiteAddresses.Add
               .Description = "# " & Date & " " & strDesc
               .IPAddress = strIP
               .Save
            End With
         Else
            With oApp.Settings.AntiSpam.GreyListingWhiteAddresses.ItemByName(strIP)
               .Description = "# " & Date & " " & strDesc
               .Save
            End With
         End If
         On Error Goto 0
         .Close
      End With
   End Sub

   Sub ExpireGreyList
      Dim j, m_Days : m_Days = 30
      Dim oApp : Set oApp = CreateObject("hMailServer.Application")
      Call oApp.Authenticate(ADMIN, PASSWORD)
      For j = oApp.Settings.AntiSpam.GreyListingWhiteAddresses.Count-1 To 0 Step -1
         If (Left(oApp.Settings.AntiSpam.GreyListingWhiteAddresses(j).Description, 1) = "#") Then
            If (DateDiff("d", Mid(oApp.Settings.AntiSpam.GreyListingWhiteAddresses(j).Description,3,10), Date) > m_Days) Then
               oApp.Settings.AntiSpam.GreyListingWhiteAddresses(j).Delete
            End If
         End If
      Next
   End Sub

'  ******************************************************************************************************************************
'  ********** hMailServer Triggers                                                                                     **********
'  ******************************************************************************************************************************

   Sub OnHELO(oClient)
      Dim strRegEx

      '
      ' Dynamic GreyList Whitelist
      '
      strRegEx = "^[a-z]+[0-9]{2}(-)[a-z]{2}[0-9](-obe\.outbound\.protection\.outlook\.com)$|" &_
                 "^(((mail134-)[0-9]{1,3})|((out22-)[0-9]{1,2}))(\.mail\.alibaba\.com)$|" &_
                 "^(a)[0-9]{1,2}(-)[0-9]{1,3}(\.smtp-out\.amazonses\.com)$|" &_
                 "^(mail-)[a-z]{2}[0-9](-f)[0-9]{1,3}(\.google\.com)$|" &_
                 "^(spring-chicken-)[a-z]{2}(\.twitter\.com)$|" &_
                 "^(mail)[a-z](-)[a-z]{2}(\.linkedin\.com)$|" &_
                 "^(mx)[0-9](\.)[a-z]{3}(\.paypal\.com)$"

      If Lookup(strRegEx, oClient.HELO) Then
         Call GreyWhiteList(oClient.IPAddress, oClient.HELO)
      End If

   End Sub
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Greylisting by subnetwork

Post by mattg » 2018-07-12 00:22

As usual, excellent work SorenR

From research I've found that the OnHELO sub is only populated on the second connection. This happens only with StartTLS connections to your mailserver.

On first OnHELO connection, oClient.helo is blank
After negotiating a successful StartTLS, OnHELO is triggered a second time, and this time oClient.helo is populated appropriately.

So, I think for this script to work, hMailserver needs a SSL certificate installed, and to have 'StartTLS (optional)' set for port 25, and for the sending server to try StartTLS.

All legitimate connections from outlook.com are StartTLS enabled were accepted
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-07-12 10:58

mattg wrote:
2018-07-12 00:22
As usual, excellent work SorenR

From research I've found that the OnHELO sub is only populated on the second connection. This happens only with StartTLS connections to your mailserver.

On first OnHELO connection, oClient.helo is blank
After negotiating a successful StartTLS, OnHELO is triggered a second time, and this time oClient.helo is populated appropriately.

So, I think for this script to work, hMailserver needs a SSL certificate installed, and to have 'StartTLS (optional)' set for port 25, and for the sending server to try StartTLS.

All legitimate connections from outlook.com are StartTLS enabled were accepted
Ah.. I don't use TLS, it's either SSL or No SSL and unsecure connections are receive-only ;-) Don't like to give people a choice when it comes to security :twisted:
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-07-12 12:19

Actually, I have switched off Greylisting since several months ago...

I use a 20 second delay on "OnClientConnect", "OnHELO" and "OnSMTPData" and that seems to filter out the worst hogs.
Then I filter the HELO/EHLO greeting ... and impose a complete rulebook of SPAM Catchers before delivering mail to recipients - that is, some mails go to my SPAM account - ONLY!

An important note ... My Rules are NOT processed if SpamAssassin/DNSBL/SURBL/SPF/DKIM found something fishy. If it's SPAM, it's SPAM. It cannot be more or less SPAM!

By not refusing everything (I Ban only when absolutely positive it's a SPAMMER), I can set up whitelists to capture mis-configured mailservers or fix my own prejudice.
This RegEx "/(iPhone (3G|4|5|6|SE|7|8|9|X)(C|S)?( Plus)?)/gmi" I commonly use in Subject or Body, mostly I'm correct in marking as SPAM, but there are execptions.

Code: Select all

   Function oLookup(strRegEx, strMatch, bGlobal)
      With CreateObject("VBScript.RegExp")
         .Global = bGlobal
         .Pattern = strRegEx
         .IgnoreCase = True
         Set oLookup = .Execute(strMatch)
      End With
   End Function

   Sub OnHELO(oClient)
      Dim strRegEx, Match, Matches

      '
      ' Exclude Backup-MX & local LAN from test
      '
      If (Left(oClient.IPAddress, 10) = "80.160.77.") Then Exit Sub
      If (Left(oClient.IPAddress, 10) = "192.168.0.") Then Exit Sub

      '
      ' Filter out "impatient" servers. Alternative to GreyListing.
      '
      If (oClient.Port = 25) Then Wait(20)

      '
      ' Deny servers with specific HELO/EHLO greetings
      '
      strRegEx = "^(\[#MY#\.#IP#\.#ADDRESS#\.#HERE#\])$|" &_
                 "^(#MY#\.#DOMAIN#)$|" &_
                 "^(#MY#\.#FQDN#\.#HERE#)$|" &_
                 "^(.*\.[a-z]{4,})$|" &_
                 "(0\.0\.0\.0)|" &_
                 "(127(?:\.[0-9]{1,3}){3})"

      Set Matches = oLookup(strRegEx, oClient.HELO, False)
      If Matches.Count > 0 Then
         For Each Match In Matches
            Result.Value = 2
            Result.Message = "5.7.1 CODE02 Your access to this mail system has been rejected due to the sending MTA's poor reputation. If you believe that this failure is in error, please contact the intended recipient via alternate means. {" & Match.Value & "}"
            Call AutoBan(oClient.IPAddress, "BLACKLIST - " & oClient.HELO, 7, "d")
            Exit Sub
         Next
      End If

      '
      ' Validate HELO/EHLO greeting
      '

      Const strFQDN = "^(?=^.{1,254}$)(^(?:(?!\.|-)([a-z0-9\-\*]{1,63}|([a-z0-9\-]{1,62}[a-z0-9]))\.)+(?:[a-z]{2,})$)$"
      Const strIPv4 = "^\[(?:[0-9]{1,3}\.){3}[0-9]{1,3}\]$"
      Const strIPv6 = "^\[(IPv6)((?:[0-9A-Fa-f]{0,4}:){1,7}(?:(?:(>25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)|[0-9A-Fa-f]{1,4}))\]$"
      Const myKnown = "^(LouisesPC)$|^(LouisesHuawei)$"

      strRegEx = strFQDN & "|" & strIPv4 & "|" & strIPv6 & "|" & myKnown

      If (Lookup(strRegEx, oClient.HELO) = False) Then
         Result.Value = 2
         Result.Message = "5.7.1 CODE03 Your access to this mail system has been rejected due to the sending MTA's poor reputation. If you believe that this failure is in error, please contact the intended recipient via alternate means."
         Call AutoBan(oClient.IPAddress, "Bad HELO - " & oClient.HELO, 7, "d")
         Exit Sub
      End If
   End Sub
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-11 04:58

Thank you for your comments!

I think some of users who commented on this missed my point.

I do NOT mean massive blacklisting, so this does not have anything to do with Windows firewall. If any, my suggestion can be seen as massive whitelisting.

I do NOT ask what other ways of blocking spam there are.

This is a feature request on how greylisting should work in hMailServer.

Currently: by default, all IPs are blocked, but if an IP such as 123.123.123.10 contacts the server, then all four bytes of this IP are stored in the DB, and next time if it contacts the server, it will pass.

My proposal: instead, store in the DB only three first bytes: store 123.123.123, so next time when any IP in the range 123.123.123.0 to 123.123.123.255 (that is, 123.123.123.*) contacts the server, it is unblocked.

This is very minor change to the code (a couple of lines of code: just set the last byte of the IP to 0 when storing it in the DB and when checking the new IP against the DB), which will GREATLY improve usability of hMailServer and the quality of life of its users.

I can provide the specific line of the code that is to be changed.

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-11 05:15

Actually here is it:

In the file GreyList.cpp, after line 45:

replace

std::shared_ptr<GreyListTriplet> pTriplet = PersistentGreyList::GetRecord(sSenderAddress, sRecipientAddress, remoteIP);

by

std::shared_ptr<GreyListTriplet> pTriplet = PersistentGreyList::GetRecord(sSenderAddress, sRecipientAddress, remoteIP & 0xFFFFFF00);

and

pTriplet->SetIPAddress(remoteIP);

by

pTriplet->SetIPAddress(remoteIP & 0xFFFFFF00);

That's all that you need to make thousands of people a lot happier and more productive. Not too complicated, either.

User avatar
katip
Senior user
Senior user
Posts: 1158
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Greylisting by subnetwork

Post by katip » 2018-09-11 08:15

gelbukh wrote:
2018-09-11 04:58
My proposal: instead, store in the DB only three first bytes: store 123.123.123, so next time when any IP in the range 123.123.123.0 to 123.123.123.255 (that is, 123.123.123.*) contacts the server, it is unblocked.

This is very minor change to the code (a couple of lines of code: just set the last byte of the IP to 0 when storing it in the DB and when checking the new IP against the DB), which will GREATLY improve usability of hMailServer and the quality of life of its users.
i agree, partially.
however particularly big providers (Hotmail, Gmail...) rotate beyond /24.
Greylisting in HMS (as it is) is useless in time/mission critical email exchange. this has been discussed :
http://www.hmailserver.com/forum/viewto ... ng#p185133
Katip
--
HMS 5.7, MariaDB 10.4.10, SA 4.0.0, ClamAV 0.103.8

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-11 23:31

Yes, big providers rotate beyond /24. My proposal will alleviate this problem 255 times, even if will not solve it completely. Ideally, the mask is to be user-configurable (maybe /16 is a better value by hit/miss balance, we don't know), if you are willing to modify more than two lines of code.

Yes, greylisting is bad for time-critical exchange (email is bad for it anyway). But without greylisting, email is useless at all -- all messages are lost in spam. Spam filters are much worse for mission-critical exchange than greylisting. So, with greylisting, a tiny fraction of users potentially may experience delays with a tiny fraction of their messages (for almost all users, almost all messages are not time-critical); without greylisting most users lose most of their messages (just not seeing them in the heap of spam) or lose mission-critical messages (caught in spam filters). I prefer to lose a tiny fraction over losing it all, but those in time-critical business can just turn it off.

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-09-12 00:02

I used to run GreyListing and a custom script to cater for a dynamic host list (5'th post in this thread ;-) ), but have now skipped it completely. I have a 20 second wait inserted for port 25 at "Sub OnClientConnect(oClient)", "Sub OnHELO(oClient)" and "Sub OnSMTPData(oClient, oMessage)" and it works just as fine, actually better :mrgreen:

After having fought the fu***ers for a couple of years now, I do see less and less meaningless traffic - except for the occational bot. :mrgreen:
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-12 01:55

Yes, there are other (and maybe better) ways of fighting spam.

However, this thread is not about "how else can we fight spam?". This is a specific feature request for this specific feature present in HMS.

Who should I address for these two lines of code to be changed? Or, is there a good argument against the requested feature? (Other than "greylisting is not the only way to fight spam".)

The 20 sec trick may be an excellent idea, but involves some risk: some legitimate server (say, from some university) can have 15 sec timeout, and then I will have no way ever to know that I am missing ALL mail from this server, and they will have no way at all to contact me. Greylisting does not have this problem.

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-12 02:00

Or, can I implement this my idea via a script? Such as in some event handler temporarily changing the incoming IP before greylisting, and then restoring the original IP? I did not find it.

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Greylisting by subnetwork

Post by mattg » 2018-09-12 06:42

If you use a custom build with OnHelo event >> http://www.hmailserver.com/forum/viewto ... 60#p203420

Then you could use the OnHelo event to do that

This is what I have done in past (note that most is commented out, I don't find this useful in the fight against spam.) You could modify the code to your needs.
If you need I can add the other subroutines that are used but not included

Code: Select all

Sub OnHELO(oClient)
	'custom hMailserver event in RVHD's builds
	'uses functions: CustomMonthlyLog, AutoBanIP, Lookup, Wait, RandomRejection
	'uses globals: g_sAdminPassword

	Dim tempString, strRegEx
	If CheckIPRange(oClient.IPAddress) = -1 Then 
		EventLog.Write("Details for OnHELO - IP Address = " & oClient.IPAddress & "  Helo sent was '" & CStr(oClient.helo) & "'")
	End If
	If oClient.helo = "ylmf-pc"  											_
	  Or oClient.helo = "User" 												
	  Then
 		TempString = "    Helo = '" & oClient.helo & "' was Autobanned for using known spammer EHLO"
		Call CustomMonthlyLog(TempString, "Port" & oClient.port)
		Call AutobanIP(oClient.ipaddress, 14, oClient.helo & " as EHLO")
		Result.Message = RandomRejection()
		EventLog.Write("Random Rejection sent to IP - " & oClient.IPAddress & "  Rejection Message was '" & Result.Message & "'")
    	Wait(14)
		Result.Value = 2
'	Else
'		Add to Greylist Whitelist if from major email sender
'		strRegEx = "^[a-z]+[0-9]{2}(-)[a-z]{2}[0-9](-obe\.outbound\.protection\.outlook\.com)$|"     &_
'		           "^(a)[0-9]{1,2}(-)[0-9]{1,3}(\.smtp-out\.amazonses\.com)$|"                       &_
'		           "^(mail-)[a-z]{2}[0-9](-f)[0-9]{1,3}(\.google\.com)$|"                            &_
'		           "^(spring-chicken-)[a-z]{2}(\.twitter\.com)$|"                                    &_
'		           "^(mail)[a-z](-)[a-z]{2}(\.linkedin\.com)$|"                                      &_
'		           "^(mx)[0-9](\.)[a-z]{3}(\.paypal\.com)$|"                                         &_
'		           "^(cyberoam)"                                         							 &_
'		           "^(mx-out\.facebook\.com)$"                                                      
'		If Lookup(strRegEx, oClient.HELO)= True Then 
'			dim oApp, NewWhitelistEntry, oWhiteList, i, checker
'			Set oApp = CreateObject("hMailServer.Application")
		
			' Give this script permission to access all
			' hMailServer settings.
'			Call oApp.Authenticate("Administrator", g_sAdminPassword)
'			Set oWhiteList = oApp.Settings.AntiSpam.GreyListingWhiteAddresses
	
'			checker = 0
'			For i = 1 To oWhiteList.count
'				If oWhiteList.item(i-1).ipAddress = oclient.IPAddress Then
'					checker = checker +1
'					Exit For
'				End If
'			Next 'i	
'			This bit below adds to the Greylist >> Whitelist
'			Probably need to add greylist triplet instead
'
'			If checker = 0 Then
'				Set NewWhitelistEntry = oWhiteList.Add
'				With NewWhitelistEntry
'					.IPAddress = oClient.IPAddress
'					.Description = "Auto Added - HELO = '" & oClient.helo & "'"
'					.Save
'				End With
'				TempString = "added '" & oClient.helo & "' to Whitelist"
'				Call CustomMonthlyLog(TempString, "Port" & oClient.port)
'			End If 'checker
'		End If
	End If    
End Sub
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-12 07:13

Thank you! Still your solution involves some things that I would avoid. Such as delays, or relying on large providers (in my case, I need to receive messages from university departments with their own servers).

But back to my request: greylisting with a mask, like /24. Is there a way to implement it with a script? Or can you (or who?) make it in the code? Or is there a reason not to do it?

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Greylisting by subnetwork

Post by mattg » 2018-09-12 10:17

gelbukh wrote:
2018-09-12 07:13
Or can you (or who?) make it in the code? Or is there a reason not to do it?
hMailserver is open source on Github (as you've found), feel free to make your own port

Is there a reason not to do this?
Yes, absolutely.

This is one of the sillies ideas that I've seen I reckon.

Consider two people with gMail addresses send you an email, first one gets bounced, second and subsequent once never get bounced. Why greylist at all??


And yes I know that I code in delays, and all sorts of other impediments - that's the point. Real mail will tolerate such things, Spammers won't.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-09-12 11:13

gelbukh wrote:
2018-09-12 02:00
Or, can I implement this my idea via a script? Such as in some event handler temporarily changing the incoming IP before greylisting, and then restoring the original IP? I did not find it.
If you look at my script near the top of this thread, you will see that I maintain the "GreyList Whitelist" by examining the HELO/EHLO message. If Sending server identifies itself in the proper manner the script does not care which IP is used or if these servers are located on each their own continent.
The Windows scheduler periodically run a "clean-up" script to keep things at a manageable level.

"Sub OnHELO(oClient)" is right before hMailServer receive recipient address(es) and does it's GreyListing thing, so this way the script can function in a "pro-active" way.

Sender is validated by RegEx to match the EHLO string like "^(mail-)[a-z]{2}[0-9](-f)[0-9]{1,3}(\.google\.com)$" and then hMailServer do not really care if it's IPv4, IPv6, earth-based or extraterrestrial.

BTW... The Wait and LockFile functions are there since VBScript do not handle record locking at all!!

Code: Select all

Option Explicit

'  ******************************************************************************************************************************
'  ********** Settings                                                                                                 **********
'  ******************************************************************************************************************************

   ' COM authentication
   Private Const ADMIN = "Administrator"
   Private Const PASSWORD = "You'd like that, eh?"

'  ******************************************************************************************************************************
'  ********** Functions                                                                                                **********
'  ******************************************************************************************************************************

   Function Wait(sec)
      With CreateObject("WScript.Shell")
         .Run "timeout /T " & Int(sec), 0, True
'        .Run "sleep -m " & Int(sec * 1000), 0, True
'        .Run "powershell Start-Sleep -Milliseconds " & Int(sec * 1000), 0, True
      End With
   End Function

   Function LockFile(strPath)
      Const Append = 8
      Const Unicode = -1
      With CreateObject("Scripting.FileSystemObject")
         Dim oFile, i
         For i = 0 To 30
            On Error Resume Next
            Set oFile = .OpenTextFile(strPath, Append, True, Unicode)
            If Not (Err.Number = 70) Then
               Set LockFile = oFile
               On Error Goto 0
               Exit For
            End If
            On Error Goto 0
            Wait(1)
         Next
      End With
      Set oFile = Nothing
      If (Err.Number = 70) Then
         EventLog.Write("ERROR: EventHandlers.vbs")
         EventLog.Write("File " & strPath & " is locked and timeout was exceeded.")
         Err.Clear
      ElseIf (Err.Number <> 0) Then
         EventLog.Write("ERROR: EventHandlers.vbs : Function LockFile")
         EventLog.Write("Error       : " & Err.Number)
         EventLog.Write("Error (hex) : 0x" & Hex(Err.Number))
         EventLog.Write("Source      : " & Err.Source)
         EventLog.Write("Description : " & Err.Description)
         Err.Clear
      End If
   End Function

   Function Lookup(strRegEx, strMatch) : Lookup = False
      With CreateObject("VBScript.RegExp")
         .Global = False
         .Pattern = strRegEx
         .IgnoreCase = True
         If .Test(strMatch) Then Lookup = True
      End With
   End Function

'  ******************************************************************************************************************************
'  ********** Subroutines                                                                                              **********
'  ******************************************************************************************************************************

   Sub GreyWhiteList(strIP, strDesc)
      Dim oApp : Set oApp = CreateObject("hMailServer.Application")
      Call oApp.Authenticate(ADMIN, PASSWORD)
      With LockFile("c:\hmailserver\temp\greywhitelist.lck")
         On Error Resume Next
         oApp.Settings.AntiSpam.GreyListingWhiteAddresses.Refresh
         If (oApp.Settings.AntiSpam.GreyListingWhiteAddresses.ItemByName(strIP) Is Nothing) Then
            With oApp.Settings.AntiSpam.GreyListingWhiteAddresses.Add
               .Description = "# " & Date & " " & strDesc
               .IPAddress = strIP
               .Save
            End With
         Else
            With oApp.Settings.AntiSpam.GreyListingWhiteAddresses.ItemByName(strIP)
               .Description = "# " & Date & " " & strDesc
               .Save
            End With
         End If
         On Error Goto 0
         .Close
      End With
   End Sub

   Sub ExpireGreyList
      Dim j, m_Days : m_Days = 30
      Dim oApp : Set oApp = CreateObject("hMailServer.Application")
      Call oApp.Authenticate(ADMIN, PASSWORD)
      For j = oApp.Settings.AntiSpam.GreyListingWhiteAddresses.Count-1 To 0 Step -1
         If (Left(oApp.Settings.AntiSpam.GreyListingWhiteAddresses(j).Description, 1) = "#") Then
            If (DateDiff("d", Mid(oApp.Settings.AntiSpam.GreyListingWhiteAddresses(j).Description,3,10), Date) > m_Days) Then
               oApp.Settings.AntiSpam.GreyListingWhiteAddresses(j).Delete
            End If
         End If
      Next
   End Sub

'  ******************************************************************************************************************************
'  ********** hMailServer Triggers                                                                                     **********
'  ******************************************************************************************************************************

   Sub OnHELO(oClient)
      Dim strRegEx

      '
      ' Dynamic GreyList Whitelist
      '
      strRegEx = "^[a-z]+[0-9]{2}(-)[a-z]{2}[0-9](-obe\.outbound\.protection\.outlook\.com)$|" &_
                 "^(((mail134-)[0-9]{1,3})|((out22-)[0-9]{1,2}))(\.mail\.alibaba\.com)$|" &_
                 "^(a)[0-9]{1,2}(-)[0-9]{1,3}(\.smtp-out\.amazonses\.com)$|" &_
                 "^(mail-)[a-z]{2}[0-9](-f)[0-9]{1,3}(\.google\.com)$|" &_
                 "^(spring-chicken-)[a-z]{2}(\.twitter\.com)$|" &_
                 "^(mail)[a-z](-)[a-z]{2}(\.linkedin\.com)$|" &_
                 "^(mx)[0-9](\.)[a-z]{3}(\.paypal\.com)$"

      If Lookup(strRegEx, oClient.HELO) Then
         Call GreyWhiteList(oClient.IPAddress, oClient.HELO)
      End If

   End Sub
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Greylisting by subnetwork

Post by RvdH » 2018-09-12 11:25

SorenR wrote:
2018-09-12 11:13
gelbukh wrote:
2018-09-12 02:00
Or, can I implement this my idea via a script? Such as in some event handler temporarily changing the incoming IP before greylisting, and then restoring the original IP? I did not find it.
If you look at my script near the top of this thread, you will see that I maintain the "GreyList Whitelist" by examining the HELO/EHLO message. If Sending server identifies itself in the proper manner the script does not care which IP is used or if these servers are located on each their own continent.
The Windows scheduler periodically run a "clean-up" script to keep things at a manageable level.

"Sub OnHELO(oClient)" is right before hMailServer receive recipient address(es) and does it's GreyListing thing, so this way the script can function in a "pro-active" way.

Sender is validated by RegEx to match the EHLO string like "^(mail-)[a-z]{2}[0-9](-f)[0-9]{1,3}(\.google\.com)$" and then hMailServer do not really care if it's IPv4, IPv6, earth-based or extraterrestrial.
Unfortunately these HELO/EHLO messages can be faked as well. So i got bit futher and enhanced SorenR's script to validate the ipaddress against know SPF ranges

So with the EHLO string like "^(mail-)[a-z]{2}[0-9](-f)[0-9]{1,3}(\.google\.com)$

I validate it against the SPF ip range for google.com

Code: Select all

Sub AddGreyList(ByVal strIP, ByVal strHELO)
  
	Dim oRegEx
	Set oRegEx = CreateObject("VBScript.RegExp")
	oRegEx.IgnoreCase = True
	oRegEx.Global = False
	
	Select Case getDomainName(strHELO)
	
	case "google.com"	
		oRegEx.Pattern= "^64\.18\.([0-9]|1[0-5])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^64\.233\.(1([6-8][0-9]|9[0-1]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^66\.102\.([0-9]|1[0-5])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^66\.249\.(8[0-9]|9[0-5])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^72\.14\.(1(9[2-9])|2([0-4][0-9]|5[0-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^74\.125\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^108\.177\.([8-9]|1[0-5])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^173\.194\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^207\.126\.(1(4[4-9]|5[0-9]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^209\.85\.(1(2[8-9]|[3-9][0-9])|2([0-4][0-9]|5[0-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^216\.58\.(1(9[2-9])|2([0-1][0-9]|2[0-3]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^216\.239\.(3[2-9]|[4-5][0-9]|6[0-3])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^172\.217\.([0-9]|[1-2][0-9]|3[0-1])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^108\.177\.(9[6-9]|1([0-1][0-9]|2[0-7]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$"
	
	....
	' other ip ranges
	....
	
	Case Else
		oRegEx.Pattern= "^(?:[0-9]{1,3}\.){3}[0-9]{1,3}$"
	End Select

	If oRegEx.Test(strIP) Then
		Dim oApp
		Set oApp = CreateObject("hMailServer.Application")
		Call oApp.Authenticate("Administrator", "YOURPASSWORD")
		With LockFile("E:\Email\greylistwhite.lck")
			On Error Resume Next
			oApp.Settings.AntiSpam.GreyListingWhiteAddresses.Refresh
			If oApp.Settings.AntiSpam.GreyListingWhiteAddresses.ItemByName(strIP) Is Nothing Then
				' REM Logging
				' REM Call LogHELO(strIP, strHELO, "E:\helo.log")
				With oApp.Settings.AntiSpam.GreyListingWhiteAddresses.Add
					.Description = Date & " Auto-Added '" & strHELO & "'"
					.IPAddress = strIP
					.Save
				End With
			Else
				With oApp.Settings.AntiSpam.GreyListingWhiteAddresses.ItemByName(strIP)
					.Description = Date & " Auto-Added '" & strHELO & "'"
					.Save
				End With
			End If
			oApp.Settings.AntiSpam.GreyListingWhiteAddresses.Refresh
			On Error Goto 0
			.Close '// Close LockFile
		End With 
		Set oApp = Nothing
	Else
		' REM Logging
		Call LogHELO(strIP, strHELO, "E:\helo_error.log")
	End If
	Set oRegEx = Nothing
End Sub


Sub LogHELO(ipaddress, helo, path)
	Dim objFSO
	Dim objTextStream
	Set objFSO = CreateObject("Scripting.FileSystemObject")
	If not objFSO.FileExists(path) Then
		'File not found.
		call objFSO.CreateTextFile(path, True) 
	end if	
	'OK to access the file.
	'Open the text file
	Set objTextStream = objFSO.OpenTextFile(path, 8)
	'Display the contents of the text file
	objTextStream.WriteLine "HELO: " & ipaddress & " From: " & helo

	'Close the file and clean up
	objTextStream.Close
	Set objTextStream = Nothing
	Set objFSO = Nothing
End Sub

Function getDomainName(strHELO)
	dim aryDomain, str2ndLevel, strTopLevel
	getDomainName = Null
	If Len(strHELO) > 0 Then  	
		aryDomain = Split(strHELO,".")
		If uBound(aryDomain) >= 1 Then
			str2ndLevel = aryDomain(uBound(aryDomain)-1)
			strTopLevel = aryDomain(uBound(aryDomain))			
			getDomainName = str2ndLevel & "." & strTopLevel
		End If
	End If
End Function
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-09-12 15:28

RvdH wrote:
2018-09-12 11:25

Unfortunately these HELO/EHLO messages can be faked as well. So i got bit futher and enhanced SorenR's script to validate the ipaddress against know SPF ranges

So with the EHLO string like "^(mail-)[a-z]{2}[0-9](-f)[0-9]{1,3}(\.google\.com)$

I validate it against the SPF ip range for google.com

Code: Select all

Sub AddGreyList(ByVal strIP, ByVal strHELO)
  
	Dim oRegEx
	Set oRegEx = CreateObject("VBScript.RegExp")
	oRegEx.IgnoreCase = True
	oRegEx.Global = False
	
	Select Case getDomainName(strHELO)
	
	case "google.com"	
		oRegEx.Pattern= "^64\.18\.([0-9]|1[0-5])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^64\.233\.(1([6-8][0-9]|9[0-1]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^66\.102\.([0-9]|1[0-5])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^66\.249\.(8[0-9]|9[0-5])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^72\.14\.(1(9[2-9])|2([0-4][0-9]|5[0-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^74\.125\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^108\.177\.([8-9]|1[0-5])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^173\.194\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^207\.126\.(1(4[4-9]|5[0-9]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^209\.85\.(1(2[8-9]|[3-9][0-9])|2([0-4][0-9]|5[0-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^216\.58\.(1(9[2-9])|2([0-1][0-9]|2[0-3]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^216\.239\.(3[2-9]|[4-5][0-9]|6[0-3])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^172\.217\.([0-9]|[1-2][0-9]|3[0-1])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
						"^108\.177\.(9[6-9]|1([0-1][0-9]|2[0-7]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$"
	
	....
	' other ip ranges
	....
	
	Case Else
		oRegEx.Pattern= "^(?:[0-9]{1,3}\.){3}[0-9]{1,3}$"
	End Select

	If oRegEx.Test(strIP) Then
		Dim oApp
		Set oApp = CreateObject("hMailServer.Application")
		Call oApp.Authenticate("Administrator", "YOURPASSWORD")
		With LockFile("E:\Email\greylistwhite.lck")
			On Error Resume Next
			oApp.Settings.AntiSpam.GreyListingWhiteAddresses.Refresh
			If oApp.Settings.AntiSpam.GreyListingWhiteAddresses.ItemByName(strIP) Is Nothing Then
				' REM Logging
				' REM Call LogHELO(strIP, strHELO, "E:\helo.log")
				With oApp.Settings.AntiSpam.GreyListingWhiteAddresses.Add
					.Description = Date & " Auto-Added '" & strHELO & "'"
					.IPAddress = strIP
					.Save
				End With
			Else
				With oApp.Settings.AntiSpam.GreyListingWhiteAddresses.ItemByName(strIP)
					.Description = Date & " Auto-Added '" & strHELO & "'"
					.Save
				End With
			End If
			oApp.Settings.AntiSpam.GreyListingWhiteAddresses.Refresh
			On Error Goto 0
			.Close '// Close LockFile
		End With 
		Set oApp = Nothing
	Else
		' REM Logging
		Call LogHELO(strIP, strHELO, "E:\helo_error.log")
	End If
	Set oRegEx = Nothing
End Sub


Sub LogHELO(ipaddress, helo, path)
	Dim objFSO
	Dim objTextStream
	Set objFSO = CreateObject("Scripting.FileSystemObject")
	If not objFSO.FileExists(path) Then
		'File not found.
		call objFSO.CreateTextFile(path, True) 
	end if	
	'OK to access the file.
	'Open the text file
	Set objTextStream = objFSO.OpenTextFile(path, 8)
	'Display the contents of the text file
	objTextStream.WriteLine "HELO: " & ipaddress & " From: " & helo

	'Close the file and clean up
	objTextStream.Close
	Set objTextStream = Nothing
	Set objFSO = Nothing
End Sub

Function getDomainName(strHELO)
	dim aryDomain, str2ndLevel, strTopLevel
	getDomainName = Null
	If Len(strHELO) > 0 Then  	
		aryDomain = Split(strHELO,".")
		If uBound(aryDomain) >= 1 Then
			str2ndLevel = aryDomain(uBound(aryDomain)-1)
			strTopLevel = aryDomain(uBound(aryDomain))			
			getDomainName = str2ndLevel & "." & strTopLevel
		End If
	End If
End Function
Oh boy... How much time do you spend maintaining that monstrum ??

I get a headache just looking at it :wink:
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Greylisting by subnetwork

Post by RvdH » 2018-09-12 15:34

Not that much...the spf range for google is unchanged for more then a year now, outlook.com range on the other hand :shock: :twisted:

hint: d-fault.nl/CIDRtoRegEx
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-09-12 20:54

I was wondering... The HELP text for the topic Whitelist in Greylisting indicate that you can use wildcards...

Anyone tried it? Like "192.168.0.*" or how?
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-12 22:42

Consider two people with gMail addresses send you an email, first one gets bounced, second and subsequent once never get bounced. Why greylist at all??
This would be exactly the desired behavior: gMail addresses should NOT be bounced.

However, since greylisting works by triples, acceptance of a message requires coincidence of THREE things: IP (subnet in my proposal), from, and to.

For a legitimate farm of round-robin servers, this will pass the message normally from the second attempt, which is exactly the desired behavior.

For a spammer, this will block the message, even if he has several computers infected on the same subnet (because the triples will be always different).
delays, and all sorts of other impediments - that's the point. Real mail will tolerate such things
No, a legitimate server does not have to respect unrealistic delays (especially not well-configured, but legitimate one). What it does respect is faults. This is why greylisting was invented, and not just delays.
EHLO string like "^(mail-)[a-z]{2}[0-9](-f)[0-9]{1,3}(\.google\.com)$
This solves the problem only of google, but not of other legitimate servers, such as the University of Chilpansingo. Unless you maintain a regex of all legitimate servers in the world -- then, indeed, why greylist at all??

Plus, this relies on that spammers don't happen to use "google" in EHLO. One day they will.
feel free to make your own port
I am not good at github. More importantly, I want this to be done for all, not only for me.
This is one of the sillies ideas that I've seen I reckon.
Special thanks for encouragement. But I am still to see a valid counterargument (other than to regex on EHLO, which, I admit, is also an exceptional idea).

User avatar
jimimaseye
Moderator
Moderator
Posts: 10053
Joined: 2011-09-08 17:48

Re: Greylisting by subnetwork

Post by jimimaseye » 2018-09-12 23:02

mattg wrote:
2018-09-12 10:17
This is one of the sillies ideas that I've seen I reckon.
+1

What smtp service will occupy 255 different ip address in the same subnet? Most will be a handful which will attempt delivery from the same server address. Bigger hosts (gmail, office365 etc), IF using round robin when greylisted, use 1000s of addresses and likely the delivery attempts will be on different ranges if it's not the same address. What is being asked for here is something to handle a very specific and rare event of an smtp 'server' using round robin to attempt redelivery following greylisting on a different address when there is only a handful of them in the first place. As rare as they are let them handle greylisting and its defined established functionally as it is. After all its not like greylisting is a mystery. Is it.

gelbukh wrote:
2018-09-12 22:42
No, a legitimate server does not have to respect unrealistic delays. What it does respect is faults. This is why greylisting was invented, and not just delays.
No but it does have to respect realistic ones. RFC 2821 informs servers are to allow a lot longer before timing out. As call backs must allow 30 seconds before timing out, 20 seconds is nothing. All serious respecting servers will wait 20 seconds for the response on connection. If they do not then they are ill configured and non RFC confirming.

[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-13 00:36

I apologize for being insistent.
a very specific and rare event of an smtp 'server' using round robin to attempt redelivery following greylisting on a different address when there is only a handful of them
Unfortunately, my logs show that both major services (gmail, outlook, etc.) and smaller university servers do this all the time.

Here is just today's example (user name hidden, but it is the same user):

Code: Select all

"SMTPD" 3128    36364   "2018-09-12 04:56:43.216"   "209.85.167.46" "RECEIVED: EHLO mail-lf1-f46.google.com"
"SMTPD" 3152    36364   "2018-09-12 04:56:43.388"   "209.85.167.46" "RECEIVED: MAIL FROM:<some-user@cs.biu.ac.il> SIZE=30303"
"SMTPD" 3128    36364   "2018-09-12 04:56:45.868"   "209.85.167.46" "SENT: 451 Please try again later."
"SMTPD" 3188    36375   "2018-09-12 05:02:41.034"   "209.85.167.48" "RECEIVED: EHLO mail-lf1-f48.google.com"
"SMTPD" 3192    36375   "2018-09-12 05:02:41.206"   "209.85.167.48" "RECEIVED: MAIL FROM:<some-user@cs.biu.ac.il> SIZE=30303"
"SMTPD" 3188    36375   "2018-09-12 05:02:42.844"   "209.85.167.48" "SENT: 451 Please try again later."
"SMTPD" 3136    36385   "2018-09-12 05:20:52.115"   "209.85.167.42" "RECEIVED: EHLO mail-lf1-f42.google.com"
"SMTPD" 3152    36385   "2018-09-12 05:20:52.287"   "209.85.167.42" "RECEIVED: MAIL FROM:<some-user@cs.biu.ac.il> SIZE=30303"
"SMTPD" 3188    36385   "2018-09-12 05:20:54.065"   "209.85.167.42" "SENT: 451 Please try again later."
"SMTPD" 3192    36418   "2018-09-12 05:58:18.145"   "209.85.167.47" "RECEIVED: EHLO mail-lf1-f47.google.com"
"SMTPD" 3172    36418   "2018-09-12 05:58:18.317"   "209.85.167.47" "RECEIVED: MAIL FROM:<some-user@cs.biu.ac.il> SIZE=30302"
"SMTPD" 3172    36418   "2018-09-12 05:58:20.126"   "209.85.167.47" "SENT: 451 Please try again later."
"SMTPD" 3140    36450   "2018-09-12 06:43:20.569"   "209.85.167.53" "RECEIVED: EHLO mail-lf1-f53.google.com"
"SMTPD" 3172    36450   "2018-09-12 06:43:20.756"   "209.85.167.53" "RECEIVED: MAIL FROM:<some-user@cs.biu.ac.il> SIZE=30303"
"SMTPD" 3188    36450   "2018-09-12 06:43:22.394"   "209.85.167.53" "SENT: 451 Please try again later."
"SMTPD" 3140    36490   "2018-09-12 07:46:49.721"   "209.85.167.46" "RECEIVED: EHLO mail-lf1-f46.google.com"
"SMTPD" 3188    36490   "2018-09-12 07:46:50.548"   "209.85.167.46" "RECEIVED: MAIL FROM:<some-user@cs.biu.ac.il> SIZE=30303"
"SMTPD" 3136    36490   "2018-09-12 07:47:00.969"   "209.85.167.46" "SENT: 250 OK"
In this example, it took 3 hours for this message to pass. Sometimes it takes days, or gmail just gives up.

The message was real, and a very important one.

And this is gmail. Not very rare 'server', is it?

With my proposal, this message would have passed in 5 minutes, at the second attempt. Instead of 3 hours (and I was lucky).
As rare as they are let them handle greylisting and its defined established functionally as it is.
I wish you were right. Nope. They don't care.
Bigger hosts (gmail, office365 etc), IF using round robin when greylisted, use 1000s of addresses and likely the delivery attempts will be on different ranges
OK, with 10,000 addresses, after some 20 attempts on average (40 in the worst case), the /24 subnet will repeat and -- with my proposal -- the message will pass in some 20 hours = 1 day (1 hour / attempt, as my log shows). Without my proposal, it will take some 5,000 hours = half a year = never. Even in this extreme case, a 20-hour delay (my proposal) is better than not getting the message at all.

And please don't say that you can regex for "goodle". If Google can do it, then others will do it, too.
RFC 2821 informs servers are to allow a lot longer before timing out.
OK, this is probably a good argument. But greylisting is a great idea, and it does not rely on whether the server is compliant.

BTW, you see that Google (and others) don't care about its users when it comes to money. Some day they will not care about this your RFC either. Greylisting is a great idea and is very reliable, and we need to modify only two lines of code to bring it to life again.

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-09-13 02:44

gelbukh wrote:
2018-09-13 00:36
Here is just today's example (user name hidden, but it is the same user):

Code: Select all

"SMTPD" 3128    36364   "2018-09-12 04:56:43.216"   "209.85.167.46" "RECEIVED: EHLO mail-lf1-f46.google.com"
"SMTPD" 3152    36364   "2018-09-12 04:56:43.388"   "209.85.167.46" "RECEIVED: MAIL FROM:<some-user@cs.biu.ac.il> SIZE=30303"
"SMTPD" 3128    36364   "2018-09-12 04:56:45.868"   "209.85.167.46" "SENT: 451 Please try again later."
"SMTPD" 3188    36375   "2018-09-12 05:02:41.034"   "209.85.167.48" "RECEIVED: EHLO mail-lf1-f48.google.com"
"SMTPD" 3192    36375   "2018-09-12 05:02:41.206"   "209.85.167.48" "RECEIVED: MAIL FROM:<some-user@cs.biu.ac.il> SIZE=30303"
"SMTPD" 3188    36375   "2018-09-12 05:02:42.844"   "209.85.167.48" "SENT: 451 Please try again later."
"SMTPD" 3136    36385   "2018-09-12 05:20:52.115"   "209.85.167.42" "RECEIVED: EHLO mail-lf1-f42.google.com"
"SMTPD" 3152    36385   "2018-09-12 05:20:52.287"   "209.85.167.42" "RECEIVED: MAIL FROM:<some-user@cs.biu.ac.il> SIZE=30303"
"SMTPD" 3188    36385   "2018-09-12 05:20:54.065"   "209.85.167.42" "SENT: 451 Please try again later."
"SMTPD" 3192    36418   "2018-09-12 05:58:18.145"   "209.85.167.47" "RECEIVED: EHLO mail-lf1-f47.google.com"
"SMTPD" 3172    36418   "2018-09-12 05:58:18.317"   "209.85.167.47" "RECEIVED: MAIL FROM:<some-user@cs.biu.ac.il> SIZE=30302"
"SMTPD" 3172    36418   "2018-09-12 05:58:20.126"   "209.85.167.47" "SENT: 451 Please try again later."
"SMTPD" 3140    36450   "2018-09-12 06:43:20.569"   "209.85.167.53" "RECEIVED: EHLO mail-lf1-f53.google.com"
"SMTPD" 3172    36450   "2018-09-12 06:43:20.756"   "209.85.167.53" "RECEIVED: MAIL FROM:<some-user@cs.biu.ac.il> SIZE=30303"
"SMTPD" 3188    36450   "2018-09-12 06:43:22.394"   "209.85.167.53" "SENT: 451 Please try again later."
"SMTPD" 3140    36490   "2018-09-12 07:46:49.721"   "209.85.167.46" "RECEIVED: EHLO mail-lf1-f46.google.com"
"SMTPD" 3188    36490   "2018-09-12 07:46:50.548"   "209.85.167.46" "RECEIVED: MAIL FROM:<some-user@cs.biu.ac.il> SIZE=30303"
"SMTPD" 3136    36490   "2018-09-12 07:47:00.969"   "209.85.167.46" "SENT: 250 OK"
In this example, it took 3 hours for this message to pass. Sometimes it takes days, or gmail just gives up.
Well, you just delivered proof that Greylisting sucks :mrgreen:

This is one of the reasons why I abandoned Greylisting - I simply got tired of having this whining noise in my ear saying "I'm trying to reset my password on XYZ and the email has not come through yet" or "Joe User just sent me a list on email 10 minutes ago but it's not in my inbox".

Greylisting once WAS good stuff... Before Social media and 1,000 CPU MailServer Farms sending mails on behalf of half the world about stuff that noone really cares about but they have to read it to be in with the right crowd...

I'm getting to the age where I don't give a sh*te. I know how to program, I can read math formulas and have hapily forgotten all about Laplace transformations and Fourier analysis, I can sew a zipper in a pair of pants, heel my shoes, Cook a decent Thai red curry with ข้าวผัดไข่, change the timing belt in my car, plaster my walls, sand the woodwork or use the lathe, do a 1/4 mile in my 1982 Audi Quattro and plow the fields with my 1952 Ferguson TEA-20. Spending 6 hours on a 16HP Lawn mover once a week gives you ample time to reflect on some of life's great adventures - like Greylisting ...

And remember, to the chemist, alchohol is not the problem, it is a solution...
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Greylisting by subnetwork

Post by mattg » 2018-09-13 03:32

SorenR wrote:
2018-09-13 02:44
Well, you just delivered proof that Greylisting sucks :mrgreen:

This is one of the reasons why I abandoned Greylisting
Me too.
gelbukh wrote:
2018-09-12 22:42
Consider two people with gMail addresses send you an email, first one gets bounced, second and subsequent once never get bounced. Why greylist at all??
This would be exactly the desired behavior: gMail addresses should NOT be bounced.
A gmail address was a poor example on my behalf.

If an ISP has their own servers on say 4 IP addresses, and own two thirds of a /24 IP range block, and all other of their own IPs are provided to users, then once you allow the /24 block through grey listing, then almost anybody could be the next to send from that /24 block.

There is a setting to bypass greylisting on SPF pass, the issue with that is two fold.
#1. Hmailserver passes SPF records that end in +all, or ~all for all IPs
#2. Most spammers send from an address that has a valid spf record
SorenR wrote:
2018-09-12 20:54
I was wondering... The HELP text for the topic Whitelist in Greylisting indicate that you can use wildcards...

Anyone tried it? Like "192.168.0.*" or how?
Pretty sure I have tried that, also FQDNs like spf.gmail.com which would in itself solve this issue fairly easily (If Office365 also used a similar method to show their sending server addresses.

I also find that Sub OnHelo doesn't always return a HELO/EHLO, It seems to when there is StartTLS involved on my system for incoming mail. ****EDIT This should say OnSMTPData, not OnHelo ****
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Greylisting by subnetwork

Post by RvdH » 2018-09-13 11:05

mattg wrote:
2018-09-13 03:32
I also find that Sub OnHelo doesn't always return a HELO/EHLO, It seems to when there is StartTLS involved on my system for incoming mail.
That is odd, you should get receive EHLO/HELO response before and after the once the STARTTLS command is received. EHLO/HELO is requested/send only for SMTPD/SMTPC
"TCPIP" 3400 "2018-09-13 10:58:06.289" "TCP - x.x.x.x connected to x.x.x.x:25."
"SMTPD" 3400 3508 "2018-09-13 10:58:06.304" "x.x.x.x" "SENT: 220 mail.server.com ESMTP"
"SMTPD" 824 3508 "2018-09-13 10:58:06.320" "x.x.x.x" "RECEIVED: EHLO something.com"
"SMTPD" 824 3508 "2018-09-13 10:58:06.351" "x.x.x.x" "SENT: 250-mail.server.com[nl]250-SIZE 41943040[nl]250-STARTTLS[nl]250-AUTH LOGIN[nl]250 HELP"
"SMTPD" 824 3508 "2018-09-13 10:58:06.367" "x.x.x.x" "RECEIVED: STARTTLS"
"SMTPD" 824 3508 "2018-09-13 10:58:06.367" "x.x.x.x" "SENT: 220 Ready to start TLS"
"TCPIP" 3416 "2018-09-13 10:58:06.398" "TCPConnection - TLS/SSL handshake completed. Session Id: 3508, Remote IP: x.x.x.x, Version: TLSv1.2, Cipher: ECDHE-RSA-AES256-GCM-SHA384, Bits: 256"
"SMTPD" 3416 3508 "2018-09-13 10:58:06.413" "x.x.x.x" "RECEIVED: EHLO something.com"
"SMTPD" 3416 3508 "2018-09-13 10:58:06.445" "x.x.x.x" "SENT: 250-mail.server.com[nl]250-SIZE 41943040[nl]250-AUTH LOGIN[nl]250 HELP"
"SMTPD" 3416 3508 "2018-09-13 10:58:06.460" "x.x.x.x" "RECEIVED: MAIL FROM:<***@***.**>"
"SMTPD" 3416 3508 "2018-09-13 10:58:06.523" "x.x.x.x" "SENT: 250 OK"
"SMTPD" 3416 3508 "2018-09-13 10:58:06.523" "x.x.x.x" "RECEIVED: RCPT TO:<***@***.**>"
"SMTPD" 3416 3508 "2018-09-13 10:58:06.538" "x.x.x.x" "SENT: 250 OK"
"SMTPD" 2320 3508 "2018-09-13 10:58:06.554" "x.x.x.x" "RECEIVED: DATA"
"SMTPD" 2320 3508 "2018-09-13 10:58:06.554" "x.x.x.x" "SENT: 354 OK, send."
"TCPIP" 3980 "2018-09-13 10:58:06.647" "Connecting to 127.0.0.1:783..."
"SMTPD" 3980 3508 "2018-09-13 10:58:10.516" "x.x.x.x" "SENT: 250 Queued (3.900 seconds)"
"SMTPD" 1284 3508 "2018-09-13 10:58:10.516" "x.x.x.x" "RECEIVED: QUIT"
"SMTPD" 1284 3508 "2018-09-13 10:58:10.516" "x.x.x.x" "SENT: 221 goodbye"
With debugging
"TCPIP" 3008 "2018-09-13 11:09:31.968" "TCP - x.x.x.x connected to x.x.x.x:25."
"DEBUG" 3008 "2018-09-13 11:09:31.968" "Executing event OnClientConnect"
"DEBUG" 3008 "2018-09-13 11:09:31.999" "Event completed"

"DEBUG" 3008 "2018-09-13 11:09:31.999" "TCP connection started for session 3714"
"SMTPD" 3008 3714 "2018-09-13 11:09:31.999" "x.x.x.x" "SENT: 220 mail.server.com ESMTP"
"SMTPD" 3008 3714 "2018-09-13 11:09:32.014" "x.x.x.x" "RECEIVED: EHLO something.com"
"DEBUG" 3008 "2018-09-13 11:09:32.014" "Executing event OnHELO"
"DEBUG" 3008 "2018-09-13 11:09:32.108" "Event completed"

"SMTPD" 3008 3714 "2018-09-13 11:09:32.108" "x.x.x.x" "SENT: 250-mail.server.com[nl]250-SIZE 41943040[nl]250-STARTTLS[nl]250-AUTH LOGIN[nl]250 HELP"
"SMTPD" 3400 3714 "2018-09-13 11:09:32.108" "x.x.x.x" "RECEIVED: STARTTLS"
"SMTPD" 3400 3714 "2018-09-13 11:09:32.124" "x.x.x.x" "SENT: 220 Ready to start TLS"
"DEBUG" 3008 "2018-09-13 11:09:32.124" "Performing SSL/TLS handshake for session 3714. Verify certificate: False"
"TCPIP" 3008 "2018-09-13 11:09:32.342" "TCPConnection - TLS/SSL handshake completed. Session Id: 3714, Remote IP: x.x.x.x, Version: TLSv1.2, Cipher: ECDHE-RSA-AES256-GCM-SHA384, Bits: 256"
"SMTPD" 3008 3714 "2018-09-13 11:09:32.342" "x.x.x.x" "RECEIVED: EHLO something.com"
"DEBUG" 3008 "2018-09-13 11:09:32.342" "Executing event OnHELO"
"DEBUG" 3008 "2018-09-13 11:09:32.576" "Event completed"

"SMTPD" 3008 3714 "2018-09-13 11:09:32.576" "x.x.x.x" "SENT: 250-mail.server.com[nl]250-SIZE 41943040[nl]250-AUTH LOGIN[nl]250 HELP"
"SMTPD" 2416 3714 "2018-09-13 11:09:32.716" "x.x.x.x" "RECEIVED: MAIL FROM:<***@***.**> SIZE=88575"
"SMTPD" 2416 3714 "2018-09-13 11:09:32.763" "x.x.x.x" "SENT: 250 OK"
"SMTPD" 3400 3714 "2018-09-13 11:09:32.810" "x.x.x.x" "RECEIVED: RCPT TO:<***@***.**>"
"SMTPD" 3400 3714 "2018-09-13 11:09:32.810" "x.x.x.x" "SENT: 250 OK"
"SMTPD" 2416 3714 "2018-09-13 11:09:32.826" "x.x.x.x" "RECEIVED: DATA"
"DEBUG" 2416 "2018-09-13 11:09:32.826" "Executing event OnSMTPData"
"DEBUG" 2416 "2018-09-13 11:09:32.904" "Event completed"

"SMTPD" 2416 3714 "2018-09-13 11:09:32.904" "x.x.x.x" "SENT: 354 OK, send."
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Greylisting by subnetwork

Post by mattg » 2018-09-13 13:50

Yes I get that in the hMailserver logs too

I meant Sub OnSMTPData, not your added Sub OnHELO. Your OnHelo works as expected

I have this code in my Eventhandlers.vbs

Code: Select all

Sub OnSMTPData(oClient, oMessage)
	'included hMailserver event
	'uses functions: CustomMonthlyLog
	'uses globals: 

	Dim TempString,i
	If oClient.Username <> "" Then 
		TempString = "              Authenticated SMTP user is '" & CStr(oClient.Username) & "'"
		Call CustomMonthlyLog(TempString, "Port" & oClient.port)
	Else
		TempString = "       Further Details for IP Address = " & oClient.IPAddress & "  Helo sent was '" & CStr(oClient.helo) & "'"
		Call CustomMonthlyLog(TempString, "Port" & oClient.port)
	End If
	TempString = "            Message FROM = '" & omessage.FromAddress & "' and has " & OMessage.Recipients.Count & " Message Recipients"
	Call CustomMonthlyLog(TempString, "Port" & oClient.port)
	For i = 0 To oMessage.Recipients.Count-1
		TempString = "                                   Message Recipient number " & i+1 & " is '" & omessage.Recipients.Item(i).Address & "'"
		Call CustomMonthlyLog(TempString, "Port" & oClient.port)
	Next 'i
End Sub
My Port25 log looks like this

Code: Select all

2018-09-13 19:49:49.148  Port Number = 25    IP Address = 24.220.46.94    Helo = ''
2018-09-13 19:50:26.585         Further Details for IP Address = 24.220.46.94  Helo sent was 'ic3.org'
2018-09-13 19:50:26.617              Message FROM = 'no-reply@ic3.org' and has 1 Message Recipients
2018-09-13 19:50:26.648                                     Message Recipient number 1 is ***REMOVED ****
2018-09-13 19:57:06.507  Port Number = 25    IP Address = 66.220.155.153    Helo = ''
2018-09-13 19:57:34.039         Further Details for IP Address = 66.220.155.153  Helo sent was 'mx-out.facebook.com'
2018-09-13 19:57:34.070              Message FROM = 'notification@facebookmail.com' and has 1 Message Recipients
2018-09-13 19:57:34.101                                     Message Recipient number 1 is ***REMOVED ****
2018-09-13 20:02:17.539  Port Number = 25    IP Address = 209.85.215.173    Helo = ''
2018-09-13 20:02:44.445         Further Details for IP Address = 209.85.215.173  Helo sent was 'mail-pg1-f173.google.com'
2018-09-13 20:02:44.460              Message FROM = 'datavation.tools@gmail.com' and has 1 Message Recipients
2018-09-13 20:02:44.492                                     Message Recipient number 1 is ***REMOVED ****
2018-09-13 20:27:31.882  Port Number = 25    IP Address = 13.210.188.175    Helo = ''
2018-09-13 20:28:05.039         Further Details for IP Address = 13.210.188.175  Helo sent was 'mg-aaus-zeta.mailguard.com.au'
2018-09-13 20:28:05.070              Message FROM = ***REMOVED **** and has 1 Message Recipients
2018-09-13 20:28:05.101                                     Message Recipient number 1 is ***REMOVED ****
2018-09-13 21:01:59.148  Port Number = 25    IP Address = 103.8.195.34    Helo = ''
2018-09-13 21:02:27.804         Further Details for IP Address = 103.8.195.34  Helo sent was 'sportoptions.com'
2018-09-13 21:02:27.835              Message FROM = 'Finnvai@sportoptions.com' and has 1 Message Recipients
2018-09-13 21:02:27.867                                     Message Recipient number 1 is ***REMOVED ****
2018-09-13 21:06:22.429  Port Number = 25    IP Address = 172.93.138.134    Helo = ''
2018-09-13 21:06:49.992         Further Details for IP Address = 172.93.138.134  Helo sent was 'thunder1.stoneyap.biz'
2018-09-13 21:06:50.023              Message FROM = 'steve@stoneyap.biz' and has 1 Message Recipients
2018-09-13 21:06:50.054                                     Message Recipient number 1 is ***REMOVED ****
As you can see sub OnSMTPData gives a null EHLO until after the STartTLS
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Greylisting by subnetwork

Post by RvdH » 2018-09-13 14:22

I don't have empty oClient.HELO entries in OnSMTPData.... with or without STARTTLS
Btw, why the CStr(oClient.HELO) ? ....oClient.HELO always is a string

I would say the error is in the part where you build the lines like: "Port Number = 25 IP Address = 13.210.188.175 Helo = ''", are you 100% sure you pass oClient.HELO value there?

Simply write EventLog.Write(oClient.HELO) in Sub OnSMTPData to verify
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-09-13 15:33

mattg wrote:
2018-09-13 13:50
Yes I get that in the hMailserver logs too

I meant Sub OnSMTPData, not your added Sub OnHELO. Your OnHelo works as expected

I have this code in my Eventhandlers.vbs

Code: Select all

Sub OnSMTPData(oClient, oMessage)
	'included hMailserver event
	'uses functions: CustomMonthlyLog
	'uses globals: 

	Dim TempString,i
	If oClient.Username <> "" Then 
		TempString = "              Authenticated SMTP user is '" & CStr(oClient.Username) & "'"
		Call CustomMonthlyLog(TempString, "Port" & oClient.port)
	Else
		TempString = "       Further Details for IP Address = " & oClient.IPAddress & "  Helo sent was '" & CStr(oClient.helo) & "'"
		Call CustomMonthlyLog(TempString, "Port" & oClient.port)
	End If
	TempString = "            Message FROM = '" & omessage.FromAddress & "' and has " & OMessage.Recipients.Count & " Message Recipients"
	Call CustomMonthlyLog(TempString, "Port" & oClient.port)
	For i = 0 To oMessage.Recipients.Count-1
		TempString = "                                   Message Recipient number " & i+1 & " is '" & omessage.Recipients.Item(i).Address & "'"
		Call CustomMonthlyLog(TempString, "Port" & oClient.port)
	Next 'i
End Sub
My Port25 log looks like this

Code: Select all

2018-09-13 19:49:49.148  Port Number = 25    IP Address = 24.220.46.94    Helo = ''
2018-09-13 19:50:26.585         Further Details for IP Address = 24.220.46.94  Helo sent was 'ic3.org'
2018-09-13 19:50:26.617              Message FROM = 'no-reply@ic3.org' and has 1 Message Recipients
2018-09-13 19:50:26.648                                     Message Recipient number 1 is ***REMOVED ****
2018-09-13 19:57:06.507  Port Number = 25    IP Address = 66.220.155.153    Helo = ''
2018-09-13 19:57:34.039         Further Details for IP Address = 66.220.155.153  Helo sent was 'mx-out.facebook.com'
2018-09-13 19:57:34.070              Message FROM = 'notification@facebookmail.com' and has 1 Message Recipients
2018-09-13 19:57:34.101                                     Message Recipient number 1 is ***REMOVED ****
2018-09-13 20:02:17.539  Port Number = 25    IP Address = 209.85.215.173    Helo = ''
2018-09-13 20:02:44.445         Further Details for IP Address = 209.85.215.173  Helo sent was 'mail-pg1-f173.google.com'
2018-09-13 20:02:44.460              Message FROM = 'datavation.tools@gmail.com' and has 1 Message Recipients
2018-09-13 20:02:44.492                                     Message Recipient number 1 is ***REMOVED ****
2018-09-13 20:27:31.882  Port Number = 25    IP Address = 13.210.188.175    Helo = ''
2018-09-13 20:28:05.039         Further Details for IP Address = 13.210.188.175  Helo sent was 'mg-aaus-zeta.mailguard.com.au'
2018-09-13 20:28:05.070              Message FROM = ***REMOVED **** and has 1 Message Recipients
2018-09-13 20:28:05.101                                     Message Recipient number 1 is ***REMOVED ****
2018-09-13 21:01:59.148  Port Number = 25    IP Address = 103.8.195.34    Helo = ''
2018-09-13 21:02:27.804         Further Details for IP Address = 103.8.195.34  Helo sent was 'sportoptions.com'
2018-09-13 21:02:27.835              Message FROM = 'Finnvai@sportoptions.com' and has 1 Message Recipients
2018-09-13 21:02:27.867                                     Message Recipient number 1 is ***REMOVED ****
2018-09-13 21:06:22.429  Port Number = 25    IP Address = 172.93.138.134    Helo = ''
2018-09-13 21:06:49.992         Further Details for IP Address = 172.93.138.134  Helo sent was 'thunder1.stoneyap.biz'
2018-09-13 21:06:50.023              Message FROM = 'steve@stoneyap.biz' and has 1 Message Recipients
2018-09-13 21:06:50.054                                     Message Recipient number 1 is ***REMOVED ****
As you can see sub OnSMTPData gives a null EHLO until after the STartTLS
So, where is the line "Port Number = 25 IP Address = 24.220.46.94 Helo = '' written ?? Sub OnHELO() or Sub OnClientConnect() ?
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Greylisting by subnetwork

Post by RvdH » 2018-09-13 15:49

SorenR wrote:
2018-09-13 15:33
So, where is the line "Port Number = 25 IP Address = 24.220.46.94 Helo = '' written ?? Sub OnHELO() or Sub OnClientConnect() ?
That is what i was thinking :)
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-09-13 16:13

RvdH wrote:
2018-09-13 15:49
SorenR wrote:
2018-09-13 15:33
So, where is the line "Port Number = 25 IP Address = 24.220.46.94 Helo = '' written ?? Sub OnHELO() or Sub OnClientConnect() ?
That is what i was thinking :)
I'm going to test Matt's logging scheme, minus the top line...

Code: Select all

   '
   ' Custom logging of SMTP port
   '
   Dim EventLogX : Set EventLogX = New LogWriter
   EventLogX.LogType = "D" ' Default is "M"onth

   EventLogX.LogDir = "c:\hmailserver\temp\mattlog\" ' Default is hMailServer logs directory
   EventLogX.LogFile = "port" & oClient.port
   If oClient.Username <> "" Then
      EventLogX.Write( "              Authenticated SMTP user is '" & oClient.Username & "'" )
   Else
      EventLogX.Write( "       Further Details for IP Address = " & oClient.IPAddress & "  Helo sent was '" & oClient.HELO & "'" )
   End If
   EventLogX.Write( "            Message FROM = '" & omessage.FromAddress & "' and has " & OMessage.Recipients.Count & " Message Recipients" )
   For i = 0 To oMessage.Recipients.Count-1
      EventLogX.Write( "                                   Message Recipient number " & i+1 & " is '" & omessage.Recipients.Item(i).Address & "'" )
   Next
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Greylisting by subnetwork

Post by RvdH » 2018-09-13 16:16

I have tested oClient.HELO in OnSMTPData() for a few hours now....none empty entries as far, so i suspect, like you suggested he makes that other logline in OnClientConnect() where the oClient.HELO value is not populated

If that is the case that will be a big, 'duh' moment for mattg :mrgreen:
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-13 23:11

I can sew a zipper in a pair of pants
Thanks for an extensive list of your virtues; please find mine on http://www.gelbukh.com.

But back to the point. This thread is not about debugging OnHELO and not about whether greylisting may be unsuitable for some types of users in some situations. Those users can disable it.

(My own feeling about greylisting is that it is the worst form of spam blocking, except for all the others. But my feelings are off-topic here.)

This thread is about a specific proposal to make greylisting work with server farms.
If an ISP has their own servers on say 4 IP addresses, and own two thirds of a /24 IP range block, and all other of their own IPs are provided to users, then once you allow the /24 block through grey listing, then almost anybody could be the next to send from that /24 block
Nope. I just re-read the documentation: it works by triples IP-from-to. So this will only unblock those spammers on the /24 subnet who happen to use the same from address as legitimate messages. No good (in theory there might possibly be some chance for this to happen), but better than disabling greylisting at all. And if this happens, there is a blacklist for those IPs, too.

So far, no convincing argument against an option for greylisting (for those of us who use it) to be able to deal with server farms in the way I proposed.

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Greylisting by subnetwork

Post by RvdH » 2018-09-14 01:12

I think I can make it...but what is your fee?

Just kidding....it shouldn't be that hard....but on the other hand there are ways to bypass your issue...so why changing something that works?
.. sorry, i don't get a lot of mail originating form University of Chilpansingo i guess
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Greylisting by subnetwork

Post by mattg » 2018-09-14 01:25

RvdH wrote:
2018-09-13 16:16
I have tested oClient.HELO in OnSMTPData() for a few hours now....none empty entries as far, so i suspect, like you suggested he makes that other logline in OnClientConnect() where the oClient.HELO value is not populated

If that is the case that will be a big, 'duh' moment for mattg :mrgreen:
As usual you guys are correct
That is exactly what I do


@gelbukh
Read this thread for an earlier discussion on this topic >> viewtopic.php?f=21&t=29238

gelbukh wrote:
2018-09-13 23:11
But back to the point. This thread is not about debugging OnHELO and not about whether greylisting may be unsuitable for some types of users in some situations. Those users can disable it.

(My own feeling about greylisting is that it is the worst form of spam blocking, except for all the others. But my feelings are off-topic here.)

This thread is about a specific proposal to make greylisting work with server farms.
If an ISP has their own servers on say 4 IP addresses, and own two thirds of a /24 IP range block, and all other of their own IPs are provided to users, then once you allow the /24 block through grey listing, then almost anybody could be the next to send from that /24 block
Nope. I just re-read the documentation: it works by triples IP-from-to. So this will only unblock those spammers on the /24 subnet who happen to use the same from address as legitimate messages. No good (in theory there might possibly be some chance for this to happen), but better than disabling greylisting at all. And if this happens, there is a blacklist for those IPs, too.

So far, no convincing argument against an option for greylisting (for those of us who use it) to be able to deal with server farms in the way I proposed.
Yes I know how greylisting works, I was offering a reason why your idea /24 block whitelisting won't work.

I'll be happy to a see (and use) a solution from RvdH / SorenR if one is forthcoming.
The discussion about OnHelo and OnSMTPData is about finding a solution to the problem that you identify and that we have discussed for many years.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Greylisting by subnetwork

Post by RvdH » 2018-09-14 01:28

And while you/we are one it...viewtopic.php?f=9&t=32375

@mattg, we all from time to time have bad days, weeks, months or whatever :mrgreen: :lol:
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-14 02:03

I think I can make it...but what is your fee?
Great, now it sounds more to the point :)
i don't get a lot of mail originating form University of Chilpansingo
Me neither, but when one arrives once in a year and is lost, it may be a very important one. I had had such cases, with a university uning a server farm (but not gmail).

My usage scenario (a small department server with a handful of users, professors and students):
  • no time-critical tasks: a delay of half an hour is OK;
  • often mission-critical tasks: losing a message can be very bad;
  • messages from many "third-party" servers, such as other university servers, not only large providers: some of them are legitimate but misconfigured;
  • lots of spam, because publishers put our addresses all over Internet (you can find hunreds of mentions of mine): even after keyword-based filtering, it takes me hours every day to go through my mail, after which I am too tired to even respond to the real ones;
  • need to concentrate: typically my inbox has some 300+ messages a day, many of them spam, and I cannot quickly grasp the important ones.
I understand that there are other usage scenarios, but surely there are other people in my situation.
why changing something that works?
Because it doesn't. Nowadays a technique that blocks server farms doesn't "work".
but on the other hand there are ways to bypass your issue...
Sadly, from this discussion none looks as an adequate replacement for greylisting. The closest is the delay, but I would not risk to mess with it my production environment.

You say it works for you? How do you know? Because nobody so far told you "Hey, I tried sending you a message but you don't reply?" Even if they did, they would have no way to tell you this. In my case, an official from the ministry will not even bother calling me on phone -- she sends me one message, and if I don't act by the deadline, she cuts my funding or rejects my application. I will never know what happened. And no, I can't whitelist all (unknown to me) officials from all relevant ministries all over the world. Properly configured greylisting will surely work, whereas all the tricks proposed on this page don't sound too reliable.

The very fact that we discuss them here indicates that they are doubtful. If the delay were a solid proven technique while greylisting a silly idea, HMS (and others) will include a prominent option of a delay on OnHELO in its settings, and would have no traces of greylisting. If delay were a solid proven technique, you would not have given me your scripts here, you would just have pointed me to the Wikipedia page on the Antispam Delay. The very fact that you have to implement your solution to a such a very common and long-standing problem as spam filtering with some kitchen-made scripts in a still-in-alfa event hook clearly hints on that it is not an obviously right thing to do.
Last edited by gelbukh on 2018-09-14 02:25, edited 3 times in total.

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-14 02:14

Read this thread for an earlier discussion on this topic >> viewtopic.php?f=21&t=29238
Yes, I've read it before opening mine. The techniques discussed there have the same fundamental flaws that we have already discussed:
  • they rely on the server's self-description in HELO, which is trivial to fake at no cost for the spammer,
  • they whitelist only a predefined set of providers, which does not solve the problem (though alleviates it, I agree).
I was offering a reason why your idea /24 block whitelisting won't work.
Thank you, but you only showed that it might have false negatives (= allow some spam) in an (I believe) quite rare situations. This is much better than false positives (= lost legitimate messages) from which all other techniques suffer (including keyword filtering, whitelisting known providers, and arguably delays), or than disabling any protection at all.

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-09-14 03:19

gelbukh wrote:
2018-09-14 02:03
You say it works for you? How do you know? Because nobody so far told you "Hey, I tried sending you a message but you don't reply?" Even if they did, they would have no way to tell you this. In my case, an official from the ministry will not even bother calling me on phone -- she sends me one message, and if I don't act by the deadline, she cuts my funding or rejects my application. I will never know what happened. And no, I can't whitelist all (unknown to me) officials from all relevant ministries all over the world. Properly configured greylisting will surely work, whereas all the tricks proposed on this page don't sound too reliable.
Well... The war on SPAM. It's not ideology or religion that controls SPAM - It's MONEY!
Whenever you think you have won - you loose, and it's not the person that read the most books who wins, it the person who gets his hands dirty in the trenches and who still have an ounce of imagination left that don't loose as much as the others.

I pity people living outside Europe, in EU we own our personal data by law and with GDPR we can fight back.

By the way, I have taken precautions on my server not to loose any mails whatsoever. I engage an offsite backup-mx for those occations where the sender gives up on my server and move on to the next MX record in DNS. Since I moved away from GreyListing the backup-mx is utilized less and less.

An other issue is, by rejecting emails you loose a big chunk of statistical data that can be used to classify SPAM. I know a few of us here reclassify SPAM in two groups; "light SPAM" that is "safe" to pass on to the user and "The Rest" that is only suitable for training SpamAssassin.

With your knowledge in AI you should easily be able to whip something up for SpamAssassin that can give us World Peace. :wink:

Oh by the way...

https://antispam.byteplant.com/products ... ilter.html
http://articles.marco.org/238
https://www.tldp.org/HOWTO/Spam-Filteri ... elays.html
http://altlasten.lutz.donnerhacke.de/mi ... be.en.html
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Greylisting by subnetwork

Post by mattg » 2018-09-14 03:30

gelbukh wrote:
2018-09-14 02:14
I was offering a reason why your idea /24 block whitelisting won't work.
Thank you, but you only showed that it might have false negatives (= allow some spam) in an (I believe) quite rare situations. This is much better than false positives (= lost legitimate messages) from which all other techniques suffer (including keyword filtering, whitelisting known providers, and arguably delays), or than disabling any protection at all.
I think you would allow much spam, not 'some spam', and I don't believe this would be a rare situation, but rather a very common one. Only a real life test would show.

If you are happy to get 'some spam' then disable greylisting, and use other spam measures. Essentially that's what we have all done here.
gelbukh wrote:
2018-09-14 02:14
The techniques discussed there have the same fundamental flaws that we have already discussed:

they rely on the server's self-description in HELO, which is trivial to fake at no cost for the spammer,
they whitelist only a predefined set of providers, which does not solve the problem (though alleviates it, I agree).
On each point
#1 not if we can somehow check the IP being used to send and compare that to the allowed sender list ie spf.gmail.com

#2 this is the point of greylisting, protecting you from unknown senders / providers. Surely the 'predefined set' will grow and change over time, and be different on all servers.

In general terms of beating SPAM, I'd like to have DMARC and DANE implemented into hMailserver, and to have more control over SPF testing
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-14 03:46

I pity people living outside Europe, in EU we own our personal data by law and with GDPR we can fight back.
In some businesses it's inevitable to have your address spread. Say, it's a European publisher Springer who requires email to be indicated on every published paper or book, by "the policy of the company." I tried to complain, but they have their rules. And it's Europe. (BTW, in all other respects the publisher that I enjoy most working with!)
With your knowledge in AI you should easily be able to whip something up for SpamAssassin that can give us World Peace. :wink:
Not quite feasible.

First, I am involved in other battles (publish or perish) and fighting spam is not my main occupation.

Second, big guys like Microsoft and Google are on it already. The bad side of the AI techniques is that they don't offer as low false positives as greylisting, which in some scenarios is unacceptable -- imagine a biochemist eagerly waiting for an acceptance decision on his paper on effects of viagra on the cardiovascular system. From my own experience, I've missed an invitation for giving a keynote lecture at the XXX International Conference on Computer Science, you guess why.

And third, the spammers will fight back with AI, too (I've got invitations from spam-sending companies for such projects).

If you have a good dataset, my students and I can try doing something reasonable on it. Though I cannot imagine a scenario of working on this topic outside a large email provider like Google. Say, you cannot reasonably share a dataset, for privacy reasons: you cannot share negative (= legitimate) examples. We have had this problem in medical research.

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-14 04:00

#2 this is the point of greylisting, protecting you from unknown senders / providers. Surely the 'predefined set' will grow and change over time, and be different on all servers.
Nope. Greylisting does not involve any predefined lists. The point of greylisting is in allowing servers to self-whitelist at a cost (without pre-existing knowledge at the side of the receiver).

There are other varieties of the idea of imposing a cost on sending messages (the delay discussed here is one of them; there are cryptographic proposals to impose the CPU cost), but they all so far have some problems (discussing which is off-topic in this thread, but we can open a new thread for it), such as relying on capabilities of the sender that are not guaranteed.
I think you would allow much spam, not 'some spam', and I don't believe this would be a rare situation, but rather a very common one. Only a real life test would show.
Maybe you are right. I have no hard data to support my feeling of that it is unlikely that a spammer on the same subnet will figure out the from address recently used from the legitimate server. But as you say, we can test it and see. I admit that in the end stats may show that you were right and I was not.

Or, some users may find it useful and some may not, depending on the from addresses they typically expect.

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Greylisting by subnetwork

Post by mattg » 2018-09-14 04:12

gelbukh wrote:
2018-09-14 04:00
#2 this is the point of greylisting, protecting you from unknown senders / providers. Surely the 'predefined set' will grow and change over time, and be different on all servers.
Nope. Greylisting does not involve any predefined lists. The point of greylisting is in allowing servers to self-whitelist at a cost (without pre-existing knowledge at the side of the receiver).
So if a server can self whitelist, won't the spammers be doing that?
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-09-14 04:21

gelbukh wrote:
2018-09-14 04:00
#2 this is the point of greylisting, protecting you from unknown senders / providers. Surely the 'predefined set' will grow and change over time, and be different on all servers.
Nope. Greylisting does not involve any predefined lists. The point of greylisting is in allowing servers to self-whitelist at a cost (without pre-existing knowledge at the side of the receiver).

There are other varieties of the idea of imposing a cost on sending messages (the delay discussed here is one of them; there are cryptographic proposals to impose the CPU cost), but they all so far have some problems (discussing which is off-topic in this thread, but we can open a new thread for it), such as relying on capabilities of the sender that are not guaranteed.
I think you would allow much spam, not 'some spam', and I don't believe this would be a rare situation, but rather a very common one. Only a real life test would show.
Maybe you are right. I have no hard data to support my feeling of that it is unlikely that a spammer on the same subnet will figure out the from address recently used from the legitimate server. But as you say, we can test it and see. I admit that in the end stats may show that you were right and I was not.

Or, some users may find it useful and some may not, depending on the from addresses they typically expect.
There is only one way to find out... Even weeks = greylisting and odd weeks = delays.
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-14 04:35

So if a server can self whitelist, won't the spammers be doing that?
Nope. The key here is the cost: self-whitelisting has a cost that spammers cannot afford (namely, the logistics of a second attempt).

What differentiates a legitimate server from a spammer is the willingness to pay a cost (in terms of logistics, delays, CPU, etc.), and greylisting exploits this difference. Other techniques such as AI miss it. Cryptographic techniques address it directly: they challenge the sender with spending considerable CPU resources per each message, which a spammer, again, cannot afford.
There is only one way to find out... Even weeks = greylisting and odd weeks = delays.
I can do it. For this, you (or somebody) need to implement the feature in question, in order to see if it passes too much spam.

So far I think my personal server blocks about 100 messages a day with greylisting (excluding those that pass after some attemts), and passes probably 300, of which some 30-50 are real; I have not analyzed this carefully. In any case not impressive, but still 25% less work for me to sort my mail. If I disable it (which I have to, unless it properly handles server farms), I will have to filter manually 100 messages more every day, apart from my main work.

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-14 04:46

BTW (though off-topic here), I did not understand why the delay you use should be applied in OnHELO and not in any other event before actual delivery? I would rather apply it after getting all the data, when I can analyze even the contents of the message to decide if I want to pass this message unconditionally (say, if it mentions words relevant for my organization, such as "conference" or "professor" in my case). Any delay at any stage would break communication with a spammer and thus prevent delivery, no?

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-14 04:53

I would rather apply it after getting all the data, when I can analyze even the contents of the message to decide if I want to pass this message unconditionally
Ah, and the same applies to greylisting and all other antispam tests, including blacklists. But this is too much to ask for, if a simple /24 masking took us 49 messages to discuss :D

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Greylisting by subnetwork

Post by mattg » 2018-09-14 05:11

There's no problem with 50 posts per topic - discussion is good.

I see a difference in what you are saying now.
You still want to greylist big senders.
I want to bypass greylisting for big senders, because I trust that if it is genuinely from one of their servers that they have already performed some spam mitigation, and that coupled with my other spam tests (especially including SpamAssassin) will detect most spam.


Also, I normally only apply delays to those message that look suspicious. If they pass all of my tests initially they get through. If they are a little off they get delayed a little, if they are a lot off they get delayed a lot and also autobanned, so they can't try again. (and a few steps in between)
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-14 07:53

You still want to greylist big senders.
I don't care of big senders. If things work correctly, they work correctly for all, and there is no reason for special tweaks for big senders. (You may want to treat them specially for the sake of efficiency; in my case I don't care about efficiency, I only care about correctness.)

Another difference is that you seem to care mostly about false negatives (= spam that passes), while I care more about false positives (= lost legitimate messages). With this, you seem to feel OK while big senders are well-attended, and for you small ones are all potential spammers not deserving attention; I care more about treating correctly legitimate small senders (then big ones will enjoy the same correct treatment, too).

Bad news is that nowadays "small" senders also use server farms, so I would have to treat them with the same precautions for the server farms, but unlike big ones, I don't have a complete list of their names. Maybe the third difference is that you consider server farm problem as specific to big senders that you can count by the fingers of one hand, while I consider it a common problem not solvable by a predefined whitelist.

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Greylisting by subnetwork

Post by mattg » 2018-09-14 08:03

I don't like missed legitimate mail, and don't believe that I've ever missed any (but of course, how would I know for sure?)

In fact, to get around the proliferation of large senders, and the ineffectiveness of greylisting at this point in time because of these large senders, I (and many others) have simply turned greylisting off.

At present, I reject lots of spam (and tell the sender that their mail was rejected), and I accept some spam, but send that to an IMAP folder where it is managed, always looking for what you call 'false positives'.

I don't really have favorites of either large or small senders. As long as they play nice and by the established rules (spf ending in -all, DKIM, DMARC) then I'm happy enough. Although fighting SPAM is nearly a full time job.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jimimaseye
Moderator
Moderator
Posts: 10053
Joined: 2011-09-08 17:48

Re: Greylisting by subnetwork

Post by jimimaseye » 2018-09-14 09:04

gelbukh wrote:
2018-09-14 07:53
You still want to greylist big senders.
I don't care of big senders. If things work correctly, they work correctly for all, and there is no reason for special tweaks for big senders.


SCENARIO:

GREYLISTING is on

Someone sends in an email

Server farm attempts delivery from ip a.b.c.d

Hits greylist.

Server reattempts delivery from j.k.l.m

Hits greylist

Server reattempts delivery from w.x.y.z

.
.


How does this proposed solution help the above situation?


The point is that your proposed solution is a non-standard tweak to the standard greylist practice and you are trying to compensate for only a portion of inadequacies of greylisting. Its only half a job and doesnt cater for all inadequacies and doesnt deal with the problem fully.

Greylist has its problems when reputable email server (admins) do not WANT to acknowledge the standards and work with them (like gnail et al). But as an admin that is something you have to accept when you choose to use Greylisting. Sure, RvdH can create a version for you as you requested, but it doesnt stop you having problems with Greylisting. Turn it on and accept that some senders will be delayed or turn it off for full confidence it will not cause occasionally problems and use alternative methods that will apply to all servers (such as the proven VERY effective 20 second delay).

And given that google servers are used more and more for businesses as their service provider its something you REALLY need to think about (its effectively making greylisting a nuisance for large scale users).

This is why you need to care about big senders.
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

palinka
Senior user
Senior user
Posts: 4455
Joined: 2017-09-12 17:57

Re: Greylisting by subnetwork

Post by palinka » 2018-09-14 12:01

gelbukh wrote:
2018-09-14 03:46
imagine a biochemist eagerly waiting for an acceptance decision on his paper on effects of viagra on the cardiovascular system.
It seems to me that people legitimately working with Viagra, etc must know that there is an issue with spam and have already worked up solutions to circumvent getting trapped by spamassassin. Otherwise they wouldn't be able to use email at all!

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-09-14 12:14

gelbukh wrote:
2018-09-14 07:53
I don't care of big senders. If things work correctly, they work correctly for all, and there is no reason for special tweaks for big senders. (You may want to treat them specially for the sake of efficiency; in my case I don't care about efficiency, I only care about correctness.)
See, this is why we are loosing the fight against SPAM ... No imagination ... No ingenuity ... No sense of adventure.

Aaron Poffenberger: Fighting Spam at the Frontline -- BSDCan 2018 (1 hour, so secure coffee and popcorn in advance)
https://www.youtube.com/watch?v=PKY6rSpzTIQ

viewtopic.php?p=202139#p202139
This guy is actually doing something interesting and RvdH picked up on it with his version of SPF based Whitelist Greylist of my dynamic EHLO based dynamic Whitelist Greylist... It does not need to be Greylist, it can be the deciding faktor whether the connection is to be delayed or not.

Part of fighting SPAM is using blacklists and I have found that the Snowshoe list is quite effective. DNS blacklist "sbl.spamhaus.org" and look for "127.0.0.3". Kill the connection on sight :!:

Code: Select all

'
' System Scripting Runtime COM object ("SScripting.IPNetwork")
' http://www.netal.com/ssr.htm
' Binary -> http://www.netal.com/software/ssr15.zip
'
Function IsSnowShoe(strIP) : IsSnowShoe = False
   Dim a
   a = Split(strIP, ".")
   With CreateObject("SScripting.IPNetwork")
      strIP = .DNSLookup(a(3) & "." & a(2) & "." & a(1) & "." & a(0) & ".sbl.spamhaus.org")
   End With
   If (strIP = "127.0.0.3") Then IsSnowShoe = True
End Function

Code: Select all

Sub OnClientConnect(oClient)
   '
   ' SnowShoe SPAM detection
   '
   If IsSnowShoe(oClient.IPAddress) Then
      Result.Value = 1
      Exit Sub
   End If
End Sub
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-09-14 12:18

palinka wrote:
2018-09-14 12:01
gelbukh wrote:
2018-09-14 03:46
imagine a biochemist eagerly waiting for an acceptance decision on his paper on effects of viagra on the cardiovascular system.
It seems to me that people legitimately working with Viagra, etc must know that there is an issue with spam and have already worked up solutions to circumvent getting trapped by spamassassin. Otherwise they wouldn't be able to use email at all!
If they are working on Viagra they probably have other priorities on their mind :mrgreen: or they would call it "sildenafil citrate" or "UK-92,480".
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-09-14 21:56

Stumbled over this today: https://dmarcian.com/spf-survey/

"google.com" ... 25 netblocks or 322.816 individual IPv4 addresses
"outlook.com" ... 82 netblocks or 654.455 individual IPv4 addresses
"facebook.com" ... 8 netblocks or 1.281 individual IPv4 addresses
"twitter.com" ... 49 netblocks or 330.296 individual IPv4 addresses
"instagram.com" ... 17 netblocks or 20.068 individual IPv4 addresses

That's 181 netblocks or 1.328.916 individual IPv4 addresses and that's only covering social media. :shock:

Damn... I forgot to check the remaining 55 Social Networking sites including Tumbl, Snapchat, Pinterest etc. etc. https://makeawebsitehub.com/social-media-sites/
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
jimimaseye
Moderator
Moderator
Posts: 10053
Joined: 2011-09-08 17:48

Re: Greylisting by subnetwork

Post by jimimaseye » 2018-09-14 22:28

I don't want to distract from the point you are making but why do you consider Google and outlook as social media servers? They are services (email providers and hosting) whilst the others exist only and are dedicated to the core for providing social media.

Still even without that 976k, with the other social media providers yet to be included it's still a lot of servers for feeding this social media nonsense.

What a world. People really have too much time to waste.
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-09-14 23:04

jimimaseye wrote:
2018-09-14 22:28
I don't want to distract from the point you are making but why do you consider Google and outlook as social media servers? They are services (email providers and hosting) whilst the others exist only and are dedicated to the core for providing social media.

Still even without that 976k, with the other social media providers yet to be included it's still a lot of servers for feeding this social media nonsense.

What a world. People really have too much time to waste.
Google+, MSN and Skype ?? What do I know :mrgreen: I'm still debating myself if I should kill my Facebook and Google+ accounts and just settle for Linkedin, Skype, Twitter, Instagram and Snapchat. ICQ closed my account (69108473) some years ago due to inactivity and Yahoo Messenger went the same way.
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

Post Reply