Real Time White Lists (dnsbl/wl)

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
palinka
Senior user
Senior user
Posts: 2172
Joined: 2017-09-12 17:57

Real Time White Lists (dnsbl/wl)

Post by palinka » 2019-12-15 00:54

Does anyone know of a good white list to check against?

palinka
Senior user
Senior user
Posts: 2172
Joined: 2017-09-12 17:57

Re: Real Time White Lists (dnsbl/wl)

Post by palinka » 2019-12-16 18:17

I looked around and I'm testing the following:

MailSpike
HostKarma
NSZones
SPFBL
SpamDonkey

There's one more I want to fit in if I have time: http://dkimwl.org. The scoring is different.

The idea is to knock out false positives for stuff like facebook's new idiot dynamic-looking PTRs. So, whitelist test first - if yes, skip the other filters. But I don't have a lot of confidence in them. For example, Chase Bank is not whitelisted on any list that I've tried so far, so I'm applying it to all incoming mail just to see what gets triggered. (Thanks to Soren for the code. :mrgreen: )

Also, I'm using geoip filter and I would like the whitelist to bypass that as well.

Code: Select all

'******************************************************************************************************************************
'********** WhiteList Tests                                                                                          **********
'******************************************************************************************************************************

Function TestLog(TestLogString)
	Const ForAppending = 8
	Dim objFSO : Set objFSO=CreateObject("Scripting.FileSystemObject")
	Dim outFile : outFile="C:\path\to\WhiteListTest.txt"
	Dim objFile : Set objFile = objFSO.OpenTextFile(outFile,ForAppending,True)
	objFile.WriteLine(TestLogString)
	objFile.Close
End Function

Function MailSpike(strIP)
	Dim a : a = Split(strIP, ".")
	With CreateObject("DNSLibrary.DNSResolver")
		Dim strLookup : strLookup = .DNSLookup(a(3) & "." & a(2) & "." & a(1) & "." & a(0) & ".rep.mailspike.net")
	End With
    Select Case strLookup
        Case "127.0.0.18"
             MailSpike = "127.0.0.18: Good Reputation"
        Case "127.0.0.19"
             MailSpike = "127.0.0.19: Very Good Reputation"
        Case "127.0.0.20"
             MailSpike = "127.0.0.20: Excellent Reputation"
        Case Else
             MailSpike = ""
    End Select
	If MailSpike <> "" Then Call TestLog(Now() & " - " & strIP & vbTab & " - MailSpike  : " & MailSpike)
End Function

Function HostKarma(strIP)
	Dim a : a = Split(strIP, ".")
	With CreateObject("DNSLibrary.DNSResolver")
		Dim strLookup : strLookup = .DNSLookup(a(3) & "." & a(2) & "." & a(1) & "." & a(0) & ".hostkarma.junkemailfilter.com")
	End With
    Select Case strLookup
        Case "127.0.0.1"
             HostKarma = "127.0.0.1: Whitelisted - Accept As Good"
        Case "127.0.0.15"
             HostKarma = "127.0.0.5: NoBL Listed - not a spam source - do not blacklist - maybe whitelist"
        Case Else
             HostKarma = ""
    End Select
	If HostKarma <> "" Then Call TestLog(Now() & " - " & strIP & vbTab & " - HostKarma  : " & HostKarma)
End Function

Function NSZones(strIP)
	Dim a : a = Split(strIP, ".")
	With CreateObject("DNSLibrary.DNSResolver")
		Dim strLookup : strLookup = .DNSLookup(a(3) & "." & a(2) & "." & a(1) & "." & a(0) & ".wl.nszones.com")
	End With
	If strLookup = "127.0.0.5" Then Call TestLog(Now() & " - " & strIP & vbTab & " - NSZones    : 127.0.0.5: Whitelisted")
End Function

Function SPFBL(strIP)
	Dim a : a = Split(strIP, ".")
	With CreateObject("DNSLibrary.DNSResolver")
		Dim strLookup : strLookup = .DNSLookup(a(3) & "." & a(2) & "." & a(1) & "." & a(0) & ".dnswl.spfbl.net")
	End With
    Select Case strLookup
        Case "127.0.0.2"
             SPFBL = "127.0.0.2: Listed as excellent reputation, confirmed by the community"
        Case "127.0.0.3"
             SPFBL = "127.0.0.3: Listed as public service or indispensable for the proper functioning of society"
        Case "127.0.0.4"
             SPFBL = "127.0.0.4: Listed as corporate message service, forbidden to use for marketing purposes"
        Case "127.0.0.5"
             SPFBL = "127.0.0.5: Listed as safe bulk service"
        Case Else
             SPFBL = ""
    End Select
	If SPFBL <> "" Then Call TestLog(Now() & " - " & strIP & vbTab & " - SPFBL      : " & SPFBL)
End Function

Function SpamDonkey(strIP)
	Dim a : a = Split(strIP, ".")
	With CreateObject("DNSLibrary.DNSResolver")
		Dim strLookup : strLookup = .DNSLookup(a(3) & "." & a(2) & "." & a(1) & "." & a(0) & ".dnsbl.spamdonkey.com")
	End With
    Select Case strLookup
        Case "126.0.0.0"
             SpamDonkey = "126.0.0.0: Whitelisted"
        Case "127.0.0.15"
             SpamDonkey = "127.0.0.0: Clean Level - no data or not a spammer"
        Case Else
             SpamDonkey = ""
    End Select
	If SpamDonkey <> "" Then Call TestLog(Now() & " - " & strIP & vbTab & " - SpamDonkey : " & SpamDonkey)
End Function


Very end of OnAcceptMessage:

Code: Select all

	'	Whitelist Tests
	Call TestLog("----------------------------------------------------------------------------------------------")
	Call TestLog(Now() & " - " & oClient.IpAddress & vbTab & " - From: " & oMessage.FromAddress & " - To: " & oMessage.Recipients(0).OriginalAddress & " - Subj: " & oMessage.Subject)
	Call MailSpike(oClient.IpAddress)
	Call HostKarma(oClient.IpAddress)
	Call NSZones(oClient.IpAddress)
	Call SPFBL(oClient.IpAddress)
	Call SpamDonkey(oClient.IpAddress)

palinka
Senior user
Senior user
Posts: 2172
Joined: 2017-09-12 17:57

Re: Real Time White Lists (dnsbl/wl)

Post by palinka » 2019-12-17 18:09

I decided to implement it.

I like the fact that it short circuits geoip, so hopefully legitimate mail from countries not on the allow list will make it through. In particular for my case, I'm thinking about messages from Samsung that have to do with work.

Code: Select all

Function Lookup(strRegEx, strMatch) : Lookup = False
   With CreateObject("VBScript.RegExp")
      .Pattern = strRegEx
      .Global = False
      .MultiLine = True
      .IgnoreCase = True
      If .Test(strMatch) Then Lookup = True
   End With
End Function


'******************************************************************************************************************************
'********** WhiteList Tests                                                                                          **********
'******************************************************************************************************************************

Function IsWLMailSpike(strIP) : IsWLMailSpike = False
	Dim a : a = Split(strIP, ".")
	With CreateObject("DNSLibrary.DNSResolver")
		' strIP = .DNSLookup(a(3) & "." & a(2) & "." & a(1) & "." & a(0) & ".rep.mailspike.net")
		Dim strLookup : strLookup = .DNSLookup(a(3) & "." & a(2) & "." & a(1) & "." & a(0) & ".rep.mailspike.net")
	End With
	Dim strRegEx : strRegEx = "^127\.0\.0\.(18|19|20)$" '18=Good Reputation, 19=Very Good Reputation, 20=Excellent Reputation
	IsWLMailSpike = Lookup(strRegEx, strLookup)
End Function

Function IsWLHostKarma(strIP) : IsWLHostKarma = False
	Dim a : a = Split(strIP, ".")
	With CreateObject("DNSLibrary.DNSResolver")
		Dim strLookup : strLookup = .DNSLookup(a(3) & "." & a(2) & "." & a(1) & "." & a(0) & ".hostkarma.junkemailfilter.com")
	End With
	Dim strRegEx : strRegEx = "^127\.0\.0\.(1|5)$" '1=Good Reputation, 5=NoBL
	IsWLHostKarma = Lookup(strRegEx, strLookup)
End Function

Function IsWLNSZones(strIP) : IsWLNSZones = False
	Dim a : a = Split(strIP, ".")
	With CreateObject("DNSLibrary.DNSResolver")
		Dim strLookup : strLookup = .DNSLookup(a(3) & "." & a(2) & "." & a(1) & "." & a(0) & ".wl.nszones.com")
	End With
	Dim strRegEx : strRegEx = "^127\.0\.0\.5$" '5=whitelisted
	IsWLNSZones = Lookup(strRegEx, strLookup)
End Function

Function IsWLSPFBL(strIP) : IsWLSPFBL = False
	Dim a : a = Split(strIP, ".")
	With CreateObject("DNSLibrary.DNSResolver")
		Dim strLookup : strLookup = .DNSLookup(a(3) & "." & a(2) & "." & a(1) & "." & a(0) & ".dnswl.spfbl.net")
	End With
	Dim strRegEx : strRegEx = "^127\.0\.0\.(2|3|4|5)$" '2=excellent rep, 3=indispensable public service, 4=corp email (no marketing), 5=safe bulk mail
	IsWLSPFBL = Lookup(strRegEx, strLookup)
End Function

Function IsWLSpamDonkey(strIP) : IsWLSpamDonkey = False
	Dim a : a = Split(strIP, ".")
	With CreateObject("DNSLibrary.DNSResolver")
		Dim strLookup : strLookup = .DNSLookup(a(3) & "." & a(2) & "." & a(1) & "." & a(0) & ".dnsbl.spamdonkey.com")
	End With
	Dim strRegEx : strRegEx = "^126\.0\.0\.0$" '126.0.0.0=whitelisted
	IsWLSpamDonkey = Lookup(strRegEx, strLookup)
End Function

Sub OnClientConnect(oClient)

	' Exclude Backup-MX & local LAN from test
	If (Left(oClient.IPAddress, 8) = "192.168.") Then Exit Sub
	If oClient.IPAddress = "127.0.0.1" Then Exit Sub

	' Test Whitelist
	If IsWLMailSpike(oClient.IPAddress) Then Exit Sub
	If IsWLHostKarma(oClient.IPAddress) Then Exit Sub
	If IsWLNSZones(oClient.IPAddress) Then Exit Sub
	If IsWLSPFBL(oClient.IPAddress) Then Exit Sub
	If IsWLSpamDonkey(oClient.IPAddress) Then Exit Sub

	' Other filters such as geoip go after whitelist tests

End Sub



User avatar
ras07
Normal user
Normal user
Posts: 228
Joined: 2010-03-11 08:51

Re: Real Time White Lists (dnsbl/wl)

Post by ras07 » 2019-12-18 00:19

palinka wrote:
2019-12-15 00:54
Does anyone know of a good white list to check against?
list.dnswl.org is pretty useful; note however that it blocks high-volume DNS servers, so if you aren't running your own DNS server it can be spotty (depending on your ISP) and if you're using Google or something for DNS if basically won't work at all.
hostkarma.junkemailfilter.com used to be pretty good (it's a combined black/white list) but I don't know whether it's being actively maintained. I don't seem to get near as many hits with it as I used to.

I've found whitelists to be kinda problematic in general. Pretty much the only spam that gets through my system anymore is from hacked accounts on legit email servers; in those cases, using whitelists to lower the spam score just serves to lower the effectiveness of other, non-IP-based spam detection methods.

I do like the idea of using it to short-circuit geoblocking, though.

ras

User avatar
mattg
Moderator
Moderator
Posts: 21103
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Real Time White Lists (dnsbl/wl)

Post by mattg » 2019-12-18 00:30

ras07 wrote:
2019-12-18 00:19
I've found whitelists to be kinda problematic in general. Pretty much the only spam that gets through my system anymore is from hacked accounts on legit email servers; in those cases, using whitelists to lower the spam score just serves to lower the effectiveness of other, non-IP-based spam detection methods.
Yep
ras07 wrote:
2019-12-18 00:19
I do like the idea of using it to short-circuit geoblocking, though.
I don't GeoIP Block (port 25), but I do spam score by GeoIP for port 25.
I also spam score based on language used

I do GeoIP block all other ports, and still require (TLSv1.2 or TLSv1.3) + AUTH for them
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

palinka
Senior user
Senior user
Posts: 2172
Joined: 2017-09-12 17:57

Re: Real Time White Lists (dnsbl/wl)

Post by palinka » 2019-12-18 01:13

Code: Select all

If NOT(IsWLMailSpike(testipaddress) OR IsWLHostKarma(testipaddress) OR IsWLNSZones(testipaddress) OR IsWLSPFBL(testipaddress) OR IsWLSpamDonkey(testipaddress)) Then
	GeoIP test...
End If
Its kind of clunky. Any ideas to un-clunk it? :mrgreen:

palinka
Senior user
Senior user
Posts: 2172
Joined: 2017-09-12 17:57

Re: Real Time White Lists (dnsbl/wl)

Post by palinka » 2019-12-18 02:25

I think this should un-clunkify it.

Code: Select all

	If IsWLMailSpike(oClient.IPAddress) Then
		Call AccRejDB(strPort, oClient.Port, "OnHELO", "Accepted", "WL-MailSpike", oClient.IPAddress, oClient.HELO)
	ElseIf IsWLHostKarma(oClient.IPAddress) Then
		Call AccRejDB(strPort, oClient.Port, "OnHELO", "Accepted", "WL-HostKarma", oClient.IPAddress, oClient.HELO)
	ElseIf IsWLNSZones(oClient.IPAddress) Then
		Call AccRejDB(strPort, oClient.Port, "OnHELO", "Accepted", "WL-NSZones", oClient.IPAddress, oClient.HELO)
	ElseIf IsWLSPFBL(oClient.IPAddress) Then
		Call AccRejDB(strPort, oClient.Port, "OnHELO", "Accepted", "WL-SPFBL", oClient.IPAddress, oClient.HELO)
	ElseIf IsWLSpamDonkey(oClient.IPAddress) Then
		Call AccRejDB(strPort, oClient.Port, "OnHELO", "Accepted", "WL-SpamDonkey", oClient.IPAddress, oClient.HELO)
	Else
		GeoIP code goes here....
	End If
The reason being, as you can see, I call my database log for these types of events. I like to keep a record of who is connecting so I can see later if someone is causing trouble, look for patterns, etc. (Plus, I actually call it at OnHELO so I'm able to grab the HELO for the logging.)

palinka
Senior user
Senior user
Posts: 2172
Joined: 2017-09-12 17:57

Re: Real Time White Lists (dnsbl/wl)

Post by palinka » 2019-12-18 16:53

More Un-Clunking. :mrgreen:

Here I have it set up so that whitelist testing is only called once, but the results can be used on any filter sequentially without having to call whitelist testing each time.

Function:

Code: Select all

Function Whitelisted(strIP, strPort, strHELO) : Whitelisted = 0

	Dim a : a = Split(strIP, ".")
	Dim strLookup, strRegEx
	Dim IsWLMailSpike, IsWLHostKarma, IsWLNSZones, IsWLSPFBL, IsWLSpamDonkey
	
	With CreateObject("DNSLibrary.DNSResolver")
		strLookup = .DNSLookup(a(3) & "." & a(2) & "." & a(1) & "." & a(0) & ".rep.mailspike.net")
	End With
	strRegEx = "^127\.0\.0\.(18|19|20)$" '18=Good, 19=Very Good, 20=Excellent Reputation
	IsWLMailSpike = Lookup(strRegEx, strLookup)
	If IsWLMailSpike Then Call AccRejDB(strPort, "OnHELO", "Accepted", "WL-MailSpike", strIP, strHELO)

	With CreateObject("DNSLibrary.DNSResolver")
		strLookup = .DNSLookup(a(3) & "." & a(2) & "." & a(1) & "." & a(0) & ".hostkarma.junkemailfilter.com")
	End With
	strRegEx = "^127\.0\.0\.(1|5)$" '1=Good, 5=NoBL
	IsWLHostKarma = Lookup(strRegEx, strLookup)
	If IsWLHostKarma Then Call AccRejDB(strPort, "OnHELO", "Accepted", "WL-HostKarma", strIP, strHELO)

	With CreateObject("DNSLibrary.DNSResolver")
		strLookup = .DNSLookup(a(3) & "." & a(2) & "." & a(1) & "." & a(0) & ".wl.nszones.com")
	End With
	strRegEx = "^127\.0\.0\.5$" '5=whitelisted
	IsWLNSZones = Lookup(strRegEx, strLookup)
	If IsWLNSZones Then Call AccRejDB(strPort, "OnHELO", "Accepted", "WL-NSZones", strIP, strHELO)

	With CreateObject("DNSLibrary.DNSResolver")
		strLookup = .DNSLookup(a(3) & "." & a(2) & "." & a(1) & "." & a(0) & ".dnswl.spfbl.net")
	End With
	strRegEx = "^127\.0\.0\.(2|3|4|5)$" '2=excellent rep, 3=indispensable public service, 4=corp email (no marketing), 5=safe bulk mail
	IsWLSPFBL = Lookup(strRegEx, strLookup)
	If IsWLSPFBL Then Call AccRejDB(strPort, "OnHELO", "Accepted", "WL-SPFBL", strIP, strHELO)

	With CreateObject("DNSLibrary.DNSResolver")
		strLookup = .DNSLookup(a(3) & "." & a(2) & "." & a(1) & "." & a(0) & ".dnsbl.spamdonkey.com")
	End With
	strRegEx = "^126\.0\.0\.0$" '126.0.0.0=whitelisted
	IsWLSpamDonkey = Lookup(strRegEx, strLookup)
	If IsWLSpamDonkey Then Call AccRejDB(strPort, "OnHELO", "Accepted", "WL-SpamDonkey", strIP, strHELO)

	If (IsWLMailSpike OR IsWLHostKarma OR IsWLNSZones OR IsWLSPFBL OR IsWLSpamDonkey) Then Whitelisted = 1

End Function

I'm calling it at OnHELO. If whitelisted by any of the whitelist tests, it gets a "1". If not, it gets a "0". Then I use the "0" later to weed out false positives from my other filters.

Call at beginning of OnHELO:

Code: Select all

	Dim IsWhitelisted : IsWhitelisted = Whitelisted(oClient.IPAddress, oClient.Port, oClient.HELO)
Then the result (0 or 1) can be used multiple times:

Code: Select all

	If IsWhitelisted = 0 Then
		GeoIP testing.....
	End If
	
	If PTR_Record <> "" AND IsWhitelisted = 0 Then
		Test for dynamic looking PTR....
	End If

	If IsWhitelisted = 0 Then
		Test for dynamic looking HELO....
	End If

	If IsWhitelisted = 0 Then
		Test if blacklisted by UCE Protect....
	End If
I DON'T know if I have ever rejected legitimate mail via GeoIP.

I DO know that certain well known servers have their full IP embedded in both PTR and HELO. Facebook does that (ex: 66-220-155-144.mail-mail.facebook.com). So stupid. I had a bunch of FPs over that.

I DO know that UCE Protect is good but very aggressive. They ban anything that has ever had a complaint. In the process many shared hosts have been listed, for which legit users get screwed. Amazon and Microsoft both are perennially listed because of the actions of a minority of users. I used to have both of them hardcoded into the UCEP test to skip if found. Now any whitelisted IP will be skipped. I'm not sure how amazon is going to work out yet, but MS/Outlook is whitelisted on a couple of those lists.

I think this is going to work out well. :mrgreen:

palinka
Senior user
Senior user
Posts: 2172
Joined: 2017-09-12 17:57

Re: Real Time White Lists (dnsbl/wl)

Post by palinka » 2019-12-20 17:13

Ran into a small issue. Probably wouldn't affect anyone else, but since I set up "CatchSpam", I was worried I might accidentally ban false positives.

I received a message that passed my filters on OnHELO, including being whitelisted by HostKarma and SPFBL. It was a poorly formed mailing list message to my dad from his small local bank, of which he is a customer. The bank outsources their email marketing to some outfit called mktomail.com, which is blacklisted by spamcop and UCE Protect. The blacklisting got picked up by spamassassin, and the BL scores + the other scores relating to the poor formatting pushed the message above the delete threshold.

So what to do? Fight fire with fire. :mrgreen:

local.cf:

Code: Select all

# Whitelist ips.whitelisted.org
header RCVD_IN_WHITELISTED eval:check_rbl('whitelisted', 'ips.whitelisted.org.')
describe RCVD_IN_WHITELISTED Sender listed in IPS-WHITELISTED-ORG
score RCVD_IN_WHITELISTED -1.5

# HostKarma DNSBL
header __RCVD_IN_HOSTKARMA eval:check_rbl('HOSTKARMA-lastexternal','hostkarma.junkemailfilter.com.')
describe __RCVD_IN_HOSTKARMA Sender listed in JunkEmailFilter
tflags __RCVD_IN_HOSTKARMA net
header RCVD_IN_HOSTKARMA_W eval:check_rbl_sub('HOSTKARMA-lastexternal', '127.0.0.1')
describe RCVD_IN_HOSTKARMA_W Sender listed in HOSTKARMA-WHITE
tflags RCVD_IN_HOSTKARMA_W net nice
score RCVD_IN_HOSTKARMA_W -1.5
header RCVD_IN_HOSTKARMA_BL eval:check_rbl_sub('HOSTKARMA-lastexternal', '127.0.0.2')
describe RCVD_IN_HOSTKARMA_BL Sender listed in HOSTKARMA-BLACK
tflags RCVD_IN_HOSTKARMA_BL net
score RCVD_IN_HOSTKARMA_BL 1.5
header RCVD_IN_HOSTKARMA_BR eval:check_rbl_sub('HOSTKARMA-lastexternal', '127.0.0.4')
describe RCVD_IN_HOSTKARMA_BR Sender listed in HOSTKARMA-BROWN
tflags RCVD_IN_HOSTKARMA_BR net
score RCVD_IN_HOSTKARMA_BR 1.0

# SPFBL whitelist
header __RCVD_IN_SPFBL_WL eval:check_rbl('spfbl-whitelist','dnswl.spfbl.net.')
describe __RCVD_IN_SPFBL_WL Sender listed in SPFBL whitelist
tflags __RCVD_IN_SPFBL_WL net
header RCVD_IN_SPFBL_WL_W eval:check_rbl_sub('spfbl-whitelist', '^127\.0\.0\.(2|3|4|5)$')
describe RCVD_IN_SPFBL_WL_W Sender listed in SPFBL whitelist
tflags RCVD_IN_SPFBL_WL_W net nice
score RCVD_IN_SPFBL_WL_W -1.5

# SpamDonkey whitelist
header __RCVD_IN_SPAMDONKEY_WL eval:check_rbl('spamdonkey-whitelist','dnsbl.spamdonkey.com.')
describe __RCVD_IN_SPAMDONKEY_WL Sender listed in SpamDonkey whitelist
tflags __RCVD_IN_SPAMDONKEY_WL net
header RCVD_IN_SPAMDONKEY_WL_W eval:check_rbl_sub('spamdonkey-whitelist', '^126\.0\.0\.0$')
describe RCVD_IN_SPAMDONKEY_WL_W Sender listed in SpamDonkey whitelist
tflags RCVD_IN_SPAMDONKEY_WL_W net nice
score RCVD_IN_SPAMDONKEY_WL_W -1.5

# NSZones whitelist
header __RCVD_IN_NSZONES_WL eval:check_rbl('nszones-whitelist','wl.nszones.com.')
describe __RCVD_IN_NSZONES_WL Sender listed in NSZones whitelist
tflags __RCVD_IN_NSZONES_WL net
header RCVD_IN_NSZONES_WL_W eval:check_rbl_sub('nszones-whitelist', '^127\.0\.0\.5$')
describe RCVD_IN_NSZONES_WL_W Sender listed in NSZones whitelist
tflags RCVD_IN_NSZONES_WL_W net nice
score RCVD_IN_NSZONES_WL_W -1.5

I'm not 100% set on the scores yet. I may increase (reduce the impact of) the whitelisting to -1 from -1.5 which I have now. Going to keep an eye on that.

Spamcop and UCE Protect are both stock dnsbl spamassassin tests. So is mailspike and dnswl.org, which have a whitelisting.

Spamcop and UCE Protect are both extremely aggressive. In the case of the message referenced above, I'm sure the bank is not sending spam, but rather some other client of mktomail.com is. Both results on that message showed mail was sent to spamtraps and I doubt the bank would do that. Many sleazy spamming marketing firms would, though.

Post Reply