Page 1 of 1

SPAM Filter not firing on some messages

Posted: 2016-08-04 14:47
by aaronwatson
I'm having trouble with the spam filter. Here's an example from AWStats log. Some messages seem to be intermittently bypassing the filter altogether. (v5.6.4)

"SMTPD" 2484 94703 "2016-07-28 14:45:23.740" "195.24.220.16" "SENT: 220 mailbox.ourdomain.com ESMTP"
"SMTPD" 2484 94703 "2016-07-28 14:45:24.115" "195.24.220.16" "RECEIVED: EHLO [195.24.220.16]"
"SMTPD" 2484 94703 "2016-07-28 14:45:24.115" "195.24.220.16" "SENT: 250-mailbox.ourdomain.com[nl]250-SIZE 10240000[nl]250-AUTH LOGIN[nl]250 HELP"
"SMTPD" 2476 94703 "2016-07-28 14:45:24.490" "195.24.220.16" "RECEIVED: MAIL FROM:<spammeraddress>"
"SMTPD" 2476 94703 "2016-07-28 14:45:24.506" "195.24.220.16" "SENT: 250 OK"
"SMTPD" 2500 94703 "2016-07-28 14:45:24.771" "195.24.220.16" "RECEIVED: RCPT TO:<internaladdress@ourdomain.com>"
"SMTPD" 2500 94703 "2016-07-28 14:45:24.771" "195.24.220.16" "SENT: 250 OK"
"SMTPD" 2536 94703 "2016-07-28 14:45:25.412" "195.24.220.16" "RECEIVED: DATA"
"SMTPD" 2536 94703 "2016-07-28 14:45:25.412" "195.24.220.16" "SENT: 354 OK, send."

Other relevant settings: When sender matches route treat sender as remote, when recipient matches route treat recipient as local.
I have no domains enabled - this is a strict Antispam relay situation so there's no authentication required. By my understanding it should just scan for spam and pass it along to our internal relay if its not rejected and 95% of the time it works flawlessly (the filters are visibly working in other areas of the log, just not on some messages). I'm just not sure why the filter's aren't being triggered the other 5% of the time.

Re: SPAM Filter not firing on some messages

Posted: 2016-08-05 01:01
by mattg
Is spam checking enabled in relevant IP range
Do you have any whitelist entries??
What logging do you have enabled?

Re: SPAM Filter not firing on some messages

Posted: 2016-08-05 16:52
by aaronwatson
Is spam checking enabled in relevant IP range
Yes. As a public filter, I have antivirus and anti-spam turned on. We only use it for inbound so internet is the only range (not including autoban entries which I set up as an added precaution)

Do you have any whitelist entries??
About 25 or so. I don't believe any of them (some including wildcards) have matched the sender domains we've been receiving but I will double check some of the more recent ones to compare.

What logging do you have enabled?
Application, SMTP, TCP/IP, AWSTATS
I just turned on Debug to see if it might help shed further light.

Re: SPAM Filter not firing on some messages

Posted: 2016-08-05 17:31
by jimimaseye
And ANTI-SPAM - General - Maximum Message Size . Check that too.

Re: SPAM Filter not firing on some messages

Posted: 2016-08-05 17:41
by aaronwatson
jimimaseye wrote:And ANTI-SPAM - General - Maximum Message Size . Check that too.
I have it set for 10240KBs so it should scan most messages under 10MBs (our ISP limit).

Re: SPAM Filter not firing on some messages

Posted: 2016-08-06 02:01
by mattg
debug logging should show tests that are tested against
aaronwatson wrote:Some messages seem to be intermittently bypassing the filter altogether. (v5.6.4)
How do you know this without Debug logging enabled?

Re: SPAM Filter not firing on some messages

Posted: 2016-08-15 14:15
by aaronwatson
I checked some of the sender domains and it's doesn't appear to be a whitelist problem. Here's the header from a sample message that made it through in case it's relevant:

Received: from hmailserver.ourdomain.com (192.168.#.#) by ourinternalmailserver.local
(192.168.#.#) with Microsoft SMTP Server (TLS) id 8.2.255.0; Sun, 14 Aug
2016 09:01:30 -0400
Received: from ourpublicIP (Unknown [112.124.57.223]) by hmailserver.ourdomain.com
with ESMTP ; Sun, 14 Aug 2016 09:01:28 -0400
Message-ID: <0649976335956-ZGLKUWFWULOSMZFZDBTHY@bhoznnkjia.beatpop.com>
From: Ben Castillo <Castillo_Ben@beatpop.com>
Subject: Re: Begin to work on binary options!
To: <myaddress@ourdomain.com>
Date: Sun, 14 Aug 2016 09:59:17 -0400
MIME-Version: 1.0
Content-Type: text/html
Content-Transfer-Encoding: 7Bit
Return-Path: hwkwaot@betterfloridaliving.com

One thing to not is that on the ones making it through, the "Received from" doesn't seem to be resolving. Here's an example received from on a message that was successfully filtered:

Received: from octa4.net.au (communigate.iinet.net.au [203.59.1.19]) by hmailserver.ourdomain.com

Re: SPAM Filter not firing on some messages

Posted: 2016-08-15 15:03
by SorenR
195.24.220.16 and 112.124.57.223 are both listed as "SnowShoe SPAM"

hMailAdmin -> Settings -> Anti-spam -> DNS blacklists -> Add ...

Enabled = Yes
DNS Host = sbl.spamhaus.org
Expected result = 127.0.0.3
Rejection message = RBL - Rejected by Spamhaus (Snowshoe)
Score = 1000 => Enough for your SPAM Delete threshold to catch it.
Snowshoe spamming is a strategy in which spam is propagated over several domains and IP addresses to weaken reputation metrics and avoid filters. The increasing number of IP addresses makes recognizing and capturing spam difficult, which means that a certain amount of spam reaches their destination email inboxes. Specialized spam trapping organizations are often hard pressed to identify and trap snowshoe spamming via conventional spam filters.

The strategy of snowshoe spamming is similar to actual snowshoes that distribute the weight of an individual over a wide area to avoid sinking into the snow. Likewise, snowshoe spamming delivers its weight over a wide area to remain clear of filters.
Lookup tool... http://multirbl.valli.org/lookup/

Re: SPAM Filter not firing on some messages

Posted: 2016-08-15 17:10
by aaronwatson
I'm currently using the zen list and have it well within delete threshold. It's recommended that if using Zen that you don't use the others (related to Spamhaus). Shouldn't that catch it?

https://www.spamhaus.org/zen/

Re: SPAM Filter not firing on some messages

Posted: 2016-08-15 20:09
by SorenR
aaronwatson wrote:I'm currently using the zen list and have it well within delete threshold. It's recommended that if using Zen that you don't use the others (related to Spamhaus). Shouldn't that catch it?

https://www.spamhaus.org/zen/
True. However many admins only tag emails as SPAM and put them into a SPAM folder. If sbl.spamhaus.org (or zen.spamhaus.org - sbl is a subset of zen) return 127.0.0.3 the email is safe to delete - that's what I do on my server.

Re: SPAM Filter not firing on some messages

Posted: 2016-08-15 23:26
by mattg
I actually query spamhaus three times each lookup, setting different scores for different results
aaronwatson wrote:I'm currently using the zen list and have it well within delete threshold. It's recommended that if using Zen that you don't use the others (related to Spamhaus). Shouldn't that catch it?
Depends on what return codes you are testing for in hMailserver.
What return codes is hMailserver scoring for a zen lookup?
If it doesn't include 3 then the snowshoe spam won't be looked up

Re: SPAM Filter not firing on some messages

Posted: 2016-08-16 01:31
by aaronwatson
Its looking for 3-11

Re: SPAM Filter not firing on some messages

Posted: 2016-08-16 02:40
by mattg
what score do you give that test?
What is your mark score?
What is your delete score?

Re: SPAM Filter not firing on some messages

Posted: 2016-08-16 02:54
by aaronwatson
What score do you give that test?
15

What is your mark score?
5

What is your delete score?
14

Intentionally aggressive which is why I was surprised so much gets through.

Re: SPAM Filter not firing on some messages

Posted: 2016-08-16 03:23
by mattg
do you use greylisting?
I find that using greylisting improves the chances of spamhaus or indeed the other antispam RBLs getting this right.

Greylist comes at cost in that mail is NOT instant though for new senders or for senders from Outlook or gmail hosted domains

Re: SPAM Filter not firing on some messages

Posted: 2016-08-18 15:11
by aaronwatson
We do use greylisting albeit a very short resend window and I've been allowing SPF and A/MX passthrough which might be making it redundant.
That said, I was watching the logs just now and saw 127.0.0.3 response code for zen.spamhaus.org.

I guess I'll have to keep tweaking the settings and see what can be done.

Re: SPAM Filter not firing on some messages

Posted: 2016-08-18 16:06
by SorenR
One thing I found to eliminate a lot of SPAM is a 20 second pause in OnClientConnect... Spammers really hate to wait :mrgreen:

Code: Select all

   Function Wait(sec)
      With CreateObject("WScript.Shell")
         .Run "sleep -m " & Int(sec * 1000), 0, True
      End With
   End Function

   Sub OnClientConnect(oClient)
      If (Left(oClient.IPAddress, 10) = "192.168.0.") Then Exit Sub ' Local LAN
      If (Left(oClient.IPAddress, 10) = "80.160.77.") Then Exit Sub ' ISP Backup-MX'es

      If (oClient.Port = 25) Then Wait(20)
   End Sub

Re: SPAM Filter not firing on some messages

Posted: 2016-08-18 17:35
by aaronwatson
I got one today. Here's a sanitized clip of the logs:

"TCPIP" 2512 "2016-08-18 10:28:49.831" "TCP - 36.84.3.229 connected to hmailserverinternalip:25."
"DEBUG" 2512 "2016-08-18 10:28:49.831" "TCP connection started for session 13262"
"SMTPD" 2512 13262 "2016-08-18 10:28:49.831" "36.84.3.229" "SENT: 220 mailbox.mydomain.com ESMTP"
"SMTPD" 2616 13262 "2016-08-18 10:28:50.128" "36.84.3.229" "RECEIVED: EHLO [36.84.3.229]"
"SMTPD" 2616 13262 "2016-08-18 10:28:50.128" "36.84.3.229" "SENT: 250-mailbox.mydomain.com[nl]250-SIZE 10240000[nl]250-AUTH LOGIN[nl]250 HELP"
"SMTPD" 2508 13262 "2016-08-18 10:28:50.425" "36.84.3.229" "RECEIVED: MAIL FROM:<Rhoda.wennerbom46@ups.es>"
"SMTPD" 2508 13262 "2016-08-18 10:28:50.425" "36.84.3.229" "SENT: 250 OK"
"SMTPD" 2528 13262 "2016-08-18 10:28:50.706" "36.84.3.229" "RECEIVED: RCPT TO:<me@mydomain.com>"
"SMTPD" 2528 13262 "2016-08-18 10:28:50.722" "36.84.3.229" "SENT: 250 OK"
"SMTPD" 2616 13262 "2016-08-18 10:28:51.019" "36.84.3.229" "RECEIVED: DATA"
"SMTPD" 2616 13262 "2016-08-18 10:28:51.019" "36.84.3.229" "SENT: 354 OK, send."
"TCPIP" 2560 "2016-08-18 10:29:00.769" "DNS - Query failure. Treating as temporary failure. Query: 229.3.84.36.in-addr.arpa, Type: 12, DnsQuery return value: 9002."
"DEBUG" 2560 "2016-08-18 10:29:00.769" "Could not retrieve PTR record for IP (false)! 36.84.3.229"
"DEBUG" 2560 "2016-08-18 10:29:00.769" "Adding task AsynchronousTask to work queue Asynchronous task queue"
"DEBUG" 2056 "2016-08-18 10:29:00.769" "Executing task AsynchronousTask in work queue Asynchronous task queue"
"DEBUG" 2056 "2016-08-18 10:29:00.769" "Saving message: {B5427132-3A4D-44E0-8B03-EB7A202C7CE6}.eml"
"DEBUG" 2056 "2016-08-18 10:29:00.769" "Requesting SMTPDeliveryManager to start message delivery"
"SMTPD" 2056 13262 "2016-08-18 10:29:00.769" "36.84.3.229" "SENT: 250 Queued (9.744 seconds)"
"DEBUG" 2344 "2016-08-18 10:29:00.769" "Adding task DeliveryTask to work queue SMTP delivery queue"
"DEBUG" 2424 "2016-08-18 10:29:00.769" "Executing task DeliveryTask in work queue SMTP delivery queue"
"DEBUG" 2424 "2016-08-18 10:29:00.769" "Delivering message..."
"APPLICATION" 2424 "2016-08-18 10:29:00.784" "SMTPDeliverer - Message 691069: Delivering message from Rhoda.wennerbom46@ups.es to me@mydomain.com. File: C:\Program Files (x86)\hMailServer\Data\{B5427132-3A4D-44E0-8B03-EB7A202C7CE6}.eml"
"DEBUG" 2424 "2016-08-18 10:29:00.784" "Connecting to ClamAV virus scanner..."
"SMTPD" 2524 13262 "2016-08-18 10:29:01.066" "36.84.3.229" "RECEIVED: QUIT"
"SMTPD" 2524 13262 "2016-08-18 10:29:01.066" "36.84.3.229" "SENT: 221 goodbye"
"DEBUG" 2524 "2016-08-18 10:29:01.066" "Ending session 13262"

Re: SPAM Filter not firing on some messages

Posted: 2016-08-18 17:44
by aaronwatson
SorenR wrote:One thing I found to eliminate a lot of SPAM is a 20 second pause in OnClientConnect... Spammers really hate to wait :mrgreen:

Code: Select all

   Function Wait(sec)
      With CreateObject("WScript.Shell")
         .Run "sleep -m " & Int(sec * 1000), 0, True
      End With
   End Function

   Sub OnClientConnect(oClient)
      If (Left(oClient.IPAddress, 10) = "192.168.0.") Then Exit Sub ' Local LAN
      If (Left(oClient.IPAddress, 10) = "80.160.77.") Then Exit Sub ' ISP Backup-MX'es

      If (oClient.Port = 25) Then Wait(20)
   End Sub
Interesting, sort of like a grey list but I can certainly see the differences as well.

Re: SPAM Filter not firing on some messages

Posted: 2016-08-18 17:55
by aaronwatson
I tried running the script. Logs are reporting an error when executing the code in which case I'm not sure if it's running correctly.
"ERROR" 2512 "2016-08-18 11:50:36.316" "Script Error: Source: (null) - Error: 80070002 - Description: (null) - Line: 3 Column: 9 - Code: (null)"
It would appear the system doesn't like this line. Do you think it might help to name the object and execute it as objName.Run?
.Run "sleep -m " & Int(sec * 1000), 0, True

Re: SPAM Filter not firing on some messages

Posted: 2016-08-18 18:38
by SorenR
aaronwatson wrote:I tried running the script. Logs are reporting an error when executing the code in which case I'm not sure if it's running correctly.
"ERROR" 2512 "2016-08-18 11:50:36.316" "Script Error: Source: (null) - Error: 80070002 - Description: (null) - Line: 3 Column: 9 - Code: (null)"
It would appear the system doesn't like this line. Do you think it might help to name the object and execute it as objName.Run?
.Run "sleep -m " & Int(sec * 1000), 0, True
It's the "sleep" command...

http://ss64.com/nt/sleep.html

The log does not show ANY RBL's being queried..

Re: SPAM Filter not firing on some messages

Posted: 2016-08-18 21:04
by aaronwatson
It's the "sleep" command...
Thanks. I did a direct copy/paste of your code and it's throwing errors I can't seem to fix. (though as I understand it's basically VBScripted Tarpitting so I'll see if I can tweak it)
The log does not show ANY RBL's being queried..
Indeed, that was the original concern I was posting about. I also have Spam Assassin configured and there were no apparent checks there either that I can tell, however it doesn't seem to effect all messages. Therein lies my dilemma.

Re: SPAM Filter not firing on some messages

Posted: 2016-08-18 21:38
by SorenR
Ahem... download the windows server 2003 resource kit, the sleep.exe is included. That will fix the errors.

Re: SPAM Filter not firing on some messages

Posted: 2016-08-18 23:44
by ^DooM^
from the log it does look like hmail does not do any spam checking.

it would be good to know hMails logic on a 9002 error for DNS. can anyone oblige?

Re: SPAM Filter not firing on some messages

Posted: 2016-08-19 01:02
by mattg
I'm not certain, but I think that the timeout is 90 seconds for DNS lookups
I believe that this is hard coded in hMailsever.

I expect that any error return code, hmailserver would simply drop the request and move on...


@aaronwatson, what DNS server is your windows machine with hMailserver installed using?

Re: SPAM Filter not firing on some messages

Posted: 2016-08-19 01:18
by aaronwatson
I use our domain controllers (two available). The primary is on a virtual machine stored on the same host. Those servers run local dns and send external queries to Google. I haven't noticed failing queries for other services (eg www browsing).

The same server also runs some http proxy services.

Re: SPAM Filter not firing on some messages

Posted: 2016-08-19 01:26
by mattg
Is it possible that the DNS is caching?
I use non-caching Bind9 running on a Ubuntu VM on the same computer (also has spamassassin and Clam with Sane Security patches on this same VM)

We've similar things when someone uses say OpenDNS which is great for limiting web browsing and other web access generally, but is really useless on a mailserver

There is ONLY 9 seconds between your hMailserver request and the error return, so it is not a hmailserver timeout, but it still smells like a DNS issue.

Can you try temporarily just setting your machine to use googles DNS directly and see if that helps...

Re: SPAM Filter not firing on some messages

Posted: 2016-08-19 13:35
by aaronwatson
I'll try disabling DNS caching on the server and see if it helps. I know it might slow things down a few ms, but it will keep things fresh in case it's a cache problem.

Re: SPAM Filter not firing on some messages

Posted: 2016-08-19 14:24
by aaronwatson
Interesting -- just checked my windows server logs and the timeouts aren't registering in the event logs. There are some there, just not as many as are actually failing and none on the date of my sample.

Re: SPAM Filter not firing on some messages

Posted: 2016-08-20 00:08
by ^DooM^
ups.es doesn't publish any records, MX or A my thoughts are hmail can't test against rdns as their isn't any and passes.

Re: SPAM Filter not firing on some messages

Posted: 2016-08-20 11:56
by abgar
I have the feeling that problem discussed here is of same origin that bothers me:
viewtopic.php?f=7&t=30185

Re: SPAM Filter not firing on some messages

Posted: 2016-08-20 13:29
by RvdH
SorenR wrote:One thing I found to eliminate a lot of SPAM is a 20 second pause in OnClientConnect... Spammers really hate to wait :mrgreen:

Code: Select all

   Function Wait(sec)
      With CreateObject("WScript.Shell")
         .Run "sleep -m " & Int(sec * 1000), 0, True
      End With
   End Function

   Sub OnClientConnect(oClient)
      If (Left(oClient.IPAddress, 10) = "192.168.0.") Then Exit Sub ' Local LAN
      If (Left(oClient.IPAddress, 10) = "80.160.77.") Then Exit Sub ' ISP Backup-MX'es

      If (oClient.Port = 25) Then Wait(20)
   End Sub

Code: Select all

Sub Wait(sec)
    dim temp
    temp=timer
    do while timer-temp<sec
    loop
end Sub

Re: RE: Re: SPAM Filter not firing on some messages

Posted: 2016-08-20 13:39
by SorenR
RvdH wrote:
SorenR wrote:One thing I found to eliminate a lot of SPAM is a 20 second pause in OnClientConnect... Spammers really hate to wait :mrgreen:

Code: Select all

   Function Wait(sec)
      With CreateObject("WScript.Shell")
         .Run "sleep -m " & Int(sec * 1000), 0, True
      End With
   End Function

   Sub OnClientConnect(oClient)
      If (Left(oClient.IPAddress, 10) = "192.168.0.") Then Exit Sub ' Local LAN
      If (Left(oClient.IPAddress, 10) = "80.160.77.") Then Exit Sub ' ISP Backup-MX'es

      If (oClient.Port = 25) Then Wait(20)
   End Sub

Code: Select all

Sub Wait(sec)
    dim temp
    temp=timer
    do while timer-temp<sec
    loop
end Sub
Two things... First... I tried somehing similar and it fails at midnight. Second... Code looks unfinished... Not using variable 'sec' ?

Re: SPAM Filter not firing on some messages

Posted: 2016-08-20 14:33
by RvdH
Variable sec is used, eg:

Sub Wait(sec)
dim temp
temp=timer
do while timer-temp<sec
loop
end Sub

Seems to work for me to pause it OnClientConnect(oClient) like in your examples...i don't know about exactly on midnight but i doubt it will make a difference, eg:

Code: Select all

  Sub OnClientConnect(oClient)
      If (Left(oClient.IPAddress, 10) = "192.168.0.") Then Exit Sub
      If (Left(oClient.IPAddress, 10) = "80.160.77.") Then Exit Sub
      If (oClient.Port = 25) Then Wait(20)
  End Sub

Re: SPAM Filter not firing on some messages

Posted: 2016-08-20 15:53
by SorenR
Variation I used before, fixed to work past midnight.

Code: Select all

Function Wait(sec)
   Dim t : t = Timer
   Do While ((Timer - t) < sec) Xor (Timer < t)
   Loop
End Function
viewtopic.php?f=20&t=27952&p=173569&hil ... ht#p173569

Re: RE: Re: SPAM Filter not firing on some messages

Posted: 2016-08-20 16:31
by aaronwatson
SorenR wrote:
RvdH wrote:
SorenR wrote:One thing I found to eliminate a lot of SPAM is a 20 second pause in OnClientConnect... Spammers really hate to wait :mrgreen:

Code: Select all

   Function Wait(sec)
      With CreateObject("WScript.Shell")
         .Run "sleep -m " & Int(sec * 1000), 0, True
      End With
   End Function

   Sub OnClientConnect(oClient)
      If (Left(oClient.IPAddress, 10) = "192.168.0.") Then Exit Sub ' Local LAN
      If (Left(oClient.IPAddress, 10) = "80.160.77.") Then Exit Sub ' ISP Backup-MX'es

      If (oClient.Port = 25) Then Wait(20)
   End Sub

Code: Select all

Sub Wait(sec)
    dim temp
    temp=timer
    do while timer-temp<sec
    loop
end Sub
Two things... First... I tried somehing similar and it fails at midnight. Second... Code looks unfinished... Not using variable 'sec' ?
Your original code is working with the toolkit installed. I'll keep an eye on the logs and user feedback to see how it works for us. Thanks everyone for your advice.

One other thought; dns checks aside, why would SpamAssassin not be called on dns lookup failure?

Re: SPAM Filter not firing on some messages

Posted: 2016-08-20 16:39
by SorenR
RvdH wrote:Variable sec is used, eg:

Sub Wait(sec)
dim temp
temp=timer
do while timer-temp<sec
loop
end Sub
For some reason I missed it on my phone :oops:

Re: RE: Re: SPAM Filter not firing on some messages

Posted: 2016-08-21 00:48
by mattg
aaronwatson wrote:One other thought; dns checks aside, why would SpamAssassin not be called on dns lookup failure?
Can only be a few reasons
1. The IP address of the connection is whitelisted
2. Spam checking is not enabled for the IP range applicable to the connection
3. The mail is authenticated
4. Your SpamAssassin is unreachable (but this should be logged)
5. external download accounts have a checkbox for spam checking

I can't think of many other reasons.
What IP address did this email connect from to your hmailserver?
What is the relevant IP range?

Actually post screen shots of all of your IP ranges, and your Anti-spam Whitelist please

Re: SPAM Filter not firing on some messages

Posted: 2016-08-22 15:49
by aaronwatson
I found one mistake on my whitelist. I'll keep an eye on anything that comes through from this point forward to see how much that takes care of. I made the rookie mistake of wildcard whitelisting *rb*. You'd be surprised how many domains/addresses use that combination. Including the one in my posted sample...
Ironically enough when I asked the user who had requested that flag to give me more detail, she said she doesn't receive anything from them anymore. :oops: