Avoid Backscatter and Improve Spam Defense

Use this forum for discussions about SpamAssassin and anti-spam in general.
Post Reply
andreasRu
New user
New user
Posts: 5
Joined: 2021-03-30 12:34

Avoid Backscatter and Improve Spam Defense

Post by andreasRu » 2021-04-14 12:20

Dear hMailServer Devs and Admins,

I've searched the forum and the internet for any related topic, but couldn't find any suitable thread that addresses our issue. I'm pretty new to this forum and also to hMailServer, so please excuse if I'm asking anything that might be obvious.

We are still in the process of migrating from another MTA to hMailserver and everything is working really beautifully at the moment. However, from my experience there is something that needs to be enhanced, but before bothering Martin or placing an issue on Githubs repository pages, I'd like to openly discuss it with experienced MTA admins here. I hope very much that I'm overseeing something in my configuration and hopefully somebody more experienced can tell me better.

At the moment hMailServer SPAM-Test adds a scoring and marks emails to be more or less "likely" to be a spam email in reference to the SPAM-Test validity checks. But the way it's implemented it will cause a good number of false positives and false negatives. Here is a test case. In this example I have the follwing SPAM-Test settings:
  • Use SPF: I've set score to 4 (triggers only if DNS of the senders email-domain has SPF implemented in DNS. If no SPF is implemented, the test will silently pass)
  • Check host in the HELO command: I've set score to 2 (triggers always with incorrect DNS entries to the IP reffering to the submitted HELO Command, but mail also fail for valid dial-in dynamic dns addresses in the HELO command )
  • Check sender has DNS-MX records: I've set score to 1 (triggers if a DNS-MX doesn't exists for the @domain.com address)
  • Verify DKIM-Signature header: I've set score to 8 (triggers only the senders DNS for the @domain.com address has DKIM implemented)
That is what hMailServer does with two different emails (1 with a valid outlook MSN address and 1 totally faked with my local PC). Here is the result:

Email received by hmailserver from valid outlook.com email address:

Code: Select all

Return-Path: xxxxxxxx@outlook.de
Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05olkn2023.outbound.protection.outlook.com
 [40.92.90.23]) ...
....
From: xxxxxxxx <xxxxxxxx@outlook.de>

X-hMailServer-Reason-1: The host name specified in HELO does not match IP address. - (Score: 2)
X-hMailServer-Reason-Score: 2
The above email fails, because outlooks smtp server envolved is sending a wrong HELO FQDN. But that SMTP server is authorized with a valid SPF!!!

Email received by hmailserver from a local vbscript MS-CDO with an invalid fake t-online.de address:

Code: Select all

Return-Path: testtts@t-online.de
Received: from mypc (cable-xxx-0-193-xxx.nc.de [xxx.0.193.xxx]) by ...
....
From: <testtts@t-online.de>
...
X-Mailer: Microsoft CDO for Windows 2000
...
X-MimeOLE: Produced By Microsoft MimeOLE
X-hMailServer-Reason-1: The host name specified in HELO does not match IP address. - (Score: 2)
X-hMailServer-Reason-Score: 2
Despite of t-online.de refusing to implement SPF ( see https://postmaster.t-online.de/#t3.5) this email would have been caught with their Forward-confirmed reverse DNS (FcrDNS) implementation. However, in this hMailServer fails too. This is a backscatters dream!!! I can't find a way how to tell hMailServer o pass the first email because of a valid SPF record and reject the second because of missing FcrDNS authorization.

For what it's worth, we would like hMailServer to strictly accept only emails from other correctly configured MTAs in a more granular way and reject everything that hasn't at least a minimal configuration to show it's legitmacy to send for a certain @domain.com address. That is also what mayor e-mail providers do. You won't get any email passed from your MTA to their MTA (and the accounts hosted there) unless you have a minimal configuration that shows you are a legimit to send on behalf of @domain.com. For that type of SPAM-Defense hMailServer doesn't get deep enough.

Instead of adding SPAM scores, an alternative would be to add deeper spam-test and add headers after SPAM-Tests in a "OR-condition" manner. Every condition check could create a X-hMailServer header accordingly:
1. ALLOW if connecting SMTP server IP is allowed by SPF to send on behalf of @domain.com as specified in the senders "From:" / "Return-Path:".
Sets header: X-hMailServer-SPFresult: 0/1 (for false/true)
OR
2. ALLOW if connecting SMTP server IP is listed as one of the valid MX IP addresses for @domain.com as specified in DNS in reference to the senders "From:" / "Return-Path:".
Sets header: X-hMailServer-MXresult: 0/1 (for false/true)
OR
3. ALLOW if connecting SMTP server PTR entry matches a valid DNS entry for @domain.com name as specified in the senders "From:" / "Return-Path:".
Sets header: X-hMailServer-FcrDNSresult: 0/1 (for false/true)
OR
4. ALLOW if DKIM is valid.
Sets header: X-hMailServer-DKIMresult: 0/1 (for false/true)

There is no SPAM scoring envolved. This check could be bypassed for users authenticating with AUTH to send their emails. This would reduce false positives and false negatives. Also, this would allow a more granular way to catch certain emails by global rules and combat SPAM and BACKSCATTER more effectively with hMailServer. Of course, there would be more overhead envolved.

We already have implemented this with our old MTA with a self programmed application that catches the email via POP3. It does all these checks in a more or less somehow dirty way, but it banned a lot of SPAM and we were able to effectively combat backscatter. However, it would be really, really nice to see hMailServer doing that and give other hMailServer admins more power to tweak their MTAs and combat spam more effectively. Please share your opinions! Thanks to all envolved in this great project!

User avatar
mattg
Moderator
Moderator
Posts: 21356
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Avoid Backscatter and Improve Spam Defense

Post by mattg » 2021-04-14 14:33

Many of use use SpamAssassin to score mail

I prefer to use ALL Spmassassin scores, and not set a value in hmailserver to a positive notification in SpamAssassin

I also accept large spam scores then silently delete them to reduce backscatter.

Some detail on your observations so far
Martin isn't around much, except for really urgent updates
andreasRu wrote:
2021-04-14 12:20
Here is a test case. In this example I have the follwing SPAM-Test settings:

Use SPF: I've set score to 4 (triggers only if DNS of the senders email-domain has SPF implemented in DNS. If no SPF is implemented, the test will silently pass)

Check host in the HELO command: I've set score to 2 (triggers always with incorrect DNS entries to the IP reffering to the submitted HELO Command, but mail also fail for valid dial-in dynamic dns addresses in the HELO command )

Check sender has DNS-MX records: I've set score to 1 (triggers if a DNS-MX doesn't exists for the @domain.com address)

Verify DKIM-Signature header: I've set score to 8 (triggers only the senders DNS for the @domain.com address has DKIM implemented)
Use SPF only triggers if there is a SPF record AND the sending IP fails to match that.
If the spf record ends in a +all, then all mail will pass
If the psf record ends in a ~all, then all mail will pass (as this is only testing spf)

In the case of Outlook.com, the way that they made the spf record uses a formula, rather than a list of IPs
We worked out recently that hMailserver was handling formula incorrectly when decoding the spf record.

This is fixed in the custom build made by RvdH >> viewtopic.php?f=10&t=30193&sid=954d5409 ... 191db02b06
It is #27


DKIM signature will also only fail where there is a DKIM record AND the message doesn't meet that record
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
SorenR
Senior user
Senior user
Posts: 4487
Joined: 2006-08-21 15:38
Location: Denmark

Re: Avoid Backscatter and Improve Spam Defense

Post by SorenR » 2021-04-14 14:55

Fighting SPAM eh...

Here's a couple of hours reading. Not all is up-to-date and ideas may change many times during a single discussion :mrgreen:

viewtopic.php?p=209541#p209541

viewtopic.php?p=175458#p175458

viewtopic.php?p=209774#p209774

I would go for blocking SnowShoe SPAM and LashBack SPAM first..

Snowshoe SPAM
DNSBL: zen.spamhaus.org
Code: 127.0.0.3

LashBack SPAM
DNSBL: ubl.unsubscore.com
Code: 127.0.0.2
SørenR.

Algorithm (noun.)
Word used by programmers when they do not want to explain what they did.

andreasRu
New user
New user
Posts: 5
Joined: 2021-03-30 12:34

Re: Avoid Backscatter and Improve Spam Defense

Post by andreasRu » 2021-04-14 16:30

That's really awesome information set together!!! Thanks for creating all that documentation and much more for sharing all of your code. I appreciate that form of contirbution really a lot. I'm just wondering why I didn't find that threads in my searches. Going to enjoy reading that all, again, thanks.

andreasRu
New user
New user
Posts: 5
Joined: 2021-03-30 12:34

Re: Avoid Backscatter and Improve Spam Defense

Post by andreasRu » 2021-04-15 12:02

I appreciate a lot all your answers.
Use SPF only triggers if there is a SPF record AND the sending IP fails to match that.
If the spf record ends in a +all, then all mail will pass
If the psf record ends in a ~all, then all mail will pass (as this is only testing spf)
We are in times where big email service providers dictate a correct SPF with "-all" set up, they ignore "+all" and "~all", so that only the specified/inludec IPs/Networks will be accepted. E.g. at MSN they dictate a correct SPF setup, they dictate PTR, and they dictate to join their SNDS and JMRP... and I even had to sign a senders liability by hand once in the past. And ONLY then they allow servers to send emails to their networks users. But still, they will monitor your servers actvitity very carefully and bann you immediately if you fail to follow their sending policy. Even if this all might be a pain, I totally understand why they do it, and I would do it just the same way to fight spam.

So my conclusion is: Looks like we need to fallback to our old solution, save the emails in a pre-process account, connect with a home grown application via POP3 and do the conditional checks then. Would have loved if hMailServer would have supported that out of the box and send an immediate "554 Reject because of policy". I'm not wining at all... I'm loving hMailServer and it's a great solution we are getting for FREE and with a very welcoming community. I also want to contribute more to this community soon in the future. Thanks!

User avatar
SorenR
Senior user
Senior user
Posts: 4487
Joined: 2006-08-21 15:38
Location: Denmark

Re: Avoid Backscatter and Improve Spam Defense

Post by SorenR » 2021-04-15 13:18

andreasRu wrote:
2021-04-15 12:02
I appreciate a lot all your answers.
Use SPF only triggers if there is a SPF record AND the sending IP fails to match that.
If the spf record ends in a +all, then all mail will pass
If the psf record ends in a ~all, then all mail will pass (as this is only testing spf)
We are in times where big email service providers dictate a correct SPF with "-all" set up, they ignore "+all" and "~all", so that only the specified/inludec IPs/Networks will be accepted. E.g. at MSN they dictate a correct SPF setup, they dictate PTR, and they dictate to join their SNDS and JMRP... and I even had to sign a senders liability by hand once in the past. And ONLY then they allow servers to send emails to their networks users. But still, they will monitor your servers actvitity very carefully and bann you immediately if you fail to follow their sending policy. Even if this all might be a pain, I totally understand why they do it, and I would do it just the same way to fight spam.

So my conclusion is: Looks like we need to fallback to our old solution, save the emails in a pre-process account, connect with a home grown application via POP3 and do the conditional checks then. Would have loved if hMailServer would have supported that out of the box and send an immediate "554 Reject because of policy". I'm not wining at all... I'm loving hMailServer and it's a great solution we are getting for FREE and with a very welcoming community. I also want to contribute more to this community soon in the future. Thanks!
I run most of the rules and scripts you see in those three links and over the past 6 month I have had maybe 10 false positives come through the server. The latest I do is block based on ASN numbers. 1 blocked ASN number means up to maybe 5 million blocked IP addresses ;-)

SPF, DKIM and DMARC only prevent impersonators.

SnowShoe, LashBack and BOT's probing your passwords are just simply annoying and should be ignored. They will NOT go away so treat them as noise.

Using POP-Fetch is fine if you do not have the drive and stamina going up against the worst scum of the earch ;-)

Most of us here are like this:
Image
SørenR.

Algorithm (noun.)
Word used by programmers when they do not want to explain what they did.

palinka
Senior user
Senior user
Posts: 2611
Joined: 2017-09-12 17:57

Re: Avoid Backscatter and Improve Spam Defense

Post by palinka » 2021-04-15 15:34

andreasRu wrote:
2021-04-15 12:02
Would have loved if hMailServer would have supported that out of the box and send an immediate "554 Reject because of policy".
What is "out of the box"? Running a well functioning email server of any flavor is never "out of the box".

"Because of policy" - what policy? Whatever policy needs to be created, massaged, tested... Even the "out of the box" hmailserver built in options.

As an example, you mentioned you score DKIM fail at 8. Hmailserver has an issue with DKIM. Its not a bug, its due to hmailserver being strictly compliant while most of the rest of the email world is non compliant. This causes a lot of false positives on DKIM. I stopped scoring DKIM altogether until recently when I started scoring DKIM fails at 1, just for curiosity's sake more than anything else.

Further reading here:
https://hmailserver.com/forum/viewtopic ... 73#p213073

By the way, spamassassin DKIM checking works better (without this 8bit issue) and is more nuanced than the sledgehammer score of 8 that you set.

Soren's scripts are fantastic. They (mostly) work right "out of the box". :D

If you're not familiar with vbs -or- simply don't have time to get over the learning curve, then it may not be for you. But if you go for it, you'll find that its as flexible and powerful as you want it to be. Plus, there's lots of support here. Your posts will get a lot of attention because you're trying - as opposed to "why not muh mailz workz?" We had one guy recently that admitted he turned his router off at night and then asked why nobody could reach his server. :roll:

User avatar
RvdH
Senior user
Senior user
Posts: 1416
Joined: 2008-06-27 14:42
Location: Netherlands

Re: Avoid Backscatter and Improve Spam Defense

Post by RvdH » 2021-04-15 17:20

SorenR wrote:
2021-04-14 14:55
LashBack SPAM
DNSBL: ubl.unsubscore.com
Code: 127.0.0.2
I just checked my spamd.logs....i had no hits for RCVD_IN_UNSUBSCORE
Strange eh, until i found this:
https://www.dnsbl.com/2021/01/status-of ... fline.html

DNS lookup for ubl.unsubscore.com returns NXDOMAIN, or is that normal?
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
SorenR
Senior user
Senior user
Posts: 4487
Joined: 2006-08-21 15:38
Location: Denmark

Re: Avoid Backscatter and Improve Spam Defense

Post by SorenR » 2021-04-15 17:59

RvdH wrote:
2021-04-15 17:20
SorenR wrote:
2021-04-14 14:55
LashBack SPAM
DNSBL: ubl.unsubscore.com
Code: 127.0.0.2
I just checked my spamd.logs....i had no hits for RCVD_IN_UNSUBSCORE
Strange eh, until i found this:
https://www.dnsbl.com/2021/01/status-of ... fline.html

DNS lookup for ubl.unsubscore.com returns NXDOMAIN, or is that normal?
Hmm... Website is live but slow.. http://blacklist.lashback.com/
SørenR.

Algorithm (noun.)
Word used by programmers when they do not want to explain what they did.

User avatar
SorenR
Senior user
Senior user
Posts: 4487
Joined: 2006-08-21 15:38
Location: Denmark

Re: Avoid Backscatter and Improve Spam Defense

Post by SorenR » 2021-04-15 20:36

Last hit in my logs with LashBack SPAM was 2020-08-31 21:39:44.843
SørenR.

Algorithm (noun.)
Word used by programmers when they do not want to explain what they did.

User avatar
RvdH
Senior user
Senior user
Posts: 1416
Joined: 2008-06-27 14:42
Location: Netherlands

Re: Avoid Backscatter and Improve Spam Defense

Post by RvdH » 2021-04-16 20:23

SorenR wrote:
2021-04-15 20:36
Last hit in my logs with LashBack SPAM was 2020-08-31 21:39:44.843
This is all i could find on the Mailop list, but it seems to be gone forever
https://www.mail-archive.com/mailop@mai ... 11607.html
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
SorenR
Senior user
Senior user
Posts: 4487
Joined: 2006-08-21 15:38
Location: Denmark

Re: Avoid Backscatter and Improve Spam Defense

Post by SorenR » 2021-04-16 20:47

RvdH wrote:
2021-04-16 20:23
SorenR wrote:
2021-04-15 20:36
Last hit in my logs with LashBack SPAM was 2020-08-31 21:39:44.843
This is all i could find on the Mailop list, but it seems to be gone forever
https://www.mail-archive.com/mailop@mai ... 11607.html
Yes, blacklist is retired.
https://www.mail-archive.com/mailop@mai ... 12312.html
SørenR.

Algorithm (noun.)
Word used by programmers when they do not want to explain what they did.

Post Reply