process rule if header doesn't exist

Use this forum for discussions about SpamAssassin and anti-spam in general.
Post Reply
User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

process rule if header doesn't exist

Post by mattg » 2018-02-23 09:09

I run with these SPF scores

# SPF scores
score SPF_SOFTFAIL 1
score SPF_NEUTRAL 3
score SPF_PASS -1
score SPF_FAIL 6

I'd like to ONLY run those test for messages that don't have a Header 'X-hMailserver-ExternalAccount'
(I think that SpamAssassin incorrectly scores SPF for mail downloaded via External Account Download, ie much mail from the external download is scored incorrectly as an SPF fail)

Any ideas on how I could go about that. Searching SpamAssassin documentation and Google haven't led me very far yet.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: process rule if header doesn't exist

Post by jimimaseye » 2018-02-23 09:34

mattg wrote: (I think that SpamAssassin incorrectly scores SPF for mail downloaded via External Account Download, ie much mail from the external download is scored incorrectly as an SPF fail)
This is not my experience. As you know al my email coes in by External Download and they all get scored correctly

eg from real email come in last night:

Code: Select all

X-Spam-Status: No, score=-6.8 required=3.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HTML_IMAGE_RATIO_04,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,
	RCVD_IN_HOSTKARMA_W,RCVD_IN_HOSTKARMA_WL,RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_SAFE,
	REMOVE_BEFORE_LINK,SPF_PASS,T_RP_MATCHES_RCVD shortcircuit=no autolearn=ham
	autolearn_force=no version=3.4.0
X-Spam-Report: 
	* -0.1 RCVD_IN_HOSTKARMA_W RBL: HostKarma: relay in white list (first pass)
	*      [185.82.77.249 listed in hostkarma.junkemailfilter.com]
	* -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no
	*      trust
	*      [185.82.77.249 listed in list.dnswl.org]
	* -3.0 RCVD_IN_RP_CERTIFIED RBL: Sender in ReturnPath Certified - Contact
	*      cert-sa@returnpath.net
	*      [Return Path SenderScore Certified {formerly]
	[Bonded Sender} - <http://www.senderscorecertified.com>]
	* -2.0 RCVD_IN_RP_SAFE RBL: Sender in ReturnPath Safe - Contact
	*      safe-sa@returnpath.net
	*      [Return Path SenderScore Safe List (formerly]
	[Habeas Safelist) - <http://www.senderscorecertified.com>]
	* -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay
	*      domain
	* -0.0 SPF_PASS SPF: sender matches SPF record
	*  1.8 REMOVE_BEFORE_LINK BODY: Removal phrase right before a link
	* -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
	*      [score: 0.0000]
	*  0.6 HTML_IMAGE_RATIO_04 BODY: HTML has a low ratio of text to image area
	*  0.0 HTML_MESSAGE BODY: HTML included in message
	* -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's
	*       domain
	* -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
	*  0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily
	*      valid
	* -2.0 RCVD_IN_HOSTKARMA_WL RBL: HostKarma: unique whitelisted
another:

Code: Select all

X-Spam-Status: No, score=0.5 required=3.0 tests=BAYES_00,HTML_IMAGE_ONLY_24,
	HTML_MESSAGE,MPART_ALT_DIFF,RCVD_IN_DNSWL_NONE,SPF_PASS,T_REMOTE_IMAGE
	shortcircuit=no autolearn=no autolearn_force=no version=3.4.0
X-Spam-Report: 
	* -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no
	*      trust
	*      [62.172.143.48 listed in list.dnswl.org]
	* -0.0 SPF_PASS SPF: sender matches SPF record
	* -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
	*      [score: 0.0000]
	*  1.6 HTML_IMAGE_ONLY_24 BODY: HTML: images with 2000-2400 bytes of words
	*  0.0 HTML_MESSAGE BODY: HTML included in message
	*  0.8 MPART_ALT_DIFF BODY: HTML and text parts are different
	*  0.0 T_REMOTE_IMAGE Message contains an external image[/code

(note the " * -0.0 SPF_PASS SPF: sender matches SPF record")
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: process rule if header doesn't exist

Post by mattg » 2018-02-23 16:42

You may well be correct that the SPF fail is accurate...I'd still like to skip SPF testing in SpamAssassin for external account downloads
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: process rule if header doesn't exist

Post by jimimaseye » 2018-02-23 17:35

mattg wrote: I'd like to ONLY run those test for messages that don't have a Header 'X-hMailserver-ExternalAccount'
Im not sure you can at HMS level maybe yes at SA level.

Isnt it a case that as an External Download message it will then have the "X-hMailserver-ExternalAccount" header added before then automatically being subjected to the Antispam/Virus checks? This process is in one single swoop so cannt be stopped/bypassed at HMS level.

As for SA, perhaps a META rule is needed (substituting the default SPF_FAIL) rule:

rule "__A"
HMSHEAD test for non-existance of header 'X-hMailserver-ExternalAccount'

rule "__B"
SPFFAIL the usual SPF_FAIL test

Meta RULE "BIGDADDY_SPFFAIL":

rule __A && __B
score 6

Therefore "BIGDADDY_SPFFAIL" will score if __A and __B are matches (where __A is only a match when there isnt 'X-hMailserver-ExternalAccount' header and __B is positive SPF_FAIL)

That should work.
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: process rule if header doesn't exist

Post by mattg » 2018-02-24 00:36

Thanks jimimaseye
That looks exactly like what I was after
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: process rule if header doesn't exist

Post by mattg » 2018-02-27 01:37

meta is working well thanks jimimaseye

Some tricks or tips that I picked up along the way.
Individual tests default to a score of 1 unless otherwise defined
So my new test of testing for header 'X-hMailserver-ExternalAccount' was scoring =1 for every match (My meta test used a NOT match contains the header)

When I set all of the base SPF tests to a score of 0, a heap of other tests mentioned this in logs on restart, so it seems that some of the SPF scoring was being applied multiple times to some messages, which wasn't my intent

And you still need write a description for meta rules
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: process rule if header doesn't exist

Post by jimimaseye » 2018-02-27 09:13

mattg wrote:My meta test used a NOT match contains the header
Out of curiosity, could you show the full rule please. Im intrigued to know what the actual syntax was.
mattg wrote:When I set all of the base SPF tests to a score of 0, a heap of other tests mentioned this in logs on restart
Yes, the trick is to score them something like 0.001. (Scoring Zero effectively disables them).
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: process rule if header doesn't exist

Post by mattg » 2018-02-27 11:51

Code: Select all

# Hmailserver External  Account download
header _HMS_EXTERNAL_ACCOUNT exists:X-hMailserver-ExternalAccount
score _HMS_EXTERNAL_ACCOUNT -0.3
describe _HMS_EXTERNAL_ACCOUNT Message was downloaded via a hMailserver External Account

#Check SPF + NOT HMS External Account
meta DIRECT_SPF_SOFTFAIL !_HMS_EXTERNAL_ACCOUNT && SPF_SOFTFAIL
meta DIRECT_SPF_NEUTRAL !_HMS_EXTERNAL_ACCOUNT && SPF_NEUTRAL
meta DIRECT_SPF_PASS !_HMS_EXTERNAL_ACCOUNT && SPF_PASS
meta DIRECT_SPF_FAIL !_HMS_EXTERNAL_ACCOUNT && SPF_FAIL

# SPF scores
score DIRECT_SPF_SOFTFAIL 1
score DIRECT_SPF_NEUTRAL 3
score DIRECT_SPF_PASS -1
score DIRECT_SPF_FAIL 6

score SPF_SOFTFAIL 0.1
score SPF_NEUTRAL 0.1
score SPF_PASS -0.1
score SPF_FAIL 0.1

Describe DIRECT_SPF_SOFTFAIL Message SPF softfail and NOT downloaded through External Account 
Describe DIRECT_SPF_NEUTRAL Message SPF NEUTRAL and NOT downloaded through External Account 
Describe DIRECT_SPF_PASS Message SPF pass and NOT downloaded through External Account
Describe DIRECT_SPF_FAIL Message SPF fail and NOT downloaded through External Account
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: process rule if header doesn't exist

Post by mattg » 2018-02-28 10:35

jimimaseye wrote:
2018-02-23 09:34
mattg wrote: (I think that SpamAssassin incorrectly scores SPF for mail downloaded via External Account Download, ie much mail from the external download is scored incorrectly as an SPF fail)
This is not my experience. As you know al my email coes in by External Download and they all get scored correctly
Do you run the spamd.exe service with a -L at the end?

I think that stops any lookups, (and makes SA scoring faster).

I don't run with a -L, and scoring takes on average 5 to 6 seconds
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: process rule if header doesn't exist

Post by jimimaseye » 2018-02-28 11:10

My Spamd parameters are: -l -s "c:\path to\spamd.log" --round-robin --timeout-child=85

My average SA scoring take about 1 to 2 seconds (all with external DNS and network lookups etc).
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: process rule if header doesn't exist

Post by mattg » 2018-02-28 13:18

mine are (on Ubuntu)

-u clamav -A 192.168.0.0/24 -A 10.10.10.0/24 -i -4 -s /var/log/clamav/spamd.log --round-robin --max-children 5 --timeout-child=85
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: process rule if header doesn't exist

Post by jimimaseye » 2018-02-28 13:39

I dont limit the children although the default is 5 anyway (so the inclusion of your parameter is ineffective).

Allowing for your Unix specific parameters, our parameters are more or less the same. (I run Spamd the windows version on the same box as HMS).

I suspect your ClamAv plugin/scanning will be slowing down the scanning (compared to mine).
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: process rule if header doesn't exist

Post by mattg » 2018-02-28 14:03

Yep

I also have a lot of custom SA rules

I found tonight that I still blocked some legitimate External Download mail with hMailserver's builtin SPF test, so I've now disabled that.
I don't ever remember it blocking any real direct downloaded mail anyway...

Wonder why I'm getting such an issue with SPF fails for externally downloaded mail?
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: process rule if header doesn't exist

Post by jimimaseye » 2018-02-28 14:42

Wonder why I'm getting such an issue with SPF fails for externally downloaded mail?
Dunno. For sure there must be a geniuine reason - maybe all of your sender are just hooky! :-)

Do you have the INTERNAL RELAYS set for those servers you are downloading from (or whatewver the ip address is of the SMTP server that is receiving the external email)?
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: process rule if header doesn't exist

Post by mattg » 2018-02-28 15:31

Incoming relays for that external account download server is set
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: process rule if header doesn't exist

Post by jimimaseye » 2018-02-28 16:30

Presumably you have manually checked that the SPF FAILS are actually FP's? And does SA spf checks concur with what HMs spf checks?
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: process rule if header doesn't exist

Post by jimimaseye » 2018-02-28 21:04

(Regarding your PM and the examples within it)

(As you know) the examples show that even Spamassassin failes the SPF test due to the nature of using a host server that doesnt have its addresses set in the senders SPF record. (ie ruraxxxxxxxxepsy.onmicrosoft.com not setting the servers of server-mx.com).

This isnt a failure of you using External Downloads - SA fails it now and will still fail it even if the email was received directly (not via a Download). Consequently there is no internal relay setting (within HMS) that will help this situation. Maybe you should whitelist this senders domain from tests in SA o, better, add a (meta) rule that checks for this particular domain as a sender and then reverse scores the SPF test (failure) score (with the knowledge that you know this sender will always fails SPF). This way it would avoid attempting to white list the complete 'server-mail.com' ip range in Spamassassin (thus still safely checking mail being relayed through them with all other tests).
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: process rule if header doesn't exist

Post by mattg » 2018-03-01 00:08

No, I don't think so

ruraxxxxxxxxepsy.onmicrosoft.com sends normally to an address hosted by server-mx.com
my hMailsevrer downloads the mail via external POP3 from server-mx.com

The received headers say last received was to server-mx.com, hmailserver and SpamAssassin fail SPF checks
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: process rule if header doesn't exist

Post by jimimaseye » 2018-03-01 00:52

I see.

I find it strange. I have a similar setup to this and yet i don't see SA failing SPF checks of the sending domain.

I don't mind sharing with you the full configs if you think it will help identify the crucial differences (privately of course). Maybe you could the sender to send a test email to one of my addresses as a comparison?
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: process rule if header doesn't exist

Post by mattg » 2018-03-01 00:59

This isn't just one sender (that would be easily handled using whitelist).
This also isn't all incoming mail, probably 2 in 5 through 2 in 10, so those with a real SPF record (that ends in a -all) is my guess.

I am pretty vicious with my SpamAssassin rules, so I sort of expected that it was just something that I'd done, but now that I see it in the hMailserver Spam checks too, I'm not so sure.
DO you do anti-spam testing in your external Account download? There are anti-spam and anti-virus checkboxes in the external account window
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

sprint
Normal user
Normal user
Posts: 31
Joined: 2018-01-31 01:36

Re: process rule if header doesn't exist

Post by sprint » 2018-03-01 05:39

Make sure you have server-mx.com IP listed in trusted_networks (local.cf).

I am using SpamAssassin with external download and SPF is working fine.

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: process rule if header doesn't exist

Post by mattg » 2018-03-01 09:07

There are many IPs that are server-mx.com, possibly hundreds of them. It is Australia's largest Telco.

With my new Meta rule, I have SpamAssassin working as I expect, it only scores on both NOT hmailsevrer external download, AND then SPF fail.

I do score SPF failure pretty high, so it does stand out...
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: process rule if header doesn't exist

Post by jimimaseye » 2018-03-01 09:18

mattg wrote:DO you do anti-spam testing in your external Account download? There are anti-spam and anti-virus checkboxes in the external account window
Yes, I have 'ANTISPAM' enabled in the External Download (they are very much checked for sure) ;-)

MY antispam:

Code: Select all

-----------------------------------------------------------------------------------------------

ANTISPAM

GENERAL                              SPAM TESTS              Score   SPAMASSASSIN
  Spam Mark:                  5       Use SPF:            True - 3    Use Spamassassin:    True
  Add X-HmailServer-Spam:     True    Check HELO host:    True - 2    Hostname:       127.0.0.1
  Add X-HmailServer-Reason:   True    Check MX records:   True - 2    Port:                 783
  Add X-HmailServer-Subject:  True    Verify DKIM:       False - 5    Use SA score: False -   5
              Subject Text: "[SPAM]"
  Spam delete threshold: 8         Maximum message size: 2048

GREYLISTING:
  Greylisting:  False

DNSBL ENTRIES:
                  zen.spamhaus.org      Score: 5     Result: 127.0.0.2-8|127.0.0.10-11
                    bl.spamcop.net      Score: 3     Result: 127.0.0.2
              zz.countries.nerd.dk      Score: 5     Result: 127.0.0.158|127.0.2.131|127.0.2.198
            b.barracudacentral.org      Score: 2     Result: 127.0.0.2
     hostkarma.junkemailfilter.com      Score: 2     Result: 127.0.0.2|127.0.0.4
           bl.spameatingmonkey.net      Score: 2     Result: 127.0.0.2-3
                   cbl.abuseat.org      Score: 2     Result: 127.0.0.2

SURBL ENTRIES:
                   multi.surbl.org      Score: 3
                      DBL Spamhaus      Score: 3
-----------------------------------------------------------------------------------------------
sprint wrote:
2018-03-01 05:39
Make sure you have server-mx.com IP listed in trusted_networks (local.cf).

I am using SpamAssassin with external download and SPF is working fine.
I dont have any external trusted_networks set (only internal devices).



I compared your headers to mine and I think I see the problem.

Ours:

Code: Select all

X-hMailServer-ExternalAccount: POPdaily
Received: from mailin2.hostvue.com (mailin2.hostvue.com [195.26.90.112])
	 (authenticated user=sue@mydomain.co.uk bits=0)
	 by ms7.hostvue.com (Cyrus v2.4.16-Kolab-2.4.16-1.el6) with LMTPSA
	 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256/256 verify=YES);
	 Wed, 28 Feb 2018 16:26:03 +0000
X-Sieve: CMU Sieve 2.4
Received: from mailj.sxxxx.com ([24.19.186.93])
	by mailin2.hostvue.com with esmtp (Exim 4.85)
	(envelope-from <xxxxxxxxxx@sxxxx.com>)
	id 1er4YM-0003ZR-Rs
	for sue@mydomain.co.uk; Wed, 28 Feb 2018 16:26:03 +0000
This too also receives from the sender and passes it to another server but that pass is via an authenticated connection. Consequently this one is ignored. That then leaves just 1 which will be ignored as the last receiving server and so the last unauthenticated server is


Yours:

Code: Select all

X-hMailServer-ExternalAccount: 	@cxxxxc.com
Received: 	from bne3-0001mmr.server-mail.com (bne3-0001mmr.server-mail.com [24.147.156.113])
 by bne3-0003mbs.server-mail.com (Postfix) with ESMTP
 id E87F55D05F for <xx@crxxxxc.com>; Wed, 28 Feb 2018 17:29:28 +1000 (EST)
Received: 	from bne3-0004mz.server-mail.com (bne3-0004mz.server-mail.com [24.147.156.212])
 by bne3-0001mmr.server-mail.com (Postfix) with ESMTP
 id C1DAA90002 for <xx@crxxxxc.com>; Wed, 28 Feb 2018 17:29:28 +1000 (EST)
Received: 	from AUS01-SY3-obe.outbound.protection.outlook.com ([104.47.117.121])
 by bne3-0004mz.server-mail.com with - 
 id G7VU1x0042dDu9k017VUiV; Wed, 28 Feb 2018 17:29:28 +1000
Note that the server receives the original email then passes it to 2 more servers of its own but they are not authenticated. So, when SA naturally ignores the last unauthenticated server, it is left with the most recent unauthenticated receipt still being from server-mail.com.
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

Post Reply