Whitelist ignored?

[palinka]

I'm finally getting around to attempting to improve spam filtering by training Bayes. I read the excellent "spamassassin bootcamp" thread and one of the bits of advice was to forward all spam to a spam@domain address to be able to view all the spam getting filtered. I've done this but haven't started training Bayes yet.

Anyway, one of the addresses that habitually ended up in spam was from a hobby forum that both I and my son belong to (x replied to your post, etc). I was receiving tons of these messages and they all end up in spam. So I changed my preferences for email so I'm only notified of PMs (to cut down the clutter) and then white listed the address in spamassassin local.cf.

A few hours later I check the spam email to see what's been filtered and this address still got picked up as spam.

From the email header:

Code: Select all

X-Spam-Report: * -100 USER_IN_WHITELIST From: address is in the user's white-list * -0.0
 SPF_PASS SPF: sender matches SPF record * -0.0 T_RP_MATCHES_RCVD Envelope
 sender domain matches handover relay *      domain * -0.0 SPF_HELO_PASS
 SPF: HELO matches SPF record *  0.2 JAM_LONG_LINK BODY: Very long link in
 mail, possibly filled up with *      random words by bulk mailer *  0.5
 JAM_DO_STH_HERE BODY: Body contains Click/Order/Press... Here *  0.0
 HTML_MESSAGE BODY: HTML included in message *  0.5 JAM_LARGE_FONT_SIZE
 RAW: Body of mail contains parts with very large *       font *  0.1
 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily *     
 valid *  0.0 T_DKIM_INVALID DKIM-Signature header exists but is not valid
 *  0.1 FROM_EXCESS_BASE64 From: base64 encoded unnecessarily *  0.0
 MSGID_FROM_MTA_HEADER Message-Id was added by a relay *

How did this happen?

[jimimaseye]

Let's see your settings too. run this and post the results: viewtopic.php?f=20&t=30914. And what was the spamassassin settings?

[palinka]

Code: Select all

2018-07-08   Hmailserver: 5.6.7-B2425.16

DOMAINS

   "Domain1.com" - 12x.dyxx.com                   Enabled: True

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: False   
                   Max message size:        0                      Plus addressing: False
                   Max size of accounts:    0                    
                                                                   Greylisting:     False

   "Domain2.com" - djxxxxx.dyxx.com               Enabled: True

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: False   
                   Max message size:        0                      Plus addressing: False
                   Max size of accounts:    0                    
                                                                   Greylisting:     False

   "Domain3.com" - lixxx.dyxx.net                 Enabled: True

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: True    
                   Max message size:        0   Header:   Relaxed  Plus addressing: False
                   Max size of accounts:    0   Body:     Relaxed
                                                Algorithm: SHA256  Greylisting:     False
                                                Private key: c:\xampp\apache\conf\altcerts\dynu.net\mail._domainkey.Domain3.com.pem
                                                Selector:    mail

   "Domain4.com" - pixxxx.us                      Enabled: True

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: True    
                   Max message size:        0   Header:   Relaxed  Plus addressing: False
                   Max size of accounts:    0   Body:     Relaxed
                                                Algorithm: SHA256  Greylisting:     False
                                                Private key: c:\xampp\apache\conf\altcerts\Domain4.com\mail.Domain4.com.pem
                                                Selector:    mail

   "Domain5.com" - rgxxxxx.com                    Enabled: True

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: True    
                   Max message size:        0   Header:   Relaxed  Plus addressing: False
                   Max size of accounts:    0   Body:     Relaxed
                                                Algorithm: SHA256  Greylisting:     False
                                                Private key: c:\xampp\apache\conf\altcerts\Domain5.com\mail.Domain5.com.pem
                                                Selector:    mail

   "Domain6.com" - spxx.dyxx.net                  Enabled: True

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: False   
                   Max message size:        0                      Plus addressing: False
                   Max size of accounts:    0                    
                                                                   Greylisting:     False

   "Domain7.com" - wax.dyxx.net                   Enabled: True

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: True    
                   Max message size:        0   Header:   Relaxed  Plus addressing: False
                   Max size of accounts:    0   Body:     Relaxed
                                                Algorithm: SHA256  Greylisting:     False
                                                Private key: c:\xampp\apache\conf\altcerts\dynu.net\mail._domainkey.Domain7.com.pem
                                                Selector:    mail
-----------------------------------------------------------------------------------------------

RULES
  1, Spam to Spam Folder          Criteria:  Use AND
     Custom: X-hMailServer-Spam        Equals          YES
     Custom: X-hMailServer-LoopCount   Less Than       1
                                  -----Actions-----
             Move To Folder                            Spam
             Forward                                   spam@Domain6.com
-----------------------------------------------------------------------------------------------

IP RANGES

IP: 127.0.0.1 - 127.0.0.1     Priority: 25     Name: My computer

  Allow connections                         Other
     SMTP:   True                              Antispam :   True
     POP3:  False                              Antivirus:   True
     IMAP:   True                              SSL/TLS:    False

  Allow Deliveries from                     Require Authentication from
     Local To Local       -  True              Local To Local       - False
     Local To External    -  True              Local To External    - False
     External To Local    -  True              External To Local    - False
     External To External -  True              External To External -  True


IP: 0.0.0.0 - 255.255.255.255     Priority: 10     Name: Internet

  Allow connections                         Other
     SMTP:   True                              Antispam :   True
     POP3:   True                              Antivirus:   True
     IMAP:   True                              SSL/TLS:     True

  Allow Deliveries from                     Require Authentication from
     Local To Local       -  True              Local To Local       -  True
     Local To External    -  True              Local To External    -  True
     External To Local    -  True              External To Local    - False
     External To External - False           


------------------------------------------------------
AUTOBANNED Local Addresses:
    No entries

-----------------------------------------------------------------------------------------------

AUTOBAN
  Autoban Enabled: True       Max invalid logon attempts:      3
                              Minutes Before Reset:           30  (0.50 hours, 0.02 days)
                              Minutes to Autoban:             60  (1.00 hours, 0.04 days)

No problems were found in the IP range configuration.
-----------------------------------------------------------------------------------------------

INCOMING RELAYS
   No entries
-----------------------------------------------------------------------------------------------

MIRRORING         Disabled
-----------------------------------------------------------------------------------------------

PROTOCOLS

SMTP
GENERAL             DELIVERY                  RFC COMPLIANCE            ADVANCED
No. Connections:  0  No Retries:  4 Mins: 60   Plain Text:        False  Bind: 
                     Host: Domain7.com         Empty sender:       True  Batch recipients:   100
Max Msg Size: 20480  Relay:-                   Incorrect endings:  True  Use STARTTLS:      True
                      EXTERNAL.TLD  (ok)       Disc. on invalid:  False  Delivered-To hdr: False
                     Port: 587                                           Loop limit:           5
                     Req Auth: True *User Entered*                       Recipient hosts:     15
                     Con. Sec.: StartTLS Required
  Routes:
     No routes defined.

POP3
  No. Connections: 0

IMAP
 GENERAL                   PUBLIC FOLDERS                    ADVANCED
  No. Connections:   0      Public folder name: #Public       IMAP sort:  True
                                                              IMAP Quota: True
                                                              IMAP Idle:  True
                                                              IMAP ACL:   True
                                                              Delim: "."
-----------------------------------------------------------------------------------------------

ANTISPAM

GENERAL                              SPAM TESTS              Score   SPAMASSASSIN
  Spam Mark:                  4       Use SPF:            True - 3    Use Spamassassin:    True
  Add X-HmailServer-Spam:     True    Check HELO host:    True - 2    Hostname:       127.0.0.1
  Add X-HmailServer-Reason:   True    Check MX records:   True - 2    Port:                 783
  Add X-HmailServer-Subject:  True    Verify DKIM:        True - 5    Use SA score:        True
              Subject Text: "***[SPAM]***"
  Spam delete threshold: 8         Maximum message size: 1024

DNSBL ENTRIES:
                  zen.spamhaus.org      Score: 3     Result: 127.0.0.2-8|127.0.0.10-11
                    bl.spamcop.net      Score: 3     Result: 127.0.0.2

SURBL ENTRIES:
                   multi.surbl.org      Score: 3

GREYLISTING:
  Greylisting:  False

WHITELISTING
   No entries
-----------------------------------------------------------------------------------------------

ANTIVIRUS

GENERAL:
  When found - Delete email. Notify Sender: True,  Notify Receiver: True

  Max Message Size: 26214
     CLAM AV:   True       Hostname: localhost    Port: 3310
     CLAMWIN:   False
     CUSTOMAV:  False

  Block Attachments: True
               *.bat             Batch processing file
               *.cmd             Command file for Windows NT
               *.com             Command
               *.cpl             Windows Control Panel extension
               *.csh             CSH script
               *.exe             Executable file
               *.inf             Setup file
               *.lnk             Windows link file
               *.msi             Windows Installer file
               *.msp             Windows Installer patch
               *.pif             Program Information file
               *.reg             Registration key
               *.scf             Windows Explorer command
               *.scr             Windows Screen saver
-----------------------------------------------------------------------------------------------

SSL CERTIFICATES
   lews-combined
       Certificate: C:\ProgramData\letsencrypt-win-simple\httpsacme-v01.api.letsencrypt.org\Domain7.com-chain.pem
       Private key: C:\ProgramData\letsencrypt-win-simple\httpsacme-v01.api.letsencrypt.org\Domain7.com-key.pem
-----------------------------------------------------------------------------------------------

SSL/TLS
             SSL 3.0 :  False
             TLS 1.0 :   True
             TLS 1.1 :   True
             TLS 1.2 :   True                Verify Remote SSL/TLS Certs:  False
SslCipherList  :

ECDHE-RSA-AES128-GCM-SHA256     - ECDHE-ECDSA-AES128-GCM-SHA256   - ECDHE-RSA-AES256-GCM-SHA384     
ECDHE-ECDSA-AES256-GCM-SHA384   - DHE-RSA-AES128-GCM-SHA256       - DHE-DSS-AES128-GCM-SHA256       
kEDH+AESGCM                     - ECDHE-RSA-AES128-SHA256         - ECDHE-ECDSA-AES128-SHA256       
ECDHE-RSA-AES128-SHA            - ECDHE-ECDSA-AES128-SHA          - ECDHE-RSA-AES256-SHA384         
ECDHE-ECDSA-AES256-SHA384       - ECDHE-RSA-AES256-SHA            - ECDHE-ECDSA-AES256-SHA          
DHE-RSA-AES128-SHA256           - DHE-RSA-AES128-SHA              - DHE-DSS-AES128-SHA256           
DHE-RSA-AES256-SHA256           - DHE-DSS-AES256-SHA              - DHE-RSA-AES256-SHA              
AES128-GCM-SHA256               - AES256-GCM-SHA384               - ECDHE-RSA-RC4-SHA               
ECDHE-ECDSA-RC4-SHA             - AES128                          - AES256                          
RC4-SHA                         - HIGH                            - !aNULL                          
!eNULL                          - !EXPORT                         - !DES                            
!3DES                           - !MD5                            - !PSK;                           
-----------------------------------------------------------------------------------------------

TCPIP PORTS                                         Connection Sec
               0.0.0.0         / 25    / SMTP   -   StartTLS Optional   Cert: lews-combined
               0.0.0.0         / 110   / POP3   -   StartTLS Optional   Cert: lews-combined
               0.0.0.0         / 143   / IMAP   -   StartTLS Optional   Cert: lews-combined
               0.0.0.0         / 465   / SMTP   -   SSL/TLS             Cert: lews-combined
               0.0.0.0         / 587   / SMTP   -   StartTLS Required   Cert: lews-combined
               0.0.0.0         / 993   / IMAP   -   SSL/TLS             Cert: lews-combined
               0.0.0.0         / 995   / POP3   -   SSL/TLS             Cert: lews-combined
-----------------------------------------------------------------------------------------------

LOGGING      Logging Enabled: True

  Paths:-
    Current:  C:\Program Files (x86)\hMailServer\Logs\hmailserver_2018-07-08.log
    Error:    C:\Program Files (x86)\hMailServer\Logs\ERROR_hmailserver_2018-07-08.log
    Event:    C:\Program Files (x86)\hMailServer\Logs\hmailserver_events.log - Last Event: 2018/01/05
    Awstats:  C:\Program Files (x86)\hMailServer\Logs\hmailserver_awstats.log
                        APPLICATION -    True
                        SMTP        -    True
                        POP3        -    True
                        IMAP        -      .
                        TCPIP       -    True
                        DEBUG       -    True
                        AWSTATS     -    True
-----------------------------------------------------------------------------------------------

SYSTEM TESTS

Database type: MySQL

IPv6 support is available in operating system.

Backup directory X:\HMS BACKUP is writable.

Relative message paths are stored in the database for all messages.

-----------------------------------------------------------------------------------------------

HMAILSERVER.INI

[Directories]
Program folder:  C:\Program Files (x86)\hMailServer\
Database folder: 
Data folder:     X:\HMS DATA\Data
Log folder:      C:\Program Files (x86)\hMailServer\Logs
Temp folder:     C:\Program Files (x86)\hMailServer\Temp
Event folder:    C:\Program Files (x86)\hMailServer\Events

[Database]
Type=              MYSQL
Username=          root
PasswordEncryption=1
Port=              3306
Server=            localhost
Internal=          0
-----------------------------------------------------------------------------------------------

Generated by HMSSettingsDiagnostics v1.92, Hmailserver Forum.


And here is C:\Program Files\JAM Software\SpamAssassin for Windows\etc\spamassassin\local.cf

Code: Select all

# This is the right place to customize your installation of SpamAssassin.
#
# See 'perldoc Mail::SpamAssassin::Conf' for details of what can be
# tweaked.
#
# Only a small subset of options are listed below
#
###########################################################################

#   Add *****SPAM***** to the Subject header of spam e-mails
#
# rewrite_header Subject *****SPAM*****


#   Save spam messages as a message/rfc822 MIME attachment instead of
#   modifying the original message (0: off, 2: use text/plain instead)
#
# report_safe 1


#   Set which networks or hosts are considered 'trusted' by your mail
#   server (i.e. not spammers)
#
# trusted_networks 212.17.35.


#   Set file-locking method (flock is not safe over NFS, but is faster)
#
# lock_method flock


#   Set the threshold at which a message is considered spam (default: 5.0)
#
# required_score 5.0


#   Use Bayesian classifier (default: 1)
#
# use_bayes 1


#   Bayesian classifier auto-learning (default: 1)
#
# bayes_auto_learn 1


#   Set headers which may provide inappropriate cues to the Bayesian
#   classifier
#
# bayes_ignore_header X-Bogosity
# bayes_ignore_header X-Spam-Flag
# bayes_ignore_header X-Spam-Status


#   Whether to decode non- UTF-8 and non-ASCII textual parts and recode
#   them to UTF-8 before the text is given over to rules processing.
#
# normalize_charset 1

#   Some shortcircuiting, if the plugin is enabled
# 
ifplugin Mail::SpamAssassin::Plugin::Shortcircuit
#
#   default: strongly-whitelisted mails are *really* whitelisted now, if the
#   shortcircuiting plugin is active, causing early exit to save CPU load.
#   Uncomment to turn this on
#
# shortcircuit USER_IN_WHITELIST       on
# shortcircuit USER_IN_DEF_WHITELIST   on
# shortcircuit USER_IN_ALL_SPAM_TO     on
# shortcircuit SUBJECT_IN_WHITELIST    on

#   the opposite; blacklisted mails can also save CPU
#
# shortcircuit USER_IN_BLACKLIST       on
# shortcircuit USER_IN_BLACKLIST_TO    on
# shortcircuit SUBJECT_IN_BLACKLIST    on

#   if you have taken the time to correctly specify your "trusted_networks",
#   this is another good way to save CPU
#
# shortcircuit ALL_TRUSTED             on

#   and a well-trained bayes DB can save running rules, too
#
# shortcircuit BAYES_99                spam
# shortcircuit BAYES_00                ham

endif # Mail::SpamAssassin::Plugin::Shortcircuit

report_safe 0
add_header all Report _REPORT_ *
score DRUGS_ERECTILE 10

# Add *****SPAM***** to the Subject header of spam e-mails
#
rewrite_header Subject [_HITS_]

# Set the threshold at which a message is considered spam (default: 5.0)
#
required_score 3.0

loadplugin Mail::SpamAssassin::Plugin::WhiteListSubject [C:\Program Files\JAM Software\SpamAssassin for Windows\runtime\lib\Mail\SpamAssassin\Plugin\WhiteListSubject.pm]

header SUBJECT_IN_WHITELIST eval:check_subject_in_whitelist()
describe SUBJECT_IN_WHITELIST Subject header is in user's white-list

header SUBJECT_IN_BLACKLIST eval:check_subject_in_blacklist()
describe SUBJECT_IN_BLACKLIST Subject header is in user's black-list

score SUBJECT_IN_WHITELIST -100
score SUBJECT_IN_BLACKLIST 100

whitelist_subject [redacted] CV Response
blacklist_subject Make Money Fast

# whitelist +[redacted]@tmomail.net:
whitelist_from  +[redacted]@tmomail.net
whitelist_from notifications@sslforfree.com
whitelist_from tims@stripersonline.com

After pasting this here, I noticed the bit about shortcircuiting. Should I unhash that?

[jimimaseye]

In the headers that you posted you didnt show whether there was a "X-HmailServer-Spam" header. I presume there is and it says "yes". (Consequently this is what made it to the Spam folder courtesy of your spam rule). Also the "X-HmailServer-Reason" and "X-HmailServer-Subject"?

What did the "X-Spam-Status" header say (as appears just before the spamassassin report you posted)?

As for the SHORTCIRCUIT question:
I use the SHORTCIRCUIT feature. Yes you could enable it (by removing the # ) against the plugin initialisation and the relevant 'shortcircuit whitelist' lines.

You could also, instead, whitelist addresses directly in Hmailserver (whitelist section) whivh will stop spamassassin and the builtin HMS checks altogether.

Unrelated:
I do note that you once had a 'trusted networks' line in there with a partial IP address but is now rem'd. Are you aware/was this intentional? (Or is it as supplied by Jam?)

[mattg]

If you whitelist in SpamAssassin, that reduces the SpamAssassin score by 100, typically to something like -99 or -98

As the required_score in your local CF is 5, a score of -98 or -99 are both well below the required_score value, and so this message will be seen as clean by SpamAssassin, and so will report 'no spam' to hMailserver, ie a 0 value

hMailserver gets all of the anti-spam score and adds them up.
hMailserver uses the Spamassassin score of 0 in the calculation

The message would only need to score 4 in the other hMailserver tests to be flagged as spam, That's just a DKIM failure on your set-up.

Many mailing lists don't handle DKIM very well, and so a fail of DKIM is likely for a hobbist mailing list.

One suggestion that I would make is to change your 'required_score' in SpamAssassin to negative 200 (-200), and then all scores above that value will be handed back to hMailserver. In this instance a -98 would keep this message out of your hmailserver spam score area

[palinka]

In the headers that you posted you didnt show whether there was a "X-HmailServer-Spam" header. I presume there is and it says "yes". (Consequently this is what made it to the Spam folder courtesy of your spam rule). Also the "X-HmailServer-Reason" and "X-HmailServer-Subject"?

What did the "X-Spam-Status" header say (as appears just before the spamassassin report you posted)?

Here is the entire header, which I did not look at in its entirety before.

Code: Select all

Return-Path: tims@stripersonline.com
X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on BrianServer
X-Spam-Level: 
X-Spam-Status: No, score=-98.6 required=3.0 tests=DKIM_SIGNED, FROM_EXCESS_BASE64,HTML_MESSAGE,JAM_DO_STH_HERE,JAM_LARGE_FONT_SIZE,
 JAM_LONG_LINK,MSGID_FROM_MTA_HEADER,SPF_HELO_PASS,SPF_PASS,T_DKIM_INVALID,
 T_RP_MATCHES_RCVD,USER_IN_WHITELIST autolearn=no autolearn_force=no
 version=3.4.1
X-Spam-Report: * -100 USER_IN_WHITELIST From: address is in the user's white-list * -0.0
 SPF_PASS SPF: sender matches SPF record * -0.0 T_RP_MATCHES_RCVD Envelope
 sender domain matches handover relay *      domain * -0.0 SPF_HELO_PASS
 SPF: HELO matches SPF record *  0.2 JAM_LONG_LINK BODY: Very long link in
 mail, possibly filled up with *      random words by bulk mailer *  0.5
 JAM_DO_STH_HERE BODY: Body contains Click/Order/Press... Here *  0.0
 HTML_MESSAGE BODY: HTML included in message *  0.5 JAM_LARGE_FONT_SIZE
 RAW: Body of mail contains parts with very large *       font *  0.1
 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily *     
 valid *  0.0 T_DKIM_INVALID DKIM-Signature header exists but is not valid
 *  0.1 FROM_EXCESS_BASE64 From: base64 encoded unnecessarily *  0.0
 MSGID_FROM_MTA_HEADER Message-Id was added by a relay *
Received: from host.stripersonline.com (stripersonline.com [72.52.250.24]) by redacted.tld
 with ESMTPS (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256) ;
 Sat, 7 Jul 2018 16:31:50 -0400
Message-ID: <B261A853-0630-4B6A-BBAD-23EF8A41EA8E@redacted.tld>
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=stripersonline.com; s=default;
 h=Content-Transfer-Encoding:Content-Type: Date:Subject:From:To:MIME-Version:Sender:Reply-To:Message-ID:Cc:Content-ID:
 Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc
 :Resent-Message-ID:In-Reply-To:References:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive; bh=o1BDZbGNS+EwQxhjLAGw5wj1+hyYG0uMu02BARF3ad8=;
 b=dzZG6d/1QdlyK30Sc0UM0jAFAH
 xjTHxR3eFWVTMzZrfP7VIEN9NdEPtFFXdLq7L4w0EeUnCqV6Bdj6ExykF77l/BhI5RrnGtuPLxTpM
 MvLW09Ta94u0L7YrqXfEkca61+IfsIeO2uYIDy80pk+u7bycQcrkptC6CuDinn0iEPFiX+5aJ7rNg
 3SVFBWbO8QLYZVEdXnenOM7WmLaxsmRJQ74+jE5KTGZOzANcrz8YXhkWdJVHfeNOyw4AnO0vxQcHN
 yW9MYAOE/Tg1CoPCVbiv1IgTpJs9sef+h0QK25swt5q1mg5468ueZVgNk9GxyPL6EjJkCfWHMWARM
 gdjFGJEQ==;
Received: from [::1] (port=51236 helo=localhost) by host.stripersonline.com with esmtp
 (Exim 4.91) (envelope-from <tims@stripersonline.com>) id 1fbtrw-0005Ac-ED
 for eddie@redacted.tld; Sat, 07 Jul 2018 16:31:48 -0400
MIME-Version: 1.0
Auto-Submitted: auto-generated
To: eddie@redacted.tld
From: =?UTF-8?B?U3VyZlRhbGs=?= <tims@stripersonline.com>
Subject: ***[SPAM]*** Free Parking
Date: Sat, 07 Jul 2018 20:31:48 +0000
Precedence: list
Content-Type: multipart/alternative; boundary="--==_mimepart_eb19891c9b55c8c9b28a3093055d07d4";
 charset=UTF-8
Content-Transfer-Encoding: 8bit
X-AntiAbuse: This header was added to track abuse, please include it with any abuse
 report
X-AntiAbuse: Primary Hostname - host.stripersonline.com
X-AntiAbuse: Original Domain - redacted.tld
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - stripersonline.com
X-Get-Message-Sender-Via: host.stripersonline.com: acl_c_authenticated_local_user: stripers
X-Authenticated-Sender: host.stripersonline.com: stripers
X-Source: 
X-Source-Args: 
X-Source-Dir: 
X-hMailServer-Spam: YES
X-hMailServer-Reason-1: Rejected by DKIM. - (Score: 5)
X-hMailServer-Reason-Score: 5
oclient: 72.52.250.24##25
X-hMailServer-LoopCount: 1

What I don't understand is why HMS overrides the -98.6 SA score and labels it spam by virtue of one test only.

You could also, instead, whitelist addresses directly in Hmailserver (whitelist section) whivh will stop spamassassin and the builtin HMS checks altogether.

OK, I will do that + keep the SA whitelist.

Unrelated:
I do note that you once had a 'trusted networks' line in there with a partial IP address but is now rem'd. Are you aware/was this intentional? (Or is it as supplied by Jam?)

I never saw it before. It must be a Jam artifact.

[palinka]

If you whitelist in SpamAssassin, that reduces the SpamAssassin score by 100, typically to something like -99 or -98

As the required_score in your local CF is 5, a score of -98 or -99 are both well below the required_score value, and so this message will be seen as clean by SpamAssassin, and so will report 'no spam' to hMailserver, ie a 0 value

hMailserver gets all of the anti-spam score and adds them up.
hMailserver uses the Spamassassin score of 0 in the calculation

Understood. I get it now. I keep forgetting that SA and HMS are not actually integrated, so the 0 "no-spam" score makes sense in that context.

Many mailing lists don't handle DKIM very well, and so a fail of DKIM is likely for a hobbist mailing list.

It is a hobby site run by the same hobbyists.

One suggestion that I would make is to change your 'required_score' in SpamAssassin to negative 200 (-200), and then all scores above that value will be handed back to hMailserver. In this instance a -98 would keep this message out of your hmailserver spam score area

OK I'm trying to wrap my head around that one. If required_score is -200, and my whitelist score is -98, and regular scores above 0, won't all messages be scored as spam since all scores are higher than -200?

[mattg]

OK I'm trying to wrap my head around that one. If required_score is -200, and my whitelist score is -98, and regular scores above 0, won't all messages be scored as spam since all scores are higher than -200?

Yes

And your hmailserver will use the Spamassassin score (as per your hmailserver settings) - for instance...

If an email scores 5 or higher in SPamAssassin, that score will be sent to hMialserver and be immediately marked as spam (as has been happening to you)

A score between 0 and 5 in SpamAssassin, will send that score to hMailserver to be added to the other hMailserver scores, so that a SpamAssassin score of 3.1 will mean that (rounded down to) 3 will be passed to hMailserver. If hMailserver also has another 2 point Spam test that is failed, this will be marked as spam by hmailserver.
If no other hmailserver tests score then a total score of 3 won't trigger your spam marking in hMailserver - message will NOT be marked as Spam

A negative score in SpamAssassin, will reduce the hMailserver total SPAM score.
A -100 in SpamAssassin, will send -100 to hMailserver, it is therefore very unlikely to ever become marked as spam by hMailserver in your set up.




Another thought, is that you could whitelist the mailing list sending IP or sending email address in hMailserver, and never score any mail from them. That would be using the hMailserver whitelist, not the spamAssassin whitelist.

[palinka]

A negative score in SpamAssassin, will reduce the hMailserver total SPAM score.
A -100 in SpamAssassin, will send -100 to hMailserver, it is therefore very unlikely to ever become marked as spam by hMailserver in your set up.

This part has me hung up because in the example email above, SA sent a score of -98.6 and then HMS transformed that into a 0.

Code: Select all

X-Spam-Status: No, score=-98.6 required=3.0 ...
------
X-Spam-Report: * -100 USER_IN_WHITELIST From: address is in the user's white-list * -0.0
------
X-hMailServer-Spam: YES
X-hMailServer-Reason-1: Rejected by DKIM. - (Score: 5)
X-hMailServer-Reason-Score: 5

[mattg]

That's correct because the required_score in your local CF is 5, so the message is marked as clean, and given a score of 0

[palinka]

Jeez this is becoming "Who's on first".

I am definitely missing a key ingredient because it still appears you're contradicting yourself (which I'm sure you're not).

I can't reconcile this (HMS spam threshold = 4:

* SA sends a negative score to HMS, which HMS interprets as 0, then adds HMS test scores for overall score
Example: SA score of -98 becomes 0 + HMS 5 for dkim fail = 5 (spam confirmed)

* SA sends a negative score to HMS, which HMS then adds to HMS test scores for overall score
Example: SA score of -98 + HMS 5 for dkim fail = -93 (ham confirmed)

What am I missing? They can't both be right. At least not for the same message.

[mattg]

the required_score in your local CF


If that is set to 5, anything lower than a 5 will pass a 0 to hmailserver, including a negative value will pass as 0

If that is set to -200, anything lower than or equal to -200 will pass as 0, but anything higher (eg a -98) will pass as actual SpamAssassin score

[jimimaseye]

Matt explains it well and clears up the scoring system: ie, SA -ve score = HMS zero score (with your current score settings.)

I see your antispam settings are close to those I use in the Spamassassin setup thread with the one difference of you choosing to use SA Score (= yes) and this is where you fell in to the trap (before matt explaining it to you).

However, to avoid the the complication of setting an "SA required = -200" requirement and all emails being 'seen as spam' by SA (and thus make using its auto-learn bayes feature problematic), I (and suggest, you) just simply use the Hmailserver WHITELIST function instead of spamassassin. This will avoid changing your existing settings. It makes sense to otherwise you are effectively saying 'I use 2 antispam systems but only whitelist in one of them'.

I do use whitelist in SA but only for addresses that get artificially caught by SA as spam due to rules that are not MX, SPF and DKIM checks (or any of the other) that HMS has been set up to check for itself. For example, emails that come in with "Dear Sir" and the word "account" in the email will fall foul of my SA (bespoke) rules - and so genuine emails will need whitelisting in SA only (I have 1 sender that matches this scenario).

[palinka]

OK now it all makes sense. Thanks, both of you.

Last question. Here I am presented with 2 scenarios: change required_score or simply use HMS whitelist. I'm sure there are pros and cons to each.

I suppose in either case, I can disable short circuiting. I assume short circuiting presents a null score to HMS.

I noticed in my spam trap several bona fide receipt emails. I haven't looked into the causes yet but if they are due to dkim/SPF issues such as the example email, wouldn't reducing required_score amplify those things and cause those messages to be even more likely to be marked spam? Such a balancing act for a few ham messages, but they are important ones and I can't whitelist those for other users because I don't know they exist until they come through and get marked as spam.

Then again, three are plenty of spammy messages (bulk marketing stuff that's not malicious spam) that get through. I guess I'll try lowering required_score for a week and see what happens. All my users are family, so notifying them is easy enough if I see ham slip through.

[jimimaseye]

You have to decide on what systems you want to use. There are 2:

a, spamassassin (and its 'required score' on what it decides IT thinks is spam) and
b, HMS (its 'spam mark score' and what IT thinks as spam). HMS is using its own determinations and including the opinion of SA (a).

SA 'required_score' purely sets its threshold - exceeding it simply feeds back to HMs its opinion. HMS then does further determinations which ultimately will then be matched against the HMS 'mark as spam' figure.

My advice: Leaves SA required_score as 3 and leave HMS 'mark as spam' score at 5. I would also change 'use SA score' in SA to "no" and just use a 'set score = 5' instead.
Why? Any SA scoring above SA 3 will for sure be spam 97% of the time (and thus hit HMS 'spam mark score of 5). Anything that doesnt hit SA scoring can still be subject to HMS double checking (such as your with your DKIM, SPF, DNSBL and SURBL checks. Its workth noting that DKIM and SPF scores in SA are more or less Zero as such checks cannot be relied upon and wont trigger spam marking).

[palinka]

Why? Any SA scoring above SA 3 will for sure be spam 97% of the time (and thus hit HMS 'spam mark score of 5). Anything that doesnt hit SA scoring can still be subject to HMS double checking (such as your with your DKIM, SPF, DNSBL and SURBL checks. Its workth noting that DKIM and SPF scores in SA are more or less Zero as such checks cannot be relied upon and wont trigger spam marking).

That sounds a lot like an argument to untick HMS spam checking options and using SA exclusively.

Edit - PS - I went through all the false positives in my spam trap and ONLY the example message above had dkim/SPF/etc issues. All the other false positives got trapped for the usual bulk mail reasons by SA - "try it", money related, very small text, etc.

[jimimaseye]

Indeed.

Other Mail systems dont have such a choice so you are blessed to have this choice. (They might, for example, only rely on Spamassassin or another single product for doing its antispam). Some people wont/dont use Spamassassin and rely only on internal functions, others use only an external function and others (like myself and you....up to now) use a combination of both.

It is all down to personal preference.