Page 1 of 1

Status Tab Spam count

Posted: 2018-11-19 19:33
by derelvis
Hi there,

first of all many thanks for the great work here in the forum. There is so much usefull information and help in here.
As I just set up my Spamassassin, sanesecurity and clamav combination from the tutorials (mostly by the examples of jimimaseye) and nearly everythings works like a charme I still wonder why the Spam count on Admin status tab ist not working for me. Spam gets detected and moved to the SPAM folders by rules according to the different email accounts but never show up on the status tab count (Spam messages: 0 <==> Spam folder contains 7 messages correctly marked as spam). For virus detection I can not say, seems it does not count the eicar test througt Test button?

HMS version 5.6.6-B2383
SA 3.4.1.36
Clamd 0.99.4

Any ideas these two counters don't show hits? All other count boxes seem to work as expected.
Thanks for any advice! Diagnostic log is waiting to be asked for :-)

Best regards
derelvis

Re: Status Tab Spam count

Posted: 2018-11-19 19:44
by jimimaseye
The spam count will increase if the spam check feature concludes there is a spam - ie if the score exceeds the Hmailserver spam delete threshold (not the spamassassin threshold). If spamassassin determines spam beyond 7 then your (my) rules just deletes the message.

run this and post the results: viewtopic.php?f=20&t=30914 so we can see the setup. Include rules when prompted.

Also show the headers of a message that you think should have triggered a spam count increase.

[Entered by mobile. Excuse my spelling.]

Re: Status Tab Spam count

Posted: 2018-11-19 20:07
by derelvis
Ok, so maybe I missed that part that SA score is not used there. But a HMS score of 7 should be counted, right? And I also tested with "use SA score" checked and newer so the counter hitting...
I thought that when HMS sees a message as Spam (threshold 5) it gets marked with [SPAM] and should increase the counter? So for my understanding every message with [SPAM] should increase the counter, if deleted or not?
(For testing reasons I changed deleting of SPAMS to as well only deleivering to SPAM folder and sending a copy to a special email adress for monitoring.)

Code: Select all

2018-11-19   Hmailserver: 5.6.6-B2383

DOMAINS

   "Domain1.com" - asxxxxxxxxxxxxxxxxxx.de        Enabled: True

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: True    
                   Max message size:        0   Header:   Relaxed  Plus addressing: False
                   Max size of accounts:    0   Body:     Relaxed
                                                Algorithm: SHA256  Greylisting:     False
                                                Private key: d:\hmailserver\Domain1.com\dkim-private.pem
                                                Selector:    dkim

   "Domain2.com" - asxxxxxxxxxx.inxx              Enabled: True

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: False   
                   Max message size:        0                      Plus addressing: False
                   Max size of accounts:    0                    
                                                                   Greylisting:     False

   "Domain3.com" - grxxxx.inxx                    Enabled: True

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: False   
                   Max message size:        0                      Plus addressing: False
                   Max size of accounts:    0                    
                                                                   Greylisting:     False

   "Domain4.com" - kexxxxxxxx.de                  Enabled: True

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: False   
                   Max message size:        0                      Plus addressing: False
                   Max size of accounts:    0                    
                                                                   Greylisting:     False

   "Domain5.com" - prxxxxxxxxx.inxx               Enabled: True

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: False   
                   Max message size:        0                      Plus addressing: False
                   Max size of accounts:    0                    
                                                                   Greylisting:     False
-----------------------------------------------------------------------------------------------

RULES
  1, Against rule loop            Criteria:  Use AND
     Custom: X-hMailServer-LoopCount   Greater Than    0
                                  -----Actions-----
             Stop Rule Processing
 ---------------------------------------------------------------------
  2, ExternalScore7               Criteria:  Use OR
     Custom: X-Spam-Level              Contains        *******
             Subject                   Regular Expr    (?i:^Virus found:.*$)
                                  -----Actions-----
             Move To Folder                            INBOX.SPAM
             Forward                                   spam@EXTERNAL.TLD
             Stop Rule Processing
 ---------------------------------------------------------------------
  3, SPAM                         Criteria:  Use OR
             Subject                   Contains        *****SPAM*****
             Subject                   Contains        [SPAM]
                                  -----Actions-----
             Move To Folder                            INBOX.SPAM
             Forward                                   spam@EXTERNAL.TLD
             Stop Rule Processing
-----------------------------------------------------------------------------------------------

IP RANGES

IP: 127.0.0.1 - 127.0.0.1     Priority: 15     Name: My computer

  Allow connections                         Other
     SMTP:   True                              Antispam :   True
     POP3:   True                              Antivirus:   True
     IMAP:   True                              SSL/TLS:    False

  Allow Deliveries from                     Require Authentication from
     Local To Local       -  True              Local To Local       - False
     Local To External    -  True              Local To External    - False
     External To Local    -  True              External To Local    - False
     External To External -  True              External To External -  True


IP: 0.0.0.0 - 255.255.255.255     Priority: 10     Name: Internet

  Allow connections                         Other
     SMTP:   True                              Antispam :   True
     POP3:   True                              Antivirus:   True
     IMAP:   True                              SSL/TLS:    False

  Allow Deliveries from                     Require Authentication from
     Local To Local       -  True              Local To Local       -  True
     Local To External    -  True              Local To External    -  True
     External To Local    - False     !! Inbound on Sub IP ranges or External Downloads only !! 
     External To External - False           


   !!  Warning:  DEFAULT DOMAIN is SET  !! - "Domain1.com"
------------------------------------------------------
AUTOBANNED Local Addresses:
    No entries

-----------------------------------------------------------------------------------------------

AUTOBAN
  Autoban Enabled: True       Max invalid logon attempts:      5
                              Minutes Before Reset:           30  (0,50 hours, 0,02 days)
                              Minutes to Autoban:             60  (1,00 hours, 0,04 days)

No problems were found in the IP range configuration.
-----------------------------------------------------------------------------------------------

INCOMING RELAYS
   No entries
-----------------------------------------------------------------------------------------------

MIRRORING         Disabled
-----------------------------------------------------------------------------------------------

PROTOCOLS

SMTP
GENERAL             DELIVERY                  RFC COMPLIANCE            ADVANCED
No. Connections:  0  No Retries:  5 Mins:  1   Plain Text:        False  Bind: 
                     Host: EXTERNAL.TLD        Empty sender:       True  Batch recipients:   100
Max Msg Size:102400  Relay:-                   Incorrect endings:  True  Use STARTTLS:      True
                      EXTERNAL.TLD  (ok)       Disc. on invalid:  False  Delivered-To hdr: False
                     Port:  25                                           Loop limit:           5
                     Req Auth: True *User Entered*                       Recipient hosts:     15
                     Con. Sec.: StartTLS Required
  Routes:
    Domain4.com              - S: Local   R: Remote - Addr: All         (ok)

POP3
  No. Connections: 0

IMAP
 GENERAL                   PUBLIC FOLDERS                    ADVANCED
  No. Connections:   0      Public folder name: #Public       IMAP sort:  True
                                                              IMAP Quota: True
                                                              IMAP Idle:  True
                                                              IMAP ACL:   True
                                                              Delim: "."
-----------------------------------------------------------------------------------------------

ANTISPAM

GENERAL                              SPAM TESTS              Score   SPAMASSASSIN
  Spam Mark:                  5       Use SPF:            True - 3    Use Spamassassin:    True
  Add X-HmailServer-Spam:     True    Check HELO host:    True - 2    Hostname:       127.0.0.1
  Add X-HmailServer-Reason:   True    Check MX records:   True - 2    Port:                 783
  Add X-HmailServer-Subject:  True    Verify DKIM:       False        Use SA score: Falsch -   5
              Subject Text: "[SPAM]"
  Spam delete threshold: 8         Maximum message size: 2048

DNSBL ENTRIES:
                  zen.spamhaus.org      Score: 5     Result: 127.0.0.2-8|127.0.0.10-11
                    bl.spamcop.net      Score: 3     Result: 127.0.0.2
            b.barracudacentral.org      Score: 2     Result: 127.0.0.2
     hostkarma.junkemailfilter.com      Score: 2     Result: 127.0.0.2|127.0.0.4
           bl.spameatingmonkey.net      Score: 2     Result: 127.0.0.2-3
                   cbl.abuseat.org      Score: 2     Result: 127.0.0.2

SURBL ENTRIES:
                   multi.surbl.org      Score: 3

GREYLISTING:
  Greylisting:  False

WHITELISTING
   No entries
-----------------------------------------------------------------------------------------------

ANTIVIRUS

GENERAL:
  When found - Delete Attachments.

  Max Message Size: 26214
     CLAM AV:   True       Hostname: localhost    Port: 3310
     CLAMWIN:   False
     CUSTOMAV:  False

  Block Attachments: True
               *.bat             Batch processing file
               *.cmd             Command file for Windows NT
               *.com             Command
               *.cpl             Windows Control Panel extension
               *.csh             CSH script
               *.exe             Executable file
               *.inf             Setup file
               *.lnk             Windows link file
               *.msi             Windows Installer file
               *.msp             Windows Installer patch
               *.reg             Registration key
               *.scf             Windows Explorer command
               *.scr             Windows Screen saver
-----------------------------------------------------------------------------------------------

SSL CERTIFICATES
   No entries
-----------------------------------------------------------------------------------------------

SSL/TLS
             SSL 3.0 :   True
             TLS 1.0 :   True
             TLS 1.1 :   True
             TLS 1.2 :   True                Verify Remote SSL/TLS Certs:   True
SslCipherList  :

ECDHE-RSA-AES128-GCM-SHA256     - ECDHE-ECDSA-AES128-GCM-SHA256   - ECDHE-RSA-AES256-GCM-SHA384     
ECDHE-ECDSA-AES256-GCM-SHA384   - DHE-RSA-AES128-GCM-SHA256       - DHE-DSS-AES128-GCM-SHA256       
kEDH+AESGCM                     - ECDHE-RSA-AES128-SHA256         - ECDHE-ECDSA-AES128-SHA256       
ECDHE-RSA-AES128-SHA            - ECDHE-ECDSA-AES128-SHA          - ECDHE-RSA-AES256-SHA384         
ECDHE-ECDSA-AES256-SHA384       - ECDHE-RSA-AES256-SHA            - ECDHE-ECDSA-AES256-SHA          
DHE-RSA-AES128-SHA256           - DHE-RSA-AES128-SHA              - DHE-DSS-AES128-SHA256           
DHE-RSA-AES256-SHA256           - DHE-DSS-AES256-SHA              - DHE-RSA-AES256-SHA              
AES128-GCM-SHA256               - AES256-GCM-SHA384               - ECDHE-RSA-RC4-SHA               
ECDHE-ECDSA-RC4-SHA             - AES128                          - AES256                          
RC4-SHA                         - HIGH                            - !aNULL                          
!eNULL                          - !EXPORT                         - !DES                            
!3DES                           - !MD5                            - !PSK;                           
-----------------------------------------------------------------------------------------------

TCPIP PORTS                                         Connection Sec
               0.0.0.0         / 110   / POP3   -   None                
               0.0.0.0         / 587   / SMTP   -   None                
               0.0.0.0         / 993   / IMAP   -   None                

    !! No SMTP Port 25 defined. Direct external SMTP inbound not possible !!

-----------------------------------------------------------------------------------------------

LOGGING      Logging Enabled: True

  Paths:-
    Current:  C:\Program Files (x86)\hMailServer\Logs\hmailserver_2018-11-19.log
    Error:    C:\Program Files (x86)\hMailServer\Logs\ERROR_hmailserver_2018-11-19.log
    Event:    C:\Program Files (x86)\hMailServer\Logs\hmailserver_events.log - Not present
    Awstats:  C:\Program Files (x86)\hMailServer\Logs\hmailserver_awstats.log
                        APPLICATION -      .
                        SMTP        -    True
                        POP3        -      .
                        IMAP        -      .
                        TCPIP       -      .
                        DEBUG       -    True
                        AWSTATS     -      .
-----------------------------------------------------------------------------------------------

SYSTEM TESTS

Database type: MSSQL Compact

IPv6 support is available in operating system.

Backup directory D:\hMailServer is writable.

Relative message paths are stored in the database for all messages.

-----------------------------------------------------------------------------------------------

HMAILSERVER.INI

[Directories]
Program folder:  C:\Program Files (x86)\hMailServer\
Database folder: C:\Program Files (x86)\hMailServer\Database
Data folder:     D:\hMailServer
Log folder:      C:\Program Files (x86)\hMailServer\Logs
Temp folder:     C:\Program Files (x86)\hMailServer\Temp
Event folder:    C:\Program Files (x86)\hMailServer\Events

[Database]
Type=              MSSQLCE
Username=           
PasswordEncryption=1
Port=              0
Server=             
Internal=          1
-----------------------------------------------------------------------------------------------

Generated by HMSSettingsDiagnostics v1.92, Hmailserver Forum.

I would have thought that this should have been counted:

Code: Select all

X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on demucsrv01.asm.intra
X-Spam-Flag: YES
X-Spam-Level: ***********
X-Spam-Status: Yes, score=11.2 required=3.0 tests=FREEMAIL_FORGED_REPLYTO, FROM_MISSP_MSFT,FSL_CTYPE_WIN1251,HTML_MESSAGE,LOTS_OF_MONEY,
 MIMEOLE_DIRECT_TO_MX,MIME_HTML_ONLY,MISSING_HEADERS,MISSING_MID,
 RCVD_IN_RP_RNBL,RCVD_IN_UCEPROTECT1,REPLYTO_WITHOUT_TO_CC,SPF_SOFTFAIL,
 TO_NO_BRKTS_FROM_MSSP,TO_NO_BRKTS_MSFT autolearn=no autolearn_force=no
 version=3.4.1
From: "Thomas"<info@aumresearchlaboratories.com>
Subject: [SPAM] [11.2] Da Thomas Grill
Date: Mon, 19 Nov 2018 00:41:40 -0800
MIME-Version: 1.0
Content-Type: text/html; charset="Windows-1251"
Content-Transfer-Encoding: 7bit
X-MSMail-Priority: Normal
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
X-Spam-Prev-Subject: Da Thomas Grill
X-hMailServer-Spam: YES
X-hMailServer-Reason-1: Tagged as Spam by SpamAssassin - (Score: 5)
X-hMailServer-Reason-2: Rejected by Barracudacentral. - (Score: 2)
X-hMailServer-Reason-Score: 7
Thank you very much so far!!

Re: Status Tab Spam count

Posted: 2018-11-19 22:10
by derelvis
Ok, so finaly at least the Virus counter got hit the first time about an hour ago ;-)

Re: Status Tab Spam count

Posted: 2018-11-19 22:11
by jimimaseye
The spam count increases when the spam DELETE threshold is exceeded. So the example (headers) you gave above will not contribute as the score hasnt hit your delete threshold of 8.

If you look in your log file you may see direct deliveries being attempted that get rejected at point of delivery.

Eg:
"APPLICATION" 3764 "2018-11-19 19:09:06.227" "hMailServer SpamProtection rejected RCPT (Sender: spameri@tiscali.it, IP:185.228.80.61, Reason: Rejected by cbl.abuseat.org)"
"APPLICATION" 2640 "2018-11-19 19:24:13.119" "hMailServer SpamProtection rejected RCPT (Sender: spameri@tiscali.it, IP:37.49.224.203, Reason: Rejected by cbl.abuseat.org)"
You will not see these emails because they would have hit your DELETE threshold (and been rejected as you can see in the log). These are the entries that appear in your count. (For example: the above will make my spam count as 2). Of course if you use external downloads to pull in emails you will not see such SMTPC entries in your logs but still might have emails contributing to the count.

Similarly, you get the same with the AV count:
"APPLICATION" 1516 "2018-11-19 12:08:17.061" "SMTPDeliverer - Message 442322: Message deleted (contained virus Sanesecurity.Malware.27391.UNOFFICIAL)."
"APPLICATION" 1516 "2018-11-19 12:27:17.501" "SMTPDeliverer - Message 442333: Message deleted (contained virus Sanesecurity.Malware.27391.UNOFFICIAL)."


The above entries are factual for my server today. Here is my evening report from my backup:
Backup Start: 19/11/2018 20:00:00.18

HMS Server Start Time: 2018-11-19 10:58:02
HMS Daily Spam Reject count: 2
HMS Daily Viruses Removed count: 2

Pausing Hmailserver.....
Hope this helps you understand.

Re: Status Tab Spam count

Posted: 2018-11-20 03:46
by palinka
I've been thinking about this recently also. In my backup report with spam counter, the counter has a small number, always fewer spams than I actually receive.

My delete threshold is a ridiculously high number 8000, so high it would never be triggered. I do that because I forward all spam to an account for sorting, and use rules to delete spam >8 before it reaches user mailboxes. Spam mark threshold is 3.

So theoretically my spam counter should always be 0 because no message ever reaches the hms delete threshold of 8000. And yet, the counter shows a number, which is always less than the actual number of spams which I can count in the spam forward inbox.

Coincidentally, I just looked and the counter shows 3 and there are also 3 spams in the spam forward inbox. That's the first time I've seen them match. So much for "always".

Re: Status Tab Spam count

Posted: 2018-11-20 10:36
by mattg
jimimaseye wrote:
2018-11-19 22:11
The spam count increases when the spam DELETE threshold is exceeded.
I'm not sure that is correct

I too have my spam delete score set higher than possible to achieve (1200)
I reject spam via a custom script. I probably should just delete it to stop backskatter, but then how would someone find out that their system has been compromised.

My spam count in status is typically 10% or so of messages processed, but in reality possibly 80 or 90 % of messages through my server are flagged as spam, and rejected via my custom script.

I don't really know what triggers the counter, but I don't believe that it is when the hMailserver DELETE threshold is exceeded.

Re: Status Tab Spam count

Posted: 2018-11-20 13:37
by jimimaseye
Possible.

I quoted it from my system statistics (as shown in my real-time example). Admittedly I found it strange and (like you guys) would have expected system scores exceeding the spam 'mark' threshold to have been the trigger (which of course reaching the delete threshold will already have hit).

Perhaps someone can check the code?

[Entered by mobile. Excuse my spelling.]

Re: Status Tab Spam count

Posted: 2018-11-20 14:57
by palinka
Here's something interesting. I just checked my backup log which said there were 4 spams processed yesterday but I only received 3 in my spam sorting account. Usually it's less, not more.

Re: Status Tab Spam count

Posted: 2018-11-20 16:17
by SorenR
If I read my version of the code (5.4.2) correctly, it's the number of SPAM messages where SPAM score is ABOVE delete threshold.

DOC Latest: https://www.hmailserver.com/documentati ... nce_status
Spam messages (before: Messages containing spam)

This is the total number of messages hMailServer has detected to contain spam. These messages may have been delivered to the recipients, depending on the server configuration. For example, if hMailServer is configured to drop all spam immediately, the count may be higher than the number of processed messages.
DOC v4.3: https://www.hmailserver.com/documentati ... ature_live
Messages containing spam
The number of messages rejected as spam since hMailServer last started or restart. The value is reset when you restart the server.
So when did that functionality change??

Re: Status Tab Spam count

Posted: 2018-11-20 17:10
by derelvis
First of all, thanks for the hint of activation the application log in addition to debug log and looking for reason "rejected" or "deleted".

So actually my status tab shows:
Server up since: 2018-11-18 15:23:47
Processed messages: 489
Viruses detected: 1
Spam messages: 0

Looking into the logs does not show any message being rejected so far (because of delete threshold being exceeded) => would confirm spam 0
One must have been deleted because of virus detection (does not show up in logs as I had not activated app logging at this time) => confirms virus 1
Log shows at least 10 messages marked as spam and can be found in SPAM folders as well (some by HMS, some by SA) => in my opinion should also be counted as spam (that's also what documentation says for my understanding) but not confirmed in my case because of 0

So I find this very confusing how the counter works, for me at all it does not count anything at the moment. But maybe due to lack of messages being rejected!? :roll:

But thanks again for contributing and discussing!

Re: Status Tab Spam count

Posted: 2018-11-20 19:09
by jimimaseye
SorenR wrote:
2018-11-20 16:17
If I read my version of the code (5.4.2) correctly, it's the number of SPAM messages where SPAM score is ABOVE delete threshold.
So my initial findings were not that far from this explanation. Still, it's wrong to be ABOVE the threshold as a message that equals the threshold is also deleted and should be included.

Or have i misunderstood something?

[Entered by mobile. Excuse my spelling.]

Re: Status Tab Spam count

Posted: 2019-03-10 15:31
by derelvis
I wanted to share news on this:
It seems to have been fixed during update from HMS 5.6.7-B2425 to HMS 5.6.7-B2427 (x64) as it asked for a database update from version 5601 to 5700.
After that I saw the SPAM counter working for the first time. Maybe some field or something else was missing in my database.
However, it finally seems to work for me now (SPAM count 5 already for toady).

Thanks to all who contributed here :-)

Re: Status Tab Spam count

Posted: 2019-08-17 08:24
by derelvis
@martin
Maybe you could shed some light on this?
My SPAM and VIRUS counters started working at some time when database has been updated during HMS update.
What I still don't understand is why I can't see any log entries like this:
"APPLICATION" 3764 "2018-11-19 19:09:06.227" "hMailServer SpamProtection rejected RCPT (Sender: spameri@tiscali.it, IP:185.228.80.61, Reason: Rejected by cbl.abuseat.org)"
"APPLICATION" 2640 "2018-11-19 19:24:13.119" "hMailServer SpamProtection rejected RCPT (Sender: spameri@tiscali.it, IP:37.49.224.203, Reason: Rejected by cbl.abuseat.org)"
Sometimes I have 3 or 4 SPAM counts but no single line in logs shwoing a rejected message!?
Does it not get logged any more or do I search for the worng term? (HMS 5.7.0 - B2469)

Thanks in advance!

Re: Status Tab Spam count

Posted: 2019-08-17 08:43
by mattg
What logging do you have enabled?

I think that is logged as debug, not application

Re: Status Tab Spam count

Posted: 2019-08-17 10:52
by derelvis
Everything except IMAP (but search in logfile for "rejected" gives 0 hits):

2019-08-17 10-31-52.jpg
Search for "rejected" says no hits

Re: Status Tab Spam count

Posted: 2019-08-19 17:28
by derelvis
Ok, maybe it is just like this:
HMS will then apply its own further checks. If it finds any itself another (minimum of) 2 will be added internally (making 7). If it hits 8 with its tests it will simply be deleted without a trace.
From here (the tutorial I used): https://www.hmailserver.com/forum/viewt ... 33#p174991

I had a closer look on yesterdays SPAM situation:
the one counted on the Status Tab has been delivered to the SPAM folder correctly.
There are about 8 hits shown in spamd.log (I know it is from Spamassassin) like this: "info: spamd: identified spam"
The one delivered gets a "info: spamd: identified spam (3.8/3.0) for (unknown)", all others have a very high spam score >7.8
So it really seems that there is no trace if it gets deleted (rejected at fetch) because of reaching the spam delete threshold! :-/