How do you check antispam settings?

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
Pedja
Normal user
Normal user
Posts: 39
Joined: 2010-10-30 05:39
Location: Serbia

How do you check antispam settings?

Post by Pedja » 2019-10-03 13:47

In old days when I used Mail Daemon it had very useful feature that all incoming mail stayed in incoming queue until user downloads it. That allowed admin to reapply all incoming mail processing limitless number of times again and again on all mail in queue.

That was very helpful for easy tweaking especially spam filters as after change in settings admin could simply reapply filters to existing mail.

hMailserver does not work that way. Incoming mail is processed and stored in user's queue and cannot be proccessed again. At least I do not know that is possible.

That said, I am interested in suggestion how could I test spam filters on hMailServer other than waiting for another similar spam to arrive. Can I somehow store example spam mail and trick hMailServer to check it against filters?

palinka
Senior user
Senior user
Posts: 1182
Joined: 2017-09-12 17:57

Re: How do you check antispam settings?

Post by palinka » 2019-10-03 14:20

I don't think that's possible.

What type of filters were you thinking about testing?

User avatar
mattg
Moderator
Moderator
Posts: 20216
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: How do you check antispam settings?

Post by mattg » 2019-10-04 02:06

I recheck with SpamAssassin

If it subsequent scans find spam it is moved to the spam@ account, or deleted if high

I have some custom logging included, and run on a schedule

(I have cut and pasted some of my script - so it may not work in isolation)

I do this

Code: Select all

Function SpamAssassin_ReCheck()
	'custom event
	'uses functions: CustomMonthlyLog
	'uses globals: g_sAdminPassword
	'called from:
	
	Const SACheckExclude = ""
	Const SACheckVerboseLogging = False
	Const Spam_email_Address = "spam@example.com"
	const Spam_email_name = "SPAM"
	
	Dim oApp, oDomain, oAccount, oIMAPFolder, oMessage, oTempDirectory, obMessage
	Dim i, j, k, l, m, s, dRetainDate, RetCode, SpamC, TempResult, OriginalScore, OriginalDate
	Dim RequiredScore, TempText, PreviousScore, PreviousDate, RetFullCode, RetTests, TestAgain
	
	Set oApp = CreateObject("hMailServer.Application")
	Call oApp.Authenticate("Administrator", g_sAdminPassword)
	OTempDirectory = oApp.Settings.Directories.TempDirectory & "\"
	
	dRetainDate = CDate(Now() - 50)
	SpamC = "%comspec% /C C:\hMailServer\Events\spamc.exe -d " & oApp.Settings.AntiSpam.SpamAssassinHost & " -p "
	SpamC = spamc & oApp.Settings.AntiSpam.SpamAssassinPort
	Call CustomMonthlyLog("***** START New reScan for SPAM using SpamAssassin *****", "Re-Spam")

	RetFullCode = "Error obtaining details"
	
	For i = 0 To oApp.Domains.count-1
		Set oDomain = oApp.Domains.item(i)
		For j = 0 To oDomain.Accounts.count-1
			Set oAccount = oDomain.Accounts.Item(j)
			If (InStr(SACheckExclude, oAccount.Address) = 0) And oAccount.Active Then
				Call CustomMonthlyLog("**** Testing mail for " & oAccount.address , "Re-Spam")
				For l = 0 To oAccount.IMAPFolders.Count-1
					Set oIMAPFolder = oAccount.IMAPFolders.Item(l)
					If oIMAPFolder.Messages.Count > 0 And oIMAPFolder.Name = "Inbox" Then
						For k = oIMAPFolder.Messages.Count -1 To 0 Step -1
							Set oMessage = oIMAPFolder.Messages.Item(k)
							If oMessage.InternalDate > dRetainDate  And oMessage.Flag(1) = False Then
								OriginalScore = 0.0
								OriginalDate = 0.0
								PreviousScore = 0.0
								PreviousDate = 0.0
								TestAgain = 0
	'							On Error Resume Next
								For m = 0 To oMessage.Headers.Count -1
									If oMessage.Headers.Item(m).Name = "X-example-re-spam" Then
										TestAgain = -100
										Call CustomMonthlyLog("Previously re-scanned so not tested again" , "Re-Spam")
									End If
									If oMessage.Headers.Item(m).Name = "X-Spam-Status" Then
										OriginalScore = oMessage.Headers.ItemByName("X-Spam-Status").Value
										OriginalScore = Left(OriginalScore,InStr(OriginalScore," required"))
										OriginalScore = Right(OriginalScore, Len(OriginalScore) - InStr(OriginalScore, ", score=")-7)
										OriginalScore = CDbl(OriginalScore)
										RequiredScore = OriginalScore
									End If
									If oMessage.Headers.Item(m).Name = "X-Spam-Recheck-Score" Then
										PreviousScore = oMessage.Headers.ItemByName("X-Spam-Recheck-Score").Value
										PreviousScore = CDbl(PreviousScore)
										PreviousDate = oMessage.Headers.ItemByName("X-Spam-Recheck-Date").Value
										PreviousDate = CDate(PreviousDate)
									End If
									If oMessage.Headers.Item(m).Name = "Date" Then
										OriginalDate = oMessage.Headers.ItemByName("Date").Value
			'							OriginalDate = CDate(OriginalDate)
									Else
										OriginalDate = Now()
									End if
								Next 'm
								
								On Error Goto 0
								If OriginalScore = 0.0 And PreviousScore = 0 Then TestAgain = TestAgain + 1
								If PreviousScore = 0.0 And OriginalDate < DateAdd("h",-4,Now()) Then TestAgain = TestAgain + 2
								If OriginalScore >= 0.0 And OriginalDate < DateAdd("h",-2,Now()) And PreviousScore >= 0 Then TestAgain = TestAgain + 4
								If OriginalScore >= 0.0 And OriginalDate < DateAdd("h",-2,Now()) And PreviousScore > 0 And PreviousDate < DateAdd("h",-4,Now()) Then TestAgain = TestAgain + 8
								If PreviousScore >= 0.0 And PreviousDate < DateAdd("h",-4,Now()) Then TestAgain = TestAgain + 16
								If PreviousScore < 0.0 And OriginalScore < 0 Then TestAgain = 0
								If TestAgain > 0 Then 
									TempText = " -c < " & Chr(34) & oMessage.FileName & Chr(34)
									TempResult = 0
									On Error Resume Next
									RetCode = RunCommandWithOutput(SpamC & TempText, 1, 0, otempdirectory & "message.txt", 1, 1)
									TempResult = CDbl(Left(RetCode,Len(RetCode)-8))
									On Error Goto 0
									'delete if very high
									Call CleanSubject(oMessage)
									If TempResult > 25.0 Then
										oIMAPFolder.Messages.DeleteByDBID(oMessage.ID) ' comment this line to NOT delete the message
									End If
									If TempResult > 4.0 Then
										On Error Resume Next
										TempText = " -R < " & Chr(34) & oMessage.FileName & Chr(34)
										RetFullCode = RunCommandWithOutput(SpamC & temptext, 1, 0, oTempDirectory & "message_long.txt", 1, 1)
										On Error Goto 0
'										oMessage.Copy(oApp.Settings.PublicFolders.ItemByName("spam").id)
										Set obMessage = CreateObject("hmailserver.Message")
										obMessage.HeaderValue("X_example_re_spam") = True
										obMessage.Body = oMessage.Body
										On Error Resume Next
											If oMessage.HTMLBody Is Not Null Then 
												obMessage.HTMLBody = oMessage.HTMLBody 
											End If
											TempText = oApp.Settings.Directories.TempDirectory & "\AttachmentDropFolder\" & oMessage.ID
											Call SaveAttachments(oMessage)
											For m = oMessage.Attachments.Count - 1 To 0 Step -1
												Call obMessage.Attachments.Add(TempText & "-" & oMessage.attachments.item(m).Filename)
											Next 'm
										On Error Goto 0
										obMessage.Charset = oMessage.Charset
										obMessage.Date = oMessage.Date
										obMessage.From = oMessage.From
										obMessage.Subject = oMessage.Subject
'										obMessage.To = oMessage.To
										obMessage.AddRecipient Spam_email_name, Spam_email_Address 
										obMessage.Flag(1) = true
										obMessage.Save
										oIMAPFolder.Messages.DeleteByDBID(oMessage.ID) ' comment this line to NOT delete the message
										Call LogItTest(obMessage)
									End If
									If OriginalScore = 0 Then
										oMessage.HeaderValue("X-Spam-Flag") = "YES"
										If Int(TempResult) > 1 Then
											oMessage.HeaderValue("X-Spam-Level") = Replace(Space(Int(TempResult))," ","*")
										End If
'										TempText = " -R < " & Chr(34) & oMessage.FileName & Chr(34)
'										TempText = RunCommandWithOutput(SpamC & TempText, 1, 0, oTempdirectory & "message_tests.txt", 1, 1)
										TempText = "Yes, score=" & TempResult & " required=" & RequiredScore & " tests=" & TempText
										oMessage.HeaderValue("X-Spam-Status") = TempText
									End If
									'Add headers
									TempText = InStr(RetFullCode, "---- ---------------------- --------------------------------------------------")
									oMessage.HeaderValue("X-Spam-Recheck-Date") = CDate(Now)
									oMessage.HeaderValue("X-Spam-Recheck-Score") = TempResult
									If Len(RetFullCode) > 100 Then
'										oMessage.HeaderValue("X-Spam-Report") = Right(RetFullCode,Len(RetFullCode)-InStr(RetFullCode, TempText) - 78)
									End if
									oMessage.Save
								End If
							End If
						Next
					End If
				Next
			End If
		Next	
	Next
	set oIMAPFolder = oApp.settings.Publicfolders.ItemByName("spam")
	Call CustomMonthlyLog("**** Testing mail already sent to SPAM folder" , "Re-Spam")
	If oIMAPFolder.Messages.Count > 0 Then
		For k = oIMAPFolder.Messages.Count -1 To 0 Step -1
			Set oMessage = oIMAPFolder.Messages.Item(k)
			If oMessage.InternalDate < dRetainDate Then
				oIMAPFolder.Messages.DeleteByDBID(oMessage.ID) ' comment this line to NOT delete the message
			Else
				If Left(oMessage.Subject,31) <> "[Resent by spam re-test script]" Then
					TestAgain = 0
					On Error Goto 0
					If OriginalScore = 0 And PreviousScore = 0 Then TestAgain = TestAgain + 1
					If PreviousScore = 0 And OriginalDate < DateAdd("h",-4,Now()) Then TestAgain = TestAgain + 2
					If OriginalScore >= 0 And OriginalDate < DateAdd("h",-2,Now()) And PreviousScore >= 0 Then TestAgain = TestAgain + 4
					If OriginalScore >= 0 And OriginalDate < DateAdd("h",-2,Now()) And PreviousScore > 0 And PreviousDate < DateAdd("h",-4,Now()) Then TestAgain = TestAgain + 8
					If PreviousScore >= 0 And PreviousDate < DateAdd("h",-4,Now()) Then TestAgain = TestAgain + 16
					If PreviousScore < 0 And OriginalScore < 0 Then TestAgain = 0
					If TestAgain > 0 Then 
						TempText = " -c < " & Chr(34) & oMessage.FileName & Chr(34)
						TempResult = 0
						TempText = Left(oMessage.Subject & Space(40),40)
						On Error Resume Next
						RetCode = RunCommandWithOutput(SpamC & TempText, 1, 0, otempdirectory & "message.txt", 1, 1)
						TempResult = CDbl(Left(RetCode,Len(RetCode)-8))
						TempText = TempText & "  " & Left(oMessage.Headers.ItemByName("From").Value & Space(55),55)
						On Error Goto 0
						If TempResult > 25.0 Then
							oIMAPFolder.Messages.DeleteByDBID(oMessage.ID) ' comment this line to NOT delete the message
						End If
						If TempResult < -95 Then
							Call CustomMonthlyLog("**** Whitelisted Message Found", "Re-Spam")
							Set obMessage = CreateObject("hmailserver.Message")
							obMessage.Body = oMessage.Body
							On Error Resume Next
								If oMessage.HTMLBody Is Not Null Then 
									obMessage.HTMLBody = oMessage.HTMLBody 
								End If
								TempText = oApp.Settings.Directories.TempDirectory & "\AttachmentDropFolder\" & oMessage.ID
								Call SaveAttachments(oMessage)
								For l = oMessage.Attachments.Count - 1 To 0 Step -1
									Call obMessage.Attachments.Add(TempText & "-" & oMessage.attachments.item(l).Filename)
								Next 'l
'							On Error Goto 0
							obMessage.Charset = oMessage.Charset
							obMessage.Date = oMessage.Date
							obMessage.From = oMessage.From
							For l = 0 To oMessage.Headers.Count -1
								obMessage.HeaderValue(oMessage.Headers.Item(l).Name) = oMessage.Headers.Item(l).Value 
							Next 'l
							obMessage.Subject = "[Resent by spam re-test script]" & oMessage.Subject
							obMessage.To = oMessage.To
							obMessage.AddRecipient "", oMessage.To
							obMessage.Flag(1) = false
							obMessage.Save
							oMessage.Save
							On Error Goto 0
						End If
						TempText = InStr(RetFullCode, "---- ---------------------- --------------------------------------------------")
						'Add headers
						oMessage.HeaderValue("X-Spam-Recheck-Date") = CDate(Now)
						oMessage.HeaderValue("X-Spam-Recheck-Score") = TempResult
						If Len(RetFullCode) > 100 Then
							Call CustomMonthlyLog(Right(RetFullCode,Len(RetFullCode)-InStr(RetFullCode, TempText) - 78), "Re-Spam")
						End if
						oMessage.Save
					End If	
				End If
			End if
		Next 'k
	End if
	Call CustomMonthlyLog("ReScan completed", "Re-Spam")
End Function
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

palinka
Senior user
Senior user
Posts: 1182
Joined: 2017-09-12 17:57

Re: How do you check antispam settings?

Post by palinka » 2019-10-04 12:11

I was thinking more along the lines of the filters i use to reject mail based on IP checking, helo, etc. I guess all of those could be recreated within spamassassin. Some are already there, like spamhaus.

That is a pretty nifty script. Please forgive my ignorance, but I'm coming up short on seeing it's usefulness, although I'm sure there is a good reason for it. Sometimes i need a slap upside the head to get set in the right direction. Could you share some insight? I mean, you're running it on a schedule so it's obviously not for merely testing new rules. Does it regularly find stuff that was missed on incoming scan?

User avatar
mattg
Moderator
Moderator
Posts: 20216
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: How do you check antispam settings?

Post by mattg » 2019-10-05 07:54

I find the sanesecurity defs for ClamAV excellent
(I run Clam tests within Spamassassin)

The one thing, is that sometimes those defs pick up something an hour or two after it is received.

What my script does is trawl all unread mail in every inbox, looking for something that wasn't flagged as spam when it arrived, and test it to see if it is NOW spam, and if so remove it.

It finds a few a week that were missed initially, mostly thanks to the excellent work of the SaneSecurity Defs
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

palinka
Senior user
Senior user
Posts: 1182
Joined: 2017-09-12 17:57

Re: How do you check antispam settings?

Post by palinka » 2019-10-05 13:04

Cool. Sounds great. I use the clamav/sane plugin too. Also download definitions hourly. I'll give that a try.

Pedja
Normal user
Normal user
Posts: 39
Joined: 2010-10-30 05:39
Location: Serbia

Re: How do you check antispam settings?

Post by Pedja » 2019-11-04 10:01

palinka wrote:
2019-10-03 14:20
What type of filters were you thinking about testing?
Custom user rules, setting points and thresholds...

Post Reply