SpamAssassin temp size off by one

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
rstarkov
New user
New user
Posts: 26
Joined: 2013-12-30 15:51

SpamAssassin temp size off by one

Post by rstarkov » 2014-01-28 17:41

I'm running hMailServer 5.4-B1950 with SpamAssassin 3.3.2. On several occasions spam messages got through without any SA headers. After enabling all the debug logging I could find, I found these:

Code: Select all

SA: Temp file size did not match what Spamd reported! (temp: 13780, spamd: 13781). Reverting to original message file.

Code: Select all

SA: Temp file size did not match what Spamd reported! (temp: 2005, spamd: 2006). Reverting to original message file.
Observe it's off by one in both cases. Is this a known issue fixed in the SVN builds? If not, can I ask hMailServer to keep those temporary files to have a closer look at what's up?

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: SpamAssassin temp size off by one

Post by Bill48105 » 2014-01-28 20:08

Hi. Yes it was a known problem before in that corrupt messages would be kept so changes were made to how hmail handles cases where what is returned is not right. Not sure why it'd be off by 1 though but maybe CR vs CRLF or LF cv CRLF maybe. It's been awhile since I worked on that SA stuff so I'll have to look at it again and see what is best option to help you out with but realize generally speaking if SA does not return an email larger than it was sent odds are it is corrupted or not worth keeping the changes anyway. If I recall one potential issue was if the email already had SA headers because they SA would strip them then add new headers & there was a chance the new message could be smaller tripping up the size check. I THOUGHT I added an INI setting to set the amount the size could be off by but maybe I never got to it. The person who ran into the problem just edited SA config adding a fake header in SA that was large enough to get by the size issue.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: SpamAssassin temp size off by one

Post by Bill48105 » 2014-01-28 20:12

Oh yeah found this:
2013-05-13 5.4-B2013051301
*** DO NOT USE THIS BUILD IF YOU USE SA - Extra header is added by SA that can cause problems **
* Upgraded to SA protocol 1.2 to allow size to be passed to/from SA for better confirming SA results (Thanks rolaids0/JamesDR for the patch!)
* IMPORTANT NOTE: If SA is desired, SA 2.70 or later will be required going forward. Since that's from 2004 seems unlikely anyone is on THAT old of a build.
So we had switched to protocol 1.2 vs 1.0 which allowed size to be passed. I'll have to look at the code to see how the size comparison is checked though.
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

rstarkov
New user
New user
Posts: 26
Joined: 2013-12-30 15:51

Re: SpamAssassin temp size off by one

Post by rstarkov » 2014-01-28 23:03

Thanks Bill. It doesn't happen very often, so treat it as low priority. In fact, now that I've found the source code (I used to think v5+ was closed source) I might have a look myself when I get sufficiently tired of this.

(my SA is configured to add the rather long X-Spam-Report header so I don't think it's a problem of the returned email not being sufficiently longer than the one going in)

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: SpamAssassin temp size off by one

Post by Bill48105 » 2014-01-28 23:37

It's a direct compare & no fudge at all.

Code: Select all

      // new way: check the result from spamd.
      if (bTestsRun && (FileUtilities::FileSize(sTempFile) == m_iSpamDSize))
So one of those 2 must be off slightly.
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

sckramer2
Normal user
Normal user
Posts: 133
Joined: 2009-07-31 21:50

Re: SpamAssassin temp size off by one

Post by sckramer2 » 2014-07-09 21:57

Hi,

"SA: Temp file size did not match what Spamd reported! (temp: 9173, spamd: 9174). Reverting to original message file."

We're getting a bunch of these everyday also, took awile to find why hmailserver\temp was filling up with scored spams, but still allowed the original message through.

Always off by 1...

Is there a way to fix this?

Thanks!

sckramer2
Normal user
Normal user
Posts: 133
Joined: 2009-07-31 21:50

Re: SpamAssassin temp size off by one

Post by sckramer2 » 2014-07-12 00:35

Anyone have a suggestion on this? It let's ridiculously high scored spams thru, and people are asking about it... thousands of emails are filtered just fine, but those 20 a day are a problem!

Thanks!

rstarkov
New user
New user
Posts: 26
Joined: 2013-12-30 15:51

Re: SpamAssassin temp size off by one

Post by rstarkov » 2014-07-12 01:20

FWIW, I've never found a fix. I still get one spam every couple of days due to this bug. I imagine it's a lot worse for you... It's annoying because it's always spam that would trivially get caught by SA; it's usually super-spammy.

I've been meaning to try and debug it myself since hMailServer is open-source, but there's never any time...

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: SpamAssassin temp size off by one

Post by Bill48105 » 2014-07-13 21:01

Did you try one of the recent experimental builds to see if there is a difference? This is unlikely an hmail bug. hmail is reporting to you in the logs the sizes. I'd guess it has to do with CRLF vs just LF type thing. Not sure how else you could end up with just 1 byte difference. If it were a stray SA header result it'd always be there for every message. Unless we know where/why most we could do is add a fudge factor ini setting that tells hmail to ignore a small difference. But ideally we find the cause to have a real solution. That'd almost require saving both so a hex compare could be done.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

sckramer2
Normal user
Normal user
Posts: 133
Joined: 2009-07-31 21:50

Re: SpamAssassin temp size off by one

Post by sckramer2 » 2014-07-14 18:37

Hi Bill,

I'm not sure how to do a compare for you to help debug... sure I have the temp file, but how do I recreate the source (from SA at that time)?

I can manually score the email again, but it's always a little different (order of blacklist lookups, bayes learning etc)--

If it's not too hard, an INI setting would be nice for now!

Thanks!

(Running latest hmailserver experimental, and jam-software SA Version 3.4.0.30)

sckramer2
Normal user
Normal user
Posts: 133
Joined: 2009-07-31 21:50

Re: SpamAssassin temp size off by one

Post by sckramer2 » 2014-07-26 01:33

this is a bit of a bump, but was thinking hmailservers "Spam mark threshold" is a little thick... would it be useful to add a decimal place 1.x... to fine tune?

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: SpamAssassin temp size off by one

Post by ^DooM^ » 2014-07-26 01:43

Just add a 0 to the end of all your spam mark numbers and you will get the same functionality with no serverside changes. Will give you much greater scope for marking spam so 1 becomes 10 and in your example 1.5 would now be 15, then in your RBLS, you can tailor to suit.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

rstarkov
New user
New user
Posts: 26
Joined: 2013-12-30 15:51

Re: SpamAssassin temp size off by one

Post by rstarkov » 2014-07-26 02:03

sckramer2 wrote:this is a bit of a bump, but was thinking hmailservers "Spam mark threshold" is a little thick... would it be useful to add a decimal place 1.x... to fine tune?
Wrong topic?

sckramer2
Normal user
Normal user
Posts: 133
Joined: 2009-07-31 21:50

Re: SpamAssassin temp size off by one

Post by sckramer2 » 2014-07-26 02:20

yep, that's why i said "bit of a bump"

sckramer2
Normal user
Normal user
Posts: 133
Joined: 2009-07-31 21:50

Re: SpamAssassin temp size off by one

Post by sckramer2 » 2014-07-26 02:24

^DooM^ wrote:Just add a 0 to the end of all your spam mark numbers and you will get the same functionality with no serverside changes. Will give you much greater scope for marking spam so 1 becomes 10 and in your example 1.5 would now be 15, then in your RBLS, you can tailor to suit.
I'll experiment, lately, I've been disabling all hmailservers built in spam protections... all redundant to the spamassassin checks (spf/dkim/rdns/dnsbl/surbl...etc. etc.) a key thing tho... is to use windows dns service (server 2012r2, you have to enable it - it queries the master dns servers)... don't use your isp's, googles, opendns, etc...

...also, spamassassin suddenly seems have gotten very effective, I use jam software Version 3.4.0.30-- where before I *had* to use greylisting... which sucks... but since this latest spamassassin build I *don't* have to use greylisting... it breaks email and the delays are absurd, i say remove it from hmail -- yeah its effective, but it breaks all the rules

sckramer2
Normal user
Normal user
Posts: 133
Joined: 2009-07-31 21:50

Re: SpamAssassin temp size off by one

Post by sckramer2 » 2014-07-26 02:45

anyway... please can we have an ini setting to fudge the off by 1?

Post Reply