ClamAv, SpamAssassin, and related scripts discussion

Forum for things that doesn't really have anything to do with hMailServer. Such as php.ini, beer, etc etc.
Post Reply
User avatar
sheffters
Senior user
Senior user
Posts: 453
Joined: 2009-07-01 20:46
Contact:

Re: ClamAv, SpamAssassin, and related scripts discussion

Post by sheffters » 2010-01-18 10:50

13 voters and 117 posts ... its one of hose topics that everyone will have a view on and they will all be different ... think that will happen wherever there is a topic on AV and the like!

S.

Tooms
Normal user
Normal user
Posts: 52
Joined: 2009-05-12 23:06
Location: Denmark
Contact:

Re: ClamAv, SpamAssassin, and related scripts discussion

Post by Tooms » 2010-01-20 10:22

I'm finally done with testing. I descoverd the reason with SpamAssassin taking all that time. I desabled the network features and I get less the 2 seconds to scan an email.

My conclusion: There is nothing wrong with SpamAssassin being slow! SA is waiting for the network to respond. So network congestion is the choke point not SA! I hope this clears up any confusion with SpamAssassin. I'm curious how effective SpamAssassin is with out the network enabled? at least some of the network tests seem to be redundant with the spam check with HMS. We need a streamlined SA config file. :)
SA Docs wrote:-L, --local

Perform only local tests on all mail. In other words, skip DNS and other network tests. Works the same as the -L flag to spamassassin(1).
So it was the network testing it wait for, now you can dig into what testes there is taken the long time and what of them that you can finetune the timeout values for

about the SpamDloader and logfile, i have put this into my todo list for it so this maybe get fixed later on.
Any comment or statements is my own and have no relationship to my workplace

Tooms
Normal user
Normal user
Posts: 52
Joined: 2009-05-12 23:06
Location: Denmark
Contact:

Re: ClamAv, SpamAssassin, and related scripts discussion

Post by Tooms » 2010-01-20 10:55

horndog wrote:
Tooms wrote:about the SpamDloader and logfile, i have put this into my todo list for it so this maybe get fixed later on.
Thanks, when I'm done testing I'll start using SpamDloader again. I need to know If I can change to the "-L" switch and have it work?
I think it will work just fine with the "-L" commandline option set inside the SpamDloader GUI, What is not work is when a commandline option is making output to console........... options like " > sometext.fil"
Any comment or statements is my own and have no relationship to my workplace

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: ClamAv, SpamAssassin, and related scripts discussion

Post by tBB » 2010-01-20 17:38

horndog wrote: My conclusion: There is nothing wrong with SpamAssassin being slow! SA is waiting for the network to respond. So network congestion is the choke point not SA!
SpamD does the network tests in parallel so it still shouldn't take that long. How fast is your connection?
horndog wrote: I hope this clears up any confusion with SpamAssassin. I'm curious how effective SpamAssassin is with out the network enabled? at least some of the network tests seem to be redundant with the spam check with HMS. We need a streamlined SA config file. :)
You can probably disable the SURBL and HELO tests in SpamAssassin as they seem to be redundant. The RBL checks in SA are more advanced as they check every host where the mail has passed thru against the RBL's and not only the delivering one so I'd rather disable them in hMS. You should probably also use the Razor2 and DCC tests. Razor2 is very effective against Spam and DCC triggers on mass mailings of all sorts (including Spam, of course). As Tooms and me suggested, try disabling the network tests one by one and you'll quickly see which one is the slow one.

As for SpamD becoming slower over time, does the memory usage increase as well?

Best regards,

Nico

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: ClamAv, SpamAssassin, and related scripts discussion

Post by tBB » 2010-01-21 16:54

If you're using the DNS of your ADSL router it might be that it can't handle the many simultaneous requests of SA. See for example this thread: http://www.mail-archive.com/users@spama ... 45136.html

You could use a local caching DNS such as http://www.maradns.org/ and forward the requests to the DNS of your ISP (this would speed up the DNS requests of SA a lot anyway).

Best regards,

Nico

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: ClamAv, SpamAssassin, and related scripts discussion

Post by tBB » 2010-01-21 22:20

horndog wrote:I lowered to time out value from 20 to 10 seconds and now the same URLs time out in 10 seconds instead of 20. So my conclusion is they are unresponsive no mater the time outs setting. I also lowered the TCP time out value (The time Spamd waits for headers from SpamC) from 30 to 25 seconds. Hopefully this will stop the HMS SA error. Ill let you know the results.
Lowering the SA timeout value(s) will maybe stop the hMS timeouts but it won't solve your problem at all.
horndog wrote:BTW would it be possible to put the URL of a cashingDNSserver in SA config instead of trying to get it done in Windows?
Yes, that is possible but why do you want to use a local DNS server only for SA? You have to configure it anyway, it would speed up most other requests as well and it's only one setting in the Windows network configuration? However:
SpamAssassin WIKI http://wiki.apache.org/spamassassin/InstallingOnWindows wrote: The following system environment variables may have to be set:

RES_NAMESERVERS=<your-dns-server-ip-address>

If you do not set this variable, SpamAssassin should use the first DNS server in your network configuration. That is useful when, for example, you move your computer between different networks and have DNS automatically assigned. If you experience problems with domain name resolution, try setting this variable to force SpamAssassin to use the specified nameserver.
Best regards,

Nico

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: ClamAv, SpamAssassin, and related scripts discussion

Post by tBB » 2010-01-21 23:12

horndog wrote:The problem is the unresponsive sites they will be unresponsive if the time out is 10 or 20 seconds.

Why install a Cashing DNS server if the sites still are unresponsive? If some one could do a test with SA from another location to show that in fact the site are responsive then that a different story.
How do you come to the conclusion that psbl, zen.spamhaus.org or dnswl would be unresponsive? These are well known and reputed RBL's and they are surely up and running.
horndog wrote:This may work without a network DNS server!
This setting just enforces a static DNS and it doesn't matter at all if it's local or not.
horndog wrote: Do you know of any good Internet DNS servers?
There are a couple of free DNS servers on the net, e.g. OpenDNS or ScrubIt. Even Google provides two but there is probably no reason to not use the one of your ISP.

Best regards,

Nico

Tooms
Normal user
Normal user
Posts: 52
Joined: 2009-05-12 23:06
Location: Denmark
Contact:

Re: ClamAv, SpamAssassin, and related scripts discussion

Post by Tooms » 2010-01-21 23:33

to verify the speed of the RBL servers, i have a tool for that call IPrblBenchmark there can benchmark the lookup speed for each RBL server you like.

http://www.tooms.dk/software/iprblbenchmark/default.asp
Any comment or statements is my own and have no relationship to my workplace

Tooms
Normal user
Normal user
Posts: 52
Joined: 2009-05-12 23:06
Location: Denmark
Contact:

Re: ClamAv, SpamAssassin, and related scripts discussion

Post by Tooms » 2010-01-22 11:40

yes, as you can see there is a big diff. between the RBL's
So if some of the slow ones are used in SA that is will slow down the mail process as it has to wait for the slow response or in the end do a timeout, so with this information you can compare this with the RBLs that SA use and if it is to slow then maybe disable that list in the SA config or change it to another RBL
Any comment or statements is my own and have no relationship to my workplace

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: ClamAv, SpamAssassin, and related scripts discussion

Post by tBB » 2010-01-22 12:08

horndog wrote:More debug info for DNS, something here might be obsolete.

Code: Select all

2010 [14512] dbg: dns: no ipv6
Thu Jan 21 18:00:27 2010 [14512] dbg: dns: is Net::DNS::Resolver available? yes
Thu Jan 21 18:00:27 2010 [14512] dbg: dns: Net::DNS version: 0.61
Thu Jan 21 18:00:29 2010 [14512] dbg: dns: dns_available set to yes in config file, skipping test
Thu Jan 21 18:00:29 2010 [14512] dbg: dns: checking RBL sa-other.bondedsender.org., set bsp-untrusted
Thu Jan 21 18:00:29 2010 [14512] dbg: dns: checking RBL plus.bondedsender.org., set ssc-firsttrusted
Thu Jan 21 18:00:29 2010 [14512] dbg: dns: checking RBL combined.njabl.org., set njabl
Thu Jan 21 18:00:29 2010 [14512] dbg: dns: checking RBL blackholes.five-ten-sg.com., set fiveten
Thu Jan 21 18:00:29 2010 [14512] dbg: dns: checking RBL blackholes.five-ten-sg.com., set fiveten-notfirsthop
Thu Jan 21 18:00:29 2010 [14512] dbg: dns: checking RBL zen.spamhaus.org., set zen-lastexternal
Thu Jan 21 18:00:29 2010 [14512] dbg: dns: checking RBL dnsbl.sorbs.net., set sorbs-lastexternal
Thu Jan 21 18:00:29 2010 [14512] dbg: dns: checking RBL dnsbl.sorbs.net., set sorbs
Thu Jan 21 18:00:29 2010 [14512] dbg: dns: checking RBL zen.spamhaus.org., set zen-lastexternal
Thu Jan 21 18:00:29 2010 [14512] dbg: dns: checking RBL psbl.surriel.com., set psbl
Thu Jan 21 18:00:29 2010 [14512] dbg: dns: checking RBL list.dnswl.org., set dnswl-firsttrusted
Thu Jan 21 18:00:29 2010 [14512] dbg: dns: checking RBL sa-accredit.habeas.com., set habeas-firsttrusted
Thu Jan 21 18:00:29 2010 [14512] dbg: dns: checking RBL sa-trusted.bondedsender.org., set bsp-firsttrusted
Thu Jan 21 18:00:29 2010 [14512] dbg: dns: checking RBL zen.spamhaus.org., set zen
Thu Jan 21 18:00:29 2010 [14512] dbg: dns: checking RBL sbl.csma.biz., set sblcsma
Thu Jan 21 18:00:29 2010 [14512] dbg: dns: checking RBL iadb.isipp.com., set iadb-firsttrusted
All of the above RBL's respond (100 random IP's, using Tooms's benchmark tool):

Summary report - Best average time (Sort by time)
——————————————————————————————————————————————————————
0,0667 Second - psbl.surriel.com
0,0694 Second - dnsbl.sorbs.net
0,0841 Second - iadb.isipp.com
0,0941 Second - combined.njabl.org
0,1144 Second - sa-accredit.habeas.com
0,1177 Second - sa-trusted.bondedsender.org
0,1241 Second - plus.bondedsender.org
0,1255 Second - sa-other.bondedsender.org
0,1295 Second - zen.spamhaus.org
0,2561 Second - blackholes.five-ten-sg.com
1,248 Second - sbl.csma.biz

sbl.csma.biz is really slow with a average response time of 1.2 sec but it didn't timeout once. The IP range which list.dnswl.org returns isn't supported by Tooms's tool but I've verified manually that it responds.

Best regards,

Nico

User avatar
DFitch
Senior user
Senior user
Posts: 258
Joined: 2006-09-16 20:40

Re: ClamAv, SpamAssassin, and related scripts discussion

Post by DFitch » 2010-01-22 16:31

So my conclusion is they are unresponsive no mater the time outs setting
My suggestion would be to check their DNSBL Usage Terms, Their FREE service is limited too a certain number of connections a day, if your over this limit your IP will be BANNED!

Also I would highly recommend using a local DNS server, especially if using the RBLs, as TBB suggested, it will help.

DCC is good but slow, I found it was not needed and use Razor as its very affective.

Just keep removing as they suggest and watch for the affects on scoring and see what you can get away with.

D
hMailServer 5.3.3: External MySql
Win2k3 Server | eWall 4.0 Anti-Spam Anti-Virus SMTP Proxy {http://sssolutions.net/}
SpamAssassin 3.31 - ClamAV on backend Ubuntu Server 10.04(VMware)

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: ClamAv, SpamAssassin, and related scripts discussion

Post by tBB » 2010-01-24 19:31

horndog wrote:...
Uhm...is there a special reason why you've replaced all of your posts with "..." :?:

Best regards,

Nico

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: ClamAv, SpamAssassin, and related scripts discussion

Post by martin » 2010-01-24 21:00

Other people told me that Horndog was being rude. I informed him about their complaints and asked him to try to be a bit more polite. We had a bit of an argument - his opinion was that he was free to be rude due to the right of speech. The hMailServer.com server is located in the U.S. and the U.S.has freedom of speech. I told him that I didn't find "freedom of speech" relevant; people should not be rude here. This probably upset him, which is likely why he replaced all his posts with "..". (I obviously like freedom of speech, but people still needs to be polite and not throw random accusations at eachother in this forum.).

After this, I deleted all his posts to clean up. Maybe this was a bad idea since it makes some threads where he has participated a bit confusing. :-\

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: ClamAv, SpamAssassin, and related scripts discussion

Post by ^DooM^ » 2010-01-24 21:08

They were just as confusing with "..." I wouldn't worry about it.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: ClamAv, SpamAssassin, and related scripts discussion

Post by tBB » 2010-01-24 22:22

martin wrote:Other people told me that Horndog was being rude. I informed him about their complaints and asked him to try to be a bit more polite. We had a bit of an argument - his opinion was that he was free to be rude due to the right of speech.
I see.... Well, I didn't notice those threads but considering that he also became somewhat rude to me (nothing worth to report though) while being completely on the wrong track and ignoring any advices he's been given, I have a suspicion about what might have happened. Still, I've never seen someone before who deleted all of his (many) posts in a forum. Horndog, should you read this: Please try decaffeinated coffee ;)

Best regards,

Nico

Post Reply