clamwin too slow - hmailserver crash

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
twaldorf
New user
New user
Posts: 22
Joined: 2009-07-13 11:29

clamwin too slow - hmailserver crash

Post by twaldorf » 2009-10-03 10:38

Hello,

yesterday I send out a newsletter (about 20.000 recipients) and found out that hmailserver wasn't able to do it. The queue was growing and growing, the cpu consumption was at 100%. I let it run over night, today in the morning the queue had 25.000 entries and it was a problem to navigate in the hmailserver administration because it crashes all the time!!!

I found out that the problem was clamwin. I disabled virus scan, restartet the hmailserver service and the queue was empty within 5 minutes.

So I urgently need a possibility to scan only incoming mails or a hint for a MUCH MUCH faster external scanner.

Can anybody help?

Thanks,

Thorsten

User avatar
martin
Developer
Developer
Posts: 6837
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: clamwin too slow - hmailserver crash

Post by martin » 2009-10-03 11:06

If you only want to scan incoming messages, you would have to create a custom script which calls an external virus scanner.

twaldorf
New user
New user
Posts: 22
Joined: 2009-07-13 11:29

Re: clamwin too slow - hmailserver crash

Post by twaldorf » 2009-10-03 11:10

Hello Martin,

thanks for you fast answer.

I hoped there is a way to use the ip-ranges for it... No chance to do it this way?

Thanks,

Thorsten

User avatar
martin
Developer
Developer
Posts: 6837
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: clamwin too slow - hmailserver crash

Post by martin » 2009-10-03 11:12

You can disable scanning on all email arrived from your local network using IP ranges. You can't disable scanning for email coming from authenticated users using IP ranges.

User avatar
mattg
Moderator
Moderator
Posts: 21183
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: clamwin too slow - hmailserver crash

Post by mattg » 2009-10-03 16:13

Some of us use ClamAv rather than ClamWin. - http://www.hmailserver.com/forum/viewto ... 12&t=13699
I tend to think that ClamWin has issues with high usage servers.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
miki75
New user
New user
Posts: 15
Joined: 2009-07-13 16:20
Location: Italy
Contact:

Re: clamwin too slow - hmailserver crash

Post by miki75 » 2009-10-04 02:01

I agree with mattg, ClamAV is better on high volume mailserver

And ClamAV for Windows - [tBB] is a very good solution!

Miki

User avatar
martin
Developer
Developer
Posts: 6837
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: clamwin too slow - hmailserver crash

Post by martin » 2009-10-05 23:50

twaldorf,
Can you tell me what hMailServer version you're using?

westdam
Senior user
Senior user
Posts: 728
Joined: 2006-08-01 21:24
Location: Padova, Italy
Contact:

Re: clamwin too slow - hmailserver crash

Post by westdam » 2009-10-06 17:24

i've got same issue over and over with clamwin.
no clue..if clamwin is scanning 5-6 email at the same moment CPU goes on 100% and server stop responding. hmailserver accept and queue email but wont send out due to clamwin scanning.

solutions?

disabling clamwin and use clamAV.

User avatar
martin
Developer
Developer
Posts: 6837
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: clamwin too slow - hmailserver crash

Post by martin » 2009-10-06 17:33

westdam,
So what hMailServer version are *you* using?
Also, what do you mean by 'wont send out"? Do you mean that ClamWin hangs while scanning? Otherwise I would assume that hMailServer will continue whenever ClamWin is done?

westdam
Senior user
Senior user
Posts: 728
Joined: 2006-08-01 21:24
Location: Padova, Italy
Contact:

Re: clamwin too slow - hmailserver crash

Post by westdam » 2009-10-06 21:44

i'm using 5.3 b363 on mssql express.
clamwin is 0.95.2 ( think is latest )

hmailserver wait clamwin has finished the scan.
i've got 5-6 clamscan.exe executable running using most of mine dualcore cpu.

this can happen only when some mail arrives in the same moment.
( for example if connection was broken and my mx send me mail )

i've to disable the clamwin scan as workaround.


for me it's not a bug.. it's simply clamwin. it's very very slow.

User avatar
martin
Developer
Developer
Posts: 6837
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: clamwin too slow - hmailserver crash

Post by martin » 2009-10-07 17:47

Hmm. But the issue in this thread was that ClamWin was slow and hMailServer crashed. But your case simply seems to be that ClamWin is slow?

westdam
Senior user
Senior user
Posts: 728
Joined: 2006-08-01 21:24
Location: Padova, Italy
Contact:

Re: clamwin too slow - hmailserver crash

Post by westdam » 2009-10-07 20:01

unfortunately i cant understand in deph.

if this situation occurs, the system is unresponsive...

sometimes hmailserver is unresponsive and doesnt accept any connection or mail , sometimes accept the connection but timed out.

what about a timeout for clamwin ?i think there is yet this kind of protection..

DeanoX
Senior user
Senior user
Posts: 480
Joined: 2005-11-05 00:07
Location: Michigan

Re: clamwin too slow - hmailserver crash

Post by DeanoX » 2009-10-07 20:15

I suggest that you run ClamAV, rather than clamwin with hmailserver.

My install of ClamAV runs without issues.

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: clamwin too slow - hmailserver crash

Post by tBB » 2009-10-07 20:18

ClamWin is simply not made for mailservers. According to your post you have several ClamScan.exe in memory. No wonder - ClamScan loads/unpacks the ClamAV signatures each time it runs which takes of course time, CPU power and a lot of memory whereas ClamDScan connects to the resident ClamD daemon, which loads the signatures only once (or when FreshClam indicates that there was a update). Therefore the solution to your issue is to use a command line version of ClamAV. Actually it's a mystery for me why hMailServer natively supports ClamWin.

Best regards,

Nico

DeanoX
Senior user
Senior user
Posts: 480
Joined: 2005-11-05 00:07
Location: Michigan

Re: clamwin too slow - hmailserver crash

Post by DeanoX » 2009-10-07 20:25

Hi Nico,

I agree, native clamwin integration should be considered for removal.

User avatar
sheffters
Senior user
Senior user
Posts: 453
Joined: 2009-07-01 20:46
Contact:

Re: clamwin too slow - hmailserver crash

Post by sheffters » 2009-10-07 20:36

Clamwin works fine on my server ... low volume servers are fine with it ... high volume servers should use an alternative ... but that doesn't mean functionality should be removed from the server.

S.

DeanoX
Senior user
Senior user
Posts: 480
Joined: 2005-11-05 00:07
Location: Michigan

Re: clamwin too slow - hmailserver crash

Post by DeanoX » 2009-10-07 20:42

No functionality would really be removed. You could run any AV program you like. There would just not be a clamwin tab in anti-virus. :)

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: clamwin too slow - hmailserver crash

Post by tBB » 2009-10-07 21:00

Well, IMO it shouldn't be removed but replaced with integration of the (much smaller) CLI version because the feature just seems to confuse people and lead to questions. Besides, there is usually no reason to have the heavy ClamWin Python-GUI loaded at a server when at the end it's just the included ClamScan.exe which is called.

Best regards,

Nico

DeanoX
Senior user
Senior user
Posts: 480
Joined: 2005-11-05 00:07
Location: Michigan

Re: clamwin too slow - hmailserver crash

Post by DeanoX » 2009-10-07 21:11

AFAIK, and I could be wrong, but hmailserver just simply detects, and then uses, an already installed version of clamwin on the same machine. There is nothing in hmailserver that includes any portion of clamwin.
You either install clamwin or clamav, or anything else. The choice is yours what to use.

User avatar
sheffters
Senior user
Senior user
Posts: 453
Joined: 2009-07-01 20:46
Contact:

Re: clamwin too slow - hmailserver crash

Post by sheffters » 2009-10-07 21:13

because the feature just seems to confuse people
only people who cant be arsed to read the forums and blindly tick everything ... it's been discussed to death on here in numerous posts about clamwin and its limitations ...

if people are sending thousands of mails out (or expecting to) then they really need to tune the software rather than blindly install everything and wonder why it doesnt run optimally ...

don't remove functionality for the rest of us because people can't be bothered to read.

S.

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: clamwin too slow - hmailserver crash

Post by tBB » 2009-10-07 21:17

Would you mind telling me the drawback if the ClamWin integration would be replaced with integration of the CLI version?

Best regards,

Nico

User avatar
sheffters
Senior user
Senior user
Posts: 453
Joined: 2009-07-01 20:46
Contact:

Re: clamwin too slow - hmailserver crash

Post by sheffters » 2009-10-07 21:23

I assume most people who use Clamwin also use it for the server AV (I do anyway, and assume I'm not completely alone) ... whereas installing some limited version for mail server only would mean 2 diff AV's having to be installed on the server or having to code some gizmo to scan files. The GUI version enables scheduled scanning and the like.

As before, IMHO high volume mail server will need to be tuned and not generic; including AV solution etc. The aim of integrating with an AV solution means people can get up and running quickly for small installs; not for people intending to use high volume mail servers which will need threading and all sorts messing with to optimise.

Cheers

S.

DeanoX
Senior user
Senior user
Posts: 480
Joined: 2005-11-05 00:07
Location: Michigan

Re: clamwin too slow - hmailserver crash

Post by DeanoX » 2009-10-07 21:28

sheffters wrote:don't remove functionality for the rest of us because people can't be bothered to read.
Even if the clamwin tab were removed, I believe that there is no loss of functionality for anyone wishing to use clamwin -vs clamav.

I think tBB's comment,
because the feature just seems to confuse people
is true.

I imagine that people see the clamwin tab, and then get clamwin, turn it on, and think all is good. But like you say sheffters, if they don't bother to read the forums, or the documentation, and just enable features, assuming they know how it works, they are just asking for trouble at some point.

To the original poster, twaldorf, it should be now evident what you need to do to resolve your specific issue. Switch to clamav, or some other mail server centric, anti-virus solution :)

User avatar
sheffters
Senior user
Senior user
Posts: 453
Joined: 2009-07-01 20:46
Contact:

Re: clamwin too slow - hmailserver crash

Post by sheffters » 2009-10-07 21:46

I imagine that people see the clamwin tab, and then get clamwin, turn it on, and think all is good.
probably.

From memory (without searching forum) the sensible limit people hit is 4-5 mails/min with clamwin ... maybe the simplest solution would be to ask martin to add a note on the clamwin integration tab about 'high volume' / greater than 4-5 mails / min don't use clamwin and look for another solution.

Re loss of functionality ... not in terms of virus scanning, but in terms of ease of use / setup then yes ... depends how functionality is defined compared to useability.

Cheers

S.

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: clamwin too slow - hmailserver crash

Post by tBB » 2009-10-07 22:04

The CLI packages of ClamAV are by far no limited versions for mail servers because running at mail servers is the basic purpose of ClamAV and not scanning HD's. In fact, ClamWin includes the limited version of ClamAV :)

Besides, in case the ClamAV integration in hMailServer would be changed, no one would stop a ClamWin user to still use ClamWin's ClamScan.exe in hMailServer if he wants to - and even if he wants to use both versions simultaneously, there wouldn't be two different AV's installed because it's of course possible to let ClamAV (ClamD) and ClamWin share the same signature databases. However, as far as I know hMailServer is well able to cope with high traffic so I wouldn't assume that it's targeted at home users anyway.

Best regards,

Nico
Last edited by tBB on 2009-10-07 22:10, edited 1 time in total.

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: clamwin too slow - hmailserver crash

Post by ^DooM^ » 2009-10-07 22:10

Clamwin is easy to install and hMail is all about usability and making things easy for everyone not just tech heads like us. It's there for small users, those guys that use it for personal mail servers between family members or intranet use in a small company and so on. I wouldn't like to see it removed but perhaps on the V5.4 GUI fixes and tweaks update there can be a warning popup about clamwins limitations.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: clamwin too slow - hmailserver crash

Post by tBB » 2009-10-07 22:38

Actually one needs to be more of a techhead if he installs ClamWin and configures it's options than he needs to be for ClamAV. If one uses ClamAV's default install directory it's not even needed to configure anything, at least not in the ClamAV distributions from the ClamWin developers or my one.

Best regards,

Nico

User avatar
sheffters
Senior user
Senior user
Posts: 453
Joined: 2009-07-01 20:46
Contact:

Re: clamwin too slow - hmailserver crash

Post by sheffters » 2009-10-07 22:44

clamwin = doubleclick an install exe ... click the thing in hmail admin ... done ...

S.

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: clamwin too slow - hmailserver crash

Post by tBB » 2009-10-07 23:21

Wow, then there would be no difference in the installation procedure for the casual user to clicking the ClamAV installer.

Seriously, I don't care if this feature is removed/changed or not but even the ClamWin developers would tell you that ClamWin is neither suitable nor meant for mailservers because each ClamScan instance requires +100Mb of Memory (if third party signatures are also loaded way more) and while loading the signatures, all available CPU time therefore it is no wonder that a few simultaneous incoming mails let the computer look like it crashed. As it usually takes only a fraction of a second and almost zero CPU power to let the ClamD daemon scan a mail, any discussion about which version should be integrated in a mailserver is IMO moot anyway.

Best regards,

Nico

User avatar
sheffters
Senior user
Senior user
Posts: 453
Joined: 2009-07-01 20:46
Contact:

Re: clamwin too slow - hmailserver crash

Post by sheffters » 2009-10-08 00:05

Im not sure were all talking about the same thing here ... I'm pritty sure mine doesnt sporn 100Mb RAM per mail ... will watch it; but think i'd have seen something weird going on by now tbh.

http://www.clamwin.com/

http://www.clamav.net/
A native port for win32 is available at http://w32.clamav.net . THIS SOFTWARE IS NO LONGER SUPPORTED. Development of ClamAV for Win32 has been stopped.
Would be a bit daft to start supporting ClamAV when its not even supported / developed / maintained under Win32 anymre (http://www.clamav.net/download/packages/packages-win32) ... imho

Cheers

S.

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: clamwin too slow - hmailserver crash

Post by tBB » 2009-10-08 00:19

ClamAV/Win32 will be officially supported again after 0.96 but there are already two well known native Win32 ports which are actively being developed and supported, that are Gianluigi Tiesi's (the developer behind the ClamWin engine) one here: http://oss.netfarm.it/clamav/ and my one here: http://hideout.ath.cx/clamav. Which one you use is a matter of personal taste. My one is smaller and might be a little faster because I'm using different compiling options (and it works out of the box with my ClamSup script for third party signatures) but Gianluigi's one has nice icons :) Feature-wise they are exactly the same.

Best regards,

Nico

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: clamwin too slow - hmailserver crash

Post by ^DooM^ » 2009-10-08 00:29

I'll agree to disagree with you tBB on the ease of Clamwin use. I think we all agree it's bad for big installs though and lets leave it at that. I'm sure Martin will do something about it if he feels the need.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

User avatar
mattg
Moderator
Moderator
Posts: 21183
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: clamwin too slow - hmailserver crash

Post by mattg » 2009-10-08 02:33

I use Nico's (tBB's) ClamAV, and I access from the ClamWin tab in Antivirus settings as per the 'tip and trick' from DFitch - http://www.hmailserver.com/forum/viewto ... 12&t=13699

Seems to work well. Low resource usage.

I use it as secondary AV protection, and it is ONLY invoked for mail scanning...
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

westdam
Senior user
Senior user
Posts: 728
Joined: 2006-08-01 21:24
Location: Padova, Italy
Contact:

Re: clamwin too slow - hmailserver crash

Post by westdam » 2009-10-08 08:47

thanks to everybody for the answer..
yes i know i can use the ClamAV with ClamD options.

i just want to know if it was possibile to integer some protection in hmail to prevent clamwin hungs the server .. what about a timeout for claminw calls?

westdam
Senior user
Senior user
Posts: 728
Joined: 2006-08-01 21:24
Location: Padova, Italy
Contact:

Re: clamwin too slow - hmailserver crash

Post by westdam » 2009-10-08 08:58

^DooM^ wrote:I'll agree to disagree with you tBB on the ease of Clamwin use. I think we all agree it's bad for big installs though and lets leave it at that. I'm sure Martin will do something about it if he feels the need.
i admit i'm often using clamwin for his ease of use. but most of mine installation can run with clamwin as well. the REAL problem infact it's not the slowness of clamwin but the slowness on concurrent scanning. It happens on high volume server , or as my example, when my primary server went down and my mx send me the mails. Lots of mail at the same time.
Clamwin hungs and hungs hmail too.

I agree on the alert box. should be a nice idea..
what about on a checkbox " Stop scanning mail if the server is on heavy load " and for heavy i mean 5-6 concurrent mail or queue becomes large..?

it's a workaround i know. i just dont want hmail becomes unresponsive due to clamwin on high load..

Also becouse if i've got a spammer attack with lots of mail on a low load server with clamwin activated the server can seriously becomes unreachable..

i'm in italy and i've got lots of customers with his own mailserver on hmail and i'm using clamwin to catch viruses.. the average is 1 domain with 10-15 accounts.. so i assume it's a low-volume mail ( no more than 100-200 mail per day.. ). Clamwin works perfectly with this rate generally..but sometimes ADSL goes down and the trouble begin.

what do you think? maybe a new feature post?

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: clamwin too slow - hmailserver crash

Post by tBB » 2009-10-08 10:45

westdam wrote: what about on a checkbox " Stop scanning mail if the server is on heavy load " and for heavy i mean 5-6 concurrent mail or queue becomes large..?
:shock: Implementing a option to stop scanning mail for viruses in case of high load is a really bad idea.

If some people insist on using ClamWin with hMailServer for reasons I still don't understand it would be probably a option to add a checkbox which lets hMailServer pass mail to ClamWin consecutively and not simultaneously.

Best regards,

Nico

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: clamwin too slow - hmailserver crash

Post by ^DooM^ » 2009-10-08 11:04

I agree with Nico that stopping scanning is a bad idea. What if your server was getting attacked by thousands of spam emails that all contained a virus? If your server is under heavy load you should use a virus scanner that can cope end of story.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

westdam
Senior user
Senior user
Posts: 728
Joined: 2006-08-01 21:24
Location: Padova, Italy
Contact:

Re: clamwin too slow - hmailserver crash

Post by westdam » 2009-10-08 11:06

I agree with you, it's a bad idea, infact a alertbox would be nice also.

Your idea it's a good idea too... but i think this will slow-down or even timeout the hmailserver if 5-6 mail arrives simult.

anyway your idea it's better than mine :D

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: clamwin too slow - hmailserver crash

Post by tBB » 2009-10-08 11:46

westdam wrote: Your idea it's a good idea too... but i think this will slow-down or even timeout the hmailserver if 5-6 mail arrives simult.
Most probably hMS would just build up a queue. Letting ClamWin (ClamScan) scan mail consecutively should be (at least on smaller machines) faster than starting several ClamScan instances simultaneously therefore I've suggested a checkbox, otherwise it should be IMO the default setting anyway.

Best regards,

Nico

westdam
Senior user
Senior user
Posts: 728
Joined: 2006-08-01 21:24
Location: Padova, Italy
Contact:

Re: clamwin too slow - hmailserver crash

Post by westdam » 2009-10-08 12:03

yes surely better.. My workaround was just to avoid server becomes unresponsive. With clamwin can happen.

Martin what think about it?

User avatar
sheffters
Senior user
Senior user
Posts: 453
Joined: 2009-07-01 20:46
Contact:

Re: clamwin too slow - hmailserver crash

Post by sheffters » 2009-10-08 15:29

random thought ... if its only to stop it being unresponsive (server wise) when it's under an attack and not normal use, maybe using processor affinity would solve it? ... not going to speed up checking and the like, but would also enable hmail to jump on another core and still queue up mails and the like ... so its just a matter of time rather than slowing the server to a crawl while it goes though the list.

Still requires a code change in hmail though ... I guess there is no easy solution ... or at least the old saying 'cant keep all the people happy all the time' applies here ... damned if you do damned if you dont.

S.

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: clamwin too slow - hmailserver crash

Post by tBB » 2009-10-08 17:12

According to the posts above hMS doesn't become unresponsive and still builds up the queue but doesn't process mail further until ClamWin has finished the scan therefore setting the affinity would make things actually worse.

Out of curiousity I've just tested how much memory/CPU ClamScan alone (excluding the ClamWin stuff) allocates with the current set of default signatures. It's 105.6 Mb and up to 99% CPU time for each thread. If the affinity would be set it would be only one core which would be occupied but still 5 threads would consume +500mb of memory and the processing would take even longer.

Passing mail to ClamWin/ClamScan consecutively would solve at least the memory issue if more than one mail comes in and would also cause the ClamScan process to finish it's scanning much faster. For small home servers it wouldn't make any difference anyway but even the smallest server can come under attack of spammers at any time and then all of a sudden it matters.

Best regards,

Nico

westdam
Senior user
Senior user
Posts: 728
Joined: 2006-08-01 21:24
Location: Padova, Italy
Contact:

Re: clamwin too slow - hmailserver crash

Post by westdam » 2009-10-09 01:23

tbb your suggestion are good. i agree with you.

User avatar
mattg
Moderator
Moderator
Posts: 21183
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: clamwin too slow - hmailserver crash

Post by mattg » 2009-10-09 01:51

Would setting delivery threads to 1 achieve this consecutive delivery?
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

westdam
Senior user
Senior user
Posts: 728
Joined: 2006-08-01 21:24
Location: Padova, Italy
Contact:

Re: clamwin too slow - hmailserver crash

Post by westdam » 2009-10-09 09:07

yep mattg but ALL mail would sent 1 per time.

anyway it's a good point. i'll check.

but i prefer the tbb suggestion.

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: clamwin too slow - hmailserver crash

Post by tBB » 2009-10-09 12:18

mattg wrote:Would setting delivery threads to 1 achieve this consecutive delivery?
Depends on where ClamWin is called but this might be at least a workaround because a little bit slower delivery would be still better than a creeping server.

I guess the best solution would be to implement something similar to the "Max number of asynchronous task threads" setting. On a side note, if hMS would connect to a ClamD daemon by TCP like it does with SpamD the same code could be probably used :wink:

Best regards,

Nico

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: clamwin too slow - hmailserver crash

Post by ^DooM^ » 2009-10-09 12:20

A spamassassin like integration with ClamD would be excellent..
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

westdam
Senior user
Senior user
Posts: 728
Joined: 2006-08-01 21:24
Location: Padova, Italy
Contact:

Re: clamwin too slow - hmailserver crash

Post by westdam » 2009-10-10 17:21

yes, great idea..

DeanoX
Senior user
Senior user
Posts: 480
Joined: 2005-11-05 00:07
Location: Michigan

Re: clamwin too slow - hmailserver crash

Post by DeanoX » 2009-10-10 19:50

^DooM^ wrote:A spamassassin like integration with ClamD would be excellent..
That would be nice.

User avatar
martin
Developer
Developer
Posts: 6837
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: clamwin too slow - hmailserver crash

Post by martin » 2009-10-16 19:13

I like the idea of integrating a ClamAV client into hMailServer. Not having to start processes to do the scanning would improve performance.

I know very little about the interface between the ClamAV client and server though. I have some vague memory of having read some source code on how to do it, but my memory is bad. tBB, were you the one who gave me the information a while ago?

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: clamwin too slow - hmailserver crash

Post by tBB » 2009-10-16 19:32

martin wrote: I know very little about the interface between the ClamAV client and server though. I have some vague memory of having read some source code on how to do it, but my memory is bad. tBB, were you the one who gave me the information a while ago?
My memory is not much better (the age, the stress, the drugs...) :lol: but yes, most probably I was the one and I'd be still glad to assist in integrating a ClamD interface into hMS.

Best regards,

Nico

User avatar
martin
Developer
Developer
Posts: 6837
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: clamwin too slow - hmailserver crash

Post by martin » 2009-10-16 22:28

You mentioned earlier that ClamAV/Win32 will be officially supported again after 0.96. Can you tell me what "officially supported" means? You have a 0.95 port on your site, so I assume 0.95 works on Windows.

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: clamwin too slow - hmailserver crash

Post by tBB » 2009-10-17 08:46

Officially supported means, after version 0.96 the ClamAV developers plan to provide Win32 executables again (also by using Gianluigi's Win32 compatibility patches). This because pages like Virustotal (which was btw. using my version as well) are since a while obliged to only use 'official' versions of ClamAV and as there is no Win32 binary provided by the ClamAV guys since a long time, ClamAV often looks bad in the tests.

However, the TCP stream protocol which ClamD uses is compatible to any OS version of ClamAV (just like SpamAssassin) and not going to change in the foreseeable future so it's safe to implement it into hMS. I'll contact you by mail.

Best regards,

Nico

westdam
Senior user
Senior user
Posts: 728
Joined: 2006-08-01 21:24
Location: Padova, Italy
Contact:

Re: clamwin too slow - hmailserver crash

Post by westdam » 2009-10-17 16:53

stange..
cant get working.also clamwin wont scan mail.
it's strange. it's all the default settings.. hmailserver use the settings but i've got even with clamdscan ( with clamd service started ) and clamwin results 0 on hmail log..


here the CLAMWIN log piece

"DEBUG" 2360 "2009-10-17 00:00:28.203" "ClamWinVirusScanner::Scan() - C:\Program Files\ClamWin\bin\clamscan.exe --database="C:\Documents and Settings\All Users\.clamwin\db" "{97228E6C-71DB-4204-B816-2A9760B45644}.eml" --tempdir="C:\WINDOWS\Temp" - Returned 0"
"DEBUG" 2360 "2009-10-17 00:00:28.203" "ClamWinVirusScanner::~Scan()"


and here the clamdscan log ( clamd service started and running )


"DEBUG" 256 "2009-10-17 16:40:25.093" "CustomVirusScanner::Scan() - "c:\clamav\clamdscan.exe" --config-file=c:\clamav\clamd.conf "E:\hMailServer\Temp\{2FAE434B-9A1C-4863-A000-A171E360C3B9}.tmp" - Returned 0"



weird. if I scan a EICAR test file with clamdscan via CMD it results with no problem. i
f hmailserver use settings above wont catch eicar test file but i even dont see clamdscan.exe in the task manager coming up during email scanning..

if I select clamwin settings and disable custom virus scanner , saving settings hmailserver wont use clamscan.exe to scan. ( or i'm unable to see it into task manager ) into the log seems hmailserver scan the file .any ideas?

westdam
Senior user
Senior user
Posts: 728
Joined: 2006-08-01 21:24
Location: Padova, Italy
Contact:

Re: clamwin too slow - hmailserver crash

Post by westdam » 2009-10-17 16:57

mmm strange..
sometimes hmailserver start clamscan.exe process to scan mail, sometimes wont.
i'm using hmailserver 5.3 1617 ( latest beta, sorry cant remember ) .

right now i've changed to clamwin settings disabling clamd scan.

clamwin run the clamscan.exe but wont catch eicar test file. (results 0 )
if I scan manually that file i found eicar test.

Tooms
Normal user
Normal user
Posts: 52
Joined: 2009-05-12 23:06
Location: Denmark
Contact:

Re: clamwin too slow - hmailserver crash

Post by Tooms » 2009-10-17 17:55

martin wrote:I like the idea of integrating a ClamAV client into hMailServer. Not having to start processes to do the scanning would improve performance.

I know very little about the interface between the ClamAV client and server though. I have some vague memory of having read some source code on how to do it, but my memory is bad. tBB, were you the one who gave me the information a while ago?
Hi Martin

I have made a app call mswclamdscan there is design as a exe plugin for a mailscanner call Mailsweeper, this plugin is using remote ClamD servers and it is not that hard to get working....



the protocol goes like this (from version 0.95.2 and newere)

Client>>> connect to ClamD server on port 3310 TCP
Client>>> send string "zINSTREAM" as bytes
Client>>> send 4 bytes with the size of the coming stream
Client>>> send the whole file as byte stream
Client>>> send 4 null byte to tell server finish with stream
Server>>> return string "stream: OK" or "Found some.virus" or "error some text"
Server>>> close the connection

ClamD dev team write about this in the manual to clamD and it is there i have it from.

use ClamD is very good idea as i think it is very fast, i have ClamD running on Debian linux server and with my plugin on a windows server i see scan speed there is between 20-100mb/sec. but this offcouse depending alot on the setup and hardware.
a normal small mail(10k or less) is having a scan time there is 0,08 sec or less


it looks like your code is just calling the clamscan.exe what about make editable so you just can enter the commandline and maybe use my exe.

mswclamdscan.exe /mail:nameofemail.eml /logfile:sometempfile.tmp

the plugin just write one line into the temp logfile for the main application to read as feedback

and the exitcodes are
0 no virus found
1 virus is found
2 some error


my app can be found here, but note it is still beta code
http://www.tooms.dk/software/mswclamdscan/default.asp

just to give my 5 cent
Any comment or statements is my own and have no relationship to my workplace

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: clamwin too slow - hmailserver crash

Post by tBB » 2009-10-17 20:55

Tooms wrote: it looks like your code is just calling the clamscan.exe what about make editable so you just can enter the commandline and maybe use my exe.
Is there any benefit in using your tool compared to ClamDScan?

Best regards,

Nico

Tooms
Normal user
Normal user
Posts: 52
Joined: 2009-05-12 23:06
Location: Denmark
Contact:

Re: clamwin too slow - hmailserver crash

Post by Tooms » 2009-10-17 21:01

tBB wrote:Officially supported means, after version 0.96 the ClamAV developers plan to provide Win32 executables again (also by using Gianluigi's Win32 compatibility patches). This because pages like Virustotal (which was btw. using my version as well) are since a while obliged to only use 'official' versions of ClamAV and as there is no Win32 binary provided by the ClamAV guys since a long time, ClamAV often looks bad in the tests.

However, the TCP stream protocol which ClamD uses is compatible to any OS version of ClamAV (just like SpamAssassin) and not going to change in the foreseeable future so it's safe to implement it into hMS. I'll contact you by mail.

Best regards, Nico
The protocol has just change in 0.95.1 or 0.95.2 there is the first to use the new protocol and they write some where that change maybe to come(read fix minor errors)

so use the new protocol "INSTREAM" and not the old "STREAM", the new one is also better has it only use 1 port and not 2 as the old one
Any comment or statements is my own and have no relationship to my workplace

Tooms
Normal user
Normal user
Posts: 52
Joined: 2009-05-12 23:06
Location: Denmark
Contact:

Re: clamwin too slow - hmailserver crash

Post by Tooms » 2009-10-17 21:08

tBB wrote:
Tooms wrote: it looks like your code is just calling the clamscan.exe what about make editable so you just can enter the commandline and maybe use my exe.
Is there any benefit in using your tool compared to ClamDScan?

Best regards,

Nico
the Clamdscan.exe is just trying to send the file to the server and if this fail it will just exit with a error, my version is having alot safety checks in the code and can use multi clamd servers so if one fails then the mswclamdscan will automatic connect to another server.

my version is design to work as plugin for high load mailsscanner and can use multi clamd servers for load sharing.
it can also log found virus to a sql database so you can have a nice webpage with some stat, this is a demo of the webpage there is include with mswclamdscan
http://www.tooms.dk/mswwebtopxviruslist/Default.aspx
Any comment or statements is my own and have no relationship to my workplace

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: clamwin too slow - hmailserver crash

Post by tBB » 2009-10-17 21:34

Tooms wrote: my version is design to work as plugin for high load mailsscanner and can use multi clamd servers for load sharing.
I see. Thanks, I was just curious.

Best regards,

Nico

Post Reply