Whilst clamwin seems to play nicer on win32 than sawin32 does (just my opinion), it would be good to be able to have an option to use clamd running on another machine, either real or virtual like can be done with spamassassin (spamd). I have spamd running on a Linux virtual machine with better perfomance than running sawin32 locally. I would like to run clamd in the same way, either on the same machine or in its own virtual or real machine if required. (optionally you could just use clamd on windows which would do away with any virtual machines if they aint your cup of tea)

Technically, you cannot do that the same way that it is done with SpamD. The client code for ClamAV passes the file path as a string to ClamD. It does not pass the entire file like SpamC does to SpamD. So you cannot offload ClamAV the same way to another machine. I wish you could, but you can't.

What you can do (what we already do) is run ClamD on the same Linux boxes as SpamD and set up a spam rule for running ClamAV. Any message that triggers the ClamAV "spam rule" gets 200 points added to its SA score. And ClamD runs on the Linux based SpamD boxes, so all of the CPU usage of SpamD and ClamD is offloaded to the Linux boxes. This is the only solution I have found so far for offloading ClamAV.

If you are interested in setting up ClamAV as a SA rule, check out this Wiki article:
http://wiki.apache.org/spamassassin/ClamAVPlugin

Recommended changes:
In the sample clamav.cf file, change 10 to 200 if you want to have virus to get an SA score of 200 instead of just 10.

That is wrong, you can send files to a remote ClamD server over a network without the need to shared storeage.

My plugin call MSWclamDscan.exe that i have made for the Mailsweeper scanner software is sending files over a TCP connection to a remote Linux server with ClamD running.

http://www.tooms.dk/Tblog/Showblog1.asp ... 2053506046

http://www.tooms.dk/software/mswclamdscan/default.asp

The software is still beta code so only use it for testing.

Tooms: I would love it if the core ClamAV development would absorb that code so this feature could be implemented. Unfortunately, the current ClamAV code (without this mod) does not allow for this. That would be a terrific improvement for ClamAV.

I have a quick question, though. Does the MSWclamDscan mod need to be implemented in both the client and server, or just the client? I have not used MSWclamDscan and I am assuming it would need to be done on both ends, but the instructions with MSWclamDscan seem to say that a standard installation of ClamD will work.

The ClamD server support this as standard, you just have to enable it in the config whene you tell it to listens on a port.
I don't know if this work in every windows port of the ClamD, but i have tryed a number of them and there it is working.
I my docs with the MSWclamDscan app there is a list of know port that i know is working or has been reported by others to work.

The code side of it is much the code they use to day, instedt of tell the ClamD server where the file is on disk then you tell it to "stream" and it will return a port number where you send the file to, so there is no big magic in this. If you look into the MAN page of ClamD there you can see information about how to do this.


Until they include a native version of ClamDscan into the Hmailserver, then you can use my MSWclamDscan as commandline scanner.
My version of ClamDscan support multi ClamD server to make it more safe as if one ClamD server fail then the MSWclamDscan will reconnect to the next server and it use then as load share so if you have high mail load then just run more ClamD servers to share the load, MSWclamDscan also have number safty checks and timeout function to make sure the scanning of content is working.

Hope you can use it.

I would also really like if i could run clamAV on a separate Linux box!

What stops you from doing it right now? ClamD doesn't have to run at the same machine, ClamDScan can connect to a remote ClamD daemon as well. If you have more than one ClamAV server it might make sense to run the above mentioned tool of Tooms instead of ClamDScan.

Best regards,

Nico

ohhh i didn't read it clearly....is there a "how to" post?

In case you're referring to how ClamDScan (or Tooms tool) can be implemented as external scanner in hMS I think there are several threads/posts here (I'd suggest a search). Just don't mix it up with ClamWin, that is something completely different.

Best regards,

Nico

I couldn't find a how-to but there is a lot of threads about clamAV and phpBB's search isn't the best so I may have just missed it.

There are no special switches for ClamDScan as it just reads the clamd.conf file. If you want ClamDScan to use a remote ClamD you have to adjust 'TCPSocket' and 'TCPAddr' in the local clamd.conf to reflect the IP/port used by the ClamD server and all should be fine.

Best regards,

Nico

Again, ClamD does NOT have to run on the same machine. It doesn't even need to be there at all, also of course databases don't have to be on the local machine if ClamD isn't running.

The stream protocol Win32 ClamD/ClamDScan uses is exactly the same as in the Linux version.

There are several third party signature update scripts for Linux, see http://www.sanesecurity.co.uk/download_ ... _linux.htm I'd recommend the one from Bill Landry.

Best regards,

Nico

Please note that i have release a new version
http://www.tooms.dk/Tblog/Showblog1.asp ... 2306513681
http://www.tooms.dk/software/mswclamdscan/default.asp

hope you can use it.

"ClamD" is the server part that can be installed on a aother pc in the network or on the same server as hmailserver.
ClamD server can be the linux version on a linux server or it can be on windows where you can use Nico's windows port of ClamAV.
On the ClamD server you simple set the option "TCPsocket=3310" in the ClamD.conf config file and restart the server, so now it can be used over the network.
if you set the "TCPaddr" you will lock it to that interface and if you not set it then it will listen on all interfaces.

"ClamDscan" is the client there is using the "clamD" server over the network
When using the ClamDscan client you then have to tell it where the server is on the network.
My app MSWclamdscan is a clone of "clamDscan" but it is design to be used in highload mailscanner and can use multi clamd server to share the load and more safe so if one clamd server breck down or is updating it patten files then mswclamdscan just fail over to another clamd server with the MTA(hmailserver) ever know it.

my tool MSWclamDscan is only the client part so you have to install a clamD server somewhere and that can be on linux or on windows with Nico's ClamAV.

if you use my MSWclamDscan tool then just follow the guide for installing and when it is working you can then setup hmailserver to use the MSWclamDscan as a antivirus scanner.

1.
go to: hmailserver gui->Settings->Antivirus->select tab "External virus scanner"

2.
check the "use external scanner"

3.
In the textbox "scanner executable" inset "d:\mswclamdscan\mswclamdscan.exe /mail:"%FILE%""
note the qwote around the %file% must be there as the mail path has space in it.

4.
set "Return value" to 1

5.
goto the Status page and restart the hmailserver to reload the new config.

6.
verify that it now detect virus mails by setting some virus test mails via hmailserver


I think this is it

Webpage with virus found.

If you download my mswclamdscan and set it up then you in the config gui will see a part to setup a database for the logging and when the mswclamdscan detect a virus when scanning then it will read the config from the reg and if it is config to log the information then it will inset a entry to the database.

The database with the virus log is very simple and you can easy make your own page to display the information the way you like or you can use the sample page that i have include and it has full sourcecode.
To see a live demo of this page then goto http://www.tooms.dk/ and click on "E-mail Virus Stat" in the menu

For now the database can only be a Mircosoft database like msde, mssql 20xx

The data is simple stored in a standard MS database so you can make a page in any format you like as long as it can read the database and build a webpage.

I have include a .aspx webpage with full sourcecode in the mswclamdscan pack.

Some day i maybe will include mysql option if the request if high for it.
You know time is limit when it is ones freetime

It was not a two-set double quotes it was one double quotes around the %FILE% and another set around the whole commandline but i can see this was misleding and wrong, but good to hear that it is working.

the "testmail_spam1.msg" is only a sample spam mail and does not include a virus so i hope when say it "detecting it with no problem" you mean that it scan the mail file with on error and return exitcode 0 (no virus found)

MAXSTREAM, instedt of setting this to 0 then it is better to set a byte vaule there is a bit higher then mswclamdscan max filesize value is set to. This way it better protect the server.

I new to hmailserver so i hope you can help me.
the mswclamdscan plugin can return a logfile with one log line with the status and virus name, do you know how to return this information back to hmailserver and use it in hmailserver like in a reply mail to sender or/and as information to the mailadmin.

Yes thanks that is a good answar

I hade hope there was a way to return the Virus name for logging information, but maybe this will come some day.

Your antivirus may log the virus name. Mine certainly does.

I think what he was asking for is if there's a way to pass the name of a found malware back to hMS so that the sender or admin can be notified.

Best regards

Nico

Yes that what i was looking for

That's what I expected.

I believe that this is NOT currently possible.

My Antivirus (Avast Corporate, via ADNM) sends me an e-mail every defined period (in my case weekly) detailing which computer, which virus, how many etc . All complete with graphs and my company letterhead.

You could parse your antivirus logs and create such a report. I do similar things with other log files, for instance I track RDP logons by username, originating IP address etc from the IIS logs.

Matt

Thanks

If external tools are allowed to write to the mail a X-ClamAV header could be added which contains the virus name. Based on this header further filtering/processing should be easily possible. IMO sending bounces to the outside is a bad idea but local users should be provided with all necessary informations in realtime (including the virus name).

Best regards,

Nico

Sorry, my bad English (and my only language at that)

I meant to say 'I believe that this is NOT possible' or 'I DON'T believe that this is possible', ie you can't do it.

You can not pass the virus name directly back to hMailserver from Clam.
You can to to headers as tBB suggest though.

AVAST also adds headers 'X-Antivirus' which is Avast + current virus database number + 'inbound' vs 'outbound' and 'X-Antivirus-Status' which is 'clean' or 'infected + virus name.

I guess this is something different. I don't use Avast but presumably it either injects itself into the LSP stack or it installs a hidden proxy over which all traffic is routed but both methods are applied before hMS even sees the mail (or in case port 110 is monitored after hMS processed the mail) so I'm still not sure if external processes are allowed to write directly to the mail when it's already in hMailServer's queue. Unfortunately I don't use hMS myself in production yet and have only played around with it so I can't tell much but perhaps someone can clarify *coughmartincough*

If external processes are able to add headers to the mail then it wouldn't be much of a problem to parse the virus name out of ClamDScan's output (I wouldn't try to parse the log) and add a header. It could be even done by a simple batch file which is called instead of ClamDScan and a external tool which adds the header. I could even provide such thing, should anyone be interested.

Best regards,

Nico

As MSWclamdscan is simply a commandline tool there is reading the reg for config and two mswclamdscan can not interact.
So the "load sharing" is simply that it pick a random server from the server list and if that fails to give a good respone back then it will pick another server from the list until it get a good response or it hit the max retry limit, but all limits and timeout value can be config from the GUI


to see what happen then try setup 2 or more ClamD servers and add then in the MSWclamDscangui and then run this commandline a number of times..

c:\mswclamdscan>mswclamdscan.exe /mail:Name_Of_Some_mail.eml /debug /devdebug

note that it will at random pick a server to do some sort of load sharing.


Now try shutdown now of the ClamD servers on the list and run the commandline again.

c:\mswclamdscan>mswclamdscan.exe /mail:Name_Of_Some_mail.eml /debug /devdebug

Now note that when it try use the server that is down then it will timeout waiting for the server to response and then pick another.


I am playing with a monitor app for clamd that dynamic can enable/disable servers from the server list so the mswclamdscan not wasting time on the down server and the app will also alart the admin about there is a issue, but this app is long away.



Hoe this made it more clear.

My MSWspamC client is also supporting multi SpamD server and will readom pick one from the list to load share.

note if your running SpamD and/or ClamD on linux platform i don't think you will have any performance issue the load sharing is more for safty so if one server goes down then it will not halt the mail trafic.

ClamD supports multiple threads so as long as the number of connections don't exceed the MaxThreads value configured in ClamD.conf it will always give a good response.

However, it might happen that ClamD is reloading it's databases if there was a update at the time a connection comes in and during this time it doesn't scan anything (of course). Depending on the hardware it might also take way more than 30 sec. to load and unpack databases, specially if there are a lot of third party signatures involved so if 30 sec. is the hardcoded timeout I would really suggest to make that configurable.

Best regards,

Nico

It does, but the upcoming hMS interaction with ClamAV probably won't and that's what I was referring to (and what this thread is about).
Do you think it's better if a mail isn't scanned at all? The definition of a loaded server depends pretty much on the hardware. A Pentium3-500 on a dialup line can be probably called 'under load' with 10 mails per minute while I call a load +10 mails per second. However, all that could happen is that hMS builds up a queue and I think there is no limit except the available space.

Therefore the processing timeout values of a mailserver should be IMO adaptable to the hardware/connection speed.

Best regards,

Nico

I guess Tooms and me would quickly monitor the new thread as well

However, as Martin stated that the feature will be integrated this poll is probably useless anyway. We could start a new one about configurable timeout values though

Best regards,

Nico

Hmmm..I fear this will have little to no effect on the timeouts because the calling of ClamDScan.exe doesn't cause that much CPU overhead. Even less if it's already in Win32's cache (which it most probably is because it's called quite often at a mailserver). What would help in this case is, well, a configurable timeout value

Hehe. Well, I don't use hMS in production yet so IMO I shouldn't start a poll. Feel free to do so if you like

Best regards,

Nico

Not sure what "Gregory House moment" means but let me quote myself:

So it's not ClamAV that times out, it's hMS!
Related to ClamAV? ClamAV has to load and unpack a huge database. If third party signatures are loaded which can be quite complex it takes even longer. Other virus scanners also don't scan anything during they reload their databases and queue the connections and other mailservers handle that as well.

Best regards,

Nico

Well, I know who Dr. House is but if you had googled for your own sentence http://www.google.com/search?hl=en&q="Gregory%20House%20moment" you would have found that there is exactly one hit which explains pretty much nothing.

Simply: It's not ClamAV which timeouts because it has it's own definition of several types of timeouts (which are btw. configurable) and it clearly says in it's output/log/stream response if it timeouts. Otherwise, applications which call ClamAV have to wait until ClamAV has finished the scan, or until ClamAV indicates that it had a timeout, or it's simply not ClamAV's problem if it finishes the scan like you said in a "timely fashion" but as I have stated now numerous times, it's IMO impossible to give a general statement (or hardcoded value) about what a "timely fashion" is because it depends on your hardware. If you have a server with a quad core and a raid system below you won't even notice if ClamD reloads it's databases so, to copy your own statement:
So, according to yourself the hardware on which your ClamAV installation runs is apparently not fast enough for hMS and it's hardcoded timeout value

Best regards,

Nico

Dude, not everyone on the planet is a fan of this TV series. I am clearly not and as such, I had no idea what you mean. However, as soon as someone starts to quote dictionaries and/or google, any discussion from this point on is usually moot anyway.
OK, I'll try to explain one last time: ClamAV is not unresponsive or hangs or whatever you try to emphasize. It queues the incoming connections until the reloading of the database is finished, then it scans them. Period.

Nico

No the MSWclamdscan is design to be fast, small, simple and safe, so there is not sort of check of the server it will simple pick one from the list and try it and if it fail then it will try another if it is below the max_scan_time value.
So if your saying the max scan time allowed in HMS is 30 sec. then config mswclamdscan to use max scan time as 30 sec or less, but you have to check that the biggest file size you allow can be scan within the 30 sec.

If you have download my mswclamdscan pack then try run the config gui mswclamdscangui.exe and add your ClamD servers and then select one server and click on verify, it will now try to test the server and tell you how long time it use to scan the biggest file size allowed by the GUI




How Martin is making the SpamC client and maybe the ClamDscan client in HMS is some thing he only can answer.

Horndag, I am still new to HMS and learning also i am only one privat person there is make my tools in my free time, so i can not keep up will all the requests

But i understand that you like to scan incoming mails and reject the bad ones before they get in, I think can easly be don as i have seen other scripts on the forum where there is run a .exe and then use it exitcode to do the action.

I use mswclamdscan at work and we have to ClamD linux servers as backend and i have jet to see a fail mail do to database updating as the mswclamdscan simple retry until it is a good scan response or max scan time is up(can be config via gui)

from time to time i also rebooting and updating the ClamD linux servers and mail flow just goes on and there is no errors.

I am only one human and will try answer what i can and yes maybe it is time to move this long talk to it own thread and not hijack this one as it seems more about my mswclamdscan then "Interface to ClamD just like the one for SpamD"



I opt for that all mails most be scan without exception.

I try to keep up but you guys are fast here

From my understading it is one topic that the ClamAV dev. team is working one to make it load faster and when there comes new updates it will load the delta updates in the background and without stopping the scan low, so there is hope that this will get better in newer versions

http://www.clamav.net/about/roadmap

Best regards,

Nico

All the information about the commandline and exitcode can be found in the docs of mswclamdscan.

mswclamdscan.exe /mail:FullPath2MailFile.eml

Exitcode 0 = no bad things found
Exitcode 1 = Virus found
Exitcode 2 = Some error happen.

Accidentally I came across this thread viewtopic.php?f=10&t=16635 which is about AV scanner timeouts in hMS. Now I'm somewhat confused if there is a hardcoded 30 sec. timeout or not. Could someone who knows for sure maybe shed some light?

Best regards,

Nico

if you look at this post then you can see most if what your asking for, so this is a good start point.
viewtopic.php?f=14&t=14916

mswclamdscan will not modify the mail file in any way and this is by design to make the app very safe.

if you use clamdscan there is maybe a option to let it add the header line, but mswclamdscan is design not to modify the mail.

Try look at the docs for clamdscan

mswclamdscan only return a exitcode and if the commandline option is used then it also can return a logfile with a single line for text.

will be better if there was a option for the enduser to set the timeout value and also set what exitcodes there is errors and not only the detected exitcode as today

Ah I see, a new hMS version was released after that date. Could you perhaps give a rough estimate about how the available 30 seconds are divided between the processes (DNSBL lookups, ClamAV, SpamAssassin)? I have a nagging suspicion that SpamAssassin is the process which takes the longest time (at least if it's the PerlApp Win32 version from sourceforge) so it might be worth trying to speed it's execution up.

Best regards,

Nico

Such batchfile would be indeed not that hard. If I remember right I have already offered somewhere in this thread to provide some batch wrapper for ClamAV which parses the output/evaluates the errorlevel and adds a X-Header to the mail which contains the virus name. I assume supporting SpamC or Tooms programs would be not much different.

Best regards,

Nico

There is a time stamp but you're probably looking in the wrong log (or it is disabled because this is the default). Look for the following lines in FreshClam.conf...

# Path to the log file (make sure it has proper permissions)
# Default: disabled
# UpdateLogFile c:\clamav\log\freshclam.log

# Log time with each message.
# Default: no
#LogTime yes

# Enable verbose logging.
# Default: no
#LogVerbose yes

..and enable what you see fit.


I thought it. This one is "compiled" by using PerlApp from Activestate. PerlApp wraps a whole 'Mini-Perl' package around the main Perl application which is unpacked to the system's temp path when the application is called (if compiled without some switches every time the application is called) and used from there. So, what you can do to speed SpamD up if you have the possibility is installing a ramdisk with a size of let's say +512 Mb and set the system's global temp path (NOT the pagefile!) to a temp folder on the ramdisk.

For such purposes I'm using the free 'Gavotte' ramdisk which is based on Microsoft's public ramdisk sources but has some nice additions. For example it can allocate the memory between 3.2Gb and 4Gb on 32 Bit systems for the ramdisk or it can create a image of the ramdisk with all paths and contents (e.g. the unpacked PerlApp SpamD) and reload it at every boot. You can get it here: http://www.megaleecher.net/RAMDisk

BTW: If you also set ClamAV's temp path (clamd.conf) to some folder on the ramdisk you will get another noticeable performance boost.

Best regards,

Nico

I get a timestamp in my clamd.log

The relevant config is clamd.config

You need to uncomment what you want by removing the leading #

Been watching this thread...

Just noticed that my clamd service wasn't running for a month. (it is my secondary AV anyway)
I haven't notice any time out issues in the hmailserver logs. What should I look for?

FreshClam.log shows the time stamp as well:

Best regards,

Nico

Best regards,

Nico

wow that was slow, i have a VM with a linux and SA and when i send a mail to it via a SpamC client then it scan in 1 sec and some times 2 sec.

I think you better try to debug your SA setup and see if it hangs in some old RBL lists or some thing like that as i have before seen SA wait and timeout on RBL or other things and that giv that long scan time and when outcoment the bad checks then SA was much faster.

That's exactly what I suspected. I was porting SA myself using PerlApp since version 2.something to 3.something for a company until I gave up because usually a week after a version was ported and implemented, the SA guys came up with a new minor version.

The Linux version is that much faster because AFAIK there's a resident Perl daemon running. Even when using the mentioned ramdisk trick the Win32 SpamD won't come even close to the processing time of the Linux version.

If you are using third party rules like for example the SARE rules from http://www.rulesemporium.com this can also slow down the processing a lot.

Last but not least, as Tooms mentioned you should check if you're using a unresponsive RBL.

Best regards,

Nico

EDIT: As far as I remember the SA port you're using also has a couple of plugins integrated. Some of them are slowing down the process a lot, e.g. the OCR plugin (which is meanwhile almost useless anyway).