ClamAV service fails periodically

[mibyge]

Hello.

I've set up ClamAV as a service as described in viewtopic.php?f=21&t=26829 and it's generally speaking working quite well.

However, I'm periodically seeing what seems to be ClamAV having an issue with loading the AV definitions. This results in the service failing, memory usage is going up and down at every attempt and also significantly increased CPU usage on the server as shown below where the issue occurred a full day from 20:00 to 20:00.

localhost_phymem-day.png (3.02 KiB) Viewed 9412 times

localhost_cpuload-day.png (3.63 KiB) Viewed 9412 times

The clamd.log file contains tons of errors like the one below:

Code: Select all

+++ Started at Sun Jan 29 18:57:11 2017
clamd daemon 0.99.1 (OS: win32, ARCH: i386, CPU: i386)
Log file size limited to 1048576 bytes.
Reading databases from C:\ProgramData\.clamwin\db
Not loading PUA signatures.
Bytecode: Security mode set to "TrustSigned".
ERROR: Invalid argument passed to function

I'm downloading the following additional AV definitions from sanesecurity:

securiteinfo.hdb
securiteinfo.ign2
javascript.ndb
securiteinfohtml.hdb
securiteinfoascii.hdb

The reason that I suspect the AV definitions is because it always begins and ends at a time where the task to update the definitions is run.

The server is a Windows Server 2008 SP2 32bit (fully patched) with hMailServer v5.6.5 build 2367 and ClamAV/ClamWin v0.99.1

Have anyone seen this issue before?

Thanks in advance.

[jimimaseye]

My reply isnt going to help much Im afraid.

I dont have the problem and the only difference (except OS 32 or 64 bit versions) is you are running some definitions that I dont.

You say:

definitions from sanesecurity:

securiteinfo.hdb
securiteinfo.ign2
javascript.ndb
securiteinfohtml.hdb
securiteinfoascii.hdb

but I see no Sanesecurity definitions and only securiteinfo definitions.

I know from discussions in the ClamAV forum/mail list that securiteinfo are extremely memory hungry - maybe this is related.

The only thing I could suggest is strip back your definitions back to using the defaults and apply them one-by-one starting with just the default, test, then add Sane Defs (as recommended in my original post), test, and then add securiteinfo defs and test again. (FWIW I dont think you need securiteinfo if you are running the Sane defs and they are a lot lower on memory resources too).

[mibyge]

The only thing I could suggest is strip back your definitions back to using the defaults and apply them one-by-one starting with just the default, test, then add Sane Defs (as recommended in my original post), test, and then add securiteinfo defs and test again. (FWIW I dont think you need securiteinfo if you are running the Sane defs and they are a lot lower on memory resources too).

My apologies. I seem to have mixed up the SaneSecurity signatures and the securiteinfo signatures

I've now removed the securiteinfo signatures and instead implemented the SaneSecurity signatures, and will monitor the performance for a couple of days.

Thanks for the tip

[agatha]

I've now removed the securiteinfo signatures and instead implemented the SaneSecurity signatures, and will monitor the performance for a couple of days.

I use those signatures for years and have no problems at all. The memory usage is about 1.3 GB.

But what about the message "invalid argument"? Maybe the problem is located in your freshclam.conf or clamd.conf?

[jimimaseye]

I use those signatures for years and have no problems at all. The memory usage is about 1.3 GB.

Without Securiteinfo, mine is 488mb. And this concurs with these findings: http://lists.clamav.net/pipermail/clama ... 03907.html. Securiteinfo is 293mb extra (in those findings) some 3.5x times bigger than the Sane sigs footprint alone. (See the whole thread from http://lists.clamav.net/pipermail/clama ... 03903.html)

Why yours is running at 1.3GB (when the above link reports 750mb total) heaven only knows - that seems a lot.

[agatha]

I am not sure. Probably the clamd.conf is the reason.

I use the recommendation from securite.info and they suggest at least 8 GB RAM for this.

Code: Select all

DetectPUA yes
IncludePUA Spy
IncludePUA Spyware
IncludePUA Game
IncludePUA Keylogger
IncludePUA Spam
IncludePUA Trojan
IncludePUA NetTool
IncludePUA Win
MaxScanSize 450M
MaxFileSize 450M
MaxEmbeddedPE 100M
MaxHTMLNoTags 10M
MaxScriptNormalize 10M
MaxHTMLNormalize 10M
MaxRecursion 30
ScanXMLDOCS yes
ScanHWP3 yes
MaxRecHWP3 32
PCREMatchLimit 20000
ScanXMLDOCS yes
DetectBrokenExecutables yes

[jimimaseye]

MaxScanSize 450M
MaxFileSize 450M

WOW!!

Clam, by default, is only set to 25MB (or something like that). It is highly unlikely you have any malware coming in on email that is remotely close to being 5MB (some say even 1MB) least alone 25. And yet, you are allowing 450mb. Do you have HMS sending large files for scanning as well or do you have it limited (Antivirus - General - Maximum Message size. I think by default it is set to Zero/unlimited). If you are sending stupidly large emails then that might explain it?

https://linux.die.net/man/5/clamd.conf

MaxScanSize SIZE

• Sets the maximum amount of data to be scanned for each input file. Archives and other containers are recursively extracted and scanned up to this value. Warning: disabling this limit or setting it too high may result in severe damage to the system.
  Default: 100M

MaxFileSize SIZE

• Files larger than this limit won't be scanned. Affects the input file itself as well as files contained inside it (when the input file is an archive, a document or some other kind of container). Warning: disabling this limit or setting it too high may result in severe damage to the system.
  Default: 25M

[agatha]

Hm. To be honest, I just copied the recommendations.

But I have just tested it with 45M each - the same RAM usage.

[jimimaseye]

Is that immediately after a clamav service restart?

[agatha]

Is that immediately after a clamav service restart?

Depends, how you define "immediately". After the service is restarted, it takes a few seconds (about 5 maybe) until this the RAM is filled.

[jimimaseye]

MY thought was that maybe the receiving of large files for scanning would increase the RAM usage and once allocated, it remains (a working cache/allocation). Meantime, after a restart and the loading of the service and definitions, and before the scanning of its first email, it may have a lower memory usage.

It is quite strange that it is so high.

You have prodded my curiosity now. Im going to turn to the ClamAV mail list for opinions. (Im not well liked there by the Clam Project leader, Joel Esler, due to my negative reviews and usefulness of their default base signatures. Good luck to me! )

EDIT: There you go. http://lists.clamav.net/pipermail/clama ... 04080.html

[agatha]

You have prodded my curiosity now.

How large is your ClamAV database folder? Mine is 666 (!) MB.

And good luck with Joel Esler - the signatures are not worthless - but not very reliable.

[jimimaseye]

How large is your ClamAV database folder? Mine is 666 (!) MB.

Well Im not a good comparison because I dont use securiteinfo (only default + Sane). But for what its worth.....




EDIT:

Oh, and by coparison for you, this is my Clamd.conf:

Code: Select all

TCPSocket 3310
MaxThreads 2
LogFile C:\Program Files (x86)\ClamWin\bin\clamd.log
DatabaseDirectory C:\ProgramData\.clamwin\db
LogTime yes

Somewhat shorter than yours.


(My sane signames.txt):

Code: Select all

sanesecurity.ftm
sigwhitelist.ign2
phish.ndb
badmacro.ndb
rogue.hdb
foxhole_filename.cdb
foxhole_generic.cdb
scam.ndb
junk.ndb
jurlbl.ndb
blurl.ndb
lott.ndb
spamattach.hdb
spamimg.hdb

[agatha]

Code: Select all

sanesecurity.ftm
sigwhitelist.ign2
rogue.hdb
junk.ndb
foxhole_filename.cdb
foxhole_generic.cdb
foxhole_all.cdb
foxhole_all.ndb
foxhole_js.cdb
foxhole_js.ndb
phish.ndb
badmacro.ndb
jurlbl.ndb
scam.ndb
mbl.ndb
winnow_malware.hdb
winnow_extended_malware.hdb
crdfam.clamav.hdb

This is quite strict, as it removes any js but i do not have many false positives.

[jimimaseye]

I am not sure. Probably the clamd.conf is the reason.

Code: Select all

DetectPUA yes
IncludePUA Spy
IncludePUA Spyware
IncludePUA Game
IncludePUA Keylogger
IncludePUA Spam
IncludePUA Trojan
IncludePUA NetTool
IncludePUA Win
.
.
.

Are you sure these are all valid?? From the various manuals and documentation I have seen they are incorrectly entered.

For example, according to http://www.clamav.net/documents/potenti ... ations-pua, the options for "IncludePUA" are only

Packed
PwTool
NetTool
P2P
IRC
RAT
Tool
Spy
Server
Script

I am trying to find references to the rest that you have listed but cant. (It is CLAM after all - never really up to date with things generally and so their documentation is VERY likely to be old and out--of-date). Where did you get your references from?

[agatha]

Where did you get your references from?

securite FAQ:

What is the best configuration for clamd.conf ?
To achieve maximum detection rates, we recommend modifying the following lines in your clamd.conf :
WARNING : These changes suggest that you have at least 8GB of RAM
DetectPUA yes
IncludePUA Spy
IncludePUA Spyware
IncludePUA Game
IncludePUA Keylogger
IncludePUA Spam
IncludePUA Trojan
IncludePUA NetTool
IncludePUA Win
MaxScanSize 450M
MaxFileSize 450M
MaxEmbeddedPE 100M
MaxHTMLNoTags 10M
MaxScriptNormalize 10M
MaxHTMLNormalize 10M
MaxRecursion 30
ScanXMLDOCS yes
ScanHWP3 yes
MaxRecHWP3 32
PCREMatchLimit 20000
ScanXMLDOCS yes
DetectBrokenExecutables yes

I am not sure, if that is correct as I did not check it. But to me it seemed plausible.

[jimimaseye]

I suspect that is the cause then. Lots of over-cautious stuff to the detriment of RAM but at least it doesnt cause you a problem.