Antivirus not delete attachment for CC account

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
paolo_c
New user
New user
Posts: 16
Joined: 2021-05-04 12:23

Antivirus not delete attachment for CC account

Post by paolo_c » 2021-05-04 13:13

I have configured hMail to use Windows Server Security Antivirus and set the option to delete attachment. When an email is received and have a CC and both recipients are on my server, one of recipient receive the email without attachment and with warning in the message, but the other recipient receive the normal email with attachment and no warning message. The attachment is a PDF received always from the same sender and it seems clean.
Antivirus is configured as: "C:\Program Files\Windows Defender\MpCmdRun.exe" -scan -scantype 3 -file "%FILE%" -disableremediation"
Return value: 2

User avatar
RvdH
Senior user
Senior user
Posts: 1422
Joined: 2008-06-27 14:42
Location: Netherlands

Re: Antivirus not delete attachment for CC account

Post by RvdH » 2021-05-04 13:39

Windows Defender flags harmless files regularly as being infected, that is caused because it makes no distinction between: infected and failing scans
eg: if a scan fails for whatever reason it will return code: 2 causing hMailserver to process as being infected

Plenty of topics here on the forums that discuss this....so it basically comes down to a bad choice as VirusScanner
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

paolo_c
New user
New user
Posts: 16
Joined: 2021-05-04 12:23

Re: Antivirus not delete attachment for CC account

Post by paolo_c » 2021-05-04 16:59

Thanks for your replay! Could be possible to use the parameter -ReturnHR instead of code? Or do you know some workaround? I searched in the forum but I found nothing, can you suggest any link. Thanks!

mikedibella
Senior user
Senior user
Posts: 464
Joined: 2016-12-08 02:21

Re: Antivirus not delete attachment for CC account

Post by mikedibella » 2021-05-04 17:10

You could wrap the execution of MpCmdRun.exe in a script and read and parse the console output, looking for output strings that differentiate between clean scan, failed scan, and infected scan results. You need to simulate each result to see what output is produced and devise the filtering criteria for each.

paolo_c
New user
New user
Posts: 16
Joined: 2021-05-04 12:23

Re: Antivirus not delete attachment for CC account

Post by paolo_c » 2021-05-04 17:40

Thanks, great idea! But I'm still unsure about the best action to take in case of failed scan (0x80070002). It is safe to consider the file clean?

mikedibella
Senior user
Senior user
Posts: 464
Joined: 2016-12-08 02:21

Re: Antivirus not delete attachment for CC account

Post by mikedibella » 2021-05-04 17:55

The only way you'll avoid the "split decision" you describe is to catch failed scans and retry them, hoping that only one retry is needed to reach a pass/fail. You can't retry infinitely without creating blocking condition in mail processing.

paolo_c
New user
New user
Posts: 16
Joined: 2021-05-04 12:23

Re: Antivirus not delete attachment for CC account

Post by paolo_c » 2021-05-04 18:15

Yes but one common reason for failed scan is files with invalid chars. Do you think if the script rename the file, retry to scan and then rename again, this can create some problem to hMailServer? And do you know if hMailServer have a timeout for return code?

mikedibella
Senior user
Senior user
Posts: 464
Joined: 2016-12-08 02:21

Re: Antivirus not delete attachment for CC account

Post by mikedibella » 2021-05-04 18:26

IIRC %FILE% passes the filename of the .eml file, not the attachment. So your script could use the hMailServer API to save each attachment to a temporary file, delete it from the message, rename the file if a problem filename is detected, scan the file, and only re-attach the file if it is clean. Repeat for multiple attachments.

paolo_c
New user
New user
Posts: 16
Joined: 2021-05-04 12:23

Re: Antivirus not delete attachment for CC account

Post by paolo_c » 2021-05-04 19:26

Antivirus normally do not scan just .eml file? I can't just rename .eml file before to scan it and let hMailServer to deal with attachment as usual?

mikedibella
Senior user
Senior user
Posts: 464
Joined: 2016-12-08 02:21

Re: Antivirus not delete attachment for CC account

Post by mikedibella » 2021-05-04 19:48

No. The .eml filename is a guid that is linked to a database record.

User avatar
RvdH
Senior user
Senior user
Posts: 1422
Joined: 2008-06-27 14:42
Location: Netherlands

Re: Antivirus not delete attachment for CC account

Post by RvdH » 2021-05-04 19:56

mikedibella wrote:
2021-05-04 18:26
IIRC %FILE% passes the filename of the .eml file, not the attachment. So your script could use the hMailServer API to save each attachment to a temporary file, delete it from the message, rename the file if a problem filename is detected, scan the file, and only re-attach the file if it is clean. Repeat for multiple attachments.
Doesn't hMailserver do that automatically? See here
I sometimes see entries like this:

Code: Select all

"DEBUG"	9820	"2021-05-04 15:55:35.564"	"Running custom virus scanner..."
"DEBUG"	9820	"2021-05-04 15:55:36.184"	"Scanner: "C:\Program Files\hMailServer\Addons\ZipScanner\ZipScanner.exe" -u="Username" -p="Password" -z -b -f="C:\Program Files\hMailServer\Data\{1D920023-D7DB-481F-A157-0C10714E0895}.eml". Return code: 0"
"DEBUG"	9820	"2021-05-04 15:55:36.355"	"Running custom virus scanner..."
"DEBUG"	9820	"2021-05-04 15:55:36.855"	"Scanner: "C:\Program Files\hMailServer\Addons\ZipScanner\ZipScanner.exe" -u="Username" -p="Password" -z -b -f="C:\Program Files\hMailServer\Temp\{DAE2DB7B-FEE4-4FBF-9E88-027D2C0BB4AE}.tmp". Return code: 0"
those *.tmp files can be more then one, I always assumed these were attachments (files, pictures etc)
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

paolo_c
New user
New user
Posts: 16
Joined: 2021-05-04 12:23

Re: Antivirus not delete attachment for CC account

Post by paolo_c » 2021-05-04 20:44

RvdH wrote:
2021-05-04 19:56
mikedibella wrote:
2021-05-04 18:26
IIRC %FILE% passes the filename of the .eml file, not the attachment. So your script could use the hMailServer API to save each attachment to a temporary file, delete it from the message, rename the file if a problem filename is detected, scan the file, and only re-attach the file if it is clean. Repeat for multiple attachments.
Doesn't hMailserver do that automatically? See here
I sometimes see entries like this:

Code: Select all

"DEBUG"	9820	"2021-05-04 15:55:35.564"	"Running custom virus scanner..."
"DEBUG"	9820	"2021-05-04 15:55:36.184"	"Scanner: "C:\Program Files\hMailServer\Addons\ZipScanner\ZipScanner.exe" -u="Username" -p="Password" -z -b -f="C:\Program Files\hMailServer\Data\{1D920023-D7DB-481F-A157-0C10714E0895}.eml". Return code: 0"
"DEBUG"	9820	"2021-05-04 15:55:36.355"	"Running custom virus scanner..."
"DEBUG"	9820	"2021-05-04 15:55:36.855"	"Scanner: "C:\Program Files\hMailServer\Addons\ZipScanner\ZipScanner.exe" -u="Username" -p="Password" -z -b -f="C:\Program Files\hMailServer\Temp\{DAE2DB7B-FEE4-4FBF-9E88-027D2C0BB4AE}.tmp". Return code: 0"
those *.tmp files can be more then one, I always assumed these were attachments (files, pictures etc)
Ok now I'm more confused...
On this thread ( https://answers.microsoft.com/en-us/pro ... bfc2657ff6 ) I found a post by [RvdH] where he claim: "...each e-mail delivered through hMailServer is saved in a temporary directory with unique name which consist of a random identifier and the email's subject."
And this log is quoted:

Code: Select all

424 "2014-06-07 13:30:04.957" "THREAT FOUND"
424 "2014-06-07 13:30:04.957" "File: "C:\Program Files (x86)\hMailServer\Temp\2014060713300489_Re ****RE****""
424 "2014-06-07 13:30:04.957" "Command: "C:\Program Files\Microsoft Security Client\MpCmdRun.exe" -Scan -ScanType 3 -File "C:\Program Files (x86)\hMailServer\Temp\2014060713300489_Re ****RE****" -DisableRemediation"
424 "2014-06-07 13:30:04.957" "Return Code: 2"
Looking at your log it seems that temporary files use guid instead.
If this is true Antivirus should not have problems about invalid chars in file name.
However it looks that the problem happen with e-mail that have invalid chats in subject.

So there is some options to set in hMailServer to change the way it save the temp files or there is some reason it use sometime guid and others time email subject?

User avatar
RvdH
Senior user
Senior user
Posts: 1422
Joined: 2008-06-27 14:42
Location: Netherlands

Re: Antivirus not delete attachment for CC account

Post by RvdH » 2021-05-04 20:52

paolo_c wrote:
2021-05-04 20:44
Ok now I'm more confused...
On this thread ( https://answers.microsoft.com/en-us/pro ... bfc2657ff6 ) I found a post by [RvdH] where he claim: "...each e-mail delivered through hMailServer is saved in a temporary directory with unique name which consist of a random identifier and the email's subject."
And this log is quoted:

Code: Select all

424 "2014-06-07 13:30:04.957" "THREAT FOUND"
424 "2014-06-07 13:30:04.957" "File: "C:\Program Files (x86)\hMailServer\Temp\2014060713300489_Re ****RE****""
424 "2014-06-07 13:30:04.957" "Command: "C:\Program Files\Microsoft Security Client\MpCmdRun.exe" -Scan -ScanType 3 -File "C:\Program Files (x86)\hMailServer\Temp\2014060713300489_Re ****RE****" -DisableRemediation"
424 "2014-06-07 13:30:04.957" "Return Code: 2"
Looking at your log it seems that temporary files use guid instead.
If this is true Antivirus should not have problems about invalid chars in file name.
However it looks that the problem happen with e-mail that have invalid chats in subject.

So there is some options to set in hMailServer to change the way it save the temp files or there is some reason it use sometime guid and others time email subject?
Ignore that post on MS site, that was ran from command line, trying to convince the people at MS to distinguish return types

I still believe the code here is responsible for the false positives reported by Windows Defender, perhaps the *.tmp file is not created if the mimepart contains "invalid/reserved" characters..eg:

Return code 2:
if malware is found and not remediated or additional user action is
required to complete remediation or there is error in scanning.
Please check History for more information.
That possibly could indicate the *.tmp is not created
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

paolo_c
New user
New user
Posts: 16
Joined: 2021-05-04 12:23

Re: Antivirus not delete attachment for CC account

Post by paolo_c » 2021-05-04 22:26

RvdH wrote:
2021-05-04 20:52
I still believe the code here is responsible for the false positives reported by Windows Defender, perhaps the *.tmp file is not created if the mimepart contains "invalid/reserved" characters..eg:

Return code 2:
if malware is found and not remediated or additional user action is
required to complete remediation or there is error in scanning.
Please check History for more information.
That possibly could indicate the *.tmp is not created
Thanks for your replay! This scenario is even worst! In this case any Antivirus will not be able to scan files that contain invalid chars, and if an antivirus return a different code for fail or positive it is very easy to bypass Antivirus just using invalid chars. Any fix?

paolo_c
New user
New user
Posts: 16
Joined: 2021-05-04 12:23

Re: Antivirus not delete attachment for CC account

Post by paolo_c » 2021-05-04 22:39

BTW... Is still not clear why for the email with CC, where all recipients are on the same email server, just for one recipient the attachment is deleted.

User avatar
RvdH
Senior user
Senior user
Posts: 1422
Joined: 2008-06-27 14:42
Location: Netherlands

Re: Antivirus not delete attachment for CC account

Post by RvdH » 2021-05-04 23:03

paolo_c wrote:
2021-05-04 22:26
RvdH wrote:
2021-05-04 20:52
I still believe the code here is responsible for the false positives reported by Windows Defender, perhaps the *.tmp file is not created if the mimepart contains "invalid/reserved" characters..eg:

Return code 2:
if malware is found and not remediated or additional user action is
required to complete remediation or there is error in scanning.
Please check History for more information.
That possibly could indicate the *.tmp is not created
Thanks for your replay! This scenario is even worst! In this case any Antivirus will not be able to scan files that contain invalid chars, and if an antivirus return a different code for fail or positive it is very easy to bypass Antivirus just using invalid chars. Any fix?
I'm not sure about that, but i have a suspicion there
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
RvdH
Senior user
Senior user
Posts: 1422
Joined: 2008-06-27 14:42
Location: Netherlands

Re: Antivirus not delete attachment for CC account

Post by RvdH » 2021-05-04 23:05

paolo_c wrote:
2021-05-04 22:39
BTW... Is still not clear why for the email with CC, where all recipients are on the same email server, just for one recipient the attachment is deleted.
No, but with a AV that triggers false positives you never know, therefor my initial reply was pick another AV
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

mikedibella
Senior user
Senior user
Posts: 464
Joined: 2016-12-08 02:21

Re: Antivirus not delete attachment for CC account

Post by mikedibella » 2021-05-05 00:29

Interesting that one of the Defender HRESULT codes is:

Code: Select all

0x80501004	ERROR_MP_NO_INTERNET_CONN	Check your Internet connection, then run the scan again.
(see https://docs.microsoft.com/en-us/micros ... -worldwide)

This leads me to believe that Defender uses, at least in part, cloud-based signature matching.

So, it is conceivable that iterative scans of the same datasets could produce split decisions if one of the scans was incomplete due to transient network issue.

User avatar
mattg
Moderator
Moderator
Posts: 21367
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Antivirus not delete attachment for CC account

Post by mattg » 2021-05-05 02:57

mikedibella wrote:
2021-05-05 00:29
This leads me to believe that Defender uses, at least in part, cloud-based signature matching.
Yes, I think that is a standard section in 'Windows Security' in Windows 10
RvdH wrote:
2021-05-04 23:05
my initial reply was pick another AV
+1
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
RvdH
Senior user
Senior user
Posts: 1422
Joined: 2008-06-27 14:42
Location: Netherlands

Re: Antivirus not delete attachment for CC account

Post by RvdH » 2021-05-05 09:39

Basically there are 2 issues with using Defender as AV
  • It doesn't properly distinguish return types
  • Nothing is logged once you enable -DisableRemediation
Return code is

0 if no malware is found or malware is successfully remediated and no additional user action is required

2 if malware is found and not remediated or additional user action is required to complete remediation or there is error in scanning. Please check History for more information.


[-DisableRemediation]

This option is valid only for custom scan.

When specified:

- File exclusions are ignored.

- Archive files are scanned.

- Actions are not applied after detection.

- Event log entries are not written after detection.

- Detections from the custom scan are not displayed in the user interface.

- The console output will show the list of detections from the custom scan.
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

paolo_c
New user
New user
Posts: 16
Joined: 2021-05-04 12:23

Re: Antivirus not delete attachment for CC account

Post by paolo_c » 2021-05-05 13:21

On a test server I used ClamAV but I had a lot of problems, it is very slow and emails in queue can thake up to 5 minutes to be delivererd, Update of signature fail many time and a lot of infected attachment are not detected.
With Windows Security I don't have problem of performance and all virus are detected. I think it is more safe to consider a fail after a scan retry as positive, because is better to explain to my clients a false positive then a virus that could infect their PC. The point is to reduce the false positive in case of invalid chars. And for this is essential to understand if the problem is caused by a real scan fail that can be solved in most cases with a scan retry, or if the problem come from the bug supposed by [RvdH]. In the second case a second scan do not solve the problem if the temp file is never created.

paolo_c
New user
New user
Posts: 16
Joined: 2021-05-04 12:23

Re: Antivirus not delete attachment for CC account

Post by paolo_c » 2021-05-05 13:34

mikedibella wrote:
2021-05-05 00:29
This leads me to believe that Defender uses, at least in part, cloud-based signature matching.

So, it is conceivable that iterative scans of the same datasets could produce split decisions if one of the scans was incomplete due to transient network issue.
If the problem is just this, a scan retry should fix it. But I'm afraid that the problem in most cases is about invalid chars. In this case the scan should fail for all recipients, but it don't. I would like to understand how hMailServer manage the AV scan for mails with multiple recipients. It scan the email just one time or for each recipient? and if it scan just one time (should be the best way for performance) how it deal with CC recipients?

mikedibella
Senior user
Senior user
Posts: 464
Joined: 2016-12-08 02:21

Re: Antivirus not delete attachment for CC account

Post by mikedibella » 2021-05-05 17:04

If the sending MTA splits the email into multiple deliveries for each recipient, even when they are at the same domain, then hMailServer will see multiple copies of the same dataset and run AV on each. You need to review logs to determine if that is what happened in the case you are describing.

User avatar
RvdH
Senior user
Senior user
Posts: 1422
Joined: 2008-06-27 14:42
Location: Netherlands

Re: Antivirus not delete attachment for CC account

Post by RvdH » 2021-05-05 17:36

mikedibella wrote:
2021-05-05 17:04
If the sending MTA splits the email into multiple deliveries for each recipient, even when they are at the same domain, then hMailServer will see multiple copies of the same dataset and run AV on each. You need to review logs to determine if that is what happened in the case you are describing.
+1

And that is very hard to do, as with -DisableRemediation nothing is logged
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

Post Reply