Did not submit bounce message with Auto-Submitted header.

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
SgtWilko
New user
New user
Posts: 7
Joined: 2016-04-01 22:38

Did not submit bounce message with Auto-Submitted header.

Post by SgtWilko » 2016-04-01 22:45

Hiya,

I've only just setup a hmailserver, it is for sending email, the server will not be holding mailboxes, nor storing mail.

As such I have a route setup to send email that is destined for the domain this instance is sending from to the real server, this works find.

However when our app send mail via the server, if it encounters a server that rejects the email (bad recipient, spam, etc.) we see the following in the log, and no delivery failure is sent to us:
Severity: 3 (Medium), Code: HM4404, Source: SMTPDeliverer::_ApplyForwarding, Description: Did not submit bounce message for message ??? from ???@???.??.>> since rule loop count was reached or Auto-Submitted header.

I know our app adds the Auto-Submitted header (which I don't want to remove because of the remove because I don't want to see messages from remote servers), but we'd still like to see the failure emails from our hmailserver, is this possible to do?


Many thanks,

User avatar
mattg
Moderator
Moderator
Posts: 21304
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Did not submit bounce message with Auto-Submitted header.

Post by mattg » 2016-04-02 00:33

There is no option that I am aware of, but you could fork the source on github
https://github.com/hmailserver/hmailserver
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

SgtWilko
New user
New user
Posts: 7
Joined: 2016-04-01 22:38

Re: Did not submit bounce message with Auto-Submitted header.

Post by SgtWilko » 2016-05-05 11:48

No way to configure a a forwarding rule to do this?

I've looked into making a script to email us, and I found this thread which shows how to get the errors emailed, but I'd want a copy of the email that caused the error and I can't see a way to get that.
Can you access the email that caused the error in anyway from the scripting system?

Thanks again.

User avatar
mattg
Moderator
Moderator
Posts: 21304
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Did not submit bounce message with Auto-Submitted header.

Post by mattg » 2016-05-05 16:42

I catch messages in the queue by re-try counts using a global rule, and then call a script that sends a reply message, where say a message hasn't been delivered after an hour.

If you catch the message before it bounces, you could do that
Once it bounces you can't
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

SgtWilko
New user
New user
Posts: 7
Joined: 2016-04-01 22:38

Re: Did not submit bounce message with Auto-Submitted header.

Post by SgtWilko » 2016-05-06 09:48

Hi Mattg,

The problem we have is that I'm certain it's being dropped because of the "Auto-Submitted" header, I'm don't think a "re-try count" rule is going to work in that instance.

Unless there's a rule that can be triggered for emails that fail to have a bounce message sent.

Basically I'm just looking for a "forward on error" rule/setting.

Thanks once again.

User avatar
mattg
Moderator
Moderator
Posts: 21304
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Did not submit bounce message with Auto-Submitted header.

Post by mattg » 2016-05-06 10:05

SgtWilko wrote:Basically I'm just looking for a "forward on error" rule/setting.
Yes I understand

This is not possible
mattg wrote:If you catch the message before it bounces, you could do that
Once it bounces you can't
You could test ALL mail, and remove that header if it exists with a script
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jimimaseye
Moderator
Moderator
Posts: 8953
Joined: 2011-09-08 17:48

Re: Did not submit bounce message with Auto-Submitted header.

Post by jimimaseye » 2016-05-06 10:06

There is an OnError event in HMS (https://www.hmailserver.com/documentati ... ng_onerror) that can be modified to do what you wish. However, the problem is that a BOUNCE is not considered an 'error', (it is a well established procedural function of a mail server) and therefore does not apply to this event.

BUT (and I havent checked here but think its correct) the condition that results in:
"Severity: 3 (Medium), Code: HM4404, Source: SMTPDeliverer::_ApplyForwarding, Description: Did not submit bounce message for message ??? from ???@???.??.>> since rule loop count was reached or Auto-Submitted header." may well apply to this event. Therefore you could try a script in that even that checks for the code number and sends you an email.

That said, what useful details you can add to that email is limited. There currently isnt a known way to capture/append the original email contents to a bounce notification (the event only really knows a message number and the email address (you have quoted) but doesnt know about the actual email so you cant refer to it).
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

SgtWilko
New user
New user
Posts: 7
Joined: 2016-04-01 22:38

Re: Did not submit bounce message with Auto-Submitted header.

Post by SgtWilko » 2016-05-06 17:29

mattg wrote:
SgtWilko wrote:Basically I'm just looking for a "forward on error" rule/setting.
Yes I understand

This is not possible
Thanks, wasn't implying that you didn't understand, more so trying to rephrase it to see if I could think about it differently :)
mattg wrote:
mattg wrote:If you catch the message before it bounces, you could do that
Once it bounces you can't
You could test ALL mail, and remove that header if it exists with a script
Would removing the header actually remove it from the email that leaves hmail, or would it just remove it for the processing of email in hmail?
If it does remove it from the outgoing mail, would it be possible to add it back in after the mail server would have processed the bounce (assuming it hasn't bounced), but before it actually is sent?

SgtWilko
New user
New user
Posts: 7
Joined: 2016-04-01 22:38

Re: Did not submit bounce message with Auto-Submitted header.

Post by SgtWilko » 2016-05-06 18:01

Hi,
jimimaseye wrote:There is an OnError event in HMS (https://www.hmailserver.com/documentati ... ng_onerror) that can be modified to do what you wish. However, the problem is that a BOUNCE is not considered an 'error', (it is a well established procedural function of a mail server) and therefore does not apply to this event.
...
jimimaseye wrote:That said, what useful details you can add to that email is limited. There currently isnt a known way to capture/append the original email contents to a bounce notification (the event only really knows a message number and the email address (you have quoted) but doesnt know about the actual email so you cant refer to it).
From what I've seen OnError would be called, this thread is talking about emailing when the error is dropped into the error log, and that log is where I'm seeing the HM4404 error.
However, as I mentioned before, I would want a copy of the email that caused the error and I can't see a way to get that.

Is there a way to identify the email in the queue on the disk?

It does seem strange that I can setup a domain in hmail, but can't configure it to forward to a "domain admin" email address for those emails which have failed to be sent from that domain.

Ok, maybe I should approach this in a different way, rather than trying to replicate the functionality in the IIS SMTP agent which allowed for a copy of the Non-Delivery report to be sent to an dedicated address, is there a way to find out why the email is being bounced from within hmail?
The error "...since rule loop count was reached or Auto-Submitted header" isn't that helpful, and from looking at the source it would seem that an actual error description is passed into SubmitErrorLog_, but then not used if the Auto-Submitted header is present.

Can I get that error somehow?


Once again, many thanks for the help.

User avatar
jimimaseye
Moderator
Moderator
Posts: 8953
Joined: 2011-09-08 17:48

Re: Did not submit bounce message with Auto-Submitted header.

Post by jimimaseye » 2016-05-06 19:06

SgtWilko wrote:
Is there a way to identify the email in the queue on the disk?
No
SgtWilko wrote: The error "...since rule loop count was reached or Auto-Submitted header" isn't that helpful, and from looking at the source it would seem that an actual error description is passed into SubmitErrorLog_, but then not used if the Auto-Submitted header is present.

Can I get that error somehow?
Look in the Log file, maybe that gives you a clue. (I dont know what level of logging is required, you would have to enable all, test and see.) Cant see it helping you much though because it still will not detail the email contents.
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
mattg
Moderator
Moderator
Posts: 21304
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Did not submit bounce message with Auto-Submitted header.

Post by mattg » 2016-05-07 02:11

SgtWilko wrote:Would removing the header actually remove it from the email that leaves hmail, or would it just remove it for the processing of email in hmail?
If it does remove it from the outgoing mail, would it be possible to add it back in after the mail server would have processed the bounce (assuming it hasn't bounced), but before it actually is sent?
It would be easier for you to fork the code on Github and do what you want at source level.

And yes it would change the email and remove the header completely.
Catching the same message again to re-add the header, after the bounce hasn't occurred is unlikely.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

pik256
New user
New user
Posts: 16
Joined: 2016-05-18 11:25
Location: Poland

Re: Did not submit bounce message with Auto-Submitted header.

Post by pik256 » 2016-10-13 19:49

SgtWilko wrote: However, as I mentioned before, I would want a copy of the email that caused the error and I can't see a way to get that.
In fact there is a way to get information from email that caused the error.

I have installed hMail and found myself in the same situation. One of my users sends a lot of "Auto-submitted: auto-generated" messages. They are sent by a process on behalf of many users. A script catches bounces, scans them to identify a valid recipient of an error for explanation. The Auto-submitted: auto-generated header was included in messages to avoid auto replies. This header is intended to stop auto-replies, not DSN bounces. A failure to not submit bounces by hmail on auto-generated messages is invalid in my opinion (based on RFC 3834 and 3461). But anyway, lets call it a "feature".

Returning to our problem: there is OnDeliveryStart event, when you have any information about the message. Especially you can use Message.ID to create unique file name with all information you need, e.g. recipients or the whole message. OnError you can scan description for message ID and use information from created file with that name. You need to delete expired files of course, but this can be done periodically by external process.
In fact I send information to external process onSMTPData, OnAcceptMessage and onDeliveryStart for anti-spam and anti virus scanning, logging and some additional processing, so onError I have information collected in a mySQL database but if you need only a copy of the email that caused the error then you can save this email to the file named as value of message ID onDeliveryStart (or maybe onDeliverMessage) and you have it onError.

User avatar
mattg
Moderator
Moderator
Posts: 21304
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Did not submit bounce message with Auto-Submitted header.

Post by mattg » 2016-10-14 00:09

pik256 wrote:[A failure to not submit bounces by hmail on auto-generated messages is invalid in my opinion (based on RFC 3834 and 3461). But anyway, lets call it a "feature".
Add it to github please>> https://github.com/hmailserver/hmailserver/issues
pik256 wrote:Returning to our problem: there is OnDeliveryStart event, when you have any information about the message. Especially you can use Message.ID to create unique file name with all information you need, e.g. recipients or the whole message. OnError you can scan description for message ID and use information from created file with that name. You need to delete expired files of course, but this can be done periodically by external process.
In fact I send information to external process onSMTPData, OnAcceptMessage and onDeliveryStart for anti-spam and anti virus scanning, logging and some additional processing, so onError I have information collected in a mySQL database but if you need only a copy of the email that caused the error then you can save this email to the file named as value of message ID onDeliveryStart (or maybe onDeliverMessage) and you have it onError.
I'm not sure that when the 'On Delivery Start' event is fired that the complete message has been downloaded. You may only be saving part of a message.

Like you say this does create a duplicate of emails that need to be cleaned up lated.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

pik256
New user
New user
Posts: 16
Joined: 2016-05-18 11:25
Location: Poland

Re: Did not submit bounce message with Auto-Submitted header.

Post by pik256 » 2016-10-14 07:10

mattg wrote: I'm not sure that when the 'On Delivery Start' event is fired that the complete message has been downloaded. You may only be saving part of a message.
It is almost complete right on OnAcceptMessage. I am not sure about a Message object but I mean the file (Message.Filename). I use it for antispam and antivirus scanning, sometimes rejecting message but copying the file to a quarantine area. After OnDeliveryStart hmail only adjusts the Return-path header and inserts Delivered-to (on local accounts).

User avatar
mattg
Moderator
Moderator
Posts: 21304
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Did not submit bounce message with Auto-Submitted header.

Post by mattg » 2016-10-14 08:10

I suspect that the message state at the time that OnAcceptMessage is fired includes message headers, but doesn't include the other message components like HTML body and body text and attachments and other MIME parts.

Probably very suitable for SPAM checking.
Unsure about Virus checking though.

Does your routine catch many viruses?

(Like many others here, I use SpamAssassin. I use ClamAV integration into SpamAssassin, and the Sane Security defs for ClamAV to detect rogue attachments. My Spamassassin removes lots of rubbish for me by scoring mail as high level spam and then hMailserver rejects the mail. I also run inline Avast virus checking)
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

SgtWilko
New user
New user
Posts: 7
Joined: 2016-04-01 22:38

Re: Did not submit bounce message with Auto-Submitted header.

Post by SgtWilko » 2016-10-14 10:03

mattg wrote:
pik256 wrote:[A failure to not submit bounces by hmail on auto-generated messages is invalid in my opinion (based on RFC 3834 and 3461). But anyway, lets call it a "feature".
Add it to github please>> https://github.com/hmailserver/hmailserver/issues
@pik256 Hiya, once raised can you post the github issue number in here, or confirm if you don't plan on raising it and I will do so (but you obviously have a more in depth understanding of this issue).

pik256
New user
New user
Posts: 16
Joined: 2016-05-18 11:25
Location: Poland

Re: Did not submit bounce message with Auto-Submitted header.

Post by pik256 » 2016-10-17 10:27

SgtWilko wrote:once raised can you post the github issue number in here, or confirm if you don't plan on raising it and I will do so
Why not, OK, I will post it. This week.
mattg wrote:I suspect that the message state at the time that OnAcceptMessage is fired includes message headers, but doesn't include the other message components like HTML body and body text and attachments and other MIME parts.

Probably very suitable for SPAM checking.
Unsure about Virus checking though.

Does your routine catch many viruses?
But of course. As I wrote I am not sure about the Message object but message file is for sure complete.
mattg wrote:(Like many others here, I use SpamAssassin. I use ClamAV integration into SpamAssassin, and the Sane Security defs for ClamAV to detect rogue attachments. My Spamassassin removes lots of rubbish for me by scoring mail as high level spam and then hMailserver rejects the mail. I also run inline Avast virus checking)
I do not use SpamAssassin. I use my own antispam script. It is fully customizable (for me) where I have implemented features unavailable in SpamAssassin. For example I use my own additional signature files with Clamav. These signatures contains some common polish spam phrases, nigerian scams etc. But I do not want to scan each message for them. My script checks an individual correspondence list maintained at each account and excludes specified signature classes detected by clamav. The same for many other antispam filters. Messages coming from unknown correspondent are treated far more restrictive than messages from people my user writes lately. Unknown users cannot advertise viagra. But friends can joke about. These correspondence lists are created and maintained automatically by script.

That is why I need to scan every and each complete message, not only with ClamAV but with my scripts even first. Before I decided to migrate to hmail I had tested if a complete message is available for custom scripting before accepting it on SMTP transaction. Of course only a few information is available OnSMTPData. This is an event that occur before the message body is received. But the whole message body is received and written to a file before OnAcceptMessage occur. Hmail need to provide this file to SpamAssasin and ClamAV. As stated in doccumentation:
Order of Execution
OnClientConnect (oClient)
spam tests >> DNSBlackilists, HELOhost, MXRecords, SPF
OnSMTPData (oClient, oMessage)
spam tests >> SURBL, DKIM, Greylisting, integrated SpamAssasin
OnAcceptMessage (oClient, oMessage)
OnDeliveryStart (oMessage)
integrated anti-virus check, Global rules
OnDeliverMessage (oMessage)
OnDeliveryFailed (oMessage, sRecipient, sErrorMessage) - if applicable
Message delivered to recipeint(s), Account level rules
So, the message file is prepared probably even before OnAcceptMessage - for SpamAssasin as I suppose. SpamAssassin does not check only headers but the whole message body. And last but not least - my scripts sometimes reject messages (returns 2 or 3 to hmail as Result.Value) but writes them to a quarantine. This helps me to adjust my scripts to avoid false positives and rarely to push a message to a user even when it was rejected (on FP). I catch a few messages to quarantine nearly each day. Yes, I am sure they are complete. Hmail only appends Delivered-to and adjusts Return-Path headers after OnDeliveryStart (I tried to alter Return-Path header with a global rule - unsuccessfully).

pik256
New user
New user
Posts: 16
Joined: 2016-05-18 11:25
Location: Poland

Re: Did not submit bounce message with Auto-Submitted header.

Post by pik256 » 2016-10-24 22:56

pik256 wrote:
SgtWilko wrote:once raised can you post the github issue number in here, or confirm if you don't plan on raising it and I will do so
Why not, OK, I will post it. This week.
Posted here

User avatar
jimimaseye
Moderator
Moderator
Posts: 8953
Joined: 2011-09-08 17:48

Re: Did not submit bounce message with Auto-Submitted header.

Post by jimimaseye » 2016-10-24 23:15

Erm. ...

...If a message comes in and an AUTOMATED (non human controlled) response is generated (for, example, because the original mailbox is unavailable for some reason), and if that automated response doesn't successfully get delivered, where exactly is the DSN apposed to go? To the very mailbox that is unable to receive the original email in the first place?? What would the point of having an automated response to prevent incoming emails and notify the sender if it doesn't stop emails coming? And that's assuming the mailbox exists and is able to receive anyway - if it doesn't, where is the DSN supposed to go? The ether? ?
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

pik256
New user
New user
Posts: 16
Joined: 2016-05-18 11:25
Location: Poland

Re: Did not submit bounce message with Auto-Submitted header.

Post by pik256 » 2016-10-25 12:28

jimimaseye wrote:Erm. ...

...If a message comes in and an AUTOMATED (non human controlled) response is generated (for, example, because the original mailbox is unavailable for some reason), and if that automated response doesn't successfully get delivered, where exactly is the DSN apposed to go? To the very mailbox that is unable to receive the original email in the first place?? What would the point of having an automated response to prevent incoming emails and notify the sender if it doesn't stop emails coming? And that's assuming the mailbox exists and is able to receive anyway - if it doesn't, where is the DSN supposed to go? The ether? ?
I am not sure if I understood this precisely and precision is important. But I will try to respond to what I understood.
DSN is nothing new, it is designed at the very beginning of SMTP. Of course, DSN is allways an automated response and possibility of mail loops was predicted by the protocol. DSNs, by default, have an empty return path. So, if a message comes in and an AUTOMATED response is generated (just DSN), and if that automated response (DSN) doesn't successfully get delivered (to the sender of an original message) then there is no next DSN to generate and delivery. Or, if you insist, the DSN is supposed to go to devnull. By design. But your doubts are also considered in RFC. As stated in RFC 2821:
(3.7) If an SMTP server has accepted the task of relaying the mail and later finds that the destination is incorrect or that the mail cannot be delivered for some other reason, then it MUST construct an "undeliverable mail" notification message and send it to the originator of the undeliverable mail (as indicated by the reverse-path).
(4.1.2.2) The reverse-path consists of the sender mailbox. Historically, that mailbox might optionally have been preceded by a list of hosts, but that behavior is now deprecated (see appendix C). In some types of reporting messages for which a reply is likely to cause a mail loop (for example, mail delivery and nondelivery notifications), the reverse-path may be null (see section 3.7).
4.5.5 Messages with a null reverse-path
There are several types of notification messages which are required by existing and proposed standards to be sent with a null reverse path, namely non-delivery notifications as discussed in section 3.7, other kinds of Delivery Status Notifications (DSNs) [24], and also Message Disposition Notifications (MDNs) [10]. All of these kinds of messages are notifications about a previous message, and they are sent to the reverse-path of the previous mail message. (If the delivery of such a notification message fails, that usually indicates a problem with the mail system of the host to which the notification message is addressed. For this reason, at some hosts the MTA is set up to forward such failed notification messages to someone who is able to fix problems with the mail system, e.g., via the postmaster alias.)
And lets go now to my (and SgtWilko's) problem. I send auto-generated mail (periodic invoices, remainders etc.). I do not need to collect common autoreplies like "out of office", so I included auto-submitted header. But I would like to know if my invoice was delivered or not. Now, some remote servers do not comply to RFC standards and send "out of office" in spite of auto-sumbitted header. This is not a big problem, I can simply ignore them. But hmail disregard for RFC standards is a bigger problem. I lose crucial information, so I have to parse hmail logs to find information about original recipient of undelivered message or to waive auto-submitted header.

And finally. If a sender (a bot) ignores DSNs and doesn't stop to send mail then of course we can do nothing more in standard (RFC-regulated) way. It probably needs a human intervention. But this is not a point to not send DSNs at all. Some robots (i.e. my script) use DSNs in proper way and stop sending mail to unavailable accounts.

User avatar
jimimaseye
Moderator
Moderator
Posts: 8953
Joined: 2011-09-08 17:48

Re: Did not submit bounce message with Auto-Submitted header.

Post by jimimaseye » 2016-10-25 16:44

So, for clarity, you are sending out an email (not a DSN), generated by command or script (rather than SMTP protocol automated) with an 'autosubmitted' header included by choice, and if it doesnt get delivered to intended destination do you :

a, want to see a non-delivery message but dont receive one or
b, dont want to see a non-delivery message but do receive one?
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

pik256
New user
New user
Posts: 16
Joined: 2016-05-18 11:25
Location: Poland

Re: Did not submit bounce message with Auto-Submitted header.

Post by pik256 » 2016-10-25 23:23

a) I want to see a non-delivery message but don't receive one.

hMail doesn't generate it due to existence of auto-submitted header (and this behaviour is incompatible with RFC2821 standard).

SgtWilko
New user
New user
Posts: 7
Joined: 2016-04-01 22:38

Re: Did not submit bounce message with Auto-Submitted header.

Post by SgtWilko » 2016-11-07 12:19

I second this: I want to see a non-delivery message but don't receive one.

The current behaviour is different from all the other MTAs we have used in the past (many that allowed a fallback admin/postmaster mail address to be configured to deal with mail loops)

@Pik256 - Thank you for raising the github issue.

Post Reply