Presumably unavoidable duplicate receipts

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
User avatar
katip
Senior user
Senior user
Posts: 624
Joined: 2006-12-22 07:58
Location: Istanbul

Presumably unavoidable duplicate receipts

Post by katip » 2019-03-28 11:55

need advise.

purpose:
recipient_2@local.com should receive a copy of mails where recipient_1@local.com is a recipient. redundant copies to recipient_2 where both together are recipients in same mail should be avoided. hence, not using built-in "Forward" in recipient_1 account setup, but a global rule such as:

From: sender@external.com
To: recipient_1@local.com, recipient_2@local.com

IF recipient list contains recipient_1@local.com
AND recipient list NOT contains recipient_2@local.com <- to avoid duplicate receipt
AND X-hMailServer-LoopCount < 1
forward to recipient_2@local.com

looks good. BUT:
some MTA's split envelope and relay for each recipient an exclusive copy. it's understandable that especially paid SMTP relay services do so.

as a result, recipient_2 recieves double as HMS reads a single recipient on each envelope, but not a list of recipients to check with above rule condition.
so, recipient_2 receives his/her own mail + another useless copy because of global rule.

if my above understanding is correct, there is nothing much to do. recipient_2 must live with duplicates.
otherwise thanks for any suggestion.
Katip
--
HMS 5.7.0-B2428-LTS-64-bit, MySQL 5.7.24, SA 3.4.2, ClamAV 0.101.2 + SaneS

User avatar
SorenR
Senior user
Senior user
Posts: 3043
Joined: 2006-08-21 15:38
Location: Denmark

Re: Presumably unavoidable duplicate receipts

Post by SorenR » 2019-03-28 13:40

Would this work?

Code: Select all

IF Recipient_list = wile.e.coyote@acme.inc, road.runner@acme.inc 
   ==> DO NOTHING

IF Recipient_list = wile.e.coyote@acme.inc 
   AND To = wile.e.coyote@acme.inc 
   ==> SEND COPY TO road.runner@acme.inc

IF Recipient_list = wile.e.coyote@acme.inc 
   AND To = wile.e.coyote@acme.inc, road.runner@acme.inc 
   ==> DO NOTHING

IF Recipient_list = road.runner@acme.inc 
   AND To = wile.e.coyote@acme.inc, road.runner@acme.inc 
   ==> DO NOTHING
SørenR.

The quantum rule of insecurity which states that the act of observing how vulnerable a host or service is changes the insecurity level of the service.

User avatar
katip
Senior user
Senior user
Posts: 624
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Presumably unavoidable duplicate receipts

Post by katip » 2019-03-28 16:34

logic is ok.
however then comes and/or CC. combinations grow. BCC is another issue.
besides i have to maintain this rules for many local user combinations.

i was thinking to keep track of all processed mails in a DB table such as:
OnAcceptMessage
for i=0 to recipients count
SQL insert Message-ID, recipient(i)
...
(ignore syntax of course)

change rule to:
IF recipient list contains recipient_1@local.com
call function checkMessageID

function would look up for this Message-ID any "Message-ID, recipient_2" record. if found, it means recipient_2 has already got his/her copy -> end function, else send one copy to recipient_2 and update table -> end function.
the problem is how to pass recipient_2 address from rule to function. does rule calls support arguments like "call function checkMessageID(recipient_2)" ?

table can be emptied during one of midnight maintenance things.
Katip
--
HMS 5.7.0-B2428-LTS-64-bit, MySQL 5.7.24, SA 3.4.2, ClamAV 0.101.2 + SaneS

User avatar
SorenR
Senior user
Senior user
Posts: 3043
Joined: 2006-08-21 15:38
Location: Denmark

Re: Presumably unavoidable duplicate receipts

Post by SorenR » 2019-03-28 17:14

katip wrote:
2019-03-28 16:34
logic is ok.
however then comes and/or CC. combinations grow. BCC is another issue.
besides i have to maintain this rules for many local user combinations.

i was thinking to keep track of all processed mails in a DB table such as:
OnAcceptMessage
for i=0 to recipients count
SQL insert Message-ID, recipient(i)
...
(ignore syntax of course)

change rule to:
IF recipient list contains recipient_1@local.com
call function checkMessageID

function would look up for this Message-ID any "Message-ID, recipient_2" record. if found, it means recipient_2 has already got his/her copy -> end function, else send one copy to recipient_2 and update table -> end function.
the problem is how to pass recipient_2 address from rule to function. does rule calls support arguments like "call function checkMessageID(recipient_2)" ?

table can be emptied during one of midnight maintenance things.
Message-ID is a globally unique identifier, so that won't work. https://tools.ietf.org/html/rfc2392
SørenR.

The quantum rule of insecurity which states that the act of observing how vulnerable a host or service is changes the insecurity level of the service.

User avatar
SorenR
Senior user
Senior user
Posts: 3043
Joined: 2006-08-21 15:38
Location: Denmark

Re: Presumably unavoidable duplicate receipts

Post by SorenR » 2019-03-28 18:58

Untested ...

Code: Select all

Dim i, j, strRecipients, strTo, Match, Matches

' Build recipient list
strRecipients = ""
For i = 0 To oMessage.Recipients.Count-1
   If strRecipients = "" Then
      strRecipients = oMessage.Recipients(i).Address
   Else
      strRecipients = strRecipients & ", " & oMessage.Recipients(i).Address
   End If
Next

' Use "To:" only?
If oMessage.CC <> "" Then
   strTo = oMessage.To & ", " & oMessage.CC
Else
   strTo = oMessage.To
End If

' Create array of "To:" and "CC:" addresses
arrTo = Split(Replace(strTo, ";", ","), ",")
strTo = ""

' Check if all "To:"s exist in "oMessage.Recipients"
For j = LBound(arrTo) To UBound(arrTo)

   ' Extract raw email address
   With CreateObject("VBScript.RegExp")
      .Pattern = "([a-zA-Z0-9._-]+@[a-zA-Z0-9._-]+\.[a-zA-Z0-9_-]+)"
      .Global = False
      .IgnoreCase = True
      Set Matches = .Execute(arrTo(j))
   End With

   ' Looks worse than it is, it is really only one match!
   For Each Match In Matches
      If (InStr(1, strRecipients, Trim(Match.Value), 1) > 0) Then
         If strTo = "" Then
            strTo = Match.Value
         Else
            strTo = strTo & ", " & Match.Value
         End If
      End If
   Next
Next

' This is where the action is ...
If strTo = "" And i = 0 Then
   '
   ' Someone is not getting his/her copy of the email ;-)
   '
End If
SørenR.

The quantum rule of insecurity which states that the act of observing how vulnerable a host or service is changes the insecurity level of the service.

User avatar
katip
Senior user
Senior user
Posts: 624
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Presumably unavoidable duplicate receipts

Post by katip » 2019-03-28 19:09

SorenR wrote:
2019-03-28 17:14
Message-ID is a globally unique identifier,
yes sure, and my idea is based on this.
MTA's split the envelope only, original Message-ID don't change on any of the relayed derivates.
i.e. 2 messages arrive to HMS. one for recipient_1 (To) and one for recipient_2 (CC), both with same Message-ID.
this makes it possible to track who and who else received the same message.

Code: Select all

From: "Selda CEYLAN" <ihracat@external.com>
To: "'Altan Bala'" <altan.bala@ourdomain.com>, =?utf-8?Q?'Mesut_A=C4=9Fao=C4=9Flu'?=
 <mesutagaoglu@external.com>
Cc: =?utf-8?Q?'Onur_Kantarc=C4=B1'?= <onur.kantarci@ourdomain.com>, <mehmet.ozkan@ourdomain.com>,
 =?utf-8?Q?'=C4=B0zabela_=C3=96ztan'?= <izabela.oztan@ourdomain.com>, =?utf-8?Q?'P=C4=B1nar_Eren'?=
 <pinar.eren@ourdomain.com>
Subject: =?utf-8?Q?RE:_ (etc...)
Date: Thu, 28 Mar 2019 18:42:20 +0300
Message-ID: <072301d4e57c$d565c8f0$80315ad0$@external.com>      <----- same in both messages
X-Envelope-From: ihracat@external.com
X-Envelope-To: altan.bala@ourdomain.com               <----- local recipient in To:

===============================================================================

From: "Selda CEYLAN" <ihracat@external.com>
To: "'Altan Bala'" <altan.bala@ourdomain.com>, =?utf-8?Q?'Mesut_A=C4=9Fao=C4=9Flu'?=
 <mesutagaoglu@external.com>
Cc: =?utf-8?Q?'Onur_Kantarc=C4=B1'?= <onur.kantarci@ourdomain.com>, <mehmet.ozkan@ourdomain.com>,
 =?utf-8?Q?'=C4=B0zabela_=C3=96ztan'?= <izabela.oztan@ourdomain.com>, =?utf-8?Q?'P=C4=B1nar_Eren'?=
 <pinar.eren@ourdomain.com>
Subject: =?utf-8?Q?RE:_ (etc...)
Date: Thu, 28 Mar 2019 18:42:20 +0300
Message-ID: <072301d4e57c$d565c8f0$80315ad0$@external.com>      <----- same in both messages
X-Envelope-From: ihracat@external.com
X-Envelope-To: pinar.eren@ourdomain.com                  <----- local recipient in CC:
Katip
--
HMS 5.7.0-B2428-LTS-64-bit, MySQL 5.7.24, SA 3.4.2, ClamAV 0.101.2 + SaneS

User avatar
katip
Senior user
Senior user
Posts: 624
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Presumably unavoidable duplicate receipts

Post by katip » 2019-03-28 19:18

SorenR wrote:
2019-03-28 18:58
Untested ...
thanks, i'll have a look.
but once again; in fact we're talking about 2 (or more) seperate but identical messages each with one unique recipient (X-Envelope-To), and the question is who were involved all together in the same message.
Katip
--
HMS 5.7.0-B2428-LTS-64-bit, MySQL 5.7.24, SA 3.4.2, ClamAV 0.101.2 + SaneS

User avatar
SorenR
Senior user
Senior user
Posts: 3043
Joined: 2006-08-21 15:38
Location: Denmark

Re: Presumably unavoidable duplicate receipts

Post by SorenR » 2019-03-28 19:37

katip wrote:
2019-03-28 19:09
SorenR wrote:
2019-03-28 17:14
Message-ID is a globally unique identifier,
yes sure, and my idea is based on this.
MTA's split the envelope only, original Message-ID don't change on any of the relayed derivates.
i.e. 2 messages arrive to HMS. one for recipient_1 (To) and one for recipient_2 (CC), both with same Message-ID.
this makes it possible to track who and who else received the same message.
Not sure you are correct. If server has to split message, it will create two new Message-ID's.
SørenR.

The quantum rule of insecurity which states that the act of observing how vulnerable a host or service is changes the insecurity level of the service.

User avatar
katip
Senior user
Senior user
Posts: 624
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Presumably unavoidable duplicate receipts

Post by katip » 2019-03-28 20:50

SorenR wrote:
2019-03-28 19:37
Not sure you are correct. If server has to split message, it will create two new Message-ID's.
see below. google servers split 1 message to 5 and send from 4 IP's. all landed with same Message-ID. this doesn't violate RFC 2392: "no two different messages will ever have the same Message-ID addr-spec". indeed, originally it WAS all in all 1 single message :wink:
i leave username part of addresses disclosed to make easier to follow the log.

Code: Select all

From: "Selda CEYLAN" <ihracat@external.com>
To: "'Altan Bala'" <altan.bala@ourdomain.com>, 'Mesut Ağaoğlu'   <mesutagaoglu@external.com>
Cc: 'Onur Kantarcı' <onur.kantarci@ourdomain.com>, <mehmet.ozkan@ourdomain.com>,   'İzabela Öztan' <izabela.oztan@ourdomain.com>, 'Pınar Eren'   <pinar.eren@ourdomain.com>
and HMS <-> google protocol

Code: Select all

"SMTPD"	4388	9701768	"2019-03-28 18:42:23.600"	"209.85.128.51"	"RECEIVED: EHLO mail-wm1-f51.google.com"
"SMTPD"	4388	9701768	"2019-03-28 18:42:23.600"	"209.85.128.51"	"SENT: 250-mail.ourdomain.com[nl]250-SIZE 50000000[nl]250-STARTTLS[nl]250 HELP"
"SMTPD"	3308	9701760	"2019-03-28 18:42:23.647"	"209.85.128.50"	"RECEIVED: STARTTLS"
"SMTPD"	3308	9701760	"2019-03-28 18:42:23.647"	"209.85.128.50"	"SENT: 220 Ready to start TLS"
"SMTPD"	3340	9701768	"2019-03-28 18:42:23.647"	"209.85.128.51"	"RECEIVED: STARTTLS"
"SMTPD"	3340	9701768	"2019-03-28 18:42:23.647"	"209.85.128.51"	"SENT: 220 Ready to start TLS"
"SMTPD"	3308	9701769	"2019-03-28 18:42:23.662"	"209.85.128.42"	"SENT: 220 mail.ourdomain.com"
"SMTPD"	4388	9701770	"2019-03-28 18:42:23.709"	"209.85.128.51"	"SENT: 220 mail.ourdomain.com"
"SMTPD"	3092	9701769	"2019-03-28 18:42:23.756"	"209.85.128.42"	"RECEIVED: EHLO mail-wm1-f42.google.com"
"SMTPD"	3092	9701769	"2019-03-28 18:42:23.756"	"209.85.128.42"	"SENT: 250-mail.ourdomain.com[nl]250-SIZE 50000000[nl]250-STARTTLS[nl]250 HELP"
"SMTPD"	3308	9701769	"2019-03-28 18:42:23.803"	"209.85.128.42"	"RECEIVED: STARTTLS"
"SMTPD"	3308	9701769	"2019-03-28 18:42:23.803"	"209.85.128.42"	"SENT: 220 Ready to start TLS"
"SMTPD"	3340	9701770	"2019-03-28 18:42:23.803"	"209.85.128.51"	"RECEIVED: EHLO mail-wm1-f51.google.com"
"SMTPD"	3092	9701760	"2019-03-28 18:42:23.803"	"209.85.128.50"	"RECEIVED: EHLO mail-wm1-f50.google.com"
"SMTPD"	3340	9701770	"2019-03-28 18:42:23.803"	"209.85.128.51"	"SENT: 250-mail.ourdomain.com[nl]250-SIZE 50000000[nl]250-STARTTLS[nl]250 HELP"
"SMTPD"	3092	9701760	"2019-03-28 18:42:23.803"	"209.85.128.50"	"SENT: 250-mail.ourdomain.com[nl]250-SIZE 50000000[nl]250 HELP"
"SMTPD"	3092	9701768	"2019-03-28 18:42:23.819"	"209.85.128.51"	"RECEIVED: EHLO mail-wm1-f51.google.com"
"SMTPD"	3092	9701768	"2019-03-28 18:42:23.819"	"209.85.128.51"	"SENT: 250-mail.ourdomain.com[nl]250-SIZE 50000000[nl]250 HELP"
"SMTPD"	3092	9701770	"2019-03-28 18:42:23.850"	"209.85.128.51"	"RECEIVED: STARTTLS"
"SMTPD"	3340	9701760	"2019-03-28 18:42:23.850"	"209.85.128.50"	"RECEIVED: MAIL FROM:<ihracat@external.com> SIZE=331252"
"SMTPD"	3092	9701770	"2019-03-28 18:42:23.850"	"209.85.128.51"	"SENT: 220 Ready to start TLS"
"SMTPD"	4388	9701768	"2019-03-28 18:42:23.881"	"209.85.128.51"	"RECEIVED: MAIL FROM:<ihracat@external.com> SIZE=331249"
"SMTPD"	3092	9701769	"2019-03-28 18:42:23.975"	"209.85.128.42"	"RECEIVED: EHLO mail-wm1-f42.google.com"
"SMTPD"	3092	9701769	"2019-03-28 18:42:23.975"	"209.85.128.42"	"SENT: 250-mail.ourdomain.com[nl]250-SIZE 50000000[nl]250 HELP"
"SMTPD"	5636	9701769	"2019-03-28 18:42:24.022"	"209.85.128.42"	"RECEIVED: MAIL FROM:<ihracat@external.com> SIZE=331249"
"SMTPD"	6080	9701770	"2019-03-28 18:42:24.022"	"209.85.128.51"	"RECEIVED: EHLO mail-wm1-f51.google.com"
"SMTPD"	6080	9701770	"2019-03-28 18:42:24.022"	"209.85.128.51"	"SENT: 250-mail.ourdomain.com[nl]250-SIZE 50000000[nl]250 HELP"
"SMTPD"	5296	9701770	"2019-03-28 18:42:24.069"	"209.85.128.51"	"RECEIVED: MAIL FROM:<ihracat@external.com> SIZE=331251"
"SMTPD"	3340	9701760	"2019-03-28 18:42:24.209"	"209.85.128.50"	"SENT: 250 OK"
"SMTPD"	3340	9701760	"2019-03-28 18:42:24.256"	"209.85.128.50"	"RECEIVED: RCPT TO:<pinar.eren@ourdomain.com>"
"SMTPD"	4388	9701768	"2019-03-28 18:42:24.569"	"209.85.128.51"	"SENT: 250 OK"
"SMTPD"	4388	9701768	"2019-03-28 18:42:24.615"	"209.85.128.51"	"RECEIVED: RCPT TO:<izabela.oztan@ourdomain.com>"
"SMTPD"	5296	9701770	"2019-03-28 18:42:24.912"	"209.85.128.51"	"SENT: 250 OK"
"SMTPD"	5296	9701770	"2019-03-28 18:42:24.959"	"209.85.128.51"	"RECEIVED: RCPT TO:<altan.bala@ourdomain.com>"
"SMTPD"	3340	9701760	"2019-03-28 18:42:25.272"	"209.85.128.50"	"SENT: 250 OK"
"SMTPD"	3340	9701760	"2019-03-28 18:42:25.319"	"209.85.128.50"	"RECEIVED: DATA"
"SMTPD"	3340	9701760	"2019-03-28 18:42:25.319"	"209.85.128.50"	"SENT: 354 OK, send."
"SMTPD"	5636	9701769	"2019-03-28 18:42:25.631"	"209.85.128.42"	"SENT: 250 OK"
"SMTPD"	5636	9701769	"2019-03-28 18:42:25.678"	"209.85.128.42"	"RECEIVED: RCPT TO:<mehmet.ozkan@ourdomain.com>"
"SMTPD"	4388	9701768	"2019-03-28 18:42:25.975"	"209.85.128.51"	"SENT: 250 OK"
"SMTPD"	4388	9701771	"2019-03-28 18:42:25.975"	"209.85.128.53"	"SENT: 220 mail.ourdomain.com"
"SMTPD"	3092	9701768	"2019-03-28 18:42:26.022"	"209.85.128.51"	"RECEIVED: DATA"
"SMTPD"	3092	9701768	"2019-03-28 18:42:26.022"	"209.85.128.51"	"SENT: 354 OK, send."
"SMTPD"	3340	9701771	"2019-03-28 18:42:26.069"	"209.85.128.53"	"RECEIVED: EHLO mail-wm1-f53.google.com"
"SMTPD"	3340	9701771	"2019-03-28 18:42:26.069"	"209.85.128.53"	"SENT: 250-mail.ourdomain.com[nl]250-SIZE 50000000[nl]250-STARTTLS[nl]250 HELP"
"SMTPD"	4388	9701771	"2019-03-28 18:42:26.115"	"209.85.128.53"	"RECEIVED: STARTTLS"
"SMTPD"	4388	9701771	"2019-03-28 18:42:26.115"	"209.85.128.53"	"SENT: 220 Ready to start TLS"
"SMTPD"	4388	9701771	"2019-03-28 18:42:26.287"	"209.85.128.53"	"RECEIVED: EHLO mail-wm1-f53.google.com"
"SMTPD"	4388	9701771	"2019-03-28 18:42:26.287"	"209.85.128.53"	"SENT: 250-mail.ourdomain.com[nl]250-SIZE 50000000[nl]250 HELP"
"SMTPD"	3092	9701771	"2019-03-28 18:42:26.319"	"209.85.128.53"	"RECEIVED: MAIL FROM:<ihracat@external.com> SIZE=331248"
"SMTPD"	3092	9701771	"2019-03-28 18:42:26.709"	"209.85.128.53"	"SENT: 250 OK"
"SMTPD"	4048	9701760	"2019-03-28 18:42:26.725"	"209.85.128.50"	"SENT: 250 Queued (1.024 seconds)"
"SMTPD"	3340	9701771	"2019-03-28 18:42:26.756"	"209.85.128.53"	"RECEIVED: RCPT TO:<onur.kantarci@ourdomain.com>"
"SMTPD"	4388	9701760	"2019-03-28 18:42:26.772"	"209.85.128.50"	"RECEIVED: QUIT"
"SMTPD"	4388	9701760	"2019-03-28 18:42:26.772"	"209.85.128.50"	"SENT: 221 goodbye"
"SMTPD"	1040	9701768	"2019-03-28 18:42:27.428"	"209.85.128.51"	"SENT: 250 Queued (1.024 seconds)"
"SMTPD"	5636	9701769	"2019-03-28 18:42:27.428"	"209.85.128.42"	"SENT: 250 OK"
"SMTPD"	3092	9701768	"2019-03-28 18:42:27.459"	"209.85.128.51"	"RECEIVED: QUIT"
"SMTPD"	3092	9701768	"2019-03-28 18:42:27.459"	"209.85.128.51"	"SENT: 221 goodbye"
"SMTPD"	3092	9701769	"2019-03-28 18:42:27.475"	"209.85.128.42"	"RECEIVED: DATA"
"SMTPD"	3092	9701769	"2019-03-28 18:42:27.475"	"209.85.128.42"	"SENT: 354 OK, send."
"SMTPD"	3340	9701771	"2019-03-28 18:42:27.772"	"209.85.128.53"	"SENT: 250 OK"
"SMTPD"	4024	9701771	"2019-03-28 18:42:27.819"	"209.85.128.53"	"RECEIVED: DATA"
"SMTPD"	4024	9701771	"2019-03-28 18:42:27.819"	"209.85.128.53"	"SENT: 354 OK, send."
"SMTPD"	5296	9701770	"2019-03-28 18:42:28.131"	"209.85.128.51"	"SENT: 250 OK"
"SMTPD"	5296	9701770	"2019-03-28 18:42:28.178"	"209.85.128.51"	"RECEIVED: DATA"
"SMTPD"	5296	9701770	"2019-03-28 18:42:28.178"	"209.85.128.51"	"SENT: 354 OK, send."
"SMTPD"	1040	9701769	"2019-03-28 18:42:28.850"	"209.85.128.42"	"SENT: 250 Queued (1.024 seconds)"
"SMTPD"	3340	9701769	"2019-03-28 18:42:29.006"	"209.85.128.42"	"RECEIVED: QUIT"
"SMTPD"	3340	9701769	"2019-03-28 18:42:29.006"	"209.85.128.42"	"SENT: 221 goodbye"
"SMTPD"	4048	9701771	"2019-03-28 18:42:29.225"	"209.85.128.53"	"SENT: 250 Queued (1.024 seconds)"
"SMTPD"	6080	9701771	"2019-03-28 18:42:29.256"	"209.85.128.53"	"RECEIVED: QUIT"
"SMTPD"	6080	9701771	"2019-03-28 18:42:29.256"	"209.85.128.53"	"SENT: 221 goodbye"
"SMTPD"	2564	9701770	"2019-03-28 18:42:29.553"	"209.85.128.51"	"SENT: 250 Queued (1.024 seconds)"
"SMTPD"	360	9701770	"2019-03-28 18:42:29.600"	"209.85.128.51"	"RECEIVED: QUIT"
"SMTPD"	360	9701770	"2019-03-28 18:42:29.600"	"209.85.128.51"	"SENT: 221 goodbye"
Katip
--
HMS 5.7.0-B2428-LTS-64-bit, MySQL 5.7.24, SA 3.4.2, ClamAV 0.101.2 + SaneS

User avatar
SorenR
Senior user
Senior user
Posts: 3043
Joined: 2006-08-21 15:38
Location: Denmark

Re: Presumably unavoidable duplicate receipts

Post by SorenR » 2019-03-28 22:04

Hmm... Your Google is different from mine :mrgreen:

Code: Select all

"SMTPD"	2792	2688	"2019-03-28 20:58:08.148"	"209.85.161.44"	"SENT: 220 mx.acme.inc ESMTP"
"SMTPD"	2792	2688	"2019-03-28 20:58:08.398"	"209.85.161.44"	"RECEIVED: EHLO mail-yw1-f44.google.com"
"SMTPD"	2792	2688	"2019-03-28 20:58:28.147"	"209.85.161.44"	"SENT: 250-mx.acme.inc[nl]250 SIZE"
"SMTPD"	2792	2688	"2019-03-28 20:58:28.366"	"209.85.161.44"	"RECEIVED: MAIL FROM:<SorenR@external.com> SIZE=2358"
"SMTPD"	2792	2688	"2019-03-28 20:58:28.366"	"209.85.161.44"	"SENT: 250 OK"
"SMTPD"	2792	2688	"2019-03-28 20:58:28.616"	"209.85.161.44"	"RECEIVED: RCPT TO:<SorenR@acme.inc>"
"SMTPD"	2792	0	"2019-03-28 20:58:28.772"	"TCP"	"DNS lookup: 44.161.85.209.zen.spamhaus.org, 0 addresses found: (none), Match: False"
"SMTPD"	2792	0	"2019-03-28 20:58:38.787"	"TCP"	"DNS lookup: 44.161.85.209.b.barracudacentral.org, 0 addresses found: (none), Match: False"
"SMTPD"	2792	0	"2019-03-28 20:58:40.803"	"TCP"	"DNS lookup: 44.161.85.209.bl.spamcop.net, 0 addresses found: (none), Match: False"
"SMTPD"	2792	2688	"2019-03-28 20:58:40.803"	"209.85.161.44"	"SENT: 250 OK"
"SMTPD"	2792	2688	"2019-03-28 20:58:41.225"	"209.85.161.44"	"RECEIVED: RCPT TO:<postmaster@acme.inc>"
"SMTPD"	2792	0	"2019-03-28 20:58:41.225"	"TCP"	"DNS lookup: 44.161.85.209.zen.spamhaus.org, 0 addresses found: (none), Match: False"
"SMTPD"	2792	0	"2019-03-28 20:58:41.225"	"TCP"	"DNS lookup: 44.161.85.209.b.barracudacentral.org, 0 addresses found: (none), Match: False"
"SMTPD"	2792	0	"2019-03-28 20:58:41.225"	"TCP"	"DNS lookup: 44.161.85.209.bl.spamcop.net, 0 addresses found: (none), Match: False"
"SMTPD"	2792	2688	"2019-03-28 20:58:41.225"	"209.85.161.44"	"SENT: 250 OK"
"SMTPD"	2792	2688	"2019-03-28 20:58:41.506"	"209.85.161.44"	"RECEIVED: RCPT TO:<spam@acme.inc>"
"SMTPD"	2792	0	"2019-03-28 20:58:41.506"	"TCP"	"DNS lookup: 44.161.85.209.zen.spamhaus.org, 0 addresses found: (none), Match: False"
"SMTPD"	2792	0	"2019-03-28 20:58:41.506"	"TCP"	"DNS lookup: 44.161.85.209.b.barracudacentral.org, 0 addresses found: (none), Match: False"
"SMTPD"	2792	0	"2019-03-28 20:58:41.506"	"TCP"	"DNS lookup: 44.161.85.209.bl.spamcop.net, 0 addresses found: (none), Match: False"
"SMTPD"	2792	2688	"2019-03-28 20:58:41.506"	"209.85.161.44"	"SENT: 250 OK"
"SMTPD"	2792	2688	"2019-03-28 20:58:41.819"	"209.85.161.44"	"RECEIVED: DATA"
"SMTPD"	2792	2688	"2019-03-28 20:59:01.193"	"209.85.161.44"	"SENT: 354 OK, send."
"SMTPD"	1476	2688	"2019-03-28 20:59:05.896"	"209.85.161.44"	"SENT: 250 Queued (4.625 seconds)"
"SMTPD"	2792	2688	"2019-03-28 20:59:06.193"	"209.85.161.44"	"RECEIVED: QUIT"
"SMTPD"	2792	2688	"2019-03-28 20:59:06.193"	"209.85.161.44"	"SENT: 221 goodbye"
SørenR.

The quantum rule of insecurity which states that the act of observing how vulnerable a host or service is changes the insecurity level of the service.

User avatar
SorenR
Senior user
Senior user
Posts: 3043
Joined: 2006-08-21 15:38
Location: Denmark

Re: Presumably unavoidable duplicate receipts

Post by SorenR » 2019-03-28 22:08

Code: Select all

Return-Path: SorenR@External.com
Delivered-To: spam@acme.inc
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on datacenter.acme.inc
X-Spam-Level: 
X-Spam-Status: No, score=-2.0 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE
 autolearn=disabled version=3.4.0
X-Spam-ASN: AS15169 209.85.128.0/17
X-Spam-Virus: No
Received: from mail-yw1-f44.google.com (mail-yw1-f44.google.com [209.85.161.44]) by
 mx.acme.inc ; Thu, 28 Mar 2019 20:59:01 +0100
Received: by mail-yw1-f44.google.com with SMTP id m207so6469487ywd.5; Thu, 28 Mar 2019
 12:58:28 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to:cc;
 bh=K0NJx9dtpjBi0aibGqDGF1NKoyEJ4XWCtk/AZ4WRoNQ=; b=BX3U4w0jgGznl5z3hTy0k5/VFJs76y/+jwjPbfZkcn0mvx+jyz6AtDmVMGj2kxqYvR
 8/Ms7qQJ64dpXmuk5tutLG1wHdp1js68wrBeEbDE9+07H9kwnrtOL3pCfvYX3mNQ5eQt
 nG0IVsdTP2cXqthXDLOwjRkDC/nwWCQUJuYbxOj3xbd2LDwxpN9dMnTR3Agi+qGVx3HM
 Rld0fVUac8oM/r1I8yCqvGeMh32k5/HnlirZbngyY9ifldqwa08BX7UW19om4t07+IMM
 YIjhxIBrT1W0G4s4v50Y/Nc7bX/oPOel+IKaWUgLQwv+RgIksDVGUMa7AzPVMU3Z9VbF H72g==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc;
 bh=K0NJx9dtpjBi0aibGqDGF1NKoyEJ4XWCtk/AZ4WRoNQ=; b=ugMcXPjz0xjxopgrbPTZ3ofoAJrkeld8op2QQFjSzIsX00Ur3N/miO6MyyQNY7+Ffm
 2YGFZxKJKUyOWVqM5j87ISsKexPO6DEMLulG72m6tXnk42gaVN0vm+2xzsJIjWpNZPBL
 MEu1CRNKvcWN8c2oAC9Fig6b3NCdmfB0mENT/ilE1Yf0w/fP4xHaYAcBHVimG7qBCYw1
 edM4uklC3reg4tZDkFviLcEgB1v7IBfHCZxBPWYW+pZlj/i/hyvOB9w20jhN13j7XNMl
 GnMz06TGF0cp3hM48dU4j592E99NuVYVccemci7Ukq+l17VZti4DZvTErhf/aAY8IuPI y/rw==
X-Gm-Message-State: APjAAAUP795iJHRrUTQYuGfAE5L98B1k1smVkMB3dPlrUjZxG8fNTcnq
 OTIeCtTLC904yaecYkWMVYAhD8R5GvyD7sDr/uLdBb05
X-Google-Smtp-Source: APXvYqzzOlGEcE93FG3bdKSR7DmlzmFHlqqCwmQ+mBFCYo21BB1OmBu6XnHyzgn3Y3b8z2DnH+krFxu7/WVN0ar3TPI=
X-Received: by 2002:a81:4b8a:: with SMTP id y132mr38118531ywa.486.1553803066154; Thu, 28
 Mar 2019 12:57:46 -0700 (PDT)
MIME-Version: 1.0
From: =?UTF-8?Q?S=C3=B8ren_Rathje?= <SorenR@External.com>
Date: Thu, 28 Mar 2019 20:57:34 +0100
Message-ID: <CALjmzwdDwXR0HcpBMwU3UZjHpm3pMNoNWtBV8Greu_-X0Y4pcA@mail.gmail.com>
Subject: test
To: =?UTF-8?Q?S=C3=B8ren_Rathje?= <soren@acme.inc>, postmaster@acme.inc
Cc: spam@acme.inc
Content-Type: multipart/alternative; boundary="000000000000cf24ad05852cf7d1"
X-Envelope-To: soren@acme.inc, postmaster@acme.inc, spam@acme.inc
X-Envelope-OriginalTo: soren@acme.inc, postmaster@acme.inc, spam@acme.inc
X-Envelope-From: SorenR@External.com
X-Envelope-HELO: mail-yw1-f44.google.com
X-Envelope-IPAddress: 209.85.161.44

--000000000000cf24ad05852cf7d1
Content-Type: text/plain; charset="UTF-8"

test

--000000000000cf24ad05852cf7d1
Content-Type: text/html; charset="UTF-8"

<div dir="ltr">test</div>

--000000000000cf24ad05852cf7d1--
SørenR.

The quantum rule of insecurity which states that the act of observing how vulnerable a host or service is changes the insecurity level of the service.

User avatar
katip
Senior user
Senior user
Posts: 624
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Presumably unavoidable duplicate receipts

Post by katip » 2019-03-29 07:57

SorenR wrote:
2019-03-28 22:04
Hmm... Your Google is different from mine :mrgreen:
:lol: probably the same.
was it your free Gmail account you sent from? i've done the same test from my Gmail and message arrived to HMS unsplitted.
however the customer from my above example sent it from their own company address via Google SMTP service, obviously a "business" package. this supports my assumption that paid services split envelopes as they charge per recipient, not per message.

Code: Select all

Return-Path: <ihracat@customers-own-domain.com>
Received: from DESKTOPT4GJT07 ([xx.xx.124.181]) by smtp.gmail.com with ESMTPSA id
 g8sm53306742wro.77.2019.03.28.08.42.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256
 bits=128/128); Thu, 28 Mar 2019 08:42:22 -0700 (PDT)
From: "Selda CEYLAN" <ihracat@customers-own-domain.com>
anyway, this weekend i'll dump all Message-ID / recipient pairs to a table and see it's usefulness.
BTW, just to save on trial/error,
does rule calls support arguments like "call function checkMessageID(recipient_2)" ?
thanks.
Katip
--
HMS 5.7.0-B2428-LTS-64-bit, MySQL 5.7.24, SA 3.4.2, ClamAV 0.101.2 + SaneS

User avatar
SorenR
Senior user
Senior user
Posts: 3043
Joined: 2006-08-21 15:38
Location: Denmark

Re: Presumably unavoidable duplicate receipts

Post by SorenR » 2019-03-29 08:51

first rule run function to lookup recipient_2@local.com and add header with result, second rule check header...
SørenR.

The quantum rule of insecurity which states that the act of observing how vulnerable a host or service is changes the insecurity level of the service.

User avatar
katip
Senior user
Senior user
Posts: 624
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Presumably unavoidable duplicate receipts

Post by katip » 2019-03-29 09:19

SorenR wrote:
2019-03-29 08:51
first rule run function to lookup recipient_2@local.com and add header with result, second rule check header...
alles klar!
thanks.
Katip
--
HMS 5.7.0-B2428-LTS-64-bit, MySQL 5.7.24, SA 3.4.2, ClamAV 0.101.2 + SaneS

User avatar
katip
Senior user
Senior user
Posts: 624
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Presumably unavoidable duplicate receipts

Post by katip » 2019-03-31 23:45

SorenR wrote:
2019-03-29 08:51
first rule run function to lookup recipient_2@local.com and add header with result, second rule check header...
it turns out it's even with a single account rule possible.

what i've done:
1.
write Message-ID and all oMessage.Recipients(i).Address from every processed message in a table at OnAcceptMessage stage.
table can be emptied during midnight maintenance.

2.
account rule:
If message size > 0
Set header value X-HMS-Forward "recipient_2, recipient_3, recipient_4..."
Run function GetSetFwdHeader

3.
function reads from message Message-Id and X-HMS-Forward addresses.
then it looks up in table if "Message-Id & address" pair exists. makes copies of message and does addRecipient to be sent to recipients who were in in X-HMS-Forward but not yet involved. those in X-HMS-Forward who were otherwise involved (CC, alias, distlist...) as found in table, get their own copies and not additional duplicates by forwarding.

summary:
subject may look trivial but is essential for me.
we use lots of internal forwards by global rules to avoid duplicates where backup staff gets copies of missing staff. many times sender adds both addresses, hence backup staff gets duplicates. global rules check recipient list but as explained above, paid SMTP services split messages in individual messages and relay one by one for each recipient. so, checking recipient list doesn't help. this may be a solution for those who suffer from same inconvenience.

Code: Select all

Sub OnAcceptMessage(oClient, oMessage)
' any other code here

Call FillMessageID(oMessage)
End Sub

'+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
' Non-Dup forwarding
'+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Sub FillMessageID(oMessage)

Dim Connection, ConnectString
Dim strMessageID, strAddress, strAddressDupC

	For i = oMessage.Headers.Count -1 To 0 Step -1
		If oMessage.Headers.Item(i).Name = "Message-ID" Then
			strMessageID = oMessage.Headers.Item(i).Value
		End If	
	Next

ConnectString = "Driver={MySQL ODBC 5.1 Driver};Server=192.168.1.10;PORT=3306;Database=test;User=<user>;Password=<password>"
Set Connection = CreateObject("ADODB.Connection")
Connection.Open ConnectString

	For i = 0 To oMessage.Recipients.Count-1
		strAddress = oMessage.Recipients(i).Address
		SqlExec = "INSERT INTO hm_messageID (messageID, address) VALUES ('" & strMessageID & "', '" & strAddress & "');"
		Connection.Execute(SqlExec)
	Next

Connection.Close

End Sub

'+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Function GetSetFwdHeader(oMessage)

Dim Connection, ConnectString, SqlGet, SqlAdd
Dim strMessageID, strFwdAddress, strUserAddress, strSubject, strCc, strTo
Dim oNewMessage
Dim oEventLog : Set oEventlog = CreateObject("hMailServer.EventLog")

    For i = oMessage.Headers.Count -1 To 0 Step -1
	    If Ucase(oMessage.Headers.Item(i).Name) = "MESSAGE-ID" Then
			strMessageID = oMessage.Headers.Item(i).Value
		ElseIf Ucase(oMessage.Headers.Item(i).Name) = "X-HMS-FORWARD" Then
			strFwdAddress = oMessage.Headers.Item(i).Value ' a comma seperated list of addresses
		ElseIf Ucase(oMessage.Headers.Item(i).Name) = "SUBJECT" Then
			strSubject = oMessage.Headers.Item(i).Value
		ElseIf Ucase(oMessage.Headers.Item(i).Name) = "CC" Then
			strCc = oMessage.Headers.Item(i).Value
		ElseIf Ucase(oMessage.Headers.Item(i).Name) = "TO" Then
			strTo = oMessage.Headers.Item(i).Value
		End If
    Next

	strUserAddress = oMessage.Recipients(0).Address 	' address of the account which called the function
	arrFwdAddress = Split(strFwdAddress,",") 		' here we get list of addresses to forward the message to

Set Connection = CreateObject("ADODB.Connection")
Set Recordset = CreateObject("ADODB.Recordset")
ConnectString = "Driver={MySQL ODBC 5.1 Driver};Server=192.168.1.10;PORT=3306;Database=test;User=<user>;Password=<password>"

	For i=0 to UBound(arrFwdAddress)
	'for each X-HMS-Forward member we check if any of them was already involved and has received the message
	
		arrFwdAdd = Trim(arrFwdAddress(i))

		SqlGet = "SELECT messageID, address FROM hm_messageID WHERE messageID = '"&strMessageID&"' and address = '"&arrFwdAdd&"';"
		Connection.Open ConnectString
		Recordset.Open SqlGet, Connection

		If Recordset.EOF Then ' now we know who was not involved, he/she has to get the message forwarded
			SqlAdd = "INSERT INTO hm_messageID (messageID, address) VALUES ('" & strMessageID & "', '" & arrFwdAdd & "');"
			Connection.Execute(SqlAdd)    'this may be useless, i don't know now. why updating the table?
			
				' now we start forwarding
				Set oNewMessage = CreateObject("hMailServer.Message")
				oNewMessage.From = oMessage.From
				oNewMessage.FromAddress = oMessage.FromAddress
				oNewMessage.AddRecipient "", arrFwdAdd	' <--- job done
				oNewMessage.Subject = oMessage.Subject
				oNewMessage.Body = oMessage.Body
				oNewMessage.HeaderValue("To") = strTo
				If strCc <> "" Then oNewMessage.HeaderValue("Cc") = strCc
				oNewMessage.Save
				
				'just for logging, can be ignored
				oEventLog.Write("Forward actions : " & strUserAddress & " -> " & arrFwdAdd & " - Subject: " & strSubject)
		End If
		
		Connection.Close
		
	Next

End Function
Katip
--
HMS 5.7.0-B2428-LTS-64-bit, MySQL 5.7.24, SA 3.4.2, ClamAV 0.101.2 + SaneS

User avatar
SorenR
Senior user
Senior user
Posts: 3043
Joined: 2006-08-21 15:38
Location: Denmark

Re: Presumably unavoidable duplicate receipts

Post by SorenR » 2019-04-01 09:09

I assume your hMailServer is using MySQL...

I have noticed by looking at my taskmanager that every time I create a new database connection windows use extra memory and sometimes this memory is not released immediately. Also it takes time (milliseconds) to load/unload the ODBC module.

I have two functions that add the connecting IP address to a table on "OnClientConnect" and delete the same IP address on "OnAcceptMessage". By using the hMailServer API to do this the memory usage stays the same and the extra load is microscopic.
I have another external script that checks the table and if the IP address is not deleted again it means that no message was sent and if this happens 3 times I ban the IP address 8)

Anyways, I assume this "message folding" is about 20% of your emails so this should reduce some of the every day load on the server.

Oh and by the way... the object "oDB" is your hMailServer database and the API calls are Write Only!

Code: Select all

Sub FillMessageID(oMessage)

   Dim oApp : Set oApp = CreateObject("hMailServer.Application")
   Call oApp.Authenticate(ADMIN, PASSWORD)
   Dim strSQL, oDB : Set oDB = oApp.Database
   If oMessage.HeaderValue("Message-ID") <> "" Then
      For i = 0 To oMessage.Recipients.Count-1
         strSQL = "INSERT INTO hm_messageID (messageID, address) VALUES ('" & oMessage.HeaderValue("Message-ID") & "', '" & oMessage.Recipients(i).Address & "');"
         Call oDB.ExecuteSQL(strSQL)
      Next
   End If
   Set oApp = Nothing
   Set oDB = Nothing

End Sub
SørenR.

The quantum rule of insecurity which states that the act of observing how vulnerable a host or service is changes the insecurity level of the service.

Post Reply