Address validation

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
User avatar
katip
Senior user
Senior user
Posts: 966
Joined: 2006-12-22 07:58
Location: Istanbul

Address validation

Post by katip » 2021-08-07 10:14

any idea how HMS validates email addresses?
for various purposes i use in my scripts:

Code: Select all

oRegEx.Pattern = "^[^\W][\w\.=-]+\@[^\W][\w\.-]+\.[\w]{2,8}$"
which works fine, but in theory, of course it's not so simple.

back to topic..
one colleague (client Thunderbird) "replied all" to an email where one of inherited recipients' address was:

Code: Select all

<"e-somecompany.cz=29'?= name.surname"@e-somecompany.cz>
HMS perfectly accepted and sent it out. NDR returned:

Code: Select all

(...)
The following recipient(s) could not be reached:

"e-somecompany.cz=29'?= name.surname"@e-somecompany.cz
   Error Type: SMTP
   Remote server (104.47.14.36) issued an error.
   hMailServer sent: RCPT TO:<"e-somecompany.cz=29'?= name.surname"@e-somecompany.cz>
   Remote server replied: 550 5.4.1 Recipient address rejected: Access denied. AS(201806281) [VI1EUR04FT036.eop-eur04.prod.protection.outlook.com]
any suggestions?
Katip
--
HMS 5.7.0, MariaDB 10.4.10, SA 3.4.4, ClamAV 0.103.2

User avatar
mattg
Moderator
Moderator
Posts: 21639
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Address validation

Post by mattg » 2021-08-07 10:40

I remember Martin once answering this question

I think it was in the context of AUTH for a domain without a TLD, ie user@example
But I can't find that post
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
SorenR
Senior user
Senior user
Posts: 4894
Joined: 2006-08-21 15:38
Location: Denmark

Re: Address validation

Post by SorenR » 2021-08-07 11:19

katip wrote:
2021-08-07 10:14
any idea how HMS validates email addresses?
for various purposes i use in my scripts:

Code: Select all

oRegEx.Pattern = "^[^\W][\w\.=-]+\@[^\W][\w\.-]+\.[\w]{2,8}$"
which works fine, but in theory, of course it's not so simple.

back to topic..
one colleague (client Thunderbird) "replied all" to an email where one of inherited recipients' address was:

Code: Select all

<"e-somecompany.cz=29'?= name.surname"@e-somecompany.cz>
HMS perfectly accepted and sent it out. NDR returned:

Code: Select all

(...)
The following recipient(s) could not be reached:

"e-somecompany.cz=29'?= name.surname"@e-somecompany.cz
   Error Type: SMTP
   Remote server (104.47.14.36) issued an error.
   hMailServer sent: RCPT TO:<"e-somecompany.cz=29'?= name.surname"@e-somecompany.cz>
   Remote server replied: 550 5.4.1 Recipient address rejected: Access denied. AS(201806281) [VI1EUR04FT036.eop-eur04.prod.protection.outlook.com]
any suggestions?
Bug in Thunderbird? Looks like a From header syntax.
SørenR.

Engineer (noun)
- I'm Not Arguing, I'm Just Explaining Why I'm Right

User avatar
katip
Senior user
Senior user
Posts: 966
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Address validation

Post by katip » 2021-08-07 11:22

mattg wrote:
2021-08-07 10:40
I remember Martin once answering this question

I think it was in the context of AUTH for a domain without a TLD, ie user@example
But I can't find that post
i think it doesn't validate external addresses at all. just puts the message in queue even if recipient address is malformed.
Katip
--
HMS 5.7.0, MariaDB 10.4.10, SA 3.4.4, ClamAV 0.103.2

User avatar
katip
Senior user
Senior user
Posts: 966
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Address validation

Post by katip » 2021-08-07 11:33

SorenR wrote:
2021-08-07 11:19
Bug in Thunderbird? Looks like a From header syntax.
no, From is fine. just one of the external recipients was malformed. it was in CC when original message arrived. colleague only did a "reply all", as a result it was taken by TB again to CC.
i.e.

Original incoming mail:

Code: Select all

From: some@external.com
To : our.colleague@local.com
CC : malformed@address
Reply All:

Code: Select all

From: our.colleague@local.com
To : some@external.com
CC : malformed@address
Katip
--
HMS 5.7.0, MariaDB 10.4.10, SA 3.4.4, ClamAV 0.103.2

User avatar
SorenR
Senior user
Senior user
Posts: 4894
Joined: 2006-08-21 15:38
Location: Denmark

Re: Address validation

Post by SorenR » 2021-08-07 12:27

katip wrote:
2021-08-07 11:33
SorenR wrote:
2021-08-07 11:19
Bug in Thunderbird? Looks like a From header syntax.
no, From is fine. just one of the external recipients was malformed. it was in CC when original message arrived. colleague only did a "reply all", as a result it was taken by TB again to CC.
i.e.

Original incoming mail:

Code: Select all

From: some@external.com
To : our.colleague@local.com
CC : malformed@address
Reply All:

Code: Select all

From: our.colleague@local.com
To : some@external.com
CC : malformed@address
This one is passed on from Thunderbird:

hMailServer sent: RCPT TO:<"e-somecompany.cz=29'?= name.surname"@e-somecompany.cz> !
SørenR.

Engineer (noun)
- I'm Not Arguing, I'm Just Explaining Why I'm Right

User avatar
jimimaseye
Moderator
Moderator
Posts: 9236
Joined: 2011-09-08 17:48

Re: Address validation

Post by jimimaseye » 2021-08-07 12:33

I once helped martin solve the regex for address format validation. It is in github under an issue somewhere if you care to search for it (i can't - I'm on my holi-bobs). There is no validation of external addresses other than an email format validation.

[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
katip
Senior user
Senior user
Posts: 966
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Address validation

Post by katip » 2021-08-07 13:15

jimimaseye wrote:
2021-08-07 12:33
I once helped martin solve the regex for address format validation. It is in github under an issue somewhere if you care to search for it (i can't - I'm on my holi-bobs). There is no validation of external addresses other than an email format validation.
sorry for the topic title. actually it should read "Adress format validation". my bad.
i was talking about external mail address format validation for outbound mails.

IF (and i don't think so, maybe some basics such as existance of "@" and ".tld") HMS does such validation prior to accepting a message, i understand it lacks at some point.
otherwise, pls ignore my question. i.e. sender must take care for external address format conformity or gets back an NDR.
thanks.
Katip
--
HMS 5.7.0, MariaDB 10.4.10, SA 3.4.4, ClamAV 0.103.2

User avatar
SorenR
Senior user
Senior user
Posts: 4894
Joined: 2006-08-21 15:38
Location: Denmark

Re: Address validation

Post by SorenR » 2021-08-07 14:12

https://github.com/hmailserver/hmailser ... 02250be4f6

Code to validate email (5.6.8 & 5.7)
SørenR.

Engineer (noun)
- I'm Not Arguing, I'm Just Explaining Why I'm Right

User avatar
katip
Senior user
Senior user
Posts: 966
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Address validation

Post by katip » 2021-08-07 14:45

SorenR wrote:
2021-08-07 14:12
https://github.com/hmailserver/hmailser ... 02250be4f6

Code to validate email (5.6.8 & 5.7)
ok, correct me pls if i'm wrong. this is the regex (line 64):

Code: Select all

^((\"[^<>@\\]+\")|([^<> @\\\"]+))@(\[([0-9]{1,3}\.){3}[0-9]{1,3}\]|(?=.{1,255}$)((?!-|\.)[a-zA-Z0-9-]{0,62}[a-zA-Z0-9])(|\.(?!-|\.)[a-zA-Z0-9-]{0,62}[a-zA-Z0-9]){1,126})$
right?
indeed it matches

Code: Select all

"e-somecompany.cz=29'?= name.surname"@e-somecompany.cz
is the address good you mean?
Katip
--
HMS 5.7.0, MariaDB 10.4.10, SA 3.4.4, ClamAV 0.103.2

User avatar
SorenR
Senior user
Senior user
Posts: 4894
Joined: 2006-08-21 15:38
Location: Denmark

Re: Address validation

Post by SorenR » 2021-08-07 15:09

String is escaped... \" = " and \\ = \ due to how C++ handles strings. Syntax is Perl IIRC.
SørenR.

Engineer (noun)
- I'm Not Arguing, I'm Just Explaining Why I'm Right

User avatar
SorenR
Senior user
Senior user
Posts: 4894
Joined: 2006-08-21 15:38
Location: Denmark

Re: Address validation

Post by SorenR » 2021-08-07 15:16

Found this... https://emailregex.com/

Code: Select all

General Email Regex (RFC 5322 Official Standard)

(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])
Image
SørenR.

Engineer (noun)
- I'm Not Arguing, I'm Just Explaining Why I'm Right

User avatar
katip
Senior user
Senior user
Posts: 966
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Address validation

Post by katip » 2021-08-07 15:25

SorenR wrote:
2021-08-07 15:09
String is escaped... \" = " and \\ = \ due to how C++ handles strings. Syntax is Perl IIRC.
yes before testing i replaced any \\ with \
anyway:
Local user <-> HMS

Code: Select all

"SMTPD"	2888	9723	"2021-08-07 09:35:21.652"	"192.168.0.4"	"RECEIVED: RCPT TO:<"e-somecompany.cz=29'?= name.surname"@e-somecompany.cz>"
"SMTPD"	2888	9723	"2021-08-07 09:35:21.652"	"192.168.0.4"	"SENT: 250 OK"
HMS <-> Recipient's system

Code: Select all

"SMTPC"	2888	9726	"2021-08-07 09:35:22.918"	"104.47.14.36"	"SENT: RCPT TO:<"e-somecompany.cz=29'?= name.surname"@e-somecompany.cz>"
"SMTPC"	4536	9726	"2021-08-07 09:35:23.012"	"104.47.14.36"	"RECEIVED: 550 5.4.1 Recipient address rejected: Access denied. AS(201806281) [VI1EUR04FT036.eop-eur04.prod.protection.outlook.com]"
Katip
--
HMS 5.7.0, MariaDB 10.4.10, SA 3.4.4, ClamAV 0.103.2

User avatar
katip
Senior user
Senior user
Posts: 966
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Address validation

Post by katip » 2021-08-07 15:31

SorenR wrote:
2021-08-07 15:16
Found this... https://emailregex.com/

Code: Select all

General Email Regex (RFC 5322 Official Standard)

(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])
we know this subject is widely debated.
that regex for example don't match "e-somecompany.cz=29'?= name.surname"@e-somecompany.cz (neither outlook.com accepts it) but HMS regex is fine with it. :?: :roll:
Katip
--
HMS 5.7.0, MariaDB 10.4.10, SA 3.4.4, ClamAV 0.103.2

User avatar
SorenR
Senior user
Senior user
Posts: 4894
Joined: 2006-08-21 15:38
Location: Denmark

Re: Address validation

Post by SorenR » 2021-08-07 15:42

katip wrote:
2021-08-07 15:31
SorenR wrote:
2021-08-07 15:16
Found this... https://emailregex.com/

Code: Select all

General Email Regex (RFC 5322 Official Standard)

(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])
we know this subject is widely debated.
that regex for example don't match "e-somecompany.cz=29'?= name.surname"@e-somecompany.cz (neither outlook.com accepts it) but HMS regex is fine with it. :?: :roll:
Create a bug report and link back to this subject. "Someone" will resolve it for your Alpha version software :mrgreen:
I think @RvdH can be talked into solving it when he's back from holiday ...
SørenR.

Engineer (noun)
- I'm Not Arguing, I'm Just Explaining Why I'm Right

User avatar
katip
Senior user
Senior user
Posts: 966
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Address validation

Post by katip » 2021-08-07 16:02

got it! valuable stuff here: https://emailregex.com/email-validation-summary/
If the quoted local part has a backslash, the following character is escaped and must not be 10 (LF), 13 (CR). This supersedes the previous rule, allowing spaces and quotation marks in the email address as long as they are escaped. (RFC 2822, section 3.4.1)
the problem is the "space" near the middle of the address. if you escape it then it matches that RFC 5322 Official Standard regex.
SorenR wrote:
2021-08-07 15:42
Create a bug report and link back to this subject. "Someone" will resolve it for your Alpha version software :mrgreen:
I think @RvdH can be talked into solving it when he's back from holiday ...
one day if it becomes a Beta and i'm still alive, then we'll see :lol:
Katip
--
HMS 5.7.0, MariaDB 10.4.10, SA 3.4.4, ClamAV 0.103.2

User avatar
SorenR
Senior user
Senior user
Posts: 4894
Joined: 2006-08-21 15:38
Location: Denmark

Re: Address validation

Post by SorenR » 2021-08-07 17:47

katip wrote:
2021-08-07 16:02
got it! valuable stuff here: https://emailregex.com/email-validation-summary/
If the quoted local part has a backslash, the following character is escaped and must not be 10 (LF), 13 (CR). This supersedes the previous rule, allowing spaces and quotation marks in the email address as long as they are escaped. (RFC 2822, section 3.4.1)
the problem is the "space" near the middle of the address. if you escape it then it matches that RFC 5322 Official Standard regex.
SorenR wrote:
2021-08-07 15:42
Create a bug report and link back to this subject. "Someone" will resolve it for your Alpha version software :mrgreen:
I think @RvdH can be talked into solving it when he's back from holiday ...
one day if it becomes a Beta and i'm still alive, then we'll see :lol:
Allthough I suspect I'm much older than you, the statistics says otherwise.

I live in Denmark and the hot topic right now is our former Prime Minister hosting a documentary on national TV about The Klaksvík battle on the Faeroe Islands (his wife is Faroese) and at the same time collecting votes to form a new political party to be introduced some time this fall.

You live in Turkey ... One wrong word in public and you are history ...

My wifes brother was married to a turkish girl for many years. She's now living in UK with her new husband. Allthough she still is part of our family we not seen them for a while, UK being UK and Covid being Covid ... :roll:
I do occational get the gossip though :wink:
SørenR.

Engineer (noun)
- I'm Not Arguing, I'm Just Explaining Why I'm Right

User avatar
katip
Senior user
Senior user
Posts: 966
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Address validation

Post by katip » 2021-08-07 18:48

SorenR wrote:
2021-08-07 17:47
Allthough I suspect I'm much older than you, the statistics says otherwise.

I live in Denmark and the hot topic right now is our former Prime Minister hosting a documentary on national TV about The Klaksvík battle on the Faeroe Islands (his wife is Faroese) and at the same time collecting votes to form a new political party to be introduced some time this fall.

You live in Turkey ... One wrong word in public and you are history ...
hmm, build '58. you?

i really don't understand how you can be so open to such perceptions.
just as an example, you know horrible wood fires here nowadays, like almost everywhere in Mediterranian area, and you should read how our goverment is under ruthless critcs bombardment as if THEY set the fire. but no, no one is hanged :lol:
anyway, at least here we can freely speak about things such as Israel -> Palestine cruelties without being stamped as antisemit and prosecuted the same day :roll:

today we live in a world where perception (i.e. mass/social media) dominates facts like never before. Joseph Goebbels couldn't even be an amateur today. alas.
Katip
--
HMS 5.7.0, MariaDB 10.4.10, SA 3.4.4, ClamAV 0.103.2

User avatar
SorenR
Senior user
Senior user
Posts: 4894
Joined: 2006-08-21 15:38
Location: Denmark

Re: Address validation

Post by SorenR » 2021-08-07 19:31

katip wrote:
2021-08-07 18:48
SorenR wrote:
2021-08-07 17:47
Allthough I suspect I'm much older than you, the statistics says otherwise.

I live in Denmark and the hot topic right now is our former Prime Minister hosting a documentary on national TV about The Klaksvík battle on the Faeroe Islands (his wife is Faroese) and at the same time collecting votes to form a new political party to be introduced some time this fall.

You live in Turkey ... One wrong word in public and you are history ...
hmm, build '58. you?

i really don't understand how you can be so open to such perceptions.
just as an example, you know horrible wood fires here nowadays, like almost everywhere in Mediterranian area, and you should read how our goverment is under ruthless critcs bombardment as if THEY set the fire. but no, no one is hanged :lol:
anyway, at least here we can freely speak about things such as Israel -> Palestine cruelties without being stamped as antisemit and prosecuted the same day :roll:

today we live in a world where perception (i.e. mass/social media) dominates facts like never before. Joseph Goebbels couldn't even be an amateur today. alas.
Build October '58.

Well, ask exile Turks why they have to look over their shoulders every time they leave their home. It is certainly real to them.

Re Israel/Palestine/racism... I've worked in 42 countries on 6 continents and lived in Asia for longer periods. Danes are generally not racists but stupidity and self promovation is not our favorite thing. In many countries people misread stupidity for racism.
The American "Cancel Culture" is something that really trigger foul language and I perfectly understand why BLM is regarded a terrorist group.

Young people today have no independance. We have a whole generation of "followers" with no inspiring thoughts of their own.

I was approached by an IT guy about a new idea (!) he got - You should have seen his face when I told him I had seen it done 20 years ago. -> "I bought a Coca Cola from a vending machine in 1999 by calling the number on the door from my mobile phone. The vending machine sat in the reception area of the finnish telephone company Sonera OY in Helsinki. My phone bill showed the purchase and accounting back in Denmark framed it and put it on the wall..."
SørenR.

Engineer (noun)
- I'm Not Arguing, I'm Just Explaining Why I'm Right

User avatar
RvdH
Senior user
Senior user
Posts: 1674
Joined: 2008-06-27 14:42
Location: Netherlands

Re: Address validation

Post by RvdH » 2021-08-12 07:42

SorenR wrote:
2021-08-07 15:42
katip wrote:
2021-08-07 15:31
SorenR wrote:
2021-08-07 15:16
Found this... https://emailregex.com/

Code: Select all

General Email Regex (RFC 5322 Official Standard)

(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])
we know this subject is widely debated.
that regex for example don't match "e-somecompany.cz=29'?= name.surname"@e-somecompany.cz (neither outlook.com accepts it) but HMS regex is fine with it. :?: :roll:
Create a bug report and link back to this subject. "Someone" will resolve it for your Alpha version software :mrgreen:
I think @RvdH can be talked into solving it when he's back from holiday ...
Note there is no perfect email regex, hence the 99.99%.
Isn't this more a problem for the sending MTA and/or script?
A (simple) single regex validating all possible email address formats doesn't exists as far as i know, a email address that might validate on one MTA can still be blocked on another MTA based on the implantation
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
jimimaseye
Moderator
Moderator
Posts: 9236
Joined: 2011-09-08 17:48

Re: Address validation

Post by jimimaseye » 2021-08-12 11:33

I remember having a conversation with both (angry) member Percepts and then with Martin about validations of email addresses.

The conclusion is that no validation of addresses is done at all, and only validation of the FORMAT is performed. I helped Martin with the regex (as I raised a discrepancy as the original weren't allowing some valid characters). Martin opted for a version which loosely did the necessary job (sufficing majority of addresses) because he wanted to keep it short and a more thorough regex matching all the complex combinations that are technically restricted would be a LOT longer and complex - his thinking being that any address that ultimately ends up being sent in comms to another server that mighty be wrong would ultimately be rejected by them anyway.

The question is, of course, whether his loose version of format validation allows us to RECEIVE such email addresses correctly or whether it is incorrectly rejecting them (maybe like the address quoted in this thread?). This, though, is a scenario that would be rare and he felt it would be an acceptable casualty given its infrequency (as 99.99% email addresses are usually normal no-fuss characters and format).

The above is all from memory of my exchange with Martin.

[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

palinka
Senior user
Senior user
Posts: 2975
Joined: 2017-09-12 17:57

Re: Address validation

Post by palinka » 2021-08-12 16:35

jimimaseye wrote:
2021-08-12 11:33
The question is, of course, whether his loose version of format validation allows us to RECEIVE such email addresses correctly or whether it is incorrectly rejecting them (maybe like the address quoted in this thread?). This, though, is a scenario that would be rare and he felt it would be an acceptable casualty given its infrequency (as 99.99% email addresses are usually normal no-fuss characters and format).
Doesn't hmailserver compare MAIL FROM to addresses in the database to conclude whether it should accept/reject? In that case, only an exact match is required.

User avatar
jimimaseye
Moderator
Moderator
Posts: 9236
Joined: 2011-09-08 17:48

Re: Address validation

Post by jimimaseye » 2021-08-12 19:48

palinka wrote:
2021-08-12 16:35
jimimaseye wrote:
2021-08-12 11:33
The question is, of course, whether his loose version of format validation allows us to RECEIVE such email addresses correctly or whether it is incorrectly rejecting them (maybe like the address quoted in this thread?). This, though, is a scenario that would be rare and he felt it would be an acceptable casualty given its infrequency (as 99.99% email addresses are usually normal no-fuss characters and format).
Doesn't hmailserver compare MAIL FROM to addresses in the database to conclude whether it should accept/reject? In that case, only an exact match is required.
I would think format validation comes first. If it doesn't pad the format validation then it doesn't get to the rest of the checking.

[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

palinka
Senior user
Senior user
Posts: 2975
Joined: 2017-09-12 17:57

Re: Address validation

Post by palinka » 2021-08-12 20:23

jimimaseye wrote:
2021-08-12 19:48
palinka wrote:
2021-08-12 16:35
jimimaseye wrote:
2021-08-12 11:33
The question is, of course, whether his loose version of format validation allows us to RECEIVE such email addresses correctly or whether it is incorrectly rejecting them (maybe like the address quoted in this thread?). This, though, is a scenario that would be rare and he felt it would be an acceptable casualty given its infrequency (as 99.99% email addresses are usually normal no-fuss characters and format).
Doesn't hmailserver compare MAIL FROM to addresses in the database to conclude whether it should accept/reject? In that case, only an exact match is required.
I would think format validation comes first. If it doesn't pad the format validation then it doesn't get to the rest of the checking.

[Entered by mobile. Excuse my spelling.]
I see your point. Maybe that's why it's intentionally loose.

What stands in the way of MAIL FROM and BAD COMMAND? Meaning what happens when MAIL FROM is not an email address?

User avatar
jimimaseye
Moderator
Moderator
Posts: 9236
Joined: 2011-09-08 17:48

Re: Address validation

Post by jimimaseye » 2021-08-12 23:56

palinka wrote:
2021-08-12 20:23
What stands in the way of MAIL FROM and BAD COMMAND? Meaning what happens when MAIL FROM is not an email address?
One for the code readers to answer.

[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

palinka
Senior user
Senior user
Posts: 2975
Joined: 2017-09-12 17:57

Re: Address validation

Post by palinka » 2021-08-13 02:20

Code: Select all

220 mydomain
HELO [MY-PC]
250 Hello.
MAIL FROM:<foo@bar.com>
250 OK
RCPT TO: foobar
550 A valid address is required.
RCPT TO: foo@bar.com
550 Delivery is not allowed to this address.
RCPT TO: real-user@mydomain
530 SMTP authentication is required.
Definitely looks like there is validation of some kind.

User avatar
katip
Senior user
Senior user
Posts: 966
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Address validation

Post by katip » 2021-08-13 05:53

jimimaseye wrote:
2021-08-12 11:33
The conclusion is that no validation of addresses is done at all, and only validation of the FORMAT is performed. I helped Martin with the regex (as I raised a discrepancy as the original weren't allowing some valid characters). Martin opted for a version which loosely did the necessary job (sufficing majority of addresses) because he wanted to keep it short and a more thorough regex matching all the complex combinations that are technically restricted would be a LOT longer and complex - his thinking being that any address that ultimately ends up being sent in comms to another server that mighty be wrong would ultimately be rejected by them anyway.
then it's been a reasonable approach IMHO.
i was asking this question just because i'd like to use the same format validation regex in my scripts, like HMS internally uses. so i assume currently (5.6.8 & 5.7) this is the one in StringParser.cpp line 64.
that one here https://emailregex.com (RFC 5322 Official???) is really exaggerated, accepts almost everything with an @ somewhere and something like a tld near the end, albeit conforming with RFC's here https://emailregex.com/email-validation-summary/
Katip
--
HMS 5.7.0, MariaDB 10.4.10, SA 3.4.4, ClamAV 0.103.2

User avatar
RvdH
Senior user
Senior user
Posts: 1674
Joined: 2008-06-27 14:42
Location: Netherlands

Re: Address validation

Post by RvdH » 2021-09-09 14:08

I had a client contacting me the other day, people where unable to fill in their email address on a online form.
The clients website is ASP.NET website, and I used the standard RegularExpressionValidator for email validation

ValidationExpression (default):

Code: Select all

\w+([-+.']\w+)*@\w+([-.]\w+)*\.\w+([-.]\w+)*
hotmail.com, live.com and probably outlook.com addresses accept email addresses like local__part@domain.com and local--part@domain.com (1) that won't pass the above Regular Expression ValidationExpression (1), go figure :!: :?:

There simply is NO Regular Expression Expression that 100% covers everything, period!

My ASP.NET RegularExpressionValidator ValidationExpression is now:

Code: Select all

(?:[a-zA-Z0-9!#$%&'*+\/\=?^_`{|}~-]+(?:\.[a-zA-Z0-9!#$%&'*+\/\=?^_`{|}~-]+)*@(?:[a-zA-Z0-9](?:[a-zA-Z0-9-]*[a-zA-Z0-9])?\.)+[a-zA-Z0-9](?:[a-zA-Z0-9-]*[a-zA-Z0-9])?)
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
katip
Senior user
Senior user
Posts: 966
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Address validation

Post by katip » 2021-09-10 11:44

RvdH wrote:
2021-09-09 14:08
There simply is NO Regular Expression Expression that 100% covers everything, period!
+1. this is clear and common consensus.

since i found it (RFC official??) i'm using in my scripts that monster:

Code: Select all

oRegEx.Pattern = "^(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|" & Chr(34) & "(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*" & Chr(34) & ")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])$"
i think it works as it caught the other day <", user"@external.com.tr"> and a simple one <user@external>
it matches your __ and -- examples too.
Katip
--
HMS 5.7.0, MariaDB 10.4.10, SA 3.4.4, ClamAV 0.103.2

User avatar
RvdH
Senior user
Senior user
Posts: 1674
Joined: 2008-06-27 14:42
Location: Netherlands

Re: Address validation

Post by RvdH » 2021-09-10 11:55

only <", user"@external.com.tr"> isn't a email address, is it? :lol:
That is FROM header
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
RvdH
Senior user
Senior user
Posts: 1674
Joined: 2008-06-27 14:42
Location: Netherlands

Re: Address validation

Post by RvdH » 2021-09-10 13:25

Code: Select all

(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])
https://regexr.com/ says that expression (PCRE) is not OK (and it only accepts lowercase names and addresses? Duh!)

Escaped + ignore case (or use .IgnoreCase option)

Code: Select all

(?:[a-zA-Z0-9!\#\$%&'*+\/\=\?\^_`{|}~-]+(?:\.[a-zA-Z0-9!\#\$%&'*+\/\=\?\^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-zA-Z0-9](?:[a-zA-Z0-9-]*[a-zA-Z0-9])?\.)+[a-zA-Z0-9](?:[a-zA-Z0-9-]*[a-zA-Z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-zA-Z0-9-]*[a-zA-Z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
RvdH
Senior user
Senior user
Posts: 1674
Joined: 2008-06-27 14:42
Location: Netherlands

Re: Address validation

Post by RvdH » 2021-09-10 15:08

Code: Select all

(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])
Freaking impossible to use this in C# is seems (or at least i can get it to work)
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
SorenR
Senior user
Senior user
Posts: 4894
Joined: 2006-08-21 15:38
Location: Denmark

Re: Address validation

Post by SorenR » 2021-09-10 15:22

RvdH wrote:
2021-09-10 15:08

Code: Select all

(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])
Freaking impossible to use this in C# is seems (or at least i can get it to work)
There are two quotes in that string you need to double-quote if using quotes around strings...
SørenR.

Engineer (noun)
- I'm Not Arguing, I'm Just Explaining Why I'm Right

User avatar
RvdH
Senior user
Senior user
Posts: 1674
Joined: 2008-06-27 14:42
Location: Netherlands

Re: Address validation

Post by RvdH » 2021-09-10 15:29

:oops:
Tried escaping, except the double quotes

Seems to work now
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
katip
Senior user
Senior user
Posts: 966
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Address validation

Post by katip » 2021-09-10 18:58

RvdH wrote:
2021-09-10 11:55
only <", user"@external.com.tr"> isn't a email address, is it? :lol:
That is FROM header
no. very likely it was a mail with some malformed multi To or CC. our user just did a "Reply All".
Katip
--
HMS 5.7.0, MariaDB 10.4.10, SA 3.4.4, ClamAV 0.103.2

User avatar
katip
Senior user
Senior user
Posts: 966
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Address validation

Post by katip » 2021-09-16 15:44

RvdH wrote:
2021-09-10 15:29
:oops:
Tried escaping, except the double quotes

Seems to work now

Code: Select all

"Mert (Izmir) mgelen"@somedomain.com
failed with that (RFC conform??) monster regex in EventHandlers.vbs.
BUT HMS accepted it. and recipient too...

Code: Select all

(...)
"SMTPD"	4092	39905	"2021-09-16 15:34:40.837"	"192.168.0.67"	"RECEIVED: RCPT TO:<"Mert (Izmir) mgelen"@somedomain.com>"
"SMTPD"	4092	39905	"2021-09-16 15:34:40.837"	"192.168.0.67"	"SENT: 250 OK"
(...)
"SMTPC"	2796	39927	"2021-09-16 15:34:50.899"	"xx.xx.xx.xx"	"SENT: RCPT TO:<"Mert (Izmir) mgelen"@somedomain.com>"
"SMTPC"	1560	39927	"2021-09-16 15:34:51.071"	"xx.xx.xx.xx"	"RECEIVED: 250 recipient <Mert (Izmir) mgelen@somedomain.com> ok"
(...)
"SMTPC"	4732	39927	"2021-09-16 15:34:56.102"	"xx.xx.xx.xx"	"RECEIVED: 250 ok:  Message 36182312 accepted"
this is really a slippery subject :roll:
Katip
--
HMS 5.7.0, MariaDB 10.4.10, SA 3.4.4, ClamAV 0.103.2

User avatar
RvdH
Senior user
Senior user
Posts: 1674
Joined: 2008-06-27 14:42
Location: Netherlands

Re: Address validation

Post by RvdH » 2021-09-17 11:39

Off course it does, i don't understand what you going with this as i do not know where you think you can tackle this within EventHandlers.vbs :?:

These are the regex's internally (as already mentioned by Soren earlier)
https://github.com/hmailserver/hmailser ... 8b1af86R59
https://github.com/hmailserver/hmailser ... 50be4f6R58

Simply give up on it :mrgreen:
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
katip
Senior user
Senior user
Posts: 966
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Address validation

Post by katip » 2021-09-17 13:07

never mind.
i changed the one in vbs to HMS internal one.
thanks.
Katip
--
HMS 5.7.0, MariaDB 10.4.10, SA 3.4.4, ClamAV 0.103.2

Post Reply