MESSAGE object in Com API not conforming to RFC on

Use this forum if you want to discuss a problem or ask a question related to a hMailServer beta release.
Post Reply
Turtleneck
New user
New user
Posts: 12
Joined: 2014-08-05 16:49

MESSAGE object in Com API not conforming to RFC on

Post by Turtleneck » 2015-01-05 18:13

I am using hmailserver 5.4.2.

I have done some scripting and discovered an indescrepancy/failure to fully adhere to RFC rule in the MESSAGE object.

RFC822 says ofr the FROM: field:
This field contains the identity of the person(s) who wished
this message to be sent. The message-creation process should
default this field to be a single, authenticated machine
address
, indicating the AGENT (person, system or process)
entering the message. If this is not done, the "Sender" field
MUST be present. If the "From" field IS defaulted this way,
the "Sender" field is optional and is redundant with the
"From" field. In all cases, addresses in the "From" field
must be machine-usable (addr-specs)
and may not contain named
lists (groups).
However, using scripting, it is possible to actually create an email that does NOT have a proper formatted email address in the FROM field, and can simply contain a narrative name.

eg,
FROM: Joe Bloggs (not allowed)

instead of

FROM: Joe Blogss <jbloggs@hishouse.com> (correct format)

In normal circumstances compiling the script should error if this was attempted. (By contrast the Microsoft CDO does exactly that if the format is incorrect as it will not allow this field to be without a full address).

If anyone wants to log it with the developers then maybe thats a good idea (I am not able to from my location).

Script to reproduce if required:

Code: Select all

HMSADMIN = "Administrator"
HMSPWD = "password"

Set oApp = CreateObject("hMailServer.Application")
Call oApp.Authenticate(HMSADMIN, HMSPWD)

BodyMsg = "Some body text etc"

Set Message = CreateObject("hMailServer.Message")
Message.FromAddress = "sender@domain.com"
Message.From = "Housekeeping process"
Message.AddRecipient "Joe Bloggs", "recipient@somedomain.com"
Message.Subject = "Housekeeping info"
Message.HTMLBody = BodyMsg
Message.Save

percepts
Senior user
Senior user
Posts: 5282
Joined: 2009-10-20 16:33
Location: Sceptred Isle

Re: MESSAGE object in Com API not conforming to RFC on

Post by percepts » 2015-01-05 18:34

since you are the programmer of your own script it is your responsibility to do your own validation and not expect the script syntax checker to do it for you. It is only a syntax checker and not a logic or RFC validator and the COM API gives you the freedom to do pretty much whatever you want and leaves it to you to get it right.

Turtleneck
New user
New user
Posts: 12
Joined: 2014-08-05 16:49

Re: MESSAGE object in Com API not conforming to RFC on

Post by Turtleneck » 2015-01-05 22:17

'Syntax checkers' exist to ensure that that the string of characters creating the command and paramters suffice the intended function. RFC rules provide the basis of the function; without the RFC rules, the theoretic function wouldnt exist and wouldnt be written or used by email servers and clients - its purpose is to ensure everything works with the same functions and to the same parameters and thereby provide the definition of a function's syntax. Any syntax checker of a function provided by the Com API (in this case the function to create a message and/or modify the relevant fields required by that function) should ensure this syntax of each provided field coded for by the programmer is correct (leaving the programmer with the responsibility to ensure that obligatory fields are included).

So I disagree with you. Programmers ensure the components ('the fields') required by a function ('create/modify a message') are provided, the function syntax checker provided by the Com API should ensure the syntax of those components is correctly coded (as expected). In much the same way it would spit out an Alpha character value where an integer is expected.

Are you suggesting that this Com API is a lot looser in it's functionality than, say, the MS CDO scripting (which to me can do the same thing whilst ensuring syntax of the same comparable command is syntaxically correct)? In other words you are saying this Com API doesnt care about nothing except controlling the Hmailserver application itself? And yet, if an email is generated within hmailserver (such as an auto-reply or out of office), it will have clearly been coded by the developer to ensure that such email is conforming to whatever RFC rules say it must do (all the necessary obligatory fields in place, correctly formatted). And yet you are saying its not possible for the same application to have those same such enforcements through its API (unlike Microsoft CDO that does). Otherwise, one would be allowed to code any value at all to any header field (invented or otherwise), whilst ignoring obligatory fields for functional success, or even create their own random header with any value they like, and then submit it to the Com API and expect it to 'work'.

Can I have the developers word on this please? I would like to know his (Martin's?) thoughts or explanation.

Thanks

EDIT:

p.s Martin, if you reply and say something like "youre right but no, I havent included message field-level syntax checking and have no intention of doing - the software is going to remain as is" then so be it. I will of course accept the software for however it works but I just want to be sure that this is your intention and that my original findings is not an error that you wish to be aware of.

User avatar
mattg
Moderator
Moderator
Posts: 20222
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: MESSAGE object in Com API not conforming to RFC on

Post by mattg » 2015-01-06 00:27

I'm wondering if the RFCs are talking about SMTP envelope FROM, or Message Header FROM
What you write with the COM API is the Message Header FROM.
What I expect the RFCs to mean is the SMTP envelope FROM.

Does hMailserver get this right in the background?
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

Turtleneck
New user
New user
Posts: 12
Joined: 2014-08-05 16:49

Re: MESSAGE object in Com API not conforming to RFC on

Post by Turtleneck » 2015-01-06 00:43

It is talking about the message header FROM field (as the same RFC article talks about all the fields making up the composition including non-smtp relevant fields such COMMENT, SUBJECT, X- type and user-defined fields). See http://www.w3.org/Protocols/rfc822/#z26

(By allowing this incorrect syntax to be formed and applied and sent, a message can be received at destination with a FROM field present and yet not with an email address contained in it.)

User avatar
mattg
Moderator
Moderator
Posts: 20222
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: MESSAGE object in Com API not conforming to RFC on

Post by mattg » 2015-01-06 01:09

Yep, just tested and confirmed that you are correct. (and has always been this way)

I couldn't even find a SPAM mail that does this.
My guess is that a malformed message like this would be rejected by most mail servers as non-compliant, but I haven't tested


Add it to the issue tracker please >> https://github.com/hmailserver/hmailserver/issues
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

Turtleneck
New user
New user
Posts: 12
Joined: 2014-08-05 16:49

Re: MESSAGE object in Com API not conforming to RFC on

Post by Turtleneck » 2015-01-06 10:38

Actually I have done a test TO a yahoo address and it doesnt get rejected (and Yahoo are normally pretty hot on conformity as I understand)

Received email source:
X-Apparently-To: user@yahoo.com; Tue, 06 Jan 2015 08:29:55 +0000
Return-Path: <sender@domain.com>
Received-SPF: pass (domain of domain.com designates xx.xxx.xxx.xx as permitted sender)
Authentication-Results: mta1086.mail.bf1.yahoo.com from=; domainkeys=neutral (no sig); from=; dkim=neutral (no sig)
Received: from 127.0.0.1 (HELO company.mydomain.com) (xx.xxx.xxx.xx)
by mta1086.mail.bf1.yahoo.com with SMTP; Tue, 06 Jan 2015 08:29:54 +0000
Content-Type: text/html; charset="utf-8"
MIME-Version: 1.0
From: Housekeeping process
To: "System Administrator" <user@yahoo.com>
Subject: House keeping info
Content-Transfer-Encoding: quoted-printable
Date: Tue, 6 Jan 2015 08:29:51 +0000
Content-Length: 18

Random body text
I have seen that some email clients have issues with such FROM addresses though (not displaying properly) if they do not have the <address> notation included (acceptable if the RFC says it should be expecting it).

The HMS log file shows thast it passes the value found in the oMessage.FromAddress as the sender and that ends up being the RETURN-PATH field (understandable as the RFC says this would happen, in the absence of a valid FROM: field).

Still would like to hear Martins thoughts on this issue.

percepts
Senior user
Senior user
Posts: 5282
Joined: 2009-10-20 16:33
Location: Sceptred Isle

Re: MESSAGE object in Com API not conforming to RFC on

Post by percepts » 2015-01-06 11:06

see "Allow Empty Sender Address" in Settings / Protocols / SMTP / RFC Compliance

https://www.hmailserver.com/documentati ... otocolsmtp

Turtleneck
New user
New user
Posts: 12
Joined: 2014-08-05 16:49

Re: MESSAGE object in Com API not conforming to RFC on

Post by Turtleneck » 2015-01-06 11:11

Turtleneck wrote: If anyone wants to log it with the developers then maybe thats a good idea (I am not able to from my location).
Im afraid I cannot.

Turtleneck
New user
New user
Posts: 12
Joined: 2014-08-05 16:49

Re: MESSAGE object in Com API not conforming to RFC on

Post by Turtleneck » 2015-01-06 11:20

percepts wrote:see "Allow Empty Sender Address" in Settings / Protocols / SMTP / RFC Compliance

https://www.hmailserver.com/documentati ... otocolsmtp
That just tells us what we already know. Empty sender addresses are allowed as RFC states that either FROM or SENDER fields should be populated with a valid address - therefore an NON-EXISTANT FROM field is allowed (ie, nothing at all) not a FROM field partly populated without a valid address. (In the event of an EMPTY sender address, the RETURN-PATH field is populated and used for the delivery transaction). And that setting you point to is simply giving the admin the choice whether to over-ride acceptance of such emails (in case of it being a common Spam symptom)

By the way, that also just shows that the Hmailserver Com API is incomplete as it doesnt allow the programmer the ability to conform the RFC. RFC says that you MUST enter a valid address in FROM or SENDER fields, and yet the Message object doesnt allow the coding of SENDER field (there is no 'message.sender' component) if he has chosen to not code or populate a proper FROM address (unless I suppose you use a 'user-defined' field "Message-header" definition. I havent tried this but but should this be the way?)

EDIT: Just tried with SENDER field (and no FROM field) and Yahoo fails to recognise this field and sees it simply as a user-defined field. As does thunderbird. Perhaps RFC does not state that a SENDER field (unlike a FROM field) is to be used for replies and is purely narative.
Last edited by Turtleneck on 2015-01-06 11:49, edited 1 time in total.

percepts
Senior user
Senior user
Posts: 5282
Joined: 2009-10-20 16:33
Location: Sceptred Isle

Re: MESSAGE object in Com API not conforming to RFC on

Post by percepts » 2015-01-06 11:45

Do you realise that RFC 822 which you linked to is obsolete? I guess not. The replacement may contain similar but I'll leave you to research and quote what is "current".

Turtleneck
New user
New user
Posts: 12
Joined: 2014-08-05 16:49

Re: MESSAGE object in Com API not conforming to RFC on

Post by Turtleneck » 2015-01-06 11:53

percepts wrote:Do you realise that RFC 822 which you linked to is obsolete? I guess not. The replacement may contain similar but I'll leave you to research and quote what is "current".
Yeah, thats the kind of help we need on forums. Helpful comments only please!

(For OTHER readers: both 2822 and 5322 state the same as I originally quoted)
from = "From:" mailbox-list CRLF

The "From:" field specifies the author(s) of the message,
that is, the mailbox(es) of the person(s) or system(s) responsible
for the writing of the message

percepts
Senior user
Senior user
Posts: 5282
Joined: 2009-10-20 16:33
Location: Sceptred Isle

Re: MESSAGE object in Com API not conforming to RFC on

Post by percepts » 2015-01-06 12:24

well standards say it should contain a valid mailbox. So please explain how you are going to validate that it is a valid mailbox. Virtually every mailserver on the planet puts all sorts of garbage in the From header. Typically <no-reply@somedomain.com> that may be correct format but its not a genuine mailbox so whats the point.

I think the problem for you is that are trying to feed your script with an un-validated list of email addresses and you are expecting hmailserver to do the validation for you. Well what are you going to do with the mail if isn't in correct format. And do you know what the correct format of a domain name is anyway or are you only interested in your specific case.

I would suggest you validate your email addresses at the point of input if you want to do a proper job and then you won't have a problem.
Last edited by percepts on 2015-01-06 12:37, edited 1 time in total.

Turtleneck
New user
New user
Posts: 12
Joined: 2014-08-05 16:49

Re: MESSAGE object in Com API not conforming to RFC on

Post by Turtleneck » 2015-01-06 12:36

My point was not about validating a mailbox - where did I ever say that was the problem??

I said that the Com API allows a VALUE to be entered that doesnt conform to the correct SYNTAX: The syntax being it must be the form of a mailbox. ie, a MAILBOX field must contain "<something@subdomain.domain>" - a MAILBOX form is clearly defined in RFC and can be coded for in the API to check that the value entered conforms to the format of a mailbox. A syntax that the same way says a Numeric field must contain numeric characters (and will error at time of compilation if not). It doesnt check that the VALUE of the numbers make any sense or are valid in context of the whole program or function.

Ill leave it for you to go back and reread and reply to what is actually said ( - or not bother. Your choice. Youre not helping either way.)

percepts
Senior user
Senior user
Posts: 5282
Joined: 2009-10-20 16:33
Location: Sceptred Isle

Re: MESSAGE object in Com API not conforming to RFC on

Post by percepts » 2015-01-06 14:51

No I'm just not giving you the answer you want. What is not helpful is asking for a data validation service to be built into a mailserver. Hmail just takes whatever it is sent and sends it to the recipient/s and stores it.

If you are writing scripts that create bad data it is your responsibility to get it right. If you are processing some input file then you do the validation on your input. Hmail will discover its bad when you try and send it. Thats where its validation takes place, it'll either reject it because it doesn't know where to deliver it or the recipient server will reject it because the address doesn't exist at their server both of which not only validate format but also whether its a real mailbox. So there is precisely zero benefit to validating format in com api except to you who can't be bothered to run any validation of your data and wants it done at the point you want it done for your own benefit.

See following for some clues on what you are asking and what you need to do.

http://en.wikipedia.org/wiki/Email_address

p.s. just about everyone knows the best way to validate an email address is to send to it and ask for a reply or weblink to be clicked which is proof of existance of mailbox. Anything else is a stab in the dark.

Oh, and don't forget about legacy formats which may not comply with current RFCs thereby rendering RFCs not so clever if you want to allow mail from non RFC compliant sources which may be perfectly valid.

Turtleneck
New user
New user
Posts: 12
Joined: 2014-08-05 16:49

Re: MESSAGE object in Com API not conforming to RFC on

Post by Turtleneck » 2015-01-06 16:22

Look, AGAIN I will say the following RELEVANT statements to this original thread of mine

1, I am asking for a SYNTAX check, not a VALIDATION service (do you see?, do you want me to say it for a FOURTH time???)
percepts wrote:What is not helpful is asking for a data validation service to be built into a mailserver
2, a SYNTAX checker is already built in to hmailserver otherwise you wouldnt get a TYPE MISMATCH error if you try to set a string of "ABC" into a field that is a Boolean! And its possible; as microsoft CDO has proven. They expect an EMAIL address form in this field and it gives a compilation error if it doesnt see one. If the Hmailserver API had defined, for example, Message.EncodeFields as a STRING and not a boolean, then it wouldnt complain about entering a string into this field. But it does, because it expects a BOOLEAN value. And a FROM: field should expect an EMAIL formatted string.

3, I am not asking for a validation of an entered email address.

4, I am not asking hmailserver to chck that the value entered into an email address is a valid email address

5, Data VALIDATION is not being asked for here, but purely a SYNTAX check of data entered in a clearly defined field format

GOT IT??

6, My original post was asking for comment from the developer
Turtleneck wrote:Can I have the developers word on this please? I would like to know his (Martin's?) thoughts or explanation.

Thanks

EDIT:

p.s Martin, if you reply and say something like "youre right but no, I havent included message field-level syntax checking and have no intention of doing - the software is going to remain as is" then so be it. I will of course accept the software for however it works but I just want to be sure that this is your intention and that my original findings is not an error that you wish to be aware of.
7, From what I can tell, you are not the developer

8, Just because the software does what it does, it doesnt make it right or BY DESIGN. Otherwise, 'development' of software would never happen because people like you believe that if things do what they do then that is the way it is to be.

9, PERCEPTS, STOP ANSWERING ON THIS TOPIC as you clearly dont have the ability to respond to what is written and purely want to answer to the questions you think should have been asked and consequently are loosing direcion on my otherwise well-intended thread. Leave it for those that have the authority to comment on the intended design of hmailserver!

p.s Did I mention I am not asking for data VALIDATION by hmailserver Com API, and simply for the ComAPI to do a field definition format check??!

percepts
Senior user
Senior user
Posts: 5282
Joined: 2009-10-20 16:33
Location: Sceptred Isle

Re: MESSAGE object in Com API not conforming to RFC on

Post by percepts » 2015-01-06 16:55

Feel free to download the source code from Github and make it do what you want yourself then.

Turtleneck
New user
New user
Posts: 12
Joined: 2014-08-05 16:49

Re: MESSAGE object in Com API not conforming to RFC on

Post by Turtleneck » 2015-01-06 17:55

Yeah, helpful as ever!
percepts wrote: "*I* am the only one that matters around here, *I* dont have the answer, and other responders opinion (Mattg) do not matter, so look it up yourself".
(It seems you have a lot of time to waste typing pointless and relevant replies.)

So.....

FAO: Martin

If you could review my original post please. If you reply and say something like "youre right but no, I havent included message field-level syntax checking and have no intention of doing - the software is going to remain as is" then so be it. I will of course accept the software for however it works but I just want to be sure that this is your intention and that my original findings is not an error that you wish to be aware of.

Thanks.

User avatar
mattg
Moderator
Moderator
Posts: 20222
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: MESSAGE object in Com API not conforming to RFC on

Post by mattg » 2015-01-07 00:48

Turtleneck wrote:FAO: Martin

If you could review my original post please.
mattg wrote:Add it to the issue tracker please >> https://github.com/hmailserver/hmailserver/issues
Turtleneck wrote:Im afraid I cannot.
I have done this.

I agree with Percepts though, I don't believe this to be a real issue. Sure, I see potential for misuse, but then there is lots of that in this world.
I use it like this

Code: Select all

	Call oApp.Authenticate("Administrator", g_sAdminPassword)

	sBackupLog = ReadFileUnicode(oApp.Settings.Backup.LogFile)
	DeleteFile(oApp.Settings.Backup.LogFile)

	Set oMessage = CreateObject("hMailServer.Message")
	oMessage.From = g_sBNFrom & " <" & g_sBNFromAddress &  ">"
	oMessage.FromAddress = g_sBNFromAddress
	oMessage.Subject = MessageSubject
	oMessage.AddRecipient MessageRecipientName, MessageRecipientAddress
	oMessage.Body = "The backup completed succesfully." & vbNewLine & vbNewLine & sBackupLog
	oMessage.Save
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

Turtleneck
New user
New user
Posts: 12
Joined: 2014-08-05 16:49

Re: MESSAGE object in Com API not conforming to RFC on

Post by Turtleneck » 2015-01-07 12:30

Thank you Mattg, most helpful. I guess we will see if anything comes of this. Cheers.

(I see how you construct the FROM field by manually constructing the correct format with the angle brackets and inserting the email address, all by script. Of course, this is the natural way and what people should be doing to conform to the RFC rules. But this was never the point of the discussion (despite Percepts' constant endless repitition of saying it is). It was always about whether the COM allows incorrectly formatted strings being entered into this field.)

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: MESSAGE object in Com API not conforming to RFC on

Post by martin » 2015-01-08 22:06

Here's what I wrote on Github.
The From property of the message gets and sets the From header in the email message. While the From header must have a specific format according to the RFC, there's nothing preventing anyone from sending email messages with incorrectly formatted headers. This happens all the time.

So if a message is received with an incorrectly formatted From header, there are two options:

- The From property in the API could return the incorrectly formatted data
- The From property in the API could return an empty string, or throwing an exception that the data is invalid

I've chosen the first alternative here, because I believe it makes the API easiest to work with.

It would be weird if the From property could return specific data, but you would not be able to set the same yourself. As a consequence, the From property can return incorrectly formatted data, and you can give it incorrectly formatted data.

So I'll close this without action.
Please let me know if you disagree with this. I agree that it's not an optimal situation, but I think it's "good enough" and changing this behavior would potentially break backwards compatibility with old scripts. When I make the next version of the API (which are currently only ideas in my head), the validation will be more strict but optional.
Martin Knafve
martin@hmailserver.com
https://twitter.com/knafve

Turtleneck
New user
New user
Posts: 12
Joined: 2014-08-05 16:49

Re: MESSAGE object in Com API not conforming to RFC on

Post by Turtleneck » 2015-01-08 23:34

Thank you Martin.

Your explanation was a pleasure to read - it gave reasoning and thought behind the reasoning (instead of "it just is and thats it!"). In someways I agree with you (particularly because of the backwards compatibility issue) but yes, of course, ideally I think that the API should recognise an email address format or reject if it not found yet not blank (microsoft CDO does that if email address format is not found). But yes, I understand your situation.

I accept this, no problems from me. Thanks for considering and replying.

Post Reply