STARTTLS feedback?

Use this forum if you want to discuss a problem or ask a question related to a hMailServer beta release.
User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

STARTTLS feedback?

Post by martin » 2014-07-28 21:59

Here's the status and some request for info:

A new setting called "Connection security", with the possible values "None", "SSL/TLS" and "STARTTLS" has been added:
  • When configuring a TCP/IP port for IMAP, SMTP and POP3 you can select connection security.
  • When setting up an external account for POP3 fetching, you can select connection security.
  • When specifying a SMTP relayer, and when setting up a SMTP route, you can select connection security.
Here's where I would appreciate some feedback:

Right now, if I have not specified a SMTP relayer and I have no routes setup, and I deliver a message to say Gmail, then STARTTLS will not be used. What I'm thinking is that it would make sense to add a new setting, maybe to "SMTP -> Advanced" with the name "Attempt to use STARTTLS during delivery". This would then affect all outgoing SMTP connections so that if I try to send an email to @gmail.com, @hotmail.com @cnn.com or whatever, then hMailServer will attempt to use STARTTLS if the server supports this. The only reason where I think it would make sense to turn this off is if you are delivering to a SMTP server which claims to support STARTTLS but where it does not work for some reason - then you could disable this explicitly.

percepts
Senior user
Senior user
Posts: 5282
Joined: 2009-10-20 16:33
Location: Sceptred Isle

Re: STARTTLS feedback?

Post by percepts » 2014-07-28 22:09

In Eudora there is a drop down under "Secure sockets" which has options:

Never
If available, STARTTLS
Required Alternate Port (SSL)
Required STARTTLS

I guess you mean something along these lines.

p.s. Yes I know Eudora is very old but it just works and I like the interface (last version before it was ruined in open source).

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: STARTTLS feedback?

Post by mattg » 2014-07-29 02:28

martin wrote:Right now, if I have not specified a SMTP relayer and I have no routes setup, and I deliver a message to say Gmail, then STARTTLS will not be used. What I'm thinking is that it would make sense to add a new setting, maybe to "SMTP -> Advanced" with the name "Attempt to use STARTTLS during delivery". This would then affect all outgoing SMTP connections so that if I try to send an email to @gmail.com, @hotmail.com @cnn.com or whatever, then hMailServer will attempt to use STARTTLS if the server supports this. The only reason where I think it would make sense to turn this off is if you are delivering to a SMTP server which claims to support STARTTLS but where it does not work for some reason - then you could disable this explicitly.
I like that concept
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: STARTTLS feedback?

Post by martin » 2014-07-29 08:54

In Eudora there is a drop down under "Secure sockets" which has options:

Never
If available, STARTTLS
Required Alternate Port (SSL)
Required STARTTLS

I guess you mean something along these lines.
If you are connecting to a specific server, which you are when you are setting up a route or SMTP relay server or when fetching from an external POP3 account, then this makes sense. But if you're not connecting to a specific SMTP server, but rather any SMTP server which the recipient happens to be hosted on, then requireing STARTTLS or required SSL doesn't make much sense. Because then your server could only deliver to those domains where they happen to have implemented these things.

So the above options would be available when setting up a route or specfiying a SMTP relay server, but then there would be another option ("Use STARTTLS if available") when doing normal SMTP delivery outside of routes/relay server scopes... if that makes sense. I've noticed that this is something Gmail does - when I send a mail from gmail to my hmailserver account, then Gmail will detect that mail.hmailserver.com supports STARTTLS and use that. But it won't attempt to find a alternate SMTP port which happens to use SSL.

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: STARTTLS feedback?

Post by mattg » 2014-07-29 09:47

While you are playing with this stuff, it would be nice to be able to block traffic on port 25 that wasn't external to local, unless it was StartTLS
In other words a way to force all local senders to use StartTLS (or SSL)

One idea that I have seen on this is adding Security level options to IP ranges.

I assume also, that there will be some way to force verification for incoming connections made via StartTLS

Matt
(PS good to have you back)
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

percepts
Senior user
Senior user
Posts: 5282
Joined: 2009-10-20 16:33
Location: Sceptred Isle

Re: STARTTLS feedback?

Post by percepts » 2014-07-29 14:08

So the above options would be available when setting up a route or specfiying a SMTP relay server, but then there would be another option ("Use STARTTLS if available") when doing normal SMTP delivery outside of routes/relay server scopes... if that makes sense. I've noticed that this is something Gmail does - when I send a mail from gmail to my hmailserver account, then Gmail will detect that mail.hmailserver.com supports STARTTLS and use that. But it won't attempt to find a alternate SMTP port which happens to use SSL.
Is it really necessary to provide a selectable option for that? When its available just use it. I guess thats the question, should hmail make the decision itself or should the hmail admin get a choice. And I guess that depends on where RFC standards are heading. If its becoming the norm to use STARTTLS then I'd say just do it without providing admin options but thats just me as I don't know any reason not to. Others may know different and have reasons for not wanting hmail to auto use STARTTLS if avaialble.

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: STARTTLS feedback?

Post by martin » 2014-07-29 14:55

mattg wrote:While you are playing with this stuff, it would be nice to be able to block traffic on port 25 that wasn't external to local, unless it was StartTLS
In other words a way to force all local senders to use StartTLS (or SSL)
mattg, that makes sense. Right now if you select STARTTLS on a TCP/IP port then that's optional. A client may connect to the port and do what it want without issuing the STARTTLS command. Splitting the STARTTLS option on TCP/IP port level into STARTTLS (Optional) and STARTTLS (Required) would be one way to go forward, but it feels like that would become a bit hairy. Because your suggestion would then imply that there will have to be a STARTTLS (Required for deliveries from local accounts) as well. Then tomorrow someone want to make it mandatory per domain.

I think it would make sense to add another setting right now to TCP/IP ports - Require authentication over SSL/TLS to force local users into using either STARTTLS or pure SSL/TLS before authenticating. Because I assume that you don't want to force it just for SMTP but also for IMAP/POP3. I guess this setting could just as well be placed on IP range level, because then you could exclude localhost from having to use SSL (or a specific web server running some script which does not support STARTTLS - or similar).
percepts wrote:Is it really necessary to provide a selectable option for that?
Regarding the setting for using STARTTLS when delivering to external servers, if it's avaialble - I'm a bit worried about backwards compatibility. I'm not sure how many SMTP servers out there that supports STARTTLS but does it in a broken way or a way which is not hMailServer compatbile (directly). I don't want users to upgrade to hMailServer 5.5 (which will have this feature) and then end up in a situation where they cant send mail anymore.

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: STARTTLS feedback?

Post by Bill48105 » 2014-07-29 16:14

Some of this was discussed in other posts but would have to find them. There almost needs to be rules available for each route and general delivery so it can be flexible enough for all the scenarios. In some cases admin may want to stop all mail unless it's secure in other cases admin may want to allow only to some domains, or if no attachments. Or admin may want to TRY secure but if it fails let it go. Etc
Bill
Ps. martin you get my skype or PM?
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

prisma
Senior user
Senior user
Posts: 325
Joined: 2010-07-09 13:16

Re: STARTTLS feedback?

Post by prisma » 2014-07-29 17:52

Bill48105 wrote:Some of this was discussed in other posts but would have to find them. There almost needs to be rules available for each route and general delivery so it can be flexible enough for all the scenarios. In some cases admin may want to stop all mail unless it's secure in other cases admin may want to allow only to some domains, or if no attachments. Or admin may want to TRY secure but if it fails let it go. Etc
Bill
Ps. martin you get my skype or PM?
+1 Enforcement or no enforcement of STARTLS has to be configurable:
  • * For every single outgoing SMTP route as well as
    * For the outgoing relayer as well as
    * For MX resolved delivery in general as well as
    * For incoming SMTP in general also.
Why that? Martin, you're partially right. There can be a sensefull default behaviour for hmailserver. But you have to image there can be many many various scenarios with higher and more restrictive security requirements. E.g. for exclusively internal (intranet) and inter enterprise communication.

So, let's speak about certificates:
  • * How to behave on certificate errors?
    * How to handle selfsigned certificates?
    * How to pin CA or host certificates, e.g. by fingerprint?
    * How to build a change of trust?
    * How to define allowed cipher suites (Bill added a simple INI setting)?
That's the much more difficult and delicate part than making encryption enforcement just configurable for the first 4 points.

For these further considerations a good point to start reading and for getting ideas is
http://www.ietf.org/proceedings/87/slid ... dane-2.pdf

But that would be the highest and final level to implement... Step-by-step.
I'm very happy that martin has started development of STARTTLS. And THX a lot to Bill for his work also!

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: STARTTLS feedback?

Post by martin » 2014-07-29 21:18

Bill48105, got the PM. Rarely on Skype now.

prisma, sure, there's a ton of different scenarios but as you say - step by step. There's no point in trying to cover all scenario in first version. Just thinking about it too much will make us end up in some kind of analysis paralysis and will just slow down things. :)

Here's the plan for version 1 of STARTTLS:
  • On TCP/IP ports you will be able to choose between:
    • Connection security: None
    • Connection security: STARTTLS (Optional)
    • Connection security: STARTTLS (Required)
    • Connection security: SSL
  • The same will be selectable for SMTP relayer, routes and external accounts. if STARTTLS (Optional) is selected, hMailServer will attempt to use STARTTLS but if the remote server does not announce support it will just unencrypted.
  • In IP ranges, you'll be able to select "Require SSL/TLS for authentication" - If you select this, local users won't be able to authenticate without using SSL or TLS. This is useful if you want to choose STARTTLS (Optional) for port 25, but you want to force local users to use it.
  • In SMTP delivery settings, it will be possible to select that hMailServer should attempt to use STARTTLS if available when delivering after a MX lookup
I think this should be a good starting point, and I'm certain it will cover at least 1 scenario (mine).

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: STARTTLS feedback?

Post by Bill48105 » 2014-07-29 22:55

martin wrote:Bill48105, got the PM. Rarely on Skype now.
Explains your lack of response on skype. I've already apologized for being extremely busy in my personal life & work lately but you should understand that, being MIA from hmail forum/irc/coding for months at a time yourself for the past 4 years. As long as we're available when convenient for you that's what matters is that how it works? Just remember many of us held down the fort when you were busy.
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: STARTTLS feedback?

Post by martin » 2014-07-30 11:19

Bill, I'm sorry if I've made you upset, that has not been my intention.

The thing is, there is nothing you need to apologize for and I'm not sure why you feel you do. I have not criticized you in any way. All I asked was if you could share the source code you've written for STARTTLS so that I could integrate it into the official release. Since I wanted to get started with it and some weeks (or so) passed, I ended up implementing it myself. I don't see this as an issue or a big deal in any way.

No person whatsoever have any obligations against me or anyone else when it comes to this site. It's all fun and nice if people want to talk in the forum and contribute in different ways - and of course it's appreciated - but it's completely optional and I expect people to come and go, join and leave as they feel like. If you would have left yesterday or tomorrow and never coming back, I would not have thought a bad thing about it. I appreciate all you do while you're here, but I don't assume anything about anyone. There's things going on outside of this forum which are much more important to everyone. The only thing I expect and hope is everyone here to behave in the way which most convenient and fun to them.

prisma
Senior user
Senior user
Posts: 325
Joined: 2010-07-09 13:16

Re: STARTTLS feedback?

Post by prisma » 2014-07-30 12:50

martin wrote:Here's the plan for version 1 of STARTTLS
Sounds like a wonderful V1 and like covering 3 of my first 4 points +1 extra point (IP ranges). Great! Only encryption enforcement after MX lookup is skipped. And, I must confess, this would be only necessary for intranet purposes or scenarios where undeliverability for unsecured communication is explicitly wanted. But, hey, this is really very special...

Question:
If I want to force encryption for delivery to special domains, could I use a route, enforce STARTTLS but leave target SMTP host empty? Is then MX resolution active? Or do I have to point explicitly to a target server? Documentation isn't 100% clear there...
If not, this would be a useful behaviour and would give the possibility to explicitly define special routes with higher security, but without pinning the servers. This scenario is much more popular.

The cipher suite part is already done by Bill, possibly he shares the code with you?

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: STARTTLS feedback?

Post by martin » 2014-07-30 13:04

If I want to force encryption for delivery to special domains, could I use a route, enforce STARTTLS but leave target SMTP host empty?
No, you cannot set up a route with empty target host name. So that would not work.

As for certificates, I'm looking into letting hMailServer rely on the Windows certificate store. So hMailServer will trust the certificates which Windows trusts.

prisma
Senior user
Senior user
Posts: 325
Joined: 2010-07-09 13:16

Re: STARTTLS feedback?

Post by prisma » 2014-07-30 15:38

martin wrote:No, you cannot set up a route with empty target host name. So that would not work.
Hmm... possibly you keep this idea at the back of your mind for a later version?.?... Putting the MX with the highest priority manually into "target server" should help in the meantime...
martin wrote:As for certificates, I'm looking into letting hMailServer rely on the Windows certificate store. So hMailServer will trust the certificates which Windows trusts.
Cool, yipee. I was afraid we have to deal with files and folders and rehashed filenames and other openssl and apache like sh** in future ;) Glad to hear that!

But p-p-p-please don't kill me: I think, enforcement of certificate trust should only be fixed added (if not configurable) for relayer and routes. If the mailserver administrator defines a relayer or a route I can expect him to add the target's certificate or it's CA certificate to windows trusted certificate store in the case it's a selfsigned certificate or from an currently untrusted CA. No big deal.
But for MX resolved targets we would possibly slam a door where encryption with an unkown selfssigned certificate or from an unknown CA is better than no encryption or no delivery.

Or do we keep it completely configurable? For the admin it wouldn't be much more than a checkbox to check or not to check for every "outgoing socket". Nothing really structural complicated.

What do you think about it? What was a good behaviour, or would you prefer to keep it configurable?

Aaaand: What about the hostname (CN)? Does it always have to fit? Some ISPs use one certificate for multiple MX... Aaaarg!

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: STARTTLS feedback?

Post by martin » 2014-07-30 21:07

I think, enforcement of certificate trust should only be fixed added (if not configurable) for relayer and routes.
Right now there's a setting called "Use STARTTLS if available" under SMTP -> Delivery of e-mail. If you de-select this, hMailServer wont' even try to do STARTTLS. This setting is not enabled by default. If you enable it, hMailServer will do a STARTTLS if the remote server announce support for it. But if it fails, the connection will be dropped because that's how Boost/Asio/OpenSSL works which hMailServer relies on. So right this second you might not want to select this if you deal with SMTP servers who claims to support STARTTLS but use self-signed certs or similar.

I have not gotten to the host name validation yet. I am honestly not sure if it's done today or not, I will have to debug to see how OpenSSL works in this aspect.

I'm going to take a closer look at what the peer certificate validation actually does.

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: STARTTLS feedback?

Post by martin » 2014-07-30 21:43

Apparently there's no real spec on how to validate the host name when it comes to email. There are drafts though. One suggestion in the RFC is to simply validate that the host name or IP address we are trying to connect to is included in the certificate. Makes sense, I guess.

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: STARTTLS feedback?

Post by mattg » 2014-07-31 02:56

Yep

That would require IP spoofing to beat.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

prisma
Senior user
Senior user
Posts: 325
Joined: 2010-07-09 13:16

Re: STARTTLS feedback?

Post by prisma » 2014-07-31 11:34

martin wrote:If you enable it, hMailServer will do a STARTTLS if the remote server announce support for it. But if it fails, the connection will be dropped because that's how Boost/Asio/OpenSSL works which hMailServer relies on. So right this second you might not want to select this if you deal with SMTP servers who claims to support STARTTLS but use self-signed certs or similar.
We don't speak about relayer and routes, right? We speak about MX resolved SMTP delivery, correct?
To deactivate use of STARTTLS for MX reolved delivery as option to get around unknown certificates is no option. Simple explanation:
We ourselves (with 5.4-B2014060501) and many other people with other mailservers are able to use selfsigned certificates on 25. Postfix, Exim and other proprietary mailservers do accept these certificates for ESMTPS delivery. They act like better delivery than no delivery (you wrote OpenSSL will cut connection), and better encryption than no encryption. Hmailserver would behave like no other mailserver system behaves at the moment. I think this would be simply wrong. The handling of this case has to be different from relayer and route (if not configurable).
martin wrote:One suggestion in the RFC is to simply validate that the host name or IP address we are trying to connect to is included in the certificate
Yes, this is how it works. But only nearly. Don't forget about wildcard certificates (CN=*.hmailserver.com). This has to be accepted although "mx.hmailserver.com" is not in "*.hmailserver.com".

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: STARTTLS feedback?

Post by martin » 2014-08-01 20:22

Ok, new suggestion:

When MX-resolved SMTP delivery is performed, hMailServer will attempt to use STARTTLS. If it is not supported, or if the TLS handshake fails, hMailServer will run the delivery without STARTTLS. (If handshake fails, this means that a new TCP/IP connection needs to be opened because the connection is in an unknown state after that).

It will be possible to configure hMailServer to never do STARTTLS. The only real benefit to this should be performance. If you are doing a large amount of newsletter delivery for example, you may not be interested in the overhead of STARTTLS. This overhead may be close to zero though (I have not measured).

When delivery is done using a route or the SMTP relayer (not MX-resolved), it's possible to choose between following connection security:
  • None
  • STARTTLS (Optional) - If selected, and STARTTLS is not supported or handshake fails, hMailServer will do the delivery using no connection security.
  • STARTTLS (Required)
  • SSL

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: STARTTLS feedback?

Post by mattg » 2014-08-02 02:29

seems OK to me
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

prisma
Senior user
Senior user
Posts: 325
Joined: 2010-07-09 13:16

Re: STARTTLS feedback?

Post by prisma » 2014-08-04 12:04

Simple Question: Will the SSL handshake fail if a selfsigned and not trusted certificate is used from the MX-resolved server?

If yes, NOK. An unencrypted connection is used where encryption would be possible.
If no, OK. The server will continue encrypted although trust is not sure.

The relay/rout part was OK in the first draft of martin and hadn't to be corrected. What's the difference now to the first draft regarding relay/route?

Any new suggestions for handling CNs?

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: STARTTLS feedback?

Post by martin » 2014-08-04 13:48

When a MX-resolved delivery is made, no actual validation of the certificate is done.

To be honest, I'm not sure what has changed since previously on relay/route. This was more of a complete description of how it will work.

(Also, I don't see this as drafts or revisions but a discussion...)

I still believe it would make sense just to validate that the certificate matches the host name, along the lines of RFC2818.

prisma
Senior user
Senior user
Posts: 325
Joined: 2010-07-09 13:16

Re: STARTTLS feedback?

Post by prisma » 2014-08-04 15:47

martin wrote:When a MX-resolved delivery is made, no actual validation of the certificate is done.
That's what most users need. Very good.

RFC2818 sounds very good! If you'd implement server identification along RFC2818 this could/should also mean:
  • certificate pinning by fingerprint (...If the client has external information as to the expected identity of the server, the hostname check MAY be omitted...)
  • wildcards (...Names may contain the wildcard character * which is considered to match any single domain name component or component fragment...)
  • configurable behaviour (...Automated clients MAY provide a configuration setting that disables this check, but MUST provide a setting which enables it...)
Why all these MAYs and not only the MUSTs? Because SMTP is not HTTP and we want to keep hmailserver working, don't we? :)

japi
New user
New user
Posts: 29
Joined: 2011-06-12 14:09
Location: Germany

Re: STARTTLS feedback?

Post by japi » 2014-08-04 18:34

prisma wrote:Simple Question: Will the SSL handshake fail if a selfsigned and not trusted certificate is used from the MX-resolved server?

If yes, NOK. An unencrypted connection is used where encryption would be possible.
If no, OK. The server will continue encrypted although trust is not sure.

The relay/rout part was OK in the first draft of martin and hadn't to be corrected. What's the difference now to the first draft regarding relay/route?

Any new suggestions for handling CNs?
Sorry, i am not sharing this point of view.
Allowing "untrusted" certificates is a major no go.
If you need it for testing or your private server you can always add the certificate into the windows trust store. But no sane administrator should disable certificate validation on production systems.
Any party which is able to intercept your traffic would then be able to just fake an SSL certificate. There are software solutions which do this on the fly, such as sslsniff (it generates an valid certificate for any hostname at the time of the ssl handshake, using an provided/selfsigned root certificate). Maybe such tools do not yet exist for STARTTLS in public, but the idea is the same: No cert validation is equal to no encryption at all, because it is as easy breakable by the intercepting party as with no encryption.
If the certificate can not be trusted, the delivery should be carried out unencrypted, retried at a later time or the mail should be bounced.

The RFC http://tools.ietf.org/html/rfc3207 is recommending certificate validation indirectly (if hMailServer is delivering to another server it is the "client"):
Both the SMTP client and server must check the result of the TLS
negotiation to see whether an acceptable degree of authentication and
privacy was achieved. Ignoring this step completely invalidates
using TLS for security.
The decision about whether acceptable
authentication or privacy was achieved is made locally, is
implementation-dependent, and is beyond the scope of this document.

The SMTP client and server should note carefully the result of the
TLS negotiation. If the negotiation results in no privacy, or if it
results in privacy using algorithms or key lengths that are deemed
not strong enough, or if the authentication is not good enough for
either party, the client may choose to end the SMTP session with an
immediate QUIT command, or the server may choose to not accept any
more SMTP commands.

[...]

If the TLS negotiation fails or if the client receives a 454
response, the client has to decide what to do next. There are three
main choices: go ahead with the rest of the SMTP session, retry TLS
at a later time, or give up and return the mail to the sender. If a
failure or error occurs, the client can assume that the server may be
able to negotiate TLS in the future, and should try negotiate TLS in
a later session, until some locally-chosen timeout occurs, at which
point, the client should return the mail to the sender. However, if
the client and server were only using TLS for authentication, the
client may want to proceed with the SMTP session, in case some of the
operations the client wanted to perform are accepted by the server
even if the client is unauthenticated.
Why does the ssl cert validation have to be handled by hMailServer anyway? I was pretty sure that openssl would handle all the checks for you? (Signature validation of any cert in the chain up to the trusted root cert, hostname validation, expiration check, revocation check by using CRL/OCSP/OCPS-Stapling, etc ...)
Checking this all "manually" is a lot of work and likely to leave security holes...

Best Regards,
Jan

japi
New user
New user
Posts: 29
Joined: 2011-06-12 14:09
Location: Germany

Re: STARTTLS feedback?

Post by japi » 2014-08-04 19:20

Let me give you another reason why not validation the certificate is a mean thing:
Virtually every postmaster is giving his certificate to one of the major CAs to have it signed and trusted by all clients.
Browsers and Mail Clients do usually not trust selfsigned certificates for a reason.
This leads me to believe that around 99,9% of public facing mail servers which use TLS are going to have a valid certificate and only internal / test servers have selfsigned certificates.

What happens now, if a mailserver is not validating the certificate by design?
- For servers with selfsigned certificates (0,1%) you enable an extremely weak encryption (which is easy to intercept/decrypt for any hacker).
- For servers with officially CA signed certificates (99,9%) you break the security of the encryption, which was previously not possible to hack.

I hope you can see why this idea outrages me. Breaking security for the masses is a bad idea.
Sorry for the double post.

Best regards,
Jan

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: STARTTLS feedback?

Post by martin » 2014-08-04 21:29

Why does the ssl cert validation have to be handled by hMailServer anyway? I was pretty sure that openssl would handle all the checks for you? (Signature validation of any cert in the chain up to the trusted root cert, hostname validation, expiration check, revocation check by using CRL/OCSP/OCPS-Stapling, etc ...)
I have not written the code to do the actual cert validation.
  • The TLS handshake is performed by OpenSSL.
  • The Windows certificate store is used. OpenSSL does not come with any actual certificates so the CA/CLR's are loaded from Windows.
  • The cert host name validation is not part of OpenSSL, but Boost/Asio which hMailServer uses implements that (by using functionality in OpenSSL)
OpenSSL has some basic low-level parts to do TLS. Boost/Asio wraps these and hMailServer uses it. hMailServer must still contain code to initiate the handshake, to decide whether the certificate should be validated and to decide if the host name should be validated and if so with what host name.
Ignoring this step completely invalidates using TLS for security.
I know that's from the spec, but as far as I can tell that's not 100% correct. If someone at my internet service provider attempts to read their mail server logs to figure out what I sent to Gmail 2 months ago, they will not be able to do that if I used TLS at that point. To be able to intercept the data, they must first perform a man-in-the-middle-attack, right? And they cannot do that retroactively.

If a MX-resolved delivery is performed, then STARTTLS is optional. This is as far as I can tell required since there's a lot of servers out there which does not support STARTTLS and requiring STARTTLS support from everyone would break interoperability completely. And if we assume that someone is intercepting the traffic and inserting a faked certificate, they can just as well intercept the traffic and filter out the STARTTLS response (that would even be easier). Then hMailServer won't know that the server supports STARTTLS and hence won't try to start a TLS session. So I don't really see how enforcing a successful validation of the remote certificate would stop a man-in-the-middle attack if the STARTTLS is optional.

I configured my own server to use an invalid cert, and both Gmail and Outlook.com delivered happily to it anyway. They ignored that I was using a self-signed cert issued for example.com. I've also confirmed what prisma has been saying about Exim (by reading the docs).

Am I missing something?

(Note that this diiffers from the behavior if a specific SMTP relayer or SMTP route is used, because then you can specify that TLS is required).

japi
New user
New user
Posts: 29
Joined: 2011-06-12 14:09
Location: Germany

Re: STARTTLS feedback?

Post by japi » 2014-08-05 00:43

Okay, I get your point now.
A Man in the middle could just prevent STARTTLS, that is correct.
And if routes with enforced STARTTLS do not allow the connection with self signed certificates i think i am okay with it :wink:
(for example if i enforce TLS to google.com then the connection is dropped if the cert is selfsigned)

But my point is: If the TLS connection is successfully established, i would expect it to be secure.
When allowing selfsigned certificates, this is not the case. People are left with a sort of placebo encryption ("could be man in the middle - who knows?").
Why are companies paying millions for CA signatures if the hassle is for nothing and it is not validated?
Please make that an option, not the default :(
Untrusted certificates should lead to handshake failure and fallback to unencrypted delivery.
I know this sounds more insecure than a breakable encrypt-anyway solution, but it leaves the majority of connections in a more secure state AND both sides notice, that their encryption is not working/not secure and can take appropriate actions.

I would absolutely want to have cert validation enabled at any time (don't care about the 0,1% unencrypted connections) and every pentest would probably mark this as a security finding (at least if STARTTLS-enforced-routes are also insecure).

I am wondering about google and hotmail allowing selfsigned certs, but i am pretty sure they have other sorts of security measures active to prevent communication eavesdropping between the big players and maybe just allow this for the small mailservers. Who knows. Comfort is maybe more important than security for free services like google and hotmail, but i guess there are a lot of companies which would expect otherwise, especially for business to business communication (at least i would expect a business to buy a Domain-validated CA signature for their public facing SMTP Server ;-) ).

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: STARTTLS feedback?

Post by mattg » 2014-08-05 00:49

martin wrote:
Ignoring this step completely invalidates using TLS for security.
I know that's from the spec, but as far as I can tell that's not 100% correct. If someone at my internet service provider attempts to read their mail server logs to figure out what I sent to Gmail 2 months ago, they will not be able to do that if I used TLS at that point. To be able to intercept the data, they must first perform a man-in-the-middle-attack, right? And they cannot do that retroactively.
No they can't retroactively do man-in-the-middle, however because we are human, we are creatures of habit, and we will use the same server again.
If you as a malignant mail admin see that I send encrypted mail to a specific server, then you can guess that I will do that again, and create a man-in-middle for next time.
martin wrote:If a MX-resolved delivery is performed, then STARTTLS is optional. This is as far as I can tell required since there's a lot of servers out there which does not support STARTTLS and requiring STARTTLS support from everyone would break interoperability completely. And if we assume that someone is intercepting the traffic and inserting a faked certificate, they can just as well intercept the traffic and filter out the STARTTLS response (that would even be easier). Then hMailServer won't know that the server supports STARTTLS and hence won't try to start a TLS session. So I don't really see how enforcing a successful validation of the remote certificate would stop a man-in-the-middle attack if the STARTTLS is optional.
You are correct.
The only way to make StartTLS work is to have a database of valid certificates, and if the certificate exists in this database then StartTLS MUST get used. Something like a DNS server does. Does such a thing already exist? This would also require a secure connection to stop similar to DNS spoofing.
martin wrote:I configured my own server to use an invalid cert, and both Gmail and Outlook.com delivered happily to it anyway. They ignored that I was using a self-signed cert issued for example.com. I've also confirmed what prisma has been saying about Exim (by reading the docs).

Am I missing something?
This shows yet again that SSL, and StartSSL is a complete waste of time. If you want message security, then use message level encryption.
martin wrote:(Note that this diiffers from the behavior if a specific SMTP relayer or SMTP route is used, because then you can specify that TLS is required).
But unless these certificates are validated, this is only a step in the right direction.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

japi
New user
New user
Posts: 29
Joined: 2011-06-12 14:09
Location: Germany

Re: STARTTLS feedback?

Post by japi » 2014-08-05 01:35

mattg wrote: The only way to make StartTLS work is to have a database of valid certificates, and if the certificate exists in this database then StartTLS MUST get used. Something like a DNS server does. Does such a thing already exist? This would also require a secure connection to stop similar to DNS spoofing.
This problem is approached ("solved") by DANE - http://en.wikipedia.org/wiki/DNS-based_ ... d_Entities
Postfix already supports it.
But my DNS Provider does not yet support DNSSEC, so this is currently not on my wish list (i have to bug them first) :mrgreen:

Another approach is to validate the certificate fingerprint using another connection. (there are services that can do this, for example https://www.grc.com/fingerprints.htm )
Google had a project for this approach, called Google Certificate Catalogue (http://googleonlinesecurity.blogspot.de ... urity.html).
And another project is http://perspectives-project.org/ .
All these are for HTTPS afaik and can not be used for SMTPS reliably.

And then there is certificate- or CA-pinning which has been mentioned before. (Saving the certificate-thumbprint (or CA) into a Database and notify on change).
mattg wrote: This shows yet again that SSL, and StartSSL is a complete waste of time. If you want message security, then use message level encryption.
No it is not. :wink:
If client to server and server to server connections are all with enforced/validated TLS, then the attackers have to be in control of either the client or server, to read the mail. Germanys biggest providers have already enforced client to server encryption and are enforcing encryption between their servers (and using STARTTLS to other servers if possible).
Message level encryption is so uncomfortable that it will not succeed any time soon (sadly).
Implementing STARTSSL will secure most of the traffic. An attacker who removes the STARTTLS command for example can be easily detected (good logging provided). No one is going to use this for mass surveillance :P

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: STARTTLS feedback?

Post by mattg » 2014-08-05 01:48

japi wrote:
mattg wrote: This shows yet again that SSL, and StartSSL is a complete waste of time. If you want message security, then use message level encryption.
No it is not. :wink:
If client to server and server to server connections are all with enforced/validated TLS, then the attackers have to be in control of either the client or server, to read the mail. Germanys biggest providers have already enforced client to server encryption and are enforcing encryption between their servers (and using STARTTLS to other servers if possible).
Will the large German ISPs start StartTLS from a connection with a deliberately bad certificate, like Martin's example above??

japi wrote:Message level encryption is so uncomfortable that it will not succeed any time soon (sadly).
+1
Although in HealthCare in Australia, we MUST use message level encryption to qualify for certain government funding, and to meet the accreditation standards that are set for us.
(There are other security issues with the model that we have adopted, but that is for a different time / place)

Matt
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

japi
New user
New user
Posts: 29
Joined: 2011-06-12 14:09
Location: Germany

Re: STARTTLS feedback?

Post by japi » 2014-08-05 02:40

mattg wrote: Will the large German ISPs start StartTLS from a connection with a deliberately bad certificate, like Martin's example above??
I whish you didn't ask.
I have now tested web.de, gmx.de, t-online.de, aim.com, gmail.com, hotmail.com and yahoo.com.
They all accept a selfsigned certificate for the CN "localhost".
Now i'm sad and go to sleep :cry:
Wondering if everyone copied the "bad behavior" of the other...

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: STARTTLS feedback?

Post by mattg » 2014-08-05 02:51

japi wrote:Wondering if everyone copied the "bad behavior" of the other...
I think that no-one is driving.
We are just doing stuff because it makes us feel like we are doing 'something', and I'm not sure that I understand why.

We have an opportunity to help drive this. If DANE is the solution, then lets look more into that, and then encourage the 'big guys' to do this too.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

prisma
Senior user
Senior user
Posts: 325
Joined: 2010-07-09 13:16

Re: STARTTLS feedback?

Post by prisma » 2014-08-05 10:31

@japi+mattg, I'll give a structured answer because I want to respond to multiple objections. Only for better reading, it not a sorted enumeration and I don't want to sound educational. And, as martin said, it's all fun and brainstorming here, so:
  • We all are IT pros and do theoretically know what would be the best to keep your message content privat.
  • Also we all live in the same reality we have to deal with. And this reality teaches us, that reality and theory are divergent.
  • End-to-end encryption will never keep the meta and traffic data private.
  • If you read this thread and others I already wrote that DANE would be the only solution to get startTLS waterproof and should be the final aim.
  • Until DNSsec is a worldwide available standard, we have to raise the bar step-by-step. It's not about just doing something.
and last but not least:
  • If you do StartTLS for MX resolved delivery with fixed enforced cert validation, hmailserver will STOP WORKING. You'd have to deactivate startTLS for MX delivery to get him working again. And that's no option too.
And yes, I wouldn't keep it that plain, because other context means other requirements. Therefore we should:
  • keep as much as possible and meaningful configurable.
  • additional to windows trust we could introduce certificate pinning to raise security in the case we know the certificate, but FQDN does not fit and/or trust is not possible (e.g. CA cert is not available for import into trusted CAs)
If it needs in future a AA or AAA hacker instead of just my granny and a wireshark to read standard (!) mail communication, we have reached something.

japi
New user
New user
Posts: 29
Joined: 2011-06-12 14:09
Location: Germany

Re: STARTTLS feedback?

Post by japi » 2014-08-05 13:42

prisma wrote:
  • If you do StartTLS for MX resolved delivery with fixed enforced cert validation, hmailserver will STOP WORKING. You'd have to deactivate startTLS for MX delivery to get him working again. And that's no option too.
No one wants a not working hmailserver. I just suggested using a fallback to unencrypted connection intsead of allowing self-signed certificates :wink:
This does not break things at all and does strengthen connections with CA-signed certificates (as in "you can be sure that an successful TLS connection is actually secure") .
But with everyone allowing self-signed certificates for random hostnames, that should be an option in hMailServer too. (even if i can not understand why everyone is doing this. This makes SMTPS somewhat insecure if compared to HTTPS, where every browser warns the user in such a case)

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: STARTTLS feedback?

Post by martin » 2014-08-09 17:17

Cert validation actually got a bit trickier than expected...

The Windows certificate store does not actually contain all trusted certificates. They are somehow automatically downloaded using Windows Update (!?) when HTTPS-requests are made using WinInet. OpenSSL does not come with a certificate store, nor does it actually has any real built-in support to handle a full CRL-check. Instead, you're supposed to download the CRL's as needed and to do that some CA's require an agreement with them. And some certs are embedded as resources in the crypt32.dll.

Reading more about this, it's a pretty sad story. Apparently Chrome no longer does CRL-requests but are using their own proprietary format CRLset instead. This file includes ~2% of all actual revocations. This appears completely broken to me (you simply cannot assume that Chrome will care about revoked certs any more since they only include a small portion of all revoked).

As I understand it, Firefox do not download the CRL-lists either. One reason is that the CRL-lists can get pretty big (some are apparently >40MB) and downloading such a list prior to a user visiting a site will make the web too slow. It would also introduce a single-point-of-failure (if the CRL-list server is down). Also, to make this actually work you would have to download the list all the time, like every 4 hour and then the required bandwidth becomes pretty high.

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: STARTTLS feedback?

Post by SorenR » 2014-08-09 18:21

martin wrote:Cert validation actually got a bit trickier than expected...

The Windows certificate store does not actually contain all trusted certificates. They are somehow automatically downloaded using Windows Update (!?) when HTTPS-requests are made using WinInet. OpenSSL does not come with a certificate store, nor does it actually has any real built-in support to handle a full CRL-check. Instead, you're supposed to download the CRL's as needed and to do that some CA's require an agreement with them. And some certs are embedded as resources in the crypt32.dll.

Reading more about this, it's a pretty sad story. Apparently Chrome no longer does CRL-requests but are using their own proprietary format CRLset instead. This file includes ~2% of all actual revocations. This appears completely broken to me (you simply cannot assume that Chrome will care about revoked certs any more since they only include a small portion of all revoked).

As I understand it, Firefox do not download the CRL-lists either. One reason is that the CRL-lists can get pretty big (some are apparently >40MB) and downloading such a list prior to a user visiting a site will make the web too slow. It would also introduce a single-point-of-failure (if the CRL-list server is down). Also, to make this actually work you would have to download the list all the time, like every 4 hour and then the required bandwidth becomes pretty high.
OCSP (Online Certificate Status Protocol) ??
https://www.ietf.org/rfc/rfc2560.txt

http://www.slsmk.com/ms-windows-2008-r2 ... -protocol/

http://news.netcraft.com/archives/2013/ ... -ocsp.html

Not that I know anything about it, but it appears to be "the thing"... :wink:
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

japi
New user
New user
Posts: 29
Joined: 2011-06-12 14:09
Location: Germany

Re: STARTTLS feedback?

Post by japi » 2014-08-09 18:36

martin wrote:The Windows certificate store does not actually contain all trusted certificates. They are somehow automatically downloaded using Windows Update (!?) when HTTPS-requests are made using WinInet.
the windows cert store should contain every trusted root certificate as far as i know.
Intermediate certificates should be sent by the server along the server-certificate, if an intermediate certificate is necessary. An additional download of certificates should not be necessary, but is necessary, if the server is not completely correctly configured and does not send the intermediate certificate. It is implementation specific that the client tries to get the intermediate certificate he does not know (i guess that is what Windows Update is used for). Many implementations will abort the connection, if the intermediate certificate is not supplied by the server and the chain to the trusted root can not be built.
martin wrote:OpenSSL does not come with a certificate store, nor does it actually has any real built-in support to handle a full CRL-check. Instead, you're supposed to download the CRL's as needed and to do that some CA's require an agreement with them. And some certs are embedded as resources in the crypt32.dll.
The CRL check is indeed a bit tricky. But i have never heard of an agreement to be necessary to do this. As long as you only validate the certificate by using the CRL-Download-Link supplied in the certificate, no agreement should be necessary, as this is a standard procedure. If an OCSP Link is present, it should be preferred to be used for validation. If the Certificate uses OCSP-Stapling, that should be preferred over both.
As CRL and OCSP need additional lookups/connections they both slow down the handshake. Also any man in the middle could block those connections and make the validation fail. For that reason no browser actually validates this during the handshake. Some browsers try the CRL/OCSP lookup after the connection is established and warn the user, if the outcome shows, that the cert is revoked. It could be too late by then, but it is better than nothing.
martin wrote:Reading more about this, it's a pretty sad story. Apparently Chrome no longer does CRL-requests but are using their own proprietary format CRLset instead. This file includes ~2% of all actual revocations. This appears completely broken to me (you simply cannot assume that Chrome will care about revoked certs any more since they only include a small portion of all revoked).

As I understand it, Firefox do not download the CRL-lists either. One reason is that the CRL-lists can get pretty big (some are apparently >40MB) and downloading such a list prior to a user visiting a site will make the web too slow. It would also introduce a single-point-of-failure (if the CRL-list server is down). Also, to make this actually work you would have to download the list all the time, like every 4 hour and then the required bandwidth becomes pretty high.
The CRLset approach by chrome is a trade-off, but Firefox is going that direction too now (https://wiki.mozilla.org/CA:RevocationPlan). Althoug it contains only a few revoked certificates, this can be enough to secure most connections, as long as it contains all revoked intermediate certificates. Revoked root certificates are removed from the trust store by windows update after some time afaik. The downside of this approach is, that it does not cover every certificate revoked by smaller sites (which has recently been done by many after the heartbleed bug).

For the caching time of CRL files and OCSP answers i would rely on the HTTP caching mechanisms like the HTTP ETag, the If-Modified-Since-Header or the Cache-Control: max-age- / Expires-Heades (if the OCSP protocol does not specify otherwise).

Anyway...
Maybe we should just invent another way of handling cert validation :mrgreen:
To not slow down the connection we should not validate any CRL/OCSP at connection time (or maybe only OCSP with 1-2 seconds timeout?).
Instead hMailServer would just log something like the following line:

Code: Select all

... SSL Connection established: IP:123.123.123.123 MX-Hostname:gmail-smtp-in.l.google.com Recipient-Hostname:gmail.com CertificateHash:sd9f87sfd98s7fd98s7df9s8df7s9d8f7s9df7 Chipher:.... KeyExchange:....OCSP-Stapling-valid:false 
The information in that line could then be used afterwards to check if the connection was secure by using any means of validation the admin would like to use in an external script for example.

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: STARTTLS feedback?

Post by martin » 2014-08-09 19:14

the windows cert store should contain every trusted root certificate as far as i know.
How many certs do you have in the Trusted Root Certification Authorities store for "Local computer" account? I have 27. And if I delete some of them, they are automatically created again. This behavior is mentioned in the Microsoft documentation in various places, such as here: http://support.microsoft.com/kb/2813430/. Also, apparantly there's been some Python bug describing the fact that the Python mechanism for listing the certs doesn't really include all of them. But I will continue to investigate this to see if I can find a way to force the Windows Crypto API to give me all CA's.

Here's the list of CA's Windows trusts:
http://social.technet.microsoft.com/wik ... l-cas.aspx

I don't have all of the in my trusted root CA store. Like the finland TeliaSonera root CA - I don't have it in my trusted certificate store. But if I open crypt32.dll in a resource editor I can find the cert embedded there.
But i have never heard of an agreement to be necessary to do this.
It's mentioned here in the below link. Also note that apparently it's quite common that certificates does not include the CRL download link or that they include invalid links.
http://etutorials.org/Programming/secur ... h+OpenSSL/
Althoug it contains only a few revoked certificates, this can be enough to secure most connections, as long as it contains all revoked intermediate certificates.
I don't think it does.

I will continue to look into using the crypto api for the cert verification. I don't have resources or skills to implement a completely own method for this.

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: STARTTLS feedback?

Post by mattg » 2014-08-10 02:15

martin wrote:
the windows cert store should contain every trusted root certificate as far as i know.
How many certs do you have in the Trusted Root Certification Authorities store for "Local computer" account? I have 27. And if I delete some of them, they are automatically created again.
Absolutely guarantee that I have to add Australian Government Health Department to the trusted root, via Management console on every computer that I want to set up trust for PKI message encryption that I need to use. The windows cert store is VERY light on what it has, and I suspect that the ones that are there by default are trusted due to some commercial arrangement with Microsoft.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: STARTTLS feedback?

Post by ^DooM^ » 2014-08-10 02:16

Yep have to do the same with KBC bank as well.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: STARTTLS feedback?

Post by mattg » 2014-08-10 02:22

And if Chrome + Firefox both don't care about revocation, then I agree with Martin
martin wrote:...This appears completely broken to me...
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: STARTTLS feedback?

Post by martin » 2014-08-10 10:29

And if Chrome + Firefox both don't care about revocation, then I agree with Martin
The CRLset file I downloaded 2-3 days ago contains a list of roughly 18 000 revoked certificates. But if you look at the actual number of revoked certificates on the Internet, it should be closer to 2 million. So Google has selected 18 000 of these which they find "important". They have also asked other CA's to provide them with "important" revocations. One problem though is that the CA's does normally not know whether a revocation is important or not (because that's not something which have been tracked).

Windows by default trusts roughly 300 CA's. In the CRLset, there are only revocations included from 56 CA's. To me, CRLset does not look like a trade-off but rather a good way to ruin the CA system. (Also, CA Security Council has stated that they are not fond of what Google has done).

Some parts of this are actually pretty funny. A security researcher revoked his certificate to a part of his website to make it easier for people to test revocation. After having done this, Google still did not show the certificate as revoked - because Google did not include any revocations from the CA the researcher was using. So some guy reported a bug with Chrome that the website was still accessible even though the certificate was revoked. Then rather than contacting the CA and asking for a list of important revocations, they "hard-coded" that specific single certificate into the CRLset and closed the bug. Full story here: https://www.grc.com/revocation/crlsets.htm
The windows cert store is VERY light on what it has, and I suspect that the ones that are there by default are trusted due to some commercial arrangement with Microsoft.
Windows trusts CA's which are part of the Microsoft Root Certificate Program. It's not really a commercial agreement as far as I can tell, but Microsoft will audit you if you want them to trust you. That makes sense I guess. Here's the member list, containing roughly 300 companies I think:
http://social.technet.microsoft.com/wik ... l-cas.aspx

Now, I don't have 300 certificates in my Trusted Root CA Store in Windows. But if I pick one company in the above member list, say Camerfirma and go to https://www.camerfirma.com/ (using Chrome or IE), Windows will automatically add this certificate to my Trusted Root CA Store store in Windows. This is done because the list of CA's which Microsoft trusts are not limited to the ones you happen to see in the Trusted Root CA Store, but there's a hard-coded list in crypt32.dll as well. So if you go to the web site of all the program members, you should end up having roughly 300 certs in your cert store (assuming that the cert program members have websites signed with their own cert).

I don't think that the Windows cert store is very light. Trusting 300 certificate authorities seems like trusting quite a lot. Any of these 300 organizations could issue false certificates. It's enough with one evil guy in one of those 300 organizations to issue a new certificate for say Google.com.

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: STARTTLS feedback?

Post by martin » 2014-08-10 21:30

I'm trying out the following approach now.

1) Use OpenSSL to initiate the SSL/TLS handshake (as before).
2) When the actual certificate verification takes place, convert the certificate from OpenSSL certificate structure to a Windows Crypto API certificate structure.
3) Ask Windows CryptoAPI to verify the complete certificate chain (including revocation checking)
4) If verification fails, tell OpenSSL that so it can fail the SSL/TLS handshake.

I've confirmed that:
- Valid certificates are verified successfully
- Revoked certificates results in a verification error (tested with revoked.grc.com:443)
- If a CA certificate is missing in the Trusted Root Certification Authorities, crypto32.dll automatically re-creates it (weird, but expected)
- Certificates placed in the Untrusted Certificates store are not trusted.

So right now at least this approach seems to work. Feels weird having to rely on both OpenSSL and the Windows Crypto API, but rather that than building something myself from scratch.

japi
New user
New user
Posts: 29
Joined: 2011-06-12 14:09
Location: Germany

Re: STARTTLS feedback?

Post by japi » 2014-08-10 22:02

Wow, great work! :D
This really sounds like the best idea if OpenSSL can't do the revocation checks automatically.

The automatic creation of certificates in the trust store was new to me, i always thought i could delete them if i don't trust them... Well, lesson learned :shock: (and i thought Microsoft did clean up that trust store :lol: )

What about self-signed certs or certs with the wrong hostname? - Are you planning on an option to allow / forbid them?
How does the crypto api handle CRL/OCSP-Timeouts? - Validation failure / or is that configurable?

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: STARTTLS feedback?

Post by martin » 2014-08-11 10:15

To be clear: The below only applies to non-mx-resolved deliveries.

If certificate validation fails for any reason, the delivery won't be performed.

If there is a failure to contact the revocation servers, the delivery will fail and an error will be logged:
Certificate verification failed. Expected host: smtp.gmail.com, Windows error code: -2146885613, Windows error message: The revocation function was unable to check revocation because the revocation server was offline.

A similar error (or same - I am a bit unsure) will occur if there's a timeout (I think). Right now this code is relying on the default Windows behavior. I guess it would make sense to add settings to allow certain parts of the process to fail, such as:
  • Untrusted certificate (self-signed)
  • Expired certificate
  • Expired immediate CA
  • Expired revocation data
  • Revocation server timeout
  • Not possible to contact revocation server
And so on. I don't think this will be included in the first version though since adding all these cases will take some time.

I'm not sure if I should change the default behavior so that if hMailServer is not able to contact the revocation servers (and the local cache has expired), an error is logged and message is delivered anyway. I think the current behavior, to only succeed with delivery if everything is OK is a pretty good first step. Soft-fail implementations seems to have received quite a lot of criticism.

hMailServer is currently using the default Windows timeouts for CRL-checks, which are 15 seconds per certificate. I think that's OK - I understand that web browsers wants a much lower timeout since they want to appear "quick", but email is asynchronous and slower than web browsing and I don't think using 15-second timeout will be an issue.

Currently, if you run in an environment where you actually need to deliver to a server which does not have a valid certificate, you can disable certificate verification globally in hMailServer. That might serve as a workaround.

prisma
Senior user
Senior user
Posts: 325
Joined: 2010-07-09 13:16

Re: STARTTLS feedback?

Post by prisma » 2014-08-11 13:04

Sounds very good. But I think we can accumulate and slash your single steps a little bit:
  • Untrusted certificate (self-signed)
This doesn't need a hmailserver configuration at all. Trust itself is configurable with windows certificate store. To have two different possibilities of configuration increases complexity needless. This applies also to untrusted intermediate and root CAs. But I could imagine a configuration within hmailserver which triggers the certificate store configuration. This would make sense.
  • Expired certificate
  • Expired intermediate CA
  • Expired root CA
I would propose to accumulate these 3 items to a single one. I see no need to handle "ignore certificate expiration" separately. But maybe I'm wrong...
  • Expired revocation data
  • Revocation server timeout
  • Not possible to contact revocation server
I would propose to accumulate these 3 items to a single one. I'm sure that "Ignore CRL errors" is enough. Many windows functions act the same way.

And last but not least one setting is completely missing:
  • Ignore hostname mismatch
;)

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: STARTTLS feedback?

Post by martin » 2014-08-11 15:38

Untrusted certificate (self-signed)
Oh, I never tried to trust a self-signed cert. I'm guessing I can just add the public certificate to my trusted CA root?

japi
New user
New user
Posts: 29
Joined: 2011-06-12 14:09
Location: Germany

Re: STARTTLS feedback?

Post by japi » 2014-08-11 23:19

Yes, you can add it in the root cert store
(For anyone interested: A HowTo can be found here under the headline "Installing a Certificate in the Trusted Root Certification Authorities Store": http://msdn.microsoft.com/en-us/library/ms733813.aspx )

Could you please clarify what you mean by non-mx-resolved deliveries? Are you Checking the certs only on specified SMTP Routes?

I have just tested on which hostname the google cert is issued... And i am again shocked.. No hostname validation is possible there at all..
The Certificate is issued to "mx.google.com", but the MX-Record for "gmail.com" points to "gmail-smtp-in.l.google.com" as the primary server.
That resolves to two IPs: "2a00:1450:4013:c01::1a" and "74.125.136.27" (in my test).
Now guess what the reverse lookup for the IPv4 address is? Right... "ea-in-f27.1e100.net" :roll: (IPv6 rdns is "ea-in-x1a.1e100.net")
And guess what the IP of "mx.google.com" is? Right... There is none.

I can now understand why the hostname should maybe not be checked by hMailServer :lol:
That setup by Google is just crazy and it is surely not the only one which is that messed up.
The only thing matching the CN of the cert is the name in the SMTP Greeting message.
I must have missed the standard which Google is following. At least i hope so. I am slowly giving up on the "security" part of STARTTLS. Especially for MITM-scenarios (hell, every open hotspot is one! Why does nobody care?) :|

EDIT: I have extracted the certificate now. "gmail-smtp-in.l.google.com" can be found in the Subject Alternative Names. Those are not shown using the command i used ("openssl s_client -starttls smtp -host gmail-smtp-in.l.google.com -port 25"). Hostname Validation is possible that way. My Bad! Google is doing OK. :mrgreen:

prisma
Senior user
Senior user
Posts: 325
Joined: 2010-07-09 13:16

Re: STARTTLS feedback?

Post by prisma » 2014-08-12 11:33

martin wrote:I'm guessing I can just add the public certificate to my trusted CA root?
Yes, of course. The cert will be trusted. But trust is not enough. Problems with OCSP and CRL distribution point are very very common. Therefore your suggestions are very meaningful.
japi wrote:Could you please clarify what you mean by non-mx-resolved deliveries? Are you Checking the certs only on specified SMTP Routes?
Yes, only for routes and relay cert validation is enabled. MX-delivery validation errors do not (not mandantory?, look downwards) lead to refusing a connection. Other mailservers behave the same way. They use encryption where encryption is possible. Although validation fails.

But I understand the background of your surprise. Therefore I proposed for MX-resolved delivery:
  • A setting for enforcing validation for MX-resolved delivery.
  • hand-in-hand with a certificate "pinning" function to pin certificates by fingerprint for explicit trust. Imagine it like your Firefox or Thunderbird allows explicit trust.

@martin: Of course validation has always be done because errors have to be logged with a warning. But a validation error hasn't to lead to refusing a connection (except otherwise configured through enforcement as I proposed above). After a log review you're able to configure windows trust or pin certs to clean up you're log for future.

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: STARTTLS feedback?

Post by martin » 2014-08-12 15:24

I will write some about this in the online documentation soon (in a few days) which will describe the complete behavior. I'm sure it will not be good enough for everyone in the first version, but it will be better what exists today, it will be fairly safe by default and it will be similar to the default behavior of other servers.

Right now, when MX-resolved delivery is performed, no certificate verification is performed. The reason behind this is that the delivery will be performed regardless of success, so spending time on verification something which will fail often (20% of the time according to stats) could be seen as a waste of resources. Of course, the downside with this behavior is that no warning will be logged if a MX-resolved delivery is made and the certificate is invalid (again, ~20% of the time).

I could change this, so that a verification is always done but if it fails, all that happens is that it's logged as an error. (This behavior was not really possible before when OpenSSL did the certificate verification, but now when I've outsourced that to the Crypto API I can control this a bit better myself)

Do you feel it would make sense to always validate the cert, but soft-fail during MX-delivery?

@prisma, I agree that these options would make sense but I don't think I'll include it in this release. There's always more things which can be add and I think the current behavior is a good starting point. Or would you consider the current funcionality 'broken' without these?

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: STARTTLS feedback?

Post by mattg » 2014-08-12 16:35

martin wrote:I could change this, so that a verification is always done but if it fails, all that happens is that it's logged as an error. (This behavior was not really possible before when OpenSSL did the certificate verification, but now when I've outsourced that to the Crypto API I can control this a bit better myself)

Do you feel it would make sense to always validate the cert, but soft-fail during MX-delivery?
As a starting point, yep, with the routes having an option to force validation
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

japi
New user
New user
Posts: 29
Joined: 2011-06-12 14:09
Location: Germany

Re: STARTTLS feedback?

Post by japi » 2014-08-12 17:29

I would really like to have a log entry on validation failures (and validate every connection, not only on specified routes).
If the connections with failures are not 20% but nearly zero, an option to enable hard-fail could be thought about.
But if it is not logged we will never know how many certificates really don't validate.

Maybe every TLS connection should be logged with all information which is necessary to validate if the connection was secure? (cert hash / validation result / chipher etc.). That would be a great plus, especially to build statistics about failed / successfull validations :)

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: STARTTLS feedback?

Post by ^DooM^ » 2014-08-12 17:39

If logging is applied to this, Might be worth adding a new logging level for SSL related log entries.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

prisma
Senior user
Senior user
Posts: 325
Joined: 2010-07-09 13:16

Re: STARTTLS feedback?

Post by prisma » 2014-08-12 17:56

martin wrote:Or would you consider the current funcionality 'broken' without these?
No, it's definitely not broken. It's a good start. For MX-delivery we should always deliver and encrypt when possible, regardless cert is valid or not. We can increase security again later.

But I would do the "soft" validation anyway and would write it to an explicit security log. Because all RFCs I've seen recommend at least a logging if the client silently steps over a validation error.

Logging brings us also statistical benefits. We will be able to see, which kind of errors appear. If statistics say there are nearly no validation errors except trust errors (which can be solved by configuring windows trust) a pinning function isn't needed that urgent. Then validation enforcement for MX-delivery can be optional introduced in future without a pinning function. But I'm sure we'll need pinning later ;)

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: STARTTLS feedback?

Post by martin » 2014-08-13 15:50

But I would do the "soft" validation anyway and would write it to an explicit security log. Because all RFCs I've seen recommend at least a logging if the client silently steps over a validation error.
I've changed so that cert verification is always done. If it fails but hMailServer proceeds, it's logged but currently to the debug log (there's no security log in hMailServer). I think it would make sense to introduce this, but I feel that a slightly larger change to the logging would make sense rather than to add something new. I feel that the logging is a bit messy now, and I don't think that most users will understand it anyway right now. In this release, it *might* make sense to add it to the Application log instead since users are more likely to have it enabled.

(I think that maybe the log should be "turned up side down" with a focus on sessions somehow. If you want to track "what happened when a user sent this email" today it's hard to follow it from start to end).

I have summarized this thread in the documentation here:
https://hmailserver.com/documentation/l ... n_security

Should not be news to you guys who've read this full thread.

percepts
Senior user
Senior user
Posts: 5282
Joined: 2009-10-20 16:33
Location: Sceptred Isle

Re: STARTTLS feedback?

Post by percepts » 2014-08-13 16:04

On logging, Bill had added in his experimental version logging of which eventhandlers event (name) was being fired which made following debug easier to know where you were in the process. Currently it just says event fired without telling you which one.

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: STARTTLS feedback?

Post by mattg » 2014-08-13 23:46

martin wrote:(I think that maybe the log should be "turned up side down" with a focus on sessions somehow. If you want to track "what happened when a user sent this email" today it's hard to follow it from start to end).
Yep
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

joshiewa
New user
New user
Posts: 2
Joined: 2014-08-13 03:25

Re: STARTTLS feedback?

Post by joshiewa » 2014-08-14 01:08

Hi guys

Just checking that in order to enable incoming STARTTLS I need to download the latest Alpha release? Or should I be able to get this working on the latest production build?

Many thanks
Josh

percepts
Senior user
Senior user
Posts: 5282
Joined: 2009-10-20 16:33
Location: Sceptred Isle

Re: STARTTLS feedback?

Post by percepts » 2014-08-14 01:20

goto http://build.hmailserver.com/ and look under artefacts

this is a development version and not yet beta but a few of us are using it to test it. There is no downgrade path. The DB layout is modified in 5.5.x so its not just a case of swapping out programs and dlls to downgrade.

Post Reply