When hMailServer communicates with other clients and servers (called peers), it is possible to enable encryption of the TCP/IP connection. In hMailServer, this is called Connection security.
There are different types of connection security which can be enabled, and what to use depends on requirements and the functionality available in servers the hMailServer installation communicates with. Before selecting connection security, make sure that the remote peers supports it.
During SSL/TLS handshake, the peers will agree on what cryptographic protocol and cipher to use. This is based on the implemented support in the two peers, and the configuration of them. For example, if a client which only supports SSLv3 connects to a server which supports both SSLv3 and TLS1.1, then SSLv3 will be used. If there is no protocol support in common, the handshake will fail.
In hMailServer, it's possible to override the default SSL/TLS cipher list. To do this, open hMailServer Administrator and navigate to Settings -> Advanced -> Security. In the SSL/TLS ciphers text box, you can enter an OpenSSL cipher list.
As a part of the SSL/TLS handshake, hMailServer will verify that the server it connects to has a correct certificate. This includes several things, for example:
If one of these steps fail, the certificate check fails.
Note that hMailServer does not verify remote client certificates - it only verifies the certificates of servers it connects to. If a end-user connects to hMailServer using for example Thunderbird or Outlook, he does not have to provide a certificate. A remote server which delivers a mail to hMailServer does not have to provide a certificate either. In these scenarios - where a client is connecting to hMailServer, hMailServer will present a certificate which will be used to encrypt the session.
During testing, you may want to disable certificate verification completely. To do this, de-select this option. Keeping certificate verification enabled in production is strongly recommended.
hMailServer uses Windows functionality to verify peer certificates and will hence trust certificates which Windows is configured to trust.
When a MX-resolved delivery is performed, hMailServer will use connection security STARTTLS (Optional). This means that hMailServer will attempt to set up an encrypted communication channel but if this fails it will fall back to a connection with no security.