When hMailServer communicates with other clients and servers (called peers), it is possible to enable encryption of the TCP/IP connection. In hMailServer, this is called Connection security.
There are different types of connection security which can be enabled, and what to use depends on requirements and the functionality available in servers the hMailServer installation communicates with. Before selecting connection security, make sure that the remote peers supports it.
During SSL/TLS handshake, the peers will agree on what cryptographic protocol and cipher to use. This is based on the implemented support in the two peers, and the configuration of them. For example, if a client which only supports SSLv3 connects to a server which supports both SSLv3 and TLS1.1, then SSLv3 will be used. If there is no protocol support in common, the handshake will fail.
In hMailServer, it's possible to override the default SSL/TLS cipher list. To do this, open hMailServer Administrator and navigate to Settings -> Advanced -> Security. In the SSL/TLS ciphers text box, you can enter an OpenSSL cipher list.
As a part of the SSL/TLS handshake, hMailServer will verify that the server it connects to has a correct certificate. This includes several things, for example:
If one of these steps fail, the certificate check fails.
Note that hMailServer does not verify remote client certificates - it only verifies the certificates of servers it connects to. If a end-user connects to hMailServer using for example Thunderbird or Outlook, he does not have to provide a certificate. A remote server which delivers a mail to hMailServer does not have to provide a certificate either. In these scenarios - where a client is connecting to hMailServer, hMailServer will present a certificate which will be used to encrypt the session.
During testing, you may want to disable certificate verification completely. To do this you need to either use the API or access the hMailServer database directly. Using the API, you can set VerifyRemoteSslCertificate = False on the Settings object. If you want to update the database manually, issue the following command and then restart the hMailServer service.
UPDATE hm_settings SET settinginteger = 0 WHERE settingname = 'VerifyRemoteSslCertificate'
Keeping certificate verification enabled in production is strongly recommended.
hMailServer uses Windows functionality to verify peer certificates and will hence trust certificates which Windows is configured to trust.
When a MX-resolved delivery is performed, hMailServer will use connection security STARTTLS (Optional). This means that hMailServer will attempt to set up an encrypted communication channel but if this fails it will fall back to a connection with no security.