3rd party application connecting to hMailServer over TLS to port 465 fails

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
codlts
New user
New user
Posts: 6
Joined: 2019-12-06 17:51

3rd party application connecting to hMailServer over TLS to port 465 fails

Post by codlts » 2021-05-14 19:08

Hello,

I am very new to configuring SMTP server and SSL certs so please bear with me.

I am running hMailServer v. 5.6.7-B2425. I need to set up hMailServer as SMTP server just for sending email notifications from various systems in my LAN to both internal and external users.

I set up SSL cert using this article (viewtopic.php?f=21&t=35965&p=226546&hilit=ssl#p226546) and port 465 and 995 using the SSL cert. Ran this test (openssl s_client -connect mail.mydomain.com:465) and it ran successfully as I got "220 mail.mydomain.com ESMTP" at the end of the output. When I ran this test, I got this in the hMailServer log:

"DEBUG" 5936 "2021-05-12 01:51:47.447" "Creating session 62"
"TCPIP" 5936 "2021-05-12 01:51:47.447" "TCP - 192.168.123.25 connected to 192.168.123.25:465."
"DEBUG" 5936 "2021-05-12 01:51:47.447" "TCP connection started for session 61"
"DEBUG" 5936 "2021-05-12 01:51:47.447" "Performing SSL/TLS handshake for session 61. Verify certificate: False"
"TCPIP" 6292 "2021-05-12 01:51:47.478" "TCPConnection - TLS/SSL handshake completed. Session Id: 61, Remote IP: 192.168.123.25, Version: TLSv1.2, Cipher: ECDHE-RSA-AES256-GCM-SHA384, Bits: 256"
"SMTPD" 6292 61 "2021-05-12 01:51:47.478" "192.168.123.25" "SENT: 220 mail.mydomain.com ESMTP"

The problem I am having is when trying to configure ADSelfService Plus to use TLS connection to the hMailServer to send out notification emails. This application has SSL and TLS as connection security options and I am using SMTP authentication.

When I use port 465 with TLS connection security, it tries to connect for some time and fails. This is the error message in the hMailServer log:

"DEBUG" 4796 "2021-05-12 18:46:45.470" "Creating session 6"
"TCPIP" 4796 "2021-05-12 18:46:45.470" "TCP - 192.168.123.100 connected to 192.168.123.25:465."
"DEBUG" 4796 "2021-05-12 18:46:45.470" "TCP connection started for session 3"
"DEBUG" 4796 "2021-05-12 18:46:45.485" "Performing SSL/TLS handshake for session 3. Verify certificate: False"
"TCPIP" 4796 "2021-05-12 18:47:15.469" "TCPConnection - TLS/SSL handshake failed. Session Id: 3, Remote IP: 192.168.123.100, Error code: 335544539, Message: short read"
"DEBUG" 4796 "2021-05-12 18:47:15.485" "Ending session 3"

I then tried to use port 465 with SSL connection security and it immediately fails. I made sure the tick SSL v3.0 box in SSL/TLS page. This is the error message in the hMailServer log:

"DEBUG" 1244 "2021-05-12 18:50:41.305" "Creating session 13"
"TCPIP" 1244 "2021-05-12 18:50:41.305" "TCP - 192.168.123.100 connected to 192.168.123.25:465."
"DEBUG" 1244 "2021-05-12 18:50:41.305" "TCP connection started for session 10"
"DEBUG" 1244 "2021-05-12 18:50:41.305" "Performing SSL/TLS handshake for session 10. Verify certificate: False"
"TCPIP" 1244 "2021-05-12 18:50:41.336" "TCPConnection - TLS/SSL handshake failed. Session Id: 10, Remote IP: 192.168.123.100, Error code: 336151574, Message: sslv3 alert certificate unknown"
"DEBUG" 1244 "2021-05-12 18:50:41.336" "Ending session 10"

I would like to use TLS connection security to connect to the hMailServer. I just tried SSL to see what would happen. Any idea why the testing is successful, but the actual connection via TLS from this 3rd party application fails? Thank you in advance.

mikedibella
Senior user
Senior user
Posts: 503
Joined: 2016-12-08 02:21

Re: 3rd party application connecting to hMailServer over TLS to port 465 fails

Post by mikedibella » 2021-05-14 20:28

SSL and TLS are misnomers in this use case. SSL means that the client will negotiate TLS before SMTP, while TLS means that the client will negoiate SMTP first and upgrade the connection to secure using the STARTTLS verb. The actual secure protocols used (SSL 3.0, TLS 1.0, 1.1, 1.2, 1.3) will depend on what the client and service mutually negotiate.

User avatar
jim.bus
Senior user
Senior user
Posts: 701
Joined: 2011-05-28 11:49
Location: US

Re: 3rd party application connecting to hMailServer over TLS to port 465 fails

Post by jim.bus » 2021-05-14 21:40

You really should show all the Logs. Make sure in hMailAdmin you have selected all your Logs to be shown.

I am guessing the problem is in your Email Client configuration. Your error seems to be a Short Read on one of your attempts, Be aware that the security connection from your Email Client to hMailServer is only for the connection to hMailServer. hMailServer will negotiate a security connection with the Receiving Email Server if both Email Servers can agree on a security protocol that both support. Your specifying the security connection from the Email Client to hMailServer has no effect on the connection from Email Server to Email Server.

You may also have a Certificate Problem in that I believe from comments I have seen from other Forum Helpers that the Certificate needs to be using the .pem file extension when you install the Certificate into hMailServer. Not all Certificates automatically have the .pem file extension format. But you would probably have seen an error in the Error Logs if this had happened. But you haven't shown all your Logs so cannot tell.

palinka
Senior user
Senior user
Posts: 2704
Joined: 2017-09-12 17:57

Re: 3rd party application connecting to hMailServer over TLS to port 465 fails

Post by palinka » 2021-05-14 23:25

https://duckduckgo.com/?q=site%3Ahmails ... +335544539

I only looked at the top result, but it seems to be a problem with the certificate. Usually that has something to do with chain certificates if nothing else seems to be wrong. If the certificate was so fubar that it couldn't load into hmailserver (like its the wrong type of certificate) - you'd know because it would error out big time.

https://duckduckgo.com/?q=site%3Ahmails ... +336151574

Second error code. Top result seems to confirm the chain certificate thing: viewtopic.php?t=30930
If there is a intermediate certificate (a certificate between your certificate and the root certificate) you also need to add that certificate in your .crt file from your own certificate.
Open your .crt file in (preferably) notepad++ also open the .crt file from the intermediate certificate and copy that information. paste the information before the certificate in the certificate you earlier saved.

it should look like this:

Code: Select all

-----BEGIN CERTIFICATE-----
<lots of gibberish from the intermediate certificate>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<even more gibberish the reply from the authority>
-----END CERTIFICATE-----

save the certificate and private key (the .key file) in a directory readable for hmailserver (preferably in a directory *only* hmailserver can read).

mikedibella
Senior user
Senior user
Posts: 503
Joined: 2016-12-08 02:21

Re: 3rd party application connecting to hMailServer over TLS to port 465 fails

Post by mikedibella » 2021-05-15 00:19

Make sure that you are using the same connection security setting on both hMailServer and with ManageEngine. Although not an official "standard", it is typical for port 465 to use what hMailserver (https://www.hmailserver.com/documentati ... n_security) calls "SSL/TLS" and what I suspect ManageEngine calls "SSL" (https://www.manageengine.com/products/s ... tings.html).

If you have both sides configured this way, ManageEngine is going to connect to hMailServer and negotiate a TLS connection before attempting any SMTP verbs.

I suspect that if you set the ManageEngine side to "TLS" that it is going to attempt to connect to hMailServer over an unencrypted TCP connection expecting to negotiate with the SMTP interface using plaintext and eventially upgrade the connection using the STARTTLS verb. But because hMailSErver is expecting TLS negotiation first, Manage Engine won't see what it expects ("220 something"). Both side will be confused and the connection aborts.

If you want to use the plaintext-then-STARTTLS approach, consider using port 587 instead and configure hMailServer to use "STARTTLS (Optional)" or "STARTTLS (Required)" and ManageEngine to use "TLS".

User avatar
mattg
Moderator
Moderator
Posts: 21454
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: 3rd party application connecting to hMailServer over TLS to port 465 fails

Post by mattg » 2021-05-15 02:07

I see that your hmailserver is suggesting TLSv1.2 in the first set of logs, and that log set looks like what I get when someone tries to connect with lower levels of encryption

You say you opened SSL3.0 - This is broken
Have you ALSO enabled TLSv1.1 or TLSv1.0? They are more likely to work than SSLv3.0

If your application is running on Windows Server 2012R2 or Windows 7 SP1 or older, you actively need to allow TLSv1.2 for TLSv1.2 to work. SSL 3.0 is also disabled by default because it is broken.
viewtopic.php?f=21&t=33149&p=207245&hil ... 24#p207245
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

codlts
New user
New user
Posts: 6
Joined: 2019-12-06 17:51

Re: 3rd party application connecting to hMailServer over TLS to port 465 fails

Post by codlts » 2021-05-15 10:30

Replying to mikedibella...

I would have to reach out to ADSelfService Plus support to see what port their SSL and TLS security connection uses. But what you say does make sense. But I did try using SSL connection in ADSelfService Plus application and once with SSL v3.0 box in SSL/TLS page in hMailServer ticked and without being ticked. But when I use SSL connection from the ADSelfService Plus application with both SSL v3.0 box in hMailServer ticked and then unticked, I get the same error message in the hMailServer log - error code: 336151574, Message: sslv3 alert certificate unknown." When I searched the forum, they say to make sure that the SSL v3.0 box was ticked, which in my case it was.

"DEBUG" 1244 "2021-05-12 18:50:41.305" "Creating session 13"
"TCPIP" 1244 "2021-05-12 18:50:41.305" "TCP - 192.168.123.100 connected to 192.168.123.25:465."
"DEBUG" 1244 "2021-05-12 18:50:41.305" "TCP connection started for session 10"
"DEBUG" 1244 "2021-05-12 18:50:41.305" "Performing SSL/TLS handshake for session 10. Verify certificate: False"
"TCPIP" 1244 "2021-05-12 18:50:41.336" "TCPConnection - TLS/SSL handshake failed. Session Id: 10, Remote IP: 192.168.123.100, Error code: 336151574, Message: sslv3 alert certificate unknown"
"DEBUG" 1244 "2021-05-12 18:50:41.336" "Ending session 10"

Tried port 587 with STARTTLS (Optional) and TLS connection in ADSelfService Plus application:

"DEBUG" 6808 "2021-05-15 03:41:21.391" "Creating session 26"
"TCPIP" 6808 "2021-05-15 03:41:21.391" "TCP - 192.168.123.100 connected to 192.168.123.25:587."
"DEBUG" 6808 "2021-05-15 03:41:21.391" "TCP connection started for session 24"
"SMTPD" 6808 24 "2021-05-15 03:41:21.391" "192.168.123.100" "SENT: 220 mail.mydomain.com ESMTP"
"SMTPD" 7100 24 "2021-05-15 03:41:21.406" "192.168.123.100" "RECEIVED: EHLO server1.mydomain.com"
"SMTPD" 7100 24 "2021-05-15 03:41:21.406" "192.168.123.100" "SENT: 250-mail.mydomain.com[nl]250-SIZE 20480000[nl]250-STARTTLS[nl]250-AUTH LOGIN PLAIN[nl]250 HELP"
"SMTPD" 6820 24 "2021-05-15 03:41:21.406" "192.168.123.100" "RECEIVED: STARTTLS"
"SMTPD" 6820 24 "2021-05-15 03:41:21.406" "192.168.123.100" "SENT: 220 Ready to start TLS"
"DEBUG" 6808 "2021-05-15 03:41:21.406" "Performing SSL/TLS handshake for session 24. Verify certificate: False"
"TCPIP" 7100 "2021-05-15 03:41:21.453" "TCPConnection - TLS/SSL handshake failed. Session Id: 24, Remote IP: 192.168.123.100, Error code: 336151574, Message: sslv3 alert certificate unknown"
"DEBUG" 7100 "2021-05-15 03:41:21.453" "Ending session 24"

Tried port 587 with STARTTLS (Required) and TLS connection in ADSelfService Plus application:

"DEBUG" 5064 "2021-05-15 03:41:42.752" "Creating session 32"
"TCPIP" 5064 "2021-05-15 03:41:42.752" "TCP - 192.168.123.100 connected to 192.168.123.25:587."
"DEBUG" 5064 "2021-05-15 03:41:42.752" "TCP connection started for session 30"
"SMTPD" 5064 30 "2021-05-15 03:41:42.752" "192.168.123.100" "SENT: 220 mail.mydomain.com ESMTP"
"SMTPD" 6980 30 "2021-05-15 03:41:42.752" "192.168.123.100" "RECEIVED: EHLO server1.mydomain.com"
"SMTPD" 6980 30 "2021-05-15 03:41:42.752" "192.168.123.100" "SENT: 250-mail.mydomain.com[nl]250-SIZE 20480000[nl]250-STARTTLS[nl]250-AUTH LOGIN PLAIN[nl]250 HELP"
"SMTPD" 5064 30 "2021-05-15 03:41:42.752" "192.168.123.100" "RECEIVED: STARTTLS"
"SMTPD" 5064 30 "2021-05-15 03:41:42.752" "192.168.123.100" "SENT: 220 Ready to start TLS"
"DEBUG" 5296 "2021-05-15 03:41:42.752" "Performing SSL/TLS handshake for session 30. Verify certificate: False"
"TCPIP" 6980 "2021-05-15 03:41:42.783" "TCPConnection - TLS/SSL handshake failed. Session Id: 30, Remote IP: 192.168.123.100, Error code: 336151574, Message: sslv3 alert certificate unknown"
"DEBUG" 6980 "2021-05-15 03:41:42.783" "Ending session 30"

I am wondering if there is a problem with my SSL certificate on the hMailServer as jim.bus and palinka alluded to in earlier posts.

I followed instructions in this article viewtopic.php?f=21&t=35965&p=226546&hilit=ssl#p226546 to create the CSR and the KEY file on the hMailServer server using OpenSSL. I then took the csr file and submitted to my Axiad CA which created the SSL certificate. It created a p7b file that contained the root, intermediate, and the mailserver certificates. The article called for combining these 3 certificates in this order: mailserver followed by intermediate followed by root certificates to create the final certificate that was used in hMailServer SSL certificates section. Then I configured port 465 and 995 under TCP/IP Ports section using the newly added SSL cert. Next step in the instruction was to test the SSL by running this on the hMailServer: openssl s_client -connect mail.mydomain.com:465.

This is the full output from this test:

Microsoft Windows [Version 10.0.17763.1911]
(c) 2018 Microsoft Corporation. All rights reserved.

C:\Windows\system32>openssl s_client -connect mail.mydomain.com:465
CONNECTED(00000190)
depth=2 O = xxx, CN = yyy Root CA
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=2 O = xxx, CN = yyy Root CA
verify return:1
depth=1 O = xxx, CN = yyy Issuing CA
verify return:1
depth=0 O = xxx, OU = zzz, CN = mail.mydomain.com.com
verify return:1
---
Certificate chain
0 s:O = xxx, OU = zzz, CN = mail.mydomain.com.com
i:O = xxx, CN = yyy Issuing CA
1 s:O = xxx, CN = yyy Root CA
i:O = xxx, CN = yyy Root CA
2 s:O = xxx, CN = yyy Issuing CA
i:O = xxx, CN = yyy Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxEw=
-----END CERTIFICATE-----
subject=O = xxx, OU = zzz, CN = mail.mydomain.com

issuer=O = xxx, CN = yyy Issuing CA

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 4350 bytes and written 456 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: BB5##################################################################3059
Session-ID-ctx:
Master-Key: FFFC###################################################################################3C61
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - a9 39 00 f7 bc 8e 84 b8-da 96 ec 27 e9 3c d3 d6 .9.........'.<..
0010 -
0020 -
0030 -
0040 -
0050 -
0060 -
0070 -
0080 -
0090 -
00a0 -
00b0 - 5c 3c 7c 96 8a 0c 5e 54-c7 56 d7 22 98 7f 53 8d \<|...^T.V."..S.

Start Time: 1620800640
Timeout : 7200 (sec)
Verify return code: 19 (self signed certificate in certificate chain)
Extended master secret: no
---
220 mail.mydomain.com ESMTP
quit
221 goodbye
read:errno=0

When I run the above openssl s_client -connect mail.mydomain.com:465 test, this is what is in the hMailServer log:

"DEBUG" 5936 "2021-05-12 01:51:47.447" "Creating session 62"
"TCPIP" 5936 "2021-05-12 01:51:47.447" "TCP - 192.168.123.25 connected to 192.168.123.25:465."
"DEBUG" 5936 "2021-05-12 01:51:47.447" "TCP connection started for session 61"
"DEBUG" 5936 "2021-05-12 01:51:47.447" "Performing SSL/TLS handshake for session 61. Verify certificate: False"
"TCPIP" 6292 "2021-05-12 01:51:47.478" "TCPConnection - TLS/SSL handshake completed. Session Id: 61, Remote IP: 192.168.123.25, Version: TLSv1.2, Cipher: ECDHE-RSA-AES256-GCM-SHA384, Bits: 256"
"SMTPD" 6292 61 "2021-05-12 01:51:47.478" "192.168.123.25" "SENT: 220 mail.mydomain.com ESMTP"

Doesn't these 2 logs indicate that the SSL certificate on the hMailServer is working?

codlts
New user
New user
Posts: 6
Joined: 2019-12-06 17:51

Re: 3rd party application connecting to hMailServer over TLS to port 465 fails

Post by codlts » 2021-05-15 10:32

Replying to mattg...

Yes, in the SSL / TLS section under Settings --> Advanced, I first enabled TLS v1.0, 1.1, and 1.2 when testing TLS security setting in the ADSelfService Plus application portal. When that didn't work, I also enabled the SSL v3.0 which also didn't work. SSL v3.0 since has been unticked.

ADSelfSerivce Plus application is running on Windows 2012 R2 server. Following your instructions, I added the registry settings for SSL v3.0, TLS v1.0. TLS v1.1, and TLS v1.2. First 3 was set to 1 and TLS v1.2 was set to 0. However, sending the test email from the ADSelfService Plus web portal afterwards failed with the same error message as before:

"DEBUG" 5876 "2021-05-15 02:46:42.204" "Creating session 27"
"TCPIP" 5876 "2021-05-15 02:46:42.204" "TCP - 192.168.123.100 connected to 192.168.123.25:465."
"DEBUG" 5876 "2021-05-15 02:46:42.204" "TCP connection started for session 26"
"DEBUG" 5876 "2021-05-15 02:46:42.204" "Performing SSL/TLS handshake for session 26. Verify certificate: False"
"TCPIP" 5876 "2021-05-15 02:47:12.214" "TCPConnection - TLS/SSL handshake failed. Session Id: 26, Remote IP: 192.168.123.100, Error code: 335544539, Message: short read"
"DEBUG" 5876 "2021-05-15 02:47:12.214" "Ending session 26"

User avatar
mattg
Moderator
Moderator
Posts: 21454
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: 3rd party application connecting to hMailServer over TLS to port 465 fails

Post by mattg » 2021-05-15 10:49

After the registry changes, you need to restart that server (with ADSelfService plus installed)


That software may try to validate certificates for connections that it makes (and not like your self signed cert)- but that is unusual.
I can't see any tech detail on the website for that software.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jim.bus
Senior user
Senior user
Posts: 701
Joined: 2011-05-28 11:49
Location: US

Re: 3rd party application connecting to hMailServer over TLS to port 465 fails

Post by jim.bus » 2021-05-15 11:39

It looks like from what I saw in your posts that you may be creating Self-Signed Certificates. It has been a long time sense I have used Self-Signed Certificates with my hMailServer so I can't remember what all the Log Entries looked like but your 'sslv3 alert certificate unknown' may not have caused your Email Client to fail. The handshake failed but the error indicates that the failure seemed to be that it didn't recognize the Certificate which could merely be because it is a Self-Signed Certificate and therefore there was no 'trust' associated with the Certificate. When I used Outlook with self-signed certificates I would always get a prompt from Outlook asking me if I wanted to trust the Self-Signed Certificate I was using. This may be the same situation at least to the extent that all that happened is the Self-Signed Certificate is untrusted which doesn't mean that the operation failed just that the 'Certificate' wasn't trusted.

Also, you still haven't shown all your Logs so we do not have the complete transaction being shown to us from the Logs so it's impossible to see if the operation totally failed or really completed though it looks like it the email transaction did stop after the handshake failure message was issued as the timestamps are identical but I saw other 'Sent'/'Received' responses all processing with the same timestamp as well so I would prefer to be able to see all the Logs to have as much information as possible. For instance, we are not seeing who ended the hmailServer session (client or hMailServer). All we know is that the session ended.

mikedibella
Senior user
Senior user
Posts: 503
Joined: 2016-12-08 02:21

Re: 3rd party application connecting to hMailServer over TLS to port 465 fails

Post by mikedibella » 2021-05-15 17:01

codlts wrote:
2021-05-15 10:30
I am wondering if there is a problem with my SSL certificate on the hMailServer as jim.bus and palinka alluded to in earlier posts.
Make sure you are sending the intermediate certificates in the CA's chain. To test, change the openssl command to include the -showcerts switch. You need to see n-1 certificate sent from the server, where n is the number of certificates in the chain. The server does not send the the root certificate, but should send the leaf (server) certificate and all intermediate certificates. If there are missing intermediate certificate in the protocol negotiation, the client (ManageEngine) will abort the connection because trust is validated at the root and the intermediates are used to the "walk" the chain from leaf to root.

If you publish the hMailserver port to the Internet, there are also multiple validation sites you can you to test and diagnose certificate issues with TLS interfaces. Can you do that?

codlts
New user
New user
Posts: 6
Joined: 2019-12-06 17:51

Re: 3rd party application connecting to hMailServer over TLS to port 465 fails

Post by codlts » 2021-05-15 19:23

mattg wrote:
2021-05-15 10:49
After the registry changes, you need to restart that server (with ADSelfService plus installed)
Yes, the server has been restarted after the registry changes.

jim.bus wrote:
2021-05-15 11:39
Also, you still haven't shown all your Logs so we do not have the complete transaction being shown to us from the Logs so it's impossible to see if the operation totally failed or really completed though it looks like it the email transaction did stop after the handshake failure message was issued as the timestamps are identical but I saw other 'Sent'/'Received' responses all processing with the same timestamp as well so I would prefer to be able to see all the Logs to have as much information as possible. For instance, we are not seeing who ended the hmailServer session (client or hMailServer). All we know is that the session ended.
Attached is the the entire contents of the log file for today 2021-05-15 when I tried to send couple of test emails from the ADSelfService Plus admin web portal following several of your suggestions. And in the Logging section under Settings, I have ticked Application, SMTP, POP3, IMAP, TCP/IP, Debug, and AWStats boxes.
hMailServerLogs_2021-05-15.zip
(6.25 KiB) Downloaded 7 times

codlts
New user
New user
Posts: 6
Joined: 2019-12-06 17:51

Re: 3rd party application connecting to hMailServer over TLS to port 465 fails

Post by codlts » 2021-05-15 19:24

mikedibella wrote:
2021-05-15 17:01
Make sure you are sending the intermediate certificates in the CA's chain. To test, change the openssl command to include the -showcerts switch. You need to see n-1 certificate sent from the server, where n is the number of certificates in the chain. The server does not send the the root certificate, but should send the leaf (server) certificate and all intermediate certificates. If there are missing intermediate certificate in the protocol negotiation, the client (ManageEngine) will abort the connection because trust is validated at the root and the intermediates are used to the "walk" the chain from leaf to root.
This is the result of the test using -showcerts:
Microsoft Windows [Version 10.0.17763.1935]
(c) 2018 Microsoft Corporation. All rights reserved.

C:\Windows\system32>openssl s_client -showcerts -connect mail.mydomain.com:465
CONNECTED(000001A8)
depth=2 O = xxx, CN = yyy Root CA
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=2 O = xxx, CN = yyy Root CA
verify return:1
depth=1 O = xxx, CN = yyy Issuing CA
verify return:1
depth=0 O = xxx, OU = zzz, CN = mail.mydomain.com
verify return:1
---
Certificate chain
0 s:O = xxx, OU = zzz, CN = mail.mydomain.com
i:O = xxx, CN = yyy Issuing CA
-----BEGIN CERTIFICATE-----
MIIEAxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxy5Ew=
-----END CERTIFICATE-----
1 s:O = AFS COD, CN = AFS TIVOD Root CA
i:O = AFS COD, CN = AFS TIVOD Root CA
-----BEGIN CERTIFICATE-----
MIIFRxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxkxBQw==
-----END CERTIFICATE-----
2 s:O = AFS COD, CN = AFS TIVOD Issuing CA
i:O = AFS COD, CN = AFS TIVOD Root CA
-----BEGIN CERTIFICATE-----
MIIE9xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxx3ziRw==
-----END CERTIFICATE-----
---
Server certificate
subject=O = xxx, OU = zzz, CN = mail.mydomain.com

issuer=O = xxx, CN = yyy Issuing CA

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 4350 bytes and written 456 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 8245########################################################2757
Session-ID-ctx:
Master-Key: 96E5########################################################################################A7A6
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 02 95 22 59 4a a0 7a 84-9c aa ba 4a 10 2d 02 3b .."YJ.z....J.-.;
0010 -
0020 -
0030 -
0040 -
0050 -
0060 -
0070 -
0080 -
0090 -
00a0 -
00b0 - 4f 00 f8 2d 17 44 bf b0-9c 24 82 76 63 03 70 ca O..-.D...$.vc.p.

Start Time: 1621095341
Timeout : 7200 (sec)
Verify return code: 19 (self signed certificate in certificate chain)
Extended master secret: no
---
220 mail.ucpsite.com ESMTP
mikedibella wrote:
2021-05-15 17:01
If you publish the hMailserver port to the Internet, there are also multiple validation sites you can you to test and diagnose certificate issues with TLS interfaces. Can you do that?
I could try this, but would have to wait until I can get our Cisco person to configure our FW, NAT, and internal routing to do this. Any additional ports other than port 465?

mikedibella
Senior user
Senior user
Posts: 503
Joined: 2016-12-08 02:21

Re: 3rd party application connecting to hMailServer over TLS to port 465 fails

Post by mikedibella » 2021-05-15 19:46

OK, the protocol trace above shows that openssl can successfully negotiate an SSL/TLS connection with hMailServer.

Now lets use openssl to test why ManageEngine connections are failing.

Stop the hMailServer service and run the following command on the hMailServer service host:

Code: Select all

openssl s_server -key keyfile -cert certfile -accept 465 -msg | find ">>>"
Make sure you replace keyfile and certfile with path and filename of the private key and cert chain file you are using with hMailserver.

Try to connect from ManageEngine and post the trace output here. A sucessful TLS 1.2 negotiation will look like this:

Code: Select all

>>> TLS 1.2 Handshake [length 0042], ServerHello
>>> TLS 1.2 Handshake [length 05cb], Certificate
>>> TLS 1.2 Handshake [length 01cd], ServerKeyExchange
>>> TLS 1.2 Handshake [length 0004], ServerHelloDone
>>> TLS 1.2 Handshake [length 00aa]???
>>> TLS 1.2 ChangeCipherSpec [length 0001]
>>> TLS 1.2 Handshake [length 0010], Finished

mikedibella
Senior user
Senior user
Posts: 503
Joined: 2016-12-08 02:21

Re: 3rd party application connecting to hMailServer over TLS to port 465 fails

Post by mikedibella » 2021-05-15 20:05

Something else I found:

https://download.manageengine.com/produ ... -guide.pdf

It looks like ADSelfService Plus is built on Java. Java uses its own trust store, not the trust store of the host OS. You may need to manually trust the root certificate of the hMailServer certificate chain. See page 4 of the PDF for procedure.

codlts
New user
New user
Posts: 6
Joined: 2019-12-06 17:51

Re: 3rd party application connecting to hMailServer over TLS to port 465 fails

Post by codlts » 2021-05-15 22:24

mikedibella wrote:
2021-05-15 20:05
Stop the hMailServer service and run the following command on the hMailServer service host:
openssl s_server -key keyfile -cert certfile -accept 465 -msg | find ">>>"
After stopping the hMailServer service, I ran the following command, but after I hit the enter button, it just sits there and there is no output. Is the syntax correct?

openssl s_server -key "C:\Program Files (x86)\hMailServer\CA\mail.mydomain.com.key" -cert "C:\Program Files (x86)\hMailServer\CA\mail.mydomain.com.crt" -accept 465 -msg | find ">>>"
mikedibella wrote:
2021-05-15 20:05
Try to connect from ManageEngine and post the trace output here.
How do I do this?
mikedibella wrote:
2021-05-15 20:05
It looks like ADSelfService Plus is built on Java. Java uses its own trust store, not the trust store of the host OS. You may need to manually trust the root certificate of the hMailServer certificate chain. See page 4 of the PDF for procedure.
This procedure is to set up SSL cert on the ADSelfService Plus web portal so when users to go the web portal to reset their password, they have a valid SSL connection to the web portal. How would I use this to trust the root certificate of the hMailServer certificate? Both hMailServer and ADSelfService Plus certs use the same root and intermediate certificates. So on the host servers for these applications have the root and the intermediate certificates installed.

mikedibella
Senior user
Senior user
Posts: 503
Joined: 2016-12-08 02:21

Re: 3rd party application connecting to hMailServer over TLS to port 465 fails

Post by mikedibella » 2021-05-15 22:49

codlts wrote:
2021-05-15 22:24
After stopping the hMailServer service, I ran the following command, but after I hit the enter button, it just sits there and there is no output. Is the syntax correct?
The command creates a TLS server listening on the -accept port. There will be no output until a connection from a remote client is attempted on that port.
codlts wrote:
2021-05-15 22:24
How do I do this?
Simulate a transaction in the ADSelfService Plus portal that would generate a message?
codlts wrote:
2021-05-15 22:24
Both hMailServer and ADSelfService Plus certs use the same root and intermediate certificates. So on the host servers for these applications have the root and the intermediate certificates installed.
Server processes do not use trust anchors unless they accept client certificates, so not having the root of the server authentication chain in the server's trust store will not cause a trust validation problem for the server. A client, however, needs the trust anchor to validate the server's certificate. In the case of the ADSelfService Plus, it is acting as both a server, for the portal it publishes, and as a client, when it submits SMTP messages for relay. So not having the trust anchor isn't going to impair the portal functionality, but it will impair the SMTP-submission functionality.

Post Reply