[DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
fbd-support
New user
New user
Posts: 15
Joined: 2018-11-19 17:29

[DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Post by fbd-support » 2018-11-19 17:34

Hey guys.

We have been using hMailserver for years on multiple servers without much issue.

Now we have a new server (Windows Server 2016 with MS SQL Server 2017), but we are experiencing issues installing hMailserver on the new server.

We keep getting the following error when the installer starts to install the hMailserver database:
[DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

I've been googling my a** of, but I haven't found any solution for this.

Can anyone tell me how to complete the hMailserver install?

Thank you!
K.

User avatar
Dravion
Senior user
Senior user
Posts: 1466
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Post by Dravion » 2018-11-19 18:48

DBNETLib is the Sybase/Microsoft TDS (Tabular Data Stream) Protocol listener which basicly listen on Port 1433 (if its a Microsoft SQL-Server Database server) or Port 5000 if its a SAP Sybase ASE-SQL-Database server. Make sure Windows Firewall is open,on Port 1433 InBound and also OutBound.

Hint: MS-SQL-Server is a fork of Sybase SQL-Server because both Companys teamed up arround 1990 and go diffrent ways a few Years later.

However:
To use MS-SQL via TCP/IP v4 or v6 you need to enable its Standard Port 1433 on all IPs or on the specific IP you want MS-SQL-Server to listen.
Dont use Dynamic SQL-Server ports and leave it blank or 0 and use a fixed/static port value 1433 for all IPs because SQL likes to randomly change it otherwise and no Client can connect.

As you can see in your Loginfo, there is an SQL
Problem as well. Even on SQL-Server 2017 with its latest updates installed, TLS/SSL encryption isnt activated in the SQL-Server TCP settings. If you switch to "Enforce encryption" any non TLS/SSL Port 1433 (TDS) compliant connection attemp will be refused and you will be informed by Windows Systemlog (Application-Category) about the rejection in full details.

Testwise switch of Enforce encryption and report bach.

ps: If you connect not via TCP/IP and by Instance
for example Localhost\SQLEXPRESS you need to switch on Shared Memory Communication in SQL-Server settings.

ps2: Only SQL-Server 2008 + SP4 installed on top of it and later versions can handle TLS 1.2.

fbd-support
New user
New user
Posts: 15
Joined: 2018-11-19 17:29

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Post by fbd-support » 2018-11-20 08:51

I'm sorry, but that didn't help me much.

Can you be more concrete about what settings I need to change to get this install working?

The SQL server has its external and 127.0.0.1 active and enabled on port 1433.
It works fine for the websites that connect to it.
No dynamic ports are being used.

I can't find any setting to enable or disable encryption on the SQL Server.
Can you explain how to do that?

Thank you.
K.

fbd-support
New user
New user
Posts: 15
Joined: 2018-11-19 17:29

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Post by fbd-support » 2018-11-20 10:06

Enabling TLS 1.0 makes the install succesful.
But when I disable TLS 1.0 again after the installation, the hMailserver stops working again...

I thought TLS 1.0 was obsolete and even leaky?
Why does hMailserver require it?

Thanks!

User avatar
mattg
Moderator
Moderator
Posts: 20224
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Post by mattg » 2018-11-20 10:44

fbd-support wrote:
2018-11-20 10:06
Why does hMailserver require it?
hMailserver doesn't

hMailserver is built with an old library for MS SQL Server so that it is compatible with more operating systems.
You don't have to use MS SQL Server

And seriously, if you have something that detects TLS1.0 between two pieces of software running on the same machine, and is able to crack that level of encryption, then you have bigger issues.

Anyhow, here is some discussion about this >> https://github.com/hmailserver/hmailserver/issues/229
Check comment on 27 July from martinknafve (he is the developer of hMailserver) he shows how to use the provider you need to get TLS1.2 connections for MS SQL Server
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

fbd-support
New user
New user
Posts: 15
Joined: 2018-11-19 17:29

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Post by fbd-support » 2018-11-20 11:42

Doesn't work.
Only enabling TLS 1.0 gets it up and running.

I will (temporarily, I hope) need to have it running like this.
When they create a TLS 1.2 compatible version of hMailserver, I will update it.

Cheers.

User avatar
Dravion
Senior user
Senior user
Posts: 1466
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Post by Dravion » 2018-11-20 11:54

You need to installl the latest Microsoft Native Client which supports TLS 1.2 and not only enabling in the Serversettings.

fbd-support
New user
New user
Posts: 15
Joined: 2018-11-19 17:29

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Post by fbd-support » 2018-11-20 12:06

Ok, latest native client installed.
How do I enable it in the "Serversettings"?

Anyway, I don't think SQL Server is the problem here... it is configured fine.

I think it is hmailserver that only supports TLS 1.0 for SQL connections!

User avatar
Dravion
Senior user
Senior user
Posts: 1466
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Post by Dravion » 2018-11-20 14:19

Take a look on the last post and use the Narca utility to restrict Windows System wide TLS 1.2 which also effects SQL-Server TLS settings.

viewtopic.php?f=21&t=33149

User avatar
mattg
Moderator
Moderator
Posts: 20224
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Post by mattg » 2018-11-20 17:17

fbd-support wrote:
2018-11-20 12:06
Ok, latest native client installed.
How do I enable it in the "Serversettings"?
Did you even read the github link I showed earlier?
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

fbd-support
New user
New user
Posts: 15
Joined: 2018-11-19 17:29

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Post by fbd-support » 2018-11-20 17:20

Of course, I tried everything you suffesred, but nothing worked.
Only enabling TLS 1.0 seems to solve the problem.

User avatar
mattg
Moderator
Moderator
Posts: 20224
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Post by mattg » 2018-11-20 17:52

You will need to be on at least the latest BETA, build 2431
https://www.hmailserver.com/download

Martin wrote
I have added an ini-file-setting which lets you override the provider to be used. Specifically, the provider MSOLEDBSQL supports TLS 1.2, so by setting Provider=MSOLEDBSQL under Database section in hMailServer.ini, TLS 1.2 will be used. For this to work, the provider must also be installed on the machine. I believe the following installs it:

https://www.microsoft.com/en-us/downloa ... x?id=56730
after you install the provider from this link (above)

Then make the changes to your hmailserver.ini


It will look something like this (although I use MySQL not the MS SQL Server)

Code: Select all

[Directories]
ProgramFolder=C:\hMailServer
DataFolder=c:\hMailServer\Data
LogFolder=c:\hMailServer\Logs
TempFolder=C:\hMailServer\Temp
EventFolder=C:\hMailServer\Events
DatabaseFolder=C:\hMailServer\Database
[GUILanguages]
ValidLanguages=english
[Security]
AdministratorPassword=***REMOVED***
[Database]
Type=MYSQL
Username=root
Password=***REMOVED***
PasswordEncryption=1
Port=3306
Server=localhost
Database=hmailserver
Internal=0
Add this last line like

Code: Select all

[Directories]
ProgramFolder=C:\hMailServer
DataFolder=c:\hMailServer\Data
LogFolder=c:\hMailServer\Logs
TempFolder=C:\hMailServer\Temp
EventFolder=C:\hMailServer\Events
DatabaseFolder=C:\hMailServer\Database
[GUILanguages]
ValidLanguages=english
[Security]
AdministratorPassword=***REMOVED***
[Database]
Type=MYSQL
Username=root
Password=***REMOVED***
PasswordEncryption=1
Port=3306
Server=localhost
Database=hmailserver
Internal=0
Provider=MSOLEDBSQL
Then restart your hmailserver, and it should be using TLSv1.2 to connect to your SQL Server
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
Dravion
Senior user
Senior user
Posts: 1466
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Post by Dravion » 2018-11-20 21:43

@Matt

Thats correct, but no guantee MSQL server doesnt downgrade the connection. I recomnend the following steps:

1) Follow your instructions above
2) Set"Enforce encryption" to "true"
(In SQL-Server Configmanager, Networking)
3)Restrict Systemwide TLS 1.2 usage
(Using the NARTAC Utility for this task)

This will make sure that no connection to SQL-Server can be dowgraded or rennegotiated below TLS 1.2 plus allowed Cypher combination and any non allowed connectionattemp will be rejected and logged in the Windows Eventlog.

fbd-support
New user
New user
Posts: 15
Joined: 2018-11-19 17:29

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Post by fbd-support » 2018-11-21 10:57

Sorry to say, but it doesn't work...

Followed all steps.
Installed the Oledb components.
Installed the beta version of hMailserver, added the "Provider" parameter.
The IISCrypto settings were already correct, but I re-applied them anyway.
And I configured the "enforce encryption".

To no avail... it doesn't work, I keep getting the following unless I enable TLS 1.0 in the Windows registry:

---------------------------
hMailServer Administrator
---------------------------
The connection to the database is not available.

ADO: [DBNETLIB][ConnectionOpen (SECCreateCredentials()).]SSL Security error.
---------------------------
OK
---------------------------

I don't have the time to further investigate this, I will leave TLS 1.0 enabled for now and hope that soon a fully compatible hMailserver version is released.
This server needs to be up and running asap.

Cheers.

User avatar
Dravion
Senior user
Senior user
Posts: 1466
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Post by Dravion » 2018-11-21 11:41

Sounds strange.

However:
You can check if your MS-SQL Server 2017 is ready for encrypted connections by using the attached TSQL Utiliy

1) Download and unzip
2) Open a Windows Command prompt (no Admin command prompt)
3) CD into the unzipped Folder
4) Type in tsql -S mssql -U sa -P <your-sql-server-password> -I freetds.conf
5) In TSQL type in: SELECT encrypt_option FROM sys.dm_exec_connections WHERE session_id = @@SPID
6) Press ENTER and type in GO and type ENTER again.

This should look like this
1> SELECT encrypt_option FROM sys.dm_exec_connections WHERE session_id = @@SPID
2> go
encrypt_option
TRUE


If its true, SQL-Server accepts the TLS 1.2 Connections. In this case your Problem is not on the SQL-Server side so we can
focus on hMailServers OLDEDB-Provider and connection string settings and if required a patch can be done.
Attachments
tsql_amd64.7z
(966.06 KiB) Downloaded 55 times

fbd-support
New user
New user
Posts: 15
Joined: 2018-11-19 17:29

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Post by fbd-support » 2018-11-21 11:48

I executed the query using Management Studio against the master database of our SQL server.

The result is:

encrypt_option
TRUE

User avatar
Dravion
Senior user
Senior user
Posts: 1466
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Post by Dravion » 2018-11-21 12:27

Ok, then its on hMailServer side.

I just setup a SQLServer 2017 Express VM and installed the hMailServer BETA B2431 like you did.
I can recreated your Error Situation and can confirm if disabling TLS 1.0 while msoledbsql_18.1.0.0_x64.msi is installed and param
Provider=MSOLEDBSQL in hMailServer.ini is set, hMailServer connection attemps is failing and Windows Eventlog SChannel TLS Errors are logged.
If TLS 1.0 is turned on again, everything works without any error.

I check the code and if possile build a patched hMailServer.exe with TLS 1.2 enabled.

fbd-support
New user
New user
Posts: 15
Joined: 2018-11-19 17:29

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Post by fbd-support » 2018-11-21 12:28

That's awesome!
Thank you very much for this!

User avatar
Dravion
Senior user
Senior user
Posts: 1466
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Post by Dravion » 2018-11-21 13:46

Ok, i created a little patch and i tested it, instructions can be found in the Readme.txt

Use this only for your specific setup

prequesites:
*Microsoft® OLE DB Driver 18 for SQL Server Provider from https://www.microsoft.com/en-us/downloa ... x?id=56730
*Pre-Installed version of hMailServer 5.6.7 - Build 2425 (32-Bit / Stable) or hMailServer 5.6.8 - Build 2431 (BETA) https://www.hmailserver.com/download

1) Backup or rename your existing hMailServer.exe, libeay32.dll and libeay32.dll
2) Replace your existing hMailServer.exe, libeay32.dll and libeay32.dll with the version from the attached archive.

ps: Dont forget in SQL-Server settings to disable "Enforce encryption" for 32-Bit because this is only needed for SSL-Certificates

Detailed instructions can be found in the readme.txt
Attachments
FIX_TLS-1.2_for_hMailServer-5.6.8-BETA-2326.7z
(1.39 MiB) Downloaded 143 times

fbd-support
New user
New user
Posts: 15
Joined: 2018-11-19 17:29

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Post by fbd-support » 2018-11-21 18:19

THANK YOU VERY MUCH!!! IT WORKS!

Will this solution be integrated in the release versions of hMailserver in the future?

Thanks again.
Kind regards.
Kris.

User avatar
Dravion
Senior user
Senior user
Posts: 1466
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Post by Dravion » 2018-11-21 23:51

fbd-support wrote:
2018-11-21 18:19
THANK YOU VERY MUCH!!! IT WORKS!
Will this solution be integrated in the release versions of hMailserver in the future?
Iam glad it works :)

To your question: I will isolate the code changes and provide a patch for Martin (which is the only official Developer with Github commit permissions). So he has to review the patch and decide if it should be part of the official version.This is a bit difficult because Martin isnt verry active lately.

However:
This fix requires also the presence of MSOLEDB Provider Version 18 to work which needs to be part of the hMailServer Installer or Users need to install it manually first.

However 2:
I forked hMailServer and this (and further patches and features) will be part of an alternative, new MSI-Installer which is in development right now.

In the long run, i try to make all Modifications as Patches avaiable for official hMailServer, but any new feature by me or other contributers to this Fork will be integrated in the new MSI-Installer first.

fbd-support
New user
New user
Posts: 15
Joined: 2018-11-19 17:29

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Post by fbd-support » 2018-11-22 09:33

Thank you for all the help!

Post Reply