Page 1 of 1

[DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2018-11-19 17:34
by fbd-support
Hey guys.

We have been using hMailserver for years on multiple servers without much issue.

Now we have a new server (Windows Server 2016 with MS SQL Server 2017), but we are experiencing issues installing hMailserver on the new server.

We keep getting the following error when the installer starts to install the hMailserver database:
[DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

I've been googling my a** of, but I haven't found any solution for this.

Can anyone tell me how to complete the hMailserver install?

Thank you!
K.

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2018-11-19 18:48
by Dravion
DBNETLib is the Sybase/Microsoft TDS (Tabular Data Stream) Protocol listener which basicly listen on Port 1433 (if its a Microsoft SQL-Server Database server) or Port 5000 if its a SAP Sybase ASE-SQL-Database server. Make sure Windows Firewall is open,on Port 1433 InBound and also OutBound.

Hint: MS-SQL-Server is a fork of Sybase SQL-Server because both Companys teamed up arround 1990 and go diffrent ways a few Years later.

However:
To use MS-SQL via TCP/IP v4 or v6 you need to enable its Standard Port 1433 on all IPs or on the specific IP you want MS-SQL-Server to listen.
Dont use Dynamic SQL-Server ports and leave it blank or 0 and use a fixed/static port value 1433 for all IPs because SQL likes to randomly change it otherwise and no Client can connect.

As you can see in your Loginfo, there is an SQL
Problem as well. Even on SQL-Server 2017 with its latest updates installed, TLS/SSL encryption isnt activated in the SQL-Server TCP settings. If you switch to "Enforce encryption" any non TLS/SSL Port 1433 (TDS) compliant connection attemp will be refused and you will be informed by Windows Systemlog (Application-Category) about the rejection in full details.

Testwise switch of Enforce encryption and report bach.

ps: If you connect not via TCP/IP and by Instance
for example Localhost\SQLEXPRESS you need to switch on Shared Memory Communication in SQL-Server settings.

ps2: Only SQL-Server 2008 + SP4 installed on top of it and later versions can handle TLS 1.2.

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2018-11-20 08:51
by fbd-support
I'm sorry, but that didn't help me much.

Can you be more concrete about what settings I need to change to get this install working?

The SQL server has its external and 127.0.0.1 active and enabled on port 1433.
It works fine for the websites that connect to it.
No dynamic ports are being used.

I can't find any setting to enable or disable encryption on the SQL Server.
Can you explain how to do that?

Thank you.
K.

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2018-11-20 10:06
by fbd-support
Enabling TLS 1.0 makes the install succesful.
But when I disable TLS 1.0 again after the installation, the hMailserver stops working again...

I thought TLS 1.0 was obsolete and even leaky?
Why does hMailserver require it?

Thanks!

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2018-11-20 10:44
by mattg
fbd-support wrote:
2018-11-20 10:06
Why does hMailserver require it?
hMailserver doesn't

hMailserver is built with an old library for MS SQL Server so that it is compatible with more operating systems.
You don't have to use MS SQL Server

And seriously, if you have something that detects TLS1.0 between two pieces of software running on the same machine, and is able to crack that level of encryption, then you have bigger issues.

Anyhow, here is some discussion about this >> https://github.com/hmailserver/hmailserver/issues/229
Check comment on 27 July from martinknafve (he is the developer of hMailserver) he shows how to use the provider you need to get TLS1.2 connections for MS SQL Server

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2018-11-20 11:42
by fbd-support
Doesn't work.
Only enabling TLS 1.0 gets it up and running.

I will (temporarily, I hope) need to have it running like this.
When they create a TLS 1.2 compatible version of hMailserver, I will update it.

Cheers.

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2018-11-20 11:54
by Dravion
You need to installl the latest Microsoft Native Client which supports TLS 1.2 and not only enabling in the Serversettings.

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2018-11-20 12:06
by fbd-support
Ok, latest native client installed.
How do I enable it in the "Serversettings"?

Anyway, I don't think SQL Server is the problem here... it is configured fine.

I think it is hmailserver that only supports TLS 1.0 for SQL connections!

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2018-11-20 14:19
by Dravion
Take a look on the last post and use the Narca utility to restrict Windows System wide TLS 1.2 which also effects SQL-Server TLS settings.

viewtopic.php?f=21&t=33149

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2018-11-20 17:17
by mattg
fbd-support wrote:
2018-11-20 12:06
Ok, latest native client installed.
How do I enable it in the "Serversettings"?
Did you even read the github link I showed earlier?

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2018-11-20 17:20
by fbd-support
Of course, I tried everything you suffesred, but nothing worked.
Only enabling TLS 1.0 seems to solve the problem.

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2018-11-20 17:52
by mattg
You will need to be on at least the latest BETA, build 2431
https://www.hmailserver.com/download

Martin wrote
I have added an ini-file-setting which lets you override the provider to be used. Specifically, the provider MSOLEDBSQL supports TLS 1.2, so by setting Provider=MSOLEDBSQL under Database section in hMailServer.ini, TLS 1.2 will be used. For this to work, the provider must also be installed on the machine. I believe the following installs it:

https://www.microsoft.com/en-us/downloa ... x?id=56730
after you install the provider from this link (above)

Then make the changes to your hmailserver.ini


It will look something like this (although I use MySQL not the MS SQL Server)

Code: Select all

[Directories]
ProgramFolder=C:\hMailServer
DataFolder=c:\hMailServer\Data
LogFolder=c:\hMailServer\Logs
TempFolder=C:\hMailServer\Temp
EventFolder=C:\hMailServer\Events
DatabaseFolder=C:\hMailServer\Database
[GUILanguages]
ValidLanguages=english
[Security]
AdministratorPassword=***REMOVED***
[Database]
Type=MYSQL
Username=root
Password=***REMOVED***
PasswordEncryption=1
Port=3306
Server=localhost
Database=hmailserver
Internal=0
Add this last line like

Code: Select all

[Directories]
ProgramFolder=C:\hMailServer
DataFolder=c:\hMailServer\Data
LogFolder=c:\hMailServer\Logs
TempFolder=C:\hMailServer\Temp
EventFolder=C:\hMailServer\Events
DatabaseFolder=C:\hMailServer\Database
[GUILanguages]
ValidLanguages=english
[Security]
AdministratorPassword=***REMOVED***
[Database]
Type=MYSQL
Username=root
Password=***REMOVED***
PasswordEncryption=1
Port=3306
Server=localhost
Database=hmailserver
Internal=0
Provider=MSOLEDBSQL
Then restart your hmailserver, and it should be using TLSv1.2 to connect to your SQL Server

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2018-11-20 21:43
by Dravion
@Matt

Thats correct, but no guantee MSQL server doesnt downgrade the connection. I recomnend the following steps:

1) Follow your instructions above
2) Set"Enforce encryption" to "true"
(In SQL-Server Configmanager, Networking)
3)Restrict Systemwide TLS 1.2 usage
(Using the NARTAC Utility for this task)

This will make sure that no connection to SQL-Server can be dowgraded or rennegotiated below TLS 1.2 plus allowed Cypher combination and any non allowed connectionattemp will be rejected and logged in the Windows Eventlog.

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2018-11-21 10:57
by fbd-support
Sorry to say, but it doesn't work...

Followed all steps.
Installed the Oledb components.
Installed the beta version of hMailserver, added the "Provider" parameter.
The IISCrypto settings were already correct, but I re-applied them anyway.
And I configured the "enforce encryption".

To no avail... it doesn't work, I keep getting the following unless I enable TLS 1.0 in the Windows registry:

---------------------------
hMailServer Administrator
---------------------------
The connection to the database is not available.

ADO: [DBNETLIB][ConnectionOpen (SECCreateCredentials()).]SSL Security error.
---------------------------
OK
---------------------------

I don't have the time to further investigate this, I will leave TLS 1.0 enabled for now and hope that soon a fully compatible hMailserver version is released.
This server needs to be up and running asap.

Cheers.

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2018-11-21 11:41
by Dravion
Sounds strange.

However:
You can check if your MS-SQL Server 2017 is ready for encrypted connections by using the attached TSQL Utiliy

1) Download and unzip
2) Open a Windows Command prompt (no Admin command prompt)
3) CD into the unzipped Folder
4) Type in tsql -S mssql -U sa -P <your-sql-server-password> -I freetds.conf
5) In TSQL type in: SELECT encrypt_option FROM sys.dm_exec_connections WHERE session_id = @@SPID
6) Press ENTER and type in GO and type ENTER again.

This should look like this
1> SELECT encrypt_option FROM sys.dm_exec_connections WHERE session_id = @@SPID
2> go
encrypt_option
TRUE


If its true, SQL-Server accepts the TLS 1.2 Connections. In this case your Problem is not on the SQL-Server side so we can
focus on hMailServers OLDEDB-Provider and connection string settings and if required a patch can be done.

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2018-11-21 11:48
by fbd-support
I executed the query using Management Studio against the master database of our SQL server.

The result is:

encrypt_option
TRUE

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2018-11-21 12:27
by Dravion
Ok, then its on hMailServer side.

I just setup a SQLServer 2017 Express VM and installed the hMailServer BETA B2431 like you did.
I can recreated your Error Situation and can confirm if disabling TLS 1.0 while msoledbsql_18.1.0.0_x64.msi is installed and param
Provider=MSOLEDBSQL in hMailServer.ini is set, hMailServer connection attemps is failing and Windows Eventlog SChannel TLS Errors are logged.
If TLS 1.0 is turned on again, everything works without any error.

I check the code and if possile build a patched hMailServer.exe with TLS 1.2 enabled.

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2018-11-21 12:28
by fbd-support
That's awesome!
Thank you very much for this!

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2018-11-21 13:46
by Dravion
Ok, i created a little patch and i tested it, instructions can be found in the Readme.txt

Use this only for your specific setup

prequesites:
*Microsoft® OLE DB Driver 18 for SQL Server Provider from https://www.microsoft.com/en-us/downloa ... x?id=56730
*Pre-Installed version of hMailServer 5.6.7 - Build 2425 (32-Bit / Stable) or hMailServer 5.6.8 - Build 2431 (BETA) https://www.hmailserver.com/download

1) Backup or rename your existing hMailServer.exe, libeay32.dll and libeay32.dll
2) Replace your existing hMailServer.exe, libeay32.dll and libeay32.dll with the version from the attached archive.

ps: Dont forget in SQL-Server settings to disable "Enforce encryption" for 32-Bit because this is only needed for SSL-Certificates

Detailed instructions can be found in the readme.txt

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2018-11-21 18:19
by fbd-support
THANK YOU VERY MUCH!!! IT WORKS!

Will this solution be integrated in the release versions of hMailserver in the future?

Thanks again.
Kind regards.
Kris.

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2018-11-21 23:51
by Dravion
fbd-support wrote:
2018-11-21 18:19
THANK YOU VERY MUCH!!! IT WORKS!
Will this solution be integrated in the release versions of hMailserver in the future?
Iam glad it works :)

To your question: I will isolate the code changes and provide a patch for Martin (which is the only official Developer with Github commit permissions). So he has to review the patch and decide if it should be part of the official version.This is a bit difficult because Martin isnt verry active lately.

However:
This fix requires also the presence of MSOLEDB Provider Version 18 to work which needs to be part of the hMailServer Installer or Users need to install it manually first.

However 2:
I forked hMailServer and this (and further patches and features) will be part of an alternative, new MSI-Installer which is in development right now.

In the long run, i try to make all Modifications as Patches avaiable for official hMailServer, but any new feature by me or other contributers to this Fork will be integrated in the new MSI-Installer first.

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2018-11-22 09:33
by fbd-support
Thank you for all the help!

Re: [DBNETLIB][ConnectionOpen (SECDoClientHandshake()).]SSL Security error.

Posted: 2020-04-08 12:22
by cvandijk
Is this already integrated? I've installed 5.6.8 - Build 2494 (BETA), and the Microsoft OLE DB Driver, but if I try to open the admin, it just hangs. If I enable TLS 1.0 on the server, it works.

@Dravion is you latest release on https://github.com/Dravion/hmailserver/releases compatible with TLS 1.2?