oApplication.Stop hanging

Use this forum if you have problems with a hMailServer script, such as hMailServer WebAdmin or code in an event handler.
Post Reply
User avatar
ras07
Normal user
Normal user
Posts: 228
Joined: 2010-03-11 08:51

oApplication.Stop hanging

Post by ras07 » 2019-10-28 20:49

My external backup script briefly stops hMS once a night to sync data and dump the database. This generally takes less than 60 seconds, and has been working fine for several years. However in the last week it's hung three times. It's hanging in the oApplication.Stop method. That method never returns, and the Current status in the Status screen of the admin client stays stuck at "Stopping".

I haven't recently changed anything at all in the system. It's running RvdH's custom build 5.6.8-B2437.21, which I've been running for over 3 months with no issues. I'm running on 32-bit Windows 7 (yeah, gotta move off that soon). I don't think it's related to a Windows update since the last one of those was almost 2 weeks before the problem started (although since the problem doesn't happen every time, it's not impossible).

The backup batch file is called by the Task Scheduler, which then calls a VBS script which waits for a lull in activity (by watching the log file) and then temporarily shuts down the server. The log doesn't show anything very useful (it says "Stopping servers..." and then nothing; I'm already running LogLevel=9 in hMailServer.INI).

Any ideas as to what might be going on, or thoughts on how to debug this?

thanks
ras

mikedibella
Normal user
Normal user
Posts: 218
Joined: 2016-12-08 02:21

Re: oApplication.Stop hanging

Post by mikedibella » 2019-10-28 21:52

Download Process Monitor (https://docs.microsoft.com/en-us/sysint ... ds/procmon) and set it up to capture, filtering events to those where Process contains the name of the hMailServer service process (hMailServer.exe). Reproduce the anomaly, stop the capture, and review the events leading up to the when the service hangs stopping.

User avatar
jimimaseye
Moderator
Moderator
Posts: 8680
Joined: 2011-09-08 17:48

Re: oApplication.Stop hanging

Post by jimimaseye » 2019-10-28 23:03

Palinka and i both occasionally had this problem. Mine was due to a windows update that needed to be reversed. I don't remember how Palinka sorted or what his cause was but it was defy telly problematic for him at the time. Maybe he will see this and explain the solution.

[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

palinka
Senior user
Senior user
Posts: 2010
Joined: 2017-09-12 17:57

Re: oApplication.Stop hanging

Post by palinka » 2019-10-29 00:12

There was a long conversation about this. One possible cause was a SYN flood attack. That may or may not have been the cause of my issues. I never really sorted out the why. But to combat the hang, I came up with few scripts to help.

http://hmailserver.com/forum/viewtopic. ... 38#p211738

Link to how I dealt with it. I changed the backup script to hand over stopping hmailserver service to powershell, which then flogs it into submission (a few tries before giving up. There's also a script to start hmailserver. Both have a notification process to let you know if this is happening, because its not every night that there is a hang.

Updated hmstop.ps1:

Code: Select all

$ErrorActionPreference = 'silentlycontinue'
$ServiceName = 'hMailServer'

Function CkSvcMail($Body) {
	$EmailFrom = "user@gmail.com"
	$EmailTo = "1234567890@tmomail.net" 
	$SMTPServer = "smtp.gmail.com" 
	$SMTPAuthUser = "user@gmail.com"
	$SMTPAuthPass = "supersecretpassword"
	$Subject = "Windows Service Failure" 
	$SMTPClient = New-Object Net.Mail.SmtpClient($SmtpServer, 587) 
	$SMTPClient.EnableSsl = $true 
	$SMTPClient.Credentials = New-Object System.Net.NetworkCredential($SMTPAuthUser, $SMTPAuthPass); 
	$SMTPClient.Send($EmailFrom, $EmailTo, $Subject, $Body)
}

Function CkSvcSMS($Body) {
	& cmd.exe /c gammu-smsd-inject -c C:\gammu\bin\smsdrc TEXT 1234567890 -text $Body
}

$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
write-output "$timestamp Now executing powershell hmstop shutdown script."

(get-service $ServiceName).Refresh()
if ((get-service $ServiceName).Status -ne 'RUNNING'){
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer already STOPPED. Nothing to shut down."
	exit
}
else { 
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer running. Preparing to shut down."
}

Stop-Service $ServiceName
# Start-Sleep -seconds 60
(get-service $ServiceName).Refresh()
if ((get-service $ServiceName).Status -ne 'STOPPED'){
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer failed to shut down. Wait 60 seconds and try again."}
else {
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer successfully STOPPED."
	exit
}

Start-Sleep -seconds 60
Stop-Service $ServiceName
(get-service $ServiceName).Refresh()
if ((get-service $ServiceName).Status -ne 'STOPPED'){
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer failed to shut down on 2nd attempt. Wait 60 seconds and try again."}
else {
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer successfully STOPPED."
	exit
}

Start-Sleep -seconds 60
Stop-Service $ServiceName
(get-service $ServiceName).Refresh()
if ((get-service $ServiceName).Status -ne 'STOPPED'){
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer failed to shut down on 3rd attempt. Giving up and sending notification."
	$Body = "ATTENTION! hMailServer backup script (powershell hmstop) could not shut down service after 3 tries. Check status NOW."
	CkSvcSMS $Body
	CkSvcMail $Body
	}
else {
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer successfully STOPPED."
}


Updated hmstart.ps1:

Code: Select all

$ErrorActionPreference = 'silentlycontinue'
$ServiceName = 'hMailServer'

Function CkSvcMail($Body) {
	$EmailFrom = "user@gmail.com"
	$EmailTo = "1234567890@tmomail.net" 
	$SMTPServer = "smtp.gmail.com" 
	$SMTPAuthUser = "user@gmail.com"
	$SMTPAuthPass = "supersecretpassword"
	$Subject = "Windows Service Failure" 
	$SMTPClient = New-Object Net.Mail.SmtpClient($SmtpServer, 587) 
	$SMTPClient.EnableSsl = $true 
	$SMTPClient.Credentials = New-Object System.Net.NetworkCredential($SMTPAuthUser, $SMTPAuthPass); 
	$SMTPClient.Send($EmailFrom, $EmailTo, $Subject, $Body)
}

Function CkSvcSMS($Body) {
	& cmd.exe /c gammu-smsd-inject -c C:\gammu\bin\smsdrc TEXT 1234567890 -text $Body
}

$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
write-output "$timestamp Now executing powershell hmstart startup script."

(get-service $ServiceName).Refresh()
if ((get-service $ServiceName).Status -ne 'STOPPED'){
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer already RUNNING. Nothing to start."
	exit
}
else {
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer not running. Preparing to start service."
}

Start-Service $ServiceName
# Start-Sleep -seconds 60
(get-service $ServiceName).Refresh()
if ((get-service $ServiceName).Status -ne 'RUNNING'){
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer failed to start. Trying again. Wait 60 seconds"}
else {
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer successfully STARTED."
	exit
}

Start-Service $ServiceName
Start-Sleep -seconds 60
(get-service $ServiceName).Refresh()
if ((get-service $ServiceName).Status -ne 'RUNNING'){
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer failed to start on 2nd attempt. Trying again. Wait 60 seconds."}
else {
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer successfully STARTED."
	exit
}

Start-Service $ServiceName
Start-Sleep -seconds 60
(get-service $ServiceName).Refresh()
if ((get-service $ServiceName).Status -ne 'RUNNING'){
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer failed to start on 3rd attempt. Giving up and sending notification."
	$Body = "ATTENTION! hMailServer backup script (powershell hmstart) could not startup service after 3 tries. Check status NOW."
	CkSvcSMS $Body
	CkSvcMail $Body
	}
else {
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer successfully STARTED."
}
I also have this one that monitors hmailserver via telnet. It runs every 5 minutes from task scheduler EXCEPT a predetermined time span reserved for the backup.

Code: Select all

$ErrorActionPreference = 'silentlycontinue'

# USER VARIABLES
$StartTime = (Get-Date 00:20)						 # check telnet start time - sometime after nightly backup ends   
$EndTime = (Get-Date 23:55)							 # check telnet end time - sometime before nightly backup begins  
$Banner = 'mydomain.net'							 # hMailServer smtp banner										  
$ServiceName = 'hMailServer'						 # name of hMailServer service - installation default
$StreamFile = 'C:\scripts\checkstate\telnetout.txt'  # location of test output (temporary file)						  

$Now = Get-Date

Function CkTelnet {
	$Socket = New-Object System.Net.Sockets.TcpClient('localhost', 25)
	If ($Socket)
	{	$Stream = $Socket.GetStream()
		$Writer = New-Object System.IO.StreamWriter($Stream)
		$Buffer = New-Object System.Byte[] 1024
		$Encoding = New-Object System.Text.AsciiEncoding
	ForEach ($Command in $Commands)
		{ $Writer.WriteLine($Command)
		  $Writer.Flush()
		}
	}   
	Start-Sleep -seconds (2)
	$Result = ""
	While($Stream.DataAvailable)
	{	$Read = $Stream.Read($Buffer, 0, 1024)
		$Result += ($Encoding.GetString($Buffer, 0, $Read)) | Out-File $StreamFile 
	}
}

Function CkSvcMail($Body) {
	$EmailFrom = "user@gmail.com"
	$EmailTo = "1234567890@tmomail.net" 
	$SMTPServer = "smtp.gmail.com" 
	$SMTPAuthUser = "user@gmail.com"
	$SMTPAuthPass = "supersecretpassword"
	$Subject = "Windows Service Failure" 
	$SMTPClient = New-Object Net.Mail.SmtpClient($SmtpServer, 587) 
	$SMTPClient.EnableSsl = $true 
	$SMTPClient.Credentials = New-Object System.Net.NetworkCredential($SMTPAuthUser, $SMTPAuthPass); 
	$SMTPClient.Send($EmailFrom, $EmailTo, $Subject, $Body)
}

Function CkSvcSMS($Body) {
	& cmd.exe /c gammu-smsd-inject -c C:\gammu\bin\smsdrc TEXT 1234567890 -text $Body
}

Function RestartRoutine {
	Restart-Service $ServiceName
	$Body = "ATTENTION! $ServiceName service is being RESTARTED due to a fault. Check status NOW!" 
	CkSvcMail $Body
	CkSvcSMS $Body
	# Wait 1 minute after service restart, refresh, then if running, exit, otherwise try restart again
	Start-Sleep -seconds 60
	(get-service $ServiceName).Refresh()
	if ((get-service $ServiceName).Status -ne 'Running'){
		Restart-Service $ServiceName
		$Body = "ATTENTION! $ServiceName service is being RESTARTED due to a fault - 2nd attempt. Check status NOW!" 
		CkSvcMail $Body
		CkSvcSMS $Body
		# Wait 1 minute after service restart, refresh, then if still not running, give up
		Start-Sleep -seconds 60
		(get-service $ServiceName).Refresh()
		if ((get-service $ServiceName).Status -ne 'Running'){
			$Body = "ATTENTION! $ServiceName service could not be started. Check status NOW!" 
			CkSvcMail $Body
			CkSvcSMS $Body
		}
	}
}

Function SecondTry {
	$Body = "ATTENTION! hMailServer is NOT RESPONDING to telnet commands. Check status NOW."
	CkSvcSMS $Body
	CkSvcMail $Body
	Start-Sleep -seconds 60
	# Test telnet connection, if success then exit, if failure then restart service
	CkTelnet
	If (Test-Path $StreamFile){
		$TestResult = Get-Content $StreamFile | Out-String
		Remove-Item -Path $StreamFile
		If ($TestResult -match $Banner){exit}
		Else {RestartRoutine}
	}
	Else {RestartRoutine}
}

# Check if being called during backup period, if yes, exit.
If ($Now -gt $EndTime){exit} 
ElseIf ($Now -lt $StartTime){exit}
Else {
	# Test telnet connection, if success then exit, if failure then wait 1 minute and try again
	CkTelnet
	If (Test-Path $StreamFile){
		$TestResult = Get-Content $StreamFile | Out-String
		Remove-Item -Path $StreamFile
		If ($TestResult -match $Banner){exit}
		Else {SecondTry}
	}
	Else {SecondTry}
}


All these scripts have an SMS function. I use gammu to send SMS. You can delete that stuff.

User avatar
jimimaseye
Moderator
Moderator
Posts: 8680
Joined: 2011-09-08 17:48

Re: oApplication.Stop hanging

Post by jimimaseye » 2019-10-29 01:03

palinka wrote:
2019-10-29 00:12
There was a long conversation about this. One possible cause was a SYN flood attack.
Ah Yes. That was it. I had or too. TCPVIEW (software) will allow these if they are apparent and ffirewall blocking of the offending ip stops the problem.

[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
ras07
Normal user
Normal user
Posts: 228
Joined: 2010-03-11 08:51

Re: oApplication.Stop hanging

Post by ras07 » 2019-10-31 00:10

jimimaseye wrote:
2019-10-29 01:03
palinka wrote:
2019-10-29 00:12
There was a long conversation about this. One possible cause was a SYN flood attack.
Interesting. after happening 3 times in 5 days, it hasn't happened since (and I've tried to trigger it manually many times). So this is a definitely possibility.

ras

User avatar
jimimaseye
Moderator
Moderator
Posts: 8680
Joined: 2011-09-08 17:48

Re: oApplication.Stop hanging

Post by jimimaseye » 2019-10-31 01:10

I found my thread that clarifies it. https://www.hmailserver.com/forum/viewtopic.php?t=33965
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
ras07
Normal user
Normal user
Posts: 228
Joined: 2010-03-11 08:51

Re: oApplication.Stop hanging

Post by ras07 » 2019-10-31 02:05

jimimaseye wrote:
2019-10-31 01:10
I found my thread that clarifies it. https://www.hmailserver.com/forum/viewtopic.php?t=33965
Very educational - thanks! For now I'm going to chalk it up to a SYN flood.

I've got an "Enable SYN cookies" setting in my router that I've never used because I didn't know what it did, but upon further reading, it looks like it's designed for this very thing. I have read about it causing some issues though, so I'm not going to enable it yet, but I'll keep it in the hip pocket.

In the meantime, I've written a simple batch file that runs TCPVCON (command line version of TCPVIEW) every 30 seconds, greps for SYN_RCVD, and logs the results. If I experience another attack it should show up.

ras

palinka
Senior user
Senior user
Posts: 2010
Joined: 2017-09-12 17:57

Re: oApplication.Stop hanging

Post by palinka » 2019-10-31 03:06

ras07 wrote:
2019-10-31 02:05
In the meantime, I've written a simple batch file that runs TCPVCON (command line version of TCPVIEW) every 30 seconds, greps for SYN_RCVD, and logs the results.
Can you share, please? I'm curious. Thanks.

User avatar
ras07
Normal user
Normal user
Posts: 228
Joined: 2010-03-11 08:51

Re: oApplication.Stop hanging

Post by ras07 » 2019-10-31 05:39

palinka wrote:
2019-10-31 03:06
ras07 wrote:
2019-10-31 02:05
In the meantime, I've written a simple batch file that runs TCPVCON (command line version of TCPVIEW) every 30 seconds, greps for SYN_RCVD, and logs the results.
Can you share, please? I'm curious. Thanks.
Sure:

Code: Select all

@echo off
SETLOCAL ENABLEDELAYEDEXPANSION

:loop
tcpvcon -a -c -n 2>nul | grep SYN_RCVD > SYNflood.tmp

FOR /F "tokens=* USEBACKQ" %%F IN (`wc SYNflood.tmp ^| awk "{print $1}"`) DO SET outvar=%%F
if 0%outvar% gtr 3 (
	echo !DATE! !TIME!
	echo !DATE! !TIME! >> SYNflood.log
	type SYNflood.tmp
	type SYNflood.tmp >> SYNflood.log
)
sleep 30
goto loop
This sends output to both the console and to SYNflood.log if there are more than 3 simultaneous half-open TCP requests (connections in SYN_RCVD state).

tcpvcon is part of tcpview from https://docs.microsoft.com/en-us/sysint ... ds/tcpview , and grep, awk, wc and sleep are Windows builds of standard Unix utilities from http://unxutils.sourceforge.net .

I just knocked this together, so no guarantees.

ras

palinka
Senior user
Senior user
Posts: 2010
Joined: 2017-09-12 17:57

Re: oApplication.Stop hanging

Post by palinka » 2019-10-31 10:54

Good idea. The 3+ thing is good. I'm going to incorporate this into my firewall ban project. Thanks.

palinka
Senior user
Senior user
Posts: 2010
Joined: 2017-09-12 17:57

Re: oApplication.Stop hanging

Post by palinka » 2019-10-31 12:59

ras07 wrote:
2019-10-31 05:39
This sends output to both the console and to SYNflood.log if there are more than 3 simultaneous half-open TCP requests (connections in SYN_RCVD state).

tcpvcon is part of tcpview from https://docs.microsoft.com/en-us/sysint ... ds/tcpview , and grep, awk, wc and sleep are Windows builds of standard Unix utilities from http://unxutils.sourceforge.net .

I just knocked this together, so no guarantees.

ras
I've been playing around with a powershell script to check syn_rcvd, but I realized something. My crappy ISP supplied router actually does protect against syn floods, but some get through before getting kicked by the router. Then they disappear. Apparently the router does something similar to your script, which is count the number of syn_rcvd before taking action.

So the question is how does this affect hmailserver? The answer is, it really doesn't EXCEPT when you're trying to shut down hmailserver process because that's the only time there seems to be an issue. Under normal operating conditions, hmailserver handles these things fine. Eventually they time out and go away (if your router does not allow them to continually renew). But when shutting down hmailserver process, hmailserver views them as open connections and wants to wait until they complete instead of killing them.

So I'm not sure what can be done about it. The issue at hand is shutting down hmailserver service during backup. I suppose if you looked for syn_rcvd right before the backup script launches and somehow kick them, then it might work. But how do you kick those connections? By creating a firewall rule right before launching the backup script? Then what happens if some more come in between the time you create the firewall rule and the time you attempt to shut down hmailserver?

I'm not sure there's any way around this. I think your best bet is to have your router do its job and hope for the best. In the meantime, you can try out the powershell scripts I posted for killing hmailserver service. They do work.

palinka
Senior user
Senior user
Posts: 2010
Joined: 2017-09-12 17:57

Re: oApplication.Stop hanging

Post by palinka » 2019-10-31 13:05

If you're interested, this is what I came up with for powershell. Before I decided it was pointless. Might as well post it. I have about an hour into it. :mrgreen:

Code: Select all

### User Variables  #############################
                                                #
$maxSyn             = 10                        #
                                                #
### Email Variables #############################
                                                #
$EmailFrom          = "sender@gmail.com"        #
$EmailTo            = "recipient@mydomain.tld"  #
$SMTPServer         = "smtp.gmail.com"          #
$SMTPAuthUser       = "sender@gmail.com"        #
$SMTPAuthPass       = "supersecretpassword"     #
                                                #
#################################################

Function EmailResults {
	$Subject = "Possible Syn Flood" 
	# $Body = "Possible Syn Flood"
	$Body = (Get-Content -Path $EmailBody | Out-String )
	$SMTPClient = New-Object Net.Mail.SmtpClient($SmtpServer, 587) 
	$SMTPClient.EnableSsl = $true 
	$SMTPClient.Credentials = New-Object System.Net.NetworkCredential($SMTPAuthUser, $SMTPAuthPass); 
	$SMTPClient.Send($EmailFrom, $EmailTo, $Subject, $Body)
}

$tcpvcon = "$PSScriptRoot\Tcpvcon.exe"
$tcpvcsv = "$PSScriptRoot\Tcpvcon.csv"
$synList = "$PSScriptRoot\synList.csv"
$synLog  = "$PSScriptRoot\synLog.csv"
$EmailBody = "$PSScriptRoot\emailbody.txt"

If (Test-Path $tcpvcsv) {Remove-Item -Force -Path $tcpvcsv}
If (Test-Path $synList) {Remove-Item -Force -Path $synList}
If (Test-Path $EmailBody) {Remove-Item -Force -Path $EmailBody}
New-Item $tcpvcsv -value "protocol,process,pid,state,localaddress,remoteaddress`n"
New-Item $synList -value "timestamp,state,remoteaddress`n"
New-Item $EmailBody -value "Possible Syn Flood from the following IPs:`n"
If (-not (Test-Path $synLog)) {New-Item $synLog -value "TimeStamp               State               Remote Address`n"}

$TimeStamp = Get-Date -f G

& cmd /c $tcpvcon -a -c -n hMailServer.exe | Out-File $tcpvcsv -Encoding ASCII -Append

Import-CSV $tcpvcsv | ForEach {
	$RemoteAddress = $_.remoteaddress
	$State = $_.state
	If ($State -eq "SYN_RCVD") {
		Write-Output "$TimeStamp,$State,$RemoteAddress" | Out-File $synList -Encoding ASCII -Append
		Write-Output ("{0,-24}{1,-20}{2}" -f $TimeStamp,$State,$RemoteAddress) | Out-File $synLog -Encoding ASCII -Append
		Write-Output "$TimeStamp - $RemoteAddress" | Out-File $EmailBody -Encoding ASCII -Append
	}
}

[int]$CountSynList = ([Linq.Enumerable]::Count([System.IO.File]::ReadLines("$synList")) - 1)
If ($CountSynList -gt $maxSyn) {EmailResults}

palinka
Senior user
Senior user
Posts: 2010
Joined: 2017-09-12 17:57

Re: oApplication.Stop hanging

Post by palinka » 2019-10-31 19:21

I noticed something else. Sometimes syn flood comes from the same IP, but most of the time it comes from multiple IPs on the same network. So for gits & shiggles, the below script outputs /24 cidr networks that have > $maxSyn connections from the same network. I set it to max 10 before listing it as a nasty net ($NetToBan). Then you can do whatever you want: setup a firewall rule or whatever.

You can see the individual IPs in synLog.csv.

powershell:

Code: Select all

### User Variables  #############################
                                                #
$maxSyn             = 10                        #
                                                #
#################################################

$tcpvcon = "$PSScriptRoot\Tcpvcon.exe"
$tcpvcsv = "$PSScriptRoot\Tcpvcon.csv"
$synList = "$PSScriptRoot\synList.csv"
$synNetList = "$PSScriptRoot\synNetList.csv"
$synLog  = "$PSScriptRoot\synLog.csv"

If (Test-Path $tcpvcsv) {Remove-Item -Force -Path $tcpvcsv}
If (Test-Path $synList) {Remove-Item -Force -Path $synList}
If (Test-Path $synNetList) {Remove-Item -Force -Path $synNetList}
New-Item $tcpvcsv -value "protocol,process,pid,state,localaddress,remoteaddress`n"
New-Item $synList -value "remoteaddress`n"
If (-not (Test-Path $synLog)) {New-Item $synLog -value "TimeStamp               State               Remote Address`n"}

$TimeStamp = Get-Date -f G

& cmd /c $tcpvcon -a -c -n hMailServer.exe | Out-File $tcpvcsv -Encoding ASCII -Append

Import-CSV $tcpvcsv | ForEach {
	$RemoteAddress = $_.remoteaddress
	$State = $_.state
	If ($State -eq "SYN_RCVD") {
		Write-Output $RemoteAddress | Out-File $synList -Encoding ASCII -Append
		Write-Output ("{0,-24}{1,-20}{2}" -f $TimeStamp,$State,$RemoteAddress) | Out-File $synLog -Encoding ASCII -Append
		Write-Output "TimeStamp - $RemoteAddress" | Out-File $EmailBody -Encoding ASCII -Append
	}
}

Get-Content $synList | ForEach {
	$NetIP = $_ -replace '\.[0-9]{1,3}$',''
	Write-Output $NetIP | Out-File $synNetList -Encoding ASCII -Append
}

Get-Content $synNetList | Group-Object | Where {
	$_.Count -gt $maxSyn
	} | ForEach {
	$NetToBan = $_.Name+".0/24"
	Write-Output $NetToBan
}

User avatar
ras07
Normal user
Normal user
Posts: 228
Joined: 2010-03-11 08:51

Re: oApplication.Stop hanging

Post by ras07 » 2019-10-31 21:49

ras07 wrote:
2019-10-31 00:10
Interesting. after happening 3 times in 5 days, it hasn't happened since (and I've tried to trigger it manually many times).
Happened again last night. My script started seeing a SYN flood at around 2am, all from IPs in the 185.90.x.x range. This went on sporadically for ~45 minutes; then it stopped and in a couple minutes I started getting them from 194.187.x.x and 185.40.x.x IPs. (All these IPs geolocate to Italy.) This went on for a couple hours and then stopped; it was quiet for about half an hour and then I started getting them from several different 47.x.x.x IPs (which all geolocate to either Japan or Hong Kong). 45 minutes later this stopped, was quiet for 5 min, and then it started up with a wide range of IPs that mostly resolved to Germany. After a while those stopped, and a few minutes later, the Italian IPs started up again. After a while that stopped, and then I got hit with IPs from France. In there somewhere my backup kicked off and hung the service, but other than that, hMS didn't appear to be affected at all. I restarted the service and hMS has been running fine all day, although the attack has been going on sporadically all day as well.

The pattern is such that they couldn't be unrelated; I'm sure this was all the same attack. But because SYN flood attacks are usually spoofed, there's no way of knowing if the attacker really controls machines in all those countries, or is just spoofing them. I've seen nearly 10,000 unique IPs in the last 12 hours, so I assume the IPs are spoofed, but it's not clear to me why they stick to specific ranges. (Probably backbone routers drop SYN requests that couldn't possibly have originated from the IPs they are claiming, so attackers have to stick to discrete IP blocks; if this is true the attacker probably has one machine in each IP range.)

Some more data: I'm seeing the SYN requests across all hMS ports (25, 465, 587, 993). On a hunch I looked at my (Linux) web server that's behind the same router, and sure enough it's simultaneously getting hammered on ports 80 and 443. (But its operation doesn't seem to be impacted either, so none of these attacks seem particularly effective.)

Incidentally I enabled my router's "Enable SYN cookies" setting, and this didn't appear to do anything at all.

User avatar
ras07
Normal user
Normal user
Posts: 228
Joined: 2010-03-11 08:51

Re: oApplication.Stop hanging

Post by ras07 » 2019-10-31 23:15

palinka wrote:
2019-10-31 12:59
So the question is how does this affect hmailserver? The answer is, it really doesn't EXCEPT when you're trying to shut down hmailserver process because that's the only time there seems to be an issue. Under normal operating conditions, hmailserver handles these things fine. Eventually they time out and go away (if your router does not allow them to continually renew). But when shutting down hmailserver process, hmailserver views them as open connections and wants to wait until they complete instead of killing them.
I'm not sure why shutting down hMS is affected by a SYN flood. I don't think hMS "sees" the half-open connections at all; I imagine there's some kind of interaction with the TCP/IP stack that's getting stuck.

Stopping the service (as opposed to calling oApplication.Stop, which just "pauses" the service but doesn't end it) seems to eventually work, although it does take a while when a SYN flood attack is going on. (I usually see "couldn't stop the service", but it does actually stop.) I may switch to that approach.

I am curious why a SYN flood attack is so ineffective as a DOS attempt.

palinka
Senior user
Senior user
Posts: 2010
Joined: 2017-09-12 17:57

Re: oApplication.Stop hanging

Post by palinka » 2019-11-01 00:53

ras07 wrote:
2019-10-31 23:15
I'm not sure why shutting down hMS is affected by a SYN flood. I don't think hMS "sees" the half-open connections at all; I imagine there's some kind of interaction with the TCP/IP stack that's getting stuck.
I don't think its a coincidence that firewall-blocking those ranges prevents the "hang".

I'm getting the same ranges: 185.36.216.0/22 (Italy) and 194.247.26.0/23 (France) hitting me with open syn requests.

I'm *pretty sure* my router is blocking syn flood attacks - BUT, it does it on a single IP basis. So when you have 1024 IPs randomly sending syn "attacks", the firewall ends up counting the number per IP and decides after a set number (unknown on my router) that its actually an attack. So with 1024 IPs coming and going, some will be determined to be an attack and some will be in the process of an attack and some will be released from blocking "after the attack". So its really a cat and mouse game. The only way to effectively stop it is to ban the entire range. Or... ignore it and use my powershell script to stop hmailserver on nightly backups. :mrgreen:

mikedibella
Normal user
Normal user
Posts: 218
Joined: 2016-12-08 02:21

Re: oApplication.Stop hanging

Post by mikedibella » 2019-11-01 04:11

I'm going to hypothesize that the .Stop method attempts to free all of the worker threads handing comms. This would include graceful closing of open TCP connections. Since the standard way to close a TCP connection is to send a FIN to the remote and wait for the the remote to reply with the FIN, if the remote is bogus the second FIN will never come that the server must wait the timeout period, which may be a while. So .Stop just hangs in this zombie state.

You may be able to force these connections to close using a utility such as CurrPorts by NirSoft (http://www.nirsoft.net/utils/cports.html). It has a command line option to close all open connections by process name (i.e. currports /close * * * * hmailserver.exe).

User avatar
ras07
Normal user
Normal user
Posts: 228
Joined: 2010-03-11 08:51

Re: oApplication.Stop hanging

Post by ras07 » 2019-11-01 04:25

palinka wrote:
2019-11-01 00:53
I don't think its a coincidence that firewall-blocking those ranges prevents the "hang".
Certainly; didn't mean to imply otherwise - my experience definitely confirms that. It's just that I don't think hMS itself does anything, or even knows anything, about half-open connections.

I imagine mikedibella's hypothesis is on the right track. (Although I'm still not sure the worker threads even have a TCP connection to close, because the TCP/IP stack hasn't finished establishing the connection yet - I envision it more like the app is trying to de-register the worker threads as handler functions, but the stack is waiting for the connection to finish handshaking so it can close it, and won't return from the de-registration call.)

Anyway, it's really curious that we're both getting attacks from the same IP ranges. What did we do to piss that guy off? :roll:

ras

palinka
Senior user
Senior user
Posts: 2010
Joined: 2017-09-12 17:57

Re: oApplication.Stop hanging

Post by palinka » 2019-11-01 04:32

ras07 wrote:
2019-11-01 04:25
Anyway, it's really curious that we're both getting attacks from the same IP ranges. What did we do to piss that guy off? :roll:

ras
They seem to be hitting the entire internet.

User avatar
ras07
Normal user
Normal user
Posts: 228
Joined: 2010-03-11 08:51

Re: oApplication.Stop hanging

Post by ras07 » 2019-11-01 04:47

palinka wrote:
2019-11-01 00:53
I'm *pretty sure* my router is blocking syn flood attacks - BUT, it does it on a single IP basis. So when you have 1024 IPs randomly sending syn "attacks", the firewall ends up counting the number per IP and decides after a set number (unknown on my router) that its actually an attack. So with 1024 IPs coming and going, some will be determined to be an attack and some will be in the process of an attack and some will be released from blocking "after the attack". So its really a cat and mouse game. The only way to effectively stop it is to ban the entire range.
Important to realize that the IP addresses you're seeing almost certainly aren't real. The point of the SYN flood is to swamp your computer without also swamping the attacking computer, so the attacking machine spoofs the IP so that it doesn't have to process the return handshake. So 1.2.3.4 says "Hey, I'm 1.2.3.97 initiating a handshake" (SYN), and your computer sends a response to 1.2.3.97 saying "OK, I'm ready to establish a connection" (SYN-ACK), and 1.2.3.79 then says (to itself) "I didn't ask, go away", and ignores you. The advantage to the attacker is that it can send hundreds of SYN requests and never has to process a single ACK, because they're all going to the wrong computers.

In fact, you could probably analyze an extended SYN flood and figure out the attacker's IP, because it will be the ONE IP address in the whole range that you DON'T see! (Not that this information would do you any good when it comes to blocking the attack.) :|

I was puzzled by the fact that the spoofed IPs are all in a narrow block. Why not just send a totally random IP every time? But I imagine most of the intermediate routers are smart enough to say "hang on, every legit IP behind me is in the 1.2.3.x block, so a packet that says it's from 5.6.7.8 is bogus; I'll just drop it". So the attacker always spoofs an IP from the same subnet.

User avatar
ras07
Normal user
Normal user
Posts: 228
Joined: 2010-03-11 08:51

Re: oApplication.Stop hanging

Post by ras07 » 2019-11-01 04:51

palinka wrote:
2019-11-01 04:32
ras07 wrote:
2019-11-01 04:25
Anyway, it's really curious that we're both getting attacks from the same IP ranges. What did we do to piss that guy off? :roll:
They seem to be hitting the entire internet.
To what end, I wonder? It's a very old attack, and it appears that most modern equipment mostly mitigates it (our hMS hang notwithstanding). Seems like trying to slow down public transportation by taking away everyone's buggy whips.

(A distributed SYN flood might be another story. But this doesn't appear to be that.)

palinka
Senior user
Senior user
Posts: 2010
Joined: 2017-09-12 17:57

Re: oApplication.Stop hanging

Post by palinka » 2019-11-01 12:42

ras07 wrote:
2019-11-01 04:47
In fact, you could probably analyze an extended SYN flood and figure out the attacker's IP, because it will be the ONE IP address in the whole range that you DON'T see! (Not that this information would do you any good when it comes to blocking the attack.) :|
That's doable. My toying around script has that information almost. I'll see if I can figure that out. Should be simple. Collect, sort, which ones are missing. It might take a few days, but it's doable.

User avatar
ras07
Normal user
Normal user
Posts: 228
Joined: 2010-03-11 08:51

Re: oApplication.Stop hanging

Post by ras07 » 2019-11-01 18:42

palinka wrote:
2019-11-01 12:42
ras07 wrote:
2019-11-01 04:47
In fact, you could probably analyze an extended SYN flood and figure out the attacker's IP, because it will be the ONE IP address in the whole range that you DON'T see! (Not that this information would do you any good when it comes to blocking the attack.) :|
That's doable. My toying around script has that information almost. I'll see if I can figure that out. Should be simple. Collect, sort, which ones are missing. It might take a few days, but it's doable.
I tried it; I actually found every possible IP in the block (including x.y.z.0 and x.y.z.255, which, for a /24 block, wouldn't even be legit IPs). So I'm guessing the attack script just randomizes the last octet and calls it a day.

palinka
Senior user
Senior user
Posts: 2010
Joined: 2017-09-12 17:57

Re: oApplication.Stop hanging

Post by palinka » 2019-11-01 18:49

ras07 wrote:
2019-11-01 18:42
palinka wrote:
2019-11-01 12:42
ras07 wrote:
2019-11-01 04:47
In fact, you could probably analyze an extended SYN flood and figure out the attacker's IP, because it will be the ONE IP address in the whole range that you DON'T see! (Not that this information would do you any good when it comes to blocking the attack.) :|
That's doable. My toying around script has that information almost. I'll see if I can figure that out. Should be simple. Collect, sort, which ones are missing. It might take a few days, but it's doable.
I tried it; I actually found every possible IP in the block (including x.y.z.0 and x.y.z.255, which, for a /24 block, wouldn't even be legit IPs). So I'm guessing the attack script just randomizes the last octet and calls it a day.
Ok. :(

I guess it's just as well. They don't really bother much except for this oapp.stop issue while backing up, for which there is a workaround. Too bad. I was looking forward to incorporating it into my firewall ban project. :mrgreen:

But now I have an excuse to upgrade my router from the simpleton one I got from my cable company. :D

palinka
Senior user
Senior user
Posts: 2010
Joined: 2017-09-12 17:57

Re: oApplication.Stop hanging

Post by palinka » 2019-11-01 19:26

http://infogalactic.com/info/SYN_cookies
Drawbacks

The use of SYN cookies does not break any protocol specifications, and therefore should be compatible with all TCP implementations. There are, however, two caveats that take effect when SYN cookies are in use. Firstly, the server is limited to only 8 unique MSS values, as that is all that can be encoded in 3 bits. Secondly, the server must reject all TCP options (such as large windows or timestamps), because the server discards the SYN queue entry where that information would otherwise be stored. [1]

While these restrictions necessarily lead to a sub-optimal experience, their effect is rarely noticed by clients because they are only applied when under attack. In such a situation, the loss of the TCP options in order to save the connection is usually considered to be a reasonable compromise.

A problem arises when the connection-finalizing ACK packet sent by the client is lost, and the application layer protocol requires the server to speak first (SMTP and SSH are two examples). In this case, the client assumes that the connection was established successfully and waits for the server to send its protocol banner, or resend the SYN+ACK packet; however, the server is not aware of the session and will not resend the SYN+ACK because it discarded the backlog queue entry that would enable it to do so. Eventually, the client will abort the connection due to an application layer timeout, but this may take a relatively long time.[2]

Version 2.6.26 of the Linux kernel added limited support of TCP options, by encoding them into the timestamp.[3]

The newer TCP Cookie Transactions (TCPCT) standard is designed to overcome these shortcomings of SYN cookies and improve it on a couple of aspects. Unlike SYN cookies, TCPCT is a TCP extension and requires support from both endpoints.

Security considerations

Simple firewalls that are configured to allow all outgoing connections but to restrict which ports an incoming connection can reach (for example, allow incoming connections to a Web server on port 80 but restrict all other ports), work by blocking only incoming SYN requests to unwanted ports. If SYN cookies are in operation, care should be taken to ensure an attacker is not able to bypass such a firewall by forging ACKs instead, trying random sequence numbers until one is accepted. SYN cookies should be switched on and off on a per-port basis, so that SYN cookies being enabled on a public port does not cause them to be recognised on a non-public port.[4]
Looks like a) syn flood filtering at the router still won't have much of an effect on the oapp.stop issue, and b) the why behind this stupid attack is occuring is some method of port scanning and getting to the app behind the port. Syn flood filtering should still be done for the second reason.

User avatar
ras07
Normal user
Normal user
Posts: 228
Joined: 2010-03-11 08:51

Re: oApplication.Stop hanging

Post by ras07 » 2019-11-02 00:05

palinka wrote:
2019-11-01 18:49
ras07 wrote:
2019-11-01 18:42
I tried it; I actually found every possible IP in the block (including x.y.z.0 and x.y.z.255, which, for a /24 block, wouldn't even be legit IPs). So I'm guessing the attack script just randomizes the last octet and calls it a day.
Too bad. I was looking forward to incorporating it into my firewall ban project. :mrgreen:
Wouldn't have helped; the router still thinks it came from the spoofed IP, so knowing the actual IP wouldn't do any good. (Unless you could geolocate the IP to a specific address and show up at their house with a baseball bat :lol: )
But now I have an excuse to upgrade my router from the simpleton one I got from my cable company. :D
I don't know of any routers that protect against SYN flood attacks (probably expensive managed commercial ones do). But if you have shell access to your router (for instance, if you've flashed it with DD-WRT or Tomato) I'm working on a script to detect a SYN flood from a Windows or Linux box and then push a script to the router to update the iptables and block it.

User avatar
ras07
Normal user
Normal user
Posts: 228
Joined: 2010-03-11 08:51

Re: oApplication.Stop hanging

Post by ras07 » 2019-11-02 00:15

palinka wrote:
2019-11-01 19:26
http://infogalactic.com/info/SYN_cookies
The use of SYN cookies ...
Actually, SYN cookies don't have anything to do with SYN flood attacks. I was confused on this point. I learned that SYN cookies help mitigate against IP spoofing in other instances, but they don't do anything to mitigate against SYN flood attacks.
the why behind this stupid attack is occuring is some method of port scanning and getting to the app behind the port. Syn flood filtering should still be done for the second reason.
I think it's really just a (rather lame) denial-of-service attempt. I don't think SYN flooding can't really do much else.

It's weird that it was so widespread. I also manage a small web site for a nonprofit; it has nothing whatsoever to do with my mail server (different IPs in different cities on different provider networks with different OSs) and yet it was being hit with the same kind of SYN floods from (roughly) the same subnets at (roughly) the same time. (I did notice that the web site performance was worse when the flood was going on, so it's not completely ineffective.)

But after 3 days of near-constant SYN floods, it's been quiet for over 5 hours ... maybe the script kiddies are taking the weekend off.

Post Reply