Check Service Status, Take Action, Notify

This section contains scripts that hMailServer has contributed with. hMailServer 5 is needed to use these.
Post Reply
palinka
Senior user
Senior user
Posts: 957
Joined: 2017-09-12 17:57

Check Service Status, Take Action, Notify

Post by palinka » 2019-02-24 18:24

POWERSHELL script to check service status, take action and notify.

The script basically uses telnet to check port 25. I didn't bother with the telnet commands because what I discovered was that if hmailserver is either STOPPED or PAUSED, the return value will be a timeout (actively refused connection on ip.ad.dr.ess:25). I figure this way, it doesn't matter what caused the malfunction because we're looking for the outcome to tell us that something is wrong.

It checks status 3 times with 5 minutes inbetween before taking action. If after the third try, its still not possible to communicate with port 25, it will attempt to restart the service and also send a notification. Then it will wait 60 seconds and check service status. If the service is not running, it will send another notification.

Obviously, if there is a problem with hmailserver, you won't be able to send an email via hmailserver, so notifications will be sent via outside smtp service. I tested it with gmail. I also send notifications via my sms gateway. Just delete that part if you don't want to use it.

Run from task scheduler.

hMSStatusActionNotify.ps1

Code: Select all

$ErrorActionPreference = 'silentlycontinue'
$remotehost = "localhost" 
$port = 25 
$ServiceName = 'hMailServer'
$EmailFrom = "user@gmail.com"
$EmailTo = "user@gmail.com" 
$SMTPServer = "smtp.gmail.com" 
$SMTPAuthUser = "user@gmail.com"
$SMTPAuthPass = "TopSecretPassword"

# Open telnet
$socket = new-object System.Net.Sockets.TcpClient($remoteHost, $port) 

# If connection refused, wait X seconds and try again
if($socket -eq $null) { 
	Start-Sleep -seconds 300
	$socket = new-object System.Net.Sockets.TcpClient($remoteHost, $port) 

	# If connection refused, wait X seconds and try again
	if($socket -eq $null) { 
		Start-Sleep -seconds 300
		$socket = new-object System.Net.Sockets.TcpClient($remoteHost, $port) 

		# If connection refused, restart HMS service and send notification about restart
		if($socket -eq $null) { 
			Restart-Service $ServiceName
			# Send Notification via Gmail
				$Subject = "Windows Service Failure" 
				$Body = "ATTENTION! MyServer reporting that the hMailServer service is being RESTARTED due to a fault. Check status NOW." 
				$SMTPClient = New-Object Net.Mail.SmtpClient($SmtpServer, 587) 
				$SMTPClient.EnableSsl = $true 
				$SMTPClient.Credentials = New-Object System.Net.NetworkCredential($SMTPAuthUser, $SMTPAuthPass); 
				$SMTPClient.Send($EmailFrom, $EmailTo, $Subject, $Body)
			# Send Notification via SMS
				$uri = "https://smsgateway/index.php/plugin/rest_api/send_sms?phoneNumber=1234567890&message=ATTENTION! hMailServer service is being RESTARTED due to a fault. Check status NOW." 
				$response = Invoke-RestMethod -Uri $uri

			# Wait X seconds, check status of service - if not running, give up trying and send notification re service fault
			Start-Sleep -seconds 60
			(get-service $ServiceName).Refresh()
			if ((get-service $ServiceName).Status -ne 'Running')
			{
			# Send Notification via Gmail
				$Subject = "Windows Service Failure" 
				$Body = "ATTENTION! MyServer reporting that the hMailServer service is not running. Check status NOW." 
				$SMTPClient = New-Object Net.Mail.SmtpClient($SmtpServer, 587) 
				$SMTPClient.EnableSsl = $true 
				$SMTPClient.Credentials = New-Object System.Net.NetworkCredential($SMTPAuthUser, $SMTPAuthPass); 
				$SMTPClient.Send($EmailFrom, $EmailTo, $Subject, $Body)
			# Send Notification via SMS
				$uri = "https://smsgateway/index.php/plugin/rest_api/send_sms?phoneNumber=1234567890&message=ATTENTION! MyServer reporting that the hMailServer service is not running. Check status NOW." 
				$response = Invoke-RestMethod -Uri $uri
			}
		}
	}
}

palinka
Senior user
Senior user
Posts: 957
Joined: 2017-09-12 17:57

Re: Check Service Status, Take Action, Notify

Post by palinka » 2019-03-25 01:17

Minor changes to address error in (get-service $ServiceName).Refresh() plus a little cleanup.

Code: Select all

$ErrorActionPreference = 'silentlycontinue'
$remotehost = "localhost" 
$port = 25 
$ServiceName = 'hMailServer'
$EmailFrom = "notify@gmail.com"
$EmailTo = "1234567890@tmomail.net" 
$SMTPServer = "smtp.gmail.com" 
$SMTPAuthUser = "notify@gmail.com"
$SMTPAuthPass = "SuperSecretPassword"

# check telnet for communication
$socket = new-object System.Net.Sockets.TcpClient($remoteHost, $port) 

# if no communication, send email warning
if($socket -eq $null) { 
# Send Notification via Gmail
	$Subject = "Windows Service Failure" 
	$Body = "ATTENTION! hMailServer service is being RESTARTED due to a fault. Check status NOW." 
	$SMTPClient = New-Object Net.Mail.SmtpClient($SmtpServer, 587) 
	$SMTPClient.EnableSsl = $true 
	$SMTPClient.Credentials = New-Object System.Net.NetworkCredential($SMTPAuthUser, $SMTPAuthPass); 
	$SMTPClient.Send($EmailFrom, $EmailTo, $Subject, $Body)

# wait 5 minutes
	Start-Sleep -seconds 300

# check telnet communication again to make sure its not a temporary network error
	$socket = new-object System.Net.Sockets.TcpClient($remoteHost, $port) 

# if no communication, send email warning
	if($socket -eq $null) { 
	# Send Notification via Gmail
		$Subject = "Windows Service Failure" 
		$Body = "ATTENTION! hMailServer service is being RESTARTED due to a fault. Check status NOW." 
		$SMTPClient = New-Object Net.Mail.SmtpClient($SmtpServer, 587) 
		$SMTPClient.EnableSsl = $true 
		$SMTPClient.Credentials = New-Object System.Net.NetworkCredential($SMTPAuthUser, $SMTPAuthPass); 
		$SMTPClient.Send($EmailFrom, $EmailTo, $Subject, $Body)
		Start-Sleep -seconds 300

# check telnet communication one last time to make sure its not a temporary network error
		$socket = new-object System.Net.Sockets.TcpClient($remoteHost, $port) 
		if($socket -eq $null) { 

# now restart hmailserver service
			Restart-Service $ServiceName
			# Send Notification via Gmail
				$Subject = "Windows Service Failure" 
				$Body = "ATTENTION! hMailServer service is being RESTARTED due to a fault. Check status NOW." 
				$SMTPClient = New-Object Net.Mail.SmtpClient($SmtpServer, 587) 
				$SMTPClient.EnableSsl = $true 
				$SMTPClient.Credentials = New-Object System.Net.NetworkCredential($SMTPAuthUser, $SMTPAuthPass); 
				$SMTPClient.Send($EmailFrom, $EmailTo, $Subject, $Body)

# wait 1 minute
			Start-Sleep -seconds 60

# refresh service status
			(get-service $ServiceName).Refresh()

# if service still not running, send notification
			if ((get-service $ServiceName).Status -ne 'Running')
			{
			# Send Notification via Gmail
				$Subject = "Windows Service Failure" 
				$Body = "ATTENTION! hMailServer service failed to restart and is not running. Check status NOW." 
				$SMTPClient = New-Object Net.Mail.SmtpClient($SmtpServer, 587) 
				$SMTPClient.EnableSsl = $true 
				$SMTPClient.Credentials = New-Object System.Net.NetworkCredential($SMTPAuthUser, $SMTPAuthPass); 
				$SMTPClient.Send($EmailFrom, $EmailTo, $Subject, $Body)
			}
		}
	}
}


User avatar
katip
Senior user
Senior user
Posts: 671
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Check Service Status, Take Action, Notify

Post by katip » 2019-03-25 08:40

Thanks, looks like a useful script.
There is a small tool here (free & portable):
http://www.den4b.com/products/shutter
once it helped me to restart a server which randomly lost network connection every few days (never been able to find out what the problem was), if ping fails.
it does a number of other things.
Katip
--
HMS 5.7.0-B2428-LTS-64-bit, MySQL 5.7.24, SA 3.4.2, ClamAV 0.101.2 + SaneS

palinka
Senior user
Senior user
Posts: 957
Joined: 2017-09-12 17:57

Re: Check Service Status, Take Action, Notify

Post by palinka » 2019-03-27 11:10

katip wrote:
2019-03-25 08:40
Thanks, looks like a useful script.
There is a small tool here (free & portable):
http://www.den4b.com/products/shutter
once it helped me to restart a server which randomly lost network connection every few days (never been able to find out what the problem was), if ping fails.
it does a number of other things.
Cool. I just looked. Winamp LOL. I had a winamp radio station a long time ago. All Johnny Cash all the time. He has such a YUUUGE discography you could listen for a week without hearing the same song. :mrgreen:

I actually have a version of this script for all the major services on my server. They run every 15 minutes. This one for hmailserver is different. I run it only once a day at 12:15 am, 15 minutes after my hmailserver backup script (Jimi's backup & clear down).

I have a peculiar issue where sometimes hmailserver hangs on shutdown. So checking for service status = not running doesn't work because the service is running, but it's not working. That's why instead i check for communication by telnet. I found a powershell script that communicates by telnet. I stripped out all the unnecessary stuff so it just looks to make a connection. If no communication then it means hmailserver got hung.

In the version that gets run every 15 minutes with all the other services, it simply notifies me without taking any action (it looks for service status, not telnet). I set it up that way so it doesn't attempt to restart during a backup. And anyway, hmailserver is so rock solid, if it weren't for this one single issue that only happens during backup, i really wouldn't need to check anything at all. That one has never been triggered in the few months I've been using it. I doubt it will ever be triggered.

But I'll have a closer look at the one you posted. The ping thing could be very useful.

Post Reply