HOW TO: Ready-To-Go Backup and Cleardown script

This section contains user-submitted tutorials.
palinka
Senior user
Senior user
Posts: 1262
Joined: 2017-09-12 17:57

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by palinka » 2019-02-19 01:14

I found a simple service notification powershell script. I commented out the bit with attempting to restart the service. Email bits are commented out too as I'm using http get to trigger the sms message. It works. Run at startup from windows scheduler.
Last edited by mattg on 2019-02-20 02:48, edited 1 time in total.
Reason: code removed as per poster's request. Modified code below.

User avatar
mattg
Moderator
Moderator
Posts: 20272
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by mattg » 2019-02-19 01:19

I get these shutdown issues almost every time I try and 'Pause' hmailserver or stop the service
That the main reason that I don't shutdown or pause to run backups, I just run backups live.

I suspect that it is because some SMTP connections are not closed properly.

RvdH shared some code saved as disconnect.exe >> viewtopic.php?f=20&t=32739&p=206184#p206167

I've used it a bit in my eventhandlers.vbs but I haven't been logging the use.
I'll do that for a while and see if that helps me...
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
mattg
Moderator
Moderator
Posts: 20272
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by mattg » 2019-02-19 01:26

palinka wrote:
2019-02-18 01:22
mattg wrote:
2019-02-18 01:15
palinka wrote:
2019-02-17 19:25
... is set up a SMS gateway, which I did last night using a spare SIM I have with my mobile plan and gammu/kalkun as the gateway.
What hardware did you use to achieve this??
I bought the cheapest usb modem i could find on Amazon. $10 huawei e352. Getting gammu to work was kind of a pain since the project is kind of old and windows was a side thought. You'll have many more options on linux. Or you can jump through hoops like me. :mrgreen:

You can use an old mobile phone too. I considered that too (free) but i tried it with an android app a long time ago and it didn't work well and caused the phone to crash after a day or 2. I think it's better to have computer hardware.
My issue is that I run hMailserver in a HyperV VM, and HyperV doesn't allow access the USB ports of the bare metal server. I do have a network device with a SIM card, a Netgear LTE Modem LB2120, but I am yet to work out how to send SMS from a network attached device via a script. I can do it from the GUI of the device, so I'm sure it will be possible...
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

palinka
Senior user
Senior user
Posts: 1262
Joined: 2017-09-12 17:57

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by palinka » 2019-02-19 01:40

mattg wrote:
2019-02-19 01:26
palinka wrote:
2019-02-18 01:22
mattg wrote:
2019-02-18 01:15

What hardware did you use to achieve this??
I bought the cheapest usb modem i could find on Amazon. $10 huawei e352. Getting gammu to work was kind of a pain since the project is kind of old and windows was a side thought. You'll have many more options on linux. Or you can jump through hoops like me. :mrgreen:

You can use an old mobile phone too. I considered that too (free) but i tried it with an android app a long time ago and it didn't work well and caused the phone to crash after a day or 2. I think it's better to have computer hardware.
My issue is that I run hMailserver in a HyperV VM, and HyperV doesn't allow access the USB ports of the bare metal server. I do have a network device with a SIM card, a Netgear LTE Modem LB2120, but I am yet to work out how to send SMS from a network attached device via a script. I can do it from the GUI of the device, so I'm sure it will be possible...
Http get

https: //yoursmsgateway.tld/index.php/plugin/rest_api/send_sms?phoneNumber=1234567890&message=whatever message you want. Including punctuation and spaces! Yay!

I put a space after https: so it wouldn't break up the url that has spaces in it in the message section.

I'm using kalkun as the front end for gammu. Gammu has a service ans kalkun is a php web app. But like i said, they're both old and there are better options for linux, but they do basically the same thing.

palinka
Senior user
Senior user
Posts: 1262
Joined: 2017-09-12 17:57

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by palinka » 2019-02-19 12:58

I worked out a way easier powershell script for checking service status. This is notify only. The other one i posted doesn't work. I jumped the gun on that one. This one is thoroughly tested. Maybe a mod can delete the other one.

Run from task scheduler. I set mine to run every hour so i don't get spammed if something breaks while I'm sleeping.

Also, this only works on one service, so i made one for each of the services i want to monitor and put them all into a single task in scheduler.

Delete the sms bit if you don't use it or use the api url from your own gateway. By the way, this throws an error because even though it works to post the message, it's a get command that is looking for a response but my gateway gives a 404 as response. Still working that out. Doesn't affect operation, but the run status in scheduler is 0x1 because of the 404 error. Need to figure out either straight post or ignore error in powershell.

Code: Select all

if((get-service hMailServer).Status -eq 'Running'){exit 1}
else {
# Send the problem services via email
	$EmailFrom = "notify@domain.net"
	$EmailTo = "admin@domain.com" 
	$Subject = "Windows Service Failure" 
	$Body = "ATTENTION! Server reporting that the hMailServer service is not running. Check status NOW." 
	$SMTPServer = "localhost" 
	$SMTPClient = New-Object Net.Mail.SmtpClient($SmtpServer, 587) 
	$SMTPClient.EnableSsl = $false 
	$SMTPClient.Credentials = New-Object System.Net.NetworkCredential("notify@domain.net", "SecretPassword"); 
	$SMTPClient.Send($EmailFrom, $EmailTo, $Subject, $Body)

# Send the problem services via SMS
	$uri = "https://mysmsgateway/sms/index.php/plugin/rest_api/send_sms?phoneNumber=1234567890&message=ATTENTION! Server reporting that the hMailServer service is not running. Check status NOW."
	$response = Invoke-RestMethod -Uri $uri
}

It took me a while to figure this one out. I don't know why, but "exit" in the if statement didn't work. Finally i ran across something on the web and changed it to "exit 1" and it worked straight off. Weird.

palinka
Senior user
Senior user
Posts: 1262
Joined: 2017-09-12 17:57

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by palinka » 2019-02-19 13:20

mattg wrote:
2019-02-19 01:26
I do have a network device with a SIM card, a Netgear LTE Modem LB2120, but I am yet to work out how to send SMS from a network attached device via a script. I can do it from the GUI of the device, so I'm sure it will be possible...
I reread your post and looked up the device. That's a hotspot modem, correct? Yeah I'm not sure if that could work. All the gateway software I've seen use at commands which require the device be connected by cable. Mine is usb and the driver fakes the serial port for access. But the usb ones are cheap if you want to go that route. Or you could try using an old phone connected by usb.

User avatar
mattg
Moderator
Moderator
Posts: 20272
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by mattg » 2019-02-20 02:56

palinka wrote:
2019-02-19 13:20
I reread your post and looked up the device. That's a hotspot modem, correct?
Some of us have NBN in Australia, which stands for National Broadband Network.
I have FTTP = Fibre to the premises

I have a fibre optic cable running to a NBN router at my premises, which offers a LAN type connector to my own router/modem commonly supplied by ISPs. This is much like cable in many regards.

Problem is that it can drop out for what ever reason.
My LTE modem fits between my router/firewall/modem and the NBN router, and offers failover to 4G via a SIM Card when the NBN connection is down. I can't get static IP on 4G, so this is only for outgoing internet traffic, not incoming traffic.

And yeah, I haven't found much on how I can achieve a message via http get, but that would be the plan
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

palinka
Senior user
Senior user
Posts: 1262
Joined: 2017-09-12 17:57

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by palinka » 2019-02-20 03:19

mattg wrote:
2019-02-20 02:56
palinka wrote:
2019-02-19 13:20
I reread your post and looked up the device. That's a hotspot modem, correct?
Some of us have NBN in Australia, which stands for National Broadband Network.
I have FTTP = Fibre to the premises

I have a fibre optic cable running to a NBN router at my premises, which offers a LAN type connector to my own router/modem commonly supplied by ISPs. This is much like cable in many regards.

Problem is that it can drop out for what ever reason.
My LTE modem fits between my router/firewall/modem and the NBN router, and offers failover to 4G via a SIM Card when the NBN connection is down. I can't get static IP on 4G, so this is only for outgoing internet traffic, not incoming traffic.

And yeah, I haven't found much on how I can achieve a message via http get, but that would be the plan
If you just want to test, there are a bunch of android gateway apps that connect via wifi. The ones i tried were not very stable. Or it could have been my phone.

For your backup internet, you could assign a backup mx record to a dynamic domain that the lte router updates. Most of these devices come with a few ddns providers built in. Shoot, you could leave it running all the time like that and never skip a beat.

I have fibre too. It's always worked even when the power goes out. I have a ups attached to the fibre hardware, all my networking gear and my server. I get about 40 minutes which is usually enough time for the power to come back on. I only remember the fibre going down once and that was very early on a Sunday morning. I think they may have been upgrading hardware.

palinka
Senior user
Senior user
Posts: 1262
Joined: 2017-09-12 17:57

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by palinka » 2019-02-20 14:04

We're waaaay off topic here but i just had to share this. I run apcupsd to monitor my ups. I had it set up to email me via my cell carrier's sms gateway so i would get text messages for power events. I already changed the scripts to use my personal sms gateway. After writing the last post and thinking about the power situation, i looked to see if the utility here accepts text messages to report outages and they do. So i registered my gateway's cell number with the utility and now when the power goes out, my server will automatically send the utility an outage text. I set it up with a 2 minute delay so it wouldn't fire on momentary outages, which are common here. Hehe.. i don't even have to be home to know about it, or if it happens while I'm sleeping they still get notified. I know my neighbors will appreciate that too. :mrgreen:

Also, one of the reasons i wanted an sms gateway is because sometimes my email > carrier gateway messages would get trapped in the carrier's sms spam filter. No matter how i experimented with "unspammy" messages, some still got trapped and i couldn't figure out why. It was random. So this solves that problem.

User avatar
jimimaseye
Moderator
Moderator
Posts: 8170
Joined: 2011-09-08 17:48

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by jimimaseye » 2019-02-20 15:27

palinka wrote:
2019-02-20 14:04
We're waaaay off topic here.....
Aren't we just.
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

palinka
Senior user
Senior user
Posts: 1262
Joined: 2017-09-12 17:57

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by palinka » 2019-02-24 14:45

jimimaseye wrote:
2019-02-20 15:27
palinka wrote:
2019-02-20 14:04
We're waaaay off topic here.....
Aren't we just.
:mrgreen:

Anyway, back on topic. It happened again last night. This time I paid a little closer attention before rebooting.

Service status was "running". My notification script didn't work because it tests "not running".

Log shows the same shutdown errors:

Code: Select all

"DEBUG"	2760	"2019-02-23 23:58:11.869"	"Still 1 remaining threads in queue Server queue. First task: IOService"
"DEBUG"	9524	"2019-02-23 23:58:11.900"	"Still 1 remaining threads in queue IOCPQueue. First task: IOCPQueueWorkerTask"
"DEBUG"	2760	"2019-02-23 23:58:12.135"	"Given up waiting for threads to join in queue Server queue."
"DEBUG"	2760	"2019-02-23 23:58:12.135"	"WorkQueueManager::RemoveQueue - Stopped Server queue"
"DEBUG"	2760	"2019-02-23 23:58:12.135"	"WorkQueueManager::RemoveQueue - Erased Server queue"
"DEBUG"	2760	"2019-02-23 23:58:12.135"	"Destructing work queue Server queue"
"DEBUG"	9524	"2019-02-23 23:58:12.166"	"Given up waiting for threads to join in queue IOCPQueue."
"DEBUG"	9524	"2019-02-23 23:58:12.166"	"WorkQueueManager::RemoveQueue - Stopped IOCPQueue"
"DEBUG"	9524	"2019-02-23 23:58:12.166"	"WorkQueueManager::RemoveQueue - Erased IOCPQueue"
"DEBUG"	9524	"2019-02-23 23:58:12.166"	"IOService::Stop() - Complete"
"DEBUG"	9524	"2019-02-23 23:58:12.166"	"Destructing work queue IOCPQueue"
Notice the last two lines. The service DID NOT stop.

Something else - maybe just a coincidence, but this seems to always happen on a Saturday night. At first I thought maybe it had something to do with me fooling around with the system on Saturdays (the time I usually do), but yesterday I did not install anything, reboot or anything like that. Weird.

palinka
Senior user
Senior user
Posts: 1262
Joined: 2017-09-12 17:57

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by palinka » 2019-02-24 17:42

The issue seems to be oapp.stop insofar as it pauses hmailserver, not stop hmailserver service. Therefore, the way I tried to detect problems didn't work as I was using service status. So I figured I would need another method - telnet would tell me if things are not working. After some goolaging and experimentation, I came up with this powershell script.

To be safe that there's not some temporary issue, it checks a full 3 times before attempting to restart hmailserver service. The script is run on the same machine as hmailserver and there's no chance for temporary network issues, but I'm playing it safe for circumstances I can't even think of at the moment.

If powershell cannot connect by telnet after 3 tries, it will restart the service and send me a text letting me know. Then it will wait 60 seconds and check service status again. If the service is not running, I'll get another message telling me.

If powershell can't connect either once or twice, but does successfully connect on the second or third try, it will finish and exit without attempting to restart hmailserver.

Code: Select all

$ErrorActionPreference = 'silentlycontinue'
$remotehost = "localhost" 
$port = 25 
$ServiceName = 'hMailServer'

$socket = new-object System.Net.Sockets.TcpClient($remoteHost, $port) 

if($socket -eq $null) { 
	Start-Sleep -seconds 300
	$socket = new-object System.Net.Sockets.TcpClient($remoteHost, $port) 
	if($socket -eq $null) { 
		Start-Sleep -seconds 300
		$socket = new-object System.Net.Sockets.TcpClient($remoteHost, $port) 
		if($socket -eq $null) { 
			Restart-Service $ServiceName
			$uri = "https://smsgateway/index.php/plugin/rest_api/send_sms?phoneNumber=1234567890&message=ATTENTION! hMailServer service is being RESTARTED due to a fault. Check status NOW." 
			$response = Invoke-RestMethod -Uri $uri
			Start-Sleep -seconds 60
			(get-service $ServiceName).Refresh()
			if ((get-service $ServiceName).Status -ne 'Running')
			{
			$uri = "https://smsgateway/index.php/plugin/rest_api/send_sms?phoneNumber=1234567890&message=ATTENTION! BrianServer reporting that the hMailServer service is not running. Check status NOW." 
			$response = Invoke-RestMethod -Uri $uri
			}
		}
	}
}


palinka
Senior user
Senior user
Posts: 1262
Joined: 2017-09-12 17:57

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by palinka » 2019-03-07 12:15

Hot damn! It worked. Not only that, but the backup script, which got hung up on oapp.stop, picked up when hmailserver service actually stopped and completed without errors.

I have my backup script set to run at 11:58pm. My telnet check script runs at 12:15. Note: it's supposed to wait 5 minutes between tries but i had it set to 10 seconds when i was testing it and forgot to change it back. Just fixed that. You'll see that in the log timestamps. The reason for the two 5 minute waits is just in case hmailserver is still running the backup, or other momentary communication errors.

Checking communication turned out to be a good idea because when it hangs like this, the service status is still "running" so checking service status won't do anything.

Real life test: PASS

Code: Select all

Backup Start: Wed 03/06/2019 23:58:01.18 


HMS Server Start Time: 2019-03-05 23:58:36
HMS Daily Spam Reject count: 16
HMS Daily Viruses Removed count: 10

 0:15:29.94 Stopping Hmailserver service...

The hMailServer service was stopped successfully.

Ok! 
 0:15:30.75 Stopping Spamassassin service...
..
The spamassassin service was stopped successfully.

~~~~SNIP~~~~

User avatar
jimimaseye
Moderator
Moderator
Posts: 8170
Joined: 2011-09-08 17:48

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by jimimaseye » 2019-03-07 15:08

Effectively your oapp.stop process is erroring/ freezing. By restarting the service your backup script then got control of the service (which is now running again) to stop it and continue. This is effectively the same as not having the oAp.stop in the first place.

[Entered by mobile. Excuse my spelling.]
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

palinka
Senior user
Senior user
Posts: 1262
Joined: 2017-09-12 17:57

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by palinka » 2019-03-07 17:07

jimimaseye wrote:
2019-03-07 15:08
Effectively your oapp.stop process is erroring/ freezing. By restarting the service your backup script then got control of the service (which is now running again) to stop it and continue. This is effectively the same as not having the oAp.stop in the first place.

[Entered by mobile. Excuse my spelling.]
The main reason its there is because the number of "hangs" was worse before I put it in. But it certainly can't hurt removing it, which I'll do tonight and see how it goes.

palinka
Senior user
Senior user
Posts: 1262
Joined: 2017-09-12 17:57

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by palinka » 2019-03-08 02:26

I looked back in this thread and found the reason i added oapp.stop.

https://hmailserver.com/forum/viewtopic ... 60#p206367

In retrospect, it appears better to leave it in. Without it, the backup script fails. With it included, plus with my telnet monitor script, the service gets restarted and the backup continues successfully. :mrgreen:

User avatar
jimimaseye
Moderator
Moderator
Posts: 8170
Joined: 2011-09-08 17:48

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by jimimaseye » 2019-03-08 10:08

So attempting to stop the service times out at 30000 milliseconds. Probably for the same reason the oApp.stop doesnt work (whatever that is). So you issue another NET STOP and net start (to then go on and issue a net stop).

OR....

Try removing the oApp.stop and changing the initial script to:

Code: Select all

:1st
:maildata
echo %time% Stopping Hmailserver service...>> %BackLog%&net stop hmailserver >> %BackLog%
set FailedHmailservice=Ok
set bold=%noBold%&set fontcolor=%fontBlack%
if errorlevel 1 echo %time% Stopping Hmailserver service (2nd try)...>> %BackLog%&net stop hmailserver >> %BackLog%
if errorlevel 1 echo %time% Stopping Hmailserver service (3rd try)...>> %BackLog%&net stop hmailserver >> %BackLog%
if errorlevel 1 set Failed=true&set FailedHmailservice=Failed to stop. Backup of emails not performed!&set bold=b&set fontcolor=%colorfill%
echo %FailedHmailservice%! >> %BackLog%
echo ^<tr^>^<td^>^<%fontcolor%^>^<%bold%^>Hmailserver service shutdown:^</td^>^<td^>^<%fontcolor%^>^<%bold%^>%FailedHmailservice%^</td^>^</tr^> >>%outf%
As you have seen that a 2nd attempt to stop seems to work then the above should more than suffice.
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

palinka
Senior user
Senior user
Posts: 1262
Joined: 2017-09-12 17:57

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by palinka » 2019-03-08 12:38

jimimaseye wrote:
2019-03-08 10:08
So attempting to stop the service times out at 30000 milliseconds. Probably for the same reason the oApp.stop doesnt work (whatever that is). So you issue another NET STOP and net start (to then go on and issue a net stop).

OR....

Try removing the oApp.stop and changing the initial script to:

Code: Select all

:1st
:maildata
echo %time% Stopping Hmailserver service...>> %BackLog%&net stop hmailserver >> %BackLog%
set FailedHmailservice=Ok
set bold=%noBold%&set fontcolor=%fontBlack%
if errorlevel 1 echo %time% Stopping Hmailserver service (2nd try)...>> %BackLog%&net stop hmailserver >> %BackLog%
if errorlevel 1 echo %time% Stopping Hmailserver service (3rd try)...>> %BackLog%&net stop hmailserver >> %BackLog%
if errorlevel 1 set Failed=true&set FailedHmailservice=Failed to stop. Backup of emails not performed!&set bold=b&set fontcolor=%colorfill%
echo %FailedHmailservice%! >> %BackLog%
echo ^<tr^>^<td^>^<%fontcolor%^>^<%bold%^>Hmailserver service shutdown:^</td^>^<td^>^<%fontcolor%^>^<%bold%^>%FailedHmailservice%^</td^>^</tr^> >>%outf%
As you have seen that a 2nd attempt to stop seems to work then the above should more than suffice.
I was actually thinking about that but i wasn't sure if the error checking resolves back to the first net stop command or if it would work sequentially the way you did it. I guess that answers my question and also simplifies things greatly. Thanks!

Maybe you should add this to the OP.

palinka
Senior user
Senior user
Posts: 1262
Joined: 2017-09-12 17:57

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by palinka » 2019-03-27 10:40

Woke up to this today:

Code: Select all

Backup Start: Tue 03/26/2019 23:58:01.14 


HMS Server Start Time: 2019-03-25 23:58:38
HMS Daily Spam Reject count: 19
HMS Daily Viruses Removed count: 12

23:58:01.23 Stopping Hmailserver service...
23:59:01.31 Stopping Hmailserver service (2nd try)...
23:59:01.34 Stopping Hmailserver service (3rd try)...
Failed to stop. Backup of emails not performed!! 

I'm stumped. There's 1 minute 8 seconds between first and second attempt but only 3 seconds between 2nd and 3rd.

User avatar
jimimaseye
Moderator
Moderator
Posts: 8170
Joined: 2011-09-08 17:48

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by jimimaseye » 2019-03-27 14:35

Not 3 seconds but 3 tenths between 2nd and 3rd.

Go to services, properties (on hmailserver service), advanced and check the "what to do" options on failures. Just tinker.

[Entered by mobile. Excuse my spelling.]
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

palinka
Senior user
Senior user
Posts: 1262
Joined: 2017-09-12 17:57

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by palinka » 2019-03-28 00:34

Will do. 👍

palinka
Senior user
Senior user
Posts: 1262
Joined: 2017-09-12 17:57

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by palinka » 2019-03-28 00:36

jimimaseye wrote:
2019-03-27 14:35
Not 3 seconds but 3 tenths between 2nd and 3rd.
Hmmm... i overlooked that. At least i got the minute part right. :lol:

User avatar
jimimaseye
Moderator
Moderator
Posts: 8170
Joined: 2011-09-08 17:48

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by jimimaseye » 2019-04-05 22:02

palinka wrote:
2019-03-28 00:36
jimimaseye wrote:
2019-03-27 14:35
Not 3 seconds but 3 tenths between 2nd and 3rd.
Hmmm... i overlooked that. At least i got the minute part right. :lol:
Next time you have a failure to stop, check out this directory (and look at the contents): C:\ProgramData\Microsoft\Windows\WER\ReportQueue

You might find it useful. (Or you might not).
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

palinka
Senior user
Senior user
Posts: 1262
Joined: 2017-09-12 17:57

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by palinka » 2019-04-06 00:52

I will. Thanks. I'll update what i find.

User avatar
jimimaseye
Moderator
Moderator
Posts: 8170
Joined: 2011-09-08 17:48

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by jimimaseye » 2019-04-06 09:57

palinka wrote:
2019-04-06 00:52
I will. Thanks. I'll update what i find.
I had a service stop failure yesterday (very rare for me). That folder had dump files and a readable xml. Perhaps useful to someone.
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

palinka
Senior user
Senior user
Posts: 1262
Joined: 2017-09-12 17:57

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by palinka » 2019-04-06 16:38

I've been super busy lately. Finally had a chance to look. …/ReportQueue and …/Temp folders were empty. …/ReportArchive had lots of entries but none related to hmailserver except one for hmailadmin.exe. Tons of apache crashes. Looks like several per day going back to Feb 8. Does this folder auto-clear? I'm wondering if this has been a regular thing for years and I never knew.

Anyway, next time it happens I'll have a look as soon as possible as well as windows event log.

Luna Moon
New user
New user
Posts: 8
Joined: 2018-05-01 17:26
Location: Europe
Contact:

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by Luna Moon » 2019-05-05 09:37

Another great tutorial! Thanks to everyone, who is helping out beginners like me, it helps a lot and I really appreciate your effort and time. :)

User avatar
jimimaseye
Moderator
Moderator
Posts: 8170
Joined: 2011-09-08 17:48

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by jimimaseye » 2019-05-07 22:22

palinka wrote:
2019-04-06 00:52
I will. Thanks. I'll update what i find.
Read this: https://www.hmailserver.com/forum/viewt ... =7&t=33965
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

palinka
Senior user
Senior user
Posts: 1262
Joined: 2017-09-12 17:57

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by palinka » 2019-05-07 22:47

jimimaseye wrote:
2019-05-07 22:22
palinka wrote:
2019-04-06 00:52
I will. Thanks. I'll update what i find.
Read this: https://www.hmailserver.com/forum/viewt ... =7&t=33965
Speak of the devil! I will check out TCPView as soon as I can.

However, just today I also became fed up with these errors. As you know, I installed RvdH's hmail version and for the past 3 days the backup script failed at NET START hmailserver (not STOP). So today I made a powershell script to stop and also one to start hmailserver. I haven't tested this yet in production except for just making sure the new parts work. I'll let you know tomorrow if I have any problems.

It seems that net stop/start command doesn't wait long enough to return a proper result. The powershell script will stomp the guts out of the service until it submits and then notify you if there were any errors. I guess the notification thing is not really necessary since the backupmail script will do the same thing. However, if, like me the past 3 days, the script errors on net start hmailserver, even though the backup completes successfully the script thinks that hmail is not running and will not attempt to email the log. So I've been receiving the cleardown email but not the backup email.

Code: Select all

:1st
:maildata
echo %time% Stopping Hmailserver service...>> %BackLog%
PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& 'C:\scripts\HMSBackupCleardown\hmstop.ps1'"  >> %BackLog%
SC query hmailserver | FIND /i "STOPPED"
set FailedHmailservice=Ok
set bold=%noBold%&set fontcolor=%fontBlack%
if errorlevel 1 set Failed=true&set FailedHmailservice=Failed to stop. Backup of emails not performed!&set bold=b&set fontcolor=%colorfill%
echo %FailedHmailservice%! >> %BackLog%
echo ^<tr^>^<td^>^<%fontcolor%^>^<%bold%^>Hmailserver service shutdown:^</td^>^<td^>^<%fontcolor%^>^<%bold%^>%FailedHmailservice%^</td^>^</tr^> >>%outf%

Code: Select all

:6th
if not "%FailedHmailservice%" == "Ok" set section=7th & goto 7th
echo %time% Starting Hmailserver service... >> %BackLog%
PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& 'C:\scripts\HMSBackupCleardown\hmstart.ps1'"  >> %BackLog%
SC query hmailserver | FIND /i "RUNNING"
set bold=%noBold%&set fontcolor=%fontBlack%
if errorlevel 1 set Failed=true&set FailedHmailservice=Failed to restart&set bold=b&set fontcolor=%colorfill%
echo %FailedHmailservice%! Errorlevel=%errorlevel% >> %BackLog%
echo ^<tr^>^<td^>^<%fontcolor%^>^<%bold%^>Hmailserver service startup:^</td^>^<td^>^<%fontcolor%^>^<%bold%^>%FailedHmailservice%^</td^>^</tr^> >>%outf%
hmstop.ps1

Code: Select all

$ErrorActionPreference = 'silentlycontinue'
$ServiceName = 'hMailServer'

Function CkSvcMail($Body) {
	$EmailFrom = "notification.account@gmail.com"
	$EmailTo = "1234567890@tmomail.net" # or any email address
	$SMTPServer = "smtp.gmail.com" 
	$SMTPAuthUser = "notification.account@gmail.com"
	$SMTPAuthPass = "supersecretpassword"
	$Subject = "Windows Service Failure" 
	$SMTPClient = New-Object Net.Mail.SmtpClient($SmtpServer, 587) 
	$SMTPClient.EnableSsl = $true 
	$SMTPClient.Credentials = New-Object System.Net.NetworkCredential($SMTPAuthUser, $SMTPAuthPass); 
	$SMTPClient.Send($EmailFrom, $EmailTo, $Subject, $Body)
}

Stop-Service $ServiceName
Start-Sleep -seconds 60
(get-service $ServiceName).Refresh()
if ((get-service $ServiceName).Status -ne 'Stopped'){
	Stop-Service $ServiceName
	Start-Sleep -seconds 60
	(get-service $ServiceName).Refresh()
	if ((get-service $ServiceName).Status -ne 'Stopped'){			
		$Body = "ATTENTION! hMailServer backup script (powershell hmstop) could not shut down service."
		CkSvcMail $Body
	}
}
hmstart.ps1

Code: Select all

$ErrorActionPreference = 'silentlycontinue'
$ServiceName = 'hMailServer'

Function CkSvcMail($Body) {
	$EmailFrom = "notification.account@gmail.com"
	$EmailTo = "1234567890@tmomail.net" # or any email address
	$SMTPServer = "smtp.gmail.com" 
	$SMTPAuthUser = "notification.account@gmail.com"
	$SMTPAuthPass = "supersecretpassword"
	$Subject = "Windows Service Failure" 
	$SMTPClient = New-Object Net.Mail.SmtpClient($SmtpServer, 587) 
	$SMTPClient.EnableSsl = $true 
	$SMTPClient.Credentials = New-Object System.Net.NetworkCredential($SMTPAuthUser, $SMTPAuthPass); 
	$SMTPClient.Send($EmailFrom, $EmailTo, $Subject, $Body)
}

Start-Service $ServiceName
Start-Sleep -seconds 60
(get-service $ServiceName).Refresh()
if ((get-service $ServiceName).Status -ne 'Running'){
	Start-Service $ServiceName
	Start-Sleep -seconds 60
	(get-service $ServiceName).Refresh()
	if ((get-service $ServiceName).Status -ne 'Running'){
		$Body = "ATTENTION! hMailServer backup script (powershell hmstart) could not restart hMailServer service."
		CkSvcSMS $Body
		CkSvcMail $Body
	}	
}

palinka
Senior user
Senior user
Posts: 1262
Joined: 2017-09-12 17:57

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by palinka » 2019-05-10 13:13

Updated hmstop and hmstart to be more verbose.

This works. It beats hmailserver service into submission. There's a bit of code in the backupmail.bat file (above) to double check if it worked by checking service status. If error on status, script resumes as with error the same as it was before.

hmstop.ps1

Code: Select all

$ErrorActionPreference = 'silentlycontinue'
$ServiceName = 'hMailServer'

Function CkSvcMail($Body) {
	$EmailFrom = "notification.account@gmail.com"
	$EmailTo = "1234567890@tmomail.net" # or other email address - i just like txt notifications
	$SMTPServer = "smtp.gmail.com" 
	$SMTPAuthUser = "notification.account@gmail.com"
	$SMTPAuthPass = "SuperSecretPassword"
	$Subject = "Windows Service Failure" 
	$SMTPClient = New-Object Net.Mail.SmtpClient($SmtpServer, 587) 
	$SMTPClient.EnableSsl = $true 
	$SMTPClient.Credentials = New-Object System.Net.NetworkCredential($SMTPAuthUser, $SMTPAuthPass); 
	$SMTPClient.Send($EmailFrom, $EmailTo, $Subject, $Body)
}

$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
write-output "$timestamp Now executing powershell hmstop shutdown script."

(get-service $ServiceName).Refresh()
if ((get-service $ServiceName).Status -ne 'RUNNING'){
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer already not RUNNING. Nothing to shut down."
	exit
}
else { 
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer RUNNING. Preparing to shut down."
}

Stop-Service $ServiceName
Start-Sleep -seconds 60
(get-service $ServiceName).Refresh()
if ((get-service $ServiceName).Status -ne 'STOPPED'){
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer failed to shut down. Wait 60 seconds and try again."}
else {
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer shut down. Good to go."
	exit
}

Start-Sleep -seconds 60
Stop-Service $ServiceName
(get-service $ServiceName).Refresh()
if ((get-service $ServiceName).Status -ne 'STOPPED'){
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer failed to shut down on 2nd attempt. Wait 60 seconds and try again."}
else {
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer shut down. Good to go."
	exit
}

Start-Sleep -seconds 60
Stop-Service $ServiceName
(get-service $ServiceName).Refresh()
if ((get-service $ServiceName).Status -ne 'STOPPED'){
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer failed to shut down on 3rd attempt. Giving up and sending notification."
	$Body = "ATTENTION! hMailServer backup script (powershell hmstop) could not shut down service."
	CkSvcMail $Body
	}
else {
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer shut down. Good to go."
}
hmstart.ps1

Code: Select all

$ErrorActionPreference = 'silentlycontinue'
$ServiceName = 'hMailServer'

Function CkSvcMail($Body) {
	$EmailFrom = "notification.account@gmail.com"
	$EmailTo = "1234567890@tmomail.net" # or other email address - i just like txt notifications
	$SMTPServer = "smtp.gmail.com" 
	$SMTPAuthUser = "notification.account@gmail.com"
	$SMTPAuthPass = "SuperSecretPassword"
	$Subject = "Windows Service Failure" 
	$SMTPClient = New-Object Net.Mail.SmtpClient($SmtpServer, 587) 
	$SMTPClient.EnableSsl = $true 
	$SMTPClient.Credentials = New-Object System.Net.NetworkCredential($SMTPAuthUser, $SMTPAuthPass); 
	$SMTPClient.Send($EmailFrom, $EmailTo, $Subject, $Body)
}

$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
write-output "$timestamp Now executing powershell hmstart startup script."

(get-service $ServiceName).Refresh()
if ((get-service $ServiceName).Status -ne 'STOPPED'){
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer already RUNNING. Nothing to start."
	exit
}
else {
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer STOPPED. Preparing to start service."
}

Start-Service $ServiceName
Start-Sleep -seconds 60
(get-service $ServiceName).Refresh()
if ((get-service $ServiceName).Status -ne 'RUNNING'){
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer failed to start. Wait 60 seconds and try again."}
else {
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer started. Good to go."
	exit
}

Start-Service $ServiceName
Start-Sleep -seconds 60
(get-service $ServiceName).Refresh()
if ((get-service $ServiceName).Status -ne 'RUNNING'){
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer failed to start on 2nd attempt. Wait 60 seconds and try again."}
else {
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer started. Good to go."
	exit
}

Start-Service $ServiceName
Start-Sleep -seconds 60
(get-service $ServiceName).Refresh()
if ((get-service $ServiceName).Status -ne 'RUNNING'){
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer failed to start on 3rd attempt. Giving up and sending notification."
	$Body = "ATTENTION! hMailServer backup script (powershell hmstart) could not shut down service."
	CkSvcMail $Body
	}
else {
	$timestamp = "{0:HH:mm:ss.ff}" -f (get-date)
	write-output "$timestamp hMailServer started. Good to go."
}

RBoy
New user
New user
Posts: 21
Joined: 2018-12-04 04:28

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by RBoy » 2019-06-07 14:28

Interesting read but I’m wondering if beating the server back into submission has any side effects on data quality or stability.

The question is why does the API to stop hang in the first place?

palinka
Senior user
Senior user
Posts: 1262
Joined: 2017-09-12 17:57

Re: HOW TO: Ready-To-Go Backup and Cleardown script

Post by palinka » 2019-06-07 17:58

RBoy wrote:
2019-06-07 14:28
Interesting read but I’m wondering if beating the server back into submission has any side effects on data quality or stability.

The question is why does the API to stop hang in the first place?
I haven't experienced any issues.

Likely caused by this: https://www.hmailserver.com/forum/viewt ... =7&t=33965

I don't understand the issue completely, but my basic understanding is that the request to pause or stop times out before all open threads are able to close. That's why my powershell thing works. Because if the first request times out, the second or third will definitely knock it out. I never got past the second request so i never had a scenario where it simply couldn't be stopped.

I have another script that runs 17 minutes after the backup starts that checks telnet for connectivity and if no communication, does effectively the same routine to shut it down. That one has only been triggered when i was using vbs oApp.stop to pause the server and got the hang. I stopped using oApp.stop to pause and the redundant one never triggered again.

Post Reply