[BUG] Backup Script Crashing Periodically

[Lee Thompson]

hMailServer: 2.6 Build 2145

I've been running hmailserver for years and there seems to be a newish issue with the backup script.

Twice now there's been a crash of some kind (dump files are produced) and after that occurs, hmailserver still works but the backup script does not.

In this case, during the 19th's backup, a DMP file is created. All subseqeuent backup attempts fail since it thinks it's still running.

Code: Select all

...
2015-01-19 02:15:01.741	Backing up data directory...
2015-01-20 02:15:00.668	Backup started
2015-01-20 02:15:00.668	BACKUP ERROR: Backup or restore operation is already started
2015-01-21 02:15:01.304	Backup started
2015-01-21 02:15:01.320	BACKUP ERROR: Backup or restore operation is already started
...

I've attached the minidumps.

(If nothing else, if there's an exception, after the dmp file is made, should probably reset the backup state?)

[percepts]

send the mini dump to Martin, he's the only one who can view it. It won't open in VSExpress 2013.

I suspect that you have too much data. How big is your hmailserver/data folder ?

You can only have one instance of bacjup running at a time.

[Lee Thompson]

Backup is nightly. When the DMP occurs it never 'terminates' that backup instance, hence the messages.

Data folder isn't that big.

[percepts]

at the time of day the backup is scheduled to run is it conflicting with any other scheduled runs such as windows updates or DB (mysql or such) dumps/restarts?

Check your windows event logs at the time the backup seems to fail.

[martin]

You write that you're running hMailServer 2.6, but I'm assuming you mean 5.6.

I looked in the dmp file and see that the error occur when the backup process copies a specific file.

Is there any error log created when the error occur? Those are located in the log directories and have a name which starts with ERROR_. This log should hopefully contain the exact error message which may explain why the error occur.

[Lee Thompson]

You're quite right, 5.6.

Here's a log entry, I've redacted the domain name and user name.

This may not be the root cause, I just noticed when it crashes, it does not clean up the DataBackup folder.

I'm going to go through some more logs and will post more detail soon.

Code: Select all

"ERROR"	4444	"2015-02-04 02:15:05.730"	"Severity: 2 (High), Code: HM4208, Source: ExceptionHandler::Run, Description: An error occured while executing 'Task-BackupTask', Error code: 80, Message: boost::filesystem::copy_file: The file exists: "C:\Program Files (x86)\hMailServer\Data\DOMAIN\USER\00\{0046E970-A407-4209-A3FF-163878C961A2}.eml", "D:\Private\Backups\hMailServer\DataBackup\DOMAIN\USER\00\{0046E970-A407-4209-A3FF-163878C961A2}.eml""
"ERROR"	4444	"2015-02-04 02:15:13.837"	"Severity: 1 (Critical), Code: HM5519, Source: StackLogger::Log, Description: An error has been detected. A mini dump has been written to C:\Program Files (x86)\hMailServer\Logs\minidump_2015-02-04 021505_{8ED33691-DDC8-4AA0-837E-7AB9864B6F9D}.dmp"

[Lee Thompson]

It looks like the backup job may be trying to backup files that are currently in a delivery queue and then crashes after the file is removed after the delivery is complete in the meantime?

Code: Select all

"ERROR"	904	"2015-01-19 02:19:44.476"	"Severity: 2 (High), Code: HM4208, Source: ExceptionHandler::Run, Description: An error occured while executing 'Task-BackupTask', Error code: 2, Message: boost::filesystem::copy_file: The system cannot find the file specified: "C:\Program Files (x86)\hMailServer\Data\{7BE0BF93-9665-4CCC-AD24-61F7FA6E2BB0}.eml", "D:\Private\Backups\hMailServer\DataBackup\{7BE0BF93-9665-4CCC-AD24-61F7FA6E2BB0}.eml""
"ERROR"	904	"2015-01-19 02:19:46.106"	"Severity: 1 (Critical), Code: HM5519, Source: StackLogger::Log, Description: An error has been detected. A mini dump has been written to C:\Program Files (x86)\hMailServer\Logs\minidump_2015-01-19 021944_{A532A4FA-DC6B-4C8E-96CE-A2DF3AAFB7FB}.dmp"

Code: Select all

"ERROR"	3356	"2015-01-07 02:31:55.929"	"Severity: 2 (High), Code: HM4208, Source: ExceptionHandler::Run, Description: An error occured while executing 'Task-BackupTask', Error code: 2, Message: boost::filesystem::copy_file: The system cannot find the file specified: "C:\Program Files (x86)\hMailServer\Data\{F0C9A3C6-71EB-4EB3-AFBB-247CA44BB3C7}.eml", "D:\Private\Backups\hMailServer\DataBackup\{F0C9A3C6-71EB-4EB3-AFBB-247CA44BB3C7}.eml""
"ERROR"	3356	"2015-01-07 02:32:01.954"	"Severity: 1 (Critical), Code: HM5519, Source: StackLogger::Log, Description: An error has been detected. A mini dump has been written to C:\Program Files (x86)\hMailServer\Logs\minidump_2015-01-07 023155_{4F86B02F-1A47-446C-88C7-1FFD9150F857}.dmp"

[jimimaseye]

You thought about writing your own backup script? Process is simple:

1, STOP HMAILSERVER service ('net stop hmailserver')
2, use ROBOCOPY to mirror your data directory to a temporary data directory
3, copy the SDF database file to a temporary database' directory (or if using an external database, do a 'dump' of the database to a .SQL file)
4, start hmailserver
(the above should take just seconds to run after the first time)
5, ZIP the temporary backup/database directories
(6, DONT delete the temporary data directory)

This process is simply in essence all the internal backup routine does. Except doing it this way you ensure that no files (incoming outgoing) get deleted whilst in process and more importantly hmailserver 'offline' time is kept to a minimum.

Create a batch file with the above, stick it in Task Scheduler.

Mine effectively does this but looks a lot more fancy as I incorporate other data directories and various checks for service stop/starting correctly, as well as incorporating notification emails as it processes. Im sure if you search the forums, or ask here, I/someone would be happy to show their ready-made versions if you dont fancy the 'challenge' (if you could call it a challenge).

[jimimaseye]

Ok, you got me. Here is a cut down version if you want it (this for using an external database, as referenced in '4th' section):

Code: Select all

set inDate=%date:~6,4%-%date:~3,2%-%date:~0,2%-%str%
rem set inDate=%date:~6,4%-%date:~3,2%-%date:~0,2%-%time:~0,2%%time:~3,2%%time:~6,2%
set Backdir=d:\backups
set Backtempdir=%Backtempdir%
set BackLog=%Backdir%\Backup_%inDate%.log
set datadir=d:\datapath

rem CHANGE VARIABLES ABOVE THIS LINE and relevant directory paths below ----

set str=%time:~0,8%&rem
set str=%str: =0%
set str=%str::=%
echo Backup Start: %date% %time% > %BackLog%

REM :: Perform backup to temporary directory

erase /Q %Backtempdir%\hMailData\*.mysql >> %BackLog%
set section=1st

:1st
:maildata
echo %time% >> %BackLog%&net stop hmailserver >> %BackLog%

:2nd
echo %time% >> %BackLog%&net stop spamassassin >> %BackLog%

:3rd
set section=3rd
robocopy "%datadir%\hMailData" %Backtempdir%\hMailData /mir /ndl /r:43200 /np /w:1 >> %BackLog%
goto ROBOerrorcheck

:4th
cd C:\Program Files\MySQL\MySQL Server 5.5\bin >> %BackLog%
mysqldump -u"root" -p"secretpassword" -q -A -l --add-drop-table -P3306 >"%Backtempdir%\hMailData\MYSQLDump_%inDate%.mysql"

:5th
cd C:\Program Files (x86)\JAM Software\SpamAssassin for Windows\
sa-update.exe -v --nogpg --channelfile UpdateChannels.txt >> %BackLog%

:6th
@rem Restart Spamassassin and Hmailserver services
echo %time% >> %BackLog%&net start spamassassin >> %BackLog%

:7th
echo %time% >> %BackLog%&net start hmailserver >> %BackLog%

:8th
set section=8th
:Sagedata
robocopy "%datadir%\SageData" %Backtempdir%\SageData /mir /ndl /r:43200 /np /w:1 >> %BackLog%
goto ROBOerrorcheck

:9th
:Decrodata
robocopy "D:\Decrodata" %Backtempdir%\Decrodata /mir /ndl /r:43200 /np /w:1 >> %BackLog%
goto ROBOerrorcheck

REM :: Zip temporary directory and email log file
cd C:\Program Files\7-Zip >> %BackLog%
echo %date% %time% 7Zip Creating archive d:\Backups\Serverdata_%inDate%.zip >> %BackLog%
7z a -tzip "d:\Backups\Serverdata_%inDate%" %Backtempdir%\*

goto Testend

:ROBOerrorcheck
if errorlevel 16 echo ***FATAL ERROR*** >> %BackLog% & goto end
if errorlevel 15 echo OKCOPY + FAIL + MISMATCHES + XTRA >> %BackLog% & goto end
if errorlevel 14 echo FAIL + MISMATCHES + XTRA >> %BackLog% & goto end
if errorlevel 13 echo OKCOPY + FAIL + MISMATCHES >> %BackLog% & goto end
if errorlevel 12 echo FAIL + MISMATCHES>> %BackLog% & goto end
if errorlevel 11 echo OKCOPY + FAIL + XTRA >> %BackLog% & goto end
if errorlevel 10 echo FAIL + XTRA >> %BackLog% & goto end
if errorlevel 9 echo OKCOPY + FAIL >> %BackLog% & goto end
if errorlevel 8 echo FAIL >> %BackLog% & goto end
if errorlevel 7 echo OKCOPY + MISMATCHES + XTRA >> %BackLog% & goto end
if errorlevel 6 echo MISMATCHES + XTRA >> %BackLog% & goto end
if errorlevel 5 echo OKCOPY + MISMATCHES >> %BackLog% & goto end
if errorlevel 4 echo MISMATCHES >> %BackLog% & goto end
if errorlevel 3 echo OKCOPY + XTRA >> %BackLog% & goto end
if errorlevel 2 echo XTRA >> %BackLog% & goto end
if errorlevel 1 echo OKCOPY  >> %BackLog% & goto end
if errorlevel 0 echo No Change >> %BackLog% & goto end
:end  
if errorlevel 8 set Failed=true
if %section%==9th set section=10th
if %section%==8th set section=9th
if %section%==7th set section=8th
if %section%==6th set section=7th
if %section%==5th set section=6th
if %section%==4th set section=5th
if %section%==3rd set section=4th
if %section%==2nd set section=3rd
if %section%==1st set section=2nd
goto %section%

:Testend

[percepts]

are you using compression option when you dump?
Where are you dumping to and is the folder being scanned by real time AV program?
If so then try switching off AV while you do backup or exclude the backup folder from your AV scan.
And check your AV logs to see if anything is happening during the backup.
Also exclude hmailserver/temp from AV scans.

[Lee Thompson]

are you using compression option when you dump?
Where are you dumping to and is the folder being scanned by real time AV program?
If so then try switching off AV while you do backup or exclude the backup folder from your AV scan.
And check your AV logs to see if anything is happening during the backup.
Also exclude hmailserver/temp from AV scans.

1. Yes
2. A different drive/folder and there is no real-time AV running on that machine.


It *looks* like it's some kind of race condition if a message is in the delivery queue at exactly the wrong time.

I've been running hMailServer on this box (which is actually a Hyper-V VM) (with a nightly backup run) for 3 years and I've had this error happen twice (both in January 2015).

[percepts]

I have always wondered about the safety of running a a backup whilst a server is still processing mail in that whilst the backup might appear to work its possible that between the xml extract from DB and backup of data folder, its possible that one or other could change which makes the restore potentially incomplete on one side or the other. i.e. DB entries with no eml or eml with no DB entry.
This probably isn't a problem on a server with low throughput and small volume of data run at night but its still a risk. But when backup starts taking a several minutes and longer the risk gets higher that a mail will be processed during backup which in the event of a restore can lead to loss of data.

Solution is to pause the mailserver during backup. Following mods to startbackup.vbs will do that. i.e. pause mail delivery (oApp.Stop) and resume (oApp.Start) either side of actual backup without actually shutting down hmailserver service.

Code: Select all

' This scripts creates a hMailServer backup using the settings 
' specified in hMailAdmin. You can double-click on the file to
' run the script.
' 
' To use the script you need to update the sAdminPassword
' setting below to reflect your own hMailServer Administrator
' password. If you want to run automatic backups, please 
' start this script using Windows Scheduled Tasks.
'
Const sAdminPassword = "password"

Dim oApp
Set oApp = CreateObject("hMailServer.Application")

' Authenticate the client.
Call oApp.Authenticate ("Administrator", sAdminPassword)

Call oApp.Stop
Call oApp.BackupManager.StartBackup()
Call oApp.Start

Infact I would suggest that hmail startbackup.vbs should do this by default. I know people might not like it but they will like it even less if they find a restore doesn't work or is missing data.
Then if people don't want to have to pause during backup they have to look at sorting out their own backup strategy such as raid mirroring to another box somewhere. What do you think Martin?

[Lee Thompson]

Solution is to pause the mailserver during backup. Following mods to startbackup.vbs will do that. i.e. pause mail delivery (oApp.Stop) and resume (oApp.Start) either side of actual backup without actually shutting down hmailserver service.

I think that's a great idea really.

Archiving an in-use file system (especially one as fluid as a mail server queue and mailboxes) is a tricky proposition in any case so pausing would make a lot of sense.

I agree it should probably pause by default but probably make it an option at the top of the script so those that want it to behave as it has in the past would have that ability.

(Might be even safer to somehow put the mailboxes in a 'read only' mode so POP3/IMAP deletes/changes would either be queued or denied.)

[Lee Thompson]

Hmm I don't think this works.

wscript/cscript returns IMMEDIATELY after issuing Call oApp.BackupManager.StartBackup() so oApp.Start occurs right away and doesn't "lock it" for the duration of the backup job.

[percepts]

The stop and start in the code I gave you does that already. Not only is mail stopped from processing but all connections are blocked too.

The com api stop and start are exact same as using PAUSE/RESUME in your hmailadmin STATUS panel. So you can see what effect they have on your email clients by hitting PAUSE and trying to do something in a client.

[percepts]

It seems you are correct and annoyingly StartBackup() doesn't seem to return anything which can be used as a test to delay the script from contnuing until the backup has finished.
Apologies for giving you a bad steer on this. I thought it worked.

That means shutting down hmailserver service before running backup. See above.

[SorenR]

It seems you are correct and annoyingly StartBackup() doesn't seem to return anything which can be used as a test to delay the script from contnuing until the backup has finished.
Apologies for giving you a bad steer on this. I thought it worked.

That means shutting down hmailserver service before running backup. See above.

Sub OnBackupCompleted() in EventHandler.vbs ??

[percepts]

good thinking I'll try it out.

[percepts]

Yup it works!

Use following for StartBackup.vbs

' This scripts creates a hMailServer backup using the settings
' specified in hMailAdmin. You can double-click on the file to
' run the script.
'
' To use the script you need to update the sAdminPassword
' setting below to reflect your own hMailServer Administrator
' password. If you want to run automatic backups, please
' start this script using Windows Scheduled Tasks.
'

Const sAdminPassword = "hmail password in here"

Dim oApp
Set oApp = CreateObject("hMailServer.Application")

' Authenticate the client.
Call oApp.Authenticate ("Administrator", sAdminPassword)

oApp.Stop

' Wait some time to give hmail time to stop itself
Dim StartTime, Elapsed
Elapsed = 0
StartTime = Now
Do While Elapsed < 10
Elapsed = DateDiff("s", StartTime, Now)
Loop

Call oApp.BackupManager.StartBackup()

AND

in eventhandlers.vbs use following to restart hmailserver. Don't forget to reload scripts.

Sub OnBackupCompleted()
Const sAdminPassword = "hmail admin password in here"

Dim oApp
Set oApp = CreateObject("hMailServer.Application")
Call oApp.Authenticate ("Administrator", sAdminPassword)
oApp.Start

End Sub

Seems to work for me.

[SorenR]

in eventhandlers.vbs use following to restart hmailserver. Don't forget to reload scripts.

Sub OnBackupCompleted()
Const sAdminPassword = "hmail admin password in here"

Dim oApp
Set oApp = CreateObject("hMailServer.Application")
Call oApp.Authenticate ("Administrator", sAdminPassword)
oApp.Start

End Sub

Seems to work for me.

You may want to include... Sub OnBackupFailed(sReason)

[percepts]

what have you used?

[SorenR]

what have you used?

I'm still on 5.4.2 and never had a failure while running automated (hot) backup.

I would imagine "OnBackupFailed" should replicate "OnBackupCompleted", PLUS send a message to Admin/Postmaster with error details.

I really need to move forward and upgrade to latest version
I just need a few changes in the code and never got past "toying" with Windows 10 Pro and Visual Studio 2013 Express

[percepts]

OK after further testing and Sorens suggestion this is a more robust solution:

For StartBackup.vbs use

' This scripts creates a hMailServer backup using the settings
' specified in hMailAdmin. You can double-click on the file to
' run the script.
'
' To use the script you need to update the sAdminPassword
' setting below to reflect your own hMailServer Administrator
' password. If you want to run automatic backups, please
' start this script using Windows Scheduled Tasks.
'

Const sAdminPassword = "password"

Dim oApp
Set oApp = CreateObject("hMailServer.Application")

' Authenticate the client.
Call oApp.Authenticate ("Administrator", sAdminPassword)

If (OApp.ServerState = 3) Then
oApp.Stop
' Wait some time for hmail to stop
Dim StartTime, Elapsed
Elapsed = 0
StartTime = Now
Do While Elapsed < 10
Elapsed = DateDiff("s", StartTime, Now)
Loop
End If

Call oApp.BackupManager.StartBackup()

In eventhandlers.vbs use:

Sub OnBackupFailed(sReason)
Const sAdminPassword = "passwordl" ' you admin password in here

Dim oApp
Set oApp = CreateObject("hMailServer.Application")
Call oApp.Authenticate ("Administrator", sAdminPassword)

If (OApp.ServerState = 1) Then
oApp.Start
End If
End Sub

Sub OnBackupCompleted()
Const sAdminPassword = "password" ' you admin password in here

Dim oApp
Set oApp = CreateObject("hMailServer.Application")
Call oApp.Authenticate ("Administrator", sAdminPassword)

If (OApp.ServerState = 1) Then
oApp.Start
End If

End Sub

[SorenR]

Just a suggestion, this is the "Timer" I use in my EventHandler...

Code: Select all

Public Function Wait(sec)
   Dim t : t = Timer
   Do While (Timer - t) < sec
   Loop
End Function

Sub ....
   ...
   ...
   wait(1)      ' wait 1 second
   ...
   ...
End Sub

[percepts]

what hapens if it spans midnight?

[SorenR]

what hapens if it spans midnight?

Hmm.... Good question, never gave it a thought.
When I Googled for timers it seemed to be the most used alternative to WScript.Sleep(1000) ...

I use it for my logging as part of the routine that make writing to a single file multithreaded.

Oh well... Back to the drawing board.

[percepts]

well now you know why I used:

Dim StartTime, Elapsed
Elapsed = 0
StartTime = Now
Do While Elapsed < 10
Elapsed = DateDiff("s", StartTime, Now)
Loop

[SorenR]

well now you know why I used:

Dim StartTime, Elapsed
Elapsed = 0
StartTime = Now
Do While Elapsed < 10
Elapsed = DateDiff("s", StartTime, Now)
Loop

Fixed it... Just had to "Xor ((Timer - t) < 0)" to catch midnight.

[percepts]

sometimes you can't see the wood for the trees. Just putting:

Wscript.Sleep(10000)

into the script does what I/you want. No need of fancy code to manually do it.

And I tried that but I think I must have had it on 1 second so didn't see that it worked when I was testing.

[SorenR]

sometimes you can't see the wood for the trees. Just putting:

Wscript.Sleep(10000)

into the script does what I/you want. No need of fancy code to manually do it.

And I tried that but I think I must have had it on 1 second so didn't see that it worked when I was testing.

Wscript.Sleep does not work from inside EventHandlers.vbs due to hMail not using a command shell, but yes... I failed to see that one too..

[funcoolio]

Hi,
recently i upgraded from v 5.4 to 5.6.2-B2234
when backup starts i get this error:

Severity: 2 (High), Code: HM4208, Source: ExceptionHandler::Run, Description: An error occured while executing 'Task-BackupTask', Error code: 80, Message: boost::filesystem::copy_file:...........

I've noticed that this error comes out only if the destination directory is not empty (or if the file being backed up already exists in the destination directory).
When this error comes out subsequents backups fail with:

"Severity: 1 (Critical), Code: HM5014, Source: BackupManager::OnBackupFailed, Description: BACKUP ERROR: Backup or restore operation is already started"

an "hmail server" service restart is needed to let the backup starts again (will fails again if the destination dir is not empty).

The problem seems to disappear if the destination directory is empty (or changed to another one), but i need to perform more tests.

[jimimaseye]

(or if the file being backed up already exists in the destination directory)

Interesting. There are only 2 methods you should be backing up with:

1, with 'Domains' and/or 'Settings' ONLY or
2, 'Domains' and/or 'Settings' + Messages....WITH COMPRESSION

Now if you do (1) then you get a "hMailserverbackup.xml" file temporarily written out which is then zipped to an original-named zip file timestamped to the SECOND in the destination directory and the original temporary mailserverbackup.xml is deleted.

If you do (2) then as well as (1) the messages are copied to a temporary directory, then everything is zipped to the timestamped zip file and the temporary data messages are deleted.

Given that in both cases the zip file in the destination directory is created with a new timestamp, at what point are you having the scenario that matches: "if the file being backed up already exists"? WHAT file is existing??

[funcoolio]

Hi,
it seems that the error comes out as soon as the procedure tries to backup the first eml file:

"C:\Program Files (x86)\hMailServer\Data\domain\user\98\{98CD6371-65ED-45C0-8E00-4CFD381C2EDE}.eml", "C:\hmailbackup\backup\DataBackup\domain\user\98\{98CD6371-65ED-45C0-8E00-4CFD381C2EDE}.eml""


i forgot to mention that with version 5.4 this error does not appear

i am backing up settings, domains and messages, compress files is not checked

[jimimaseye]

i am backing up settings, domains and messages, compress files is not checked

And there is your problem.

a, ok, you have identified it is causing you a minor problem now but
b, EVEN IF IT 'worked' this backup would be no good to you.

If you DO NOT compress the messages, then you WILL NOT BE ABLE to restore from this backup. Fullstop. (It is a known bug/feature/non-sensical garbage, but netertheless it is fact. Read this thread fro the beginning. Trust me on this....or try a restore for yourself and see. )

So.... COMPRESS your messages and solve both problems........(or DOES it??? )

[funcoolio]

ok, i will give it a try and let you know
thanks for your support

[funcoolio]

Backup problem solved! Many thanks, when reading all the posts i did not realized that compressing is necessary (generally i speak/read/write a different language than english)

Later i will also check restore process (never had the need to perform it .....)

thanks again for you support

[jimimaseye]

Youre welcome.

Remember: it is always a good idea to test your restore procedures BEFORE actually needing them in a live environment. Your are lucky that this 'problem' came along and actually highlighted a fundamental error in your procedures.

[Lee Thompson]

There still seems to be an issue with this crashing periodically. Fortunately, it's not occurring very often.

While the backup is taking place, users can check/delete mail, the server can receive mail etc, and that's all good, however, hMailServer's backup function throws an exception if a file it thought it was going to copy gets deleted in the meantime.

e.g.

Code: Select all

"ERROR"	908	"2015-05-18 00:09:23.521"	"Severity: 2 (High), Code: HM4208, Source: ExceptionHandler::Run, Description: An error occured while executing 'Task-BackupTask', Error code: 2, Message: boost::filesystem::copy_file: The system cannot find the file specified: "C:\Program Files (x86)\hMailServer\Data\{74D91783-AAC1-444E-85E2-D48D54CDABC3}.eml", "D:\Private\Backups\hMailServer\DataBackup\{74D91783-AAC1-444E-85E2-D48D54CDABC3}.eml""

(NOTE: occurred is misspelled in the error message.)

This is further compounded because:

• The DataBackup folder (and it's contents) is left in the backup directory.
• Future backup attempts fail because it still thinks backup is running. (BACKUP ERROR: Backup or restore operation is already started).

I would suggest perhaps the following ideas for a solution:

• During the data directory backup, basically 'pause' the system. Perhaps still answer on the ports but with a 400 Server Busy or something. Unfortunately this can be a 10+ minute window depending on the volume of data.
• Use Windows Shadow Copy to copy the data files.
• Instead of throwing an exception if the file can't be found when it actually does the copy_file function, just log it and skip it.
• This seems to almost always be the transient message queue, perhaps exclude it from backup? Not ideal but..

It would also probably be a good idea if an exception is thrown to log the event, clean up the temporary files, reset the backup status and alert the administrator (e-mail, popup, foghorn, etc). (Or make this an option.)


v5.6.3-B2249

[mattg]

@Lee

Is that using the scripts presented in this thread, or using the built-in backup functionality?

[Lee Thompson]

I'm using the built-in function that's triggered by the included StartBackup.vbs script in Addons\Utilities.

[Lee Thompson]

I've written a batch file to trigger hMailServer backups and recover if the backup process has crashed on the prior run as a workaround.

I thought I'd share it in case other people are running into this issue.

The script, on my system (Windows Server 2008 R2), is called "backup_hmailserver.bat". It's launched by Task Scheduler at 12:05AM each morning.


Overview

If it sees the DataBackup folder in hMailServer's backup target folder, it will initiate a recovery process which consists of removing that folder and it's contents and re-start hMailServer service.

In either case, it then runs the StartBackup.vbs script.

The script logs it's own output and the vbs interpreter output to a file.


Configuration:

You will need to edit set backupfolder= and set hmailpath= to match your installation.

By default, logs will go into hMailServer's log folder, but you can change this by altering set log_base=.

Code: Select all

@echo off
setlocal

set script_ver=201505240729
set fname=Backup hMailServer
set sname=backup_hmailserver

REM	Options

set backupfolder=C:\Backups\hMailServer
set hmailpath=C:\Program Files (x86)\hMailServer

REM	Logging

set log_separator=--------------------
set log_base=%hmailpath%\logs
set log=%log_base%\%sname%.log
set lastrun=%log_base%\%sname%-lastrun.log

REM	Tools

set tool_service_start=net start
set tool_service_stop=net stop
set tool_removedirectory=rmdir

REM	Service Name

set servicename=hmailserver

REM	Temp Backup Folder 

set tempfolder=DataBackup

REM	StartBackup Script Options and Locations

set scriptpath=%hmailpath%\Addons\Utilities
set scriptengine=cscript.exe
set scriptengineopt=//B
set scriptname=StartBackup.vbs
set engineerror=0

REM	Initialize

if exist "%log%" echo %date% %time% %log_separator% >> "%log%"

echo %date% %time% %fname% %script_ver% Initialized >> "%log%"

if exist "%backupfolder%\%tempfolder%" call :recover_backup

call :run_backup

:end
echo %date% %time% %fname% Exiting >> "%log%"
goto :eof

:recover_backup
echo %date% %time% %fname% Previous Backup Process Crashed >> "%log%"
echo %date% %time% %fname% Cleanup >> "%log%"
echo %date% %time% %fname% Deleting Temporary Files >> "%log%"
%tool_removedirectory% /S /Q "%backupfolder%\%tempfolder%"
echo %date% %time% %fname% Restarting Service >> "%log%"
%tool_service_stop% %servicename%
%tool_service_start% %servicename%
echo %date% %time% %fname% Finished Recovery >> "%log%"
exit /b 1

:run_backup
echo %date% %time% %fname% Starting Backup >> "%log%"
echo %date% %time% %fname% Running %scriptname% >> "%log%"
echo %date% %time% %fname% Script Engine: %scriptengine% >> "%log%"
echo %date% %time% %fname% Script File: %scriptpath%\%scriptname% >> "%log%"
set t_dir=%cd%
cd /d "%scriptpath%"
echo %date% %time% %fname% start %scriptname% > "%lastrun%"
"%scriptengine%" %scriptengineopt% "%scriptname%" >> "%lastrun%"
set engineerror=%ERRORLEVEL%
echo %date% %time% %fname% finish %scriptname% (%engineerror%) >> "%lastrun%"
echo %date% %time% %fname% Script Exited with %engineerror% >> "%log%"
cd /d "%t_dir%"
exit /b 1

[prisma]

Guys, I'm not able to understand why we have to write batches and patches, do tricks, read hints about bugs, write post after post.

This software has a build in backup and should be able to do a hot backup compressed, not compressed or completely without messages. But it is not able to. This should be improved. I'd test it and I'd help whereever my help is wanted. But I'm not able to code it. I'm telling this anybody everytime. But nobody seems to listen...

[jimimaseye]

@Lee

Well done on your script and thanks. FYI (and others) I also did one here: viewtopic.php?f=21&t=28139

[Lee Thompson]

Code: Select all

hMailServer 5.6.3-B2249
Windows Server 2008 R2 (Standard Edition)
MSSQL CE

Well, my workaround batch file to clean up when hMailServer's backup job crashes is working, however, the exceptions do seem to be occurring more often than you might expect. I have the backup scheduled to run the nightly at 5 minutes past midnight and through the month of July, the backup job has crashed 4 times. This is a little alarming because my mail server is very low traffic. (It's for me, some friends, and some family members.) So far it's always a message that has obviously been deleted or delivered by the time hMailServer gets to the file.


I think this is a pretty serious bug actually as it basically leaves hMailServer in a state where it can no longer run the backup job and just restarting the service won't fix it because the temporary files are all still in the backup location and hMailServer will throw a fit if it tries to run the backup job.

In my opinion, the options for a fix could be several things:

1. Pause the queue at the start of the backup job, and resume at the end.
2. Use VSS service to do a shadow copy.
3. Simply move on if a file isn't there instead of throwing an exception. (Perhaps log a warning with the pathname.)

Frankly, the last option is the best one as even if #1 and #2 are both implemented, this problem could still occur. Pausing the queue could have it's own issues if for some reason it didn't resume.


(The data storage location should also not be under Program Files but that's another issue entirely. Technically it should be under %ProgramData% or an administrator-specified path.)

Code: Select all

"ERROR"	4108	"2015-07-02 00:11:41.569"	"Severity: 2 (High), Code: HM4208, Source: ExceptionHandler::Run, Description: An error occured while executing 'Task-BackupTask', Error code: 2, Message: boost::filesystem::copy_file: The system cannot find the file specified: "C:\Program Files (x86)\hMailServer\Data\{E3C00A02-1FB3-4565-9745-852A2EAABCAF}.eml", "D:\Private\Backups\hMailServer\DataBackup\{E3C00A02-1FB3-4565-9745-852A2EAABCAF}.eml""

"ERROR"	4620	"2015-07-15 00:16:26.363"	"Severity: 2 (High), Code: HM4208, Source: ExceptionHandler::Run, Description: An error occured while executing 'Task-BackupTask', Error code: 2, Message: boost::filesystem::copy_file: The system cannot find the file specified: "C:\Program Files (x86)\hMailServer\Data\{23FA5CFB-E544-45AA-A897-132A9E86FC34}.eml", "D:\Private\Backups\hMailServer\DataBackup\{23FA5CFB-E544-45AA-A897-132A9E86FC34}.eml""

"ERROR"	3632	"2015-07-23 00:09:38.716"	"Severity: 2 (High), Code: HM4208, Source: ExceptionHandler::Run, Description: An error occured while executing 'Task-BackupTask', Error code: 2, Message: boost::filesystem::copy_file: The system cannot find the file specified: "C:\Program Files (x86)\hMailServer\Data\{659C4F17-545D-489A-A175-F3CAF680C77D}.eml", "D:\Private\Backups\hMailServer\DataBackup\{659C4F17-545D-489A-A175-F3CAF680C77D}.eml""

"ERROR"	3412	"2015-07-30 00:10:49.545"	"Severity: 2 (High), Code: HM4208, Source: ExceptionHandler::Run, Description: An error occured while executing 'Task-BackupTask', Error code: 2, Message: boost::filesystem::copy_file: The system cannot find the file specified: "C:\Program Files (x86)\hMailServer\Data\{7E554CD7-D87E-4905-A44E-AAF3550DDD43}.eml", "D:\Private\Backups\hMailServer\DataBackup\{7E554CD7-D87E-4905-A44E-AAF3550DDD43}.eml""

[jimimaseye]

I physically stop and start the service (rather than pause it) in my script to prevent this happening, and the service is offline for about 30 seconds whilst it does an MIRROR update of the files and takes a dump of MySQL database. I think 30 seconds in the dead of night is is far more acceptable (and highly unlikely to be a problem) than taking alternative methods which do cause this missing file problem.

[Lee Thompson]

I physically stop and start the service (rather than pause it) in my script to prevent this happening, and the service is offline for about 30 seconds whilst it does an MIRROR update of the files and takes a dump of MySQL database. I think 30 seconds in the dead of night is is far more acceptable (and highly unlikely to be a problem) than taking alternative methods which do cause this missing file problem.

I thought about doing it that way as well. I kinda wish I had gone with a MySQL install since it's easier to database dump that than MSSQL CE.

Still, my point is that hMailServer's backup function should just handle it without us having to work around it. It's overall such an excellent program, it's disappointing that this function is pretty much busted.

[mattg]

you do realise that the built in MS SQL CE is intended only for very small home installations, and it is against the EULA of that product to use this database in a commercial environment.

If the builtin hmailserver backup works, to change database you simply need to:-
1. back up (compressed including all messages)
2. installing the new database type (ie MySQL)
3. running the DBsetup.exe to create the tables and modify the hmailserver.ini
4. restore the backup

[Lee Thompson]

you do realise that the built in MS SQL CE is intended only for very small home installations, and it is against the EULA of that product to use this database in a commercial environment.

If the builtin hmailserver backup works, to change database you simply need to:-
1. back up (compressed including all messages)
2. installing the new database type (ie MySQL)
3. running the DBsetup.exe to create the tables and modify the hmailserver.ini
4. restore the backup

This is a home. My wife and I use it and a couple of friends and family members. I have a total of 6 email accounts on hMailServer.

(I have a windows domain setup under MSDN OS because I do a lot of programming/IT work as a consultant and stay in practice that way. Well, and I'm a geek.)

But thanks for the instructions -- might switch it to MySQL.


But anyway, the backup function shouldn't be crashing out if it can't find a file that's been deleted/moved from normal routing operations.

[mattg]

But anyway, the backup function shouldn't be crashing out if it can't find a file that's been deleted/moved from normal routing operations.

Agreed....

btw I'm not the police; I don't care how you use hmailserver (as long as you aren't a spammer - then I care). I was trying to say that a small install with only a few accounts would be unlucky to experience the backup trying to handle a mail message that is in the delivery queue in the few seconds taken to do the backup. The mini dump is new, so maybe this has been happening a while and no-one ever noticed, because the error message doesn't say that backup stops and won't re-start.

[Lee Thompson]

btw I'm not the police; I don't care how you use hmailserver (as long as you aren't a spammer - then I care). I was trying to say that a small install with only a few accounts would be unlucky to experience the backup trying to handle a mail message that is in the delivery queue in the few seconds taken to do the backup. The mini dump is new, so maybe this has been happening a while and no-one ever noticed, because the error message doesn't say that backup stops and won't re-start.

Oh phew (Believe me I'm not a spammer, I *&#$*( hate spam.)

That's what struck me too, if it's happening on my tiny little server... then this could be a potential serious issue and it can be hard to even tell it's failing unless you're really paying attention. When the backup job tries to run again in the crashed state, it just logs that it's already running.

If you need the dmp files I can get them to you.


(By the way, I did used to work on a commercial mail server from about 1996-2001. We bought it from another developer, it was a 16-bit application mode and we built it into a 32-bit service and the code was an utter mess. A lesson in the importance of due diligence.)

[Lee Thompson]

Just a follow up.

The backup function is still crashing periodically but not as often:

Code: Select all

minidump_2015-10-16 001443_{5E617E59-8A32-48B4-9C07-0A42BC782381}
minidump_2015-12-01 001624_{7C8AB35F-4DBC-4D26-BFDB-5C64129B2093}
minidump_2015-12-04 001730_{75845246-1D84-46FF-A3DD-58C3C81CA3D9}

My batch file now cleans up if there was a crash so I just miss one night's backup, worst case.

v5.6.4-B2283