Best backup and recovery practice for hMS ?

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
melto
New user
New user
Posts: 9
Joined: 2017-01-10 16:45

Best backup and recovery practice for hMS ?

Post by melto » 2017-05-10 12:16

Hi,

I see hMS does a fantastic job dealing with emails, my main concern now is with the ability to quickly restore/fix it if it does fall over and die, may be power surge or dead OS ending with corruption.
I read that the internal backup can only deal with database size less than 1GB, as this is very tiny compare to actual usage today, this is not going to work in most cases.

I normally would install some form of imaging software such as Acronis to backup the whole drive, but as this is normally done once a day in the evening, say there is a problem at 5pm and the server died, I can restore to yesterday but all emails from yesterday to 5pm today are not part of the backup, how do I extract these emails and inject them into the server that's restored from yesterday ? or is there a simple way of just copying the whole SQL folder(s) and then place them directly over the backup from yesterday ?

With Exchange I can extract the edb manually per accounts but before I start using hMS I would really like to know about recovery and repairs before I am stuck.

Thanks everyone for your help.

User avatar
jimimaseye
Moderator
Moderator
Posts: 8094
Joined: 2011-09-08 17:48

Re: Best backup and recovery practice for hMS ?

Post by jimimaseye » 2017-05-10 12:19

What version of database are you using - internal or external?
I read that the internal backup can only deal with database size less than 1GB
This is incorrect. it is 15GB in the latest versions and the restriction is completely removed in upcoming 5.7. The 15GB refers to the DATA directory size.

FYI: I do a nightly backup of data (across to a LAN storage disk and to offiste backup). But I also have all my incoming emails come in to an external collecting mail server of which I then pickup via external download with a 'delete from server' of 30 days. (I only send out via direct smtp). This serves as a backup for me in that if there is a hmailserver total failure any emails that come in between 'last nights backup' and full system restoration will still be there and come in again after restoration thereby losing no imcoming emails at all. (Admittedly I will lose any record of SENT emails since backup - not such a big deal because what is important is that at least the recipients of those emails will already have had them).
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

melto
New user
New user
Posts: 9
Joined: 2017-01-10 16:45

Re: Best backup and recovery practice for hMS ?

Post by melto » 2017-05-10 13:37

Thank you for all your help as always.

My other questions is that should the server hardware died. but the drives are ok, can I somehow extract the SQL database/email files off the drive and inject them into a new database or the existing database restored from the backup ?

User avatar
mattg
Moderator
Moderator
Posts: 20025
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Best backup and recovery practice for hMS ?

Post by mattg » 2017-05-11 01:06

melto wrote:My other questions is that should the server hardware died. but the drives are ok, can I somehow extract the SQL database/email files off the drive and inject them into a new database or the existing database restored from the backup ?
Depends on the database used, but yes

There is a DataDirecttorySynchroniser that will go through all accounts and add mail not in the database to the inbox of each account, so as a worst case scenario, if you can access the file storage where the data directory is then you can at least put all messages back into the inbox.

You can use a network storage device for Data Directory if you have access to a SAN or NAS, and you can even use a database hosted on another machine. Large setups have a machine to run the hMailserver service, a machine to host the database and a Storage device for the data directory.

(By large we are talking 100 000's messages per day, with 1000's of domains being hosted)
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

2014peter2014
New user
New user
Posts: 12
Joined: 2014-04-20 18:27

Re: Best backup and recovery practice for hMS ?

Post by 2014peter2014 » 2017-07-02 12:35

I am having a back up error. Using 5.6.6B2383
The backup hangs after one successful back.
On the second attempt it hangs at the data directory and no other backups will occur unless the computer is rebooted.
Any ideas??
2017-07-01 16:53:24.991 Backing up data directory...
2017-07-01 16:54:20.086 Backup started
2017-07-01 16:54:20.091 BACKUP ERROR: Backup or restore operation is already started
2017-07-02 20:30:24.369 Backup started
2017-07-02 20:30:24.385 BACKUP ERROR: Backup or restore operation is already started

User avatar
jimimaseye
Moderator
Moderator
Posts: 8094
Joined: 2011-09-08 17:48

Re: Best backup and recovery practice for hMS ?

Post by jimimaseye » 2017-07-02 13:29

When you say it "hangs", why do you say this? Backups can take a while to complete and you simply dont give it enough time.

I suspect you havent waited long enough for it to complete and then subsequent attempts now leave you with this error because it has the residue from the (now forcefuly aborted) previous backup attempt. So:

1, go to you Backup destination directory and empty it completely
2, open Task Manager
3, Start a new backup
4, watch taskmanager for the subprocess of hmailserver.exe (in particular the '7z' program if you are compressing) and watch and wait patiently. To give you an idea of time: On my system a 9gb data directory can take 50 minutes to complete the backup.
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

2014peter2014
New user
New user
Posts: 12
Joined: 2014-04-20 18:27

Re: Best backup and recovery practice for hMS ?

Post by 2014peter2014 » 2017-07-03 10:00

Hi Will have a go testing as you say but..

1. My database has 7 email addresses and 5 emails only with no attachments
2. The first time I do a back up it takes all off 20-30 seconds to complete.

2014peter2014
New user
New user
Posts: 12
Joined: 2014-04-20 18:27

Re: Best backup and recovery practice for hMS ?

Post by 2014peter2014 » 2017-07-05 15:49

Is there anyway to actually cancel the backup run without rebooting?

2014peter2014
New user
New user
Posts: 12
Joined: 2014-04-20 18:27

Re: Best backup and recovery practice for hMS ?

Post by 2014peter2014 » 2017-07-30 09:32

Backups no longer work at all even after a reboot.
The back up starts but stalls at "backing up data directory"

Even after days

User avatar
Dravion
Senior user
Senior user
Posts: 1423
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: Best backup and recovery practice for hMS ?

Post by Dravion » 2017-07-30 10:24

Do you have any Log entries or Error Messages?

helps us to solve the problem if you can provide as much technical


Please answer the following questions as best you can:

If you run backup from hMailadmin:
*Did the Program freeze, can you do other tasks while backup is running?
*Can you quit the Program the normal way or do you have to kill it with the Windows Taskmanager?
*Did the program crash or do you get a specific Error Message?
*Do you use a Networkshare or a local Harddrive for as Data and/or DataDirectory?

2014peter2014
New user
New user
Posts: 12
Joined: 2014-04-20 18:27

Re: Best backup and recovery practice for hMS ?

Post by 2014peter2014 » 2017-08-07 10:47

Log entries as below ( modded the account name for privacy)

2017-07-30 17:42:14.190 Backup started
2017-07-30 17:42:14.205 Loading backup settings....
2017-07-30 17:42:14.205 Backing up domains...
2017-07-30 17:42:14.221 Backing up account cp.local...
2017-07-30 17:42:14.236 Backing up account .local...
2017-07-30 17:42:14.252 Backing up account .local...
2017-07-30 17:42:14.268 Backing up account j.local...
2017-07-30 17:42:14.283 Backing up account .local...
2017-07-30 17:42:14.299 Backing up account .local...
2017-07-30 17:42:14.299 Backing up account local...
2017-07-30 17:42:14.314 Backing up account p.local...
2017-07-30 17:42:14.330 Backing up account pp.local...
2017-07-30 17:42:14.330 Backing up account rp.local...
2017-07-30 17:42:14.377 Backing up account ocal...
2017-07-30 17:42:14.533 Backing up account ocal...
2017-07-30 17:42:14.548 Backing up account local...
2017-07-30 17:42:14.548 Backing up account .local...
2017-07-30 17:42:14.564 Backing up data directory...
2017-08-07 18:42:10.964 Backup started
2017-08-07 18:42:10.964 BACKUP ERROR: Backup or restore operation is already started


Program works fine, no errors, no hanging,
Local drive (default) for install
Local drive for the back up as well.
The back up just doesnt end.
See the log after 7 days!

User avatar
mattg
Moderator
Moderator
Posts: 20025
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Best backup and recovery practice for hMS ?

Post by mattg » 2017-08-07 11:55

What are the paths set in the .ini file?
What options are you selecting when doing a backup?

I think that the hmailserver /tmp/ directory is used to create the backup. Can you confirm that this folder is empty and normal Windows permissions apply to that folder...
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jimimaseye
Moderator
Moderator
Posts: 8094
Joined: 2011-09-08 17:48

Re: Best backup and recovery practice for hMS ?

Post by jimimaseye » 2017-08-07 22:07

In fairness I have seen this behaviour before a couple of times. In any case, I actually conneced to the system to see it for myself and confirmed that it was doing as reported here (backup started, never ends, but the backup process terminates without reason or clue).

It confused me too and after investigation it was down to some sort of database corruption I think - I deleted one specific account and then things worked ok. In another case I was unable to get to the end of the investigation or resolve the [rpblem as the system it was running on was seriously under-powered and starting the backup process simp[ly sent the CPU to nearly 100%. I presume it was a similar thing - that being some sort of data integrity/corruption.
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

storpotaten
New user
New user
Posts: 5
Joined: 2017-08-19 11:33

Re: Best backup and recovery practice for hMS ?

Post by storpotaten » 2017-08-19 14:31

Anyone who has solved this issue yet?

I have the same problem with win server 2012 R2,
works fine with win server 2008 R2.

// Mike

User avatar
jimimaseye
Moderator
Moderator
Posts: 8094
Joined: 2011-09-08 17:48

Re: Best backup and recovery practice for hMS ?

Post by jimimaseye » 2017-08-19 15:15

I don't think our is anything to do with the OS but is data specific. If you backup and restore the database to 2008 does it all stall?
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

storpotaten
New user
New user
Posts: 5
Joined: 2017-08-19 11:33

Re: Best backup and recovery practice for hMS ?

Post by storpotaten » 2017-08-19 17:48

I have only backed up on 2008 an it doesn´t stall.

User avatar
jimimaseye
Moderator
Moderator
Posts: 8094
Joined: 2011-09-08 17:48

Re: Best backup and recovery practice for hMS ?

Post by jimimaseye » 2017-08-19 20:25

And restoring the 2008 to the 2012 in full, does the 2012 then fail to backup itself?
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

storpotaten
New user
New user
Posts: 5
Joined: 2017-08-19 11:33

Re: Best backup and recovery practice for hMS ?

Post by storpotaten » 2017-08-19 21:15

Yes!

User avatar
mattg
Moderator
Moderator
Posts: 20025
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Best backup and recovery practice for hMS ?

Post by mattg » 2017-08-19 22:18

interesting

Can you check the database for the 2012 machine and see which is the last domain and account added to the database?
if all there, then check message folders on disk inside each account - there should be up to 256 XX folders per account...

Also what type of databases on 2008 and 2012?
What encoding is set for each table?

Perhaps this is something to do with emojies or some non-latin characters...
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

storpotaten
New user
New user
Posts: 5
Joined: 2017-08-19 11:33

Re: Best backup and recovery practice for hMS ?

Post by storpotaten » 2017-08-20 09:53

Don´t know which domain or account created last but i don´t think anything is missing.
Some account´s contains less than 256 folders.
2008: Mysql 5.1.6.8, 2012: Mysql 5.7.19
Encoding:utf8_general_ci

// Mike

storpotaten
New user
New user
Posts: 5
Joined: 2017-08-19 11:33

Re: Best backup and recovery practice for hMS ?

Post by storpotaten » 2017-08-20 10:11

I found some account folders in the data directory that didn´t exist in hmailserver,
i deleted these and the backup worked.
Don´t know if that was the cause or if it was just luck.

// Mike

User avatar
mattg
Moderator
Moderator
Posts: 20025
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Best backup and recovery practice for hMS ?

Post by mattg » 2017-08-20 10:25

That may well be the cause

Folders are normally deleted when an account is removed
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jimimaseye
Moderator
Moderator
Posts: 8094
Joined: 2011-09-08 17:48

Re: Best backup and recovery practice for hMS ?

Post by jimimaseye » 2017-08-20 10:37

storpotaten wrote:I found some account folders in the data directory that didn´t exist in hmailserver,
i deleted these and the backup worked.
Don´t know if that was the cause or if it was just luck.

// Mike
Test.

Remove all domains and delete the data directory contents on the 2012 server. Then restore from the 2008 backup.

Only directories relating to the restored accounts will now exist.

Then try the backup again.

If your observation is right then it could potentially mean this problem would be more common if the restore process is not emptying the data directory first.
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
jimimaseye
Moderator
Moderator
Posts: 8094
Joined: 2011-09-08 17:48

Re: Best backup and recovery practice for hMS ?

Post by jimimaseye » 2017-08-20 11:43

Hmmmm....

I was unable to reproduce it. Various tests and combinations of creating (from a copy of an existing) a defunct account (eg, "user2notused") and then backing up and restoring with it. The defunct account data also gets restored despite not being referenced in the database. (This is expected as I beloeve the backup procedure is a simply COPY of data directory and everything within it and then zipped) and the restore will be a reversal).

However, I was able to break the restore at seemingly random intervals. I found often if I clicked RESTORE it would restore correctly but then clicking restore immediately again it would then fail (simply stopping without warning with "Restoring data directory..." and leading to future "BACKUP ERROR: Backup or restore operation is already started".) Only stopping and restarting the service would allow the restore to work correctly again. BUT THEN, just as I though I had nailed a specific set of actions to always replicate a problem, I did another backup of the same data and tried the actions again and then couldnt reproduce it at will (only managing to reproduce it randomly). And remeber, the backup file contents are identical.

So there is something temperamental, this I am sure of. But understanding what and why is tricky. But I wouldnt be surprised if this sporadic and unidentified condition isnt the same cause of the backup failures too.
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
mattg
Moderator
Moderator
Posts: 20025
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Best backup and recovery practice for hMS ?

Post by mattg » 2017-08-20 23:35

I was thinking the user was reporting extra folders on the machine being backed up, not on the machine being restored to
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jimimaseye
Moderator
Moderator
Posts: 8094
Joined: 2011-09-08 17:48

Re: Best backup and recovery practice for hMS ?

Post by jimimaseye » 2017-08-20 23:41

Yes. I did a range of tests including that. I couldn't get the backup to fail.
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

2014peter2014
New user
New user
Posts: 12
Joined: 2014-04-20 18:27

Re: Best backup and recovery practice for hMS ?

Post by 2014peter2014 » 2017-08-21 11:56

Hi,
My problem was a rather simple fix , but still don't have an explanation for the problem cause.

Because my Database is so small (takes under 60seconds to do a backup) I was using uncompressed backup. This allowed an initial backup but always got "stuck" when doing a second back up requiring a reboot.

If I choose to compress then each backup is created as a new unique named file and the backup goes through with no errors.

After experimentation, I can do repeated uncompressed back up but have to nominate a new empty folder every time otherwise it stalls.

Hope this helps someone else. (using windows 7)

Cheers

User avatar
mattg
Moderator
Moderator
Posts: 20025
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Best backup and recovery practice for hMS ?

Post by mattg » 2017-08-22 01:28

I've also long suspected that uncompressed restores are broken as well. I recall that hmailsevrer will ONLY restore from a compressed backup.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jimimaseye
Moderator
Moderator
Posts: 8094
Joined: 2011-09-08 17:48

Re: Best backup and recovery practice for hMS ?

Post by jimimaseye » 2017-08-22 09:01

2014peter2014 wrote:Hi,
My problem was a rather simple fix , but still don't have an explanation for the problem cause.

Because my Database is so small (takes under 60seconds to do a backup) I was using uncompressed backup. This allowed an initial backup but always got "stuck" when doing a second back up requiring a reboot.
This is because that effectively you are creating a backup folder called 'DataBackup' (which is a copy of your data folder) in your nominated backup directory. Trying to run another to the same finds conflict and it is unable to handle the existing folder (ie, it doesnt seem to overwrite or delete it first). In my tests stopping and starting of the service doesnt make a difference.

Logged, more info: https://github.com/hmailserver/hmailserver/issues/234
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
jimimaseye
Moderator
Moderator
Posts: 8094
Joined: 2011-09-08 17:48

Re: Best backup and recovery practice for hMS ?

Post by jimimaseye » 2017-08-22 09:07

mattg wrote:I recall that hmailsevrer will ONLY restore from a compressed backup.
Just tested. A restore of all (including messages) from an uncompressed backup is fine.
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

Snatch
New user
New user
Posts: 2
Joined: 2017-10-01 14:58

Re: Best backup and recovery practice for hMS ?

Post by Snatch » 2017-10-11 16:30

i have the same problem. backup hangs after one successful back.

User avatar
mattg
Moderator
Moderator
Posts: 20025
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Best backup and recovery practice for hMS ?

Post by mattg » 2017-10-11 22:44

Snatch wrote:i have the same problem. backup hangs after one successful back.
Please start a new thread and link to this if you think it relevant as per forum rules
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

anoshwadia
New user
New user
Posts: 2
Joined: 2019-08-17 18:05

Re: Best backup and recovery practice for hMS ?

Post by anoshwadia » 2019-08-17 18:19

Just started using HmailServer yesterday and have been running into the same error where the backup logs would get stuck on:
Backing up data directory...

In my case it turns out that my backup directory had a subfolder called Databackup. Apparently the backup method uses this subfolder to create the backup files, zip them up and then deletes the folder. Deleting the Databackup folder seemed to fix the problem. I'm guessing if a backup fails or gets stuck, that temporary folder lingers around and causes subsequent backups to fail. Hope this helps someone else!

Anosh

Post Reply