5.4 Backup feature

Use this forum if you want to discuss a problem or ask a question related to a hMailServer beta release.
Post Reply
westminster
Normal user
Normal user
Posts: 31
Joined: 2011-10-28 17:00

5.4 Backup feature

Post by westminster » 2012-02-15 23:19

Are the backups created with the new experimental facility in 5.4 something that can be restored to a 5.3 installation? The bigger backup and restore feature would be a BIG help. I'm on the verge of converting a 3GB database from MSSQL CE to MySQL and would really like to just use the HMA built-in Backup & Restore.

What is the expected time frame for when the release version of 5.4 will be out?

Thanks!

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: 5.4 Backup feature

Post by ^DooM^ » 2012-02-16 01:16

They may be compatible but I wouldn't count on it. Best way is to setup a test environment and try it. Virtualbox is free and easy to setup VM's in.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: 5.4 Backup feature

Post by Bill48105 » 2012-02-16 17:52

Since you should only restore to exact same version the backup was from not sure it matters.. (IE- you restore 5.3.3 backup to 5.3.3 & upgrade to 5.4 if needed etc)

No time frame on official 5.4 release, waiting on martin & he's been too busy with work/life.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

prisma
Senior user
Senior user
Posts: 310
Joined: 2010-07-09 13:16

Re: 5.4 Backup feature

Post by prisma » 2012-02-20 11:12

westminster wrote:I'm on the verge of converting a 3GB database from MSSQL CE to MySQL
Hey Bill,

I'm a little bit confused:

1. I always thought the 2GB limitation of the built-in backup is concerning the total amount of database+filefolder respectively the packed size of both. Is this wrong? What is right?
2. I read in this forum about the switch to 7zip to improve the build-in backup. A little later you wrote the limitation exists anyway. What's the current status?

Cheers!

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: 5.4 Backup feature

Post by Bill48105 » 2012-02-20 18:04

prisma: The limitation was in place because of ZIP limitations (issues after 2GB) so if you had over 1.5G of messages built-in backup would ABORT & refuse to run. Once martin changed the backup to use 7z there was no physical restriction to keep the 2G (1.5G) limit in EXCEPT as a sanity check so I modified hmail 5.4 to LOG if over 1.5G of messages was being backed up & STOP backups if it was > 15G. So yes the restriction is still there but it has been increased to a level not likely to be commonly reached (and at that level one is likely using a different method to backup). The idea is that since backup runs as a background process (as in can't tell status except by looking at logs) it made sense to still have some sanity check restriction in there. If the new 1.5G log -> 15G stop levels become an issue they could be changed/removed or even have a setting available for the user to set but they seemed reasonable to be & certainly better than 1.5G which could be quite common. Really anyone having over a couple G of emails should be looking for better backup options than the built-in in either case. ;)
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

prisma
Senior user
Senior user
Posts: 310
Joined: 2010-07-09 13:16

Re: 5.4 Backup feature

Post by prisma » 2012-02-21 12:41

Understand. I'm not asking because I want to use backup for a regular backup. I'm asking, because the build-in backup is the easiest way to migrate from MSSQL CE to postgres. And for this reason I need a full backup including the messages, not only the database dump. Am I right? :?

... and I really have to switch, the server becomes slower and slower and slower and sl...

westminster
Normal user
Normal user
Posts: 31
Joined: 2011-10-28 17:00

Re: 5.4 Backup feature

Post by westminster » 2012-02-21 15:32

I noticed that the current 5.3 will back up all message and folder info into the hMailServerBackup.xml file even if Messages is UNchecked. I also am trying to migrate from MSSQL CE to MySQL. The .elm files in /data are fine. I'd like to use backup/restore to export the database records from the MS SQL CE database, and then I could import the <message> records into an HMS MySQL setup.

When I import the backup, it does not import the message data that is in the .xml file in the <Message> records. Since the .eml files are all keeping the same name and are intact in the /data folder, it would be a big help if I could import the <Message> records from the .xml file.

Hope that makes sense!
Michael

prisma
Senior user
Senior user
Posts: 310
Joined: 2010-07-09 13:16

Re: 5.4 Backup feature

Post by prisma » 2012-02-21 15:50

This absolutely makes sense! This would be the solution I need also.
The question is: storing the paths in the XML file anyway, is it a bug or a feature?

If it's a feature, it would make sense to have the option to import them.

So, Bill, what do you think about us ? ;)

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: 5.4 Backup feature

Post by Bill48105 » 2012-02-21 16:58

The built-in backup consists of 2 parts: Database (settings & message info) + Message files. To migrate you need both. One thing you need to be careful of is if you do a backup w/o the messages then restore the entire Data tree is WIPED so you lose them all. So currently the only real way to migrate with existing messages is by including the messages. It has been discussed a few times there should be a way to not include the FILES but includes everything else in the backup to allow migrations/restores with the files in-place (meaning you handle the Data tree manually) but it hasn't seemed like it'd be useful enough to people to bother because most people who need to do it just do it manually (like sql dump/import). Obviously that becomes more of an issue for migration due to compatibility & tools available.

Btw one option to migrate is to backup & restore w/o the messages then restore Data files then run the data folder sync tool BUT you lose all IMAP folder info & messages end up in INBOX for each user. Far from ideal unless you are a POP-only shop.

Anyway if I get time I'll look at the source code & see how hard it'd be to work in a way to backup & restore with in-place messages. Could be as simple as skipping the Data folder kill before restore.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: 5.4 Backup feature

Post by Bill48105 » 2012-02-21 20:30

FYI: Testing a build here with option to skip backup/restore of message FILES but still backup/restore the messages in DATABASE. So far seems to be working OK except during restore it wipes the Data folder so I rename Data, create new empty Data, restore, delete the temp empty Data & rename the original data back & voila. Not ideal but hey if it works it is a simple solution to fulfill a need for now & can pretty it up later if needed. ;)
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

westminster
Normal user
Normal user
Posts: 31
Joined: 2011-10-28 17:00

Re: 5.4 Backup feature

Post by westminster » 2012-02-21 20:34

Bill, that would be a BIG help. Thanks for checking into it!

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: 5.4 Backup feature

Post by Bill48105 » 2012-02-21 21:01

Who wants to volunteer to test it. :D
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

westminster
Normal user
Normal user
Posts: 31
Joined: 2011-10-28 17:00

Re: 5.4 Backup feature

Post by westminster » 2012-02-21 21:37

I can test it!

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: 5.4 Backup feature

Post by Bill48105 » 2012-02-21 22:29

westminster wrote:I can test it!
Ok cool! I'm wrapping up & getting a new archive ready to upload so will let you know soon.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: 5.4 Backup feature

Post by Bill48105 » 2012-02-21 23:24

Okie dokie here it is:
http://www.hmailserver.com/forum/viewto ... 10&t=21420

Info on it is posted showing the INI value that needs to be set, install directions, warnings, etc.

As noted during restore domains that are restored have their folder tree deleted from Data folder. Not ideal but in reality no one should be restoring without backing up 1st right. ;) The trick I found was to backup (then backup again with a different method lol) then rename the Data folder then restore then rename the restored data folder then rename the temp Data folder back to Data. (Sounds confusing but it's not really.)

Anyway don't think of this as an in-place restore but rather a way to restore your settings & message info THEN you restore your Data folder tree as you see fit.

I'd HIGHLY recommend you make sure no connections can come into hmail during the restore process especially while restoring data files! But that was true before. You can either change the ports hmail listens on temporarily or delete port forwards in router etc. But you should be super careful to not have any mail flow until ready.

I will note that I would try this on TEST MACHINES & not on a live production server. At least not until we know for sure if it is working right. :D
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

westminster
Normal user
Normal user
Posts: 31
Joined: 2011-10-28 17:00

Re: 5.4 Backup feature

Post by westminster » 2012-02-22 16:34

I can do the test, but unfortunately I need a solution that will work for 5.3 (unless 5.4 can be deemed stable and advisable for production environments). Any chance of folding this in to a 5.3 variant?

I'll do the 5.4 test and let you know how it goes!

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: 5.4 Backup feature

Post by Bill48105 » 2012-02-22 17:12

westminster wrote:I can do the test, but unfortunately I need a solution that will work for 5.3 (unless 5.4 can be deemed stable and advisable for production environments). Any chance of folding this in to a 5.3 variant?

I'll do the 5.4 test and let you know how it goes!
5.3 is closed source plus should be considered dead really. (martin is barely around for anything let alone working on 5.3) I HIGHLY doubt any more 5.3 releases will come.. So with that said 5.4 is looking quite good eh lol

We can't make promises for 5.4 but check out the 5.4 poll & drop into irc & see how it's going for all of us. I've had ZERO problems since going live back in Nov 2010 with ~300 users. And that was on PRE-ALPHA! lol

If you wanted to use to help migrate 5.3 you could with enough work using test/temp machine but honestly I wouldn't be using 5.3.x myself due to all known issues, closed source, etc.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

prisma
Senior user
Senior user
Posts: 310
Joined: 2010-07-09 13:16

Re: 5.4 Backup feature

Post by prisma » 2012-02-22 19:16

Hey Bill,

first of all, thanks for your work. The solution you offered is a possible way. But wasn't it perfect if you skip over the deletion/creation part also if the INI value is set? Would it be a big thing to change this behaviour too? I think it would be the safer way...

Cheers

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: 5.4 Backup feature

Post by Bill48105 » 2012-02-22 20:09

prisma wrote:Hey Bill,

first of all, thanks for your work. The solution you offered is a possible way. But wasn't it perfect if you skip over the deletion/creation part also if the INI value is set? Would it be a big thing to change this behaviour too? I think it would be the safer way...

Cheers
prisma: Have at it. ;)

But you're welcome. Hope it will be useful to you & others :)

My original plan was to disable deletion completely but the 1/2 hr I allocated on it turned into 2+ hrs chasing crap in the code. I did disable the deleteall that wipes the Data folder but during the XML import there is also code that wipes domains as they are created and I already spent more time on this than I really wanted. It was supposed to be a quick useful hack & turned into wasting a good part of my day on something not even sure many people will use so I said 'meh good enough for now'. I might re-visit it later but choosing your battles when you have 100 things on your plate at once is essential to get anything done.

Anyway, you guys wanted a way & I gave a way. It is isn't ideal but it works based on my testing here. Even with the restore 'issue' I don't find it a big issue as the GUI warns about your data being deleted 1st plus anyone with any sense should backup before restoring so.. :D Besides the BACKUP portion is what I'll actually use myself in conjunction with sql dumps. Now I'll be able to schedule 'full' backups including message info without the actual messages since they are backed up using robocopy & archiving.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

westminster
Normal user
Normal user
Posts: 31
Joined: 2011-10-28 17:00

Re: 5.4 Backup feature

Post by westminster » 2012-02-22 21:15

OK, I just tested it on a test server using a database with about 25000 messages (3GB of data). It seems to work fine! Thank you, Bill, for your work on this!! I think it will be a big help to others who want to convert an existing MS SQL CE installation to MySQL.

So if I understand you correctly, there are versions of 5.4 that seem safe to run in a production environment. Is that correct? Is there a particular version of 5.4 that is considered "production safe"?

Thanks! Michael

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: 5.4 Backup feature

Post by Bill48105 » 2012-02-22 21:34

westminster wrote:OK, I just tested it on a test server using a database with about 25000 messages (3GB of data). It seems to work fine! Thank you, Bill, for your work on this!! I think it will be a big help to others who want to convert an existing MS SQL CE installation to MySQL.

So if I understand you correctly, there are versions of 5.4 that seem safe to run in a production environment. Is that correct? Is there a particular version of 5.4 that is considered "production safe"?

Thanks! Michael
Ok cool! Yeah my testing here showed it was working. (Heck I spend more time on testing than the code changes usually. lol) You should be able to tell if it worked since the 7z will be JUST the XML file & the file will be FARRRR from 3GB. If you extract the XML & view it you should see all the message info in there. (Normally when you backup w/o messages they aren't there, just settings are so the XML is MUCH smaller.) Ideally you'd try to restore on a spare test box & load up admin. Even w/o the data folder populated/restored it should show the users have the messages (since it just goes by the database) but once you try to download a message you could the dreaded 'missing message file' replacement email. At least you'd know it restore should be good though.

I keep my sig updated with the version I have LIVE. It doesn't mean others are not safe but unless I am having issues I tend to leave my server using whatever is working at that time. Heck my pre-alpha build had been running for almost an entire year before upgrading. In reality ANY of the 5.4's AFTER 1ST ALPHA are pretty much OK besides issues as noted in the change logs which only hit certain people under certain conditions. I believe any of the last few builds should be OK but if you are unsure the 5.4-B2011120101 build I've been using since December has been fine for me. Of course we always recommend you TEST it on another computer 1st (backup & restore same version you have then upgrade that test server) and never hurts to make sure we are available in IRC 1st in case you run into issue doesn't hurt either. ;)

Btw in terms of 'safe' no one can guarantee that with ANY software. All we can say is check out the voting poll, look at the forum for 5.4 history & hang out in irc & go with what you are comfortable with. Personally I have a hard time recommending to anyone to NOT be on 5.4 because of the known issues with previous versions & 5.4 being open source vs closed to me it is a no brainer but I understand the hesitation many have.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

prisma
Senior user
Senior user
Posts: 310
Joined: 2010-07-09 13:16

Re: 5.4 Backup feature

Post by prisma » 2012-03-05 12:52

Bill48105 wrote:My original plan was to disable deletion completely but the 1/2 hr I allocated on it turned into 2+ hrs chasing crap in the code. I did disable the deleteall that wipes the Data folder but during the XML import there is also code that wipes domains as they are created and I already spent more time on this than I really wanted.
I've noticed Martin is putting the pieces together for a (I hope stable?) release. Possibly it's much more easier for him to implement the skip of the deletion than for you. I really really think in conjunction with a checkbox in the GUI a type independent database dump and restore would be a huge benefit for a little bit more users than for us two lonely souls :)

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: 5.4 Backup feature

Post by Bill48105 » 2012-03-05 16:35

prisma: Yes we're working on getting 5.4 released finally but it was decided nothing new will be added, just bug fixes so it'll likely have to wait.
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

prisma
Senior user
Senior user
Posts: 310
Joined: 2010-07-09 13:16

Re: 5.4 Backup feature

Post by prisma » 2012-03-05 16:47

Where do I have to submit the feature request? Or is it already done?

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: 5.4 Backup feature

Post by Bill48105 » 2012-03-05 17:08

The Feature Requests forum. ;)
http://www.hmailserver.com/forum/viewforum.php?f=2
Not sure if has been submitted or not, you'd need to search.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

prisma
Senior user
Senior user
Posts: 310
Joined: 2010-07-09 13:16

Re: 5.4 Backup feature

Post by prisma » 2012-03-06 10:13

Hey Bill,

thank you for deleting the duplicated feature request (I could imagine you did this). But accidentally you deleted the wrong post :( The post you left is the post I tried to delete myself and it has no content. Would you please restore the "good" one?

... sometimes this forum appears to be a little bit buggy. deleting a post deletes only the content and lefts the headline. While creating an answer I'm often getting an error "service unavailable". after that the post is duplicated... horrible... possibly it's because I'm behind a dual wan router with a load balancer. But other forums don't show this behaviour...

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: 5.4 Backup feature

Post by Bill48105 » 2012-03-06 17:02

prisma, No sorry I did not delete your post. Maybe a mod did. Hate to say I don't think there is a way to restore deleted posts so you'd need to do it again I believe, sorry.

Yes the forum has had issues since moving to new servers. martin said he is working on it but been unable to figure out what the issue is I guess. Just know even if you get the service unavailable error your post most likely did get saved. Always check before posting again!
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: 5.4 Backup feature

Post by ^DooM^ » 2012-03-06 17:17

What was your feature request and when did you post it?

Also hopefully I have fixed the service unavailable errors, so far so good.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

prisma
Senior user
Senior user
Posts: 310
Joined: 2010-07-09 13:16

Re: 5.4 Backup feature

Post by prisma » 2012-03-06 17:23

The request directly BEFORE this one:
http://www.hmailserver.com/forum/viewto ... =2&t=22358

It has nearly the same title, similar to the empty request above, but without the mistyping.
It's named "Backup only database structure without messagesfolder"

Thanx doom

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: 5.4 Backup feature

Post by ^DooM^ » 2012-03-06 17:35

Cant see it mate and nothing in admin about it being deleted. Forums been playing up though. I suggest you start a new thread.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

prisma
Senior user
Senior user
Posts: 310
Joined: 2010-07-09 13:16

Re: 5.4 Backup feature

Post by prisma » 2012-03-06 18:07

Done:
http://www.hmailserver.com/forum/viewto ... =2&t=22365

Please Mr. Admin, delete the right item this time.

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: 5.4 Backup feature

Post by ^DooM^ » 2012-03-06 18:32

The old one wasn't deleted by a mod, if it was deleted it would have been by the OP i.e. you. Moderator deletes are logged. There is no log for your message.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

westminster
Normal user
Normal user
Posts: 31
Joined: 2011-10-28 17:00

Re: 5.4 Backup feature

Post by westminster » 2013-04-08 21:50

Bill, I know this is a crazy question, but back when you posted this helpful link for hmailbin-5.4-B2011120101.7z, was it supposed to be applied to an installation of hMailServer-5.3.3-B1879.exe or hmailserver-5.4-b1931.exe? I've just rebuilt our server and couldn't remember which installation was the base for the hmailbin-5.4-B2011120101.7z. It appears that I have had it running of over a year on 5.3.3, but I wanted to be sure.

Thanks!
Michael

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: 5.4 Backup feature

Post by Bill48105 » 2013-04-08 22:03

westminster wrote:Bill, I know this is a crazy question, but back when you posted this helpful link for hmailbin-5.4-B2011120101.7z, was it supposed to be applied to an installation of hMailServer-5.3.3-B1879.exe or hmailserver-5.4-b1931.exe? I've just rebuilt our server and couldn't remember which installation was the base for the hmailbin-5.4-B2011120101.7z. It appears that I have had it running of over a year on 5.3.3, but I wanted to be sure.

Thanks!
Michael
My experimentals should ONLY be run over existing working 5.4 and preferably using the official build closest to that experimental due to what it's based on. (I've been trying to note the official build each are based on or you can usually tell by the dates) In other words if you are on 5.3.3 now you should install an official 5.4 to do proper upgrade then if you need a feature from one of my experimentals follow the install steps shown on experimental thread. B2011120101 is quite old now so not sure I'd be using it but I can't think of any known issues with it off hand to sat NOT to though. What feature was it you needed, maybe it's in official 5.4 betas now as I try to get my changes into official source once they are tested unless there is an objection from martin. If you're looking for the db only backup feature that is in the official code but there has not been an update since so yes you'd need one of my experimentals until then. Since there is a known bug in B1944 & B1946 with forwarding automated emails such as google calendar events odds are your best bet is hmailbin-5.4-B2012052801 I should have the bug fixed but have yet had time to test it or post an updated B1946-based experimental with the changes yet. So you'd install B1942 official beta then drop in 5.4-B2012052801.7z per the instructions on the experimental thread http://www.hmailserver.com/forum/viewto ... 10&t=21420
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

westminster
Normal user
Normal user
Posts: 31
Joined: 2011-10-28 17:00

Re: 5.4 Backup feature

Post by westminster » 2013-04-08 22:13

Thanks, Bill. So, back when I installed the 20111201 experimental in Feb. 2012 I should have installed the experimental files over hMailServer-5.4-B1931.exe, not over hMailServer-5.3.3-B1879.exe. Is that correct?

At the time (Feb. 2012) I had a 3GB database running in 5.3.3, and needed to convert it from MSSQL CE to MySQL.

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: 5.4 Backup feature

Post by Bill48105 » 2013-04-08 23:20

westminster wrote:Thanks, Bill. So, back when I installed the 20111201 experimental in Feb. 2012 I should have installed the experimental files over hMailServer-5.4-B1931.exe, not over hMailServer-5.3.3-B1879.exe. Is that correct?

At the time (Feb. 2012) I had a 3GB database running in 5.3.3, and needed to convert it from MSSQL CE to MySQL.
Yes definitely over 5.4 not 5.3.3. I'd be surprised it worked over 5.3.x because of changes in the database. Granted there were not many changes but the file paths was a big one & you should have got nagged about full paths in database.

WOW 3GB database? :o How is that even possible? You sure you didn't mean 3GB of EMAIL FILES?
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

westminster
Normal user
Normal user
Posts: 31
Joined: 2011-10-28 17:00

Re: 5.4 Backup feature

Post by westminster » 2013-04-08 23:30

Bill48105 wrote: WOW 3GB database? :o How is that even possible? You sure you didn't mean 3GB of EMAIL FILES?
Bill
Whups! Yes, it was 3GB of mail files!!

When I set up the new server this weekend, I copied over the old server's hmailserver folder, installed 5.3.3. and immediately copied the 2011120101 experimental files into place, so I guess the main differences would be some scripts and php in PHPWebAdmin/ and DBScripts/. Perhaps that's why nothing burped?

Thanks for the help.

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: 5.4 Backup feature

Post by Bill48105 » 2013-04-09 01:41

westminster wrote:
Bill48105 wrote: WOW 3GB database? :o How is that even possible? You sure you didn't mean 3GB of EMAIL FILES?
Bill
Whups! Yes, it was 3GB of mail files!!

When I set up the new server this weekend, I copied over the old server's hmailserver folder, installed 5.3.3. and immediately copied the 2011120101 experimental files into place, so I guess the main differences would be some scripts and php in PHPWebAdmin/ and DBScripts/. Perhaps that's why nothing burped?

Thanks for the help.
Surprised it works unless you upgrade to 5.4 using official installer because during upgrade some database conversions take place.
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

westminster
Normal user
Normal user
Posts: 31
Joined: 2011-10-28 17:00

Re: 5.4 Backup feature

Post by westminster » 2013-04-09 02:29

I think it worked because I exported the MySQL db from the old server and imported it into MySQL on the new server . The hms folders were just copied over from the old server. When the hms install was originally created on the old server I ran the first 5.4 installer--that's when the 5.3.3 to 5.4 conversion took place.

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: 5.4 Backup feature

Post by Bill48105 » 2013-04-09 02:59

westminster wrote:I think it worked because I exported the MySQL db from the old server and imported it into MySQL on the new server . The hms folders were just copied over from the old server. When the hms install was originally created on the old server I ran the first 5.4 installer--that's when the 5.3.3 to 5.4 conversion took place.
Not sure i follow.. The hm_messages table would need to be there before installer ran for the installer to do the conversion.. That's why the migrate & upgrade process is to upgrade 1st then migrate. On top of paths being truncated in hm_messages during upgrade an upgrade script is run depending on db type. For mysql it'd be Upgrade5320to5400MySQL.sql for 5.3.2+ (5.3.3 & 5.3.4 included) which if you read it shows these changes:

Code: Select all

insert into hm_settings (settingname, settingstring, settinginteger) values ('EnableWhitelisting', '', 1);
update hm_whitelist set whiteemailaddress = replace(whiteemailaddress, '/', '//');
update hm_whitelist set whiteemailaddress = replace(whiteemailaddress, '%', '/%');
update hm_whitelist set whiteemailaddress = replace(whiteemailaddress, '_', '/_');
update hm_whitelist set whiteemailaddress = replace(whiteemailaddress, '?', '_');
update hm_whitelist set whiteemailaddress = replace(whiteemailaddress, '*', '%');
update hm_greylisting_whiteaddresses set whiteipaddress = replace(whiteipaddress, '/', '//');
update hm_greylisting_whiteaddresses set whiteipaddress = replace(whiteipaddress, '%', '/%');
update hm_greylisting_whiteaddresses set whiteipaddress = replace(whiteipaddress, '_', '/_');
update hm_greylisting_whiteaddresses set whiteipaddress = replace(whiteipaddress, '?', '_');
update hm_greylisting_whiteaddresses set whiteipaddress = replace(whiteipaddress, '*', '%');
delete from hm_settings where settingname = 'tarpitdelay';
delete from hm_settings where settingname = 'tarpitcount';
insert into hm_settings (settingname, settingstring, settinginteger) values ('MaxNumberOfMXHosts', '', 15);
insert into hm_settings (settingname, settingstring, settinginteger) values ('ClamAVEnabled', '', 0);
insert into hm_settings (settingname, settingstring, settinginteger) values ('ClamAVHost', 'localhost', 0);
insert into hm_settings (settingname, settingstring, settinginteger) values ('ClamAVPort', '', 3310);
update hm_dbversion set value = 5400;
If you look closely that does a few things including fixing whitelist & greylist syntax of existing settings, removes tarpit settings, adds ClamAV (clamd client ) settings and updates the db version.

I imagine you can see why I'd be surprised 5.4 dropping into 5.3.x would work.. I'm quite certain hmailserver.exe won't even start 1st complaining about incorrect dbversion upon startup. Now if you did those changes manually or if you dropped in the exe's then ran the 5.4 installer to upgrade I'd imagine it'd work but at that point I'd figure the experimental bins would be replaced by the ones in the installer.

Anyway, as long as it's working I guess that's all that matters. Btw I posted a new experimental today that should fix the forwarding bug that is in B1944-B1946 & my experimentals based on those.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

westminster
Normal user
Normal user
Posts: 31
Joined: 2011-10-28 17:00

Re: 5.4 Backup feature

Post by westminster » 2013-04-09 15:02

I'm afraid I'm making a hash of the description. Let me describe it chronologically:

Feb-2012
--------------
I have a HMS installation running 5.3.3. I need to convert it from MSSQL CE to MySQL.

Bill helpfully offers a 2011120101 experimental build which will support full export and import with a new message option (enabling me to export from my MSSQL CE DB, re-install HMS, and import the DB into MySQL).

So, I install 5.4-B1931 over 5.3.3, which upgrades my 5.3.3 installation. Then I copy over Bill's 2011120101 experimental binaries. With these, I successfully convert from MSSQL CE to MySQL.

This installation runs fine for a year.

April-2013
--------------
I build a new server and need to move the HMS to the new server.

I copy over the HMS folder from the old server to the new server. I export the HMS MySQL db from the old server and import it into MySQL on the new server.

I install 5.3.3 on the new server, and copy over the 2011120101 experimental build binaries. This is not a correct setup, but it works. After doing a diff, it appears that the differences are some php scripts for the web interface and the DB upgrade scripts. With the 2011120101 binaries in place, however, everything else is basically 5.4 so it works.

A day later, I realize the mistake and install 5.4-B1931. Over that installation, I copy the experimental 2011120101 binaries. All now correctly duplicates the old server's setup.

I hope that makes sense.

Now my question is this: Should I leave things as is, or should I apply a newer 5.4 build? I assume it will upgrade the database format (I think the B1931 was still using the 5320 database format ).

Michael

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: 5.4 Backup feature

Post by Bill48105 » 2013-04-09 16:11

AHHH OK that makes more sense. Yeah your mistake was when you moved you should have installed 5.4 but sure you realize that now.

As far as which build you should use, that's up to you. All I can say is look at the change logs to see if you can benefit from the changes and to see where the potential for problems might be. For the most part new releases have been stable but there were changes between B1942 & B1944 that no one realized until recently. The B1946 build posted still has the issue but martin is posting a new build shortly (hopefully today still or this weekend). In the meantime I posted a new experimental yesterday that is based on B1946 but has a fix for the forwarding bug. 2011120101 is quite old now but at same time if it works & you don't want/need any of the changes since then probably guess no reason to upgrade really.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

Post Reply