Internal Backup/Restore - Reasonable Restrictions?

Use this forum if you want to discuss a problem or ask a question related to a hMailServer beta release.
Post Reply
User avatar
jimimaseye
Moderator
Moderator
Posts: 7520
Joined: 2011-09-08 17:48

Internal Backup/Restore - Reasonable Restrictions?

Post by jimimaseye » 2015-03-25 16:41

FAO Martin

As has previously been eluded to in another thread (viewtopic.php?f=10&t=27950) we are hitting problems with the built-in restrictions on the internal backup function. As I understand it, it is now built-in that it WILL NOT start if the data directory exceeds a hard-coded limit of 16GB.

This causes people with larger more successful installations problems and an example (and by no means the only one) of this is here: viewtopic.php?p=173918#p173918. Due to the size of the data directory the internal backup procedure cannot be used.

Now, it is my belief that the procedure (when including messages) makes a copy of the messages and then compresses them with 7Zip. Now, as far as I know there is no builtin filesize limit in 7Zip (well, technically it is "Supporting files with sizes up to 16000000000 GB", a billion times more than the program limit: http://www.7-zip.org/7z.html), and that the only filesize restriction would be at OS/filesystem level or physical hardware disk availability, and in the case of MOST systems, (ie, NOT using FAT/FAT32), 16GB is not a limit. And consequently there would be nothing stopping a user from MANUALLY copying their data directory to a temporary location, and then MANUALLY 7Zipping it to a compressed archive which is, as far as I understand, exactly what HMS does within its built in function. I also understand HMS writes an XML file with each of the messages' details but again this output file should only be restricted by the filesystem so data CONTENT shouldnt be a problem due to size.

I would like to ask if you could explain why this restriction of 16gb datasize allowance is in place and if there is possibility of removing such restriction (which in turn helps administration of larger installations).
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
mattg
Moderator
Moderator
Posts: 19101
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Internal Backup/Restore - Reasonable Restrictions?

Post by mattg » 2015-03-26 01:17

here is some other discussion

viewtopic.php?f=10&t=25571
viewtopic.php?f=10&t=22192
viewtopic.php?f=10&t=3074

viewtopic.php?f=2&t=22365


Also, there has been some discussion over the years about restoring single domains, accounts or even messages.
Perhaps a domain by domain backup process too...
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

prisma
Senior user
Senior user
Posts: 308
Joined: 2010-07-09 13:16

Re: Internal Backup/Restore - Reasonable Restrictions?

Post by prisma » 2015-03-26 18:35

Martin had created a poll for future improvements:
viewtopic.php?t=27191
I already wrote there the backup system was a little bit a step child of hmailserver. Unfortunately it not a own item to choose. I think Martin sees no reason for improvements regarding backup.

User avatar
martin
Developer
Developer
Posts: 6777
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: Internal Backup/Restore - Reasonable Restrictions?

Post by martin » 2015-03-29 20:39

I have already removed that code for version 5.7 (but 5.7 is currently still alpha-at-best).

I agree that it does not really make any sense. I suspect Bill added it to prevent things from going amok, but 7zip is good at handling large files.

Here's the reference to the original request:
https://github.com/hmailserver/hmailserver/issues/69
Martin Knafve
martin@hmailserver.com
https://twitter.com/knafve

User avatar
mattg
Moderator
Moderator
Posts: 19101
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Internal Backup/Restore - Reasonable Restrictions?

Post by mattg » 2015-03-30 00:11

Martin, removing the warning is only a small part of the issue.

I think that the question about the backup NOT working is a much bigger issue. (particularly the uncompressed backup not including messages)
Also, the limit of 15 Gb isn't very large in today's world.

Perhaps the time taken to compress backups could also be logged.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
SorenR
Senior user
Senior user
Posts: 2624
Joined: 2006-08-21 15:38
Location: Denmark

Re: Internal Backup/Restore - Reasonable Restrictions?

Post by SorenR » 2015-03-30 00:39

mattg wrote:Martin, removing the warning is only a small part of the issue.

I think that the question about the backup NOT working is a much bigger issue. (particularly the uncompressed backup not including messages)
Also, the limit of 15 Gb isn't very large in today's world.

Perhaps the time taken to compress backups could also be logged.

Code: Select all

2015-03-29 05:00:02.581	Backing up data directory...
2015-03-29 05:12:24.057	Backing up settings...
2015-03-29 05:12:24.276	Writing XML file...
2015-03-29 05:12:28.354	Compressing message files...
2015-03-29 05:25:36.362	Backup completed successfully
05:25:36.362 - 05:12:28.354 = 13 minutes 8 seconds..
SørenR.

The quantum rule of insecurity which states that the act of observing how vulnerable a host or service is changes the insecurity level of the service.

User avatar
mattg
Moderator
Moderator
Posts: 19101
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Internal Backup/Restore - Reasonable Restrictions?

Post by mattg » 2015-03-30 02:06

:oops:
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jimimaseye
Moderator
Moderator
Posts: 7520
Joined: 2011-09-08 17:48

Re: Internal Backup/Restore - Reasonable Restrictions?

Post by jimimaseye » 2015-03-30 11:01

SorenR wrote:05:25:36.362 - 05:12:28.354 = 13 minutes 8 seconds..
That's a long time for 4 emails! :lol:

Seriously, though, I just did a test on mine: 7.5GB data took 41 minutes 26 secs to complete (resultant zip was 4.9GB). And thats on Server 2008 R2.

Code: Select all

2015-03-30 09:17:30.836	Backup started
2015-03-30 09:17:30.851	Loading backup settings....
2015-03-30 09:17:30.914	The size of the data directory exceeds the maximum RECOMMENDED size for the built in backup (1.5GB) so LOGGING. Please consult the backup documentation
2015-03-30 09:17:30.929	Backing up domains...
2015-03-30 09:17:32.723	Backing up data directory...
2015-03-30 09:30:24.379	Backing up settings...
2015-03-30 09:30:24.379	Writing XML file...
2015-03-30 09:30:28.450	Compressing message files...
2015-03-30 09:58:56.188	Backup completed successfully
Out of interest how many GB was your data Soren? (Because the 'backing up data' time was comparable to mine, and yet in total my zipping was twice as long)
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
SorenR
Senior user
Senior user
Posts: 2624
Joined: 2006-08-21 15:38
Location: Denmark

Re: Internal Backup/Restore - Reasonable Restrictions?

Post by SorenR » 2015-03-30 12:56

jimimaseye wrote:
SorenR wrote:05:25:36.362 - 05:12:28.354 = 13 minutes 8 seconds..
That's a long time for 4 emails! :lol:

Seriously, though, I just did a test on mine: 7.5GB data took 41 minutes 26 secs to complete (resultant zip was 4.9GB). And thats on Server 2008 R2.

Code: Select all

2015-03-30 09:17:30.836	Backup started
2015-03-30 09:17:30.851	Loading backup settings....
2015-03-30 09:17:30.914	The size of the data directory exceeds the maximum RECOMMENDED size for the built in backup (1.5GB) so LOGGING. Please consult the backup documentation
2015-03-30 09:17:30.929	Backing up domains...
2015-03-30 09:17:32.723	Backing up data directory...
2015-03-30 09:30:24.379	Backing up settings...
2015-03-30 09:30:24.379	Writing XML file...
2015-03-30 09:30:28.450	Compressing message files...
2015-03-30 09:58:56.188	Backup completed successfully
Out of interest how many GB was your data Soren? (Because the 'backing up data' time was comparable to mine, and yet in total my zipping was twice as long)
Image
SørenR.

The quantum rule of insecurity which states that the act of observing how vulnerable a host or service is changes the insecurity level of the service.

User avatar
jimimaseye
Moderator
Moderator
Posts: 7520
Joined: 2011-09-08 17:48

Re: Internal Backup/Restore - Reasonable Restrictions?

Post by jimimaseye » 2015-03-30 14:10

Ah so it is comparable by ratio. I have 60000 files opposed to your 17000. And 7.5GB opposed to your 1.8GB. So it sort of makes sense.

(BTW, you know I was joking about the "4 emails", right?) :oops:
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
jimimaseye
Moderator
Moderator
Posts: 7520
Joined: 2011-09-08 17:48

Re: Internal Backup/Restore - Reasonable Restrictions?

Post by jimimaseye » 2015-03-30 14:13

martin wrote:I have already removed that code for version 5.7 (but 5.7 is currently still alpha-at-best).

I agree that it does not really make any sense. I suspect Bill added it to prevent things from going amok, but 7zip is good at handling large files.

Here's the reference to the original request:
https://github.com/hmailserver/hmailserver/issues/69
This is good to hear.

As matt is suggesting, I hope its more than the 'warning' that is removed, and that also the limit is removed too (as well as other complications currently existing).

Maybe Bill introduced the 16GB limit after reading the 7zip page and thinking the upper file limit meant 16GB and not understanding it actually reads 16,000,000,000 GB.
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
jimimaseye
Moderator
Moderator
Posts: 7520
Joined: 2011-09-08 17:48

Re: Internal Backup/Restore - Reasonable Restrictions?

Post by jimimaseye » 2017-01-14 14:19

Ok, so Ive tested myself for clarity. In version 5.7 the 16GB limit has been completely removed (and not just the erroneous warning).

The test:

1, Enter hm_MESSAGES and set the 'messagesize' of one message to 140 GB (yes, one hundred and 40).
2, Try complete backup (all ticked + compress)

Results:
under 5.6.6 - receive " Data Directory exceeds 15GB limit" message and the backup aborts (140 is far bigger than 15 and 15 is the limit)
under 5.7.0 B2373 - backup runs and completes as desired - no limit or restriction encountered.

Conclusions:

a, YEY!!!!!! AT LAST!!!! Finally this nonsensical restriction is no longer there.
b, (oh, and the calculation for determining the data directory size was clearly done by the sum of the hm_messages 'messagesize' field and not a physical detection of the files).
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

Post Reply