hMailCleaner (Compare hMaildatabase with files on disk)

This section contains scripts that hMailServer has contributed with. hMailServer 4 is needed to use these.
Post Reply

Was this script usefull to you?

Yes
13
87%
No
1
7%
Files on harddrive and hmaildatabase match
1
7%
 
Total votes: 15

large
New user
New user
Posts: 27
Joined: 2004-12-31 12:47
Contact:

hMailCleaner (Compare hMaildatabase with files on disk)

Post by large » 2007-08-02 09:03

I have been using hMailserver from the early betas and love the job Martin does! Thanks; keep up the good work. This script is how I can give something back to Martin & the community :)

I recently changed the server hardware and copied the %programfiles%\hMailServer\Data\ to the new disk. After checking the filecount against the SQL-database the actual filecount was 20906 against 4655 in the database. Meaning that I had 16251 files taking up space without a reference. That was a waste of space (around 1,5GB).

I dunno how this happened, but I had a faulty RAID controller that made random files writeprotected.

Anyway, here is a Description of the script:
It makes a recursive list of a given directory (usally where you store the *.eml files :)) and then it compares to the filenames that are stored in the hmailserver database. If there is a mismatch the file will be renamed to *.deleteme

Who need this?
If you're running a hmailserver and had a couple of lockups or simular I guess you might have some "leftovers" on the disk.

The script (hMailCleaner.vbs):
Please run the script in a command prompt using cscript.exe!

Code: Select all

' ====================================================
' hMailCleaner by Lars Werner (http://lars.werner.no/)
' ====================================================
' Usage: cscript hmailcleaner.vbs "data-path" adminpassword [/delete]
'
' How does it work?
' -----------------
' The script makes a recursive list of a given directory (the place where you store the *.eml files) and
' then it compares to the filenames that are stored in the hmailserver database.
' If there is a mismatch the file will be renamed to *.deleteme
'
'WARNING: If /delete option is given then all the mismatched files will be permanently deleted!!!
'
' Who need this?
' --------------
' If you're running a hmailserver and had a couple of lockups or simular I guess you might have some "leftovers" on the disk.
' This script marks the files that aren't in the database or even delete them directly!
' Pretty usefull if you have a crappy RAID controller that makes everything writeprotected, like I did ;)
'
'Disclaimer: I use hMailserver in a small scale! This script has only been tested with my server & setup and it might crash or harm your system. So please take backup, and don't come to me crying about it!

Option Explicit

Dim aFiles() 'Array of all the files in the given path
Dim aMailFiles() 'Array of files in the database
Dim iCount, iArrayCount, iPercentage, iLastPercentage
Dim bDelete  'Really delete?
Dim objFSO, objFolder, colFiles, objFile

'===========================
'== SUBS & FUNCTIONS
'= Store each file from a given directory into an array
Function ArrayOfFiles(Folder) 'Returns an object of objFolder (used in recursive mode)
	Dim SubFolder
	For Each Subfolder in Folder.SubFolders
		Set objFolder = objFSO.GetFolder(Subfolder.Path)
		Set colFiles = objFolder.Files
		Redim preserve aFiles(UBound(aFiles) + colFiles.Count) 'Resize the array with the new files we got
		For Each objFile in colFiles
			if StrComp(Right(objFile.Name, 9), ".deleteme") <> 0 then
				aFiles(iCount) = objFolder.Path & "\" & objFile.Name
			end if
			iCount = iCount + 1
		Next
		ArrayOfFiles Subfolder 'Recursive call
	Next
End Function

'===========================
'= Store each db "file" into an array
Sub FillMailArray(sAdminPw)
	Dim oApp
	Set oApp = CreateObject("hMailServer.Application")

	' Authenticate the client.
	Call oApp.Authenticate ("Administrator", sAdminPw)

	Dim iDomains, iAccounts, iMessages
	iArrayCount = 0
	ReDim aMailFiles(iArrayCount)

	for iDomains = 0 to oApp.Domains.Count - 1
		Dim oDomain
		Set oDomain = oApp.Domains.Item(iDomains)
		WScript.Echo "Domain: " & oDomain.Name
	
		for iAccounts = 0 to oDomain.Accounts.Count - 1
			Dim oAccount
			set oAccount = oDomain.Accounts.Item(iAccounts)
			WScript.Echo "	-> " & oAccount.Address
			WScript.Echo "		> Messages: " & oAccount.Messages.Count
			Redim preserve aMailFiles(UBound(aMailFiles) + oAccount.Messages.Count)
			for iMessages = 0 to oAccount.Messages.Count - 1
				Dim oMessage
				set oMessage = oAccount.Messages.Item(iMessages)
				aMailFiles(iArrayCount) = oMessage.Filename
				iArrayCount = iArrayCount + 1
				set oMessage = nothing
			Next 'End iMessages
			set oAccount = nothing
		Next 'End iAccounts
		set oDomain = nothing
	Next 'End iDomains

	set oApp = nothing
End Sub

'===========================
'== MAIN
bDelete = false

'Check if we're using wscript (quit) or cscript (ok)
if InStr(lcase(WScript.Fullname), "wscript") > 0 then
	WScript.Echo "Please do not run this program from windows! Use the command prompt and cscript.exe instead"
	WScript.Quit
end if

'Check how many arguments we have
if WScript.Arguments.Count >=2 then
	Set objFSO = CreateObject("Scripting.FileSystemObject")
	iCount = 0 'Overall counter to know where we are in the file-array
	Set objFolder = objFSO.GetFolder(Wscript.Arguments(0))
	Set colFiles = objFolder.Files
	Redim aFiles(colFiles.Count)
	For Each objFile in colFiles
		'Ignore already "deleted" messages
		if StrComp(Right(objFile.Name, 9), ".deleteme") <> 0 then
			aFiles(iCount) = objFolder.Path & "\" & objFile.Name
		end if
		iCount = iCount + 1
	Next

	'Just wait message for the user
	WScript.Echo "Please wait, creating filelist for: " & Wscript.Arguments(0)

	'Now get an recursive list of files
	ArrayOfFiles objFSO.GetFolder(Wscript.Arguments(0))

	'Now create an array of files from the database
	FillMailArray(Wscript.Arguments(1))

	'Print out the info before we start renaming
	WScript.Echo
	WScript.Echo " Number of physical files on disk: " & UBound(aFiles)
	WScript.Echo "Number of files in hmail database: " & UBound(aMailFiles)
	WScript.Echo

	'Check if aMailFiles is bigger than aFiles and give a warning!
	if UBound(aFiles) < UBound(aMailFiles) then
		WScript.Echo "WARNING: hMaildatabase contains more files than there actually is on the hard drive, this will hMailServer to give ""failed to open"" messages in IMAP"
		WScript.Echo
	end if

	'Check if we're gonna delete & Write warning if we're gonna delete the file
	if WScript.Arguments.Count = 3 then
		if InStr(lcase(WScript.Arguments(2)), "/delete") then
			bDelete = true
			WScript.Echo "WARNING: Mismatched files will be permanently deleted! (To abort press CTRL-C, 3 sec delay)"
			WScript.Echo
			WScript.Sleep 3000
		else
			WScript.Echo "WARNING: Unknown third argument, files will be renamed!"
			WScript.Echo
		end if
	end if

	'Now loop through the files & compare with the database files
	iLastPercentage = 0 'Stores the last procentage so it does not update so much
	WScript.Echo "Please wait while the files are compared..."
	for iCount = 0 to UBound(aFiles) - 1

		'Check if we can find the filename in the database
		Dim bFound, sComp
		bFound = false

		'We only do the check when there is an actual file!
		if Len(aFiles(iCount)) <> 0 then
			'Get the filename we are gonna to compare with (remove the directory)
			sComp = right(lcase(aFiles(iCount)), len(aFiles(iCount)) - len(Wscript.Arguments(0))) 'We only compare the info behind selected dir :)
			if Left(sComp,1) = "\" then sComp = Right(sComp, len(sComp)-1) 'Remove the \ if it exists...

			'Now check with the databaselist
			for iArrayCount = 0 to UBound(aMailFiles)
				'Check if the entry actually is an entry
				if(Len(aMailFiles(iArrayCount))) <> 0 then
					if StrComp(sComp, lcase(Right(aMailFiles(iArrayCount), Len(sComp))), 0) = 0 then
						aMailFiles(iArrayCount) = ""
						bFound = true
						exit for
					end if
				end if
			Next

			'If we haven't found the file we are going to rename/delete it
			if bFound = false then
				if bDelete = false then
					call objFSO.MoveFile(aFiles(iCount), aFiles(iCount) & ".deleteme")
					WScript.Echo "Renamed: " & aFiles(iCount)
				else
					call objFSO.DeleteFile(aFiles(iCount))
					WScript.Echo "Deleted: " & aFiles(iCount)
				end if
			end if
		end if 'Len(aFiles(iCount) <> 0

		'Here we write the progress (5% at the time)
		iPercentage = FormatNumber((iCount * 100) / UBound(aFiles), 0)
		if iPercentage - iLastPercentage = 5 then
			WScript.Echo "Progress: " & iPercentage & "%"
			iLastPercentage = iPercentage
		end if

	Next

	WScript.Echo
	WScript.Echo "Finished!"

	'Cleanup of objects
	Set colFiles = nothing
	Set objFolder = nothing
	set objFSO = nothing
else
	WScript.Echo "===================================================="
	WScript.Echo "hMailCleaner by Lars Werner (http://lars.werner.no/)"
	WScript.Echo "===================================================="
	WScript.Echo
	WScript.Echo "Usage: cscript hmailcleaner.vbs ""data-path"" adminpassword [/delete]"
	WScript.Echo
	WScript.Echo "Example: cscript hmailcleaner.vbs ""C:\Program Files\hMailServer\Data"" AdminPw"
	WScript.Echo "With logfile: cscript hmailcleaner.vbs ""C:\Program Files\hMailServer\Data"" AdminPw >log.txt"
	WScript.Echo "(The log.txt will be placed the current directory that you're running the script from, no output will be shown)"
	WScript.Echo
	WScript.Echo "How does it work?"
	WScript.Echo "-----------------"
	WScript.Echo "The script makes a recursive list of a given directory (the place where you store the *.eml files) and then it compares to the filenames that are stored in the hmailserver database."
	WScript.Echo "If there is a mismatch the file will be renamed to *.deleteme (default behaviour)"
	WScript.Echo
	WScript.Echo "WARNING: If /delete option is given then all the mismatched files will be permanently deleted!!!"
end if
Please give feedback on the poll!

Warning: I use hMailserver in a small scale! This script has only been tested with my server & setup and it might crash or harm your system. So please take backup, and don't come to me crying about it!
Last edited by large on 2007-08-02 10:51, edited 4 times in total.
Lars Werner
http://lars.werner.no
Check out my tools:
http://lars.werner.no/unpacker/ - 100% automated extraction tool
http://lars.werner.no/sizeme/ - Maximize the output on a given media (like CD/DVD ect)

User avatar
bagu
Normal user
Normal user
Posts: 212
Joined: 2005-06-17 03:08
Location: France
Contact:

Post by bagu » 2007-08-02 09:24

Usefull but popups are really anoying

large
New user
New user
Posts: 27
Joined: 2004-12-31 12:47
Contact:

Post by large » 2007-08-02 09:27

Please do not run this script directly from Windows (by doubleclicking on it).

Run it from a command prompt with cscript.exe command :)

Edit:
I've updated the script to notify the user if he doubleclicked on the hMailCleaner.vbs file
Lars Werner
http://lars.werner.no
Check out my tools:
http://lars.werner.no/unpacker/ - 100% automated extraction tool
http://lars.werner.no/sizeme/ - Maximize the output on a given media (like CD/DVD ect)

GlenC
Senior user
Senior user
Posts: 680
Joined: 2004-08-17 23:31
Location: Santiago, Chile

Post by GlenC » 2007-08-03 01:53

Hey that was really nice. I always wondered if I was in sync or not (mostly due to a lot of experimentation I've done in the past). For the record, I only had two orphans.. both 0 byte files.

Thanks large... and Martin too for the reliable mailserver. :)

cnsjfruiz
New user
New user
Posts: 1
Joined: 2007-11-15 07:25

Re: hMailCleaner (Compare hMaildatabase with files on disk)

Post by cnsjfruiz » 2007-11-16 08:41

Could you provide me assistance for using your script to remove the entire directory instead of just removing the eml files? For example if you deleted an account but for some reason the directory did not get deleted you could run this script to remove orphan directories? By the way the script is awesome!! Thank you in advance!

large
New user
New user
Posts: 27
Joined: 2004-12-31 12:47
Contact:

Post by large » 2007-11-16 09:40

cnsjfruiz:
Well if you want to delete empty directories on a subuser you should modify the code like this:

Code: Select all

         'If we haven't found the file we are going to rename/delete it
         if bFound = false then
            if bDelete = false then
               call objFSO.MoveFile(aFiles(iCount), aFiles(iCount) & ".deleteme")
               WScript.Echo "Renamed: " & aFiles(iCount)
            else
               call objFSO.DeleteFile(aFiles(iCount))
               RmDir Left(aFiles(iCount), InStrRev(aFiles(iCount), "\")-1) 'Removes the directory if it is empty.
               WScript.Echo "Deleted: " & aFiles(iCount)
            end if
         end if
      end if 'Len(aFiles(iCount) <> 0 
It has not been tested, but it should work.

If you actually delete an user then a simple deltree of that username should be the simplest solution (but thats not what this script is all about)
Lars Werner
http://lars.werner.no
Check out my tools:
http://lars.werner.no/unpacker/ - 100% automated extraction tool
http://lars.werner.no/sizeme/ - Maximize the output on a given media (like CD/DVD ect)

Kaan1983
Senior user
Senior user
Posts: 595
Joined: 2007-01-30 16:26
Location: TÜRKIYE

Post by Kaan1983 » 2007-11-16 10:43

how is it different than hMailSynchronizer?

large
New user
New user
Posts: 27
Joined: 2004-12-31 12:47
Contact:

Post by large » 2007-11-16 11:01

I guess you are referring to Data directory synchronizer (DDS): http://www.hmailserver.com/documentatio ... rguide_dds ?

This script does not import any e-mails and it does not alter your database. It just cleans up the mails on the harddrive that don't have a match in the database.

DDS is used when you have had a database error or simular and want to import mail that exists on the harddrive but not in the database.

On my server all the "leftovers" where spam (or files contained virus). I guess a file-lock or similar accrued when external programs checks a file sometimes.

Kaan1983
Senior user
Senior user
Posts: 595
Joined: 2007-01-30 16:26
Location: TÜRKIYE

Post by Kaan1983 » 2007-11-16 14:25

Well I don't have any "leftovers" yet but if I have one day how can I be sure that it's spam. Or even if it's a spam and if I get it to the database with DDS it'll be one spam for one of the users maybe. I mean what's the ratio of these leftovers to the users..

Or maybe instead of deleting em immediately you can send em again to the recipient and hope the anti-spam and virus scanner work fine this time.

Also if you find such an e-mail that none of the spam scanners recognize neither virus scanners and doesn't reach to the recipient then you can use that file for hacking :D

large
New user
New user
Posts: 27
Joined: 2004-12-31 12:47
Contact:

Post by large » 2007-11-16 14:51

Some people have leftovers for different reasons. You don't know if it's spam before you take some samples of the leftovers.

The script renames (or deletes based on your choice) the files to <oldfilename>.deleteme, so you can check the content afterwards.

Remember that the DDS import mails under a user-directory without a filter-, spam- or viruscheck. You can risk importing a bunch of unwanted mails into the users account.
If a file is present on the harddrive and does not exists on the database changes are high that the content is unwanted, if you haven't experienced any database problems.



When you look at my stats earlier; 16251 eml-files that are not listed in the database. Cause on my system where;
* Realtime virusscan locking files after clam-win check. hMailserver couldn't delete the file, but it cleaned it from the database of cause)
* Faulty raid-system that sometimes made files locked (unlocked on next reboot). hMailserver cannot delete, but cleans up in the database.

On my account alone it would give me over 2000 spam/virus mails that I actually needed to delete manually if I used DDS (because of the bypass of checks). So this script wasn't made for fun you know. It is a handy tool if you want to check if there are leftovers and you can reuse the array-procedures to actually make a tree-view for instance in an asp/php-page :)

Kaan1983
Senior user
Senior user
Posts: 595
Joined: 2007-01-30 16:26
Location: TÜRKIYE

Post by Kaan1983 » 2007-11-16 19:50

Please don't get me wrong, beyond being unexperienced I didn't even study anything related to computer sciences... I'm trying to learn.

I only have 200 eml files at the total and 0 leftovers :) But, in your case, over 2000 leftovers should be boring.

Btw, I don't wanna pass over my boudaries but the same think could've be done in a more effiecient way by querying the DB.

duke16
Normal user
Normal user
Posts: 40
Joined: 2007-03-20 12:13

Post by duke16 » 2007-12-18 16:09

Is this differerent from the addon that comes with hMailserver? DataDirectorySynchronizer.exe

rodolfor
Senior user
Senior user
Posts: 282
Joined: 2005-06-30 09:05
Location: Gubbio - Italy

Post by rodolfor » 2007-12-25 11:57

Warning:
1. the script check also files in \data and could rename/delete emails also if there are in normal transit (and not inserted in database).
2. if you have a lot of messages, it could take hours to complete, comparing files and database as they are in very different times.

To avoid this during this script hmailserver must be isolated from the users.

I suggest to check only files older more than a day.

thanks

BrandonH75
Normal user
Normal user
Posts: 41
Joined: 2007-06-18 20:46
Location: MN

Post by BrandonH75 » 2007-12-28 00:56

I've been thinking I needed something like this for a while. I'm glad I finally found it. :)

I'm having an issue with it though. First, I don't know anything about VB. When I run the script it gets up to "Please wait while the files are compared...", then I get the error...

"C:\hMailServer\Addons\Utilities\hmailcleaner.vbs(177, 16) Microsoft VBScript runtime error: File not found"

...and the script stops.

The first time I ran it, it found some leftovers in the Data directory and a few in a couple of the accounts folders before it gave that error. I deleted those files, and every run after that it doesn't find anything before it errors out.

There is about a 30k difference in the file count. :P Am I doing something wrong?

Thanks!

redrummy
Senior user
Senior user
Posts: 370
Joined: 2007-06-21 06:52
Location: Alaska

Post by redrummy » 2007-12-28 05:23

Line 177 is in the code that renames or deletes orphaned files:

Code: Select all

         'If we haven't found the file we are going to rename/delete it 
         if bFound = false then 
            if bDelete = false then 
               call objFSO.MoveFile(aFiles(iCount), aFiles(iCount) & ".deleteme") 
               WScript.Echo "Renamed: " & aFiles(iCount) 
            else 
               call objFSO.DeleteFile(aFiles(iCount)) 
               WScript.Echo "Deleted: " & aFiles(iCount) 
            end if 
         end if 
      end if 'Len(aFiles(iCount) <> 0 
I suspect the server contents change before the script completes, thus corrupting the results (see rodolfor's post before yours). This does seem a bit dangerous as hMS has to be running for the script to work. If you have physical/console access to the server maybe use a firewall to temporarily block traffic or disable the NIC while the script runs?

P.S. I don't know anything about VB either, which became painfully obvious once I started trying to write my own scripts. ;)

Greta
Senior user
Senior user
Posts: 306
Joined: 2007-01-02 13:23
Contact:

Re: hMailCleaner (Compare hMaildatabase with files on disk)

Post by Greta » 2011-04-07 10:00

This script is for version 4. But can I run it also on hmail version 5.3?

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: hMailCleaner (Compare hMaildatabase with files on disk)

Post by ^DooM^ » 2011-04-07 12:26

If at first you don't succeed, bomb disposal probably isn't for you! ヅ

Greta
Senior user
Senior user
Posts: 306
Joined: 2007-01-02 13:23
Contact:

Re: hMailCleaner (Compare hMaildatabase with files on disk)

Post by Greta » 2011-04-07 13:33

That PHP script doesn't work because php is not installed on the server.

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: hMailCleaner (Compare hMaildatabase with files on disk)

Post by ^DooM^ » 2011-04-07 20:30

Then you will just have to test the script to find out.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

Post Reply