Very old POP3 messages in message directories

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
tohare
Normal user
Normal user
Posts: 180
Joined: 2014-12-07 13:35
Location: Florida!

Very old POP3 messages in message directories

Post by tohare » 2015-09-28 10:24

I browsed the server mail directories (i.e.: E3, F4, etc.) and see messages listed that are very old (years). I can't help but believe these messages were already read but the system did not delete them after they were read by the user. Is there a way to delete messages in the directory structures that are over "n" number of years old? There are too many directories for each user to do it manually. Also, if deleted in directory will this cause a problem elsewhere (i.e.: MySQL)???
Thanks,
Thomas

Hazark
New user
New user
Posts: 26
Joined: 2014-05-08 23:17

Re: Very old POP3 messages in message directories

Post by Hazark » 2015-09-28 10:29

There is a custom script that do what you want.

viewtopic.php?p=175114#p175114
(You probably need to change MESSAGES_FOLDER to "INBOX")

But beware, many users do expect these "old" mails stays there and do not have any backup.

User avatar
jimimaseye
Moderator
Moderator
Posts: 8172
Joined: 2011-09-08 17:48

Re: Very old POP3 messages in message directories

Post by jimimaseye » 2015-09-28 10:34

DO NOT MESS with the data files at OS level! If you do you ask for trouble that will need remedial actions.

The best way is to identify what the files are. If they are Inbound messages that should have appeared at some point in the relevant account inbox at some point in the past. Take a look at them to see if they are inbound.

Then run DDS. If they are missing from the database they will then be imported (and reappear in the relevant inbox). From here you can then delete them using your client.

The above works if you are running clients by IMAP connection (recommended!). Make a connection by IMAP and do this work.

If however you are using POP then its likely that you do not have the correct 'delete after....' setting on the email client (such as "delete from server when/after....").

(IF YOU STILL CONTINUE to delete the files manually you will cause problems because the Database will still think they exist and if they are referenced then you will get a whole lot of messy 'emails missing' emails appear. You will need to clean up your database using a script.)

EDIT: Yes, you could also use my script (as Hazark has suggested) - that should work well but BE CAREFUL because its definitive and it does not care about content.
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

tohare
Normal user
Normal user
Posts: 180
Joined: 2014-12-07 13:35
Location: Florida!

Re: Very old POP3 messages in message directories

Post by tohare » 2015-09-28 10:50

Thanks for the replies guys. I just looked and the messages in question are POP3 and at least 2 years old and some are clearly marked "*SPAM*". Not sure if this is a SA blooper or what. If it is a Spam Assassin blooper then is there a way to fix it? I used DDS every few months and have run it in both directions. So for some unknown reason even after years and many DDS sessions SPAM messages still exist in the user POP3 system directories. I dare say (w/o looking) that those messages are no longer in MySQL data stores.

I would love to reduce the size of the compressed files by eliminating junk (BU is run every day). If nothing else it would reduce the time required to do the daily BU.
Thanks,
Thomas

User avatar
jimimaseye
Moderator
Moderator
Posts: 8172
Joined: 2011-09-08 17:48

Re: Very old POP3 messages in message directories

Post by jimimaseye » 2015-09-28 11:16

Well,.....
..... if the messages are MARKED as spam, then Spamassassin is doing its job (it seems). SA doesnt DELETE messages, it scores them (if the messages have been scored then SA work is done). However, HMS does the job of deleting messages 'at source' if messages exceeds a threshold. The question is: WHAT has physically marked the messages? Was it SA or was it HMS (in the AntiSpam settings - because HMS can also do scoring and marking as 'SPAM'). Review your HMS settings to see what it says (post them here).

I have a question: why are your clients connecting using POP? By its nature, it is open to mis-configuration or opportunity to not perform as you think it will (ESPECIALLY if there is more than 1 client connecting to the same account). As for these messages, could it be that some PC/client based antivirus software is deleting these messages as they download (protecting the OS) and as such these are not hitting your client and subsequently the client is not reporting back to the server that they have been deleted?

In short: USE IMAP!! :D You will not get these problems.

I think the best thing for you to do i this case is to load up an IMAP connection to your accounts and view the messages. Then delete them either manually via this client or use the script mentioned above (once your happy they all qualify). Then review your choice of using POP and/or review the configuration of the POP clients on ALL machines that connect.
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

tohare
Normal user
Normal user
Posts: 180
Joined: 2014-12-07 13:35
Location: Florida!

Re: Very old POP3 messages in message directories

Post by tohare » 2015-09-28 11:56

I Am a IMAP junkie but the users are older and set in ways. They seem to be pop3 die hard and I ain't gonna argue...

As time permits I will try to find what marks them spam. In the mean time I will ask them to manually set a "delete after...." option and hope that works. I really don't want to look at their email for privacy reasons.
Thanks,
Thomas

User avatar
jimimaseye
Moderator
Moderator
Posts: 8172
Joined: 2011-09-08 17:48

Re: Very old POP3 messages in message directories

Post by jimimaseye » 2015-09-28 12:01

If you changed them from POP to IMAP, how would it affect them/how would they know? (Im curious)
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

tohare
Normal user
Normal user
Posts: 180
Joined: 2014-12-07 13:35
Location: Florida!

Re: Very old POP3 messages in message directories

Post by tohare » 2015-09-29 09:12

Jimmy

They need to define whether POP or IMAP in their client setup. Then they get a list of default folders for the IMAP. I'd rather not deal with users even though I have no doubt it would be better to move them to IMAP. ;-)
Thanks,
Thomas

tohare
Normal user
Normal user
Posts: 180
Joined: 2014-12-07 13:35
Location: Florida!

Re: Very old POP3 messages in message directories

Post by tohare » 2015-09-29 09:25

FWIW...

It appears all the emails are only in the POP3 folders and mainly from early 2015 and before. All are SPAM. Possibly mainly before the upgrade to SA 3.4 (was 3.3 in some headers).

IMAP folders appear clean. I noticed most appeared to be SA inbound. Some were from the relay service and many were the local SA doing the flagging.

To just throw it out there, if anyone still has POP account on HMS you may want to periodically poke around for old files. Some seem as if were from when we first transitioned from another email server to HMS. Bottom line is you can get a "lot" of "flotsom and jetsom" in the mail folders if you do not look once in a while.

I hate to admit it but I manually went through one user after I developed a pattern and deleted all the suspect emails (all of them). The date stamps were key and any recent mails I inspected and the "all" were flagged as Spam. I ran the BU again and ZIP file is ~12 megs smaller now.. Ouch...

I will deal with it in a way that gets me the least amount of user blow back.

Just for grins I set the SA service permissions (SPAMD) to admin. That will hopefully stop any possible future problems. But I really suspect it was the upgrade from 3.3 to 3.4 juding from approx date stamps and best recollections.

What had happened is the Win OS flaked and I had to transfer to BU MS while I rebuilt new Win Server OS from scratch. Then transferred back with latest updates installed. That appears to show the approx date stamp.
Thanks,
Thomas

User avatar
mattg
Moderator
Moderator
Posts: 20294
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Very old POP3 messages in message directories

Post by mattg » 2015-09-29 13:27

Many inline AntiVirus software (on client computers) will not download a SPAM or Virus mail from a POP3 server.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jimimaseye
Moderator
Moderator
Posts: 8172
Joined: 2011-09-08 17:48

Re: Very old POP3 messages in message directories

Post by jimimaseye » 2015-09-29 13:33

I concur.
jimimaseye wrote:are your clients connecting using POP? By its nature, it is open to mis-configuration or opportunity to not perform as you think it will (ESPECIALLY if there is more than 1 client connecting to the same account). As for these messages, could it be that some PC/client based antivirus software is deleting these messages as they download (protecting the OS) and as such these are not hitting your client
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

Post Reply