Email archiving

This forum contains features that has been archived. This section contains implemented features, duplicate requests, and requests which we have decided not to implement.
Post Reply

Do you need this feature?

Yes
33
83%
No
7
18%
 
Total votes: 40

User avatar
mcampbell
Normal user
Normal user
Posts: 148
Joined: 2004-03-30 20:31
Location: North East US

Email archiving

Post by mcampbell » 2005-11-07 17:25

With all the talk of government coming down on companies who don't archive and retain emails (sent and received) I was wondering if there were any plans to include a way to archive emails with retention rules for deleteing, etc? I've looked at 3rd party solutions but before I commit to something like that I'd like to see if HMail is planning anything.
HM 4.2B195 :: MySQL 4.1 :: PHP 4.3.11 :: Win2k
Power over "them"

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2005-11-07 21:18

Do you know what these needs are?

What needs to be stored? The message file, timestamp, from address, sender address?

User avatar
mcampbell
Normal user
Normal user
Posts: 148
Joined: 2004-03-30 20:31
Location: North East US

Post by mcampbell » 2005-11-07 23:28

The message must be stored including headers. Any incoming or outgoing email is subject to archiving and the messages should (must) be stored using encryption so that the orignal contents can not be altered. There should also be an audit trail showing who has done what with the archive and when. This would include searches, deletions and updates. There should be a few articles floating around out there to read on it but I'll see if I can find an online version of the one I was reading last night.

scracha
New user
New user
Posts: 15
Joined: 2004-05-24 02:55

Post by scracha » 2005-12-02 06:02

mcampbell wrote:The message must be stored including headers. Any incoming or outgoing email is subject to archiving and the messages should (must) be stored using encryption so that the orignal contents can not be altered. There should also be an audit trail showing who has done what with the archive and when. This would include searches, deletions and updates. There should be a few articles floating around out there to read on it but I'll see if I can find an online version of the one I was reading last night.
Oh...sounds fancy. I'm just after some way of safely moving emails over a certain date to a different IMAP folder.

User avatar
kenpem
Normal user
Normal user
Posts: 56
Joined: 2005-09-24 11:33

Post by kenpem » 2005-12-06 18:15

Scary! Is this becoming a legal requirement? I presume in the US, land of more lawyers than plumbers? (which always made me giggle - everybody needs a loo, do more people need to sue?) Sounds very Big Brother. Also sounds like a GREAT way to up hosting bills - where's all that data going to be stored? Any idea how long for?

It wouldn't surprise me if this practice is actually illegal in the UK, with the Data Protection Act.

So either way, if this "feature" has to be added, please let us switch it off?

jmilton
New user
New user
Posts: 1
Joined: 2006-04-17 17:05

Post by jmilton » 2006-04-17 17:12

In the financial services world, we need to do this to monitor what is being said to investors to ensure they are not mislead. The retention requirement is, depending on the type of operation, 3 to 7 years.

Tekl
Normal user
Normal user
Posts: 30
Joined: 2006-01-05 13:19

Post by Tekl » 2006-08-23 16:33

Beside the requirement for the encryption. Is there a way to create searchable archives? I want to archive every year on a dvd and there must be the possibillity to search through the messages. So with the archive there must be stored an index-file. Does anyone know about a solution?

Is it possible to use ZOE, copy Mails to it and then burn it onto a DVD so that it directly can be launched from dvd?

mbreitba
Senior user
Senior user
Posts: 340
Joined: 2006-04-14 22:25

Post by mbreitba » 2006-08-23 21:50

Does anyone know if this is actually a requirement? With our email services as large as they are, archiving that much mail would be a nightmare, not to mention expensive over time!

If this is indeed becoming law in the US, we will need this capability, without a doubt.

fozzybear
New user
New user
Posts: 8
Joined: 2006-08-22 22:04

Post by fozzybear » 2006-08-24 21:19

as stated, financial/investment services ARE required to keep such information. I have not seen requirements for other *yet* (and I am not like the legal guru or anything) however, since we do have more lawyer than plumbers, its conceivable that a client of your hosting company can be involved in a suit or criminal investigation and need to produce email related records, such as logs and emails. IMHO from the stuff I have been seeing, its important to have a defined policy reguarding data retention of logs and even emails and make those clearly stated to your clients in things such as the TOS/AUP etc. That way, if you do end up having to legally produce material for court, you can clearly cite your policies and comply within them. If you declare all email logs are detroyed after 30 days and no email archiving is done, you are in a safer position when the client is requesting logs from a year ago, etc. Mainly its about developing the plans a staying with it... you ARE doing backups, right?? Whats your policy on how long you keep them, where they are, and what you do with them after the time you keep them... You could be required to mine 5 years of your CD/DVD backups looking for emails related xyz@domain.com if you have not determined your policy and do not destroy after x time. Just my thoughts and considerations.

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2006-08-24 22:11

as stated, financial/investment services ARE required to keep such information
I think that's depending on what country you live in. I'm pretty sure that the nigerian government doesn't require it. ;-) And as far as I know, no companies are required to keep ALL email. They are required to keep the email which contains information regarding business decisions and so on. (Might be wrong on that)

I don't think you actually need to declare that email logs are destoryed after X days, and that email aren't archived. It's ofcourse a good idéa to inform your clients/customers regarding these things, but I really doubt you have some legal requirement doing so in any country in the world. Say for example that you live in the US and work at a company which needs to comply with the Sarbanes-Oxley Act. Then if you host all your email at some 3:rd-party company, it's your responsibility to make sure that the email which is hosted by the 3:rd-party company is archived for the required time. It's not like you can just expect that your email will be properly archived by some hosting company. Even if the email IS archived, I think making such an assumption would be violating the SOX act..

(That is, if you live in the US where the SOX act actually applies. Which I don't. :)

westdam
Senior user
Senior user
Posts: 728
Joined: 2006-08-01 21:24
Location: Padova, Italy
Contact:

Post by westdam » 2006-08-25 00:17

hi martin, well, i'm italian and the EU recently has passed a law for the privacy and it's important to create single user backup for mail, password, data etc etc..
actually i'm done it with ewall, just saving all the mail in/out for every account..

Tekl
Normal user
Normal user
Posts: 30
Joined: 2006-01-05 13:19

Post by Tekl » 2006-08-25 14:56

Does anyone know an alternative to ZEO? It should provide fast search of all Mails with a web-client.

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2006-08-25 17:06

hi martin, well, i'm italian and the EU recently has passed a law for the privacy and it's important to create single user backup for mail, password, data etc etc..
actually i'm done it with ewall, just saving all the mail in/out for every account..
I doubt that law (which law are you talking about?) applies to all email accounts for all email messages over the entire EU. I live in the EU as well and I know there has been discussions about such laws but I'm sure no laws that has been passed which saves that all email should be stored for everyone...

westdam
Senior user
Senior user
Posts: 728
Joined: 2006-08-01 21:24
Location: Padova, Italy
Contact:

Post by westdam » 2006-08-25 17:18

yeah sorry martin it was a EU proposal of law. but in italy it's a LAW .
i've to create the backup of EVERY mail day by day for each user..
it's for privacy reason...

johndow
Normal user
Normal user
Posts: 79
Joined: 2005-07-24 23:35
Location: Planet earth

Post by johndow » 2006-09-05 00:12

Situation in Germany (Just for information)
Archive:
Tax Law say: Any email with a "possible" tax relevant content must be stored/archived for at least 6 years. If your archive only the relevant emails you have to backup all other emails to show that you have not forget some relevant mails... :roll:

Privacy: Yes, it must be keept.. but to prevent tax crime you have to archive all private emails with a "possible" tax relevant content (privacy ends where tax matters begin). All the others does not need to be archived but you have to make a backup to show that you have only deleted private emails. Privacy law says: do not allow private emails on company equipment.

If you don't make the backups you can not shown that you have not deleted some tax relevant emails. - Don't come in this situation. -

Requirements about encryption etc. are not given. The only requirement is, that you have to provide the emails in an import format (.eml is currently accepted) for automatic parsing by the german tax authority.

jts
New user
New user
Posts: 4
Joined: 2006-12-10 06:57

An absolute requirement.

Post by jts » 2006-12-10 07:02

Seconding all that has been mentioned above. While hMailServer is a fantastic mail server, no business in the US could use it solely for their corporate email because of required retention policies. Recently, the laws have become stricter. Please see: http://www.latimes.com/technology/la-fi ... technology

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2006-12-10 11:31

jts, there's a lot of businesses in the US which uses it. Most companies doesn't have to keep ALL their email, just email which is relevant for tracking business decisions.

Also, if you're working in a business where all electronic information needs to be stored, I don't think you want every software system you use to have it's own way of storing this information considering that you want to store it many years. Would be a nightmare to maintain for larger organizations. They will most likely buy a central tracking system which their other systems can interact with.
Last edited by martin on 2006-12-10 13:27, edited 1 time in total.

bazporter
Normal user
Normal user
Posts: 98
Joined: 2005-06-03 16:14

Post by bazporter » 2006-12-10 12:09

The company I work for supplies document management and archiving solutions for global organisations, and we work in many countries around the world. All the companies we work with have different requirements placed on them for retention of records (including email) depending on the markets they work in and their business type; financial has the most requirements right now it seems.

I have not found any instance where an ISP is required to carry out archiving on behalf of a customer; it is the customer's responsibility to ensure that they retain their own relevant records. If you are providing an ASP service then that would be a whole different ball game, but then you would be charging to provide the retention solution and working to specific SLA's and contractual dates, etc.

Typically our customers choose one of two ways to handle email archiving:
  • 1. They archive every outbound email. No questions.
    2. They educate their users and provide guidlines on the types of emails that are to be archived for the purposes of satisfying regulatory compliance.
I think that it would be a massive undertaking for Martin to start trying to build an archiving mechanism into hMailServer. hMailServer is an excellent mail server,it should not attempt to become an archiving solution.

To make it easier for those who have to comply with regulations requiring the archiving of email maybe Martin could create a mechanism (selectable) that would make a copy of every outgoing email and place it in a folder at the point at which the the mail is delivered to an external mail server. If the folder is selectable, that will allow administrators to set a convenient location, and the extension also configurable for those who would require .eml instead of .txt or .msg then that would also help. The file name would only need to consisit of a timestamp as the content should be searcheable by the ultimate storage solution.

It is then up to the Administrator to configure a mechanism to move the files to a structured location, such as a document management system like Captaris Alchemy, Documentum AX5, Sharepoint, etc. or even a specific email archiving tool like KVS Vault. The files could even be kept as discrete text files in a folder location organised by date, which would be relatively easy to achieve using scripting, for those not wishing to invest in a specific solution.

Just my two pence.

--
Regards
Barry

jts
New user
New user
Posts: 4
Joined: 2006-12-10 06:57

Basic Retention

Post by jts » 2006-12-10 18:21

Martin, just to clarify, I would never leave the decision to keep relevant emails up to an end-user. What about cases of sour employees?

It actually would not be a nightmare to maintain, as it may cost more in legal fees for a company to defend against. Barry, I'm looking more at a SMB business than an ISP based model. As all the messages are stored in an EML format, could a simple flag and archive mode simply be added to make a readonly copy to another folder? It would then be up to the admin/corporate retention policy to archive the mail.

A small brokerage in the US, for instance, is required to keep all correspondences of a stock trade for six years. This is all incoming/outgoing.

Please forgive my ignorance about the EU, but I thought there was a new retention law that required keeping all sorts of material? I guess my long winded point is: with storage being extremely cheap (long term/short), what would be the detriment of adding this basic feature? hMailServer could then start touting the fact that they are in compliance with a majority of the Sarbanes-Oxley ACT policies (http://articles.techrepublic.com.com/51 ... 34345.html).

While I have six (gray hair growing years) of experience with MS Exchange, my limited experience with hMailServer has absolutely blown me away. It is a fantastic product in all regards, but is only missing this one little feature. Cheers on a fantastic product!

(Note: I am also very impressed with the civility and intelligence of this forum. A very rare find amongst other Open Source projects!)

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2006-12-10 18:35

Agree that letting end-users be responsible for this would be a a risk.

Using the OnDeliverMessage-event, it wouldn't take more than some hour to write a script which copies all outgoing and incoming email to a directory structure and have the script sort the email by year/month/date/sender-domain or something similar.

Even among smaller organizations and companies, I suspect that there are many different wishes on how this should be done. Because of this, I think it would be wiser to implement this as a simple hMailServer script as a first step. This would give much more flexibility.

Even if disk space is cheap, storing something in 6 years requires a bit of work. You need to ensure backups of this directory is working properly etc, so even if some standard functionality was added to hMailServer, some manual work by the server admin would be required.

When such a script has been created (I might create one myself) it could be put up here in the scripts section of the forum. Perhaps it could be added to the installation as well. From a server admin perspective, I don't see any big immediate benefit of having this as a standard feature. Enabling this as a script wouldn't take many minutes for the sysadmin anyway.

jts
New user
New user
Posts: 4
Joined: 2006-12-10 06:57

Post by jts » 2006-12-11 07:14

martin, good suggestion. I have started a very very basic one here: http://www.hmailserver.com/forum/viewto ... 9481#39481

I apologize for the cross post, however I thought it was very relevant to the discussion and to Martin's suggestion.

chameenz
New user
New user
Posts: 29
Joined: 2004-06-23 16:48
Location: Baltimore, MD

Post by chameenz » 2006-12-22 03:51

I looked at hmail's scripting because I need to accomplish a sorted archiving exactly like what you are describing. I could not find a way to move messages into imap folders using hmails functions. Is this possible?

johndow
Normal user
Normal user
Posts: 79
Joined: 2005-07-24 23:35
Location: Planet earth

Post by johndow » 2006-12-23 01:23

Just my two cent on archiving:

Martin has a mirror function to send every mail from hmailserver to a second email adress (local or somewhere else).

Curently I use this function and copy manual each month the emails via IMAP to a montly Subfolder.

Question to Martin:
How much work would it be to modify the currently existing mirror function in a way that, if sending on a local email account, you create automaticly subfolders (weekly, montly, yearly) via IMAP in the incomming mails?

There is a commercial mailserver who works in this way and this mailserver is sold and used in the USA. The manufacturer is in the USA...

Name of the Mailserver wil be given only to martin via PM.

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2006-12-23 01:31

Come on... "Martin has a mirror function"? The software is named hMailServer - not Martin. I'm not a piece of software. =)

Would take somewhere between 4 and 20 hours. Easiest way is probably to write a short script which updates a rule once a day. Then it will only take some hour at most to implement it. Anyway, I'm not so sure it sounds like a good idea. Doing long-time archiving of email from a system in the system itself sounds risky. Both from a legal and architectural perspective, it sounds like a worst-case solution.

Long-term backups and archives for legal purposes should be stored physically separated from the system you are performing backup/archiving of. Feels much more safe to have them placed in files on disk, so that they can be easily compressed and moved to backup tapes / disks if needed.

johndow
Normal user
Normal user
Posts: 79
Joined: 2005-07-24 23:35
Location: Planet earth

Post by johndow » 2006-12-23 01:45

Come on... "Martin has a mirror function"? The software is named hMailServer - not Martin. I'm not a piece of software. =)
Sorry, but a good sample whats happen when I am writing to early in a morning. :lol:
Would take somewhere between 4 and 20 hours. Anyway, I'm not so sure it sounds like a good idea. Doing long-time archiving of email from a system in the system itself sounds risky. Both from a legal and architectural perspective, it sounds like a worst-case solution.

Backup and archiving should be physically separated from the system you are performing backup / archiving of. Feels more safe to have them placed in files on disk, so that they can be easily compressed and moved to backup tapes / disks if needed.
Totaly agree. But take a look at the commercial demo.
They store the archive on another hard disk (if wanted) and store everything in .eml files. The Software is only used to have it more comfortable to take a look at them.
Nevertheless, a cheap, not complete professional archiving but better as nothing.

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2006-12-23 01:58

Nevertheless, a cheap, not complete professional archiving but better as nothing.
If the person who uses it knows that it's incomplete, may not be enough to be compliant with the legal requirements, and that it's an unsecure way to do it. Which they never do. :) Anyway, since I had 5 minutes to spare, I've added support for using %YEAR%, %MONTH% and %DAY% macros in the "Move to IMAP folder" rule action for 4.4. This way you can move messages to IMAP sub folders using a simple rule and you don't have to do it manually.

User avatar
Slug
Moderator
Moderator
Posts: 1369
Joined: 2005-03-13 05:42
Location: Sydney Australia
Contact:

Post by Slug » 2006-12-23 05:33

martin wrote:Anyway, since I had 5 minutes to spare, I've added support for using %YEAR%, %MONTH% and %DAY% macros in the "Move to IMAP folder" rule action for 4.4. This way you can move messages to IMAP sub folders using a simple rule and you don't have to do it manually.
I like that ..

Thanks
Michael
Missing Hmailserver ... Now running Debian servers

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2007-02-20 23:22

bazporter,
Just a bit curious, is Sharepoint really suitable to handle large amounts of email - for example all email for a domain? (I've just started to use Sharepoint a few weeks ago so have very little experience in it so far)

User avatar
harddiskman
New user
New user
Posts: 7
Joined: 2007-04-07 19:09

Post by harddiskman » 2007-04-21 21:07

Martin,

Could you please explain us a little bit more about Sharepoint.. Can we use MS Sharepoint 3.0 with hmailserver? ...

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2007-04-22 10:29

harddiskman,
No, sorry I have no clue. :) I've used Sharepoint a little bit but still have very little experience. There's definitively no integration between hMailServer and Sharepoint if that's what you're wondering..

Henry Sværke
Normal user
Normal user
Posts: 46
Joined: 2007-09-08 16:46

Post by Henry Sværke » 2007-09-08 17:40

First off, thank you heaps for this great software and second but not the least, hello forum.

Now to the point. One of my employees is now facing charges for CP distribution. Apparently he operated from his place, but I was contacted by the local police and they told me I had to let them access the mail archive because "we *suspect* he could have abused your smtp server", words from the law. Unfortunately I was keeping none as I thought it wasn't all that necessary, hell, I didn't even bother keeping any logs (afteralls, the company was just 5 people I trusted blindly) but they took that as uncooperativeness. A week later the whole situation was really hittin' the fan..

Anyways, I'd like to show thumbs up for this feature request since I'd feel safer if my server could store mail for manual approval and later delivery (keep in mind we work with lots of graphics and even videos, so not allowing attachments would be rather counterproductive)

Regards.

nstevens
Normal user
Normal user
Posts: 70
Joined: 2005-04-10 18:00
Location: California, USA

Re: Email archiving

Post by nstevens » 2008-04-22 04:47

Does any one know of any third party software that could work with HmailServer and capture incoming and outgoing emails for archiving, or any updated scripts that could be used?

User avatar
mattg
Moderator
Moderator
Posts: 19979
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Email archiving

Post by mattg » 2008-04-22 06:21

nstevens wrote:Does any one know of any third party software that could work with HmailServer and capture incoming and outgoing emails for archiving, or any updated scripts that could be used?
There is the mirror function. You could send all incoming and outgoing email to a separate server.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
krescenilo
New user
New user
Posts: 1
Joined: 2008-04-23 19:43
Location: Illinois

Re: Email archiving

Post by krescenilo » 2008-04-23 19:48

First hello all,

I've been messing around with hMailServer. I tried an archive software called MailArchiva with hMailServer + the mirror function. And it seems to be working. Though I'm still experimenting with it using a virtual machine. And have only used it with the Open Source Edition myself.

mattg wrote:
nstevens wrote:Does any one know of any third party software that could work with HmailServer and capture incoming and outgoing emails for archiving, or any updated scripts that could be used?
There is the mirror function. You could send all incoming and outgoing email to a separate server.
"Life is but a walking shadow a poor player that struts and frets his hour upon the stage and then is heard no more! It is a tale told by an idiot full of sound and fury signifying nothing!"

at0mic
Normal user
Normal user
Posts: 34
Joined: 2007-10-20 18:53

Re: Email archiving

Post by at0mic » 2008-04-25 19:11

In the last place I worked at, I used Exchange for email. There's an option to send a copy of all incoming/outgoing mail to the user account of your choice. I set this to the administrator email account.

In the administrator's imap account at the end of each month, I put all the email for that month into a folder named after the month. Next, I moved the month from 4 months ago into a PST file and saved it to DVD. Each DVD could hold 3 or 4 months worth of email (each month had its own PST file).

I always kept the last three months worth of email in the administrators imap email account in case it was needed. People often accidentally deleted an important email that they needed. It was very quick and easy to find this email in the administrator account and drag it to the users account.

Post Reply