Moving many messages between IMAP folders = duplicate hell

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
Magnus W
Normal user
Normal user
Posts: 162
Joined: 2007-04-17 15:30

Moving many messages between IMAP folders = duplicate hell

Post by Magnus W » 2009-01-26 20:02

Hi all... something that's been bugging me for a while (1.5 years in fact):

If you select about a thousand mails or more (in Thunderbird) and move them to a different IMAP folder, the server seems to queue the moving operation many times, copying the files and easily filling the server hard drive.

1) Is this a known bug?
2) I'm curious about this behavior: What business has it creating new files anyway, I thought it was a pointer in a database and it should be a (number of) simple (hum) UPDATE query(s) not involving any actual file operations?

User avatar
mattg
Moderator
Moderator
Posts: 20960
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Moving many messages between IMAP folders = duplicate hell

Post by mattg » 2009-01-27 08:33

This is a Thunderbird IMAP implementation issue.

With IMAP, the original file is ONLY marked for deletion, not actually deleted, whilst a copy is created an placed in the new directory

Search in Thunderbird advanced settings for 'purge' and see if forcing a 'purge' helps you.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

Magnus W
Normal user
Normal user
Posts: 162
Joined: 2007-04-17 15:30

Re: Moving many messages between IMAP folders = duplicate hell

Post by Magnus W » 2009-01-27 15:19

Yes, of course, there is no "move" command in IMAP, I keep forgetting this :-( Thunderbird must get some kind of timeout, initiating the copy again?

The relevant setting should be mail.imap.expunge_after_delete set to TRUE. Hope this fixes it.

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: Moving many messages between IMAP folders = duplicate hell

Post by martin » 2009-01-27 16:03

Have heard this before but not been able to reproduce the "re-queing".

Typically, hMailServer doesn't do anything unless told so by a client. So I'm guessing that you may be right that there's a timeout and Thunderbird then tells hMailServer to do the operation again.

Try moving a smaller number of messages, like 30 and see if that works. If it does, I guess that's a confirmation of Thunderbird asks hMailServer to "redo" the message copying.

Magnus W
Normal user
Normal user
Posts: 162
Joined: 2007-04-17 15:30

Re: Moving many messages between IMAP folders = duplicate hell

Post by Magnus W » 2009-01-27 16:25

I'll try to check under which circumstances this is happening (my clients get it all the time) but IIRC the threshold is ~1000 messages.

Just curious about the internal workings of hMailServer; obviously it makes an actual copy of the file when issued a copy command. Wouldn't it be more storage, I/O and time efficient to make a SQL table row copy pointing at the same file, and then at deletion, check for other references to the same file before removing the actual file?

(of course, in this case, the condition experienced would fill the database instead of the disk, but anyway...)

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: Moving many messages between IMAP folders = duplicate hell

Post by martin » 2009-01-27 16:31

Yes, I've thought about reusing message files for multiple rows. Kind of complex to implement in a good way but far from impossible. Haven't had time to do it though. There's other similar things as well (reusing the same physical message file for multiple accounts as an example).

I just tried it here with 2500 messages and did not experience the problem. But maybe since my server is not under high load and I'm connecting from localhost, the response is faster and does not cause any timeout. :-\

If you could provide me with IMAP logs covering the problem it would be nice. Can be sent to martin@hmailserver.com.

Magnus W
Normal user
Normal user
Posts: 162
Joined: 2007-04-17 15:30

Re: Moving many messages between IMAP folders = duplicate hell

Post by Magnus W » 2009-01-27 18:12

martin wrote:Yes, I've thought about reusing message files for multiple rows. Kind of complex to implement in a good way but far from impossible. Haven't had time to do it though. There's other similar things as well (reusing the same physical message file for multiple accounts as an example).
Yes, for messages delivered to many people at the same time it sure makes a lot of sense. Exchange did this, which was a plus in my opinion, but they largely removed this functionality in Exchange 2007, single-instance-storing attachments but not message bodies. With hMailServer's current architecture, separating attachments is hardly possible. An interesting discussion of the change and why it was made is here:
http://msexchangeteam.com/archive/2008/ ... 48095.aspx

I can't quite understand why they say that this reduced I/O. Maybe they are using a file database for the index too? I know too little about the internals of Exchange Message Store.
martin wrote:I just tried it here with 2500 messages and did not experience the problem. But maybe since my server is not under high load and I'm connecting from localhost, the response is faster and does not cause any timeout. :-\

If you could provide me with IMAP logs covering the problem it would be nice. Can be sent to martin@hmailserver.com.
Are you also using Thunderbird? I'll try to nail down this issue, had a small downage yesterday when the server completely filled up and everything went kinda sour :-P

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: Moving many messages between IMAP folders = duplicate hell

Post by martin » 2009-01-27 18:30

Yes, I'm using Thunderbird as well to test it...

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: Moving many messages between IMAP folders = duplicate hell

Post by martin » 2009-01-27 19:09

I saw some strange behavior when moving 10 000 messages. I will see if I can create a log out of it...

Edit: False alarm. I thought Thunderbird has hanged but I enabled logging in Thunderbird and saw that it was just slow. It spent 2-3 minutes parsing response from hMailServer and during that time it appeared as "hanged".

Magnus W
Normal user
Normal user
Posts: 162
Joined: 2007-04-17 15:30

Re: Moving many messages between IMAP folders = duplicate hell

Post by Magnus W » 2009-01-27 22:59

Was this in/from the inbox or another folder? Have you set mail.imap.expunge_after_delete, or any other purging-related settings? Are you using the stable Thunderbird or the beta?

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: Moving many messages between IMAP folders = duplicate hell

Post by martin » 2009-01-28 18:44

I've tried different combinations of folders (both including and not including inbox). Imap.expunge_after_delete is set to off I believe (the default value at least). I've tried both using the latest stable (2.0.0.20) and 3.0 beta.

About the performance issue I saw in Thunderbird - it was reported as a bug 4 years ago and is not fixed yet. Reading the long history of the bug report, it looks like the developers consider it to be a boring bug to fix.
https://bugzilla.mozilla.org/show_bug.cgi?id=296453

Bladeinger
Normal user
Normal user
Posts: 49
Joined: 2008-06-28 07:11

Re: Moving many messages between IMAP folders = duplicate hell

Post by Bladeinger » 2009-02-18 21:08

I encountered the same problem with a fresh hmailserver 5 installation last week. Customers gave me a call, because the server, on which hmailserver is running (it's a server 2003) was not responding anymore. We never had that in 3 years before. We had to hard-reset the server. After reboot I could see, that in one account the amount of mails had grown from about 6 GB to 50 GB! Investigation shows, that one user wanted to move about 2500 mails in his inbox to a newly created subfolder of inbox. When I looked in this folder, there were 150.000 (!!!) mails in it, every mail was multiplied 50-60 times.

The problem is reproducable on this server and also takes place when only a few hundred mails are selected for moving. Mail client is Thunderbird 2.0.0.19, runing under XP Professional.

When the mails are moved to another account on the same server, within the same thunderbird-installation, the problem doesn't happen, only moving mails within the same account shows this symptom.
The problem can also be reproduced when connecting with thunderbird to hmailserver from any other client computer.

Is there already a solution for this problem?

DeanoX
Senior user
Senior user
Posts: 480
Joined: 2005-11-05 00:07
Location: Michigan

Re: Moving many messages between IMAP folders = duplicate hell

Post by DeanoX » 2009-02-18 21:39

Looking at Martin's last post, this is a bug in Thunderbird that has not yet been fixed.

Try using another client, or a webmail program.

I know that using SM under hmail IMAP, I can move several k's of email without issue.


Thanks,
-Dean
hMailServer 5.4.2-1964, mysql, ClamAV, SpamAssassin, SquirrelMail, GeoIP.
hMailServer Support Services for US Based Clients.
Low Rates, Quick Service. Send a Private Message for More Information.

Bladeinger
Normal user
Normal user
Posts: 49
Joined: 2008-06-28 07:11

Re: Moving many messages between IMAP folders = duplicate hell

Post by Bladeinger » 2009-02-19 06:35

I am not sure if this is a thunderbird bug at all. I'd rather assume , that it's a kind of compatibility issue between thunderbird and hmailserver 5.
I'll tell you, why I believe that.

I take care for a few thousand workstations of small and mid-size companies, about 90 percent of the clients are using thunderbird as mail client.
In the last 3 years we switched nearly all of them from pop3 to imap mail. After doing that, moving large ammounts of mail in subdirectories is a very common action taken by the users to reduce the response times when switching between directories within thunderbird.
And we never encountered this problem before, even if thousands of mails were moved. And as we are using hmailserver at all new installations since spring 2008 and didn't have any of these problems before, I wonder, if hmailserver 5 handles this action in another way than hmailserver 4 does? It's our first installation of hmailserver 5.

When I find the time this weekend, I'll try two different things: I'll install another mail client to see, if the problem also occurs. And I'll install a second mail server as well, to see if this problem can be reproduced in the same enviroment with thunderbird as mail client.
My guess is, that in both cases the problem will not happen. We will see...

If anyone is interested in server logs or other information, I will be glad to send any information I can.

westdam
Senior user
Senior user
Posts: 728
Joined: 2006-08-01 21:24
Location: Padova, Italy
Contact:

Re: Moving many messages between IMAP folders = duplicate hell

Post by westdam » 2009-02-19 18:47

so, anyone can suggest a good imap client?
outlook express is quite buggy
thurnerbird is slow.


on linux side i've use evolution and it's not so bad but still slow...

Bladeinger
Normal user
Normal user
Posts: 49
Joined: 2008-06-28 07:11

Re: Moving many messages between IMAP folders = duplicate hell

Post by Bladeinger » 2009-02-23 15:17

Ok, yesterday and today I found time to do some tests.
First of all: My guess was wrong! It's NO incompatibility between Thunderbird and hmailserver 5, it's definitely a Thunderbird bug.

I installed another mail client (Pegasus) to access the mails, performed the same mail-move operations, and it was done without any problem. After that, I installed a second mail server on that machine, copied the mails from the hmailserver-account to a new account on the new mailserver, accessed this account via Thunderbird and performed again the same move-operations. Immediately the mails were copied over and over again. So the result was clear: Thunderbird was the reason for the problem.

The bug is already listed at Bugzilla: https://bugzilla.mozilla.org/show_bug.cgi?id=381867 They categorized the importance of this bug as "minor", which gives very few hope, that the bug will be fixed soon.

But nevertheless, in the comments to this bug, I found a possibility to avoid this behaviour: After establishing and IMAP-SSL connection between Thunderbird and hmailserver, the problem did not happen again.

charleslyk
New user
New user
Posts: 10
Joined: 2008-01-01 05:37

Re: Moving many messages between IMAP folders = duplicate hell

Post by charleslyk » 2009-05-16 12:20

In case anybody faces the same issue using Thunderbird, the resolution will be to use

Remove Duplicate Messages (Alternate)
https://addons.mozilla.org/en-US/thunderbird/addon/4654

Works well for me..

matthew1471
New user
New user
Posts: 17
Joined: 2010-06-20 21:59

Re: Moving many messages between IMAP folders = duplicate hell

Post by matthew1471 » 2019-03-07 16:27

I don't trust add-ons and wanted to have control over the process (performing full content hashing etc. to ensure I wasn't deleting some that just look similar from the headers alone). For those who care for C# and use MySQL, there's some source code here : viewtopic.php?f=21&t=33675

shapeshifter910
New user
New user
Posts: 19
Joined: 2011-11-08 12:16

Re: Moving many messages between IMAP folders = duplicate hell

Post by shapeshifter910 » 2019-03-28 12:14

Still a problem.
Only hmailserver does this.
It's super annoying and will force me to abandon hmailserver.
Never had this problem with servers from Bluehost, GoDaddy and many others.
Even deleting more than a few hundred emails will time out hmailserver and the emails come back again.
I'm using the latest stable hmailserver and mariadb that came with XAMPP 7.2.7

User avatar
mattg
Moderator
Moderator
Posts: 20960
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Moving many messages between IMAP folders = duplicate hell

Post by mattg » 2019-03-29 02:26

shapeshifter910 wrote:
2019-03-28 12:14
Even deleting more than a few hundred emails will time out hmailserver and the emails come back again.
Not for me

Not ever

Have you set timeout options for IMAP and POP3 in your hMailserver.ini

I think these still work - and no there is no documentation other then what is shown below

Code: Select all

[settings]
;***********************************************************************
; these are from Bill'a Alpha builds and may or may not coninue to work
;***********************************************************************
; still current as at 18/12/2015 

POP3DMinTimeout=30
; 10 seconds is default which becomes 15 internally

POP3DMaxTimeout=90
; 600 seconds is default

SMTPDMinTimeout=90
; 10 seconds is default which becomes 15 internally

SMTPDMaxTimeout=300
; 1800 seconds is default

POP3CMinTimeout=300
; 30 seconds is default

POP3CMaxTimeout=900
; 900 seconds is default

SMTPCMinTimeout=90
; 30 seconds is default

SMTPCMaxTimeout=300
; 600 seconds is default
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jimimaseye
Moderator
Moderator
Posts: 8677
Joined: 2011-09-08 17:48

Re: Moving many messages between IMAP folders = duplicate hell

Post by jimimaseye » 2019-03-29 09:57

I have experienced this condition myself some years ago where trying to move (drag/drop) 5000 emails from a folder in to another folder (or doing an IMAP delete). My conclusion was that my server wasnt of high enough power and I wasn't patient enough for it to finish but frankly I wouldn't really know. Consequently, now, I only ever move or delete up to 800 (arbitrary number) at a time to avoid pushing my luck and monitor the process to ensure that each 'move' has completed before I do the next block.

Is it TB? Is it HMS (doubt it)? Or is it the server performance? Or maybe its a combination.
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

shapeshifter910
New user
New user
Posts: 19
Joined: 2011-11-08 12:16

Re: Moving many messages between IMAP folders = duplicate hell

Post by shapeshifter910 » 2019-04-02 04:35

Well, yeah, you could say the server is under-powered by today's standards (it's a quad core Pentium class system) with 4 Gb of RAM and running Server 2008 R2.
But nothing else is installed and/or running on it, so it's a "lean" install, and it's hosting only couple email domains with no more than a dozen accounts ( and most are used very sparingly).
But even if it is under powered, there is no reason for software to malfunction like that. At most it should run slow but not exhibit this kind of screw-up.
It's a software fault and it's gotta be hmail, because like I said, TB works fine with every other email service.
I'll check the time-outs, but I have never had to ask any other provider to adjust any timeouts on their server.

User avatar
mattg
Moderator
Moderator
Posts: 20960
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Moving many messages between IMAP folders = duplicate hell

Post by mattg » 2019-04-02 11:25

shapeshifter910 wrote:
2019-04-02 04:35
It's a software fault and it's gotta be hmail,
so you've tested other MTAs on the same hardware..?
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

westdam
Senior user
Senior user
Posts: 728
Joined: 2006-08-01 21:24
Location: Padova, Italy
Contact:

Re: Moving many messages between IMAP folders = duplicate hell

Post by westdam » 2019-04-02 11:40

It's an old problem. It's related with imap protocol and multithread client ..especially Thunderbird. Also can improve a separate DB with decent performance on writing. In the past I had same issue I've made a Linux MySQL separate installation with a low latency kernel and moving small crunch of email max 300/500. But it's passed a lot of.time , maybe right now is different. Never tried with mariadb.

Also try to move mail from webmail.

shapeshifter910
New user
New user
Posts: 19
Joined: 2011-11-08 12:16

Re: Moving many messages between IMAP folders = duplicate hell

Post by shapeshifter910 » 2019-04-05 00:31

I have tried webmail too and same thing.

shapeshifter910
New user
New user
Posts: 19
Joined: 2011-11-08 12:16

Re: Moving many messages between IMAP folders = duplicate hell

Post by shapeshifter910 » 2019-04-05 00:39

mattg wrote:
2019-04-02 11:25
shapeshifter910 wrote:
2019-04-02 04:35
It's a software fault and it's gotta be hmail,
so you've tested other MTAs on the same hardware..?
Not the exact same hardware, but even slower, older hardware, running postfix and sendmail and never had this problem.
And I've noticed, as an IT tech, I am often tasked with clean up of my clients emails. So I've spent countless hours moving, copying, deleting volumes of emails using most of the common email clients - Outlook, TB, Mac mail etc.
My observation is that when the volume of emails is large, the process slows down, but never stops, and never I have seen the emails just show back in the original folder where they were moved/deleted from.
Could it be the MariaDB? Has Hmail been tested with MariaDB?
Thanks for at least participating in the discussion :)

Virinum
Normal user
Normal user
Posts: 105
Joined: 2018-11-23 14:42
Location: Germany

Re: Moving many messages between IMAP folders = duplicate hell

Post by Virinum » 2019-04-05 09:25

Maybe an antivirus scanner is messing around with your email files?

My setup is using MariaDB (10.3.14) and it works like a charm. My VPS has 4 GB RAM and 2 vCores.

shapeshifter910
New user
New user
Posts: 19
Joined: 2011-11-08 12:16

Re: Moving many messages between IMAP folders = duplicate hell

Post by shapeshifter910 » 2019-04-07 18:13

Virinum wrote:
2019-04-05 09:25
Maybe an antivirus scanner is messing around with your email files?

My setup is using MariaDB (10.3.14) and it works like a charm. My VPS has 4 GB RAM and 2 vCores.
Thanks. That's good to know.
I will update to the latest XAMPP and see whether it'll help.

Post Reply