BUG: memory leak

[prisma]

Server: 5.4-B2014042101 + postgresql 9.2
Client: Thunderbird 24.6.0 message indexing default (on)

When the user moves a huge number of files around, especially when he does this in an asynchronous way (*), hmailserver memory usage crows permanently. A user let our server run out of memory by doing this.

Tested with 5.4-B1950 and 5.4-B2014042101. But B1950 seems to have less problems, the effect is better to reproduce with B2014042101. But maybe I did not do the right or wrong things with B1950.

(*) Let thunderbird archive all messages (6000+) in incoming to archive subfolders. Move all messages back from single subfolders to incoming, delete all messages in incoming, move them back to incoming, around and around. Do not wait until all headlines or messages are downloaded and hmailserver is idle again.

[Bill48105]

I notice you didn't mention this is on 32bit windows, what version of windows or how much memory the server has. You also did not mention this only happens with thunderbird.

It is a known issue that some users are hitting 32bit memory ceiling with some email clients doing some actions. It might be possible to dig into the code & try to optimize memory usage a bit but unlikely that will be done any time soon. We're working on getting 64bit hmail but that too isn't likely to be anytime soon either so at this point the best thing to do is enforce mailbox quotas or limit client usage. I posted a special build with LARGEMEMORY support but that is not tested & might only help on 64bit windows with enough ram.
Bill

[prisma]

Right Bill, some information are missing:
* It happens with thunderbird not with roundcube.
* It's a 32-bit Server 2008 core with 2 Gig memory
* Memory is not freed after client disconnect

If the client disconnects and the memory is not freed, this is a memory leak and has nothing to do with optional memory optimization. Whatever the client makes "wrong","right" or "different" than other clients. Therefore 64-bit and more memory is no solution.

[Bill48105]

Right Bill, some information are missing:
* It happens with thunderbird not with roundcube.
* It's a 32-bit Server 2008 core with 2 Gig memory
* Memory is not freed after client disconnect

If the client disconnects and the memory is not freed, this is a memory leak and has nothing to do with optional memory optimization. Whatever the client makes "wrong","right" or "different" than other clients. Therefore 64-bit and more memory is no solution.

Thanks for add'l info. Does the memory only leak if there is error or crash or always? There are MANY MANY people who using Thunderbird without complaint.
Bill

[prisma]

Always. No errors on client side nor on server side (until the crash). I'd say, give it 15 minutes. If you aren't able to reproduce it within this time (no debugging, just trying) I'll take a really big spate to dig deeper

Don't know, but only 36 people use the experimental builds (and voted for it). What do the download stats say from mediafire?

[Bill48105]

Always. No errors on client side nor on server side (until the crash). I'd say, give it 15 minutes. If you aren't able to reproduce it within this time (no debugging, just trying) I'll take a really big spate to dig deeper

Don't know, but only 36 people use the experimental builds (and voted for it). What do the download stats say from mediafire?

lol is that a challenge?

The fact you say it happens with official builds too but "worse" with experimentals leads me to believe there is something else exterior coming into play. Not to mention if it only happens with tbird. Sounds like you are implying it's a bug in experimental builds but can't be the case if it happens with official releases too. Especially since I have not submitted any code changes to official releases

Indeed very few people vote in the poll. mediafire stats can be misleading as no way to know how many that download actually use it builds but there have been 480 downloads for 2014 builds & 322 downloads for all 2013 builds. I didn't add them up but looks like around another 500 downloads for 2011-2012 builds combined. So with roughly 1300 downloads hard to say how many actually use it.
Bill

[prisma]

When I see 1950 growing a few KB and I see an experimental build growing multiple MB, I have to report it truthfully. Maybe that's a fault?

[Bill48105]

When I see 1950 growing a few KB and I see an experimental build growing multiple MB, I have to report it truthfully. Maybe that's a fault?

More like it's hard to believe anything changed between them that would be IMAP memory related

[prisma]

Yes.

[percepts]

A lot of people would have this problem I suspect if your setup was relatively common. You are using Postgre which is not so common so just a thought is that hmail is opening a connection or perhaps thousands to postgre and not dropping them when it should.

Just a thought but is there any way you can check what hmail/postgre is doing when this happens and how many open connections there are? Is there a postgre log or open connection inspector?

[prisma]

Hi percepts,

good idea, I thought already into the same direction. And I little bit waited for somebody asking about it

Connections to postgres are very very easy to see. Every connection creates an own server working process. Unix style.
I have seen nothing noticeable in taskmgr during monitoring memory consumption.

Uses martin ODBC or native database functions for postgres? Against which version of postgres components is hmailserver linked?