SEARCH CHARSET US-ASCII ALL UNDELETED Subject ...

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
evb
New user
New user
Posts: 14
Joined: 2008-11-05 09:33
Location: Belgium

SEARCH CHARSET US-ASCII ALL UNDELETED Subject ...

Post by evb » 2008-11-06 01:30

I can't connect anymore to hmailserver (4.4)
I'm using squirrelmail to connect. It did work with no problems till a week ago.
Now I get following error in the browser :

ERROR: Connection broken with the IMAP server (Verbinding verbroken door IMAP server)
Query: SEARCH CHARSET US-ASCII ALL UNDELETED Subject Jombackup

--> JomBackup is an IMAP folder created by me

What is the query about?
He is searching all the undeleted mails? in the IMAP folder JomBackup?
hmailserver 5.3.3-B1879
MySQL server Ver 5.5, for Win32

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: SEARCH CHARSET US-ASCII ALL UNDELETED Subject ...

Post by martin » 2008-11-06 01:38

Do you get that error directly when trying to log on?

The client seems to search for all message with the text JomBackup in the subject...

Is JomBackup just an IMAP folder you created using your email client? Or have you entered "JomBackup" in other places? (In some configuration perhaps?)

User avatar
dzekas
Senior user
Senior user
Posts: 2486
Joined: 2005-10-13 21:28
Location: Lithuania

Re: SEARCH CHARSET US-ASCII ALL UNDELETED Subject ...

Post by dzekas » 2008-11-06 10:15

hMailServer failed to respond fast enough. PHP closed connection.

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: SEARCH CHARSET US-ASCII ALL UNDELETED Subject ...

Post by martin » 2008-11-06 10:16

Yes, but why would it search for a specific subject string when you "connect"?

evb
New user
New user
Posts: 14
Joined: 2008-11-05 09:33
Location: Belgium

Re: SEARCH CHARSET US-ASCII ALL UNDELETED Subject ...

Post by evb » 2008-11-06 10:39

Your remark about entering the word "jombackup" did ring a bell :-)

I did use this word in a squirrelmail rule!
So I deleted the squirrelmail rules directly in the user prefs table of the squirrelmail database and bingo, squirrelmail did do a login.
I saw that the inbox did contain 1500 mails...
In the manual of squirrelmail, they say that serverside filtering is more performant, so I will give it a try with the rules in hmailserver (must do yet RTFM in the manual ;-) )

Remark : because of the 1500 mails, I clicked in squirrelmail the option to sort on the from field, and again the error :
FOUT: Verbinding verbroken door IMAP server
Query: SORT (FROM) UTF-8 ALL

I don't know anything about the IMAP protocol between the mailclient and the IMAP server, but if I understand it well, squirrelmail does send a sort of query to the IMAP server?
Apparently if too many mails then hmailserver rejects the connection?
I have the IMAP sort extension actif and in squirrelmail I activated the option to use the server IMAP sort extension.

Or is it PHP who closes the connection? Can I configure this?
(did put max_execution_time all to 120 seconds, but no change)
hmailserver 5.3.3-B1879
MySQL server Ver 5.5, for Win32

User avatar
dzekas
Senior user
Senior user
Posts: 2486
Joined: 2005-10-13 21:28
Location: Lithuania

Re: SEARCH CHARSET US-ASCII ALL UNDELETED Subject ...

Post by dzekas » 2008-11-06 11:11

evb wrote: I saw that the inbox did contain 1500 mails...
In the manual of squirrelmail, they say that serverside filtering is more performant, so I will give it a try with the rules in hmailserver (must do yet RTFM in the manual ;-) )

Remark : because of the 1500 mails, I clicked in squirrelmail the option to sort on the from field, and again the error :
FOUT: Verbinding verbroken door IMAP server
Query: SORT (FROM) UTF-8 ALL

I don't know anything about the IMAP protocol between the mailclient and the IMAP server, but if I understand it well, squirrelmail does send a sort of query to the IMAP server?
Apparently if too many mails then hmailserver rejects the connection?
I have the IMAP sort extension actif and in squirrelmail I activated the option to use the server IMAP sort extension.

Or is it PHP who closes the connection? Can I configure this?
(did put max_execution_time all to 120 seconds, but no change)
Socket timeouts are hardcoded in SquirrelMail scripts and PHP. SquirrelMail sets timeout only on connect operation. Read and write operations use default timeout values, because functions used to control stream timeouts are introduced only in PHP 4.3.0. I have never used or tested them. Don't know if functions are stable enough and they don't have issues or quirks.

"Verbinding verbroken door IMAP server" is "Connection dropped by IMAP server". You just have to switch from Dutch to English in order to see original error message. If it happens after some long operation, connection is usually dropped by PHP and not by IMAP server.

SORT extension won't help in SEARCH operations and 1500 mailbox is not large one. You should not see errors even when server side sorting is turned off. Code would use unoptimized PHP based filtering, but it should not hit PHP memory limits on 1.5K mailbox.

evb
New user
New user
Posts: 14
Joined: 2008-11-05 09:33
Location: Belgium

Re: SEARCH CHARSET US-ASCII ALL UNDELETED Subject ...

Post by evb » 2008-11-07 09:26

So, like already said, with no filtering actif, squirrelmail was able to read my inbox of 1500 mails.
I deleted my mails to have a 200 mails left and then again I activated the sort.
Now I did not receive an error. It works like before.

Because I wanted to use this system for all the mailaccounts (yet only one mailaccount is activated on the hmailserver to test it), I must now what's the reason of this not wanted feature. I don't want this to happen on the users mailboxes.

I did read the php log and find out that the error happens on the php line :
[04-Nov-2008 20:15:25] PHP Fatal error: Maximum execution time of 30 seconds exceeded in C:\Inetpub\wwwroot\CangrejoMail-2.0.2\functions\imap_general.php on line 92
--> if (!($read = fgets($imap_stream, $buffer)))

Code: Select all

/**
 * Custom fgets function: gets a line from the IMAP server,
 * no matter how big it may be.
 * @param stream imap_stream the stream to read from
 * @return string a line
 */
function sqimap_fgets($imap_stream) {
    $read = '';
    $buffer = 4096;
    $results = '';
    $offset = 0;
    while (strpos($results, "\r\n", $offset) === false) {
        if (!($read = fgets($imap_stream, $buffer))) {
        /* this happens in case of an error */
        /* reset $results because it's useless */
        $results = false;
            break;
        }


Because this forum is about the hmailserver, I will not ask for help on the squirrelmail side ;-)
But my question about the error received is not yet answered:
Connection dropped by IMAP server
Query: SORT (FROM) UTF-8 ALL

For me this is the query sent to the imap server when I did click the sort on the FROM field in squirrelmail.
Does the IMAP server don't understand this?
Or goes something wrong with a 1500 mails mailbox? (I did the same with a 200 mails mailbox and there no problem)
hmailserver 5.3.3-B1879
MySQL server Ver 5.5, for Win32

User avatar
dzekas
Senior user
Senior user
Posts: 2486
Joined: 2005-10-13 21:28
Location: Lithuania

Re: SEARCH CHARSET US-ASCII ALL UNDELETED Subject ...

Post by dzekas » 2008-11-07 09:44

evb wrote: Because this forum is about the hmailserver, I will not ask for help on the squirrelmail side ;-)
Other IMAP servers (Courier, Cyrus, UW, Dovecot) don't trigger this error. Only hMailServer does.

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: SEARCH CHARSET US-ASCII ALL UNDELETED Subject ...

Post by martin » 2008-11-07 10:03

Other IMAP servers (Courier, Cyrus, UW, Dovecot) don't trigger this error. Only hMailServer does.
Sounds like I need to look into this. :)

Dzekas, so the problem is that hMailServer is too slow on responding to the SORT command. And it's not possible to fix it on the SquirrelMail side since the timeout is hardcoded, right? If I understand that properly, the problem may occur on Courier, Cyrus, UW, Dovecot as well if the mailbox is large enough. At least in theory? (Since in theory, sorting an infinite number of messages will take longer than the hardcoded SquirrelMail-timeout).

Maybe creating some kind of indexing on these fields would help. :-\

User avatar
dzekas
Senior user
Senior user
Posts: 2486
Joined: 2005-10-13 21:28
Location: Lithuania

Re: SEARCH CHARSET US-ASCII ALL UNDELETED Subject ...

Post by dzekas » 2008-11-07 10:22

martin wrote:
Other IMAP servers (Courier, Cyrus, UW, Dovecot) don't trigger this error. Only hMailServer does.
Sounds like I need to look into this. :)
http://home.gna.org/nasmail/profiling/imap.shtml
Dzekas, so the problem is that hMailServer is too slow on responding to the SORT command. And it's not possible to fix it on the SquirrelMail side since the timeout is hardcoded, right?
It might be possible to catch this error, but result would be SquirrelMail generated error box instead of PHP errors. SquirrelMail should handle these "PHP and not IMAP server dropped connection" errors, but catching them won't fix the issue.
If I understand that properly, the problem may occur on Courier, Cyrus, UW, Dovecot as well if the mailbox is large enough. At least in theory? (Since in theory, sorting an infinite number of messages will take longer than the hardcoded SquirrelMail-timeout).

Maybe creating some kind of indexing on these fields would help. :-\
Some cache stored in non-volatile location would help. RAM is volatile location. I suspect that hMailServer caches message envelope info in memory. When cached info is not available, it causes lots on small disk I/O operations.

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: SEARCH CHARSET US-ASCII ALL UNDELETED Subject ...

Post by martin » 2008-11-07 20:55

I agree that fixing the error message doesn't actually solve the problem, but it could help reducing the confusion a bit.

And yes, when sorting on for example the Subject header, hMailServer will do a lot of small disk reads and MIME-decoding which are both expensive operations. The reason I haven't added a index is that such a thing would make the code more complex. I try to keep it as simple as possible but in this case it's apparently not enough.

I'll look into it and try to come up with some way to cache the info. I guess I could decode the message content and put the header values in the database together in a cache-table. And when the user asks for a sorted message list I let the database engine do the sorting. I did a quick test on MSSQL where I simulated caching header fields of 50 000 messages in 100 different mailboxes (5 000 000 messages). For every message, I stored 5 header fields, each containing a 50 character "random" string (25 000 000 header fields in total). Sorting 50 000 messages in a mailbox this way takes between 0.5 and 2 seconds on my computer (depending on the MSSQL cache) .

The downside with this solution would be that the size of the database increases quite a lot (probably a few GB in the case above). Also, it's kind of tricky to keep such a big table up to date. New messages needs to be added when they arrive, after restoring a backup the cache needs to be rebuilt and so on. Or hMailServer could just index data when it's requested.

dzekas, do you know if other mail servers does something similar? I assume they do, I don't see how they could get the performance you list without it..

User avatar
dzekas
Senior user
Senior user
Posts: 2486
Joined: 2005-10-13 21:28
Location: Lithuania

Re: SEARCH CHARSET US-ASCII ALL UNDELETED Subject ...

Post by dzekas » 2008-11-07 22:11

Cyrus uses own local delivery agent and can update cached information when message is delivered. Dovecot, UW and Courier use mailboxes that can be created or accessed by other programs and messages are delivered by third party local delivery agents.

Cyrus and Dovecot have file based caches. Looks like Cyrus cache stores message mime structure and some headers. Not sure what is stored in dovecot cache, because I have deleted mailbox after tests.

Courier has smaller cache files. It stores only uid information.

UW mailbox formats differ from the one used by hMailServer. All messages are in one file and there is no cache. mbx mailbox might have some cache or index in file header.

All these servers were tested on Linux and there might be performance differences between disk or filesystem access functions used on Linux by those servers and functions used by hMailServer on Windows.

It is possible that hMailServer performance was affected by used PHP setup. Other servers are tested on PHP setup with Zend Platform extensions and higher memory limits. If I remember correctly, hMailServer was tested on standard PHP setup. hMailServer was tested on standard desktop machine and other servers were tested on laptop with slower disk and cpu. Zend Platform might improve script performance, but I have only one license and I choose to install it on Linux machine. If I test hMailServer on same PHP setup, scripts will be affected by LAN speed.

You are free to run those tests yourself on your set of mailboxes. All mailbox times are recorded by NaSMail test plugin. SELECT and EXAMINE tests are performed in development version of same plugin. You might be able to run that plugin in SquirrelMail, but it was written for NaSMail and some used functions are not available in SquirrelMail.

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: SEARCH CHARSET US-ASCII ALL UNDELETED Subject ...

Post by martin » 2008-11-08 01:26

Thanks for the feedback. Helpful, since I would have to spend a lot of time if I wanted to set up those environments and check myself.

Performance improvements won't make it in version 5 of hMailServer since this is closed to be released. Guessing it will be included in the next version though (5.1).

evb
New user
New user
Posts: 14
Joined: 2008-11-05 09:33
Location: Belgium

Re: SEARCH CHARSET US-ASCII ALL UNDELETED Subject ...

Post by evb » 2008-11-10 21:49

I did follow your conversation.
So if I'm correct, there will not be a solution for this problem before version 5.1?

I've transferred the mail rules from the squirrelmail platform to the hmailserver.
Will do the same on my other testserver which has still the 1500 mails to see if this changes anything.

But probably this will not be a solution for the 'sort' performance problem.
hmailserver 5.3.3-B1879
MySQL server Ver 5.5, for Win32

Post Reply