Issue 264: Searching for strings in UTF8 format did not work

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
Tim Josten
New user
New user
Posts: 7
Joined: 2012-03-28 06:28

Issue 264: Searching for strings in UTF8 format did not work

Post by Tim Josten » 2012-10-16 11:55

Hello Martin!

Looks like UTF-8 searching still doesn't work in 5.3.3 or newer versions. I am using Roundcube 0.8 beta. I am russian. If I search for email subjects written in english it works fine. Once I enter any russian word, it returns empty list. I think this is not a Roundcube problem. Here's the log:

Code: Select all

"IMAPD"	4376	85538	"2012-10-16 16:29:02.365"	"127.0.0.1"	"SENT: * OK IMAPrev1"
"IMAPD"	4376	85538	"2012-10-16 16:29:02.367"	"127.0.0.1"	"RECEIVED: A0001 CAPABILITY"
"IMAPD"	4376	85538	"2012-10-16 16:29:02.367"	"127.0.0.1"	"SENT: * CAPABILITY IMAP4 IMAP4rev1 CHILDREN IDLE QUOTA SORT ACL NAMESPACE RIGHTS=texk[nl]A0001 OK CAPABILITY completed"
"IMAPD"	4412	85538	"2012-10-16 16:29:02.368"	"127.0.0.1"	"RECEIVED: A0002 LOGIN tim@***.ru ***"
"IMAPD"	4412	85538	"2012-10-16 16:29:02.371"	"127.0.0.1"	"SENT: A0002 OK LOGIN completed"
"IMAPD"	4412	85538	"2012-10-16 16:29:02.376"	"127.0.0.1"	"RECEIVED: A0003 SELECT INBOX"
"IMAPD"	4412	85538	"2012-10-16 16:29:02.377"	"127.0.0.1"	"SENT: * 55 EXISTS[nl]* 0 RECENT[nl]* FLAGS (\Deleted \Seen \Draft \Answered \Flagged)[nl]* OK [UIDVALIDITY 1332767657][nl]* OK [UIDNEXT 325][nl]* OK [PERMANENTFLAGS (\Deleted \Seen \Draft \Answered \Flagged)][nl]A0003 OK [READ-WRITE] SELECT completed"
"IMAPD"	4412	85538	"2012-10-16 16:29:02.378"	"127.0.0.1"	"RECEIVED: A0004 UID SORT (DATE) UTF-8 ALL TEXT {12}"
"IMAPD"	4412	85538	"2012-10-16 16:29:02.379"	"127.0.0.1"	"SENT: + Ready for additional command text."
"IMAPD"	4376	85538	"2012-10-16 16:29:02.380"	"127.0.0.1"	"RECEIVED: запрос"
"IMAPD"	4376	85538	"2012-10-16 16:29:03.687"	"127.0.0.1"	"SENT: * SORT"
"IMAPD"	4376	85538	"2012-10-16 16:29:03.689"	"127.0.0.1"	"SENT: A0004 OK UID completed"
"IMAPD"	4416	85538	"2012-10-16 16:29:03.691"	"127.0.0.1"	"RECEIVED: A0005 LOGOUT"
"IMAPD"	4416	85538	"2012-10-16 16:29:03.692"	"127.0.0.1"	"SENT: * BYE Have a nice day[nl]A0005 OK Logout completed"
hMailServer correctly outputs UTF-8 string in log (RECEIVED: запрос), but no search results. I found this code in hMailServer source:

Code: Select all

   bool ContainsNoCase(PCMYSTR szSub) const
   {
      MYTYPE sFindIn = this->c_str();
      MYTYPE sLookFor = szSub;

      // Convert to lower case so that we'll do the search in 
      // an in-case sensitive manner.
      sFindIn.ToLower();
      sLookFor.ToLower();

      if (sFindIn.Find(sLookFor) >= 0)
         return true;
      else
         return false;
   }
Maybe it incorrectly compares strings in Unicode? Or lowercase function corrupts data?

Tim Josten
New user
New user
Posts: 7
Joined: 2012-03-28 06:28

Re: Issue 264: Searching for strings in UTF8 format did not

Post by Tim Josten » 2012-10-17 18:21

Well, I figured it by myself.
Roundcube sends the incorrect command. :) Totally no offense to the author of hMailServer.
It miss the CHARSET word:
A0004 UID SORT (DATE) UTF-8 ALL TEXT ...

Correct command is:
A0004 UID SORT (DATE) CHARSET UTF-8 ALL TEXT ...

Now it works fine!

Tim Josten
New user
New user
Posts: 7
Joined: 2012-03-28 06:28

Re: Issue 264: Searching for strings in UTF8 format did not

Post by Tim Josten » 2012-10-18 06:51

Here comes another problem, now on the server side. There are lots of messages in my inbox folder. Some of them are in UTF-8, some are in Windows-1251, some are in KOI8-R codepage. I can only search for UTF-8 messages.
For example:
Subject: =?KOI8-R?B?UmU6IO/v7yDu4ec=?=
From: =?KOI8-R?B?7sHUwczY0SDn0sHCz9fTy8HR?=

Notice the KOI8-R prefix. hMailServer doesn't handle this currently.

User avatar
dzekas
Senior user
Senior user
Posts: 2486
Joined: 2005-10-13 21:28
Location: Lithuania

Re: Issue 264: Searching for strings in UTF8 format did not

Post by dzekas » 2012-10-18 07:31

Tim Josten wrote:Here comes another problem, now on the server side. There are lots of messages in my inbox folder. Some of them are in UTF-8, some are in Windows-1251, some are in KOI8-R codepage. I can only search for UTF-8 messages.
For example:
Subject: =?KOI8-R?B?UmU6IO/v7yDu4ec=?=
From: =?KOI8-R?B?7sHUwczY0SDn0sHCz9fTy8HR?=

Notice the KOI8-R prefix. hMailServer doesn't handle this currently.
If you think that you have found deficiency in hmailserver implementation and have a way to reproduce it in latest published dev version, could you register ticket in bug tracker?

Post Reply