Could not read the message header, since the file was not available

[itsmesri]

I am seeing these errors in error log files. I could find those files in data folder also. But don't know why these error coming... What could be wrong.

"ERROR" 6180 "2016-04-04 12:44:12.415" "Severity: 3 (Medium), Code: HM4403, Source: Message::GetHeader, Description: Could not read the message header, since the file was not available. File: D:\Program Files\hMailServer\Data\domain.com\subha\F4\{F468F148-7139-4EA8-B4BD-B40CA7B2E35F}.eml"
"ERROR" 6180 "2016-04-04 12:44:12.681" "Severity: 3 (Medium), Code: HM4403, Source: Message::GetHeader, Description: Could not read the message header, since the file was not available. File: D:\Program Files\hMailServer\Data\domain.com\subha\6A\{6A5253DE-0CAC-4328-83C4-B1D35A663EA6}.eml"
"ERROR" 6180 "2016-04-04 12:44:12.931" "Severity: 3 (Medium), Code: HM4403, Source: Message::GetHeader, Description: Could not read the message header, since the file was not available. File: D:\Program Files\hMailServer\Data\domain.com\subha\12\{12FA0032-7A6A-4650-B0EC-59F7AD324761}.eml"
"ERROR" 6180 "2016-04-04 12:44:12.931" "Severity: 3 (Medium), Code: HM4403, Source: Message::GetHeader, Description: Could not read the message header, since the file was not available. File: D:\Program Files\hMailServer\Data\domain.com\subha\28\{28EDACD7-10F7-4884-9E8B-4B67DE0B91D0}.eml"
"ERROR" 6180 "2016-04-04 12:44:12.931" "Severity: 3 (Medium), Code: HM4403, Source: Message::GetHeader, Description: Could not read the message header, since the file was not available. File: D:\Program Files\hMailServer\Data\domain.com\subha\72\{721B5890-810B-4114-80CC-5222AB05A791}.eml"
"ERROR" 6180 "2016-04-04 12:44:13.009" "Severity: 3 (Medium), Code: HM4403, Source: Message::GetHeader, Description: Could not read the message header, since the file was not available. File: D:\Program Files\hMailServer\Data\domain.com\subha\58\{589AAD41-47F2-4E9D-BC45-E9A4CBFAAED9}.eml"

[SorenR]

Real-time antivirus locking files ??

[itsmesri]

I don't know. But when I check on virus... no activity on this. I don't see any errors there.

[SorenR]

I don't know. But when I check on virus... no activity on this. I don't see any errors there.

You MUST exclude hMailServer data directory from real-time antivirus. If hMailServer and real-time antivirus fight for file access, hMailServer loose every time

[itsmesri]

What if any data file comes with virus. How should I eliminate them?

[SorenR]

What if any data file comes with virus. How should I eliminate them?

hMailServer have anti-virus settings... Your client workstations should also have anti-virus.

Any virus/trojan/malware require a workstation/client to open the email, not the server.

[pik256]

I am a new user in this forum. I have found this topic using google.

I have a lot of these errors in my logs and I know when they appear. I have a script to check free space on a few accounts at OnDeliveryStart. When there is not enough space and some messages are older more than N days before last POP or IMAP logon time, then script deletes oldest messages to make a room for a new one. Messages are deleted by hMail from disk and from database. They are unavailable (do not exist) for scripting interface and for any external SQL script.

Unfortunately hMail have some kind of internal memory cache for messages I think. hMail does not remove messages from this cache. Not allways but sometimes it happens. Probably when the user is logged in via POP or IMAP at the time the script is fired, I am not sure. Anyway - each next time when user logs in via IMAP and sends "STATUS INBOX" he receives a number including deleted messages and when he tries to get sorted list of UIDs then these errors appear. There is no way to correct this using script because scripting interface does not see these messages. The list of messages retrieved by script is different then the list retrieved with IMAP.

The only way I know to correct it is to restart hMailserver. The internal message cache is rebuilt using database and everything is OK (no more errors in logs when user logs in).
Oh, there is another way. When a user tries to fetch such message then hMail creates a message file with a current time and a warning (you can set it) and logs an error "PersistentMessage::_WriteDataToMessageFile, Description: Message retrieval failed because message file xxx.eml did not exist". Unfortunately then the user often complains not understanding what it is.

[mattg]

How do you delete the messages?

[pik256]

With this javascript:

function OnDeliveryStart(oMessage) {
var fso = new ActiveXObject("Scripting.FileSystemObject");
var rcpt = "d:\\mailroot\\h-filters\\" + oMessage.Filename.substr(oMessage.Filename.lastIndexOf("\\")+1) + '.guard';
var f = fso.OpenTextFile(rcpt, 2, true);
for(i = 0; i < oMessage.Recipients.Count; i++) f.Write( oMessage.Recipients(i).Address + '\n');
f.close();
var WshShell = new ActiveXObject("WScript.Shell");
WshShell.CurrentDirectory="d:\\mailroot\\h-filters";
if (WshShell.Run("create_a_list_of_message_IDs_to_delete (querying database)" + rcpt, 0, true)) {
var oApp = new ActiveXObject("hMailServer.Application");
oApp.Authenticate(...);
f = fso.OpenTextFile(rcpt, 1, false);
var line, account = false;
while (!f.AtEndOfStream) {
line = f.ReadLine().split(" ");
if (!account || account.ID != line[0]) account = oApp.Domains.ItemByDBID(line[1]).Accounts.ItemByDBID(line[0]);
account.Messages.DeleteByDBID(line[2]);
}
f.close();
}
fso.DeleteFile(rcpt);
Result.Value = 0;
}

[mattg]

And there is your issue

You need to remove messages using the COM API.

NOT doing that (like you have done, deleting files directly with javascript) is not supported

[SorenR]

And there is your issue

You need to remove messages using the COM API.

NOT doing that (like you have done, deleting files directly with javascript) is not supported

He is... He's just using Javascript where we use VBScript..

Code seems legit too...

Will dig into it after CSI is finished on the telly..

[SorenR]

With this javascript:

function OnDeliveryStart(oMessage) {
bla bla bla
}

A few things...

What version of hMailServer ?

This line...
var rcpt = "d:\\mailroot\\h-filters\\" + oMessage.Filename.substr(oMessage.Filename.lastIndexOf("\\")+1) + '.guard';

Where does this come from? "d:\mailroot\h-filters\" <==EDIT== Ah... You are saving file {bla.bla.bla}.eml.guard here...

And... "lastIndexOf("\\")" ... only old version of hMailServer have path information in "oMessage.FileName" <==EDIT== Forget it... was thinking of DB field ...

viewtopic.php?t=21688

This partial line...
if (WshShell.Run("create_a_list_of_message_IDs_to_delete (querying database)" + rcpt, 0, true))

I assume you are running some sort of external job to query something that you cannot do with the COM API ??

Oh.. Make sure your line[0] to line[2] are either long (UID) or long long (ID)

Code properly enbedded in "Code" tags..

Code: Select all

function OnDeliveryStart(oMessage) {
   var fso = new ActiveXObject("Scripting.FileSystemObject");
   var rcpt = "d:\\mailroot\\h-filters\\" + oMessage.Filename.substr(oMessage.Filename.lastIndexOf("\\")+1) + '.guard';
   var f = fso.OpenTextFile(rcpt, 2, true);
   for(i = 0; i < oMessage.Recipients.Count; i++) f.Write( oMessage.Recipients(i).Address + '\n');
   f.close();
   var WshShell = new ActiveXObject("WScript.Shell");
   WshShell.CurrentDirectory="d:\\mailroot\\h-filters";
   if (WshShell.Run("create_a_list_of_message_IDs_to_delete (querying database)" + rcpt, 0, true)) {
      var oApp = new ActiveXObject("hMailServer.Application");
      oApp.Authenticate(...);
      f = fso.OpenTextFile(rcpt, 1, false);
      var line, account = false;
      while (!f.AtEndOfStream) {
         line = f.ReadLine().split(" ");
         if (!account || account.ID != line[0]) account = oApp.Domains.ItemByDBID(line[1]).Accounts.ItemByDBID(line[0]);
         account.Messages.DeleteByDBID(line[2]);
      }
      f.close();
   }
   fso.DeleteFile(rcpt);
   Result.Value = 0;
}

[pik256]

What version of hMailServer ?

5.6.4-B2283
OK, I see newer one, I will upgrade soon and will check if this bug is corrected. But I suspect no, because it was not reported yet.

This partial line...
if (WshShell.Run("create_a_list_of_message_IDs_to_delete (querying database)" + rcpt, 0, true))

I assume you are running some sort of external job to query something that you cannot do with the COM API ??

Yes. It was easier for me than struggling in JS. In fact I wrote a complex application for spam and virus scanning, greylisting, bandwidth limiting, autoreplies etc. I used and improved it for more than 10 years with xMailserver but now I migrated to hMail because of lack of IMAP in xMail. The code for removing expired messages on selected accounts is a part of my application. I am trying to adapt it to hMail without unneccessary additional coding. For adaptation I had to use some useful undocumented hMail features I found in sources (e.g. SMTP 4xx reject code in OnSMTPData, oClient.HELO property etc.) and I hope they will not disappear in next versions.

Oh.. Make sure your line[0] to line[2] are either long (UID) or long long (ID)

OK, I will do my best.

Code properly embedded in "Code" tags..

Yes, I learnt it. Sorry, I realized to late to correct it.

[SorenR]

Back to your problem... Something is fishy. Yes, hMailServer do cache some information but as long as you manipulate data using the COM API you should be in the clear.

Some of us have found in another case a while back that deleting more than 1 message is best done top down (count-1 to 0 step -1) when using DeleteByDBID... For some reason... The matter was not investigated further but accepted as "as is"...

[pik256]

Thanks a lot. I will reverse the list of messages to delete and I will report if it solved the problem or not.

[pik256]

OK, I have upgraded to the latest 5.6.5-B-2367 and tested it for a few days.
When I delete messages with DeleteByDBID ordering them with ascending Ids then it appears far less errors than with descending order.
But sometimes they still appears. It is independent if the user is logged in at that time because the errors appears on accounts where nobody logged in since a few weeks, too.
Simply messages are not deleted from internal cache. When user logs and try to fetch an ordered list of messages then the error

Code: Select all

HM4403, Source: Message::GetHeader, Description: Could not read the message header, since the file was not available.

appears. This is the first problem. But there is the second one yet: when the user try to fetch such message then the error

Code: Select all

Code: HM5026, Source: PersistentMessage::_WriteDataToMessageFile, Description: Message retrieval failed because message file XX did not exist

appears and hmail creates a message file on disk.
Next, if you restart a service this message file become orphaned. Hmail do not know about it anything. There is no such file name in hm_messages table, neither in internal cache.

[SorenR]

Hmm.... How do you handle errors ?? It is not clear from your code.

Reason I'm asking is if you receive two emails to this account and trigger two events simultaneously, you will end up with two ".guard" files containing equal DBID's and this in turn will result in two events deleting the same DBID... That will most certainly fail.

Going through the source sode, more precise InterfaceMessages.cpp it is obvious that an Expunge() is performed to remove the message from the cache...

Code: Select all

STDMETHODIMP InterfaceMessages::DeleteByDBID(hyper lDBID)
{
   try
   {
      if (!m_pMessages)
         return GetAccessDenied();

      shared_ptr<HM::Message> pMsg = m_pMessages->GetItemByDBID(lDBID);	
      
      if (!pMsg)
      {
         // No such message exists
         return DISP_E_BADINDEX;
      }
   
      // Mark the message for deletion.
      if (!m_pMessages->DeleteMessageByDBID(lDBID))
         return DISP_E_BADINDEX;
   
      // Expunge the mailbox. Will cause the message to be
      // deleted from disk and database.
      m_pMessages->Expunge();
   
      // If we're aren't browsing in the message cache already, 
      // we need to delete the message from the cache now. This
      // is needed if the messages are accessed using the
      // Account.Messages property. When messages are accessed
      // via Account.IMAPFolders, we access the message cache
      // directly, and when we don't need to remove message here.
      // (Since it was done like 10 lines up).
   
      __int64 iFolderID = m_pMessages->GetFolderID();
      if (iFolderID == -1)
      {
         // We're not browsing the message cache.
         __int64 iAccountID = m_pMessages->GetAccountID();
   
   
         std::vector<__int64> affectedMessages;
         affectedMessages.push_back(lDBID);
   
         // Notify clients that the message has been dropped.
         shared_ptr<HM::ChangeNotification> notification = 
            shared_ptr<HM::ChangeNotification>(new HM::ChangeNotification(m_pMessages->GetAccountID(), m_pMessages->GetFolderID(), HM::ChangeNotification::NotificationMessageDeleted, affectedMessages));
   
         HM::Application::Instance()->GetNotificationServer()->SendNotification(notification);
      }
   
      return S_OK;
   }
   catch (...)
   {
      return COMError::GenerateGenericMessage();
   }
}

I understand your desire to code in JavaScript, unfortunately not that many of us here excel at JavaScript... We'r mostly VBScript buffs

[pik256]

if you receive two emails to this account and trigger two events simultaneously, you will end up with two ".guard" files containing equal DBID's and this in turn will result in two events deleting the same DBID... That will most certainly fail.

This is the case I should have considered... I am not handling it properly. On the other hand, if it happened, then an error in log would appear. I have not seen such a thing so far. But sure, it is possible, I have to handle it.

Going through the source sode, more precise InterfaceMessages.cpp ...

This source gave me one more idea: to not delete messages directly from Account.Messages but via Account.IMAPFolders. I will recode it and report results here.

[pik256]

This source gave me one more idea: to not delete messages directly from Account.Messages but via Account.IMAPFolders. I will recode it and report results here.

OK, after a few days I can confirm: deleting messages with Domains.ItemByDBID().Accounts.ItemByDBID().IMAPFolders.ItemByDBID().Messages.DeleteByDBID() is better than Domains.ItemByDBID().Accounts.ItemByDBID().Messages.DeleteByDBID().
I noticed hundreds of errors with the former script every day and I have not noticed any error since a week, when I changed Account.Messages.DeleteByDBID to Account.IMAPFolders.Messages.DeleteByDBID

[SorenR]