Could not read the message header, since the file was not available

Use this forum if you want to discuss a problem or ask a question related to a hMailServer beta release.
itsmesri
Normal user
Normal user
Posts: 59
Joined: 2014-11-07 08:30

Could not read the message header, since the file was not available

Postby itsmesri » 2016-04-05 06:29

I am seeing these errors in error log files. I could find those files in data folder also. But don't know why these error coming... What could be wrong.


"ERROR" 6180 "2016-04-04 12:44:12.415" "Severity: 3 (Medium), Code: HM4403, Source: Message::GetHeader, Description: Could not read the message header, since the file was not available. File: D:\Program Files\hMailServer\Data\domain.com\subha\F4\{F468F148-7139-4EA8-B4BD-B40CA7B2E35F}.eml"
"ERROR" 6180 "2016-04-04 12:44:12.681" "Severity: 3 (Medium), Code: HM4403, Source: Message::GetHeader, Description: Could not read the message header, since the file was not available. File: D:\Program Files\hMailServer\Data\domain.com\subha\6A\{6A5253DE-0CAC-4328-83C4-B1D35A663EA6}.eml"
"ERROR" 6180 "2016-04-04 12:44:12.931" "Severity: 3 (Medium), Code: HM4403, Source: Message::GetHeader, Description: Could not read the message header, since the file was not available. File: D:\Program Files\hMailServer\Data\domain.com\subha\12\{12FA0032-7A6A-4650-B0EC-59F7AD324761}.eml"
"ERROR" 6180 "2016-04-04 12:44:12.931" "Severity: 3 (Medium), Code: HM4403, Source: Message::GetHeader, Description: Could not read the message header, since the file was not available. File: D:\Program Files\hMailServer\Data\domain.com\subha\28\{28EDACD7-10F7-4884-9E8B-4B67DE0B91D0}.eml"
"ERROR" 6180 "2016-04-04 12:44:12.931" "Severity: 3 (Medium), Code: HM4403, Source: Message::GetHeader, Description: Could not read the message header, since the file was not available. File: D:\Program Files\hMailServer\Data\domain.com\subha\72\{721B5890-810B-4114-80CC-5222AB05A791}.eml"
"ERROR" 6180 "2016-04-04 12:44:13.009" "Severity: 3 (Medium), Code: HM4403, Source: Message::GetHeader, Description: Could not read the message header, since the file was not available. File: D:\Program Files\hMailServer\Data\domain.com\subha\58\{589AAD41-47F2-4E9D-BC45-E9A4CBFAAED9}.eml"

User avatar
SorenR
Senior user
Senior user
Posts: 2327
Joined: 2006-08-21 15:38
Location: Denmark

Re: Could not read the message header, since the file was not available

Postby SorenR » 2016-04-05 13:54

Real-time antivirus locking files ??
SørenR.

The quantum rule of insecurity which states that the act of observing how vulnerable a host or service is changes the insecurity level of the service.

itsmesri
Normal user
Normal user
Posts: 59
Joined: 2014-11-07 08:30

Re: Could not read the message header, since the file was not available

Postby itsmesri » 2016-04-05 20:32

I don't know. But when I check on virus... no activity on this. I don't see any errors there.

User avatar
SorenR
Senior user
Senior user
Posts: 2327
Joined: 2006-08-21 15:38
Location: Denmark

Re: Could not read the message header, since the file was not available

Postby SorenR » 2016-04-05 20:45

itsmesri wrote:I don't know. But when I check on virus... no activity on this. I don't see any errors there.

You MUST exclude hMailServer data directory from real-time antivirus. If hMailServer and real-time antivirus fight for file access, hMailServer loose every time ;-)
SørenR.

The quantum rule of insecurity which states that the act of observing how vulnerable a host or service is changes the insecurity level of the service.

itsmesri
Normal user
Normal user
Posts: 59
Joined: 2014-11-07 08:30

Re: Could not read the message header, since the file was not available

Postby itsmesri » 2016-04-05 22:00

What if any data file comes with virus. How should I eliminate them?

User avatar
SorenR
Senior user
Senior user
Posts: 2327
Joined: 2006-08-21 15:38
Location: Denmark

Re: Could not read the message header, since the file was not available

Postby SorenR » 2016-04-05 22:41

itsmesri wrote:What if any data file comes with virus. How should I eliminate them?

hMailServer have anti-virus settings... Your client workstations should also have anti-virus.

Any virus/trojan/malware require a workstation/client to open the email, not the server.
SørenR.

The quantum rule of insecurity which states that the act of observing how vulnerable a host or service is changes the insecurity level of the service.

pik256
New user
New user
Posts: 16
Joined: 2016-05-18 11:25
Location: Poland

Re: Could not read the message header, since the file was not available

Postby pik256 » 2016-09-19 10:18

I am a new user in this forum. I have found this topic using google.

I have a lot of these errors in my logs and I know when they appear. I have a script to check free space on a few accounts at OnDeliveryStart. When there is not enough space and some messages are older more than N days before last POP or IMAP logon time, then script deletes oldest messages to make a room for a new one. Messages are deleted by hMail from disk and from database. They are unavailable (do not exist) for scripting interface and for any external SQL script.

Unfortunately hMail have some kind of internal memory cache for messages I think. hMail does not remove messages from this cache. Not allways but sometimes it happens. Probably when the user is logged in via POP or IMAP at the time the script is fired, I am not sure. Anyway - each next time when user logs in via IMAP and sends "STATUS INBOX" he receives a number including deleted messages and when he tries to get sorted list of UIDs then these errors appear. There is no way to correct this using script because scripting interface does not see these messages. The list of messages retrieved by script is different then the list retrieved with IMAP.

The only way I know to correct it is to restart hMailserver. The internal message cache is rebuilt using database and everything is OK (no more errors in logs when user logs in).
Oh, there is another way. When a user tries to fetch such message then hMail creates a message file with a current time and a warning (you can set it) and logs an error "PersistentMessage::_WriteDataToMessageFile, Description: Message retrieval failed because message file xxx.eml did not exist". Unfortunately then the user often complains not understanding what it is.

User avatar
mattg
Moderator
Moderator
Posts: 17596
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Could not read the message header, since the file was not available

Postby mattg » 2016-09-19 15:10

How do you delete the messages?
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

pik256
New user
New user
Posts: 16
Joined: 2016-05-18 11:25
Location: Poland

Re: Could not read the message header, since the file was not available

Postby pik256 » 2016-09-19 16:58

With this javascript:

function OnDeliveryStart(oMessage) {
var fso = new ActiveXObject("Scripting.FileSystemObject");
var rcpt = "d:\\mailroot\\h-filters\\" + oMessage.Filename.substr(oMessage.Filename.lastIndexOf("\\")+1) + '.guard';
var f = fso.OpenTextFile(rcpt, 2, true);
for(i = 0; i < oMessage.Recipients.Count; i++) f.Write( oMessage.Recipients(i).Address + '\n');
f.close();
var WshShell = new ActiveXObject("WScript.Shell");
WshShell.CurrentDirectory="d:\\mailroot\\h-filters";
if (WshShell.Run("create_a_list_of_message_IDs_to_delete (querying database)" + rcpt, 0, true)) {
var oApp = new ActiveXObject("hMailServer.Application");
oApp.Authenticate(...);
f = fso.OpenTextFile(rcpt, 1, false);
var line, account = false;
while (!f.AtEndOfStream) {
line = f.ReadLine().split(" ");
if (!account || account.ID != line[0]) account = oApp.Domains.ItemByDBID(line[1]).Accounts.ItemByDBID(line[0]);
account.Messages.DeleteByDBID(line[2]);
}
f.close();
}
fso.DeleteFile(rcpt);
Result.Value = 0;
}

User avatar
mattg
Moderator
Moderator
Posts: 17596
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Could not read the message header, since the file was not available

Postby mattg » 2016-09-19 17:12

And there is your issue

You need to remove messages using the COM API.

NOT doing that (like you have done, deleting files directly with javascript) is not supported
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
SorenR
Senior user
Senior user
Posts: 2327
Joined: 2006-08-21 15:38
Location: Denmark

Re: Could not read the message header, since the file was not available

Postby SorenR » 2016-09-19 21:48

mattg wrote:And there is your issue

You need to remove messages using the COM API.

NOT doing that (like you have done, deleting files directly with javascript) is not supported

He is... He's just using Javascript where we use VBScript..

Code seems legit too... :mrgreen:

Will dig into it after CSI is finished on the telly..
SørenR.

The quantum rule of insecurity which states that the act of observing how vulnerable a host or service is changes the insecurity level of the service.

User avatar
SorenR
Senior user
Senior user
Posts: 2327
Joined: 2006-08-21 15:38
Location: Denmark

Re: Could not read the message header, since the file was not available

Postby SorenR » 2016-09-19 22:22

pik256 wrote:With this javascript:

function OnDeliveryStart(oMessage) {
bla bla bla
}

A few things...

What version of hMailServer ?

This line...
var rcpt = "d:\\mailroot\\h-filters\\" + oMessage.Filename.substr(oMessage.Filename.lastIndexOf("\\")+1) + '.guard';

Where does this come from? "d:\mailroot\h-filters\" <==EDIT== Ah... You are saving file {bla.bla.bla}.eml.guard here...

And... "lastIndexOf("\\")" ... only old version of hMailServer have path information in "oMessage.FileName" <==EDIT== Forget it... :roll: was thinking of DB field ...

https://www.hmailserver.com/forum/viewtopic.php?t=21688

This partial line...
if (WshShell.Run("create_a_list_of_message_IDs_to_delete (querying database)" + rcpt, 0, true))

I assume you are running some sort of external job to query something that you cannot do with the COM API ??

Oh.. Make sure your line[0] to line[2] are either long (UID) or long long (ID)

Code properly enbedded in "Code" tags.. :mrgreen:

Code: Select all

function OnDeliveryStart(oMessage) {
   var fso = new ActiveXObject("Scripting.FileSystemObject");
   var rcpt = "d:\\mailroot\\h-filters\\" + oMessage.Filename.substr(oMessage.Filename.lastIndexOf("\\")+1) + '.guard';
   var f = fso.OpenTextFile(rcpt, 2, true);
   for(i = 0; i < oMessage.Recipients.Count; i++) f.Write( oMessage.Recipients(i).Address + '\n');
   f.close();
   var WshShell = new ActiveXObject("WScript.Shell");
   WshShell.CurrentDirectory="d:\\mailroot\\h-filters";
   if (WshShell.Run("create_a_list_of_message_IDs_to_delete (querying database)" + rcpt, 0, true)) {
      var oApp = new ActiveXObject("hMailServer.Application");
      oApp.Authenticate(...);
      f = fso.OpenTextFile(rcpt, 1, false);
      var line, account = false;
      while (!f.AtEndOfStream) {
         line = f.ReadLine().split(" ");
         if (!account || account.ID != line[0]) account = oApp.Domains.ItemByDBID(line[1]).Accounts.ItemByDBID(line[0]);
         account.Messages.DeleteByDBID(line[2]);
      }
      f.close();
   }
   fso.DeleteFile(rcpt);
   Result.Value = 0;
}
SørenR.

The quantum rule of insecurity which states that the act of observing how vulnerable a host or service is changes the insecurity level of the service.

pik256
New user
New user
Posts: 16
Joined: 2016-05-18 11:25
Location: Poland

Re: Could not read the message header, since the file was not available

Postby pik256 » 2016-09-20 15:32

SorenR wrote:What version of hMailServer ?

5.6.4-B2283
OK, I see newer one, I will upgrade soon and will check if this bug is corrected. But I suspect no, because it was not reported yet.

This partial line...
if (WshShell.Run("create_a_list_of_message_IDs_to_delete (querying database)" + rcpt, 0, true))

I assume you are running some sort of external job to query something that you cannot do with the COM API ??


Yes. It was easier for me than struggling in JS. In fact I wrote a complex application for spam and virus scanning, greylisting, bandwidth limiting, autoreplies etc. I used and improved it for more than 10 years with xMailserver but now I migrated to hMail because of lack of IMAP in xMail. The code for removing expired messages on selected accounts is a part of my application. I am trying to adapt it to hMail without unneccessary additional coding. For adaptation I had to use some useful undocumented hMail features I found in sources (e.g. SMTP 4xx reject code in OnSMTPData, oClient.HELO property etc.) and I hope they will not disappear in next versions.

Oh.. Make sure your line[0] to line[2] are either long (UID) or long long (ID)

OK, I will do my best. :wink:

Code properly embedded in "Code" tags.. :mrgreen:

Yes, I learnt it. Sorry, I realized to late to correct it.

User avatar
SorenR
Senior user
Senior user
Posts: 2327
Joined: 2006-08-21 15:38
Location: Denmark

Re: Could not read the message header, since the file was not available

Postby SorenR » 2016-09-20 20:46

Back to your problem... Something is fishy. Yes, hMailServer do cache some information but as long as you manipulate data using the COM API you should be in the clear.

Some of us have found in another case a while back that deleting more than 1 message is best done top down (count-1 to 0 step -1) when using DeleteByDBID... For some reason... The matter was not investigated further but accepted as "as is"...
SørenR.

The quantum rule of insecurity which states that the act of observing how vulnerable a host or service is changes the insecurity level of the service.

pik256
New user
New user
Posts: 16
Joined: 2016-05-18 11:25
Location: Poland

Re: Could not read the message header, since the file was not available

Postby pik256 » 2016-09-20 23:05

Thanks a lot. I will reverse the list of messages to delete and I will report if it solved the problem or not.

pik256
New user
New user
Posts: 16
Joined: 2016-05-18 11:25
Location: Poland

Re: Could not read the message header, since the file was not available

Postby pik256 » 2016-09-28 10:33

OK, I have upgraded to the latest 5.6.5-B-2367 and tested it for a few days.
When I delete messages with DeleteByDBID ordering them with ascending Ids then it appears far less errors than with descending order.
But sometimes they still appears. It is independent if the user is logged in at that time because the errors appears on accounts where nobody logged in since a few weeks, too.
Simply messages are not deleted from internal cache. When user logs and try to fetch an ordered list of messages then the error

Code: Select all

HM4403, Source: Message::GetHeader, Description: Could not read the message header, since the file was not available.

appears. This is the first problem. But there is the second one yet: when the user try to fetch such message then the error

Code: Select all

Code: HM5026, Source: PersistentMessage::_WriteDataToMessageFile, Description: Message retrieval failed because message file XX did not exist

appears and hmail creates a message file on disk.
Next, if you restart a service this message file become orphaned. Hmail do not know about it anything. There is no such file name in hm_messages table, neither in internal cache.

User avatar
SorenR
Senior user
Senior user
Posts: 2327
Joined: 2006-08-21 15:38
Location: Denmark

Re: Could not read the message header, since the file was not available

Postby SorenR » 2016-09-28 18:25

Hmm.... How do you handle errors ?? It is not clear from your code.

Reason I'm asking is if you receive two emails to this account and trigger two events simultaneously, you will end up with two ".guard" files containing equal DBID's and this in turn will result in two events deleting the same DBID... That will most certainly fail.

Going through the source sode, more precise InterfaceMessages.cpp it is obvious that an Expunge() is performed to remove the message from the cache...

Code: Select all

STDMETHODIMP InterfaceMessages::DeleteByDBID(hyper lDBID)
{
   try
   {
      if (!m_pMessages)
         return GetAccessDenied();

      shared_ptr<HM::Message> pMsg = m_pMessages->GetItemByDBID(lDBID);   
     
      if (!pMsg)
      {
         // No such message exists
         return DISP_E_BADINDEX;
      }
   
      // Mark the message for deletion.
      if (!m_pMessages->DeleteMessageByDBID(lDBID))
         return DISP_E_BADINDEX;
   
      // Expunge the mailbox. Will cause the message to be
      // deleted from disk and database.
      m_pMessages->Expunge();
   
      // If we're aren't browsing in the message cache already,
      // we need to delete the message from the cache now. This
      // is needed if the messages are accessed using the
      // Account.Messages property. When messages are accessed
      // via Account.IMAPFolders, we access the message cache
      // directly, and when we don't need to remove message here.
      // (Since it was done like 10 lines up).
   
      __int64 iFolderID = m_pMessages->GetFolderID();
      if (iFolderID == -1)
      {
         // We're not browsing the message cache.
         __int64 iAccountID = m_pMessages->GetAccountID();
   
   
         std::vector<__int64> affectedMessages;
         affectedMessages.push_back(lDBID);
   
         // Notify clients that the message has been dropped.
         shared_ptr<HM::ChangeNotification> notification =
            shared_ptr<HM::ChangeNotification>(new HM::ChangeNotification(m_pMessages->GetAccountID(), m_pMessages->GetFolderID(), HM::ChangeNotification::NotificationMessageDeleted, affectedMessages));
   
         HM::Application::Instance()->GetNotificationServer()->SendNotification(notification);
      }
   
      return S_OK;
   }
   catch (...)
   {
      return COMError::GenerateGenericMessage();
   }
}


I understand your desire to code in JavaScript, unfortunately not that many of us here excel at JavaScript... We'r mostly VBScript buffs :mrgreen:
SørenR.

The quantum rule of insecurity which states that the act of observing how vulnerable a host or service is changes the insecurity level of the service.

pik256
New user
New user
Posts: 16
Joined: 2016-05-18 11:25
Location: Poland

Re: Could not read the message header, since the file was not available

Postby pik256 » 2016-09-28 21:54

if you receive two emails to this account and trigger two events simultaneously, you will end up with two ".guard" files containing equal DBID's and this in turn will result in two events deleting the same DBID... That will most certainly fail.

This is the case I should have considered... I am not handling it properly. On the other hand, if it happened, then an error in log would appear. I have not seen such a thing so far. But sure, it is possible, I have to handle it.

Going through the source sode, more precise InterfaceMessages.cpp ...

This source gave me one more idea: to not delete messages directly from Account.Messages but via Account.IMAPFolders. I will recode it and report results here.

pik256
New user
New user
Posts: 16
Joined: 2016-05-18 11:25
Location: Poland

Re: Could not read the message header, since the file was not available

Postby pik256 » 2016-10-05 12:40

pik256 wrote:This source gave me one more idea: to not delete messages directly from Account.Messages but via Account.IMAPFolders. I will recode it and report results here.


OK, after a few days I can confirm: deleting messages with Domains.ItemByDBID().Accounts.ItemByDBID().IMAPFolders.ItemByDBID().Messages.DeleteByDBID() is better than Domains.ItemByDBID().Accounts.ItemByDBID().Messages.DeleteByDBID().
I noticed hundreds of errors with the former script every day and I have not noticed any error since a week, when I changed Account.Messages.DeleteByDBID to Account.IMAPFolders.Messages.DeleteByDBID

User avatar
SorenR
Senior user
Senior user
Posts: 2327
Joined: 2006-08-21 15:38
Location: Denmark

Re: Could not read the message header, since the file was not available

Postby SorenR » 2016-10-05 13:47

Image
SørenR.

The quantum rule of insecurity which states that the act of observing how vulnerable a host or service is changes the insecurity level of the service.


Return to “Development & alpha discussions”



Who is online

Users browsing this forum: No registered users and 2 guests