Page 1 of 1

QuickRetries & QuickRetriesMinutes

Posted: 2016-09-26 11:47
by RvdH
Can someone shed some light what these INI setting actually do?

Is the QuickRetries value counted on top of the normal Retries setting in GUI?

So lets say whenever QuickRetries=5 and QuickRetriesMinutes=10 and the default Retries settings in Settings -> Protocols -> SMTP -> Delivery of e-mail = 4 (60 min) the external delivery of e-mail is tried 5 times within the first 50 minutes and after that the normal 4 attempts are tried each hour?

And what is QueueRandomnessMinutes?

Re: QuickRetries & QuickRetriesMinutes

Posted: 2016-09-26 12:25
by SorenR
From what I can read in the code it has to do with resending message if greylisted... I am guessing that if you have 10 retries in 30 minute intervals under SMTP protocol, you can alter this by setting QuickRetries=2 and QuickRetriesMinutes=5 so that delivery will be @5 minutes, @10 minutes, @40 minutes, @70 minutes, @100 minutes, @130 minutes ... etc...
RvdH wrote:And what is QueueRandomnessMinutes?
Er... It's to fix a Windows 2000 bug :mrgreen:

Code: Select all

      if (m_iQueueRandomnessMinutes > 0)
      {

         // Get our random #
         // LOG_DEBUG("Windows 2000 does not support rand_s & pukes here");
         rnd_err = (rand_s(&tmp_rnd));

         // If no error getting random # use it
         if (rnd_err == 0)
            iRandomAdjust = (unsigned int) ((double)tmp_rnd / (double) UINT_MAX * m_iQueueRandomnessMinutes) + 1;
      }

Re: QuickRetries & QuickRetriesMinutes

Posted: 2016-09-26 14:20
by mattg
These were added by Bill with one of his experimental builds >> viewtopic.php?f=10&t=21420#p132238

pretty much as SorenR says
RvdH wrote:And what is QueueRandomnessMinutes?
Instead of being exact every 10 minutes, that changes by up to the 'random' figure. The idea is to NOT be predicable with re-tries.

Re: QuickRetries & QuickRetriesMinutes

Posted: 2016-09-26 14:49
by RvdH
One question remains, are the QuickRetries counted on top of the normal retries or should i change the value (sum) of the normal reties with the value specified in QuickRetries?

Re: QuickRetries & QuickRetriesMinutes

Posted: 2016-09-26 15:42
by mattg
included in the total (I think hMailserver just counts the number of retries and if less than or equal to the retries then tries again)

to be honest it is very rare for mail to reach my retry settings

Re: QuickRetries & QuickRetriesMinutes

Posted: 2016-09-26 17:24
by RvdH
mattg wrote:included in the total (I think hMailserver just counts the number of retries and if less than or equal to the retries then tries again)

to be honest it is very rare for mail to reach my retry settings
Yup, it's included in total (tested it)

QuickRetries=2
QuickRetriesMinutes=10

Code: Select all

"APPLICATION" 2792 "2016-09-26 16:01:56.768" "SMTPDeliverer - Message 3879: Delivering message from something@something.com to something@somethingelse.com. File: C:\Program Files (x86)\hMailServer\Data\{7EDAF7C5-382E-4767-AFF2-3B263F9AF795}.eml"
"APPLICATION" 2792 "2016-09-26 16:01:57.284" "SMTPDeliverer - Message 3879: No mail servers could be found for the address something@somethingelse.com."
"APPLICATION" 2792 "2016-09-26 16:01:57.284" "SMTPDeliverer - Message 3879: Message could not be delivered. Greylisting? Scheduling it for quick retry 1 of 2 in 10 minutes."
"APPLICATION" 2792 "2016-09-26 16:01:57.284" "SMTPDeliverer - Message 3879: Message delivery thread completed."

"APPLICATION" 2792 "2016-09-26 16:12:56.817" "SMTPDeliverer - Message 3879: Delivering message from something@something.com to something@somethingelse.com. File: C:\Program Files (x86)\hMailServer\Data\{7EDAF7C5-382E-4767-AFF2-3B263F9AF795}.eml"
"APPLICATION" 2792 "2016-09-26 16:12:57.161" "SMTPDeliverer - Message 3879: No mail servers could be found for the address something@somethingelse.com."
"APPLICATION" 2792 "2016-09-26 16:12:57.161" "SMTPDeliverer - Message 3879: Message could not be delivered. Greylisting? Scheduling it for quick retry 2 of 2 in 10 minutes."
"APPLICATION" 2792 "2016-09-26 16:12:57.176" "SMTPDeliverer - Message 3879: Message delivery thread completed."

"APPLICATION" 2792 "2016-09-26 16:23:56.900" "SMTPDeliverer - Message 3879: Delivering message from something@something.com to something@somethingelse.com. File: C:\Program Files (x86)\hMailServer\Data\{7EDAF7C5-382E-4767-AFF2-3B263F9AF795}.eml"
"APPLICATION" 2792 "2016-09-26 16:24:00.795" "SMTPDeliverer - Message 3879: No mail servers could be found for the address something@somethingelse.com."
"APPLICATION" 2792 "2016-09-26 16:24:00.810" "SMTPDeliverer - Message 3879: Message could not be delivered. Scheduling it for later delivery in 60 minutes."
"APPLICATION" 2792 "2016-09-26 16:24:00.810" "SMTPDeliverer - Message 3879: Message delivery thread completed."

"APPLICATION" 2792 "2016-09-26 17:24:51.888" "SMTPDeliverer - Message 3879: Delivering message from something@something.com to something@somethingelse.com. File: C:\Program Files (x86)\hMailServer\Data\{7EDAF7C5-382E-4767-AFF2-3B263F9AF795}.eml"
"APPLICATION" 2792 "2016-09-26 17:24:52.325" "SMTPDeliverer - Message 3879: No mail servers could be found for the address something@somethingelse.com."
"APPLICATION" 2792 "2016-09-26 17:24:52.325" "SMTPDeliverer - Message 3879: Message could not be delivered. Scheduling it for later delivery in 60 minutes."
"APPLICATION" 2792 "2016-09-26 17:24:52.325" "SMTPDeliverer - Message 3879: Message delivery thread completed."
Retry count in GUI currently = 4

Re: QuickRetries & QuickRetriesMinutes

Posted: 2017-02-02 19:51
by Bill48105
Cool to see people are using the special INI settings I made but sad that martin never added them to the GUI. The INI settings were meant as a way to tweak the basic delivery queue that's built into hmailserver. If you're interested you can read more about many of my changes back in the old 5.4 days:
viewtopic.php?p=132238#p132238

You'll see detailed descriptions of each INI setting & WHY they were there. Perhaps QuickRetries wasn't the best term to name them but the though was eventually they'd be moved into the GUI & not matter. The short version is if you set hmail retries too low you waste bandwidth retrying too often & sometimes even get blocked by remotes thinking you're a spammer. If you set too high the mail waits too long when it might have been brief too busy or greylist. Those settings give you the ability to set the hamil retries HIGH like 90 minutes but force hmail to retry at a shorter interval for the 1st few tries. So you can have retries such as 5 min, 5 min, 90 min, 90 min ...

LOL at the windows 2000 note. Someone had my build crashing with randomness turned on & come find out 2k didnt support the random function used. It was decided to not worry about it because so few used 2k still so it was left alone & just say 2k no longer supported.

Re: QuickRetries & QuickRetriesMinutes

Posted: 2017-02-02 20:02
by jimimaseye
Bill48105 wrote:Cool to see people are using the special INI settings I made but sad that martin never added them to the GUI. The INI settings were meant as a way to tweak the basic delivery queue that's built into hmailserver. If you're interested you can read more about many of my changes back in the old 5.4 days:
viewtopic.php?p=132238#p132238

You'll see detailed descriptions of each INI setting & WHY they were there. ......
Not easy to tell how many of them have been retained and active in the latest builds.

Re: QuickRetries & QuickRetriesMinutes

Posted: 2017-02-02 23:11
by mattg
https://github.com/hmailserver/hmailser ... ttings.cpp

Code: Select all

   IniFileSettings::IniFileSettings() :
      is_internal_database_(false),
      dbport_(0),
      no_of_dbconnections_(0),
      add_xauth_user_header_(false),
      no_of_dbconnection_attempts_(6),
      no_of_dbconnection_attempts_Delay(5),
      max_no_of_external_fetch_threads_(15),
      greylisting_enabled_during_record_expiration_(true),
      greylisting_expiration_interval_(240),
      preferred_hash_algorithm_(3),
      dnsbl_checks_after_mail_from_(false),
      log_level_(0),
      max_log_line_len_(500),
      quick_retries_(0),
      quick_retries_Minutes(0),
      queue_randomness_minutes_(0),
      mxtries_factor_(0),
      sqldbtype_(HM::DatabaseSettings::TypeUnknown),
      sep_svc_logs_(false),
	  rewrite_envelope_from_when_forwarding_(false),
      archive_hardlinks_(false),
      pop3dmin_timeout_(false),
      pop3dmax_timeout_(false),
      pop3cmin_timeout_(false),
      pop3cmax_timeout_(false),
      smtpdmin_timeout_(false),
      smtpdmax_timeout_(false),
      smtpcmin_timeout_(false),
      smtpcmax_timeout_(false),
      samin_timeout_(false),
      samax_timeout_(false),
      clam_min_timeout_(false),
      clam_max_timeout_(false),
      samove_vs_copy_(false),
      indexer_full_minutes_(0),
      indexer_full_limit_(0),
      indexer_quick_limit_(0),
      load_header_read_size_(0),
      load_body_read_size_(0),
      blocked_iphold_seconds_(0),
      smtpdmax_size_drop_(0),
      backup_messages_dbonly_(false),
      add_xauth_user_ip_(false)

Re: QuickRetries & QuickRetriesMinutes

Posted: 2017-02-02 23:15
by mattg
Bill48105 wrote:Cool to see people are using the special INI settings I made but sad that martin never added them to the GUI.
Please make a Github fork or branches. Branches are good so that they can be added back to main easily.

We have a couple of others doing some minor development here and there.
I'd love to see the work being done on the WebAdmin panel, and the event OnHELO being added to main for a start.

Re: QuickRetries & QuickRetriesMinutes

Posted: 2017-02-02 23:22
by jimimaseye
Indeed. But can we be sure that was is listed in that list ACTUALLY has a function buried in the code? Could they be INI fil settings/key words that no longer have functionality within the code (due to Martin not including the actual function). For example, a search for "greylisting_enabled_during_record_expiration" does show up in Bills list of INI settings so it isnt one of Bills (and is standard by Martin) but we have no way of knowing what it is. And Billslist of settings list, for example, "ValidEmailPattern" - but Ive seen the code and from what I can tell this is definitely not used to determine a valid email address.

Re: QuickRetries & QuickRetriesMinutes

Posted: 2017-02-02 23:34
by mattg
Under the bit I quoted there is more which detail default values.

You can search the entire code for each of them and see where else it is being used. There may well be some that aren't used elsewhere in the code (like say SAMoveVsCopy)

Re: QuickRetries & QuickRetriesMinutes

Posted: 2017-02-03 00:14
by jimimaseye
Oooh! Ive just tried a couple of them.

ArchiveDir=C:\Pathto\Archive

This is perfect for those that dont want to actual create/state an actual email address in the MIRRORING function. It quite literally create a copy of all inbound and outbound emails in the archive folder (that folder containing a domain subfolder with all emails). eg, inbound: "{F4917B04-F8AB-4FE2-8B11-603659BF48B5}.eml" and outbound file: "Sent-{F4917B04-F8AB-4FE2-8B11-603659BF48B5}.eml". The benfit is it doesnt then rely on smtp sending those copies (freeing up resources - but only good if youre happy with the archive being local).

SepSvcLogs=1
It createes separate log files - 1 for each logging type (IMAP, SMTP etc). eg, "hmailserver_SMTP_2017-02-02.log" contains all SMTP traffic logging only. Makes reading the sequence easier for those that dont have access/ability to read the log analysers.

(And confirmed: INI setting of Bills "ValidEmailPattern", despite listed in Bills writeup, is not included in final build. This would suggest that if it isnt in the IniFileSettings.cpp then it isnt used.)

@Mattg
If a detailed posting was made of these things, is it possible to make a 'sticky' on this forum? Or is that Martin's decision only?

Re: QuickRetries & QuickRetriesMinutes

Posted: 2017-02-03 00:48
by mattg
I don't think I can make a sticky, but I recall that someone else (other than Martin) does, we can Ask.

I was also going to submit a new branch on github removing all of the unused old stuff.

Re: QuickRetries & QuickRetriesMinutes

Posted: 2017-02-03 00:49
by jimimaseye
Ok, Ive checked against Bills initial list of INI': the following are no longer included/valid:

ValidEmailPattern
AutoReplyReturnPath
SPFDefaultPolicy
SPFPolicyOverride
DnsServers
SSLOptionList
SSLCipherList

Ill redo Bills list in to a HOW TO style documentation in to DEV/ALPHA section without these exceptions.

Re: QuickRetries & QuickRetriesMinutes

Posted: 2017-02-03 01:51
by jimimaseye
mattg wrote:I don't think I can make a sticky, but I recall that someone else (other than Martin) does, we can Ask.

I was also going to submit a new branch on github removing all of the unused old stuff.
viewtopic.php?f=10&t=30900 - What do you think?

Re: QuickRetries & QuickRetriesMinutes

Posted: 2017-02-03 05:15
by mattg
I'm pretty certain that this one has been dropped >> SAMoveVsCopy

I'll check the others, but Yes I like what you have done.

Re: QuickRetries & QuickRetriesMinutes

Posted: 2017-02-03 07:24
by Bill48105
mattg wrote:
Bill48105 wrote:Cool to see people are using the special INI settings I made but sad that martin never added them to the GUI.
Please make a Github fork or branches. Branches are good so that they can be added back to main easily.

We have a couple of others doing some minor development here and there.
I'd love to see the work being done on the WebAdmin panel, and the event OnHELO being added to main for a start.
HUH?!? I did a full commit of ALL my changes forever ago before I went on hiatus. 3 years ago in fact:
https://github.com/Bill48105/hmailserver
martin just didn't want to accept the changes to be merged so instead he said he'd pick the stuff he wanted to include & left it at that.

Re: QuickRetries & QuickRetriesMinutes

Posted: 2017-02-03 07:43
by Bill48105
jimimaseye wrote:
Bill48105 wrote:Cool to see people are using the special INI settings I made but sad that martin never added them to the GUI. The INI settings were meant as a way to tweak the basic delivery queue that's built into hmailserver. If you're interested you can read more about many of my changes back in the old 5.4 days:
viewtopic.php?p=132238#p132238

You'll see detailed descriptions of each INI setting & WHY they were there. ......
Not easy to tell how many of them have been retained and active in the latest builds.
If I tried hard enough I could figure it out because anything I committed via SVN would likely still be in the code because with SVN he didn't have to approve & it'd require him to roll back my changes or change individual code lines. (I always detailed my changes to SVN but that's gone but I always put the changes in my experimental post) Once he made the move to github he started from that last SVN including my changes to that point that I had got into the SVN & he started making his own changes from that point on github without me knowing so I had changes on my end that hadnt been committed yet. So I had to fork his official on github then it took me forever to manually merge his with mine & I committed that. But I don't think he ever accepted or used any of my changes between SVN & Github so it ended up being a giant waste of my time & I moved onto other projects including my personal life which had suffered due to amount of time I had dedicated to hmailserver. Good to see so many of you still around volunteering your time.

Re: QuickRetries & QuickRetriesMinutes

Posted: 2017-02-03 12:42
by jimimaseye
mattg wrote:I'm pretty certain that this one has been dropped >> SAMoveVsCopy

I'll check the others, but Yes I like what you have done.
From what I see, this function is still there:

https://github.com/hmailserver/hmailser ... Client.cpp (line 182):

Code: Select all

      if (IniFileSettings::Instance()->GetSAMoveVsCopy())
         {
            // Move temp file overwriting message file
            FileUtilities::Move(sTempFile, message_file_, true);
            LOG_DEBUG("SA - Move used");
         }
         else
         {
            // Copy temp file to message file
            FileUtilities::Copy(sTempFile, message_file_, false);
            LOG_DEBUG("SA - Copy+Delete used");
Cant say I know what the point/benefit of it is though (default 'copy+delete' would avoid file locking issues).

Re: QuickRetries & QuickRetriesMinutes

Posted: 2017-02-03 12:52
by jimimaseye
Bill48105 wrote: But I don't think he ever accepted or used any of my changes between SVN & Github so it ended up being a giant waste of my time & I moved onto other projects including my personal life.....
I think you will be surprised Bill. As you can see from the above I have reviewed your INI's and sanitised it by looking at what is stil there (here) and all except (mainly) the DNS, SSL and SPF related ones are still functional. (From what I read on page 4 of your builds thread he just chose to not accept them as 'official' without his testing environment, documentation and GUI inclusion - but they are still there unofficially. That doesnt mean they are of no use or unrequired and if he didnt want them he would have removed them as he did with the DNS, SPF and SSL stuff.)

Re: QuickRetries & QuickRetriesMinutes

Posted: 2017-02-03 16:51
by Bill48105
jimimaseye wrote:
mattg wrote:I'm pretty certain that this one has been dropped >> SAMoveVsCopy

I'll check the others, but Yes I like what you have done.
From what I see, this function is still there:

https://github.com/hmailserver/hmailser ... Client.cpp (line 182):

Code: Select all

      if (IniFileSettings::Instance()->GetSAMoveVsCopy())
         {
            // Move temp file overwriting message file
            FileUtilities::Move(sTempFile, message_file_, true);
            LOG_DEBUG("SA - Move used");
         }
         else
         {
            // Copy temp file to message file
            FileUtilities::Copy(sTempFile, message_file_, false);
            LOG_DEBUG("SA - Copy+Delete used");
Cant say I know what the point/benefit of it is though (default 'copy+delete' would avoid file locking issues).
Cool it's still there it makes a big difference in load. hmail creates tmp files EVERY time a message is created. Be it incoming email or moving between imap folders etc. As you can imagine that adds up to a lot of read/writes to begin with but then hmail would COPY that temp file to it's final destination. So pretty much everything was written twice, read once for something that only really requires written once.. And with enough emails especially larger ones it makes a HUGE difference in not only CPU usage but HDD activity including saturation depending on the storage system. Yes there are risks mostly with permissions but as long as the temp folder doesn't have special permissions the final destination file has OK perms too. It's one of those behind the scene things I found in hmail when trying to make it more efficient & IMO should have been made the default long ago but wasn't my call. But yeah do some testing with it on & off you'll see what i mean. no comparison of 2W+R vs 1W+move.. Night and day.

Re: QuickRetries & QuickRetriesMinutes

Posted: 2017-02-03 17:01
by Bill48105
jimimaseye wrote:
Bill48105 wrote: But I don't think he ever accepted or used any of my changes between SVN & Github so it ended up being a giant waste of my time & I moved onto other projects including my personal life.....
I think you will be surprised Bill. As you can see from the above I have reviewed your INI's and sanitised it by looking at what is stil there (here) and all except (mainly) the DNS, SSL and SPF related ones are still functional. (From what I read on page 4 of your builds thread he just chose to not accept them as 'official' without his testing environment, documentation and GUI inclusion - but they are still there unofficially. That doesnt mean they are of no use or unrequired and if he didnt want them he would have removed them as he did with the DNS, SPF and SSL stuff.)
What i meant were the changes between move from SVN to github. I'm not sure any of those changes ever got included.. I spent &^% ton of time doing STARTTLS & he did it over himself without my code. (The SSL stuff was mainly for that) The other missing things were my usual attempts to improve hmail in little ways while still remaining backward compatible. I'd have to look back WHY I made the changes but I always had a reason.. Usually trying to solve a problem in the forum or optimize something. For example I recall the SPF setting let you choose to reject soft fails which isn't really an acceptable thing to do but so many lazy people leave their spf on ~all transitioning mode forever that their spf is essentially worthless & gets abused. I had also made it so you could specify your own dns server since hmail uses system DNS by default which can be problematic for some people. I actually think there were others I done on my local builds that I never shared but at this point couldn't say for sure. It's been good 3 years since I looked at the source.

Re: QuickRetries & QuickRetriesMinutes

Posted: 2017-02-03 17:05
by jimimaseye
Bill48105 wrote: Cool it's still there it makes a big difference in load. hmail creates tmp files EVERY time a message is created. Be it incoming email or moving between imap folders etc. As you can imagine that adds up to a lot of read/writes to begin with but then hmail would COPY that temp file to it's final destination. So pretty much everything was written twice, read once for something that only really requires written once.. And with enough emails especially larger ones it makes a HUGE difference in not only CPU usage but HDD activity including saturation depending on the storage system. Yes there are risks mostly with permissions but as long as the temp folder doesn't have special permissions the final destination file has OK perms too. It's one of those behind the scene things I found in hmail when trying to make it more efficient & IMO should have been made the default long ago but wasn't my call. But yeah do some testing with it on & off you'll see what i mean. no comparison of 2W+R vs 1W+move.. Night and day.
Would there be a valid argument that doing copy+delete avoids filelocking (access) problems (which would prevent a successful MOVE)? And in the worse case scenario, the temp directory just has redundant temporary files in it (in such file locking situations) if Copy+Delete is used (whereas a MOVE would have created a more fundamental error)? Or isnt there a situation where that file could still be locked when it comes to that section of code?

Re: QuickRetries & QuickRetriesMinutes

Posted: 2017-02-06 06:59
by Bill48105
jimimaseye wrote:
Bill48105 wrote: Cool it's still there it makes a big difference in load. hmail creates tmp files EVERY time a message is created. Be it incoming email or moving between imap folders etc. As you can imagine that adds up to a lot of read/writes to begin with but then hmail would COPY that temp file to it's final destination. So pretty much everything was written twice, read once for something that only really requires written once.. And with enough emails especially larger ones it makes a HUGE difference in not only CPU usage but HDD activity including saturation depending on the storage system. Yes there are risks mostly with permissions but as long as the temp folder doesn't have special permissions the final destination file has OK perms too. It's one of those behind the scene things I found in hmail when trying to make it more efficient & IMO should have been made the default long ago but wasn't my call. But yeah do some testing with it on & off you'll see what i mean. no comparison of 2W+R vs 1W+move.. Night and day.
Would there be a valid argument that doing copy+delete avoids filelocking (access) problems (which would prevent a successful MOVE)? And in the worse case scenario, the temp directory just has redundant temporary files in it (in such file locking situations) if Copy+Delete is used (whereas a MOVE would have created a more fundamental error)? Or isnt there a situation where that file could still be locked when it comes to that section of code?
Maybe in theory but there should never be a case where 2 threads end up with the same filename since they are GUID generated. In the years of running multiple hmail servers handling millions of emails using my custom build I can't say I ever ran into a scenario where I believe there was contention for a file due to move vs copy/delete. It could also be argued there's less chance of issue since it's faster to write once then move than write twice & delete since the file(s) is/are in-use less time.