QuickRetries & QuickRetriesMinutes

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

QuickRetries & QuickRetriesMinutes

Post by RvdH » 2016-09-26 11:47

Can someone shed some light what these INI setting actually do?

Is the QuickRetries value counted on top of the normal Retries setting in GUI?

So lets say whenever QuickRetries=5 and QuickRetriesMinutes=10 and the default Retries settings in Settings -> Protocols -> SMTP -> Delivery of e-mail = 4 (60 min) the external delivery of e-mail is tried 5 times within the first 50 minutes and after that the normal 4 attempts are tried each hour?

And what is QueueRandomnessMinutes?
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6315
Joined: 2006-08-21 15:38
Location: Denmark

Re: QuickRetries & QuickRetriesMinutes

Post by SorenR » 2016-09-26 12:25

From what I can read in the code it has to do with resending message if greylisted... I am guessing that if you have 10 retries in 30 minute intervals under SMTP protocol, you can alter this by setting QuickRetries=2 and QuickRetriesMinutes=5 so that delivery will be @5 minutes, @10 minutes, @40 minutes, @70 minutes, @100 minutes, @130 minutes ... etc...
RvdH wrote:And what is QueueRandomnessMinutes?
Er... It's to fix a Windows 2000 bug :mrgreen:

Code: Select all

      if (m_iQueueRandomnessMinutes > 0)
      {

         // Get our random #
         // LOG_DEBUG("Windows 2000 does not support rand_s & pukes here");
         rnd_err = (rand_s(&tmp_rnd));

         // If no error getting random # use it
         if (rnd_err == 0)
            iRandomAdjust = (unsigned int) ((double)tmp_rnd / (double) UINT_MAX * m_iQueueRandomnessMinutes) + 1;
      }
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: QuickRetries & QuickRetriesMinutes

Post by mattg » 2016-09-26 14:20

These were added by Bill with one of his experimental builds >> viewtopic.php?f=10&t=21420#p132238

pretty much as SorenR says
RvdH wrote:And what is QueueRandomnessMinutes?
Instead of being exact every 10 minutes, that changes by up to the 'random' figure. The idea is to NOT be predicable with re-tries.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: QuickRetries & QuickRetriesMinutes

Post by RvdH » 2016-09-26 14:49

One question remains, are the QuickRetries counted on top of the normal retries or should i change the value (sum) of the normal reties with the value specified in QuickRetries?
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: QuickRetries & QuickRetriesMinutes

Post by mattg » 2016-09-26 15:42

included in the total (I think hMailserver just counts the number of retries and if less than or equal to the retries then tries again)

to be honest it is very rare for mail to reach my retry settings
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: QuickRetries & QuickRetriesMinutes

Post by RvdH » 2016-09-26 17:24

mattg wrote:included in the total (I think hMailserver just counts the number of retries and if less than or equal to the retries then tries again)

to be honest it is very rare for mail to reach my retry settings
Yup, it's included in total (tested it)

QuickRetries=2
QuickRetriesMinutes=10

Code: Select all

"APPLICATION" 2792 "2016-09-26 16:01:56.768" "SMTPDeliverer - Message 3879: Delivering message from something@something.com to something@somethingelse.com. File: C:\Program Files (x86)\hMailServer\Data\{7EDAF7C5-382E-4767-AFF2-3B263F9AF795}.eml"
"APPLICATION" 2792 "2016-09-26 16:01:57.284" "SMTPDeliverer - Message 3879: No mail servers could be found for the address something@somethingelse.com."
"APPLICATION" 2792 "2016-09-26 16:01:57.284" "SMTPDeliverer - Message 3879: Message could not be delivered. Greylisting? Scheduling it for quick retry 1 of 2 in 10 minutes."
"APPLICATION" 2792 "2016-09-26 16:01:57.284" "SMTPDeliverer - Message 3879: Message delivery thread completed."

"APPLICATION" 2792 "2016-09-26 16:12:56.817" "SMTPDeliverer - Message 3879: Delivering message from something@something.com to something@somethingelse.com. File: C:\Program Files (x86)\hMailServer\Data\{7EDAF7C5-382E-4767-AFF2-3B263F9AF795}.eml"
"APPLICATION" 2792 "2016-09-26 16:12:57.161" "SMTPDeliverer - Message 3879: No mail servers could be found for the address something@somethingelse.com."
"APPLICATION" 2792 "2016-09-26 16:12:57.161" "SMTPDeliverer - Message 3879: Message could not be delivered. Greylisting? Scheduling it for quick retry 2 of 2 in 10 minutes."
"APPLICATION" 2792 "2016-09-26 16:12:57.176" "SMTPDeliverer - Message 3879: Message delivery thread completed."

"APPLICATION" 2792 "2016-09-26 16:23:56.900" "SMTPDeliverer - Message 3879: Delivering message from something@something.com to something@somethingelse.com. File: C:\Program Files (x86)\hMailServer\Data\{7EDAF7C5-382E-4767-AFF2-3B263F9AF795}.eml"
"APPLICATION" 2792 "2016-09-26 16:24:00.795" "SMTPDeliverer - Message 3879: No mail servers could be found for the address something@somethingelse.com."
"APPLICATION" 2792 "2016-09-26 16:24:00.810" "SMTPDeliverer - Message 3879: Message could not be delivered. Scheduling it for later delivery in 60 minutes."
"APPLICATION" 2792 "2016-09-26 16:24:00.810" "SMTPDeliverer - Message 3879: Message delivery thread completed."

"APPLICATION" 2792 "2016-09-26 17:24:51.888" "SMTPDeliverer - Message 3879: Delivering message from something@something.com to something@somethingelse.com. File: C:\Program Files (x86)\hMailServer\Data\{7EDAF7C5-382E-4767-AFF2-3B263F9AF795}.eml"
"APPLICATION" 2792 "2016-09-26 17:24:52.325" "SMTPDeliverer - Message 3879: No mail servers could be found for the address something@somethingelse.com."
"APPLICATION" 2792 "2016-09-26 17:24:52.325" "SMTPDeliverer - Message 3879: Message could not be delivered. Scheduling it for later delivery in 60 minutes."
"APPLICATION" 2792 "2016-09-26 17:24:52.325" "SMTPDeliverer - Message 3879: Message delivery thread completed."
Retry count in GUI currently = 4
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: QuickRetries & QuickRetriesMinutes

Post by Bill48105 » 2017-02-02 19:51

Cool to see people are using the special INI settings I made but sad that martin never added them to the GUI. The INI settings were meant as a way to tweak the basic delivery queue that's built into hmailserver. If you're interested you can read more about many of my changes back in the old 5.4 days:
viewtopic.php?p=132238#p132238

You'll see detailed descriptions of each INI setting & WHY they were there. Perhaps QuickRetries wasn't the best term to name them but the though was eventually they'd be moved into the GUI & not matter. The short version is if you set hmail retries too low you waste bandwidth retrying too often & sometimes even get blocked by remotes thinking you're a spammer. If you set too high the mail waits too long when it might have been brief too busy or greylist. Those settings give you the ability to set the hamil retries HIGH like 90 minutes but force hmail to retry at a shorter interval for the 1st few tries. So you can have retries such as 5 min, 5 min, 90 min, 90 min ...

LOL at the windows 2000 note. Someone had my build crashing with randomness turned on & come find out 2k didnt support the random function used. It was decided to not worry about it because so few used 2k still so it was left alone & just say 2k no longer supported.
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: QuickRetries & QuickRetriesMinutes

Post by jimimaseye » 2017-02-02 20:02

Bill48105 wrote:Cool to see people are using the special INI settings I made but sad that martin never added them to the GUI. The INI settings were meant as a way to tweak the basic delivery queue that's built into hmailserver. If you're interested you can read more about many of my changes back in the old 5.4 days:
viewtopic.php?p=132238#p132238

You'll see detailed descriptions of each INI setting & WHY they were there. ......
Not easy to tell how many of them have been retained and active in the latest builds.
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: QuickRetries & QuickRetriesMinutes

Post by mattg » 2017-02-02 23:11

https://github.com/hmailserver/hmailser ... ttings.cpp

Code: Select all

   IniFileSettings::IniFileSettings() :
      is_internal_database_(false),
      dbport_(0),
      no_of_dbconnections_(0),
      add_xauth_user_header_(false),
      no_of_dbconnection_attempts_(6),
      no_of_dbconnection_attempts_Delay(5),
      max_no_of_external_fetch_threads_(15),
      greylisting_enabled_during_record_expiration_(true),
      greylisting_expiration_interval_(240),
      preferred_hash_algorithm_(3),
      dnsbl_checks_after_mail_from_(false),
      log_level_(0),
      max_log_line_len_(500),
      quick_retries_(0),
      quick_retries_Minutes(0),
      queue_randomness_minutes_(0),
      mxtries_factor_(0),
      sqldbtype_(HM::DatabaseSettings::TypeUnknown),
      sep_svc_logs_(false),
	  rewrite_envelope_from_when_forwarding_(false),
      archive_hardlinks_(false),
      pop3dmin_timeout_(false),
      pop3dmax_timeout_(false),
      pop3cmin_timeout_(false),
      pop3cmax_timeout_(false),
      smtpdmin_timeout_(false),
      smtpdmax_timeout_(false),
      smtpcmin_timeout_(false),
      smtpcmax_timeout_(false),
      samin_timeout_(false),
      samax_timeout_(false),
      clam_min_timeout_(false),
      clam_max_timeout_(false),
      samove_vs_copy_(false),
      indexer_full_minutes_(0),
      indexer_full_limit_(0),
      indexer_quick_limit_(0),
      load_header_read_size_(0),
      load_body_read_size_(0),
      blocked_iphold_seconds_(0),
      smtpdmax_size_drop_(0),
      backup_messages_dbonly_(false),
      add_xauth_user_ip_(false)
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: QuickRetries & QuickRetriesMinutes

Post by mattg » 2017-02-02 23:15

Bill48105 wrote:Cool to see people are using the special INI settings I made but sad that martin never added them to the GUI.
Please make a Github fork or branches. Branches are good so that they can be added back to main easily.

We have a couple of others doing some minor development here and there.
I'd love to see the work being done on the WebAdmin panel, and the event OnHELO being added to main for a start.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: QuickRetries & QuickRetriesMinutes

Post by jimimaseye » 2017-02-02 23:22

Indeed. But can we be sure that was is listed in that list ACTUALLY has a function buried in the code? Could they be INI fil settings/key words that no longer have functionality within the code (due to Martin not including the actual function). For example, a search for "greylisting_enabled_during_record_expiration" does show up in Bills list of INI settings so it isnt one of Bills (and is standard by Martin) but we have no way of knowing what it is. And Billslist of settings list, for example, "ValidEmailPattern" - but Ive seen the code and from what I can tell this is definitely not used to determine a valid email address.
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: QuickRetries & QuickRetriesMinutes

Post by mattg » 2017-02-02 23:34

Under the bit I quoted there is more which detail default values.

You can search the entire code for each of them and see where else it is being used. There may well be some that aren't used elsewhere in the code (like say SAMoveVsCopy)
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: QuickRetries & QuickRetriesMinutes

Post by jimimaseye » 2017-02-03 00:14

Oooh! Ive just tried a couple of them.

ArchiveDir=C:\Pathto\Archive

This is perfect for those that dont want to actual create/state an actual email address in the MIRRORING function. It quite literally create a copy of all inbound and outbound emails in the archive folder (that folder containing a domain subfolder with all emails). eg, inbound: "{F4917B04-F8AB-4FE2-8B11-603659BF48B5}.eml" and outbound file: "Sent-{F4917B04-F8AB-4FE2-8B11-603659BF48B5}.eml". The benfit is it doesnt then rely on smtp sending those copies (freeing up resources - but only good if youre happy with the archive being local).

SepSvcLogs=1
It createes separate log files - 1 for each logging type (IMAP, SMTP etc). eg, "hmailserver_SMTP_2017-02-02.log" contains all SMTP traffic logging only. Makes reading the sequence easier for those that dont have access/ability to read the log analysers.

(And confirmed: INI setting of Bills "ValidEmailPattern", despite listed in Bills writeup, is not included in final build. This would suggest that if it isnt in the IniFileSettings.cpp then it isnt used.)

@Mattg
If a detailed posting was made of these things, is it possible to make a 'sticky' on this forum? Or is that Martin's decision only?
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: QuickRetries & QuickRetriesMinutes

Post by mattg » 2017-02-03 00:48

I don't think I can make a sticky, but I recall that someone else (other than Martin) does, we can Ask.

I was also going to submit a new branch on github removing all of the unused old stuff.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: QuickRetries & QuickRetriesMinutes

Post by jimimaseye » 2017-02-03 00:49

Ok, Ive checked against Bills initial list of INI': the following are no longer included/valid:

ValidEmailPattern
AutoReplyReturnPath
SPFDefaultPolicy
SPFPolicyOverride
DnsServers
SSLOptionList
SSLCipherList

Ill redo Bills list in to a HOW TO style documentation in to DEV/ALPHA section without these exceptions.
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: QuickRetries & QuickRetriesMinutes

Post by jimimaseye » 2017-02-03 01:51

mattg wrote:I don't think I can make a sticky, but I recall that someone else (other than Martin) does, we can Ask.

I was also going to submit a new branch on github removing all of the unused old stuff.
viewtopic.php?f=10&t=30900 - What do you think?
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: QuickRetries & QuickRetriesMinutes

Post by mattg » 2017-02-03 05:15

I'm pretty certain that this one has been dropped >> SAMoveVsCopy

I'll check the others, but Yes I like what you have done.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: QuickRetries & QuickRetriesMinutes

Post by Bill48105 » 2017-02-03 07:24

mattg wrote:
Bill48105 wrote:Cool to see people are using the special INI settings I made but sad that martin never added them to the GUI.
Please make a Github fork or branches. Branches are good so that they can be added back to main easily.

We have a couple of others doing some minor development here and there.
I'd love to see the work being done on the WebAdmin panel, and the event OnHELO being added to main for a start.
HUH?!? I did a full commit of ALL my changes forever ago before I went on hiatus. 3 years ago in fact:
https://github.com/Bill48105/hmailserver
martin just didn't want to accept the changes to be merged so instead he said he'd pick the stuff he wanted to include & left it at that.
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: QuickRetries & QuickRetriesMinutes

Post by Bill48105 » 2017-02-03 07:43

jimimaseye wrote:
Bill48105 wrote:Cool to see people are using the special INI settings I made but sad that martin never added them to the GUI. The INI settings were meant as a way to tweak the basic delivery queue that's built into hmailserver. If you're interested you can read more about many of my changes back in the old 5.4 days:
viewtopic.php?p=132238#p132238

You'll see detailed descriptions of each INI setting & WHY they were there. ......
Not easy to tell how many of them have been retained and active in the latest builds.
If I tried hard enough I could figure it out because anything I committed via SVN would likely still be in the code because with SVN he didn't have to approve & it'd require him to roll back my changes or change individual code lines. (I always detailed my changes to SVN but that's gone but I always put the changes in my experimental post) Once he made the move to github he started from that last SVN including my changes to that point that I had got into the SVN & he started making his own changes from that point on github without me knowing so I had changes on my end that hadnt been committed yet. So I had to fork his official on github then it took me forever to manually merge his with mine & I committed that. But I don't think he ever accepted or used any of my changes between SVN & Github so it ended up being a giant waste of my time & I moved onto other projects including my personal life which had suffered due to amount of time I had dedicated to hmailserver. Good to see so many of you still around volunteering your time.
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: QuickRetries & QuickRetriesMinutes

Post by jimimaseye » 2017-02-03 12:42

mattg wrote:I'm pretty certain that this one has been dropped >> SAMoveVsCopy

I'll check the others, but Yes I like what you have done.
From what I see, this function is still there:

https://github.com/hmailserver/hmailser ... Client.cpp (line 182):

Code: Select all

      if (IniFileSettings::Instance()->GetSAMoveVsCopy())
         {
            // Move temp file overwriting message file
            FileUtilities::Move(sTempFile, message_file_, true);
            LOG_DEBUG("SA - Move used");
         }
         else
         {
            // Copy temp file to message file
            FileUtilities::Copy(sTempFile, message_file_, false);
            LOG_DEBUG("SA - Copy+Delete used");
Cant say I know what the point/benefit of it is though (default 'copy+delete' would avoid file locking issues).
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: QuickRetries & QuickRetriesMinutes

Post by jimimaseye » 2017-02-03 12:52

Bill48105 wrote: But I don't think he ever accepted or used any of my changes between SVN & Github so it ended up being a giant waste of my time & I moved onto other projects including my personal life.....
I think you will be surprised Bill. As you can see from the above I have reviewed your INI's and sanitised it by looking at what is stil there (here) and all except (mainly) the DNS, SSL and SPF related ones are still functional. (From what I read on page 4 of your builds thread he just chose to not accept them as 'official' without his testing environment, documentation and GUI inclusion - but they are still there unofficially. That doesnt mean they are of no use or unrequired and if he didnt want them he would have removed them as he did with the DNS, SPF and SSL stuff.)
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: QuickRetries & QuickRetriesMinutes

Post by Bill48105 » 2017-02-03 16:51

jimimaseye wrote:
mattg wrote:I'm pretty certain that this one has been dropped >> SAMoveVsCopy

I'll check the others, but Yes I like what you have done.
From what I see, this function is still there:

https://github.com/hmailserver/hmailser ... Client.cpp (line 182):

Code: Select all

      if (IniFileSettings::Instance()->GetSAMoveVsCopy())
         {
            // Move temp file overwriting message file
            FileUtilities::Move(sTempFile, message_file_, true);
            LOG_DEBUG("SA - Move used");
         }
         else
         {
            // Copy temp file to message file
            FileUtilities::Copy(sTempFile, message_file_, false);
            LOG_DEBUG("SA - Copy+Delete used");
Cant say I know what the point/benefit of it is though (default 'copy+delete' would avoid file locking issues).
Cool it's still there it makes a big difference in load. hmail creates tmp files EVERY time a message is created. Be it incoming email or moving between imap folders etc. As you can imagine that adds up to a lot of read/writes to begin with but then hmail would COPY that temp file to it's final destination. So pretty much everything was written twice, read once for something that only really requires written once.. And with enough emails especially larger ones it makes a HUGE difference in not only CPU usage but HDD activity including saturation depending on the storage system. Yes there are risks mostly with permissions but as long as the temp folder doesn't have special permissions the final destination file has OK perms too. It's one of those behind the scene things I found in hmail when trying to make it more efficient & IMO should have been made the default long ago but wasn't my call. But yeah do some testing with it on & off you'll see what i mean. no comparison of 2W+R vs 1W+move.. Night and day.
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: QuickRetries & QuickRetriesMinutes

Post by Bill48105 » 2017-02-03 17:01

jimimaseye wrote:
Bill48105 wrote: But I don't think he ever accepted or used any of my changes between SVN & Github so it ended up being a giant waste of my time & I moved onto other projects including my personal life.....
I think you will be surprised Bill. As you can see from the above I have reviewed your INI's and sanitised it by looking at what is stil there (here) and all except (mainly) the DNS, SSL and SPF related ones are still functional. (From what I read on page 4 of your builds thread he just chose to not accept them as 'official' without his testing environment, documentation and GUI inclusion - but they are still there unofficially. That doesnt mean they are of no use or unrequired and if he didnt want them he would have removed them as he did with the DNS, SPF and SSL stuff.)
What i meant were the changes between move from SVN to github. I'm not sure any of those changes ever got included.. I spent &^% ton of time doing STARTTLS & he did it over himself without my code. (The SSL stuff was mainly for that) The other missing things were my usual attempts to improve hmail in little ways while still remaining backward compatible. I'd have to look back WHY I made the changes but I always had a reason.. Usually trying to solve a problem in the forum or optimize something. For example I recall the SPF setting let you choose to reject soft fails which isn't really an acceptable thing to do but so many lazy people leave their spf on ~all transitioning mode forever that their spf is essentially worthless & gets abused. I had also made it so you could specify your own dns server since hmail uses system DNS by default which can be problematic for some people. I actually think there were others I done on my local builds that I never shared but at this point couldn't say for sure. It's been good 3 years since I looked at the source.
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: QuickRetries & QuickRetriesMinutes

Post by jimimaseye » 2017-02-03 17:05

Bill48105 wrote: Cool it's still there it makes a big difference in load. hmail creates tmp files EVERY time a message is created. Be it incoming email or moving between imap folders etc. As you can imagine that adds up to a lot of read/writes to begin with but then hmail would COPY that temp file to it's final destination. So pretty much everything was written twice, read once for something that only really requires written once.. And with enough emails especially larger ones it makes a HUGE difference in not only CPU usage but HDD activity including saturation depending on the storage system. Yes there are risks mostly with permissions but as long as the temp folder doesn't have special permissions the final destination file has OK perms too. It's one of those behind the scene things I found in hmail when trying to make it more efficient & IMO should have been made the default long ago but wasn't my call. But yeah do some testing with it on & off you'll see what i mean. no comparison of 2W+R vs 1W+move.. Night and day.
Would there be a valid argument that doing copy+delete avoids filelocking (access) problems (which would prevent a successful MOVE)? And in the worse case scenario, the temp directory just has redundant temporary files in it (in such file locking situations) if Copy+Delete is used (whereas a MOVE would have created a more fundamental error)? Or isnt there a situation where that file could still be locked when it comes to that section of code?
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: QuickRetries & QuickRetriesMinutes

Post by Bill48105 » 2017-02-06 06:59

jimimaseye wrote:
Bill48105 wrote: Cool it's still there it makes a big difference in load. hmail creates tmp files EVERY time a message is created. Be it incoming email or moving between imap folders etc. As you can imagine that adds up to a lot of read/writes to begin with but then hmail would COPY that temp file to it's final destination. So pretty much everything was written twice, read once for something that only really requires written once.. And with enough emails especially larger ones it makes a HUGE difference in not only CPU usage but HDD activity including saturation depending on the storage system. Yes there are risks mostly with permissions but as long as the temp folder doesn't have special permissions the final destination file has OK perms too. It's one of those behind the scene things I found in hmail when trying to make it more efficient & IMO should have been made the default long ago but wasn't my call. But yeah do some testing with it on & off you'll see what i mean. no comparison of 2W+R vs 1W+move.. Night and day.
Would there be a valid argument that doing copy+delete avoids filelocking (access) problems (which would prevent a successful MOVE)? And in the worse case scenario, the temp directory just has redundant temporary files in it (in such file locking situations) if Copy+Delete is used (whereas a MOVE would have created a more fundamental error)? Or isnt there a situation where that file could still be locked when it comes to that section of code?
Maybe in theory but there should never be a case where 2 threads end up with the same filename since they are GUID generated. In the years of running multiple hmail servers handling millions of emails using my custom build I can't say I ever ran into a scenario where I believe there was contention for a file due to move vs copy/delete. It could also be argued there's less chance of issue since it's faster to write once then move than write twice & delete since the file(s) is/are in-use less time.
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

Post Reply