Better queue processing [90%]

Use this forum if you want to suggest a new feature to hMailServer. Before posting, please search the forum to confirm that it has not already been suggested.
Post Reply

Do you need this feature?

Yes
56
93%
No
4
7%
 
Total votes: 60

Aldoir
Normal user
Normal user
Posts: 59
Joined: 2005-12-01 12:20

Better queue processing [90%]

Post by Aldoir » 2006-05-11 21:43

I'm with a trouble, that I imagine can be solved easily

Once a user send a lot of mails (mailing list, by eg), other users can't receive messagens, since all messages get into the same queue for processing

My suggestion is

1) have a separated thread/queue to process outgoing messages and delivery of local messages

2) a priority scheme, that delivers one message per user.

Eg. if a user sent 100 emails, and other send 1 single email, the second should not wait all 100 emails be delivered to get its message delivered, instead, after the first mail from fist user, its email will be delivered

Confusing? yes.. my english is not so well as it should :)

comments are welcome




MOD EDIT: Mostly done in 5.4 with something like this in hmailserver.ini:
(Still needs to be added to GUI admin & perhaps further settings such as a multiplier)

Code: Select all

[Options]
QuickRetries=3
QuickRetriesMinutes=7 
QueueRandomnessMinutes=7
(See posts below for details)
Note: Local deliveries are not queued

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Post by ^DooM^ » 2006-05-11 22:47

I figured out that sending 10k emails to my userbase seriously disrupted my incoming and outgoing emails too ;)

I installed a second local copy of hmail on a seperate server for mass newsletter emails and left my main hmail server have a nice rest :)

It would be nice not to have to do this though if there was some way to set the priority low for mass emails which instructed hmail to pause every x seconds to allow incoming and outgoing of normal priority emails.

Not sure how this would be accomplished though..

-Jon
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

theTerran
Senior user
Senior user
Posts: 289
Joined: 2004-06-22 18:07
Location: Florida

Post by theTerran » 2006-05-12 07:58

For those of us with mailing lists, it would be nice to have the option of handling these bulk mails at a lower priority. Sure, we want all mail to go out quickly, but user mail should be first and fastest if given the choice!

Just my $.02
Daniel
Terran Enterprises LLC

timb0311
Normal user
Normal user
Posts: 50
Joined: 2006-09-22 20:15

Post by timb0311 » 2006-09-22 20:27

Software Developer here, new to hmailserver.

From a design perspective, each protocol (SMTP, IMAP, POP3) should have it's own Processing Queue. Whether each is its own Windows service or just a background thread running for each... accepting connections and handing connections over to worker threads for that protocol.

For a workaround, which is a common setup in a high volume evironment, you would have multiple servers anyway. A front end server to accept incoming, scan it for viruses etc. Then a backend server when users log in and read emails and send them. As long as you can share the config info from the db across servers.

This should fix your ability to accept incoming on the front end while the backend is sending.

Not sure if you could setup mutliple instance of hmailserver on a single machine or if you would need 2 machines. Someone would need to test that.

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2006-09-22 21:20

timb0311, I don't see how your suggestions would solve the problems that Aldoir describes...

hMailServer 4.2 already has seperate threads for the different protocols. In hMailServer 4.3, IOCP is used instead which gives better performance. But this doesn't have any effect on Aldoirs problem..

timb0311
Normal user
Normal user
Posts: 50
Joined: 2006-09-22 20:15

Post by timb0311 » 2006-10-27 18:58

martin wrote:timb0311, I don't see how your suggestions would solve the problems that Aldoir describes...

hMailServer 4.2 already has seperate threads for the different protocols. In hMailServer 4.3, IOCP is used instead which gives better performance. But this doesn't have any effect on Aldoirs problem..
Did you even read my reply? As previsouly stated, I am new to hMailserver. So I don't know the queuing implementation. I merely stated, from a design perspective, how servers should be implemented.

As for the other setup... I am not going into detail. You can google it to find out more.

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2006-10-28 10:59

Yes, I actually read your reply. I just didn't see what you were trying to say. I didn't see how your suggestion had anything to do with the feature requested in this thread. :)

iprat
Normal user
Normal user
Posts: 247
Joined: 2005-05-20 16:50
Location: Barcelona, EU
Contact:

Post by iprat » 2006-10-30 23:42

Just some ideas:

To make two diferent engines for internal and external email seems a bit overkill for me, probably could be made on IP ranges basis, but this would only free local users from this kind of delivery saturation and would lead to hide the problem instead on solving it.

It could help but that doesn't solve entirely the problem.

Maybe as SMTP server has a multithreaded design a smart way would be the following:

The basic idea would be that a single user by default could only use one thread unless there are no more requests for SMTP threads usage.

Lets say A wants to send 500 emails and there are 5 threads. If no one else is using the server A will have 5 SMTP threads for him to send the 500 emails at a good rate.

But lets say B, C, and D begin to work and want to send emails to other users, if they send a message their messages will be processed almost immediately when any of the 5 SMTP threads finishes any of the currently A jobs, and A jobs would only enter again in the SMTP threads when other users jobs have finished or no A user thread is in the SMTP threads stack.

This makes sense for me in the idea of a huge job is done as fast as possible while the server is IDLE, and when the server is loaded the job will be relatively slowly served. I say relatively because when the massive job arrives to the top of the queue they will have one SMTP thread assured for him while enabling other users to work normally.

This system seems to maintain the idea of the first to arrive is served first while assuring that a huge work doesn't block the entire server. Only when several huge jobs (exactly when as much huge jobs as SMTP threads) are sent to the SMTP stack the server will be effectively blocked.

This could be coded as:

1)Loop here until there is at least one SMTP thread available
2)Have a look at the first message in queue
3)If the message taken from queue is from a user that already uses any of the SMTP threads in use it is temporarily skipped ELSE then process message in the free SMTP thread.
4)Loop through rule #3 until we reach the end of the queue or we find a match of the ELSE case in rule #3.
5)In the case we find the end of the queue then just assign the job to the first iteration of rule #3.

The problem will probably Martin have to face will be that the jobs inserted in the queue will have to be for unique destinies for this rules to have effect and I don't know if this is really how it is done now. It makes sense to me but I really don't know if hMailServer is really designed in that way.

Does that make sense to you Martin ?
My perfect combination:
hMailServer 5.6.1 (B2208), ASSP 1.3.3.8 (antispam), Clamav 0.98.6 (antivirus)

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Post by ^DooM^ » 2006-10-31 12:54

Sounds great in theory. Another possibility is have some sort of iprange(s) that is dedicated to sending a lot of email that will be given a lower priority than other email. Anything matching this iprange(s) when sending email will be flagged internally. When it comes to sending hmail checks for this flag in the queue and depending on how many emails are flagged a set number of threads are assigned to these emails freeing up the servers other threads to normal operations.

Something to think about at least :)

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2006-10-31 13:07

Yeah I agree that that might be a good way to go. Will think about it. :)

chanas
Normal user
Normal user
Posts: 57
Joined: 2006-04-08 00:27
Location: Athens/Greece
Contact:

Post by chanas » 2006-12-23 11:23

Another thing I found out while delivering 10K messages. The worker threads/delivery threads is not a "live" setting. It seems to need a restart of hmail to work. So when I was about half-way in the queue full with yahoo graylisted pending messages, I restarted hmail with 20 threads (which resulted in starting the queue from the begining) but eventually it ended faster.

The thing is that delivery to mailbox and delivery to external (SMTP) is done by the same thread. I had incoming messages that were accepted but the since the delivery event was queued the message wasnt delivered to the mailbox and pop3 showed 0 mails.
So I could send e-mails, receive e-mails, but they were delivered to local mailboxes a bit late.

Apart for that during the whole time memory usage and CPU was quite low, which was quite good for my poor P4/1GB mail server :D

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2006-12-23 12:15

It's correct that changing the number of workers requires server restart. Changing the number of delivery threads doesn't though.

ke
New user
New user
Posts: 2
Joined: 2006-12-23 12:07

Domain pooling / grouping

Post by ke » 2006-12-23 12:23

Hello,

I've recently started trialling hM as a replacement for Communigate (and it's expensive licenses). So far so good, it has not yet been pushed mailing list wise but performance seems ok and I like the SQL datasource.

One thing I have noticed in the logs is that smtp delivery does not seem to be grouped via domain - it seems as though hM initiates a new connection for each delivery (which if so, is very inefficient).

A major speed increase could be obtained by simply delivering multiple messages with same connection per domain. So if you have 10 messages to delivery to me.com connect to me.com and send all of them (or in smaller groups if need be). This would save having to reinitiate the connection for each message.

This was based on simple log monitoring, so please do let me know if something similar is already in place. So far hM seems very promising and fills a gap in the windows market.

Ke

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2006-12-23 12:49

It's correct that multiple message deliveries aren't grouped by domain. But I'm not so sure that grouping would give much better performance. From a simple TCP/IP perspective, it would be more efficient. But grouping delivery of email would increase the complexity of the rest of the SMTP server. hMailServer would have to keep track of much more things at the same time. Also, the connection to remote SMTP server is somewhat time-consuming, but not much compared to the other parts of the SMTP delivery. Also, with multiple parallel delivery threads, hMailServer can initiate many SMTP connections at the same time.

This together, I don't think there would be a 'major speed increase'. Not sure though, since I don't have any metrics. Do you have any metrics on this really showing that there would be a major speed increase?

ke
New user
New user
Posts: 2
Joined: 2006-12-23 12:07

Post by ke » 2006-12-23 13:45

No numerical stats - this is based on experience with other mail packages.

Communigate, Postfix and Visnetic/Merek all use this technique - and all (esp Communigate) have excellent delivery performance.

http://www.stalker.com/CommuniGatePro/S ... ltiChannel
http://www.postfix.org/CONNECTION_CACHE_README.html

The major mail services also prefer you to re-use your SMTP connections, more so if running an active list service.

Comparing hM with something like Communigate isn't exactly fair footing, although I guess it depends on how high you want to raise the bar or what the target sector is.

]Pablo[
New user
New user
Posts: 13
Joined: 2005-09-11 12:28
Location: Bologna, Italy
Contact:

Post by ]Pablo[ » 2007-06-15 17:59

Hello,
any news about this feature request?
I'm very interested on it...but I can't see any statement in the changelog or documentation.

thanks!

Thomas Parvais
Normal user
Normal user
Posts: 111
Joined: 2004-12-17 12:21
Contact:

Re: Better queue processing

Post by Thomas Parvais » 2008-10-07 11:41

Hello,

If I'm not wrong, it will be a bad idea to send all emails "in batch" per domain as the SMTP receiver might consider this as spamming (too many email in certain time frame).

This is very close to my request to be able to add Hmailserver a throtle option for SMTP out (distribution list) for which I've many issue of this kind.

Thomas
Interrested by Law & new technologies ?
Intéressé par le droit de l'internet et des nouvelles technologies ?
Visit/Visitez http://www.droit-technologie.org

ObiWan
Senior user
Senior user
Posts: 278
Joined: 2010-07-21 14:30
Location: Halfway between Germany and Egypt

Re:

Post by ObiWan » 2010-07-22 17:25

^DooM^ wrote:Sounds great in theory. Another possibility is have some sort of iprange(s) that is dedicated to sending a lot of email that will be given a lower priority than other email. Anything matching this iprange(s) when sending email will be flagged internally. When it comes to sending hmail checks for this flag in the queue and depending on how many emails are flagged a set number of threads are assigned to these emails freeing up the servers other threads to normal operations.

Something to think about at least :)
Or just "rehash" the queues; that is, assign lower priority to messages belonging to distribution lists so that they won't go out "all together" and also "shuffle" the queue so that messages won't go out in FIFO order; an approach to implement such a thing may be the following

setup 5 outbound queues, let's call then A, B, C, D and E

messages belonging to "distribution lists" will go to queue A
messages above a given size will go to queue B
all the other messages will be evenly distributed amongst queues B, C and D

the delivery thread(s) will then use a roundrobin approach so that
it will fetch a message from queue A, delivery it, fetch a message
from queue B, delivery it and so on until reaching queue D then it
will restart from A; this will allow "shuffling" the delivery and solve
or at least relieve the issue

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Better queue processing

Post by Bill48105 » 2010-07-22 19:47

Yeah ObiWan, that sounds probably like the 'ideal' setup but likely requires quite a bit of work on the source to get done and I recall many ideas being discussed at length with various work vs results. In other threads I mentioned something with a similar effect with 'randomization' and staggering of the queue 'nexttrytime' to give a similar result with the existing queue system in place now but likely many fold easier to implement because it just requires adding calculated value to the time to pace it into the future leaving gaps for other mail to go. I guess whoever works on improving the queue code can decide how much effort they want to put into it. If it's me I think I know which I'd do. :D Of course there'd be nothing stopping a combination either now or later in the future either.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

ObiWan
Senior user
Senior user
Posts: 278
Joined: 2010-07-21 14:30
Location: Halfway between Germany and Egypt

Re: Re:

Post by ObiWan » 2010-07-23 09:49

ObiWan wrote:
setup 5 outbound queues, let's call then A, B, C, D and E

messages belonging to "distribution lists" will go to queue A
messages above a given size will go to queue B
all the other messages will be evenly distributed amongst queues B, C and D
Sorry, I messed up the above, it should read

messages belonging to "distribution lists" will go to queue A
messages above a given size will go to queue E
all the other messages will be evenly distributed amongst queues B, C and D

ok, not it sounds right :)

gpanucci
New user
New user
Posts: 10
Joined: 2010-08-19 13:36

Re: Better queue processing

Post by gpanucci » 2010-08-21 03:40

I know i'm a newbie but doesn't it make sense because lots of people that use mailing lists don't send them from primary emails e.g. first.last@whoever.com
So doesn't it make sense rather than de-prioritising a whole IP address(because lots of people dont have computers to soley handle mailing lists), that you just get the admin to specify when making the account if it will be a mailing list account then just make the priority for that account lower? That is to say, while IDLE send messages, if not wait.

ObiWan
Senior user
Senior user
Posts: 278
Joined: 2010-07-21 14:30
Location: Halfway between Germany and Egypt

Re: Better queue processing

Post by ObiWan » 2010-08-23 09:56

gpanucci wrote:I know i'm a newbie but doesn't it make sense because lots of people that use mailing lists don't send them from primary emails e.g. first.last@whoever.com
So doesn't it make sense rather than de-prioritising a whole IP address(because lots of people dont have computers to soley handle mailing lists), that you just get the admin to specify when making the account if it will be a mailing list account then just make the priority for that account lower? That is to say, while IDLE send messages, if not wait.
Uhm... sorry but I can't understand your point

A message sent to a mailing list will be sent from whatever email address has been
registered and using THAT address mailserver (e.g. gmail or whatever); the ML server
will then receive such a message and proceed redistributing it to the list members (and
rewrite the envelope from and the reply-to so that the first will point to the ML admin
and the second one to the ML address), so, the server knows if a given address belongs
to a mailing list and will be able to deal with it w/o problems; the sender address does
not play any role in this case, so, again, I can't understand your point

gpanucci
New user
New user
Posts: 10
Joined: 2010-08-19 13:36

Re: Better queue processing

Post by gpanucci » 2010-08-23 11:04

The point is rather than force the server to send the message at all,
Specify if automailer@domain.com or whatever the outward email address is, as a mailing list address, and in turn reduce the priority of the emails that the address is sending, so that if 1000 people on a mailing list get sent a message other users don't have to wait until the server has finished processing the initial message before it handles other higher priority email.

ObiWan
Senior user
Senior user
Posts: 278
Joined: 2010-07-21 14:30
Location: Halfway between Germany and Egypt

Re: Better queue processing

Post by ObiWan » 2010-08-23 11:13

gpanucci wrote:The point is rather than force the server to send the message at all,
Specify if automailer@domain.com or whatever the outward email address is, as a mailing list address, and in turn reduce the priority of the emails that the address is sending, so that if 1000 people on a mailing list get sent a message other users don't have to wait until the server has finished processing the initial message before it handles other higher priority email.
Hm... if the ML is handled by hMailServer, then it will receive messages to someml@example.com
and then send them to all the list members, so, in such a case, using the approach I suggested
(that is separate queues) will deal with this and avoid outbound congestions; if, on the other hand, an hMailServer user is sending a message to an external ML then the problem doesn't
exist, the email will just flow to the ML external server which will just deal with it, so there's
no need to handle it in a special way nor to have any knowledge of whatever external ML

ObiWan
Senior user
Senior user
Posts: 278
Joined: 2010-07-21 14:30
Location: Halfway between Germany and Egypt

Re: Better queue processing

Post by ObiWan » 2010-10-21 18:52

Aldoir wrote: Eg. if a user sent 100 emails, and other send 1 single email, the second should not wait all 100 emails be delivered to get its message delivered, instead, after the first mail from fist user, its email will be delivered
Quick workaround; install the IIS SMTP service, configure it with a "dummy"
local domain and set it up to listen on an alternate port (e.g. 125) or on a
separate, dedicated IP, then configure hMailServer to send all the outbound
mail to the IIS SMTP, this will quickly offload the hMail queue and will speed
up delivery since the IIS SMTP doesn't have the same outbound limitation

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Better queue processing

Post by Bill48105 » 2010-10-21 21:31

So ObiWan.. How does IIS queue work that is better? LIFO? Send them all at once? Or is the point just to have all outbound mail offloaded from hmail so hmail is free to deliver local? Does IIS add new mail to front of queue or something otherwise not sure how the 2nd person's 1 email could be helped unless it was local.. I'm not familiar with IIS SMTP to know how the queue/delivery mechanism works so that's why I'm asking. ;)
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

ObiWan
Senior user
Senior user
Posts: 278
Joined: 2010-07-21 14:30
Location: Halfway between Germany and Egypt

Re: Better queue processing

Post by ObiWan » 2010-10-22 10:03

Bill48105 wrote:So ObiWan.. How does IIS queue work that is better? LIFO? Send them all at once? Or is the point just to have all outbound mail offloaded from hmail so hmail is free to deliver local? Does IIS add new mail to front of queue or something otherwise not sure how the 2nd person's 1 email could be helped unless it was local.. I'm not familiar with IIS SMTP to know how the queue/delivery mechanism works so that's why I'm asking. ;)
Bill
Bill, the whole idea is about external email, that is messages
which need to be routed "to the internet" - now it sounds like hMS
has some issues with the outbound queue management so, since
solving it would mean changing the code and since that may/will
take some time, I'm suggesting the above (IIS SMTP as smarthost)
as a workaround for this issue, the idea is that hMailServer will
quickly offload its outbound queue passing messages over
to the local IIS SMTP (that would be fast), the latter will then take
care of routing those to whatever external MX and since IIS SMTP
can be configured with a number of parallel delivery threads (1000
by default), that could also help avoiding the current delivery issues
and speed up the whole delivery process

Again, the above isn't a fix but just a workaround but it still
worth a try imVVVHo

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: Better queue processing

Post by ^DooM^ » 2010-10-22 10:18

You can set hmail to use 1000 delivery threads. I don't see why using IIS SMTP is any different to using hmails SMTP queue. It would only be better if IIS SMTP Sent all yahoo emails, all hotmail emails, all gmail emails etc etc etc in one session from the queue instead of opening up a new connection for each email.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

ObiWan
Senior user
Senior user
Posts: 278
Joined: 2010-07-21 14:30
Location: Halfway between Germany and Egypt

Re: Better queue processing

Post by ObiWan » 2010-10-22 10:49

^DooM^ wrote:You can set hmail to use 1000 delivery threads. I don't see why using IIS SMTP is any different to using hmails SMTP queue. It would only be better if IIS SMTP Sent all yahoo emails, all hotmail emails, all gmail emails etc etc etc in one session from the queue instead of opening up a new connection for each email.
The whole thread started due to some apparent issues with hMS queue handling
Once a user send a lot of mails (mailing list, by eg), other users can't receive messages, since all messages get into the same queue for processing
so, it sounds like even if you set hMail to handle 1000 delivery threads it still has
issues if messages are over a given size; that's why I'm suggesting (again, just
as a workaround) to use IIS SMTP to quickly offload the hMS queue and relieve
the issue

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: Better queue processing

Post by ^DooM^ » 2010-10-22 11:17

As long as hmail doesn't offload mail bound for local accounts then I can see the advantage of using a secondary SMTP smarthost. I send newsletter mail through a secondary hmail setup on a separate server.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

ObiWan
Senior user
Senior user
Posts: 278
Joined: 2010-07-21 14:30
Location: Halfway between Germany and Egypt

Re: Better queue processing

Post by ObiWan » 2010-10-22 12:19

^DooM^ wrote:As long as hmail doesn't offload mail bound for local accounts then I can see the advantage of using a secondary SMTP smarthost. I send newsletter mail through a secondary hmail setup on a separate server.
That's exactly the point, my suggestion is to set up hMailServer to use
the IIS SMTP as a "smarthost" - this means that hMS will send to the
IIS SMTP all (and only) the mail which needs to be routed to external
destinations, all the internal mail will still be handled as usual - also,
any "event script" will still work since hMail will still call it when it will
send out the messages to IIS SMTP :)

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Better queue processing

Post by Bill48105 » 2010-10-22 14:33

LOL, sorry to stir up anything but thx ObiWan you did clarify what I had asked & I agree it could helpful for some scenarios but certainly wouldn't advocate everyone do it (nor do I believe you suggested such a thing either).

Queue stuff is on my immediate list of things to work on but I've been holding off to avoid delaying 5.4 release any further. I've looked at the code & believe there are a few things I can do to help the queue a lot. Many I've mentioned before but with code in hand I see these things should be relatively easy to do & low risk of breaking anything:

- Apply user-defined randomness factor to spread the messages out & likely option per type such as lists (higher to spread further & cause delivery over time instead of all at once), local-to-local (lower to keep at front of queue & therefor quicker delivery) etc

- Option for *1st* retry delay aside from normal retry delay to help with greylisting & not need retry schedule to be so aggressive (WAY less retries per day!)

- Max MX tries & possibly Randomness factor in MX-tries to avoid queue tie-ups for places like yahoo who are shitheads with TONS of MX records, accept entire message before rejecting/delaying (mailbox full, try again later, etc). Quick-fix is not try every MX record but try them randomly per queue try but should track which ones are working or not ideally. Might require some type of "order" in the randomness to avoid appearing as spammer.

- At some point possibly ability to have multiple queues but I'd do all of the above 1st & see how much improvement there is 1st before I spent the effort. (I already have ideas how I'd do it but have a ton of other more important things I'd like to get to.)

I really think those should help tremendously based on watching current queue delivery & what types of stuff 'hog' the queue. & why.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

ObiWan
Senior user
Senior user
Posts: 278
Joined: 2010-07-21 14:30
Location: Halfway between Germany and Egypt

Re: Better queue processing

Post by ObiWan » 2010-10-22 16:55

Bill48105 wrote: LOL, sorry to stir up anything but thx ObiWan you did clarify what I had asked & I agree it could helpful for some scenarios but certainly wouldn't advocate everyone do it (nor do I believe you suggested such a thing either).
Aye aye, sir, never wrote it should always be used, just... since some users complained about
such an issue, thought that the "IIS SMTP" may be a decent workaround for the time being :)
Bill48105 wrote: - Apply user-defined randomness factor to spread the messages out & likely option per type such as lists (higher to spread further & cause delivery over time instead of all at once), local-to-local (lower to keep at front of queue & therefor quicker delivery) etc
Hm... what about "shuffling" the queue so that "big messages" will get "intermixed" with small ones so allowing better delivery ? Sure that won't work as a FIFO but afaik there isn't any RFC dictating that a mailserver has to use a FIFO approach for the delivery queue :D
Bill48105 wrote: - Option for *1st* retry delay aside from normal retry delay to help with greylisting
& not need retry schedule to be so aggressive (WAY less retries per day!)
I think that a separate queue to hold "retries" may be a good idea although I'm not
sure it's feasible; using a separate queue will allow you to move on such a queue
the "send retries" and leaving the main delivery queue free to process its stuff
Bill48105 wrote: - At some point possibly ability to have multiple queues but I'd do all of the above 1st & see how much improvement there is 1st before I spent the effort. (I already have ideas how I'd do it but have a ton of other more important things I'd like to get to.)
Since we're at queues, I'm attaching some "C" code which implements a quite "smart" queue
it's not mine... I grabbed that code years ago from some ML (don't even remember where), all
I can say is that I used/use it a lot and it works like a charm, the main advantage is that the
code allows you to "push" (enqueue) elements w/o holding a lock, and this dramatically speeds things up, the lock is just hold when "popping" (dequeueing) elements and since usually such an operation is "atomic" that doesn't impact so much on performances... I hope the code may be
useful to you as it was and is to me

HTH
Attachments
queue.zip
Non-Blocking queue
(1.97 KiB) Downloaded 401 times

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Better queue processing

Post by Bill48105 » 2010-10-22 17:24

Oh yeah Obi, there are lots of ways to do things but I'm trying to work within the current code with reasonably quick-fixes that help without any major overhauls.

I don't think the size of the message is known at the points I'm working at so not sure adjusting queue by size will be feasible with quick-fixes..

I'll check out the code but again I doubt anyone is motivated enough or has the time to do overhaul so tweaking & being smarter with what we have is the plan Stan. :D

For now I have made this minor tweak & am testing:
For 1st 3 delivery attempts, the delay is hard-coded at 5 min (i picked a value that seemed reasonable for greylisting & help our cause). Then after that messages are retried according to the normal schedule. IMO this will help a ton because people often set retry real low because of greylisting but then for servers with issues it causes way too many tries to occur tying up the queue. So far in my testing it works well.

Code: Select all

"DEBUG" 1268 "2010-10-22 09:57:58.293" "SD::_RescheduleDelivery"
"SMTPC" 1268 0 "2010-10-22 09:57:58.293" "APP" "SMTPDeliverer - Message 4478: Message could not be delivered. Greylisting? Scheduling it for quick retry in 5 min."
"DEBUG" 1268 "2010-10-22 09:57:58.293" "Message rescheduled for later quick delivery. (Greylisting?)"
"SMTPC" 1268 0 "2010-10-22 09:57:58.293" "APP" "SMTPDeliverer - Message 4478: Message delivery thread completed."

here was next try: (note retries are set to 60 minutes) 
"SMTPC" 1848 0 "2010-10-22 10:03:43.029" "APP" "SMTPDeliverer - Message 4478: Message could not be delivered. Greylisting? Scheduling it for quick retry in 5 min."

after 3rd 5-min-delay retry it goes back to normal retry schedule (60 minutes by default): "SMTPC" 1848 0 "2010-10-22 10:15:44.496" "APP" "SMTPDeliverer - Message 4478: Message could not be delivered. Scheduling it for later delivery."

Now queue shows next try in 60 minutes per user settings.
Anyway that change took me less time to write than this post & should have decent results assuming people can now set retry a lot higher without fear of huge greylisting delays. I just need to decide if that quick-retry over-ride should be applied to routes (which have their own retry schedule) and if there is reason to put effort into making this quick-retry optional or user-definable or not..
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

ObiWan
Senior user
Senior user
Posts: 278
Joined: 2010-07-21 14:30
Location: Halfway between Germany and Egypt

Re: Better queue processing

Post by ObiWan » 2010-10-22 17:42

Bill48105 wrote: Anyway that change took me less time to write than this post & should have decent results assuming people can now set retry a lot higher without fear of huge greylisting delays. I just need to decide if that quick-retry over-ride should be applied to routes (which have their own retry schedule) and if there is reason to put effort into making this quick-retry optional or user-definable or not..
Sorry for cutting so much, I see your point and understand it; about the delay...
some docs floating around use 5 minutes as the "example" for the GL delay so
if you set your retry to exactly 5 minutes you may hit the GL a couple times
before being able to deliver the message; imVVVHo it would be a good idea
to increase the delay to 6 minutes and then, to avoid "collisions" add a random
value ranging from 30 to 60 seconds to the "base value" (after the first attemtp)

Notice that the default values shown here aren't generally used instead the
general setup uses a delay ranging from 0 (just reject the first, immediately
allow the second) to 5 minutes so, again, 6 minutes may be a viable value

Also, and since we're at graylisting, I think you may be interested in the
attached critter, it's a simple "vbs" which loads into the hMailServer GL
whitelist a number of IPs known to "react badly" to graylisting ;-) the
script usage is pretty straightforward and the comments at the top should
help understanding how that list has been built and where to look for
updates ;)

HTH
Attachments
glwhite.zip
(3.68 KiB) Downloaded 394 times

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Better queue processing

Post by Bill48105 » 2010-10-22 18:16

Potential new 5.4 hmailserver.ini values: (as in subject to change without notice)
[Settings]
QuickRetries=3
'(default=0 or disabled. Anything <0 is disabled. Common sense alert - Set less than main retries setting..)
QuickRetriesMinutes=2
'(default=6. Common sense alert - Set less than retry minutes..)
("Default" means if not defined as in normal install.. If at some point it could default to enabled and moved to GUI)

So far working fine in my testing. Again this is NOT in current 5.3.x hmail and *might* be in 5.4 alpha.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

ObiWan
Senior user
Senior user
Posts: 278
Joined: 2010-07-21 14:30
Location: Halfway between Germany and Egypt

Re: Better queue processing

Post by ObiWan » 2010-10-22 18:32

Bill48105 wrote:
[Settings]
QuickRetries=3
QuickRetriesMinutes=2
A bit of warning Bill, some spamfilters (e.g. ASSP - but others do too) will "block"
you for a given amount of time in case you retry too-fast on a GL reply, this
means that retrying 3 times with a 2 minutes interval may result in the sending
(your) IP being "penalized" so that any further connection attempt to the
receiving MX (filter) will be refused for some "long" interval of time

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Better queue processing

Post by Bill48105 » 2010-10-22 18:40

Yeah I know Obi, I use ASSP myself. I wasn't suggesting anyone use 2 minutes.. Just showing an example of the setting. And that is partly why defaults are what they are.. Disabled & 6 minutes for defaults (if it were enabled) should be reasonably safe. Like ANY mail server setting, it is up to admin to decide best settings for their needs using a bit of common sense as well. ;)
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Better queue processing

Post by Bill48105 » 2010-10-22 20:11

OK I got wild hair.. Added some randomness to retry delays.
"SMTPC" 3184 0 "2010-10-22 13:45:34.720" "APP" "SMTPDeliverer - Message 4528: Message could not be delivered. Greylisting? Scheduling it for quick retry 2 of 3 in 10 minutes."
"SMTPC" 3432 0 "2010-10-22 13:47:19.761" "APP" "SMTPDeliverer - Message 4529: Message could not be delivered. Greylisting? Scheduling it for quick retry 2 of 3 in 6 minutes."
"SMTPC" 3576 0 "2010-10-22 13:47:34.763" "APP" "SMTPDeliverer - Message 4530: Message could not be delivered. Greylisting? Scheduling it for quick retry 2 of 3 in 14 minutes."
"SMTPC" 1468 0 "2010-10-22 13:54:44.521" "APP" "SMTPDeliverer - Message 4491: Message could not be delivered. Scheduling it for later delivery in 24 minutes.."
"SMTPC" 3184 0 "2010-10-22 13:54:59.512" "APP" "SMTPDeliverer - Message 4509: Message could not be delivered. Scheduling it for later delivery in 27 minutes.."
"SMTPC" 1532 0 "2010-10-22 13:55:14.524" "APP" "SMTPDeliverer - Message 4522: Message could not be delivered. Scheduling it for later delivery in 25 minutes.."
No delay is added to 'fresh' emails, they are delivered immediately as before (for now). The idea is that it should help create gaps in the queue to allow room for 'fresh' messages to be added & still have a chance to get delivered. It will default to 0 (off) like the other new options & be enabled in INI if someone desires to use it. Will post up that info later once it's in place but figured I'd give a preview. :D
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: Better queue processing

Post by tBB » 2010-10-22 20:23

ObiWan wrote:Since we're at queues, I'm attaching some "C" code which implements a quite "smart" queue
it's not mine... I grabbed that code years ago from some ML (don't even remember where), all
I can say is that I used/use it a lot and it works like a charm, the main advantage is that the
code allows you to "push" (enqueue) elements w/o holding a lock, and this dramatically speeds things up, the lock is just hold when "popping" (dequeueing) elements and since usually such an operation is "atomic" that doesn't impact so much on performances... I hope the code may be useful to you as it was and is to me
HTH
Nifty one! I don't know the related code in hMS but that implementation looks quite good and seems not that hard to implement.

Thanks and best regards,

Nico

ObiWan
Senior user
Senior user
Posts: 278
Joined: 2010-07-21 14:30
Location: Halfway between Germany and Egypt

Re: Better queue processing

Post by ObiWan » 2010-10-22 22:12

tBB wrote:
ObiWan wrote:Since we're at queues, I'm attaching some "C" code which implements a quite "smart" queue
it's not mine... I grabbed that code years ago from some ML (don't even remember where), all
I can say is that I used/use it a lot and it works like a charm, the main advantage is that the
code allows you to "push" (enqueue) elements w/o holding a lock, and this dramatically speeds things up, the lock is just hold when "popping" (dequeueing) elements and since usually such an operation is "atomic" that doesn't impact so much on performances... I hope the code may be useful to you as it was and is to me
HTH
Nifty one! I don't know the related code in hMS but that implementation looks quite good and seems not that hard to implement.

Thanks and best regards,

Nico
Thanks, hope it will be of help; just in case have a look at an example of usage, this is some code I wrote which puts that
queue code ad *good* use :) (I think you'll realize by yourself what that critter is :D)

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Better queue processing

Post by Bill48105 » 2010-10-22 23:45

Can be "World's Best" queue implementation but until someone has the time or inclination to do some major rewriting of hmail it's just gibberish really. ;)

Anyway, I got the "Quick Retry" & "Queue Retry Randomness Factor" code added in & been testing all day. Doesn't appear I borked anything. hehe

I committed my changes to SVN so we'll see what martin thinks. Once 5.4 alpha is out people can enable the new options in the INI and see how they work. I do believe those changes will help out in a lot of situations and other things can be tweaked later as well. If all else fails people can ignore the options if they want as they are not enabled by default but from my tests so far so good.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: Better queue processing

Post by tBB » 2010-10-23 01:11

ObiWan wrote:Thanks, hope it will be of help; just in case have a look at an example of usage, this is some code I wrote which puts that
queue code ad *good* use :) (I think you'll realize by yourself what that critter is :D)
Yeah, I've read about that at the SpamAssassin Wiki some time ago and it sounded reasonable but nice that someone went ahead and wrote it. Not sure if I'm going to use it because I'm lazy and also found some good ways to identify and deal with spambots at the SMTP level but in any case it's now in my toolbox along with the queue source :)
Bill48105 wrote:Can be "World's Best" queue implementation but until someone has the time or inclination to do some major rewriting of hmail it's just gibberish really. ;)
Many features and improvements of OSS projects start with what you call "gibberish". If a project maintainer would call good code snippets along with example cases which were provided by a user the same there would be hardly anybody contributing his "gibberish" to the project. ;)

Best regards,

Nico

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Better queue processing

Post by Bill48105 » 2010-10-23 03:24

tBB wrote:Many features and improvements of OSS projects start with what you call "gibberish". If a project maintainer would call good code snippets along with example cases which were provided by a user the same there would be hardly anybody contributing his "gibberish" to the project. ;)

Best regards,

Nico
If you got the inclination download the source & integrate that code snippet. ;)

Point was not to dis the snippet or the idea as if there was something wrong with it but it would be too time consuming and huge risk of really breaking something to bother trying. IMO anyways, but if anyone thinks otherwise it's open now so have at it. :D Until then I'm using my time to improve what's there in my spare time under my choice of priorities and sharing my efforts with everyone rather than rewriting the book that's essentially 99.x% done as far as I'm concerned.
Cheers
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: Better queue processing

Post by tBB » 2010-10-23 10:00

Bill48105 wrote:If you got the inclination download the source & integrate that code snippet. ;)

Point was not to dis the snippet or the idea as if there was something wrong with it but it would be too time consuming and huge risk of really breaking something to bother trying. IMO anyways, but if anyone thinks otherwise it's open now so have at it. :D
When the source of the new hMS version is officially released I'll surely have a look at it. Otherwise, someone said:
"I've been holding off to avoid delaying 5.4 release any further"
Oh wait, that was you ;)

Bill, I know what you mean but a implement-it-or-shut-up-style response is not always appropriate (there are definitely cases where it is though :D ) However, I think it would be a good idea to create a closed developer section of the forum where interested people can discuss new features and ways to implement them. It should be closed because sooner or later people would start posting OT stuff like in the current 'Development & alpha discussions' section.

Best regards,

Nico

ObiWan
Senior user
Senior user
Posts: 278
Joined: 2010-07-21 14:30
Location: Halfway between Germany and Egypt

Re: Better queue processing

Post by ObiWan » 2010-10-23 10:03

tBB wrote:
ObiWan wrote:Thanks, hope it will be of help; just in case have a look at an example of usage, this is some code I wrote which puts that
queue code ad *good* use :) (I think you'll realize by yourself what that critter is :D)
Yeah, I've read about that at the SpamAssassin Wiki some time ago and it sounded reasonable but nice that someone went ahead and wrote it. Not sure if I'm going to use it because I'm lazy and also found some good ways to identify and deal with spambots at the SMTP level but in any case it's now in my toolbox along with the queue source :)
Well... it's not just about "bots" :) see, time ago I had to deal with a mailserver really overwhelmed by
incoming critters trying to fire out their junk, so while seeking for a solution I stumbled upon the so
called "MX sandwich" idea... the problem was that all I had available were Win32 boxes and there
wasn't any decent "fake MX" critter running on such a platform, so I decided to jump the gun and
put together my own one, then, since I was at it, I expanded the idea letting the critter "emulate"
a mailserver (but still rejecting messages with a tempfail) and then gathering some infos about the
connecting IPs and logging them for further "data gathering and analysis" ;-)

As for the setup, it's quite easy, let's say you have a domain with a single MX (just to keep things plain)
so you'll have something like this in DNS (assuming the domain is example.com)

mail A 192.0.2.11

@ MX 10 mail.example.com

now, to activate the sandwich all you'll need are a couple more IPs, then you'll change your
DNS zone to look like

mx0 A 192.0.2.10
mail A 192.0.2.11
mx9 A 192.0.2.12

@ MX 10 mx0.example.com
@ MX 20 mail.example.com
@ MX 50 mx9.example.com

then you ensure that port 25 on 192.0.2.10 (mx0) will be filtered (firewalled) so that
it won't answer to connection attemtps and go on installing fakemx to listen on port
25 on 192.0.2.12 - that's all :) it will also work in case you have multiple MX, just set
up the first and the last as shown, then there are several tweaks, like inserting some
fakemx between various valid MX records or "rotating" the records, but even the
simple setup shown above will work ... and the logs will be interesting, believe me :)

ObiWan
Senior user
Senior user
Posts: 278
Joined: 2010-07-21 14:30
Location: Halfway between Germany and Egypt

Re: Better queue processing

Post by ObiWan » 2010-10-23 10:06

Bill48105 wrote: Can be "World's Best" queue implementation but until someone has the time or inclination to do some major rewriting of hmail it's just gibberish really. ;)
Up to you, never wrote "you must implement it" nor trying to learn you how to write code; I'm not that kind of folk
I just thought that, since I had that code working quite well and since we were discussing about queues and how
to speed them up, it may have been interesting having a look at that code, that's all; then, by the way use it or
discard it or whatever, no problem on this side

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Better queue processing

Post by Bill48105 » 2010-10-23 17:03

Wow I love how simple non confrontational statements get turned around into some huge thing. Not sure if it is the "Forum" format or what but it is annoying as hell that's for sure. Sometimes it feels like the text is getting changed on it's way to someone else's screen in some sort of language translator or something..

tBB/Nico: Source has been available for awhile. Link was posted by martin so can't get more official than that. ;) What is not official is the release of the compiled source as binary/installer. While I realize your point, if martin released 5.4 today your compiled copy from the SVN as of now would be functionally identical. The delay comment was when martin told me he'd be releasing 5.4 alpha but then was told it wasn't going to happen until later so I took the time to do more. Obviously you had no way of knowing but more reason to not make your statement. Please don't make it more than it is. I didn't delay anything..

ObiWan: I'm sorry if I hurt your feelings or whatever is source of your posts. I looked at the code you posted. I said thank you and that I'll leave it for someone else. Please don't read or make more into it than that.

I really hope this "discussion" is over.. The topic is better queue processing, I posted up what I spent time on related to it & it's been all twisted & turned and totally off-topic with personal stuff. In the time it took for 3 of us to make all these posts here the queue could have been re-written twice. lol

Anyway, my hope is that with the Quick Retry, Queue Randomness Factor and MX/A Limit Factor changes I made, some of the headaches of the current queue will be reduced but only real-world testing will tell.
Peace.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

User avatar
tBB
Senior user
Senior user
Posts: 268
Joined: 2009-04-17 18:10
Location: The land of Beer and Sauerkraut!
Contact:

Re: Better queue processing

Post by tBB » 2010-10-23 19:46

Bill48105 wrote:tBB/Nico: Source has been available for awhile. Link was posted by martin so can't get more official than that. ;) What is not official is the release of the compiled source as binary/installer. While I realize your point, if martin released 5.4 today your compiled copy from the SVN as of now would be functionally identical. The delay comment was when martin told me he'd be releasing 5.4 alpha but then was told it wasn't going to happen until later so I took the time to do more. Obviously you had no way of knowing but more reason to not make your statement. Please don't make it more than it is. I didn't delay anything..Bill
Calm down mate. I wasn't implying at all that you're delaying the release. What I meant is that I'm following your statement to not propose changes until the new version/source gets released. I didn't notice that Martin posted the link to the svn because it wasn't under the announcement section nor at the main page which is what I would call official.

As for the language translator, I don't think that the term "gibberish" means anything different in English, German or any other language on the planet and at least in my book it qualifies as confrontational.

Best regards,

Nico

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Better queue processing

Post by Bill48105 » 2010-10-23 20:08

I am calm. Last I checked we don't probes attached to us taking our vitals.. ;)

Your response on the gibberish stuff is perfect example of what I am talking about because: A. It was not a jab or directed toward any person and B. It was not intended to be confrontational to anyone. Clearly there was miscommunication or misunderstanding of what I said but interpret how you will as it's not worth the effort to try & clarify as likely that would just be mixed up as well and be an endless circle.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

ObiWan
Senior user
Senior user
Posts: 278
Joined: 2010-07-21 14:30
Location: Halfway between Germany and Egypt

Re: Better queue processing

Post by ObiWan » 2010-10-24 12:21

Bill48105 wrote:
ObiWan: I'm sorry if I hurt your feelings or whatever is source of your posts. I looked at the code you posted. I said thank you and that I'll leave it for someone else. Please don't read or make more into it than that.
No offense/hurt taken, don't worry, I was just trying to make it clear that we're just discussing about development, it isn't
imHo some kind of competition as who "writes/proposes better code", just a general effort to try helping the project be
better and better and this sometimes includes code examples/snippets and discussions like this one; but please let's
avoid the usual "developers competitive spirit", it's unneeded, won't bring any advantage and I don't think this place
is about such kind of stuff as I see it we're all here to try making hMailServer better and better so let's keep focused
on that, please :)

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Better queue processing

Post by Bill48105 » 2010-10-24 18:00

GAME ON MAN!! LOL Just kidding! I don't view it as a competition at all and since I'm not a developer (it's a hobby of mine) I have no developer's competitive spirit I suppose. Heck not even sure what that is or what the purpose would be.. I figured we were all on the same team. ;) I've said it before & I'll say it again, please don't read too much into things I say especially in the forum as too often it seems I am misunderstood.. I am here to help anyone who needs it when I can and help improve hmail in my spare time and have no desire for competition or drama. I am one to work on many small things rather than one big one and that's what I am doing. Focusing mainly on my personal needs but things others have asked for as well which will be apparent when 5.4 comes out as there is some of both but hopefully all turn out useful to many. :)
Cheers
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Better queue processing

Post by Bill48105 » 2010-11-02 06:49

FYI: This could be considered done in 5.4 alpha:
http://www.hmailserver.com/forum/viewto ... 12#p117612

Partial list relating to queue improvements in 5.4 alpha:

Code: Select all

QuickRetries=3
; This key tells hmail to over-ride default wait/retry schedule for 1st defined # of tries which should
; allow one to set retry delay higher while not causing greylisted messages to be stuck too long.
; This defaults to 0 or disabled if not defined

QuickRetriesMinutes=7
; Defines # of minutes to delay for each retry if QuickRetries is enabled otherwise normal delivery schedule is used
; This defaults to 0 or disabled

QueueRandomnessMinutes=7
; This key defines how many minutes to use for 'QuickRetries' that over-rides the default retry schedule.
; This does not apply to NEW messages as they are still queued for immediate delivery as before.
; This defaults to 0 or disabled

MXTriesFactor=5
; This key defines a multiplier to determine how many MX/A server results to try per retry.
; For example: If set to 2 & the recipient has 11 servers, on 1st try, 1 x MXTriesFactor or 2 are tried.
; On 2nd try, 2 x MXTriesFactor or 4 are tried. On 3rd 6 are tried & so on. Eventually all 11
; would be tried if the message was not successively delivered enough times.
; This defaults to disabled if not defined
I realize since it is alpha perhaps it will be tough to get a real sense of how much difference these changes make but I suspect they should help quite a bit and always room for improvement as well. It would be great to get some feedback on any results people are finding to know if better queue processing was achieved. :)
Thx
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

ObiWan
Senior user
Senior user
Posts: 278
Joined: 2010-07-21 14:30
Location: Halfway between Germany and Egypt

Re: Better queue processing

Post by ObiWan » 2010-11-02 09:40

Bill48105 wrote:FYI: This could be considered done in 5.4 alpha:
http://www.hmailserver.com/forum/viewto ... 12#p117612

[..snipped...]

I realize since it is alpha perhaps it will be tough to get a real sense of how much difference these changes make but I suspect they should help quite a bit and always room for improvement as well. It would be great to get some feedback on any results people are finding to know if better queue processing was achieved. :)
Thanks Bill !

Post Reply