Greylisting - allow the host IP after success

Use this forum if you want to suggest a new feature to hMailServer. Before posting, please search the forum to confirm that it has not already been suggested.
Post Reply

Would you like this feature?

Yes
9
90%
No
1
10%
 
Total votes: 10

Shiloh
Normal user
Normal user
Posts: 163
Joined: 2006-04-14 00:00

Greylisting - allow the host IP after success

Post by Shiloh » 2008-06-17 20:55

After an IP has successfully gotten through greylisting, that IP should be whitelisted so it can bypass future greylisting delays. What hMailServer currently does is only allow the email through for the same triplet (To, From, IP). For example, bill@remotedomain.com might send an email to larry@localdomain.com. The To, From, and IP are stored as a triplet, and the remote email server is given a temp delay message. When the remote server tries again, the triplet is validated and the message is let through.

But when lisa@remotedomain.com tries to send to tom@localdomain.com, then greylisting will also delay that email until the remote server retries even though that same server has already successfully validated a different email message already. Once a given IP has successfully done a retry and gotten a message accepted through greylisting, it would be logical to assume that IP will be able to successfully get through greylisting each time. There is no reason to delay that remote IP with greylisting. The delay on the first email (from bill to larry) is expected, but the delay on the second email (lisa to tom) should be eliminated.

What I think we should do is automatically whitelist (within the greylisting system) the IP of the remote server on its first successful message triplet validation. This would eliminate most of the greylisting related delays.

rodolfor
Senior user
Senior user
Posts: 282
Joined: 2005-06-30 09:05
Location: Gubbio - Italy

Re: Greylisting - allow the host IP after success

Post by rodolfor » 2008-06-19 13:25

but...if the system allow an email from (i.e.) gmail.com, alla users of gmail.com are allowed too?
Hmailserver [lastversion] + MSSQL

Shiloh
Normal user
Normal user
Posts: 163
Joined: 2006-04-14 00:00

Re: Greylisting - allow the host IP after success

Post by Shiloh » 2008-06-19 20:37

rodolfor: Yes. Keep in mind that greylisting is not spam filtering. Greylisting is merely a way to check to see if the remote IP is an actual SMTP server. That is the only purpose of greylisting. It does this by forcing the remote IP to retry email deliveries. Once an IP has proven that it is in fact a real SMTP server capable of retrying email deliveries, then there is nothing additional that greylisting can do or needs to do. It is a safe bet that a remote SMTP server will always succeed at retrying the email if it can successfully handle that task even one time. There is no sensible reason to force a greylisting delay for each unique IP, From, and To triplet once the IP has had a successful retry of an email delivery. At that point in time, the IP should just be automatically added to the greylisting system's whitelist so that the IP does not need to go through the delay/retry process again.

This change would help reduce the delays often associated with greylisting. This change would not affect the amount of spam filtered by the greylisting system. Spammers who send millions of messages from an IP would still be stopped by greylisting, because those spammers would not be retrying the delayed emails.

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: Greylisting - allow the host IP after success

Post by ^DooM^ » 2008-06-19 22:52

I'm not so sure.

If spammer1 sends emails from abc@gmail.com and bcd@gmail.com and cde@gmail.com and def@gmail.com all from the same IP then I would want all of those to be delayed otherwise one gets delayed and the rest get through whereby I end up with a stackload more spam than if this wasn't implemented.

If this feature was implemented I would at the very least like an option to turn it off.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

User avatar
mattg
Moderator
Moderator
Posts: 20794
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Greylisting - allow the host IP after success

Post by mattg » 2008-06-20 02:37

Can this functionality be added using scripting?

http://www.hmailserver.com/documentatio ... eaddresses

Function Add()
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

Shiloh
Normal user
Normal user
Posts: 163
Joined: 2006-04-14 00:00

Re: Greylisting - allow the host IP after success

Post by Shiloh » 2008-06-20 20:59

Doom: If a spammer sends spam through gmail servers, then ALL of the spam is going to get through greylisting no matter what. The current behavior and the proposed change will both allow that spam through the greylisting system. This is because greylisting only purpose is to make sure the remote server is a real SMTP server capable of retrying email deliveries. My proposed change does not affect that issue. In the event that a spammer is sending spam through gmail, then it is up to other spam filtering technologies to stop those spams.

Additionally, most of us (including you) are already whitelisting gmail's servers within the greylisting system, because gmail has a pool of servers that handle SMTP sending duties. So gmail is a poor example. Since gmail is already whitelisted in the greylisting system, gmail does not get the delays anyway. My proposed change does not have any impact on gmail or any other service that is already whitelisted in the greylisting system.

The change I have proposed will not increase the amount of spam, but it will reduce many of the delays on legitimate email. The current implementation delays a lot of email that will eventually be let through anyway. We need to make some smart tweak to the greylisting system to try to eliminate delays that are obviously unneeded. If an IP has already successfully got even one message through the greylisting system on a retry, then it is quite safe to assume all of the email from that IP will get through the greylisting system. This is because the only function that greylisting does is delay email so the remote host has to prove that it is a real SMTP relay. Greylisting is not a spam filtering solution. It is just a way of filtering out delivery attempts from IP addresses that are not real SMTP servers.

Shiloh
Normal user
Normal user
Posts: 163
Joined: 2006-04-14 00:00

Re: Greylisting - allow the host IP after success

Post by Shiloh » 2008-06-20 21:14

mattg: I think this could be done using the API. The COM object contains an Add function for the GreyListingWhiteAddresses collection, so it is possible for us to add IP addresses to greylisting's whitelist dynamically from the VBS code. I think the OnAcceptMessage is executed after the greylisting. So I think code could be added in this section. The code would need to make sure the connection was not an SMTP AUTH connection (make sure oClient.Username=""), because those SMTP AUTH connections are client PCs instead of servers. No reason to add every client PC to greylisting's whitelist because client PCs will use SMTP AUTH anyway.

I think we can assume that any connection that makes it to the OnAcceptMessage routine without using SMTP AUTH is a server that has successfully gotten through the greylisting system. Maybe Martin will comment here on this assumption. Anyway, if this assumption is true, then we could grab oClient.IPAddress in the OnAcceptMessage routine and store the value into greylisting's whitelist using the COM API.

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: Greylisting - allow the host IP after success

Post by ^DooM^ » 2008-06-21 01:27

Ok gmail was a bad example but the greylisting triplet uses the from address, the to address and the IP address. currently as you well know all three of these have to match before it passes even after it has been sent once. If a spammer is altering his from address for every message which they frequently do from the same IP then every one of those messages will get through.

You say it is not an anti spam feature, I disagree. I think this is a great anti spam feature and I am guessing martin does as well as greylisting is listed under the antispam section of the administrator. Don't get me wrong here I see your point but I also see its flaw.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

GlenC
Senior user
Senior user
Posts: 680
Joined: 2004-08-17 23:31
Location: Santiago, Chile

Re: Greylisting - allow the host IP after success

Post by GlenC » 2008-06-21 04:43

I think Doom has a good point. I can imagine that some spammer, knowing this behavior about hMailserver could exploit it by retrying a few times. So I think maybe an option would be a better idea than making this standard behavior.

Shiloh
Normal user
Normal user
Posts: 163
Joined: 2006-04-14 00:00

Re: Greylisting - allow the host IP after success

Post by Shiloh » 2008-06-21 17:56

Doom: I do not want to change the triplet validation behavior. That should remain a three part object. What I am talking about is adding the IP to the whitelist automatically after a triplet has been successfully validated. Because if a remote IP is able to validate a triplet (successfully retry a delayed message) even one time, then it is safe to assume the IP could do that successfully each time.

A spammer should not be able to exploit this any differently than they currently can. If a spammer is sending from a million different FROM addresses on the same IP, the spam will still receive the temporary delay message (just like it does not). An entire triplet would still need to be matched in order to accept the email. The difference I am talking about is assuming the IP will always be able to validate a triplet if it is able to do it at least once. The only way a spammer could do that is to send a email, wait a while (several minutes), and then send the exact same email again. If a spammer is willing to do that, they can already get their spam through.

But I think I see what you are saying. "What if the spammer knows the feature exists on the email server and then sets up a the first message to send and then wait 30 minutes and send the same message again?" Is that the special case you are referring to? In that case, it would be possible to get through the greylist to cause some trouble. Maybe a solution would be to only whitelist the IP after X number of triplet validation successes from that IP. That would make it much more time consuming for the spammer to try to exploit the feature. What do you think of that idea?

GlenC
Senior user
Senior user
Posts: 680
Joined: 2004-08-17 23:31
Location: Santiago, Chile

Re: Greylisting - allow the host IP after success

Post by GlenC » 2008-06-21 23:31

Yes, that was the special case I was thinking about. I've seen the stats on your mail server, so I know you are looking to improve performance in any way you can. And this would probably help you out a lot. But, I think some might feel uncomfortable whitelisting in that way. That's why I think it might be better to make it an option as Doom suggested.

I'm sure Martin is in the background here thinking "GAH! another option!!"

Personally, given my low volume of mail, either way or none at all would be ok with me. I'm just playing Devil's Advocate here :twisted:

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: Greylisting - allow the host IP after success

Post by ^DooM^ » 2008-06-22 02:39

That is a nice compromise whitelisting an IP after X successful attempts through the greylist.

Perhaps another option to add to this is whitelist expire after X days option those that are automatically set. This would help stop returning spammers in the future just bypassing the greylist.

Whether this be a rolling limit as in X days from last successful access or a static from first bypass I am not sure. I guess it all Depends on how often a spammer sends from the same IP.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

rodolfor
Senior user
Senior user
Posts: 282
Joined: 2005-06-30 09:05
Location: Gubbio - Italy

Re: Greylisting - allow the host IP after success

Post by rodolfor » 2008-06-22 08:08

Why wait for X success ?
The purpose of graylist is to avoid connections from SMTP servers (not user) who does not retry automatically.
If one account is capable to traverse the graylist then all accounts of the same server are capable too.
Then, after ONE success, the SMTP server MUST be whitelisted.
I vote yes
Hmailserver [lastversion] + MSSQL

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: Greylisting - allow the host IP after success

Post by ^DooM^ » 2008-06-22 10:06

did you read this whole thread rudolfor because it doesn't sound like you have from your response.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

rodolfor
Senior user
Senior user
Posts: 282
Joined: 2005-06-30 09:05
Location: Gubbio - Italy

Re: Greylisting - allow the host IP after success

Post by rodolfor » 2008-06-22 10:17

""What if the spammer knows the feature exists on the email server and then sets up a the first message to send and then wait 30 minutes and send the same message again?"
...ops!
Hmailserver [lastversion] + MSSQL

Post Reply