Selective Greylisting

Use this forum if you have problems with a hMailServer script, such as hMailServer WebAdmin or code in an event handler.
Post Reply
Hotlanta
Normal user
Normal user
Posts: 46
Joined: 2006-12-04 20:46

Selective Greylisting

Post by Hotlanta » 2006-12-08 07:42

Hi everyone.

I'm trying to figure out a way to selectively greylist incoming mail that I think may be spam. It seems to me that greylisting is a very good way to handle mail that has been marked as spam since rejecting it can cause problems with losing mail that has generated a false positive, and delivering all mail marked as spam just fills up user mailboxes with trash. On the other hand, delaying mail that I don't think is spam seems like a great way to annoy people.

That said, I'm trying to come up with a scripting strategy to solve this problem, but I don't know enough about how things work internally to figure this out on my own. Some ideas I'm looking at include:

1) Turn off automatic greylisting, and bounce the possible spam myself while implimenting my own whitelisting method. Yuck! I could probably use the data structures already in place, but what a mess to implement from my perspective (I'm sure Martin could do it easily!).

2) Turn on automatic greylisting, but whitelist any email that doesn't look like spam as it comes through. This would be pretty easy to do, but I'm not sure if this would work since the greylisting rejection probably takes place before the evenhandler.vbs subroutines are called.

3) Some other methodology that I'm not aware of that Martin may share that allows me to submit an email to be greylisted.

I look forward to any suggestions.

Regards,

John

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2006-12-11 17:40

2)
Grey listing takes place beforethe email has been transmitted. hMailServer only knows the senders IP address and email address when grey listing is performed. So I doubt that's possible.
that allows me to submit an email to be greylisted.
Not exactly sure what you mean with that. :-\

Anyway, in hMailServer 4.4, you will be able to whitelist senders based on their domain, email address and IP address.

Hotlanta
Normal user
Normal user
Posts: 46
Joined: 2006-12-04 20:46

Post by Hotlanta » 2006-12-12 16:26

What I want to be able to do is to return mail temporarily based on whether or not I suspect the mail is spam. In other words, I don't want to return everything as the greylisting works now, I want to be selective about what I send back. Is there a way for me to do this manually without using the internal greylisting system? Can I set the return code? If I can do that, then I can set the whitelist at the same time.

Regards,

John

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2006-12-12 16:29

> based on whether or not I suspect the mail is spam

What method do you plan to use to determine whether the email is spam or not?

mbreitba
Senior user
Senior user
Posts: 340
Joined: 2006-04-14 22:25

Post by mbreitba » 2006-12-12 19:35

They way greylisting works, it sends an error message before any of the message data has been transmitted. My guess is that you want to use ASSP or SpamAssassin to try and determine if the message is SPAM. If this is the case, then you'll have to accept the entire message before you can return an error code. This would be far outside the way that Greylisting is intended to work, and would most likely violate RFC's in some way.

Hotlanta
Normal user
Normal user
Posts: 46
Joined: 2006-12-04 20:46

Post by Hotlanta » 2006-12-12 19:45

mbreitba wrote:They way greylisting works, it sends an error message before any of the message data has been transmitted. My guess is that you want to use ASSP or SpamAssassin to try and determine if the message is SPAM. If this is the case, then you'll have to accept the entire message before you can return an error code. This would be far outside the way that Greylisting is intended to work, and would most likely violate RFC's in some way.
Actually, I have a few other methods that I'm testing.

So returning the error code after getting the message isn't possible to do? What if the server was busy in the middle of the transmission of the message? Wouldn't it return the same error code? It seems to me that any RFC would have to account for that possibility.

Hotlanta
Normal user
Normal user
Posts: 46
Joined: 2006-12-04 20:46

Post by Hotlanta » 2006-12-13 17:11

It appears to me from reading the specs on Greylisting that if I had a way to return "451 4.7.1 Please try again later" at the onClientConnect stage when the message hasn't been accepted, that manual greylisting could be achieved. Is there a way we can set the return codes from onClientConnect?

Hotlanta
Normal user
Normal user
Posts: 46
Joined: 2006-12-04 20:46

The Usefulness of Selective Greylisting

Post by Hotlanta » 2006-12-13 17:53

Ok, so why is selective greylisting useful? First, it appears from reading the posts in this forum that greylisting indiscriminately can cause loss of email and as well as complaints by end users about slow mail. Although we may set our return interval to only 5 minutes, the sender's server may not try again for several hours. End users have gotten spoiled and expect their mail to be delivered almost immediately. Doing business on the Internet is very fast, and waiting 4 hours to get an important email isn't going to make many people happy. We can't ignore this reality.

In addition, handling probable spam is tricky, too. There are basically two ways to handle suspected spam; you can bounce it with a reject message (and take the chance that important mail might not get through), or you can mark it as spam in the headers, and then make your users download and filter it with their email client (major slowdown in getting mail). Truth is, most users don't bother to check their bulk mail very often, so if they're not specifically looking for that important email, it may still get ignored. We've all had that case where we found out a week later that someone emailed us but we "ignored" their mail, only to then find it in our bulk mail folder.

I would like to take the email that I suspect as to be spam, and greylist it, but allow the other valid email through immediately. That seems to be an intelligent way to deal with this problem. Greylisting probable spam is less likely to delay or lose valid email while getting rid of most of the spam.

By allowing for selective greylisting, we also allow for user interfaces that can be built for automatic whitelisting. For example, on a user by user basis, we would be able to look at their outbound email and build automatic whitelists for reply mail coming back from specific email addresses. We could also allow end users to whitelist their address books. This would be a very nice feature to offer end users since it could be done automatically from their perspective.

What happens in the middle of an SMTP transaction if the receiving server is suddenly busy? Suppose the server allows the connection, gets the message, but instead of either doing a hard rejection (onAcceptMessage allows for this), we have a "451 4.7.1 Please try again later" message? Is there actually a prohibition in the RFC from doing this at this stage? If so, what happens if the receiving server doesn't acknowledge reciept of the message or loses the connection? Does the sending server resend it? If that's the case, couldn't we do a "legitimate" greylisting at that point?

I have to believe there's a way to do this if we think about it.

Am I just missing something or does having programming control over greylisting and whitelisting make sense to anyone else, too?

John

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Post by ^DooM^ » 2006-12-13 18:22

I like what you are proposing and if it can be done then I am all for it but I feel that would take quite a bit of implementation.

Hows this for an idea which I think would be relitivley easy to implement. Adding an option to Check DNS Blacklists before greylisting. First instead of bouncing straight away check DNS blacklists. If the IP is on a blacklist then bounce with a try later but adding the usual records to greylisting triplets. If the mail comes back in and matches the triplets append the [SPAM] message to the topic and headers but pass through anyway.
Any emails that come through that do not match a blacklist would be passed with no delay and do virus scanning etc, then users can sort through Junk later.

I realise that doing it this way falls back on the DNS blacklists again which could also bring about false positives but at least this way you would still have some control over spam ( by your choice of blacklists ) without delaying that important email ( as long as their server isn't blacklisted and configured correctly that is )

doable? Can anyone see any possible flaws with this technique?

mbreitba
Senior user
Senior user
Posts: 340
Joined: 2006-04-14 22:25

Post by mbreitba » 2006-12-13 20:37

I know someone posted a flowchart style representation of how Hmail handles things - I can't find it right off hand, but does anyone know where it is? That'd answer some of our questions as to where stuff occurs (is greylisting before or after DNS checks, etc).

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Post by martin » 2006-12-17 20:40

Flow chart is available here:
http://img278.imageshack.us/img278/3862 ... tes3fo.png

Somewhat outdated though. I'm not the author of this one.

Edit: Started to create a new one which is more up to date:
http://download.hmailserver.com/dev/hMa ... t-v0.1.pdf
(yeah, I know it's ugly)

ewhitmor
Normal user
Normal user
Posts: 30
Joined: 2008-04-09 19:25

Re: Selective Greylisting

Post by ewhitmor » 2008-04-17 17:14

When i download the PDF flowchart adobe throws an error "Acrobat could not open because it is either not a supported file or because the file has been damaged". Anyone else having this problem?

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: Selective Greylisting

Post by ^DooM^ » 2008-04-17 20:04

I just get a 404
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

ewhitmor
Normal user
Normal user
Posts: 30
Joined: 2008-04-09 19:25

Re: Selective Greylisting

Post by ewhitmor » 2008-04-17 21:10

Martin,

can you repost the PDF?

Thx
Eric

User avatar
SorenR
Senior user
Senior user
Posts: 3220
Joined: 2006-08-21 15:38
Location: Denmark

Re: Selective Greylisting

Post by SorenR » 2008-04-17 21:56

SørenR.

“With age comes wisdom, but sometimes age comes alone.”
- Oscar Wilde

Post Reply