Spammer has bypassed my greylist in build 245 -- How?

dloomer · Post by **dloomer** » 2006-11-09 18:58

Before getting into it, I think I might know the answer: the Return-Path header is set to an address at my own domain (or in fact, the same as the recipient the spam is directed to). And I would assume that hMail's greylist implementation would not put a user from its own domain in the greylist queue, correct?

Anyway, in case I'm barking up the wrong tree here, this is all the info that I believe is relevant:

First, the relevant parts of the header:

Code: Select all

From - Thu Nov 09 09:31:19 2006
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
Return-Path: <dave@######.org>
Received: from p54800396.dip0.t-ipconnect.de ([84.128.20.132])
	by mail.######.org
	with hMailServer ; Thu, 9 Nov 2006 09:31:17 -0600
Received: from cusdomain.iinet.net.au (port=12303 helo=wmsffqmhtu)
	by p54800396.dip0.t-ipconnect.de with smtp
	id 822-1Q66o-f7i
	for dave@######.org; Thu, 09 Nov 2006 16:30:59 +0100
Message-ID: <000901c70414$0b50f320$00afaa3c@wmsffqmhtu>
From: "Benjamin Phillips" <yncfgobske@dmecontractors.com.au>
To: dave@######.org

The IP is not in my whitelist (select * from hm_greylisting_whiteaddresses where whiteipaddress='84.128.20.132' reveals nothing; I only use dotted quad in my hm_greylisting_whiteaddresses table).

The IP is also has not appeared previously in my greylist (select * from hm_greylisting_triplets where glipaddress=1417680004 reveals nothing; unused records not removed until 1 day and this message was received just a couple hours ago).

Greylist active, 5 minutes to defer delivery attempts, 1 day before removing unused records, 72 to remove used records. Greylisting had been working with 100% success and no false positives before this spammer showed up.

From my log:

Code: Select all

"TCPIP"	3640	"2006-11-09 09:31:15.423"	"Created accept socket 2028 on listening socket 1232"
"SMTPD"	3640	4389	"2006-11-09 09:31:15.423"	"84.128.20.132"	"SENT: 220 mail.######.org ESMTP"
"SMTPD"	3640	4389	"2006-11-09 09:31:15.688"	"84.128.20.132"	"RECEIVED: EHLO p54800396.dip0.t-ipconnect.de"
"SMTPD"	3640	4389	"2006-11-09 09:31:15.688"	"84.128.20.132"	"SENT: 250-hmailserver[nl]250-SIZE[nl]250 AUTH LOGIN"
"TCPIP"	3640	"2006-11-09 09:31:15.907"	"Created accept socket 1476 on listening socket 1232"
"SMTPD"	3640	4390	"2006-11-09 09:31:15.923"	"84.128.20.132"	"SENT: 220 mail.######.org ESMTP"
"SMTPD"	3640	4389	"2006-11-09 09:31:15.923"	"84.128.20.132"	"RECEIVED: MAIL FROM:<dave@######.net>"
"TCPIP"	3640	"2006-11-09 09:31:16.126"	"DNS - INAddr lookup: 132.20.128.84.sbl-xbl.spamhaus.org"
"SMTPD"	1968	4390	"2006-11-09 09:31:16.157"	"84.128.20.132"	"RECEIVED: EHLO p54800396.dip0.t-ipconnect.de"
"SMTPD"	1968	4390	"2006-11-09 09:31:16.157"	"84.128.20.132"	"SENT: 250-hmailserver[nl]250-SIZE[nl]250 AUTH LOGIN"
"TCPIP"	3640	"2006-11-09 09:31:16.204"	"DNSResolver - INAddr Lookup result for 132.20.128.84.sbl-xbl.spamhaus.org: 0 responses"
"TCPIP"	3640	"2006-11-09 09:31:16.204"	"DNS - INAddr lookup: 132.20.128.84.relays.ordb.org"
"TCPIP"	3640	"2006-11-09 09:31:16.360"	"DNSResolver - INAddr Lookup result for 132.20.128.84.relays.ordb.org: 0 responses"
"TCPIP"	3640	"2006-11-09 09:31:16.360"	"DNS - INAddr lookup: 132.20.128.84.bl.spamcop.net"
"SMTPD"	1968	4390	"2006-11-09 09:31:16.423"	"84.128.20.132"	"RECEIVED: MAIL FROM:<dave@######.org>"
"TCPIP"	1968	"2006-11-09 09:31:16.423"	"DNS - INAddr lookup: 132.20.128.84.sbl-xbl.spamhaus.org"
"TCPIP"	1968	"2006-11-09 09:31:16.423"	"DNSResolver - INAddr Lookup result for 132.20.128.84.sbl-xbl.spamhaus.org: 0 responses"
"TCPIP"	1968	"2006-11-09 09:31:16.423"	"DNS - INAddr lookup: 132.20.128.84.relays.ordb.org"
"TCPIP"	1968	"2006-11-09 09:31:16.423"	"DNSResolver - INAddr Lookup result for 132.20.128.84.relays.ordb.org: 0 responses"
"TCPIP"	1968	"2006-11-09 09:31:16.423"	"DNS - INAddr lookup: 132.20.128.84.bl.spamcop.net"
"TCPIP"	3640	"2006-11-09 09:31:16.438"	"DNSResolver - INAddr Lookup result for 132.20.128.84.bl.spamcop.net: 0 responses"
"SMTPD"	3640	4389	"2006-11-09 09:31:16.454"	"84.128.20.132"	"SENT: 250 OK"
"TCPIP"	1968	"2006-11-09 09:31:16.516"	"DNSResolver - INAddr Lookup result for 132.20.128.84.bl.spamcop.net: 0 responses"
"SMTPD"	1968	4390	"2006-11-09 09:31:16.516"	"84.128.20.132"	"SENT: 250 OK"
"SMTPD"	3640	4389	"2006-11-09 09:31:16.688"	"84.128.20.132"	"RECEIVED: RCPT TO:<dave@######.net>"
"SMTPD"	3640	4389	"2006-11-09 09:31:16.688"	"84.128.20.132"	"SENT: 250 OK"
"SMTPD"	1968	4390	"2006-11-09 09:31:16.735"	"84.128.20.132"	"RECEIVED: RCPT TO:<dave@######.org>"
"SMTPD"	1968	4390	"2006-11-09 09:31:16.751"	"84.128.20.132"	"SENT: 250 OK"
"SMTPD"	3640	4389	"2006-11-09 09:31:16.985"	"84.128.20.132"	"RECEIVED: DATA"
"SMTPD"	3640	4389	"2006-11-09 09:31:16.985"	"84.128.20.132"	"SENT: 354 OK, send."
"SMTPD"	1968	4390	"2006-11-09 09:31:17.001"	"84.128.20.132"	"RECEIVED: DATA"
"SMTPD"	1968	4390	"2006-11-09 09:31:17.016"	"84.128.20.132"	"SENT: 354 OK, send."
"SMTPD"	3640	4389	"2006-11-09 09:31:17.579"	"84.128.20.132"	"SENT: 250 Queued (0.594 seconds)"
"APPLICATION"	1896	"2006-11-09 09:31:17.595"	"SMTPDeliverer - Message 45639: Delivering message from dave@######.net to dave@######.org. File: F:\hMailServer\{36B61447-900A-4B65-B0FE-0F53F50ED21C}.eml"
"SMTPD"	1968	4390	"2006-11-09 09:31:17.845"	"84.128.20.132"	"SENT: 250 Queued (0.828 seconds)"
"APPLICATION"	1896	"2006-11-09 09:31:17.845"	"SMTPDeliverer - Message 45639: Message delivery thread completed."
"APPLICATION"	1896	"2006-11-09 09:31:17.860"	"SMTPDeliverer -

Is this as designed?

dloomer · Post by **dloomer** » 2006-11-09 23:28

Here's an update: Looking at my logs for the three days since installing build 245 (previously I had 199 and thus no greylisting), spammers have sent about 15 e-mails impersonating me in the From envelope header. Of these, 10 were rejected by SpamCop or other blacklist, and the remainder all passed through, no questions asked. No "450 Please try again later" (which I do see from all other non-whitelisted first-time senders) or anything in the logs for these. I'm not sure if the solution is so easy as putting all envelope From addresses from non-whitelisted IPs in the greylist queue (regardless of "friendly" domain name or not), but from the looks of this based on what understanding I have of the problem, there is a very easily exploitable loophole here. Or maybe I'm missing something...

Post by **martin** » 2006-11-10 01:48

Have you set up SPF records for your domain? And have you enabled SPF in hMailServer?

dloomer · Post by **dloomer** » 2006-11-10 01:59

Yes, and yes.

If I enter the Return-path e-mail along with the offending IP address at http://www.dnsstuff.com/pages/spf.htm, I get a response of SOFTFAIL.

Here is my SPF record:

v=spf1 ptr ptr:usinternet.com include:authsmtp.com ~all

"Use SPF" is checked and has always been checked in the General tab of my Spam Protection settings.

But you're right, SPF should prevent this -- and to be honest, should be a requirement in order to effectively utilize greylisting if you're a mail admin (not sure if you were planning on having hMail warn the user if they set things up otherwise). Why it isn't working for me, I'm not sure.

Post by **martin** » 2006-11-10 02:03

No, SPF should not prevent this.

If you read the documentation, you'll see that hMailServer allows soft-fail email to go through. The SPF record needs to be of type fail to work.

dloomer · Post by **dloomer** » 2006-11-10 02:11

I'll try setting my SPF record to -all instead of ~all, even though guides I've read online have advised against this. Most of the well-known domains I've checked have seemed to use ~all (or even ?all) as well.

It seems then that using SPF with -all on your own DNS would be a requirement in order for greylisting to work 100% effectively, right? As long as some people (either out of necessity or ignorance) leave theirs at ~all, it seems that spammers will be able to defeat greylisting just by imitating the domain they are sending to, and once they catch on to that anyone left with a ~all record will be left back at square one, if I am understanding this correctly.

Or could hMail be optionally configured to check all sender domains against the greylist table, without skipping mail that is (apparently) from your own domain? You could then whitelist your own IP addresses to prevent any delays.

dloomer · Post by **dloomer** » 2006-11-10 02:26

I suppose another solution might be (note that I'm new at this, so pardon any naivete or idealism): The greylist algorithm first checks the domain in the From envelope header; if it is a "friendly" domain, don't check against the greylist as long as SPF passes. If SPF returns SOFTFAIL or worse, and the IP is not whitelisted, then continue greylist logic.

Again, I'm very humble about my knowledge on this, but figured I'd throw it by you anyway.

cveillon · Post by **cveillon** » 2006-11-11 02:38

I believe greylisting only looks for a three paired match. the entire from address, the sending server IP address, and the recipient address. Greylisting doesn't look at just the domain name in the 'from' address. It's looking for a complete match of all three things as it's criteria. I also believe that greylisting runs separately from SPF checking and the two are not connected.

Anyone jump in if they know differently!

with best regards,
Chuck

dloomer · Post by **dloomer** » 2006-11-11 03:20

It does seem that at least in hMailserver's implementation, the local domain is automatically whitelisted. And it's whitelisted by the domain name in the e-maill address in the From envelope header, not by IP address (which would not be practical anyway unless it used SPF, which I suggested even though it somehow doesn't feel right). Martin hasn't directly confirmed this, and I haven't looked at the code (I might if I get a chance), but the behavior I've seen seems to strongly suggest this is what is happening. And in all other cases -- in other words, when the From domain is not a hosted domain -- hMail definitely appears to perform greylisting the usual way, looking up by the full triplet, or in the case of the whitelist by IP address. My greylisting setup is extremely successful (I'm doing better than the 95% cited in the white paper) in all cases EXCEPT when the spammer impersonates my hosted domain in the From envelope e-mail address, where my success rate is 0%.

Going back to Martin's suggestion (and this also backs up my guess that it is whitelisting the hosted domain), I thought about my SPF setup and decided that since I am the only person who sends mail from my domain, and I know exactly which SMTP servers I could potentially send from, I feel comfortable with changing my SPF to -all (vs. ~all). I did that, and it solves the problem as all spammers impersonating my domain in the From envelope header now get rejected by SPF (albeit not by the greylist). When I was using ~all, the best I could hope for was a SOFTFAIL which would not trigger an SPF rejection and rightly so.

This works for me in my situation, but my guess is that it won't be an option for everybody.

matty · Post by **matty** » 2006-11-13 18:48

be careful with the -all option! See my post here (it is toward the bottom):
http://www.hmailserver.com/forum/viewtopic.php?t=6101

I thought it would be fine too but quickly found a lot of legit email was blocked.

dloomer · Post by **dloomer** » 2006-11-13 19:04

Interesting point, I'll just need to keep an eye on things when I fill out web forms that send e-mails on my behalf -- that should be my only concern. I'm comfortable keeping the -all for now -- in my particular case, which I know wouldn't necessarily apply to someone else.

But just another reason why the SPF solution to this purported problem with hMail's greylist implementation isn't going to be an option for everyone. I suppose the programmers are waiting for a complaint from an actual person who can't find a workaround (which isn't me, since I worked around it), but I figure that has to be an issue more of "when" than "if."

Post by **martin** » 2006-11-13 19:17

It's correct that local users bypasses grey listing. We had a discussion a long time ago about this and the best suggestion to minimize issues anyone came up with was to let local users bypass grey listing. I can't really remember what the issue was. It feels it would be enough to let SMTP authenticated users bypass the grey listing. :-\

dloomer · Post by **dloomer** » 2006-11-13 19:19

Any chance of just making it configurable whether to let those users (whether pretending or not) through? I realize there's a fine balance between simplicity and functionality, just thought I'd suggest it.

Post by **martin** » 2006-11-13 19:20

I'll add that to the next version..

GotNet · Post by **GotNet** » 2006-11-13 20:20

martin wrote:It's correct that local users bypasses grey listing. We had a discussion a long time ago about this and the best suggestion to minimize issues anyone came up with was to let local users bypass grey listing. I can't really remember what the issue was. It feels it would be enough to let SMTP authenticated users bypass the grey listing. :-\

I had to whitelist local users early on so that read receipts would function.

chanas · Post by **chanas** » 2006-11-26 14:27

Authentication can solve the bypass problem. I found two possible ways but
I dont think we can implement them yet.

A. Spammer forges "from" to be from local domain but the account is a non-existant one. So a check if the "from" account is a valid one before accepting it kills this one.
B. A user pretends he is a valid local one. Since I have authentication
on for all my users that could solve the problem if local->local required
authentication. But we don't have such a choice. The checkbox "Require auth for delivery to local accounts" would affect the "External to local" too, so no e-mails, and I guess nobody will appreciate it

So a local -> local with forced auth solves it I guess. Could we do that in scripting if and until Martin implements that?

Is there logic to all this or should I drink more cofee?