Greylisting by subnetwork
- jimimaseye
- Moderator
- Posts: 10053
- Joined: 2011-09-08 17:48
Re: Greylisting by subnetwork
I have WhatsApp as an advanced replacement for sms. And a Facebook account locked out to literally everyone (even myself .... nearly) just for those God awful software companies that insist on being contactable by Facebook only. Everything else is email and telephones.
There is nothing SOCIAL about modern social networks. 'Posting' a broadcast message in a platform on the hope someone MIGHT be interested, to a load of 'friends' you probably don't remember meeting or have never even met or never will, with the misguided belief that your words are important to their life and never comprehending or believing that NO ONE would have read it.
A bit like I'm doing on this post.
[Entered by mobile. Excuse my spelling.]
There is nothing SOCIAL about modern social networks. 'Posting' a broadcast message in a platform on the hope someone MIGHT be interested, to a load of 'friends' you probably don't remember meeting or have never even met or never will, with the misguided belief that your words are important to their life and never comprehending or believing that NO ONE would have read it.
A bit like I'm doing on this post.
[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829
Re: Greylisting by subnetwork
Caedite eos. Novit enim Dominus qui sunt eius.
Kill them. Kill them all! Spike their hearts with a silver stake and douse them with holy water. ALL of them!
- jimimaseye
- Moderator
- Posts: 10053
- Joined: 2011-09-08 17:48
Re: Greylisting by subnetwork
Twitter, Instagram and Snapchat. Really? The epitome of wasting time and space. Can't wait for the day they follow bebo and MySpace .....and 3d TVs.
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829
Re: Greylisting by subnetwork
I don't work on Viagra but I did miss my very important invitation to the XXX International Conference on Computer Science (i.e., 30th conference). My argument was the danger of content filtering (call it AI if you want), and that it can bite completely unexpectedly (so whitelists are of no use).It seems to me that people legitimately working with Viagra, etc must know that there is an issue with spam and have already worked up solutions
The day when I see a server behaving in this way, greylisting is dead.Someone sends in an email
Server farm attempts delivery from ip a.b.c.d
Hits greylist.
Server reattempts delivery from j.k.l.m
Hits greylist
Server reattempts delivery from w.x.y.z
However, my strong assumption is that server farms allocate IPs mostly by subets, not completely random.
This confirms my assumption! You see: my technique reduces a million of IPs to two hundred. Worst case is 82, which means that typically their messages would pass after some 40 attempts. Not great, but better than half million attempts as it is now.That's 181 netblocks or 1.328.916 individual IPv4 addresses
Ideally, if the subnet mask is made configurable, such as oMessage.GreylistingMask = 24; then I might want to try /16 and see what happens. Probably this is indeed a case when a big provider needs a special treatment: if IP/16 is in a predefined list of 82 known subnets then oMessage.GreylistingMask = 16 else oMessage.GreylistingMask = 24. Still safe for less monstrous servers and almost solves the problem with Outlook.
(I can't agree more!)What a world. People really have too much time to waste.
And still please let me remind you that this thread is not about each of us's proven recipes of how best to fight spam (we will not solve this problem here). It's about making greylisting in HMS a bit more configurable, for whatever it's worth.
Re: Greylisting by subnetwork
Sorry for an off-topic comment... but there is much reason in social networks, actually an interplay of many reasons.There is nothing SOCIAL about modern social networks.
The big (SOCIAL) advantage of social networks over old-school subscription-based mailing lists is re-tweeting (re-posting or how it is called): a mechanism for people to vote for some messages, so of all this heap of garbage, some information floats up to the surface -- the information that is selected in a social/democratic way by the users. After all, the humankind has no way to know what is important other than by re-tweeting. In academia this is called citation: many say garbage, but when one says something worthy, others repeat it, and it becomes common knowledge.
Another SOCIAL function of social networks is a substitute for religion: a human being wants to be a part of eternity and to be in the center of the Universe. In the past, religion catered for this need. Now, when it is mostly lost, social networks picked up the function of a supernatural ether to which our souls belong. With this, any worth-for-nothing person having nothing to do with his useless life has an illusion of spiritual fusion with eternity when he posts his stupid selfie in The Cloud (= The Universe) where Somebody surely watches it. Anyway, better than narcotics, and serves the same SOCIAL goal.
(I have to admit that when I write my own webpage, I have this feeling: somewhere in archive.net my life is preserved for the eternity, and Somebody Someday will read all I wrote there.)
Re: Greylisting by subnetwork
Oops no. With 82 subnets, with probability of 0.5 the message will pass by 10th attempt. With probability of 0.94, it will pass by 20th attempt, i.e., in one day. The probability of not passing by 40th attempt is 0.000005, and by 50th attempt (2 days) is 0.000000001. Given that a decent server should make at least 50 attempts, in non-time-critical applications no special treatment is needed.Worst case is 82, which means that typically their messages would pass after some 40 attempts.
For the original greylisting, that is, of 600,000 IPs, the probability of the message being bounced after 100th attempt is 0.99.
Assuming 100 attempts as the server's threshold, in this extreme situation my method gives 100% guaranteed success, while traditional greylisting gives 99% guaranteed failure.
Re: Greylisting by subnetwork
not sure what religion you mean, but common religions - particularly Abrahamic 3 - strictly impose modesty and fairness, forbid egocentrism and illusionism.gelbukh wrote: ↑2018-09-15 06:00Another SOCIAL function of social networks is a substitute for religion: a human being wants to be a part of eternity and to be in the center of the Universe. In the past, religion catered for this need. Now, when it is mostly lost, social networks picked up the function of a supernatural ether to which our souls belong. With this, any worth-for-nothing person having nothing to do with his useless life has an illusion of spiritual fusion with eternity when he posts his stupid selfie in The Cloud (= The Universe) where Somebody surely watches it. Anyway, better than narcotics, and serves the same SOCIAL goal.
it's generally true that religion is mostly lost and social media became one of major "tin gods" (communication technologies, IT) of Neo-Paganism, which in fact can no way substitute that "lost" religion as it has no eschatology.
yes, for those who need, better than narcotics, not that effective though
Katip
--
HMS 5.7, MariaDB 10.4.10, SA 4.0.0, ClamAV 0.103.8
--
HMS 5.7, MariaDB 10.4.10, SA 4.0.0, ClamAV 0.103.8
- jimimaseye
- Moderator
- Posts: 10053
- Joined: 2011-09-08 17:48
Re: Greylisting by subnetwork
How would God/Allah/Yahweh/Buddah/Donald Trump handle ineffective greylisting?
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829
Re: Greylisting by subnetwork
I do indeed have a decent server, and my decent server gives up after 3 attempts. Gmail tries for 3 days but a quick (incomplete) Google search did not yield the number of attempts. The number of attempts is up to each administrator, so there's no golden rule to follow. You can't rely on what "a decent server" should do.
Re: Greylisting by subnetwork
I was going through my old logs (from 2011 and up) and found that I only used a "clean" Greylisting from march 2014 to august 2015. About that time I added the OnHELO trigger to my 5.4.2 and modified the Admin GUI - the value is actually stored in minutes in the DB but the GUI changes it to hours - and introduced my Dynamic Greylisting Whitelist based on HELO/EHLO greeting.
"Minutes to defer delivery attempts = 4" <== Changed hours to minutes
"Hours before removing unused records = 12"
"Days before removing unused records = 32"
hmailserver.ini:
[Settings]
GreylistingRecordExpirationInterval=30
I got really - really - really fed up having to wait up to 3 hours for emails to come through from Google and others. It got so bad that if I was on the phone with someone and they said they would send me an email, I would switch OFF greylisting until I got the email...
Anyways... Google and Outlook are generally within the same subnet but I have seen variations. Amazon and their "mail for hire - amazonses.com" service is all over the map and so is Linkdin. This could be a geographical issue as we are only ~5 million people in Denmark in total so they can squeeze us in where they have spare bandwidth.
"Minutes to defer delivery attempts = 4" <== Changed hours to minutes
"Hours before removing unused records = 12"
"Days before removing unused records = 32"
hmailserver.ini:
[Settings]
GreylistingRecordExpirationInterval=30
I got really - really - really fed up having to wait up to 3 hours for emails to come through from Google and others. It got so bad that if I was on the phone with someone and they said they would send me an email, I would switch OFF greylisting until I got the email...
Anyways... Google and Outlook are generally within the same subnet but I have seen variations. Amazon and their "mail for hire - amazonses.com" service is all over the map and so is Linkdin. This could be a geographical issue as we are only ~5 million people in Denmark in total so they can squeeze us in where they have spare bandwidth.
SørenR.
Woke is Marxism advancing through Maoist cultural revolution.
Woke is Marxism advancing through Maoist cultural revolution.
Re: Greylisting by subnetwork
So, these services do not use a reasonably limited number of subnets?Amazon and their "mail for hire - amazonses.com" service is all over the map and so is Linkdin.
If so, then greylisting is dead. Nothing to discuss. I will experiment with delays.
My fear about delays is that 20 sec can be too long for some servers, and then such servers will have absolutely no chance to pass a message, even after many attempts. Can't we have the best of the two worlds? I mean, to vary the action for greylisting: not to drop the connection, but instead challenge the sender with a 20 sec delay. That is, apply delays only for first attempt and skip it for the second attempt. A spammer will give up and not return. A legitimate server will most probably wait and deliver the message. But if the delay is too long and it drops the connection, then it may have a second chance (and in this case, subnet masking is still useful).
With this, a message can be lost only if (1) the server has too short connection timeout, AND (2) it uses too many IPs (or subnets); then bad luck. No guarantee, but probably better than unconditional delay and than greylisting as is.
Re: Greylisting by subnetwork
Ah, well, with delays there is no second try.
I did some investigation when i switched to delays and 9 out of 10 times SPAM only got as far as the first delay.
I would claim that on any given day the delays and the Snowshoe detection take care of 80% of my traffic, the rest is managed by Spamassassin.
I did some investigation when i switched to delays and 9 out of 10 times SPAM only got as far as the first delay.
I would claim that on any given day the delays and the Snowshoe detection take care of 80% of my traffic, the rest is managed by Spamassassin.
SørenR.
Woke is Marxism advancing through Maoist cultural revolution.
Woke is Marxism advancing through Maoist cultural revolution.
- jimimaseye
- Moderator
- Posts: 10053
- Joined: 2011-09-08 17:48
Re: Greylisting by subnetwork
Many of us use the 20 second relay including myself. I haven't had a single complaint, evidence or reason to think that a mail had failed to be delivered. But the spam count is dramatically reduced.
https://www.tldp.org/HOWTO/Spam-Filteri ... elays.html
https://www.tldp.org/HOWTO/Spam-Filteri ... elays.html
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829
Re: Greylisting by subnetwork
Yes they do, https://aws.amazon.com/blogs/messaging- ... addresses/
HELO/EHLO check
Code: Select all
^(a)[0-9]{1,2}(-)[0-9]{1,3}(\.smtp-out\.([a-z]{2}\-[a-z]{4,5}\-\d{1,2}\.)?amazonses\.com)$
Code: Select all
^(mail[a-z]{1}\-[a-z]{2})(\.linkedin\.com)$
Code: Select all
Case "amazonses.com"
oRegEx.Pattern= "^54\.240\.([0-9]|[1-5][0-9]|6[0-3])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
"^199\.127\.(2(3[2-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
"^199\.255\.(1(9[2-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$"
case "linkedin.com"
oRegEx.Pattern= "^199\.101\.162\.([0-9]|[1-9][0-9]|1([0-1][0-9]|2[0-7]))$|" &_
"^108\.174\.3\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
"^108\.174\.6\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
"^108\.174\.0\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
"^199\.101\.161\.130$"
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup
Re: Greylisting by subnetwork
@RvdH
Would it be hard to get hMailserver to greylist >> Whitelist either by FQDN (eg spf.gmail.com) or by regex (like the examples that you just listed), and have these then get checked on the fly?
Easier perhaps than changing greylisting triplets to be one of these things instead of IP address only, and again checking them on the fly (would probably need some way to enter these manually via the GUI too)
Would it be hard to get hMailserver to greylist >> Whitelist either by FQDN (eg spf.gmail.com) or by regex (like the examples that you just listed), and have these then get checked on the fly?
Easier perhaps than changing greylisting triplets to be one of these things instead of IP address only, and again checking them on the fly (would probably need some way to enter these manually via the GUI too)
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation
https://www.hmailserver.com/documentation
Re: Greylisting by subnetwork
GreyWhitelisting by FQDN (EHLO/HELO) alone is unreliable, these can easily be faked, so basically you need to do both, first check if the HELO/EHLO banner matches known values and then verify if sender ip is within allowed ip range
To do that inside hmailserver? mmm...Probably, but that takes away the fun of scripting (and way above my head, skill wise)
To do that inside hmailserver? mmm...Probably, but that takes away the fun of scripting (and way above my head, skill wise)
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup
Re: Greylisting by subnetwork
OK thanks for the answer
This is the only bit I was after, just with various ways to show the allowed IP range
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation
https://www.hmailserver.com/documentation
Re: Greylisting by subnetwork
OK, just to let you know that I've disabled greylisting and implemented some delays (without any whitelisting for large senders). Will let you know if I observe something unexpected.
For greylisting, there are options to skip on SPF pass and on messages from A or MX. I did not find how to detect this info from a script, to bypass the delays. Maybe by presence/absence of some headers? Also, is it easy to check if the sender is whitelisted? Again, to omit the delays.
If greylisting is dead and the delays are the way to go, then it would be better to have the delays as one of the anti-spam mechanisms in HMS itself, to enjoy the same treatment (skip on SPF pass, A or MX, whitelisting) as other anti-spam mechanisms. The minimum modification would be to select the action for greylisting: rejection (as now) or delay. Nearly one line of code changes, in the file greylisting.cpp.
For greylisting, there are options to skip on SPF pass and on messages from A or MX. I did not find how to detect this info from a script, to bypass the delays. Maybe by presence/absence of some headers? Also, is it easy to check if the sender is whitelisted? Again, to omit the delays.
If greylisting is dead and the delays are the way to go, then it would be better to have the delays as one of the anti-spam mechanisms in HMS itself, to enjoy the same treatment (skip on SPF pass, A or MX, whitelisting) as other anti-spam mechanisms. The minimum modification would be to select the action for greylisting: rejection (as now) or delay. Nearly one line of code changes, in the file greylisting.cpp.
Re: Greylisting by subnetwork
Greylisting was great until the mass email domain hosters started up - now it just doesn't work.
If we could find a way to use greylisting for mass senders, or better to greylist / whitelist them, this would be ideal.
Delay is the poor cousin at best of greylisting - but it works OK
If we could find a way to use greylisting for mass senders, or better to greylist / whitelist them, this would be ideal.
Delay is the poor cousin at best of greylisting - but it works OK
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation
https://www.hmailserver.com/documentation
Re: Greylisting by subnetwork
1 line of code? Go ahead and make the changes yourself then... the project is on github for a reasongelbukh wrote: ↑2018-09-16 04:01If greylisting is dead and the delays are the way to go, then it would be better to have the delays as one of the anti-spam mechanisms in HMS itself, to enjoy the same treatment (skip on SPF pass, A or MX, whitelisting) as other anti-spam mechanisms. The minimum modification would be to select the action for greylisting: rejection (as now) or delay. Nearly one line of code changes, in the file greylisting.cpp.
Once you have something that works, you can send me a pull request
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup
Re: Greylisting by subnetwork
FYI, I think you changed:
"Minutes to defer delivery attempts = 4"
"Hours before removing unused records = 12" <== Changed days to hours
"Days before removing unused records = 32"
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup
Re: Greylisting by subnetwork
Btw, i just checked the hm_greylisting_triplets db table, but in both my running instances all 'glipaddress2' are NULL values..
What is the 'glipaddress2' for? Is it a leftover from earlier version?
EDIT: Ah, nevermind.... i think it is for IP6
What is the 'glipaddress2' for? Is it a leftover from earlier version?
EDIT: Ah, nevermind.... i think it is for IP6
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup
Re: Greylisting by subnetwork
Yes, exactly... Not sure what I was looking at when I whote that. My diff's say exactly the same as you wrote.
In hindsight that modification may not have been the smartest move considering how the large mailserver farms work.
I don't know if any of you watched the video I posted earlier. One of the things Aaron Poffenberger speaks about is "walking the spf" to collect IP address for the whitelist. He also made some tools to do it ('nix clearly).
https://github.com/akpoff
He also mention the idea of whitelisting the domains you send mail TO... I believe there is a script somewhere in the forum that does this for a general whitelist. It should be possible to modify this to "GreyWhitelist" in stead of "global whitelist".
There is no doubt that a successfull greylist is a lot of work and require a lot of ressources. For that reason alone, using delays is favorable
SørenR.
Woke is Marxism advancing through Maoist cultural revolution.
Woke is Marxism advancing through Maoist cultural revolution.
Re: Greylisting by subnetwork
Came across this ... Sometimes Perl looks is sooo much easier
Save to SPFList.pl and execute with "SPFList spf.protection.outlook.com"
Not 100% sure what to change to make it run on Windows... I've killed Perl on all my boxes except my NAS.
"-q=TXT" may need to be changed to "-type=TXT".
Save to SPFList.pl and execute with "SPFList spf.protection.outlook.com"
Not 100% sure what to change to make it run on Windows... I've killed Perl on all my boxes except my NAS.
"-q=TXT" may need to be changed to "-type=TXT".
Code: Select all
#!/usr/bin/perl
$domain=shift @ARGV;
@results=getit($domain);
sub getit {
my $domain=shift;
my @foo=`nslookup -q=TXT $domain`;
my @results=();
foreach (@foo) {
next if not /$domain\ttext/;
s/$domain\ttext = "v=spf1//;
@results=split /\s+/;
foreach (@results) {
next if /-all/;
print "$_\n";
if (/include:/) {
s/include://;
getit($_);
}
}
}
}
SørenR.
Woke is Marxism advancing through Maoist cultural revolution.
Woke is Marxism advancing through Maoist cultural revolution.
Re: Greylisting by subnetwork
This works on Windows. Changes are marked by CHANGED.Not 100% sure what to change to make it run on Windows...
Code: Select all
#!/usr/bin/perl
$domain=shift @ARGV;
@results=getit($domain);
sub getit {
my $domain=shift;
my @foo=`nslookup -q=TXT $domain`;
my @results=();
foreach (@foo) {
next if not /^\s*"v=spf1/; # CHANGED
s/^\s*"v=spf1//; # CHANGED
@results=split /\s+/;
foreach (@results) {
next if /-all/ or /^$/; # CHANGED
if (/include:/) {
s/include://;
getit($_);
}
else { # CHANGED
print "$_\n"; # CHANGED
}
}
}
}
Re: Greylisting by subnetwork
Well, can we fix it? Here is an idea:Greylisting was great until the mass email domain hosters started up - now it just doesn't work.
The logic of greylisting is: "reject the message; if the same message is attempted again, pass it". The triples only serve the purpose of identifying whether it is the same message. What does not work anymore is the IP as part of identification of the message, right? But we can identify a message in other ways. A trivial way is by the contents: in OnAcceptMessage, take a checksum of the message (excluding the headers) and use it in the triple instead of the sender IP. Of course, you can mix in there some headers, such as probably the sending timestamp (or, just use this timestamp alone instead of the IP). Maybe simply the message size could be used for this, though I suspect it includes headers that can change from attempt to attempt.
I realize that this would not save bandwith, but nowadays bandwith is cheap, what we care about is the time of the user on reading the message. Anyway all "AI methods", including keyword-based filtering, work on the full text of the message.
For this to work from scripts, it could be good to change the following in the code:
- To move greylisting check after OnAcceptMessage. Actually, all antispam checks can be moved at the end, to give the script a chance to whitelist the message before any checks. We do not need to save bandwith, so no point in early rejection;
- To add property to the Message object: oMessage.GreyListingID, which will be used instead of IP in greylisting.cpp. It would be initially populated with the IP, but the script can mask it by /24, or assign a completely different string, such as the message MD5 sum, timestamp, or whatever;
- Ideally, add oMessage.BypassAntispam, so that the script could opt to bypass any antispam checks by, say, some keywords in the message.
Re: Greylisting by subnetwork
I don't think changing the way Greylisting works is the way forward. Anyways it would be violating the RFC.
I feel that akpoff is on the right path, how little we want it, but none the less... The solution is to work our way around greylisting for the major senders by "walking the SPF" and adding these IP addresses to the whitelist. SPF records do change frequently, so running a batch job every sunday should probably be fine.
We can easily argue that whitelisting should be modified to allow for CIDR notated addressing.
You _should_ really spend the hour to watch the speech he is giving at BSDCan.
Aaron Poffenberger: Fighting Spam at the Frontline -- BSDCan 2018.
https://www.youtube.com/watch?v=PKY6rSpzTIQ
I feel that akpoff is on the right path, how little we want it, but none the less... The solution is to work our way around greylisting for the major senders by "walking the SPF" and adding these IP addresses to the whitelist. SPF records do change frequently, so running a batch job every sunday should probably be fine.
We can easily argue that whitelisting should be modified to allow for CIDR notated addressing.
You _should_ really spend the hour to watch the speech he is giving at BSDCan.
Aaron Poffenberger: Fighting Spam at the Frontline -- BSDCan 2018.
https://www.youtube.com/watch?v=PKY6rSpzTIQ
SørenR.
Woke is Marxism advancing through Maoist cultural revolution.
Woke is Marxism advancing through Maoist cultural revolution.
Re: Greylisting by subnetwork
@SorenR
Isn't this doing exactly that, walking the SPF?
Downside of this method is the fact the greylistwhitelist could get very long and causing performance issues in hmailserver, and it could hold entries one might never get a connection from
Anyway, besides that i liked the idea off walking the SPF, so i'm working on something in C# to walk the (ip4) spf records and be able to dynamically add entries to the greylistwhitelist
What the program does is basically doing what i did manually first, you still validate HELO/EHLO like before using regexp, then you call the program which first walks the SPF records, adds them to a regexp pattern and verify the connecting ip address matched against this...eliminating the need of predefined SPF IP regexp ranges, inspired by mattg's question if this could be done on the fly
Isn't this doing exactly that, walking the SPF?
Downside of this method is the fact the greylistwhitelist could get very long and causing performance issues in hmailserver, and it could hold entries one might never get a connection from
Anyway, besides that i liked the idea off walking the SPF, so i'm working on something in C# to walk the (ip4) spf records and be able to dynamically add entries to the greylistwhitelist
Code: Select all
With CreateObject("WScript.Shell")
iReturn = .Run("""spfverify.exe"" 174.125.82.54 google.com", 0, True)
if iReturn = 0 Then
EventLog.Write("pass")
REM add to the greylistwhitelist
elseif iReturn = 1 Then
EventLog.Write("not pass")
else
EventLog.Write("Command error")
End if
End With
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup
Re: Greylisting by subnetwork
Did you look at this ?RvdH wrote: ↑2018-09-17 13:22@SorenR
Isn't this doing exactly that, walking the SPF?
Downside of this method is the fact the greylistwhitelist could get very long and causing performance issues in hmailserver, and it could hold entries one might never get a connection from
Anyway, besides that i liked the idea off walking the SPF, so i'm working on something in C# to walk the (ip4) spf records and be able to dynamically add entries to the greylistwhitelist
What the program does is basically doing what i did manually first, you still validate HELO/EHLO like before using regexp, then you call the program which first walks the SPF records, adds them to a regexp pattern and verify the connecting ip address matched against this...eliminating the need of predefined SPF IP regexp ranges, inspired by mattg's question if this could be done on the flyCode: Select all
With CreateObject("WScript.Shell") iReturn = .Run("""spfverify.exe"" 174.125.82.54 google.com", 0, True) if iReturn = 0 Then EventLog.Write("pass") REM add to the greylistwhitelist elseif iReturn = 1 Then EventLog.Write("not pass") else EventLog.Write("Command error") End if End With
https://github.com/akpoff/spfwalk
SørenR.
Woke is Marxism advancing through Maoist cultural revolution.
Woke is Marxism advancing through Maoist cultural revolution.
Re: Greylisting by subnetwork
No, but basically it does the same... ip4 only though, hmailserver does not support ip6 in greywhitelisting to my knowledge
Im currently testing...but it looks to work as expected, Example from Eventlog
Code: Select all
3416 "2018-09-17 15:45:47.555" "spfverify.exe 40.107.13.82 passed for: EUR01-HE1-obe.outbound.protection.outlook.com"
3408 "2018-09-17 15:48:11.127" "spfverify.exe 209.85.219.172 passed for: mail-yb1-f172.google.com"
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup
Re: Greylisting by subnetwork
So you are whitelisting then as they come ??
What prevents a spammer with valid SPF settings from being whitelisted?
What prevents a spammer with valid SPF settings from being whitelisted?
SørenR.
Woke is Marxism advancing through Maoist cultural revolution.
Woke is Marxism advancing through Maoist cultural revolution.
Re: Greylisting by subnetwork
Huh
We are talking about dynamic greylistwhitelisting based on ip and host, aren't we? If a spammer uses a valid gmail account they won't be greylisted either, do they?
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup
Re: Greylisting by subnetwork
Had to re-read it again...
So HELO/EHLO is validated against SPF because it is statistically very unlikely that a spammer would fake HELO/EHLO greeting AND the SPF information at the same time?
That would work, but I'm not sure that it resource wise is better than doing a daily/weekly SPF walk and update the whitelist ... How often do SPF data change?
I was wondering if outbound mail should automatically whitelist the domain but I'm thinking that for safety X number of sent mails in Y days is needed to qualify for a whitelist entry.
Whitelisting will need to support CIDR/Netblocks regardless for simplicity.
Whitelists added outside the SPF walk should have an expiration date for inactivity.
SørenR.
Woke is Marxism advancing through Maoist cultural revolution.
Woke is Marxism advancing through Maoist cultural revolution.
Re: Greylisting by subnetwork
Don't know if you can use this, but since I use an off-site BackUp-MX, I receive mail from banned IP Ranges via this "backdoor"...
This is how I scan for them using the IP address from the "Received" header with my Backup-MX (backup-mx.post.tele.dk) and hMailServer "From-To" IP Range...
Actually it this part that is interesting
This is how I scan for them using the IP address from the "Received" header with my Backup-MX (backup-mx.post.tele.dk) and hMailServer "From-To" IP Range...
Code: Select all
Function Lookup(strRegEx, strMatch) : Lookup = False
With CreateObject("VBScript.RegExp")
.Pattern = strRegEx
.Global = False
.MultiLine = True
.IgnoreCase = True
If .Test(strMatch) Then Lookup = True
End With
End Function
Function oLookup(strRegEx, strMatch, bGlobal)
With CreateObject("VBScript.RegExp")
.Pattern = strRegEx
.Global = bGlobal
.MultiLine = True
.IgnoreCase = True
Set oLookup = .Execute(strMatch)
End With
End Function
Function LongIntegerFromIP(p_strIP)
Dim arrTemp, i, lngTemp
arrTemp = Split(p_strIP, ".")
For i = 0 To UBound(arrTemp)
lngTemp = lngTemp + CLng(arrTemp(i)) * (256 ^ (3 - i))
Next
LongIntegerFromIP = lngTemp
End Function
Function isAutoBan(oMessage)
Dim strRegEx, i, a, Match, Matches, m_strIP, m_strLowerIP, m_strUpperIP
Dim oApp : Set oApp = CreateObject("hMailServer.Application")
Call oApp.Authenticate(ADMIN, PASSWORD)
strRegEx = "(?:[0-9]{1,3}\.){3}[0-9]{1,3}"
For i = 0 To oMessage.Headers.Count-1
If oMessage.Headers(i).Name = "Received" Then
If Lookup("by backup-mx.post.tele.dk", oMessage.Headers(i).Value) Then
Set Matches = oLookup(strRegEx, oMessage.Headers(i).Value, False)
If Matches.Count > 0 Then
For Each Match In Matches
m_strIP = LongIntegerFromIP(Match.Value)
For a = 0 To oApp.Settings.SecurityRanges.Count-1
If (oApp.Settings.SecurityRanges.Item(a).Priority = 20) Then
m_strLowerIP = LongIntegerFromIP(oApp.Settings.SecurityRanges.Item(a).LowerIP)
m_strUpperIP = LongIntegerFromIP(oApp.Settings.SecurityRanges.Item(a).UpperIP)
If (m_strIP >= m_strLowerIP) And (m_strIP <= m_strUpperIP) Then
Result.Value = 2
Result.Message = "5.7.1 CODE08 The SMTP service on IP address (" & Match.Value & ") is not welcome here."
Exit Function
End If
End If
Next
Next
Exit Function
End If
End If
End If
Next
End Function
Code: Select all
m_strLowerIP = LongIntegerFromIP(oApp.Settings.SecurityRanges.Item(a).LowerIP)
m_strUpperIP = LongIntegerFromIP(oApp.Settings.SecurityRanges.Item(a).UpperIP)
If (m_strIP >= m_strLowerIP) And (m_strIP <= m_strUpperIP) Then
Result.Value = 2
Result.Message = "5.7.1 CODE08 The SMTP service on IP address (" & Match.Value & ") is not welcome here."
Exit Function
End If
SørenR.
Woke is Marxism advancing through Maoist cultural revolution.
Woke is Marxism advancing through Maoist cultural revolution.
Re: Greylisting by subnetwork
The program is bigger than intentionally planned, i had to use a external lib (ARSoft.Tools.Net) because the classes i tried to use didn't always behave as expected, resulting in failures
In Sub OnHELO(oClient) in the experimental build or Sub OnSMTPData(oClient, oMessage) do something like:
Sub AddGreyList() and used functions
Call the program like:
Requirements
.Net 4.5
Download
https://d-fault.nl/files/spfverify.zip
In Sub OnHELO(oClient) in the experimental build or Sub OnSMTPData(oClient, oMessage) do something like:
Code: Select all
Dim oRegEx
Set oRegEx = CreateObject("VBScript.RegExp")
oRegEx.IgnoreCase = True
oRegEx.Global = False
oRegEx.Pattern= "^([a-z]{3}[\d]{2}\-[a-z]{2}[\d]\-)(obe\.outbound\.protection\.outlook\.com)$|" &_
"^(mail\-[a-z]{2}[\d]\-f[\d]{1,3})(\.google\.com)$|" &_
"^(o[\d]{1,2})\.(email\.wetransfer\.com)$|" &_
"^(o[\d]{1,2})\.(email\.airbnb\.com)$|" &_
"^(o[\d]{1,2}\.sg|mail\-[\w]{2,3}[\d]+|mailout\-[\w]{2,3}\-[\w]{2,3})(\.booking\.com)$|" &_
"^(mail[a-z]{1}\-[a-z]{2})(\.linkedin\.com)$|" &_
"^(spruce\-goose\-[a-z]{2}|spring\-chicken\-[a-z]{2})(\.twitter\.com)$|" &_
"^(mx\-out\.facebook\.com)$|" &_
"^(cpsmtpb\-ews[\d]{2,3}|cpsps\-ews[\d]]{2,3}|cpdelvb\-safe[\d]]{2,3})(\.kpnxchange\.com)$|" &_
"^(lb[\d]{1}\-smtp\-cloud[\d]{1})(\.xs4all\.net)$|" &_
"^(marktplaats\.nl)$|" &_
"^(a)[0-9]{1,2}(-)[0-9]{1,3}(\.smtp-out\.([a-z]{2}\-[a-z]{4,5}\-\d{1,2}\.)?amazonses\.com)$|" &_
"^(o[\d]{1,2})(\.em\.spotify\.com)$|" &_
"^(webgrid[a-z]{1}\d{3}\.emsecure\.net)$|" &_
"^(mailrelay|mailsec)\d{3}(\.isp\.belgacom\.be)$"
If oRegEx.Test(oClient.HELO) Then
Call AddGreyList(oClient.IPAddress, oClient.HELO)
Result.Value = 0
Exit Sub
End If
Set oRegEx = Nothing
Code: Select all
Sub AddGreyList(ByVal strIP, ByVal strHELO)
dim iReturn : iReturn = 2
dim hostname : hostname = getDomainName(strHELO)
Dim oApp
Set oApp = CreateObject("hMailServer.Application")
Call oApp.Authenticate("Administrator", sAdminPassword)
With LockFile("C:\Program Files (x86)\hMailServer\Temp\greylistwhite.lck")
On Error Resume Next
oApp.Settings.AntiSpam.GreyListingWhiteAddresses.Refresh
If oApp.Settings.AntiSpam.GreyListingWhiteAddresses.ItemByName(strIP) Is Nothing Then
With CreateObject("WScript.Shell")
iReturn = .Run("""C:\Program Files (x86)\hMailServer\Events\spfverify.exe"" " & strIP & " " & hostname & "", 0, True)
End With
if iReturn = 0 Then
EventLog.Write("spfverify.exe " & strIP & " passed for: " & hostname)
With oApp.Settings.AntiSpam.GreyListingWhiteAddresses.Add
.Description = Date & " Auto-Added '" & strHELO & "'"
.IPAddress = strIP
.Save
End With
ElseIf iReturn = 1 Then
EventLog.Write("spfverify.exe " & strIP & " failed for: " & hostname)
Else
EventLog.Write("spfverify.exe command error, spfverify.exe " & strIP & " failed for: " & hostname)
End if
Else
With oApp.Settings.AntiSpam.GreyListingWhiteAddresses.ItemByName(strIP)
.Description = Date & " Auto-Added '" & strHELO & "'"
.Save
End With
End If
oApp.Settings.AntiSpam.GreyListingWhiteAddresses.Refresh
On Error Goto 0
.Close '// Close LockFile
End With
Set oApp = Nothing
End Sub
Function getDomainName(byVal strHELO)
dim aryDomain, str2ndLevel, strTopLevel
getDomainName = Null
If Len(strHELO) > 0 Then
aryDomain = Split(strHELO,".")
If uBound(aryDomain) >= 1 Then
str2ndLevel = aryDomain(uBound(aryDomain)-1)
strTopLevel = aryDomain(uBound(aryDomain))
getDomainName = str2ndLevel & "." & strTopLevel
End If
End If
End Function
Function LockFile(strPath)
Const Append = 8
Const Unicode = -1
With CreateObject("Scripting.FileSystemObject")
Dim oFile, i
For i = 0 To 30
On Error Resume Next
Set oFile = .OpenTextFile(strPath, Append, True, Unicode)
If (Not Err.Number = 70) Then
Set LockFile = oFile
On Error Goto 0
Exit For
End If
On Error Goto 0
Wait(1)
Next
End With
Set oFile = Nothing
If (Err.Number = 70) Then
EventLog.Write("ERROR: EventHandlers.vbs")
EventLog.Write("File " & strPath & " is locked and timeout was exceeded.")
Err.Clear
ElseIf (Err.Number <> 0) Then
EventLog.Write("ERROR: EventHandlers.vbs : Function LockFile")
EventLog.Write("Error : " & Err.Number)
EventLog.Write("Error (hex) : 0x" & Hex(Err.Number))
EventLog.Write("Source : " & Err.Source)
EventLog.Write("Description : " & Err.Description)
Err.Clear
End If
End Function
Function Wait(sec)
With CreateObject("WScript.Shell")
.Run "timeout /NOBREAK /T " & Int(sec), 0, True
' REM .Run "sleep -m " & Int(sec * 1000), 0, True
' REM .Run "powershell Start-Sleep -Milliseconds " & Int(sec * 1000), 0, True
End With
End Function
Code: Select all
spfverify.exe ipaddress hostname [-verbose]
- ipaddress is required
- hostname is required
- -verbose is optional
Code: Select all
spfverify.exe 5.57.20.177 booking.com
IP address matches SPF record(s) ip range for domain (return code = 0)
.Net 4.5
Download
https://d-fault.nl/files/spfverify.zip
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup
Re: Greylisting by subnetwork
I managed to build the app without using the ARSoft.Tools.Net Library after all, bringing it down from 929kb to just 20kb
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup
Re: Greylisting by subnetwork
- fixed a bug in mx:domain.com lookup
- added support for exists:%{i} and exists:%{ir} SPF records
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup