Greylisting by subnetwork

Use this forum if you want to suggest a new feature to hMailServer. Before posting, please search the forum to confirm that it has not already been suggested.
User avatar
jimimaseye
Moderator
Moderator
Posts: 10053
Joined: 2011-09-08 17:48

Re: Greylisting by subnetwork

Post by jimimaseye » 2018-09-14 23:25

I have WhatsApp as an advanced replacement for sms. And a Facebook account locked out to literally everyone (even myself .... nearly) just for those God awful software companies that insist on being contactable by Facebook only. Everything else is email and telephones.

There is nothing SOCIAL about modern social networks. 'Posting' a broadcast message in a platform on the hope someone MIGHT be interested, to a load of 'friends' you probably don't remember meeting or have never even met or never will, with the misguided belief that your words are important to their life and never comprehending or believing that NO ONE would have read it.

A bit like I'm doing on this post. 😁

[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

palinka
Senior user
Senior user
Posts: 4455
Joined: 2017-09-12 17:57

Re: Greylisting by subnetwork

Post by palinka » 2018-09-15 00:19

SorenR wrote:
2018-09-14 23:04
I'm still debating myself if I should kill my Facebook and Google+ accounts and just settle for Linkedin, Skype, Twitter, Instagram and Snapchat.
Caedite eos. Novit enim Dominus qui sunt eius.

Kill them. Kill them all! Spike their hearts with a silver stake and douse them with holy water. ALL of them!

User avatar
jimimaseye
Moderator
Moderator
Posts: 10053
Joined: 2011-09-08 17:48

Re: Greylisting by subnetwork

Post by jimimaseye » 2018-09-15 00:32

Twitter, Instagram and Snapchat. Really? The epitome of wasting time and space. Can't wait for the day they follow bebo and MySpace .....and 3d TVs.
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-15 05:35

It seems to me that people legitimately working with Viagra, etc must know that there is an issue with spam and have already worked up solutions
I don't work on Viagra but I did miss my very important invitation to the XXX International Conference on Computer Science (i.e., 30th conference). My argument was the danger of content filtering (call it AI if you want), and that it can bite completely unexpectedly (so whitelists are of no use).
Someone sends in an email

Server farm attempts delivery from ip a.b.c.d

Hits greylist.

Server reattempts delivery from j.k.l.m

Hits greylist

Server reattempts delivery from w.x.y.z
The day when I see a server behaving in this way, greylisting is dead.

However, my strong assumption is that server farms allocate IPs mostly by subets, not completely random.
That's 181 netblocks or 1.328.916 individual IPv4 addresses
This confirms my assumption! You see: my technique reduces a million of IPs to two hundred. Worst case is 82, which means that typically their messages would pass after some 40 attempts. Not great, but better than half million attempts as it is now.

Ideally, if the subnet mask is made configurable, such as oMessage.GreylistingMask = 24; then I might want to try /16 and see what happens. Probably this is indeed a case when a big provider needs a special treatment: if IP/16 is in a predefined list of 82 known subnets then oMessage.GreylistingMask = 16 else oMessage.GreylistingMask = 24. Still safe for less monstrous servers and almost solves the problem with Outlook.
What a world. People really have too much time to waste.
(I can't agree more!)

And still please let me remind you that this thread is not about each of us's proven recipes of how best to fight spam (we will not solve this problem here). It's about making greylisting in HMS a bit more configurable, for whatever it's worth.

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-15 06:00

There is nothing SOCIAL about modern social networks.
Sorry for an off-topic comment... but there is much reason in social networks, actually an interplay of many reasons.

The big (SOCIAL) advantage of social networks over old-school subscription-based mailing lists is re-tweeting (re-posting or how it is called): a mechanism for people to vote for some messages, so of all this heap of garbage, some information floats up to the surface -- the information that is selected in a social/democratic way by the users. After all, the humankind has no way to know what is important other than by re-tweeting. In academia this is called citation: many say garbage, but when one says something worthy, others repeat it, and it becomes common knowledge.

Another SOCIAL function of social networks is a substitute for religion: a human being wants to be a part of eternity and to be in the center of the Universe. In the past, religion catered for this need. Now, when it is mostly lost, social networks picked up the function of a supernatural ether to which our souls belong. With this, any worth-for-nothing person having nothing to do with his useless life has an illusion of spiritual fusion with eternity when he posts his stupid selfie in The Cloud (= The Universe) where Somebody surely watches it. Anyway, better than narcotics, and serves the same SOCIAL goal.

(I have to admit that when I write my own webpage, I have this feeling: somewhere in archive.net my life is preserved for the eternity, and Somebody Someday will read all I wrote there.)

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-15 06:35

Worst case is 82, which means that typically their messages would pass after some 40 attempts.
Oops no. With 82 subnets, with probability of 0.5 the message will pass by 10th attempt. With probability of 0.94, it will pass by 20th attempt, i.e., in one day. The probability of not passing by 40th attempt is 0.000005, and by 50th attempt (2 days) is 0.000000001. Given that a decent server should make at least 50 attempts, in non-time-critical applications no special treatment is needed.

For the original greylisting, that is, of 600,000 IPs, the probability of the message being bounced after 100th attempt is 0.99.

Assuming 100 attempts as the server's threshold, in this extreme situation my method gives 100% guaranteed success, while traditional greylisting gives 99% guaranteed failure.

User avatar
katip
Senior user
Senior user
Posts: 1158
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Greylisting by subnetwork

Post by katip » 2018-09-15 10:18

gelbukh wrote:
2018-09-15 06:00
Another SOCIAL function of social networks is a substitute for religion: a human being wants to be a part of eternity and to be in the center of the Universe. In the past, religion catered for this need. Now, when it is mostly lost, social networks picked up the function of a supernatural ether to which our souls belong. With this, any worth-for-nothing person having nothing to do with his useless life has an illusion of spiritual fusion with eternity when he posts his stupid selfie in The Cloud (= The Universe) where Somebody surely watches it. Anyway, better than narcotics, and serves the same SOCIAL goal.
not sure what religion you mean, but common religions - particularly Abrahamic 3 - strictly impose modesty and fairness, forbid egocentrism and illusionism.
it's generally true that religion is mostly lost and social media became one of major "tin gods" (communication technologies, IT) of Neo-Paganism, which in fact can no way substitute that "lost" religion as it has no eschatology.
yes, for those who need, better than narcotics, not that effective though :roll:
Katip
--
HMS 5.7, MariaDB 10.4.10, SA 4.0.0, ClamAV 0.103.8

User avatar
jimimaseye
Moderator
Moderator
Posts: 10053
Joined: 2011-09-08 17:48

Re: Greylisting by subnetwork

Post by jimimaseye » 2018-09-15 10:27

How would God/Allah/Yahweh/Buddah/Donald Trump handle ineffective greylisting?

:mrgreen:
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

palinka
Senior user
Senior user
Posts: 4455
Joined: 2017-09-12 17:57

Re: Greylisting by subnetwork

Post by palinka » 2018-09-15 10:30

gelbukh wrote:
2018-09-15 06:35
Given that a decent server should make at least 50 attempts
I do indeed have a decent server, and my decent server gives up after 3 attempts. Gmail tries for 3 days but a quick (incomplete) Google search did not yield the number of attempts. The number of attempts is up to each administrator, so there's no golden rule to follow. You can't rely on what "a decent server" should do.

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-09-15 13:56

I was going through my old logs (from 2011 and up) and found that I only used a "clean" Greylisting from march 2014 to august 2015. About that time I added the OnHELO trigger to my 5.4.2 and modified the Admin GUI - the value is actually stored in minutes in the DB but the GUI changes it to hours - and introduced my Dynamic Greylisting Whitelist based on HELO/EHLO greeting.

"Minutes to defer delivery attempts = 4" <== Changed hours to minutes
"Hours before removing unused records = 12"
"Days before removing unused records = 32"

hmailserver.ini:
[Settings]
GreylistingRecordExpirationInterval=30

I got really - really - really fed up having to wait up to 3 hours for emails to come through from Google and others. It got so bad that if I was on the phone with someone and they said they would send me an email, I would switch OFF greylisting until I got the email...

Anyways... Google and Outlook are generally within the same subnet but I have seen variations. Amazon and their "mail for hire - amazonses.com" service is all over the map and so is Linkdin. This could be a geographical issue as we are only ~5 million people in Denmark in total so they can squeeze us in where they have spare bandwidth.
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-15 23:47

Amazon and their "mail for hire - amazonses.com" service is all over the map and so is Linkdin.
So, these services do not use a reasonably limited number of subnets?

If so, then greylisting is dead. Nothing to discuss. I will experiment with delays.

My fear about delays is that 20 sec can be too long for some servers, and then such servers will have absolutely no chance to pass a message, even after many attempts. Can't we have the best of the two worlds? I mean, to vary the action for greylisting: not to drop the connection, but instead challenge the sender with a 20 sec delay. That is, apply delays only for first attempt and skip it for the second attempt. A spammer will give up and not return. A legitimate server will most probably wait and deliver the message. But if the delay is too long and it drops the connection, then it may have a second chance (and in this case, subnet masking is still useful).

With this, a message can be lost only if (1) the server has too short connection timeout, AND (2) it uses too many IPs (or subnets); then bad luck. No guarantee, but probably better than unconditional delay and than greylisting as is.

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-09-16 00:20

Ah, well, with delays there is no second try.

I did some investigation when i switched to delays and 9 out of 10 times SPAM only got as far as the first delay.

I would claim that on any given day the delays and the Snowshoe detection take care of 80% of my traffic, the rest is managed by Spamassassin.
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
jimimaseye
Moderator
Moderator
Posts: 10053
Joined: 2011-09-08 17:48

Re: Greylisting by subnetwork

Post by jimimaseye » 2018-09-16 00:22

Many of us use the 20 second relay including myself. I haven't had a single complaint, evidence or reason to think that a mail had failed to be delivered. But the spam count is dramatically reduced.

https://www.tldp.org/HOWTO/Spam-Filteri ... elays.html
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Greylisting by subnetwork

Post by RvdH » 2018-09-16 01:42

gelbukh wrote:
2018-09-15 23:47
Amazon and their "mail for hire - amazonses.com" service is all over the map and so is Linkdin.
So, these services do not use a reasonably limited number of subnets?

If so, then greylisting is dead. Nothing to discuss. I will experiment with delays.
Yes they do, https://aws.amazon.com/blogs/messaging- ... addresses/

HELO/EHLO check

Code: Select all

^(a)[0-9]{1,2}(-)[0-9]{1,3}(\.smtp-out\.([a-z]{2}\-[a-z]{4,5}\-\d{1,2}\.)?amazonses\.com)$

Code: Select all

^(mail[a-z]{1}\-[a-z]{2})(\.linkedin\.com)$
IP Range

Code: Select all

Case "amazonses.com"
oRegEx.Pattern= "^54\.240\.([0-9]|[1-5][0-9]|6[0-3])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
"^199\.127\.(2(3[2-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
"^199\.255\.(1(9[2-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$"
case "linkedin.com"	
oRegEx.Pattern= "^199\.101\.162\.([0-9]|[1-9][0-9]|1([0-1][0-9]|2[0-7]))$|" &_
"^108\.174\.3\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
"^108\.174\.6\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
"^108\.174\.0\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$|" &_
"^199\.101\.161\.130$"
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Greylisting by subnetwork

Post by mattg » 2018-09-16 02:02

@RvdH

Would it be hard to get hMailserver to greylist >> Whitelist either by FQDN (eg spf.gmail.com) or by regex (like the examples that you just listed), and have these then get checked on the fly?

Easier perhaps than changing greylisting triplets to be one of these things instead of IP address only, and again checking them on the fly (would probably need some way to enter these manually via the GUI too)
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Greylisting by subnetwork

Post by RvdH » 2018-09-16 02:25

GreyWhitelisting by FQDN (EHLO/HELO) alone is unreliable, these can easily be faked, so basically you need to do both, first check if the HELO/EHLO banner matches known values and then verify if sender ip is within allowed ip range


To do that inside hmailserver? mmm...Probably, but that takes away the fun of scripting (and way above my head, skill wise) :)
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Greylisting by subnetwork

Post by mattg » 2018-09-16 02:29

OK thanks for the answer
RvdH wrote:
2018-09-16 02:25
... then verify if sender ip is within allowed ip range...
This is the only bit I was after, just with various ways to show the allowed IP range
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-16 04:01

OK, just to let you know that I've disabled greylisting and implemented some delays (without any whitelisting for large senders). Will let you know if I observe something unexpected.

For greylisting, there are options to skip on SPF pass and on messages from A or MX. I did not find how to detect this info from a script, to bypass the delays. Maybe by presence/absence of some headers? Also, is it easy to check if the sender is whitelisted? Again, to omit the delays.

If greylisting is dead and the delays are the way to go, then it would be better to have the delays as one of the anti-spam mechanisms in HMS itself, to enjoy the same treatment (skip on SPF pass, A or MX, whitelisting) as other anti-spam mechanisms. The minimum modification would be to select the action for greylisting: rejection (as now) or delay. Nearly one line of code changes, in the file greylisting.cpp.

User avatar
mattg
Moderator
Moderator
Posts: 22435
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Greylisting by subnetwork

Post by mattg » 2018-09-16 08:10

Greylisting was great until the mass email domain hosters started up - now it just doesn't work.

If we could find a way to use greylisting for mass senders, or better to greylist / whitelist them, this would be ideal.
Delay is the poor cousin at best of greylisting - but it works OK
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Greylisting by subnetwork

Post by RvdH » 2018-09-16 08:19

gelbukh wrote:
2018-09-16 04:01
If greylisting is dead and the delays are the way to go, then it would be better to have the delays as one of the anti-spam mechanisms in HMS itself, to enjoy the same treatment (skip on SPF pass, A or MX, whitelisting) as other anti-spam mechanisms. The minimum modification would be to select the action for greylisting: rejection (as now) or delay. Nearly one line of code changes, in the file greylisting.cpp.
1 line of code? Go ahead and make the changes yourself then... the project is on github for a reason
Once you have something that works, you can send me a pull request :lol:
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Greylisting by subnetwork

Post by RvdH » 2018-09-16 08:43

SorenR wrote:
2018-09-15 13:56
"Minutes to defer delivery attempts = 4" <== Changed hours to minutes
"Hours before removing unused records = 12"
"Days before removing unused records = 32"
FYI, I think you changed:

"Minutes to defer delivery attempts = 4"
"Hours before removing unused records = 12" <== Changed days to hours
"Days before removing unused records = 32"
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Greylisting by subnetwork

Post by RvdH » 2018-09-16 12:16

Btw, i just checked the hm_greylisting_triplets db table, but in both my running instances all 'glipaddress2' are NULL values..
What is the 'glipaddress2' for? Is it a leftover from earlier version?


EDIT: Ah, nevermind.... i think it is for IP6
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-09-16 12:57

RvdH wrote:
2018-09-16 08:43
SorenR wrote:
2018-09-15 13:56
"Minutes to defer delivery attempts = 4" <== Changed hours to minutes
"Hours before removing unused records = 12"
"Days before removing unused records = 32"
FYI, I think you changed:

"Minutes to defer delivery attempts = 4"
"Hours before removing unused records = 12" <== Changed days to hours
"Days before removing unused records = 32"
Yes, exactly... Not sure what I was looking at when I whote that. My diff's say exactly the same as you wrote.

In hindsight that modification may not have been the smartest move considering how the large mailserver farms work.

I don't know if any of you watched the video I posted earlier. One of the things Aaron Poffenberger speaks about is "walking the spf" to collect IP address for the whitelist. He also made some tools to do it ('nix clearly).
https://github.com/akpoff

He also mention the idea of whitelisting the domains you send mail TO... I believe there is a script somewhere in the forum that does this for a general whitelist. It should be possible to modify this to "GreyWhitelist" in stead of "global whitelist".

There is no doubt that a successfull greylist is a lot of work and require a lot of ressources. For that reason alone, using delays is favorable :mrgreen:
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-09-17 03:18

Came across this ... Sometimes Perl looks is sooo much easier :roll:

Save to SPFList.pl and execute with "SPFList spf.protection.outlook.com"
Not 100% sure what to change to make it run on Windows... I've killed Perl on all my boxes except my NAS.
"-q=TXT" may need to be changed to "-type=TXT".

Code: Select all

#!/usr/bin/perl
$domain=shift @ARGV;
@results=getit($domain);


sub getit {
  my $domain=shift;

  my @foo=`nslookup -q=TXT $domain`;
  my @results=();
  foreach (@foo) {
   next if not /$domain\ttext/;
   s/$domain\ttext = "v=spf1//;
   @results=split /\s+/;
   foreach (@results) {
    next if /-all/;
    print "$_\n";
    if (/include:/) {
     s/include://;
     getit($_);
    }
   } 
  } 
}
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-17 04:15

Not 100% sure what to change to make it run on Windows...
This works on Windows. Changes are marked by CHANGED.

Code: Select all

#!/usr/bin/perl
$domain=shift @ARGV;
@results=getit($domain);

sub getit {
  my $domain=shift;

  my @foo=`nslookup -q=TXT $domain`;
  my @results=();
  foreach (@foo) {
   next if not /^\s*"v=spf1/;       # CHANGED
   s/^\s*"v=spf1//;                 # CHANGED
   @results=split /\s+/;
   foreach (@results) {
    next if /-all/ or /^$/;         # CHANGED
    if (/include:/) {
     s/include://;
     getit($_);
    }
    else {                          # CHANGED
    print "$_\n";                   # CHANGED
    }
   } 
  } 
}

gelbukh
New user
New user
Posts: 25
Joined: 2018-06-13 21:52

Re: Greylisting by subnetwork

Post by gelbukh » 2018-09-17 06:42

Greylisting was great until the mass email domain hosters started up - now it just doesn't work.
Well, can we fix it? Here is an idea:

The logic of greylisting is: "reject the message; if the same message is attempted again, pass it". The triples only serve the purpose of identifying whether it is the same message. What does not work anymore is the IP as part of identification of the message, right? But we can identify a message in other ways. A trivial way is by the contents: in OnAcceptMessage, take a checksum of the message (excluding the headers) and use it in the triple instead of the sender IP. Of course, you can mix in there some headers, such as probably the sending timestamp (or, just use this timestamp alone instead of the IP). Maybe simply the message size could be used for this, though I suspect it includes headers that can change from attempt to attempt.

I realize that this would not save bandwith, but nowadays bandwith is cheap, what we care about is the time of the user on reading the message. Anyway all "AI methods", including keyword-based filtering, work on the full text of the message.

For this to work from scripts, it could be good to change the following in the code:
  • To move greylisting check after OnAcceptMessage. Actually, all antispam checks can be moved at the end, to give the script a chance to whitelist the message before any checks. We do not need to save bandwith, so no point in early rejection;
  • To add property to the Message object: oMessage.GreyListingID, which will be used instead of IP in greylisting.cpp. It would be initially populated with the IP, but the script can mask it by /24, or assign a completely different string, such as the message MD5 sum, timestamp, or whatever;
  • Ideally, add oMessage.BypassAntispam, so that the script could opt to bypass any antispam checks by, say, some keywords in the message.
Again, looks as very little change to the code, for good benefit.

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-09-17 10:34

I don't think changing the way Greylisting works is the way forward. Anyways it would be violating the RFC.

I feel that akpoff is on the right path, how little we want it, but none the less... The solution is to work our way around greylisting for the major senders by "walking the SPF" and adding these IP addresses to the whitelist. SPF records do change frequently, so running a batch job every sunday should probably be fine.

We can easily argue that whitelisting should be modified to allow for CIDR notated addressing.

You _should_ really spend the hour to watch the speech he is giving at BSDCan.

Aaron Poffenberger: Fighting Spam at the Frontline -- BSDCan 2018.
https://www.youtube.com/watch?v=PKY6rSpzTIQ
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Greylisting by subnetwork

Post by RvdH » 2018-09-17 13:22

@SorenR
Isn't this doing exactly that, walking the SPF?

Downside of this method is the fact the greylistwhitelist could get very long and causing performance issues in hmailserver, and it could hold entries one might never get a connection from


Anyway, besides that i liked the idea off walking the SPF, so i'm working on something in C# to walk the (ip4) spf records and be able to dynamically add entries to the greylistwhitelist

Code: Select all

With CreateObject("WScript.Shell")
	iReturn = .Run("""spfverify.exe"" 174.125.82.54 google.com", 0, True)
	if iReturn = 0 Then
		EventLog.Write("pass")
                REM add to the greylistwhitelist 
	elseif iReturn = 1 Then 		
		EventLog.Write("not pass")
	else		
		EventLog.Write("Command error")
	End if
End With
What the program does is basically doing what i did manually first, you still validate HELO/EHLO like before using regexp, then you call the program which first walks the SPF records, adds them to a regexp pattern and verify the connecting ip address matched against this...eliminating the need of predefined SPF IP regexp ranges, inspired by mattg's question if this could be done on the fly :)
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-09-17 15:42

RvdH wrote:
2018-09-17 13:22
@SorenR
Isn't this doing exactly that, walking the SPF?

Downside of this method is the fact the greylistwhitelist could get very long and causing performance issues in hmailserver, and it could hold entries one might never get a connection from


Anyway, besides that i liked the idea off walking the SPF, so i'm working on something in C# to walk the (ip4) spf records and be able to dynamically add entries to the greylistwhitelist

Code: Select all

With CreateObject("WScript.Shell")
	iReturn = .Run("""spfverify.exe"" 174.125.82.54 google.com", 0, True)
	if iReturn = 0 Then
		EventLog.Write("pass")
                REM add to the greylistwhitelist 
	elseif iReturn = 1 Then 		
		EventLog.Write("not pass")
	else		
		EventLog.Write("Command error")
	End if
End With
What the program does is basically doing what i did manually first, you still validate HELO/EHLO like before using regexp, then you call the program which first walks the SPF records, adds them to a regexp pattern and verify the connecting ip address matched against this...eliminating the need of predefined SPF IP regexp ranges, inspired by mattg's question if this could be done on the fly :)
Did you look at this ?

https://github.com/akpoff/spfwalk
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Greylisting by subnetwork

Post by RvdH » 2018-09-17 15:46

SorenR wrote:
2018-09-17 15:42
Did you look at this ?

https://github.com/akpoff/spfwalk
No, but basically it does the same... ip4 only though, hmailserver does not support ip6 in greywhitelisting to my knowledge

Im currently testing...but it looks to work as expected, Example from Eventlog

Code: Select all

3416	"2018-09-17 15:45:47.555"	"spfverify.exe 40.107.13.82 passed for: EUR01-HE1-obe.outbound.protection.outlook.com"
3408	"2018-09-17 15:48:11.127"	"spfverify.exe 209.85.219.172 passed for: mail-yb1-f172.google.com"
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-09-17 16:25

So you are whitelisting then as they come ??

What prevents a spammer with valid SPF settings from being whitelisted?
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Greylisting by subnetwork

Post by RvdH » 2018-09-17 16:59

SorenR wrote:
2018-09-17 16:25
So you are whitelisting then as they come ??

What prevents a spammer with valid SPF settings from being whitelisted?
Huh :?

We are talking about dynamic greylistwhitelisting based on ip and host, aren't we? If a spammer uses a valid gmail account they won't be greylisted either, do they? :mrgreen:
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-09-17 21:38

RvdH wrote:
2018-09-17 16:59
SorenR wrote:
2018-09-17 16:25
So you are whitelisting then as they come ??

What prevents a spammer with valid SPF settings from being whitelisted?
Huh :?

We are talking about dynamic greylistwhitelisting based on ip and host, aren't we? If a spammer uses a valid gmail account they won't be greylisted either, do they? :mrgreen:
Had to re-read it again... :oops:

So HELO/EHLO is validated against SPF because it is statistically very unlikely that a spammer would fake HELO/EHLO greeting AND the SPF information at the same time?

That would work, but I'm not sure that it resource wise is better than doing a daily/weekly SPF walk and update the whitelist ... How often do SPF data change?

I was wondering if outbound mail should automatically whitelist the domain but I'm thinking that for safety X number of sent mails in Y days is needed to qualify for a whitelist entry.

Whitelisting will need to support CIDR/Netblocks regardless for simplicity.

Whitelists added outside the SPF walk should have an expiration date for inactivity.
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
SorenR
Senior user
Senior user
Posts: 6308
Joined: 2006-08-21 15:38
Location: Denmark

Re: Greylisting by subnetwork

Post by SorenR » 2018-09-17 21:57

Don't know if you can use this, but since I use an off-site BackUp-MX, I receive mail from banned IP Ranges via this "backdoor"...

This is how I scan for them using the IP address from the "Received" header with my Backup-MX (backup-mx.post.tele.dk) and hMailServer "From-To" IP Range...

Code: Select all

Function Lookup(strRegEx, strMatch) : Lookup = False
   With CreateObject("VBScript.RegExp")
      .Pattern = strRegEx
      .Global = False
      .MultiLine = True
      .IgnoreCase = True
      If .Test(strMatch) Then Lookup = True
   End With
End Function

Function oLookup(strRegEx, strMatch, bGlobal)
   With CreateObject("VBScript.RegExp")
      .Pattern = strRegEx
      .Global = bGlobal
      .MultiLine = True
      .IgnoreCase = True
      Set oLookup = .Execute(strMatch)
   End With
End Function

Function LongIntegerFromIP(p_strIP)
   Dim arrTemp, i, lngTemp
   arrTemp = Split(p_strIP, ".")
   For i = 0 To UBound(arrTemp)
      lngTemp = lngTemp + CLng(arrTemp(i)) * (256 ^ (3 - i))
   Next
   LongIntegerFromIP = lngTemp
End Function

Function isAutoBan(oMessage)
   Dim strRegEx, i, a, Match, Matches, m_strIP, m_strLowerIP, m_strUpperIP
   Dim oApp : Set oApp = CreateObject("hMailServer.Application")
   Call oApp.Authenticate(ADMIN, PASSWORD)
   strRegEx = "(?:[0-9]{1,3}\.){3}[0-9]{1,3}"
   For i = 0 To oMessage.Headers.Count-1
      If oMessage.Headers(i).Name = "Received" Then
         If Lookup("by backup-mx.post.tele.dk", oMessage.Headers(i).Value) Then
            Set Matches = oLookup(strRegEx, oMessage.Headers(i).Value, False)
            If Matches.Count > 0 Then
               For Each Match In Matches
                  m_strIP = LongIntegerFromIP(Match.Value)
                  For a = 0 To oApp.Settings.SecurityRanges.Count-1
                     If (oApp.Settings.SecurityRanges.Item(a).Priority = 20) Then
                        m_strLowerIP = LongIntegerFromIP(oApp.Settings.SecurityRanges.Item(a).LowerIP)
                        m_strUpperIP = LongIntegerFromIP(oApp.Settings.SecurityRanges.Item(a).UpperIP)
                        If (m_strIP >= m_strLowerIP) And (m_strIP <= m_strUpperIP) Then
                           Result.Value = 2
                           Result.Message = "5.7.1 CODE08 The SMTP service on IP address (" & Match.Value & ") is not welcome here."
                           Exit Function
                        End If
                     End If
                  Next
               Next
               Exit Function
            End If
         End If
      End If
   Next
End Function
Actually it this part that is interesting

Code: Select all

m_strLowerIP = LongIntegerFromIP(oApp.Settings.SecurityRanges.Item(a).LowerIP)
m_strUpperIP = LongIntegerFromIP(oApp.Settings.SecurityRanges.Item(a).UpperIP)
If (m_strIP >= m_strLowerIP) And (m_strIP <= m_strUpperIP) Then
   Result.Value = 2
   Result.Message = "5.7.1 CODE08 The SMTP service on IP address (" & Match.Value & ") is not welcome here."
   Exit Function
End If
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Greylisting by subnetwork

Post by RvdH » 2018-09-18 16:16

The program is bigger than intentionally planned, i had to use a external lib (ARSoft.Tools.Net) because the classes i tried to use didn't always behave as expected, resulting in failures

In Sub OnHELO(oClient) in the experimental build or Sub OnSMTPData(oClient, oMessage) do something like:

Code: Select all

Dim oRegEx
Set oRegEx = CreateObject("VBScript.RegExp")
oRegEx.IgnoreCase = True
oRegEx.Global = False
oRegEx.Pattern= "^([a-z]{3}[\d]{2}\-[a-z]{2}[\d]\-)(obe\.outbound\.protection\.outlook\.com)$|" &_
				"^(mail\-[a-z]{2}[\d]\-f[\d]{1,3})(\.google\.com)$|" &_
				"^(o[\d]{1,2})\.(email\.wetransfer\.com)$|" &_
				"^(o[\d]{1,2})\.(email\.airbnb\.com)$|" &_
				"^(o[\d]{1,2}\.sg|mail\-[\w]{2,3}[\d]+|mailout\-[\w]{2,3}\-[\w]{2,3})(\.booking\.com)$|" &_
				"^(mail[a-z]{1}\-[a-z]{2})(\.linkedin\.com)$|" &_
				"^(spruce\-goose\-[a-z]{2}|spring\-chicken\-[a-z]{2})(\.twitter\.com)$|" &_
				"^(mx\-out\.facebook\.com)$|" &_
				"^(cpsmtpb\-ews[\d]{2,3}|cpsps\-ews[\d]]{2,3}|cpdelvb\-safe[\d]]{2,3})(\.kpnxchange\.com)$|" &_
				"^(lb[\d]{1}\-smtp\-cloud[\d]{1})(\.xs4all\.net)$|" &_
				"^(marktplaats\.nl)$|" &_
				"^(a)[0-9]{1,2}(-)[0-9]{1,3}(\.smtp-out\.([a-z]{2}\-[a-z]{4,5}\-\d{1,2}\.)?amazonses\.com)$|" &_
				"^(o[\d]{1,2})(\.em\.spotify\.com)$|" &_
				"^(webgrid[a-z]{1}\d{3}\.emsecure\.net)$|" &_
				"^(mailrelay|mailsec)\d{3}(\.isp\.belgacom\.be)$"
If oRegEx.Test(oClient.HELO) Then 
	Call AddGreyList(oClient.IPAddress, oClient.HELO)
	Result.Value = 0
	Exit Sub
End If
Set oRegEx = Nothing
Sub AddGreyList() and used functions

Code: Select all

Sub AddGreyList(ByVal strIP, ByVal strHELO)
	dim iReturn : iReturn = 2
	dim hostname : hostname = getDomainName(strHELO) 
	Dim oApp
	Set oApp = CreateObject("hMailServer.Application")
	Call oApp.Authenticate("Administrator", sAdminPassword)
	With LockFile("C:\Program Files (x86)\hMailServer\Temp\greylistwhite.lck")
		On Error Resume Next
		oApp.Settings.AntiSpam.GreyListingWhiteAddresses.Refresh
		If oApp.Settings.AntiSpam.GreyListingWhiteAddresses.ItemByName(strIP) Is Nothing Then
			With CreateObject("WScript.Shell")
				iReturn = .Run("""C:\Program Files (x86)\hMailServer\Events\spfverify.exe"" " & strIP & " " & hostname & "", 0, True)	
			End With
			if iReturn = 0 Then
				EventLog.Write("spfverify.exe " & strIP & " passed for: " & hostname)
				With oApp.Settings.AntiSpam.GreyListingWhiteAddresses.Add
					.Description = Date & " Auto-Added '" & strHELO & "'"
					.IPAddress = strIP
					.Save
				End With
			ElseIf iReturn = 1 Then 		
				EventLog.Write("spfverify.exe " & strIP & " failed for: " & hostname)
			Else		
				EventLog.Write("spfverify.exe command error, spfverify.exe " & strIP & " failed for: " & hostname)
			End if			
		Else
			With oApp.Settings.AntiSpam.GreyListingWhiteAddresses.ItemByName(strIP)
				.Description = Date & " Auto-Added '" & strHELO & "'"
				.Save
			End With
		End If
		oApp.Settings.AntiSpam.GreyListingWhiteAddresses.Refresh
		On Error Goto 0
		.Close '// Close LockFile
	End With 
	Set oApp = Nothing
End Sub

Function getDomainName(byVal strHELO)
	dim aryDomain, str2ndLevel, strTopLevel
	getDomainName = Null
	If Len(strHELO) > 0 Then  	
		aryDomain = Split(strHELO,".")
		If uBound(aryDomain) >= 1 Then
			str2ndLevel = aryDomain(uBound(aryDomain)-1)
			strTopLevel = aryDomain(uBound(aryDomain))			
			getDomainName = str2ndLevel & "." & strTopLevel
		End If
	End If
End Function

Function LockFile(strPath)
	Const Append = 8
	Const Unicode = -1
	With CreateObject("Scripting.FileSystemObject")
		Dim oFile, i
		For i = 0 To 30
			On Error Resume Next
			Set oFile = .OpenTextFile(strPath, Append, True, Unicode)
			If (Not Err.Number = 70) Then
				Set LockFile = oFile
				On Error Goto 0
				Exit For
			End If
			On Error Goto 0
			Wait(1)
		Next
	End With
	Set oFile = Nothing
	If (Err.Number = 70) Then
		EventLog.Write("ERROR: EventHandlers.vbs")
		EventLog.Write("File " & strPath & " is locked and timeout was exceeded.")
		Err.Clear
	ElseIf (Err.Number <> 0) Then
		EventLog.Write("ERROR: EventHandlers.vbs : Function LockFile")
		EventLog.Write("Error       : " & Err.Number)
		EventLog.Write("Error (hex) : 0x" & Hex(Err.Number))
		EventLog.Write("Source      : " & Err.Source)
		EventLog.Write("Description : " & Err.Description)
		Err.Clear
	End If
End Function

Function Wait(sec)
	With CreateObject("WScript.Shell")
		.Run "timeout /NOBREAK /T " & Int(sec), 0, True
		' REM .Run "sleep -m " & Int(sec * 1000), 0, True
		' REM .Run "powershell Start-Sleep -Milliseconds " & Int(sec * 1000), 0, True
	End With
End Function
Call the program like:

Code: Select all

spfverify.exe ipaddress hostname [-verbose]
  • ipaddress is required
  • hostname is required
  • -verbose is optional
Example:

Code: Select all

spfverify.exe 5.57.20.177 booking.com
IP address matches SPF record(s) ip range for domain (return code = 0)
Requirements
.Net 4.5

Download
https://d-fault.nl/files/spfverify.zip
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Greylisting by subnetwork

Post by RvdH » 2018-09-23 00:18

I managed to build the app without using the ARSoft.Tools.Net Library after all, bringing it down from 929kb to just 20kb
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
RvdH
Senior user
Senior user
Posts: 3231
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Greylisting by subnetwork

Post by RvdH » 2020-02-24 10:44

  • fixed a bug in mx:domain.com lookup
  • added support for exists:%{i} and exists:%{ir} SPF records
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

Post Reply