SURBL bug?

[palinka]

I noticed this today:

Code: Select all

DEBUG 4296 2024-05-13 10:49:10.787 SURBL: Execute
DEBUG 4296 2024-05-13 10:49:10.787 SURBL: Found URL: googleapis.com
DEBUG 4296 2024-05-13 10:49:10.787 SURBL: Found URL: windows.net
DEBUG 4296 2024-05-13 10:49:10.787 SURBL: 2 unique addresses found.
DEBUG 4296 2024-05-13 10:49:10.787 SURBL: Lookup: googleapis.com.multi.surbl.org
DEBUG 4296 2024-05-13 10:49:10.896 SURBL: Lookup: windows.net.multi.surbl.org
DEBUG 4296 2024-05-13 10:49:11.021 SURBL: Match not found
DEBUG 4296 2024-05-13 10:49:11.021 SURBL: Execute
DEBUG 4296 2024-05-13 10:49:11.021 SURBL: Found URL: googleapis.com
DEBUG 4296 2024-05-13 10:49:11.021 SURBL: Found URL: windows.net
DEBUG 4296 2024-05-13 10:49:11.021 SURBL: 2 unique addresses found.
DEBUG 4296 2024-05-13 10:49:11.021 SURBL: Lookup: googleapis.com.dbl.spamhaus.org
DEBUG 4296 2024-05-13 10:49:11.099 SURBL: Lookup: windows.net.dbl.spamhaus.org
DEBUG 4296 2024-05-13 10:49:11.131 SURBL: Match not found
DEBUG 4296 2024-05-13 10:49:11.131 Spam test: SpamTestSURBL, Score: 0

However, I have scripted SURBL testing as part of my shortlink expansion project. I also had this for the same message in my event log:

Code: Select all

4296 2024-05-13 10:49:18.438 SURBL hit on message URL: dsfsd32wsfes.blob.core.windows.net

Hmailserver seems to be parsing only the highest domain level, i.e. windows.net vs dsfsd32wsfes.blob.core.windows.net. Of course windows.net did not hit on SURBL.

This is my regex for finding whole domains:

Code: Select all

(?:https?:\/\/)([a-zA-Z0-9-.]+)(?:\/[^\s]+)?

[RvdH]

Seems to be by design, eg: https://github.com/hmailserver/hmailser ... pp#L70-L74

[palinka]

That makes SURBL testing almost useless.

But at least there is an alternative. https://hmailserver.com/forum/viewtopic.php?f=9&t=41891

I use Soren's blacklist on hits. So far there was only 1 false positive, which was a legit debt collection agency (gym membership). I assume those messages were marked as spam by many, many message recipients on tons of mail servers which is why the URL ended up on SURBL. Not listed on Spamhaus DBL, however. Gyms are notorious for shady business practices with their membership contracts. They make a lot of money from auto-renewals where the client isn't even aware, and then they make you jump through hoops to get the renewal cancelled, all the while billing you for that time. Very shady.

[RvdH]

That makes SURBL testing almost useless.

But at least there is an alternative. https://hmailserver.com/forum/viewtopic.php?f=9&t=41891

I use Soren's blacklist on hits. So far there was only 1 false positive, which was a legit debt collection agency (gym membership). I assume those messages were marked as spam by many, many message recipients on tons of mail servers which is why the URL ended up on SURBL. Not listed on Spamhaus DBL, however. Gyms are notorious for shady business practices with their membership contracts. They make a lot of money from auto-renewals where the client isn't even aware, and then they make you jump through hoops to get the renewal cancelled, all the while billing you for that time. Very shady.

SpamAssassin

[RvdH]

That makes SURBL testing almost useless.

But how many SURBL servers actually list subdomains? Have a number? I can imagine for the sake of simplicity SURBL list would use toplevel domains

[palinka]

That makes SURBL testing almost useless.

But how many SURBL servers actually list subdomains? Have a number? I can imagine for the sake of simplicity SURBL list would use toplevel domains

I do have a number - or at least i can find it out, since i get more hits on whole domains than I do on main/top domains. Without actually counting, I'm guessing its at least 3 times more on whole domains than main/top domains. That's a significant difference.

SURBL.org sees a difference, obviously, between whole domains and main/top domains. For example, the one I showed above is listed on SURBL.org while windows.net was not.

I only use two SURBL servers: surbl.org and dbl.spamhaus.net. Spamhaus is extremely conservative, so its hard to tell whether they restrict listings to main/top domains or not. But surbl.org definitely does differentiate between whole and main/top domains.

[palinka]

That makes SURBL testing almost useless.

But at least there is an alternative. https://hmailserver.com/forum/viewtopic.php?f=9&t=41891

I use Soren's blacklist on hits. So far there was only 1 false positive, which was a legit debt collection agency (gym membership). I assume those messages were marked as spam by many, many message recipients on tons of mail servers which is why the URL ended up on SURBL. Not listed on Spamhaus DBL, however. Gyms are notorious for shady business practices with their membership contracts. They make a lot of money from auto-renewals where the client isn't even aware, and then they make you jump through hoops to get the renewal cancelled, all the while billing you for that time. Very shady.

SpamAssassin

SA correctly scored that particular message as ham.

[RvdH]

SURBL is a bit unclear what to use, eg:

in https://surbl.org/guidelines they say:

Extract domains or subdomains from those URIs. When using the wildcarded version of multi (e.g., for public DNS queries), it is not necessary to reduce domains to a specific level. Due to wildcards, subdomains of a blacklisted domain will match a blacklisted domain. For matching queries against the non-wildcarded version of multi, domains will need to be reduced in order to match the level of blacklisted domains. We recommend and use the wildcarded version of multi for DNS.

And i don't see any usage/description to differentiate between normal and wildcarded multi.surbl.org lookups


But in https://surbl.org/faqs#testpoints they state:

But note that only the last, two-level domain surbl-org-permanent-test-point.com will work as the base domain for a URI in a test message for SpamAssassin. This is because URIs with test.multi.surbl.org.multi.surbl.org, etc., won't be detected by most SURBL-using programs because they're supposed to be reduced down to a two-level domain which would be surbl.org for those.

[RvdH]

https://github.com/hmailserver/hmailser ... il/TLD.cpp

The tlds.txt, used to determine the toplevel TLD's for domainnames to extract in SURBL in %ProgramFiles%\hMailServer\Bin seems very outdated, anyone knows where to get a updated copy?

[RvdH]

https://github.com/hmailserver/hmailser ... il/TLD.cpp

The tlds.txt, used to determine the toplevel TLD's for domainnames to extract in SURBL in %ProgramFiles%\hMailServer\Bin seems very outdated, anyone knows where to get a updated copy?

https://www.surbl.org/static/two-level-tlds

[katip]

https://github.com/hmailserver/hmailser ... il/TLD.cpp

The tlds.txt, used to determine the toplevel TLD's for domainnames to extract in SURBL in %ProgramFiles%\hMailServer\Bin seems very outdated, anyone knows where to get a updated copy?

Mozilla maintains such a list:
https://publicsuffix.org/list/
https://en.wikipedia.org/wiki/Wikipedia ... uffix_List
https://wiki.mozilla.org/TLD_List

[RvdH]

https://github.com/hmailserver/hmailser ... il/TLD.cpp

The tlds.txt, used to determine the toplevel TLD's for domainnames to extract in SURBL in %ProgramFiles%\hMailServer\Bin seems very outdated, anyone knows where to get a updated copy?

Mozilla maintains such a list:
https://publicsuffix.org/list/
https://en.wikipedia.org/wiki/Wikipedia ... uffix_List
https://wiki.mozilla.org/TLD_List

Think the tlds.txt needs to hold two-level-tlds only, eg: not single-level, like .nl, .se, .com

[RvdH]

That makes SURBL testing almost useless.

Don't agree with that

Have been examining my BIND9 caching & forwarder DNS server logs, looks like for some list SA uses the full domain, and with other only top-level domain

Code: Select all

15-mei-2024 13:15:38.973 info: client @00000212C6B8CC38 127.0.0.1#65497 (vdhout.nl.multi.uribl.com): query: vdhout.nl.multi.uribl.com IN A +E(0) (127.0.0.1)
15-mei-2024 13:15:38.973 info: client @00000212C6B870C8 127.0.0.1#65497 (blog.vdhout.nl.multi.surbl.org): query: blog.vdhout.nl.multi.surbl.org IN A +E(0) (127.0.0.1)
15-mei-2024 13:15:38.973 info: client @00000212C6B8D948 127.0.0.1#65497 (vdhout.nl.fresh30.spameatingmonkey.net): query: vdhout.nl.fresh30.spameatingmonkey.net IN A +E(0) (127.0.0.1)
15-mei-2024 13:15:38.973 info: client @00000212C6B8E658 127.0.0.1#65497 (vdhout.nl.fresh15.spameatingmonkey.net): query: vdhout.nl.fresh15.spameatingmonkey.net IN A +E(0) (127.0.0.1)
15-mei-2024 13:15:38.973 info: client @00000212C6B90078 127.0.0.1#65497 (vdhout.nl.fresh.spameatingmonkey.net): query: vdhout.nl.fresh.spameatingmonkey.net IN A +E(0) (127.0.0.1)
15-mei-2024 13:15:38.979 info: client @00000212C6B91A98 127.0.0.1#65497 (vdhout.nl.urired.spameatingmonkey.net): query: vdhout.nl.urired.spameatingmonkey.net IN A +E(0) (127.0.0.1)
15-mei-2024 13:15:38.979 info: client @00000212C6B927A8 127.0.0.1#65497 (vdhout.nl.{APIKEY}.white.mail.abusix.zone): query: vdhout.nl.{APIKEY}.white.mail.abusix.zone IN A +E(0) (127.0.0.1)
15-mei-2024 13:15:38.979 info: client @00000212C6B8F368 127.0.0.1#65497 (vdhout.nl.dob.sibl.support-intelligence.net): query: vdhout.nl.dob.sibl.support-intelligence.net IN A +E(0) (127.0.0.1)
15-mei-2024 13:15:38.979 info: client @00000212C6B90D88 127.0.0.1#65497 (vdhout.nl.{APIKEY}.dblack.mail.abusix.zone): query: vdhout.nl.{APIKEY}.dblack.mail.abusix.zone IN A +E(0) (127.0.0.1)
15-mei-2024 13:15:38.979 info: client @00000212C6B90D88 127.0.0.1#65497 (vdhout.nl.dbl.nordspam.com): query: vdhout.nl.dbl.nordspam.com IN A +E(0) (127.0.0.1)
15-mei-2024 13:15:38.979 info: client @00000212C6D323E8 127.0.0.1#65497 (vdhout.nl.uribl.spameatingmonkey.net): query: vdhout.nl.uribl.spameatingmonkey.net IN A +E(0) (127.0.0.1)
15-mei-2024 13:15:38.979 info: client @00000212C6D323E8 127.0.0.1#65497 (blog.vdhout.nl.{APIKEY}.dbl.dq.spamhaus.net): query: blog.vdhout.nl.{APIKEY}.dbl.dq.spamhaus.net IN A +E(0) (127.0.0.1)
15-mei-2024 13:15:38.979 info: client @00000212C6D330F8 127.0.0.1#65497 (vdhout.nl.fresh10.spameatingmonkey.net): query: vdhout.nl.fresh10.spameatingmonkey.net IN A +E(0) (127.0.0.1)
15-mei-2024 13:15:38.979 info: client @00000212C6D316D8 127.0.0.1#65497 (vdhout.nl.{APIKEY}.zrd.dq.spamhaus.net): query: vdhout.nl.{APIKEY}.zrd.dq.spamhaus.net IN A +E(0) (127.0.0.1)

multi.uribl.com and dbl.dq.spamhaus.net seems to check both blog.vdhout.nl as vdhout.nl (the link used in the test message was https://blog.vdhout.nl), whereas others only check vdhout.nl
Remarkable enough list like spameatingmonkey.net, abusix.zone and others are custom additionally added SURBL server in my SA local.cf

Still need to find where my custom SURBL rules differentiate with the default rules, and also why for example dbl.dq.spamhaus.net vs zrd.dq.spamhaus.net differentiate

[palinka]

That makes SURBL testing almost useless.

Don't agree with that

Well, I now have some hard data to back up my claim.

Item 1 - As described above, the full domain (dsfsd32wsfes.blob.core.windows.net) hit on SURBL while the main/top domain (windows.net) did not. That's proof that SURBL does check subdomains separately from main/top domains. [edit - I have no way of knowing if SURBL disregarded any part of the "full" domain, per the info you posted earlier, or if they used examined the entire domain]

Item 2 - I checked my custom log for some statistics. Since I implemented my shortlink expansion checks (April 16):

hmailserver: 1 message detected
scripted full domain checks: 13 messages detected

That's proof that its more effective to use whole domains. Using the full domain is (so far) 1,200% more effective than only the main/top domain. That's a significant difference.

[palinka]

https://github.com/hmailserver/hmailser ... il/TLD.cpp

The tlds.txt, used to determine the toplevel TLD's for domainnames to extract in SURBL in %ProgramFiles%\hMailServer\Bin seems very outdated, anyone knows where to get a updated copy?

viewtopic.php?t=35044

I use the publicsuffix.org list. I made a script to convert it to a VBS regex string, and function that uses that regex to reduce any full domain down to the name+tld domain. Its the opposite of what we're looking at here, but you might find it interesting since we're on the topic.

[RvdH]

That makes SURBL testing almost useless.

Don't agree with that

Well, I now have some hard data to back up my claim.

Item 1 - As described above, the full domain (dsfsd32wsfes.blob.core.windows.net) hit on SURBL while the main/top domain (windows.net) did not. That's proof that SURBL does check subdomains separately from main/top domains. [edit - I have no way of knowing if SURBL disregarded any part of the "full" domain, per the info you posted earlier, or if they used examined the entire domain]

Item 2 - I checked my custom log for some statistics. Since I implemented my shortlink expansion checks (April 16):

hmailserver: 1 message detected
scripted full domain checks: 13 messages detected

That's proof that its more effective to use whole domains. Using the full domain is (so far) 1,200% more effective than only the main/top domain. That's a significant difference.

That result doesn't really agree with their own description/instructions given
viewtopic.php?p=253197#p253197

The hit on dsfsd32wsfes.blob.core.windows.net, should also hit on windows.net if i read/understand their description properly

[palinka]

Don't agree with that

Well, I now have some hard data to back up my claim.

Item 1 - As described above, the full domain (dsfsd32wsfes.blob.core.windows.net) hit on SURBL while the main/top domain (windows.net) did not. That's proof that SURBL does check subdomains separately from main/top domains. [edit - I have no way of knowing if SURBL disregarded any part of the "full" domain, per the info you posted earlier, or if they used examined the entire domain]

Item 2 - I checked my custom log for some statistics. Since I implemented my shortlink expansion checks (April 16):

hmailserver: 1 message detected
scripted full domain checks: 13 messages detected

That's proof that its more effective to use whole domains. Using the full domain is (so far) 1,200% more effective than only the main/top domain. That's a significant difference.

That result doesn't really agree with their own description/instructions given
viewtopic.php?p=253197#p253197

The hit on dsfsd32wsfes.blob.core.windows.net, should also hit on windows.net if i read/understand their description properly

I just tested every level of that particular domain on multirbl.valli.org. Results:

Code: Select all

hits	domain
====	======
0	windows.net
0	core.windows.net
0	blob.core.windows.net
3	dsfsd32wsfes.blob.core.windows.net

The hits were on Abusix, Invaluement and SURBL.org.

This is conclusive proof that SURBL (and others) check beyond 1 level of subdomains.

Therefore, I humbly request that hmailserver get updated to check whole domains on SURBL testing.

[palinka]

https://github.com/hmailserver/hmailser ... il/TLD.cpp

The tlds.txt, used to determine the toplevel TLD's for domainnames to extract in SURBL in %ProgramFiles%\hMailServer\Bin seems very outdated, anyone knows where to get a updated copy?

Mozilla maintains such a list:
https://publicsuffix.org/list/
https://en.wikipedia.org/wiki/Wikipedia ... uffix_List
https://wiki.mozilla.org/TLD_List

Think the tlds.txt needs to hold two-level-tlds only, eg: not single-level, like .nl, .se, .com

There are 3 and even more level TLDs. Example:

Code: Select all

// gov.br 26 states + df https://en.wikipedia.org/wiki/States_of_Brazil
ac.gov.br
al.gov.br
am.gov.br
ap.gov.br
ba.gov.br
...

There are six 6-part TLDs. Example:

Code: Select all

s3.dualstack.cn-north-1.amazonaws.com.cn

There are 193 5-part TLDs
There are 470 4-part TLDs
There are 2,420 3-part TLDs

I counted using regex in notepad++.

[RvdH]

Mozilla maintains such a list:
https://publicsuffix.org/list/
https://en.wikipedia.org/wiki/Wikipedia ... uffix_List
https://wiki.mozilla.org/TLD_List

Think the tlds.txt needs to hold two-level-tlds only, eg: not single-level, like .nl, .se, .com

There are 3 and even more level TLDs. Example:

Code: Select all

// gov.br 26 states + df https://en.wikipedia.org/wiki/States_of_Brazil
ac.gov.br
al.gov.br
am.gov.br
ap.gov.br
ba.gov.br
...

There are six 6-part TLDs. Example:

Code: Select all

s3.dualstack.cn-north-1.amazonaws.com.cn

There are 193 5-part TLDs
There are 470 4-part TLDs
There are 2,420 3-part TLDs

I counted using regex in notepad++.

Probably exactly why hmailserver only checks the top-level domain
https://www.surbl.org/static/three-level-tlds (4, 5 and 6 level lists are non-existent on surbl.org)

[RvdH]

For SA, magic word seems to be notrim (and maybe also usefull domains_only?)


https://spamassassin.apache.org/full/4. ... IDNSBL.txt

tflags NAME_OF_RULE notrim
The full hostname component will be matched against the named
"urirhsbl"/"urirhssub" rule, instead of using the trimmed domain.
This works better, but the specific uribl must support this method.

Code: Select all

ifplugin Mail::SpamAssassin::Plugin::URIDNSBL
    urirhsbl  URIBL_AMI_DBLACK    <APIKEY>.dblack.mail.abusix.zone. A
    body      URIBL_AMI_DBLACK    eval:check_uridnsbl('URIBL_AMI_DBLACK')
    describe  URIBL_AMI_DBLACK    Contains a spam URL listed in the Abusix Mail Intelligence domain blocklist
    score     URIBL_AMI_DBLACK    3.0
    tflags    URIBL_AMI_DBLACK    net

    urirhssub URIBL_AMI_WHITE     <APIKEY>.white.mail.abusix.zone. A 127.0.2.1
    body      URIBL_AMI_WHITE     eval:check_uridnsbl('URIBL_AMI_WHITE')
    describe  URIBL_AMI_WHITE     Contains a domain listed in the Abusix Mail Intelligence domain whitelist
    score     URIBL_AMI_WHITE     -0.1
    tflags    URIBL_AMI_WHITE     nice net
endif

Code: Select all

ifplugin Mail::SpamAssassin::Plugin::URIDNSBL
    urirhsbl  URIBL_AMI_DBLACK    <APIKEY>.dblack.mail.abusix.zone. A
    body      URIBL_AMI_DBLACK    eval:check_uridnsbl('URIBL_AMI_DBLACK')
    describe  URIBL_AMI_DBLACK    Contains a spam URL listed in the Abusix Mail Intelligence domain blocklist
    score     URIBL_AMI_DBLACK    3.0
    tflags    URIBL_AMI_DBLACK    net notrim

    urirhssub URIBL_AMI_WHITE     <APIKEY>.white.mail.abusix.zone. A 127.0.2.1
    body      URIBL_AMI_WHITE     eval:check_uridnsbl('URIBL_AMI_WHITE')
    describe  URIBL_AMI_WHITE     Contains a domain listed in the Abusix Mail Intelligence domain whitelist
    score     URIBL_AMI_WHITE     -0.1
    tflags    URIBL_AMI_WHITE     nice net notrim
endif

[palinka]

In any case, it doesn't matter because the full domain should be sent to SURBL. There's no need whatsoever to parse it. Let SURBL parse it however they like. The worst thing that can happen is they return NX domain. They won't even yell at you or slap your hand.

[RvdH]

In any case, it doesn't matter because the full domain should be sent to SURBL. There's no need whatsoever to parse it. Let SURBL parse it however they like. The worst thing that can happen is they return NX domain. They won't even yell at you or slap your hand.

Uh, but how effective would that be knowing not all SURBL server support full url's

[palinka]

In any case, it doesn't matter because the full domain should be sent to SURBL. There's no need whatsoever to parse it. Let SURBL parse it however they like. The worst thing that can happen is they return NX domain. They won't even yell at you or slap your hand.

Uh, but how effective would that be knowing not all SURBL server support full url's

viewtopic.php?p=253206#p253206

Very, apparently.

[RvdH]

In any case, it doesn't matter because the full domain should be sent to SURBL. There's no need whatsoever to parse it. Let SURBL parse it however they like. The worst thing that can happen is they return NX domain. They won't even yell at you or slap your hand.

Uh, but how effective would that be knowing not all SURBL server support full url's

viewtopic.php?p=253206#p253206

Very, apparently.

That is YOUR opinion, entirely based on a single hit

You do realize if you lookup the full url against a SURBL server that only list two-level domains you likely never get any hits? Hence the warning in SA URIDNSBL, eg:

tflags NAME_OF_RULE notrim
The full hostname component will be matched against the named
"urirhsbl"/"urirhssub" rule, instead of using the trimmed domain.
This works better, but the specific uribl must support this method.

[palinka]

That is YOUR opinion, entirely based on a single hit

One hit is all it takes to prove that the most important SURBL (and others) look deeper than 2 levels. There are probably more - probably a lot more that look deeper than 2 levels - that simply didn't list that particular URL at the time I checked Valli multirbl.

You do realize if you lookup the full url against a SURBL server that only list two-level domains you likely never get any hits? Hence the warning in SA URIDNSBL, eg:

tflags NAME_OF_RULE notrim
The full hostname component will be matched against the named
"urirhsbl"/"urirhssub" rule, instead of using the trimmed domain.
This works better, but the specific uribl must support this method.

How about instead of speculating, go and find one that doesn't support it? At least 3 do, including the most important one. And Spamhaus DBL also supports it - we just don't know if they trim the domain or not, but that's irrelevant to actually using the service.

Also, any service that only looks at 2 levels is by definition ignoring URLs from the 8,230 TLDs that have 2 parts or more. Maybe 30 years ago this kind of SURBL service was common and made sense, but not in current year. In fact, I bet that NOTRIM documentation is at least 15 years old. Its not necessary to update it because its still true, even if its not relevant.

1,200% more effective. Its a no-brainer.

1,200% is not speculation. The most important RBLs looking deeper than 2 levels is not speculation. The only speculation is your assertion that maybe somewhere out in the forgotten zombie internet there's a very old RBL that only 3 people use that still can't accept multi-level domains as input.

Anyway, what is the downside? 3 people that don't even use hmailserver could lose their beloved ancient RBL effectiveness *IF* they were to switch to hmailserver. Maybe. In the future. The benefit clearly outweighs the cost here.

[RvdH]

And now you literally are posting nothing more then YOUR assumptions... talking about a no-brainer, you classify seamlessly

anyway, good luck!

hint: search for eval:check_uridnsbl in all *.cf of SA, plenty even in default rules that do not use/support notrim, like validity.com, uribl.com
abusix.zone requires a API KEY and is limited up to a certain amount of queries so that list isn't usable for everyone, and even then in their own proved SA rules they do not use notrim

[palinka]

And now you literally are posting nothing more then YOUR assumptions... talking about a no-brainer, you classify seamlessly

anyway, good luck!

hint: search for eval:check_uridnsbl in all *.cf of SA, plenty even in default rules that do not use/support notrim, like validity.com, uribl.com
abusix.zone requires a API KEY and is limited up to a certain amount of queries so that list isn't usable for everyone, and even then in their own proved SA rules they do not use notrim

"Do not use" and "Do not support" are very different things. How do we know those that don't use notrim actually don't support it? You haven't proven a single thing, while I have. And on top of that, we're talking about hmailserver where only a tiny minority of users will use any SURBL other than surbl.org and dbl.spamhaus.net.

But if you don't want a 1,200% improvement, well, I guess that's your call. It can still be scripted with excellent results, and it cannot be denied that a 1,200% improvement is significant. That's no assumption - that's based on empirical evidence. 1,200% improvement over 30 days. I bet that improves over time.

That's about all I have to say on the matter. The data speaks for itself.

[RvdH]

Hilarious

[palinka]

Hilarious

Expected

[RvdH]

Trim Host Parts
If NO, hostnames will be considered invalid. If YES, the host part will be trimmed.

https://admin.uribl.com/?section=lookup

[palinka]

Trim Host Parts
If NO, hostnames will be considered invalid. If YES, the host part will be trimmed.

https://admin.uribl.com/?section=lookup

Using the example domain - dsfsd32wsfes.blob.core.windows.net:
No Trim - Not listed
Yes Trim - Listed on URIBL white

Is that the response you were expecting? Its not helping your case.

[RvdH]

Trim Host Parts
If NO, hostnames will be considered invalid. If YES, the host part will be trimmed.

https://admin.uribl.com/?section=lookup

Using the example domain - dsfsd32wsfes.blob.core.windows.net:
No Trim - Not listed
Yes Trim - Listed on URIBL white

Is that the response you were expecting? Its not helping your case.

So you now reached the denial fase?
It in described in their test-tool description ffs + notrim missing for their SA rules (1 + 1 = ?)

[RvdH]

https://admin.uribl.com/?section=lookup

Using the example domain - dsfsd32wsfes.blob.core.windows.net:
No Trim - Not listed
Yes Trim - Listed on URIBL white

Is that the response you were expecting? Its not helping your case.

So you now reached the denial fase?
It in described in their test-tool description ffs + notrim missing for their SA rules (simple math that proves my case)

Another one who only lists trimmed domains
fresh*.spameatingmonkey.net (eg: SEM-FRESHZERO,SEM-FRESH,SEM-FRESH10,SEM-FRESH15,SEM-FRESH30)

SEM-URIRED and SEM-URI take full urls, but only trimmed URLs' are listed, see MultiRBL.valli.org, probably not looking for a exact match but if it contains/ends with listed domain)

[palinka]

What am I in denial of? That trimming produces false negatives on some surbl servers?

Anyway, I already said I would do it my way, because its 1,200% more effective. And you can do it any way you want. Change the code, don't change the code... I don't care. I already put my 1,200% more effective method into use a month ago. Its free to use under the palinka license - you may copy/paste/change however you like - or just ignore it. Its up to you.

[RvdH]

In any case, it doesn't matter because the full domain should be sent to SURBL. There's no need whatsoever to parse it. Let SURBL parse it however they like. The worst thing that can happen is they return NX domain. They won't even yell at you or slap your hand.

You stated that simply using the full url would solve ALL issues, which clearly is not the case and is plain wrong as this entirely depends on the SURBL server capabilities, some accept the full URL, some do not as shown, some accept both but list only the trimmed urls (and therefor could trigger double scoring!)

You can agree that by using the full url can also produce false negatives or double positives on some surbl servers (eg: the ones that expect trimmed urls)?
The changes required for SURBL checking are therefor not that straightforward as you claimed, eg: fixing one, breaking the other meaning your math doesn't add up

[RvdH]

The way i see it we can only make this work if there would be a conditional switch (boolean) added to SURBL server list in a database/GUI, similar to the notrim argument in spamassassin, that way one could/would be able to specify if the full or trimmed down version of a domain has to be used against a SURBL server, as checking both full and trimmed down URL can result in unwanted (double) scoring, see the SEM-URI, SEM-URIRED examples given above

Is it worth the change? I doubt it in current hMailserver state... and therefor i would suggest to keep it the what it is, eg: let hmailserver check the trimmed down list, and additionally use spamassassin to check against the notrim list or use script (until VBScript support disappears)

[RvdH]

The way i see it we can only make this work if there would be a conditional switch (boolean) added to SURBL server list in a database/GUI, similar to the notrim argument in spamassassin, that way one could/would be able to specify if the full or trimmed down version of a domain has to be used against a SURBL server, as checking both full and trimmed down URL can result in unwanted (double) scoring, see the SEM-URI, SEM-URIRED examples given above

Is it worth the change? I doubt it in current hMailserver state... and therefor i would suggest to keep it the what it is, eg: let hmailserver check the trimmed down list, and additionally use spamassassin to check against the notrim list or use script (until VBScript support disappears)

Correction: Double scoring within hMailserver seems to be impossible as the SURBL check seems to break out the loop if a match is found against that particular SURBL server, eg: not checking remaining extracted URL's. If i can confirm above, possibility would be to check both trimmed and non-trimmed URL's.... downside this will at least double (but could be more!) the amount of lookups, especially for SURBL server with query limits, like abusix this could have impact and push you above the query limits, example:

This is current behavior:

Code: Select all

"DEBUG"	5876	"2024-05-17 07:02:25.459"	"SURBL: Execute"
"DEBUG"	5876	"2024-05-17 07:02:25.459"	"SURBL: Found URL: primera.nl"
"DEBUG"	5876	"2024-05-17 07:02:25.459"	"SURBL: Found URL: w3.org"
"DEBUG"	5876	"2024-05-17 07:02:25.459"	"SURBL: Found URL: typekit.net"
"DEBUG"	5876	"2024-05-17 07:02:25.459"	"SURBL: Found URL: closecontact.eu"
"DEBUG"	5876	"2024-05-17 07:02:25.459"	"SURBL: 4 unique addresses found."

Same message, check both trimmed and non-trimmed URL's

Code: Select all

"DEBUG"	5876	"2024-05-17 07:02:25.459"	"SURBL: Execute"
"DEBUG"	5876	"2024-05-17 07:02:25.459"	"SURBL: Found URL: view.email.primera.nl"
"DEBUG"	5876	"2024-05-17 07:02:25.459"	"SURBL: Found URL: primera.nl"
"DEBUG"	5876	"2024-05-17 07:02:25.459"	"SURBL: Found URL: click.email.primera.nl"
"DEBUG"	5876	"2024-05-17 07:02:25.459"	"SURBL: Found URL: w3.org"
"DEBUG"	5876	"2024-05-17 07:02:25.459"	"SURBL: Found URL: use.typekit.net"
"DEBUG"	5876	"2024-05-17 07:02:25.459"	"SURBL: Found URL: typekit.net"
"DEBUG"	5876	"2024-05-17 07:02:25.459"	"SURBL: Found URL: closecontact.eu"
"DEBUG"	5876	"2024-05-17 07:02:25.459"	"SURBL: Found URL: image.email.primera.nl"
"DEBUG"	5876	"2024-05-17 07:02:25.459"	"SURBL: 8 unique addresses found."

eg: instead of only checking primera.nl, it checks 3 more subdomains within the primera.nl top-level domain

[RvdH]

While on this topic, modified abusix rule set for SpamAssassin that both checks trimmed URL's (default) and full URL's (but only scores one time, due to meta rule)

Code: Select all

ifplugin Mail::SpamAssassin::Plugin::DNSEval
    header    __RCVD_IN_AMI       eval:check_rbl('ami', '<APIKEY>.combined.mail.abusix.zone.')
    describe  __RCVD_IN_AMI       Received via a relay in Abusix Mail Intelligence
    tflags    __RCVD_IN_AMI       net

    header    RCVD_IN_AMI_BLACK   eval:check_rbl_sub('ami', '^127\.0\.0\.(?:[23]|200)$')
    describe  RCVD_IN_AMI_BLACK   Received via a relay in Abusix Mail Intelligence Black
    score     RCVD_IN_AMI_BLACK   3.0
    tflags    RCVD_IN_AMI_BLACK   net

    header    RCVD_IN_AMI_EXPLOIT eval:check_rbl_sub('ami', '127.0.0.4')
    describe  RCVD_IN_AMI_EXPLOIT Received via a relay in Abusix Mail Intelligence Exploit
    score     RCVD_IN_AMI_EXPLOIT 3.0
    tflags    RCVD_IN_AMI_EXPLOIT net

    header    RCVD_IN_AMI_DYN     eval:check_rbl('ami-lastexternal', '<APIKEY>.combined.mail.abusix.zone.', '^127\.0\.0\.1[12]$')
    describe  RCVD_IN_AMI_DYN     Received via a relay in Abusix Mail Intelligence Dynamic
    score     RCVD_IN_AMI_DYN     3.0
    tflags    RCVD_IN_AMI_DYN     net

    header    RCVD_IN_AMI_WHITE   eval:check_rbl('ami-firsttrusted', '<APIKEY>.combined.mail.abusix.zone.', '127.0.2.1')
    describe  RCVD_IN_AMI_WHITE   Received via a relay in Abusix Mail Intelligence White
    score     RCVD_IN_AMI_WHITE   -0.1
    tflags    RCVD_IN_AMI_WHITE   nice net
endif

ifplugin Mail::SpamAssassin::Plugin::URIDNSBL
    urirhsbl  URIBL_AMI_DBLACK    <APIKEY>.dblack.mail.abusix.zone. A
    body      URIBL_AMI_DBLACK    eval:check_uridnsbl('URIBL_AMI_DBLACK')
    describe  URIBL_AMI_DBLACK    Contains a spam URL listed in the Abusix Mail Intelligence domain blocklist
    score     URIBL_AMI_DBLACK    3.0
    tflags    URIBL_AMI_DBLACK    net

    urirhssub URIBL_AMI_WHITE     <APIKEY>.white.mail.abusix.zone. A 127.0.2.1
    body      URIBL_AMI_WHITE     eval:check_uridnsbl('URIBL_AMI_WHITE')
    describe  URIBL_AMI_WHITE     Contains a domain listed in the Abusix Mail Intelligence domain whitelist
    score     URIBL_AMI_WHITE     -0.1
    tflags    URIBL_AMI_WHITE     nice net
	
	
    # fullhost
	
    urirhsbl  __URIBL_AMI_DBLACK_FULLHOST    <APIKEY>.dblack.mail.abusix.zone. A
    body      __URIBL_AMI_DBLACK_FULLHOST    eval:check_uridnsbl('URIBL_AMI_DBLACK_FULLHOST')
    tflags    __URIBL_AMI_DBLACK_FULLHOST    net notrim

    urirhssub __URIBL_AMI_WHITE_FULLHOST     <APIKEY>.white.mail.abusix.zone. A 127.0.2.1
    body      __URIBL_AMI_WHITE_FULLHOST     eval:check_uridnsbl('URIBL_AMI_WHITE_FULLHOST')
    tflags    __URIBL_AMI_WHITE_FULLHOST     nice net notrim
	
    meta      URIBL_AMI_DBLACK_FULLHOST      ( __URIBL_AMI_DBLACK_FULLHOST && !URIBL_AMI_DBLACK )
    describe  URIBL_AMI_DBLACK_FULLHOST      Contains a spam URL listed in the Abusix Mail Intelligence domain blocklist
    score     URIBL_AMI_DBLACK_FULLHOST      3.0

    meta      URIBL_AMI_WHITE_FULLHOST       ( __URIBL_AMI_WHITE_FULLHOST && !URIBL_AMI_WHITE )
    describe  URIBL_AMI_WHITE_FULLHOST       Contains a domain listed in the Abusix Mail Intelligence domain whitelist
    score     URIBL_AMI_WHITE_FULLHOST       -0.1
endif

[mikedibella]

I get @plinka's point and want to point out an increasing trend I'm seeing in UCE received at my server.

Increasingly, spammers are using storage buckets on Azure, AWS and GCP to host content. The FQDNs for these buckets are on 2nd level domains that it would be imprudent to blacklist because they will also host ham.

Bottom line, there is a very compelling use case for FQDN blacklisting capability.

[RvdH]

I think you know, but in case you do not I'll post it once again: https://www.hmailserver.com/state
Not interested in flogging a dead horse...

If you serious about fighting spam, use SpamAssassin... no need to reinvent wheel, SpamAssassin is able to handle non-trimmed URI's just fine

[palinka]

I'm serious about fighting spam. https://hmailserver.com/forum/viewtopic.php?f=9&t=41891

I'm like Bruce Lee dragon style spam fighting. That spam doesn't stand a chance. Pussy ass bitch spam goes down in flames on a daily basis. I hardly get any spam at all. That spam is too pussy ass bitch to try to knock on the door. They know I'm going to numchuck those bitches straight back to whatever dirty hole they came from.

[RvdH]

It was a reply to mikedibella though...

that said: SpamAsassasin handles/expands shorturls as well... once again no need to reinvent the wheel

[RvdH]

Had some spare time over the weekend, only minimal changes (not even changes, line added actually) are needed to make SURBL take full and trimmed down domain

Code: Select all

"DEBUG"	3656	"2024-08-19 11:43:25.338"	"SURBL: Found URL: contact.dela.nl"
"DEBUG"	3656	"2024-08-19 11:43:25.338"	"SURBL: Found URL: dela.nl"
"DEBUG"	3656	"2024-08-19 11:43:25.338"	"SURBL: Found URL: w3.org"
"DEBUG"	3656	"2024-08-19 11:43:25.338"	"SURBL: 3 unique addresses found."

Code: Select all

"DEBUG"	3656	"2024-08-19 11:04:18.869"	"SURBL: Found URL: click.tomtom.com"
"DEBUG"	3656	"2024-08-19 11:04:18.869"	"SURBL: Found URL: tomtom.com"
"DEBUG"	3656	"2024-08-19 11:04:18.869"	"SURBL: Found URL: dl.asnapieu.com"
"DEBUG"	3656	"2024-08-19 11:04:18.869"	"SURBL: Found URL: asnapieu.com"
"DEBUG"	3656	"2024-08-19 11:04:18.869"	"SURBL: Found URL: asemailmgmteu.com"
"DEBUG"	3656	"2024-08-19 11:04:18.869"	"SURBL: Found URL: pages.airsp.eu"
"DEBUG"	3656	"2024-08-19 11:04:18.869"	"SURBL: Found URL: airsp.eu"
"DEBUG"	3656	"2024-08-19 11:04:18.869"	"SURBL: Found URL: w3.org"
"DEBUG"	3656	"2024-08-19 11:04:18.869"	"SURBL: Found URL: download.tomtom.com"
"DEBUG"	3656	"2024-08-19 11:04:18.869"	"SURBL: Found URL: stylecampaign.com"
"DEBUG"	3656	"2024-08-19 11:04:18.869"	"SURBL: Found URL: library.tomtom.com"
"DEBUG"	3656	"2024-08-19 11:04:18.869"	"SURBL: 11 unique addresses found."

These could ideally be ignored i think, those are triggered on DOCTYPE from some HTML mails

Code: Select all

"DEBUG"	3656	"2024-08-19 10:31:48.786"	"SURBL: Found URL: schemas.microsoft.com"
"DEBUG"	3656	"2024-08-19 10:31:48.786"	"SURBL: Found URL: microsoft.com"
"DEBUG"	3656	"2024-08-19 10:31:48.786"	"SURBL: Found URL: w3.org"
"DEBUG"	3656	"2024-08-19 10:31:48.786"	"SURBL: 3 unique addresses found."

the Regex which is used right now, ignores www. prefix, no expert here but if a domain is blocked i would assume SURBL providers will put in the domain without www. no? (this also save a lookup)

Code: Select all

(?:(?>https?)?(?>:\/\/|\%3A\%2F\%2F))(?:www\.)?([a-z0-9\-\.\=\r\n]+)

https://github.com/hmailserver/hmailser ... pp#L67-L74

Code: Select all

...
            // Clean the URL from linefeeds
            CleanURL_(sURL);

            if (addresses.find(sURL) == addresses.end())
            {
               String slogMessage;
               slogMessage.Format(_T("SURBL: Found URL: %s"), sURL.c_str());
               LOG_DEBUG(slogMessage);

               addresses.insert(sURL);

               if (addresses.size() > maxURLsToProcess)
               {
                  break;
               }
            }

            // Trim away top domain
            if (!CleanHost_(sURL))
            {
               sRemainingSearchSpace = matches.suffix();
               continue;
            }
...

[glenluo]

I won't use hmail SURBL,as it can not check the result value,even query return public IP, or 127.255.255.255,hmail will deem it as listed.
The best is SURBL should has Expected result like RBL function.

[RvdH]

I won't use hmail SURBL,as it can not check the result value,even query return public IP, or 127.255.255.255,hmail will deem it as listed.
The best is SURBL should has Expected result like RBL function.

Or simply never query DNS with public DNS server when running a mailserver/spamfilters... or even better use a local caching/forwarding DNS server
If you configure to query DNS with public DNS you do not qualify as a good system admin (in the context when running mailserver/spamfilter)

These could ideally be ignored i think, those are triggered on DOCTYPE from some HTML mails

Code: Select all

"DEBUG"	3656	"2024-08-19 10:31:48.786"	"SURBL: Found URL: schemas.microsoft.com"
"DEBUG"	3656	"2024-08-19 10:31:48.786"	"SURBL: Found URL: microsoft.com"
"DEBUG"	3656	"2024-08-19 10:31:48.786"	"SURBL: Found URL: w3.org"
"DEBUG"	3656	"2024-08-19 10:31:48.786"	"SURBL: 3 unique addresses found."

Code: Select all

            // ignore some HTML doctype url's
            sRegex = "(?=.*)(w3\\.org|schemas\\.microsoft\\.com)";
            boost::wregex expr(sRegex, boost::wregex::icase);
            if (boost::regex_match(sURL, expr))
            {
               sRemainingSearchSpace = matches.suffix();
               continue;
            }

[RvdH]

https://regex101.com/r/eXLnIK/1

Code: Select all

            // ignore some HTML doctype url's, which are default in for example Outlook composed mails
            // www.w3.org
            // www.w3c.org
            // schemas.microsoft.com
            // fonts.googleapis.com
            // fonts.gstatic.com
            sRegex = "(?=.*)(w3(?:c)?\\.org|(?:schemas\\.microsoft|fonts\\.(?:googleapis|gstatic))\\.com)";
            boost::wregex expr(sRegex, boost::wregex::icase);
            if (boost::regex_match(sURL, expr))
            {
               sRemainingSearchSpace = matches.suffix();
               continue;
            }

These are, while monitoring, the most frequent hits on what can be considered 'safe' static/common URI's used in HTML mails

To test this build: *.71 / *.14