URL send to SURBL

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
tunis
Normal user
Normal user
Posts: 245
Joined: 2015-01-05 20:22
Location: Sweden

URL send to SURBL

Post by tunis » 2020-01-23 09:32

Hello, I saw that the URL for SURBL lookup sometimes is faulty.

Here is an example from my log.

Code: Select all

"DEBUG"	3168	"2020-01-23 08:09:03.919"	"SURBL: Execute"
"DEBUG"	3168	"2020-01-23 08:09:03.919"	"SURBL: Found URL: webbdagarna.se"
"DEBUG"	3168	"2020-01-23 08:09:03.919"	"SURBL: Found URL: trippus.se"
"DEBUG"	3168	"2020-01-23 08:09:03.919"	"SURBL: Found URL: rebelandbird.com)conversionista"
"DEBUG"	3168	"2020-01-23 08:09:03.935"	"SURBL: Found URL: conversionista.com)bleckout"
"DEBUG"	3168	"2020-01-23 08:09:03.935"	"SURBL: Found URL: bleckout.co)"
"DEBUG"	3168	"2020-01-23 08:09:03.935"	"SURBL: Found URL: nitor.com"
"DEBUG"	3168	"2020-01-23 08:09:03.935"	"SURBL: Found URL: storykit.io"
"DEBUG"	3168	"2020-01-23 08:09:03.935"	"SURBL: Found URL: umbraco.com)"
"DEBUG"	3168	"2020-01-23 08:09:03.935"	"SURBL: Found URL: zenloop.com"
"DEBUG"	3168	"2020-01-23 08:09:03.935"	"SURBL: Found URL: siteimprove.com)adform"
"DEBUG"	3168	"2020-01-23 08:09:03.935"	"SURBL: Found URL: adform.com)"
"DEBUG"	3168	"2020-01-23 08:09:03.935"	"SURBL: Found URL: ugglamassage.se)"
"DEBUG"	3168	"2020-01-23 08:09:03.935"	"SURBL: Found URL: stickerapp.se)"
"DEBUG"	3168	"2020-01-23 08:09:03.935"	"SURBL: Found URL: kakdegsfabriken.se)"
"DEBUG"	3168	"2020-01-23 08:09:03.935"	"SURBL: Found URL: idg.se"
"DEBUG"	3168	"2020-01-23 08:09:03.935"	"SURBL: Found URL: twitter.com"
"DEBUG"	3168	"2020-01-23 08:09:03.935"	"SURBL: 16 unique addresses found."
"DEBUG"	3168	"2020-01-23 08:09:03.935"	"SURBL: Lookup: adform.com).multi.surbl.org"
"DEBUG"	3168	"2020-01-23 08:09:03.935"	"SURBL: Lookup: bleckout.co).multi.surbl.org"
"DEBUG"	3168	"2020-01-23 08:09:03.935"	"SURBL: Lookup: conversionista.com)bleckout.multi.surbl.org"
"DEBUG"	3168	"2020-01-23 08:09:03.935"	"SURBL: Lookup: idg.se.multi.surbl.org"
"DEBUG"	3168	"2020-01-23 08:09:04.075"	"SURBL: Lookup: kakdegsfabriken.se).multi.surbl.org"
"DEBUG"	3168	"2020-01-23 08:09:04.075"	"SURBL: Lookup: nitor.com.multi.surbl.org"
"DEBUG"	3168	"2020-01-23 08:09:04.207"	"SURBL: Lookup: rebelandbird.com)conversionista.multi.surbl.org"
"DEBUG"	3168	"2020-01-23 08:09:04.207"	"SURBL: Lookup: siteimprove.com)adform.multi.surbl.org"
"DEBUG"	3168	"2020-01-23 08:09:04.207"	"SURBL: Lookup: stickerapp.se).multi.surbl.org"
"DEBUG"	3168	"2020-01-23 08:09:04.207"	"SURBL: Lookup: storykit.io.multi.surbl.org"
"DEBUG"	3168	"2020-01-23 08:09:04.287"	"SURBL: Lookup: trippus.se.multi.surbl.org"
"DEBUG"	3168	"2020-01-23 08:09:04.456"	"SURBL: Lookup: twitter.com.multi.surbl.org"
"DEBUG"	3168	"2020-01-23 08:09:04.563"	"SURBL: Lookup: ugglamassage.se).multi.surbl.org"
"DEBUG"	3168	"2020-01-23 08:09:04.563"	"SURBL: Lookup: umbraco.com).multi.surbl.org"
"DEBUG"	3168	"2020-01-23 08:09:04.563"	"SURBL: Lookup: webbdagarna.se.multi.surbl.org"
"DEBUG"	3168	"2020-01-23 08:09:04.610"	"SURBL: Lookup: zenloop.com.multi.surbl.org"
"DEBUG"	3168	"2020-01-23 08:09:04.790"	"SURBL: Match not found"
Some URL contains ")", that not should be there.

Is anybody else got this in the logs?
HMS 5.6.8 B2494.25 on Windows Server 2019 Core VM.
HMS 5.6.8 B2494.24 on Windows Server 2016 Core VM.
HMS 5.6.7 B2425.16 on Windows Server 2012 R2 Core VM.

User avatar
jimimaseye
Moderator
Moderator
Posts: 8518
Joined: 2011-09-08 17:48

Re: URL send to SURBL

Post by jimimaseye » 2020-01-23 09:58

Ideally, you should find the email that contains the link in the body to see how that link is formed. Then we can determine if it is a parsing issue (and therefore potentially problematic) or just a display issue with the actual url being correctly parsed and checked as required.

Can you find examples?

[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

tunis
Normal user
Normal user
Posts: 245
Joined: 2015-01-05 20:22
Location: Sweden

Re: URL send to SURBL

Post by tunis » 2020-01-23 11:07

I checked the email and all links are like this one.

Code: Select all

href="https://dmd.idg.se/x/c/?LU.RbsMgDPyVvtCnqQ0hbMokq2q3L1gf9jgRQAldDJEDivb1DLLJtnQn_3w6DbLNCB3vmz7PILnseSbgont_EXmCKcZlfT2ft207kR3srLwZHJmTDnhJEb90WmNAJm.bwuT7v.BJtsd9qXBRbvTwaYfBqFGRV6xt7jHo7ynMWHAtVlvcH6kA.1Mv1vhgb4JdO.KBCndlFgojKUQb999rSKQtWEM7RWtcQrCo3PznHXy0PkI1aHgrsoGPGuBwPNxKgF8A38"
This redirects to rebelandbird.com "rebelandbird.com)conversionista".
HMS 5.6.8 B2494.25 on Windows Server 2019 Core VM.
HMS 5.6.8 B2494.24 on Windows Server 2016 Core VM.
HMS 5.6.7 B2425.16 on Windows Server 2012 R2 Core VM.

User avatar
mattg
Moderator
Moderator
Posts: 20786
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: URL send to SURBL

Post by mattg » 2020-01-28 01:20

I noticed some with square braces too

]


hMailserver should remove all braces / brackets before sending to SURBL testing
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
RvdH
Senior user
Senior user
Posts: 1042
Joined: 2008-06-27 14:42
Location: Netherlands

Re: URL send to SURBL

Post by RvdH » 2020-01-28 13:47

What HMS version?

I think te regex used to detect URL's for SURBL checking is faulty,
https://github.com/hmailserver/hmailser ... BL.cpp#L43

Or does BOOST use some weird/custom regex?
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
SorenR
Senior user
Senior user
Posts: 3562
Joined: 2006-08-21 15:38
Location: Denmark

Re: URL send to SURBL

Post by SorenR » 2020-01-28 16:03

RvdH wrote:
2020-01-28 13:47
What HMS version?

I think te regex used to detect URL's for SURBL checking is faulty,
https://github.com/hmailserver/hmailser ... BL.cpp#L43

Or does BOOST use some weird/custom regex?
https://www.regular-expressions.info/boost.html
SørenR.

“With age comes wisdom, but sometimes age comes alone.”
- Oscar Wilde

User avatar
RvdH
Senior user
Senior user
Posts: 1042
Joined: 2008-06-27 14:42
Location: Netherlands

Re: URL send to SURBL

Post by RvdH » 2020-01-28 20:58

OK, as it seems BOOST needs some weird extra escaping

But i would say, using a regex like: (?:https?:\\/\\/)([^\\?&><\\[\\(#\\\\ \\\"'\\/\\)\\]]*) would solve the issues, see example here: https://regexr.com/4t4ap
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

palinka
Senior user
Senior user
Posts: 1915
Joined: 2017-09-12 17:57

Re: URL send to SURBL

Post by palinka » 2020-01-28 21:28

RvdH wrote:
2020-01-28 20:58
OK, as it seems BOOST needs some weird extra escaping

But i would say, using a regex like: (?:https?:\\/\\/)([^\\?&><\\[\\(#\\\\ \\\"'\\/\\)\\]]*) would solve the issues, see example here: https://regexr.com/4t4ap
Could the X-Spam-Report (bad) formatting also be caused by a malformed regex? It appears to delete line breaks from the report.

User avatar
RvdH
Senior user
Senior user
Posts: 1042
Joined: 2008-06-27 14:42
Location: Netherlands

Re: URL send to SURBL

Post by RvdH » 2020-01-28 21:44

palinka wrote:
2020-01-28 21:28
RvdH wrote:
2020-01-28 20:58
OK, as it seems BOOST needs some weird extra escaping

But i would say, using a regex like: (?:https?:\\/\\/)([^\\?&><\\[\\(#\\\\ \\\"'\\/\\)\\]]*) would solve the issues, see example here: https://regexr.com/4t4ap
Could the X-Spam-Report (bad) formatting also be caused by a malformed regex? It appears to delete line breaks from the report.
No

Have been trying to fix that as well, but i am unable to find where that goes wrong, looks like the spam report and hmailserver use different line-endings and thus destroying the spam report formatting
https://github.com/hmailserver/hmailserver/issues/115
https://github.com/hmailserver/hmailserver/issues/138
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
RvdH
Senior user
Senior user
Posts: 1042
Joined: 2008-06-27 14:42
Location: Netherlands

Re: URL send to SURBL

Post by RvdH » 2020-01-29 02:42

tunis wrote:
2020-01-23 11:07
I checked the email and all links are like this one.

Code: Select all

href="https://dmd.idg.se/x/c/?LU.RbsMgDPyVvtCnqQ0hbMokq2q3L1gf9jgRQAldDJEDivb1DLLJtnQn_3w6DbLNCB3vmz7PILnseSbgont_EXmCKcZlfT2ft207kR3srLwZHJmTDnhJEb90WmNAJm.bwuT7v.BJtsd9qXBRbvTwaYfBqFGRV6xt7jHo7ynMWHAtVlvcH6kA.1Mv1vhgb4JdO.KBCndlFgojKUQb999rSKQtWEM7RWtcQrCo3PznHXy0PkI1aHgrsoGPGuBwPNxKgF8A38"
This redirects to rebelandbird.com "rebelandbird.com)conversionista".
Redirect? Is mailserver supposed to follow redirect url's? I doubt it does, so that seems to be a wrong example and assumption
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
RvdH
Senior user
Senior user
Posts: 1042
Joined: 2008-06-27 14:42
Location: Netherlands

Re: URL send to SURBL

Post by RvdH » 2020-01-29 04:44

This regex thingy is keeping me awake :)

Instead of defining every character that is NOT allowed, like:

Code: Select all

(?:https?:\/\/)(?:[^@\s]+@)?(?:www\.)?([^\?:#<>\s{}[\]()\/\\'\"]+)

https://regexr.com/4t4u3

Wouldn't it be easier to specify what characters are allowed? (eg: alphanumeric, dot and dash)

Code: Select all

(?:https?:\/\/)(?:[^@\s]+@)?(?:www\.)?([\w\-\.]+)
https://regexr.com/4t4u6
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
mattg
Moderator
Moderator
Posts: 20786
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: URL send to SURBL

Post by mattg » 2020-01-29 04:48

I have a government newsletter sent to me yesterday

@ RvdH, I'll PM you both the log, plus the source of the email

I'm using 5.7.0-B2497 (x64) = absolute bleeding edge
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

tunis
Normal user
Normal user
Posts: 245
Joined: 2015-01-05 20:22
Location: Sweden

Re: URL send to SURBL

Post by tunis » 2020-01-29 10:18

RvdH wrote:
2020-01-29 02:42
tunis wrote:
2020-01-23 11:07
I checked the email and all links are like this one.

Code: Select all

href="https://dmd.idg.se/x/c/?LU.RbsMgDPyVvtCnqQ0hbMokq2q3L1gf9jgRQAldDJEDivb1DLLJtnQn_3w6DbLNCB3vmz7PILnseSbgont_EXmCKcZlfT2ft207kR3srLwZHJmTDnhJEb90WmNAJm.bwuT7v.BJtsd9qXBRbvTwaYfBqFGRV6xt7jHo7ynMWHAtVlvcH6kA.1Mv1vhgb4JdO.KBCndlFgojKUQb999rSKQtWEM7RWtcQrCo3PznHXy0PkI1aHgrsoGPGuBwPNxKgF8A38"
This redirects to rebelandbird.com "rebelandbird.com)conversionista".
Redirect? Is mailserver supposed to follow redirect url's? I doubt it does, so that seems to be a wrong example and assumption
All link (except two) in that email are like this and I get lots of SURBL checks for that email. Look at the first post.
Is hmailServer doing any encoding that course and the link encode to a readable link?
HMS 5.6.8 B2494.25 on Windows Server 2019 Core VM.
HMS 5.6.8 B2494.24 on Windows Server 2016 Core VM.
HMS 5.6.7 B2425.16 on Windows Server 2012 R2 Core VM.

tunis
Normal user
Normal user
Posts: 245
Joined: 2015-01-05 20:22
Location: Sweden

Re: URL send to SURBL

Post by tunis » 2020-01-29 12:39

I tried to resend that email and then it's only find the URL I found.

Code: Select all

"DEBUG"	2684	"2020-01-29 11:23:04.817"	"SURBL: Execute"
"DEBUG"	2684	"2020-01-29 11:23:04.833"	"SURBL: Found URL: idg.se"
"DEBUG"	2684	"2020-01-29 11:23:04.833"	"SURBL: Found URL: ugglamassage.se"
"DEBUG"	2684	"2020-01-29 11:23:04.833"	"SURBL: Found URL: stickerapp.se"
"DEBUG"	2684	"2020-01-29 11:23:04.833"	"SURBL: 3 unique addresses found."
"DEBUG"	2684	"2020-01-29 11:23:04.833"	"SURBL: Lookup: idg.se.multi.surbl.org"
"DEBUG"	2684	"2020-01-29 11:23:04.972"	"SURBL: Lookup: stickerapp.se.multi.surbl.org"
"DEBUG"	2684	"2020-01-29 11:23:05.004"	"SURBL: Lookup: ugglamassage.se.multi.surbl.org"
"DEBUG"	2684	"2020-01-29 11:23:05.035"	"SURBL: Match not found"
I found more emails with ")" in the log, but I can't find that URL in emails. In some emails like in the first post I can't even find the domain name.
HMS 5.6.8 B2494.25 on Windows Server 2019 Core VM.
HMS 5.6.8 B2494.24 on Windows Server 2016 Core VM.
HMS 5.6.7 B2425.16 on Windows Server 2012 R2 Core VM.

User avatar
RvdH
Senior user
Senior user
Posts: 1042
Joined: 2008-06-27 14:42
Location: Netherlands

Re: URL send to SURBL

Post by RvdH » 2020-01-29 15:51

Base64 Encoded maybe?


This regex ain't never gonna be 100% i think, to much quirks
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
SorenR
Senior user
Senior user
Posts: 3562
Joined: 2006-08-21 15:38
Location: Denmark

Re: URL send to SURBL

Post by SorenR » 2020-01-29 18:42

RvdH wrote:
2020-01-28 21:44
palinka wrote:
2020-01-28 21:28
RvdH wrote:
2020-01-28 20:58
OK, as it seems BOOST needs some weird extra escaping

But i would say, using a regex like: (?:https?:\\/\\/)([^\\?&><\\[\\(#\\\\ \\\"'\\/\\)\\]]*) would solve the issues, see example here: https://regexr.com/4t4ap
Could the X-Spam-Report (bad) formatting also be caused by a malformed regex? It appears to delete line breaks from the report.
No

Have been trying to fix that as well, but i am unable to find where that goes wrong, looks like the spam report and hmailserver use different line-endings and thus destroying the spam report formatting
https://github.com/hmailserver/hmailserver/issues/115
https://github.com/hmailserver/hmailserver/issues/138
Looking at this from a different angle... Windows use CRLF and Unix use LF... I found myself in a situation where only half of a .BAT file would work labal wise (Compiling BOOST 1.72) and I ended up converting all .BAT files from LF line endings to CRLF line endings.
https://www.hmailserver.com/forum/viewt ... ST#p216909

On the other hand SpamAssassin have had dealings with the CRLF/LF policy before... And... Since 90% of SpamAssassin developers use Unix they don't really care about Windows ;-)

https://bz.apache.org/SpamAssassin/show_bug.cgi?id=4068
SørenR.

“With age comes wisdom, but sometimes age comes alone.”
- Oscar Wilde

User avatar
RvdH
Senior user
Senior user
Posts: 1042
Joined: 2008-06-27 14:42
Location: Netherlands

Re: URL send to SURBL

Post by RvdH » 2020-01-30 10:41

Please stay on topic, this post is about SURBL url detection

I think the 'best' regex i have come up and tested is:

Code: Select all

(?:(?>https?)?(?>:\/\/|\%3A\%2F\%2F))(?:www\.)?([a-z0-9\-\.\=\r\n]+)
To have a perfect regex seems impossible, as the are so many possibilities with formatting, encoding, line-breaks in the e-mail source

You can test this regexp by using the online testers like: https://regex101.com/ and/or https://regexr.com/ using PCRE with the flags: case insensitive, global
Copy paste the regex above and simply paste the e-mail message source, (almost ;)) all occurrences of urls in the mails should be highlighted

I have posted a pull request for martin, now just hope he's still around ;)
https://github.com/hmailserver/hmailserver/pull/312
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
RvdH
Senior user
Senior user
Posts: 1042
Joined: 2008-06-27 14:42
Location: Netherlands

Re: URL send to SURBL

Post by RvdH » 2020-01-30 12:45

@tunis

If your signature is right you are still using 5.6.x, right?
I have a 5.6.8-B2494.24 build (at the usual place) using the above regex if you like to give it a test run
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

tunis
Normal user
Normal user
Posts: 245
Joined: 2015-01-05 20:22
Location: Sweden

Re: URL send to SURBL

Post by tunis » 2020-01-30 15:45

@RvdH

I updated hmailServer almost the same email came again without ")".

Code: Select all

"DEBUG"	1116	"2020-01-30 13:22:53.636"	"SURBL: Execute"
"DEBUG"	1116	"2020-01-30 13:22:53.636"	"SURBL: Found URL: webbdagarna.se"
"DEBUG"	1116	"2020-01-30 13:22:53.636"	"SURBL: Found URL: trippus.se"
"DEBUG"	1116	"2020-01-30 13:22:53.636"	"SURBL: Found URL: rebelandbird.com"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: conversionista.com"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: bleckout.co"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: socialview.se"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: nitor.com"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: storykit.io"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: umbraco.com"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: creuna.com"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: zenloop.com"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: siteimprove.com"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: adform.com"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: mis.se"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: comprend.com"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: stickermule.com"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: 16 unique addresses found."
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Lookup: adform.com.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:53.831"	"SURBL: Lookup: bleckout.co.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:53.863"	"SURBL: Lookup: comprend.com.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:53.988"	"SURBL: Lookup: conversionista.com.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.131"	"SURBL: Lookup: creuna.com.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.262"	"SURBL: Lookup: mis.se.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.294"	"SURBL: Lookup: nitor.com.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.309"	"SURBL: Lookup: rebelandbird.com.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.356"	"SURBL: Lookup: siteimprove.com.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.372"	"SURBL: Lookup: socialview.se.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.403"	"SURBL: Lookup: stickermule.com.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.450"	"SURBL: Lookup: storykit.io.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.481"	"SURBL: Lookup: trippus.se.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.607"	"SURBL: Lookup: umbraco.com.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.716"	"SURBL: Lookup: webbdagarna.se.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.763"	"SURBL: Lookup: zenloop.com.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.794"	"SURBL: Match not found"
I still wonder how it detects URLs? I can't find more than 2-3 domain and SURBL find 16. But no ")" is found now. :D
HMS 5.6.8 B2494.25 on Windows Server 2019 Core VM.
HMS 5.6.8 B2494.24 on Windows Server 2016 Core VM.
HMS 5.6.7 B2425.16 on Windows Server 2012 R2 Core VM.

User avatar
RvdH
Senior user
Senior user
Posts: 1042
Joined: 2008-06-27 14:42
Location: Netherlands

Re: URL send to SURBL

Post by RvdH » 2020-01-30 16:39

tunis wrote:
2020-01-30 15:45
@RvdH

I updated hmailServer almost the same email came again without ")".

Code: Select all

"DEBUG"	1116	"2020-01-30 13:22:53.636"	"SURBL: Execute"
"DEBUG"	1116	"2020-01-30 13:22:53.636"	"SURBL: Found URL: webbdagarna.se"
"DEBUG"	1116	"2020-01-30 13:22:53.636"	"SURBL: Found URL: trippus.se"
"DEBUG"	1116	"2020-01-30 13:22:53.636"	"SURBL: Found URL: rebelandbird.com"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: conversionista.com"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: bleckout.co"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: socialview.se"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: nitor.com"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: storykit.io"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: umbraco.com"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: creuna.com"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: zenloop.com"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: siteimprove.com"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: adform.com"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: mis.se"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: comprend.com"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Found URL: stickermule.com"
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: 16 unique addresses found."
"DEBUG"	1116	"2020-01-30 13:22:53.652"	"SURBL: Lookup: adform.com.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:53.831"	"SURBL: Lookup: bleckout.co.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:53.863"	"SURBL: Lookup: comprend.com.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:53.988"	"SURBL: Lookup: conversionista.com.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.131"	"SURBL: Lookup: creuna.com.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.262"	"SURBL: Lookup: mis.se.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.294"	"SURBL: Lookup: nitor.com.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.309"	"SURBL: Lookup: rebelandbird.com.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.356"	"SURBL: Lookup: siteimprove.com.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.372"	"SURBL: Lookup: socialview.se.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.403"	"SURBL: Lookup: stickermule.com.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.450"	"SURBL: Lookup: storykit.io.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.481"	"SURBL: Lookup: trippus.se.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.607"	"SURBL: Lookup: umbraco.com.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.716"	"SURBL: Lookup: webbdagarna.se.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.763"	"SURBL: Lookup: zenloop.com.multi.surbl.org"
"DEBUG"	1116	"2020-01-30 13:22:54.794"	"SURBL: Match not found"
I still wonder how it detects URLs? I can't find more than 2-3 domain and SURBL find 16. But no ")" is found now. :D
Cool 8)

It literally takes the mail (source!) and tries to lookup url entries in it line by line, this could be in the headers, plain text part, html part (incl style blocks)
Maybe you should try what i described above, eg: take the mail source and paste it into https://regex101.com/ and/or https://regexr.com/ using described regex, all occurrences of urls in the mails should be highlighted
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

tunis
Normal user
Normal user
Posts: 245
Joined: 2015-01-05 20:22
Location: Sweden

Re: URL send to SURBL

Post by tunis » 2020-01-30 16:59

RvdH wrote:
2020-01-30 16:39
It literally takes the mail (source!) and tries to lookup url entries in it line by line, this could be in the headers, plain text part, html part (incl style blocks)
Maybe you should try what i described above, eg: take the mail source and paste it into https://regex101.com/ and/or https://regexr.com/ using described regex, all occurrences of urls in the mails should be highlighted
I have taken the source of the email and the URLs are not there but SURBL find them.
HMS 5.6.8 B2494.25 on Windows Server 2019 Core VM.
HMS 5.6.8 B2494.24 on Windows Server 2016 Core VM.
HMS 5.6.7 B2425.16 on Windows Server 2012 R2 Core VM.

User avatar
RvdH
Senior user
Senior user
Posts: 1042
Joined: 2008-06-27 14:42
Location: Netherlands

Re: URL send to SURBL

Post by RvdH » 2020-01-30 17:02

Can you pm the source to me so i could have a look?

I am a bit surprised to see 16 matches and lookups, especially because a hardcoded limit of 15 is defined in the code

Code: Select all

const int maxURLsToProcess = 15;
:o
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

palinka
Senior user
Senior user
Posts: 1915
Joined: 2017-09-12 17:57

Re: URL send to SURBL

Post by palinka » 2020-01-30 18:21

RvdH wrote:
2020-01-30 17:02
Can you pm the source to me so i could have a look?

I am a bit surprised to see 16 matches and lookups, especially because a hardcoded limit of 15 is defined in the code

Code: Select all

const int maxURLsToProcess = 15;
:o
0-15 = 16 iterations. :D

User avatar
RvdH
Senior user
Senior user
Posts: 1042
Joined: 2008-06-27 14:42
Location: Netherlands

Re: URL send to SURBL

Post by RvdH » 2020-01-30 19:12

palinka wrote:
2020-01-30 18:21
RvdH wrote:
2020-01-30 17:02
Can you pm the source to me so i could have a look?

I am a bit surprised to see 16 matches and lookups, especially because a hardcoded limit of 15 is defined in the code

Code: Select all

const int maxURLsToProcess = 15;
:o
0-15 = 16 iterations. :D
:oops:
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
RvdH
Senior user
Senior user
Posts: 1042
Joined: 2008-06-27 14:42
Location: Netherlands

Re: URL send to SURBL

Post by RvdH » 2020-03-07 16:55

This change is now included in latest artifact for 5.7.x
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

Post Reply