Cache results of RBL's for user defined period of time

This forum contains features that has been archived. This section contains implemented features, duplicate requests, and requests which we have decided not to implement.
Post Reply

Do you want this feature

Yes
18
90%
No
2
10%
 
Total votes: 20

^DooM^
Site Admin
Posts: 13862
Joined: 2005-07-29 16:18
Location: UK

Cache results of RBL's for user defined period of time

Post by ^DooM^ » 2010-03-09 03:26

Whilst reading a users logs earlier from an open relay test looking at TCPIP it showed that hMail contacted 4 blacklists with the same IP for each attempt (around 20 attempts) so that was 80 lookups for one test over the space of about 5 minutes. This to me seems like a lot of wasted bandwidth. Now I know DNS servers cache results but you still have the overhead of contacting the DNS servers.

I propose having a new table for RBL Lookups, somewhat like the greylist table. IP's that have been checked are added to DB and expired after a few minutes/hours/days/however long the user wants.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

DeanoX
Senior user
Senior user
Posts: 480
Joined: 2005-11-05 00:07
Location: Michigan

Re: Cache results of RBL's for user defined period of time

Post by DeanoX » 2010-03-09 16:43

This is a good idea. I had made mention of something like this once before in an earlier discussion, http://www.hmailserver.com/forum/viewto ... =7&t=16159 but I never got around to adding a feature request. Thanks ^DooM^.

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Cache results of RBL's for user defined period of time

Post by Bill48105 » 2010-04-25 00:01

Yeah some sort of caching would be nice but doesn't your DNS server cache lookups? I use BIND & it does for me.. First lookup takes a lot longer than subsequent ones as seen by the timestamp in the logs. If you use your ISP's DNS server (or indirectly using your routers) it isn't too hard to setup BIND as a caching-only server which would avoid having to add anything to hmail..
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

^DooM^
Site Admin
Posts: 13862
Joined: 2005-07-29 16:18
Location: UK

Re: Cache results of RBL's for user defined period of time

Post by ^DooM^ » 2010-04-25 01:23

Bill48105 wrote:Yeah some sort of caching would be nice but doesn't your DNS server cache lookups? I use BIND & it does for me.. First lookup takes a lot longer than subsequent ones as seen by the timestamp in the logs. If you use your ISP's DNS server (or indirectly using your routers) it isn't too hard to setup BIND as a caching-only server which would avoid having to add anything to hmail..
Bill
Please read the full post before making comments.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Cache results of RBL's for user defined period of time

Post by Bill48105 » 2010-04-25 03:41

^DooM^ wrote: Please read the full post before making comments.
I apologize if I somehow offended you but I did read the entire post and perhaps my thoughts did not come out as I intended because I was distracted at the time. I guess my point was a local DNS server caching the lookups should provide good results reducing lookups to downstream DNS servers without having to mod the mail server & creating more overhead on the SQL server but seems my fingers typed something else & it pissed you off. My bad. You could have at least posted a useful response (perhaps commenting on pros/cons of caching the lookups in SQL vs DNS caching & performance testing you did to compare them) instead of scolding me like my dad, but seems you'd rather do the later. :? Anyway I have not done any such testing but suspect dns caching creates a lot less overhead & server load than sql lookups especially if the table gets huge but maybe constructive dialog would help prove or disprove either case.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

DeanoX
Senior user
Senior user
Posts: 480
Joined: 2005-11-05 00:07
Location: Michigan

Re: Cache results of RBL's for user defined period of time

Post by DeanoX » 2010-04-25 04:21

I run local bind dns services on the same machines as hMail to save overhead, but traffic could be reduced even further. Using slow dns servers, or other such geographically dispersed dns servers, potential problem points could be averted.

In a lot of cases, I see IP addresses connect several hundred times, or more, that are rbl positive. I would like to turn them away at the door, for the lowest possible cost.

Rather than do repeated lookup's to dns, it's proposed that rbl positive listings be handled in a similar way to how greylisting, currently works.

So, if an IP is confirmed as being listed, take that IP address, and apply user defined settings, similar to greylisting, and issue a 45X error, try again later.

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Cache results of RBL's for user defined period of time

Post by Bill48105 » 2010-04-25 05:44

DeanoX wrote:I run local bind dns services on the same machines as hMail to save overhead, but traffic could be reduced even further. Using slow dns servers, or other such geographically dispersed dns servers, potential problem points could be averted.

In a lot of cases, I see IP addresses connect several hundred times, or more, that are rbl positive. I would like to turn them away at the door, for the lowest possible cost.

Rather than do repeated lookup's to dns, it's proposed that rbl positive listings be handled in a similar way to how greylisting, currently works.

So, if an IP is confirmed as being listed, take that IP address, and apply user defined settings, similar to greylisting, and issue a 45X error, try again later.
Hey DeanoX,
I'm new to hmail but I was reading about a setting to change when the RBL lookup occurs in the SMTP conversation:
http://www.hmailserver.com/forum/viewto ... 05#p107705
Granted it's not exactly what you're talking about but might cut down on some lookups if the spam is to unknown users & you don't have catchall set.

As far as your 400-level response comment, I've not dug deep enough into hmail to know the ins & out of how it works but yeah if it doesn't do that now that should be useful. Personally I've used ASSP as a front-end for years and it does a great job of beating back spam connections before they even get to the mail server so most of the IP-related features of hMail are not helpful to me since all connections coming into it appear to be from my ASSP IP anyway.

I'd be interested in thoughts on 'work' related to dns lookups to your caching dns server vs building a table of IP's, queries of that data and eventually expiry management as well as drawbacks related to false positives during delays in data cleanup after an IP has been removed from the original RBL. (Of course the same can be said of caching & DNS TTL but that's what productive discussions are for. ;) BIND seems very efficient at lookups (especially if they are cached) so I'm guessing caching in SQL database would require a lot more work from the server but with less internet traffic especially if the records were kept longer than the TTL.

Btw for those super sensitive to DNS queries to the outside (especially the response times or if you're worried they'll be down at any given point) many RBL's allow zone transfers so you can keep a local copy via rsync but I'd imagine you need to do a crapload of lookups to make that worthwhile. lol Anyhow figured I'd throw that out there as a possibility.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

DeanoX
Senior user
Senior user
Posts: 480
Joined: 2005-11-05 00:07
Location: Michigan

Re: Cache results of RBL's for user defined period of time

Post by DeanoX » 2010-04-25 06:23

Bill48105 wrote: I'm new to hmail but I was reading about a setting to change when the RBL lookup occurs in the SMTP conversation:
http://www.hmailserver.com/forum/viewto ... 05#p107705
Granted it's not exactly what you're talking about but might cut down on some lookups if the spam is to unknown users & you don't have catchall set.
I prefer not reveal the existence of real users, -vs unknown users. Keep them guessing, and make them do the work.
Bill48105 wrote: As far as your 400-level response comment, I've not dug deep enough into hmail to know the ins & out of how it works but yeah if it doesn't do that now that should be useful. Personally I've used ASSP as a front-end for years and it does a great job of beating back spam connections before they even get to the mail server so most of the IP-related features of hMail are not helpful to me since all connections coming into it appear to be from my ASSP IP anyway.
Right now, in my anti-spam configuration, rbl positives are given a 550 error, but they hammer away anyway. I have used assp in the past, but I prefer to use the mail servers capabilities if at all possible. Central management, less work.
Bill48105 wrote: I'd be interested in thoughts on 'work' related to dns lookups to your caching dns server vs building a table of IP's, queries of that data and eventually expiry management as well as drawbacks related to false positives during delays in data cleanup after an IP has been removed from the original RBL. (Of course the same can be said of caching & DNS TTL but that's what productive discussions are for. ;) BIND seems very efficient at lookups (especially if they are cached) so I'm guessing caching in SQL database would require a lot more work from the server but with less internet traffic especially if the records were kept longer than the TTL.
While I agree bind is efficient as a caching server, queries are still made, and quite a lot of them when you are getting hammered. That's when the offending IP gets blocked at the gateway routers. :twisted:
My goal is to save as much bandwidth everywhere I can. Effective bandwidth control saves $$ in the long run here.
Bill48105 wrote: Btw for those super sensitive to DNS queries to the outside (especially the response times or if you're worried they'll be down at any given point) many RBL's allow zone transfers so you can keep a local copy via rsync but I'd imagine you need to do a crapload of lookups to make that worthwhile. lol Anyhow figured I'd throw that out there as a possibility.
Bill
I would recommend a local, caching only name server to anyone running a mail server locally. And the more rbl's in use locally, I would imagine that you would see a pretty good improvement in all dns response times, not just rbl lookups.

Since greylisting works rather nicely in hMail, I believe that a large benefit could be derived by handling rbl positives in a similar fashion. And since the greylisting framework is already in hMail, I am 'assuming', it could be integrated nicely for use with the rbl. Another anti-spam fighting tool at the least.

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Cache results of RBL's for user defined period of time

Post by Bill48105 » 2010-04-25 07:01

DeanoX wrote: I prefer not reveal the existence of real users, -vs unknown users. Keep them guessing, and make them do the work.
Oh so true, don't want to give the spammers any more info than needed while giving legit bounces enough info to be useful to legit senders. I guess this is where being able to set SMTP responses would be useful so that you could customize them to a certain extent as long you weren't straying too far from RFC's. Then you could set the INI value & instead of 'user uknown' say 'mailbox unavailable' or such to not tip them off. Sneaky. :D
DeanoX wrote: Right now, in my anti-spam configuration, rbl positives are given a 550 error, but they hammer away anyway. I have used assp in the past, but I prefer to use the mail servers capabilities if at all possible. Central management, less work.
I'd have to brush up on RFC's and if permanent vs temporary response is supposed to happen for RBL detection but I'd sooner say 500-level is best so that false positives bounce back to the sender & they know there's a problem vs legit servers retrying until it expires before bouncing. Odds are spammers are going to ignore level anyway.

I've been using MTS Professional for many years now (likely soon to be migrated to hmail so RIP trusty friend) & unfortunately it's been unable to hold up to SMTP D.O.S. attacks too well & ASSP really saved me many times and has allowed my setup to work out far longer than it would have without it. Yeah ideally you'd like to not have to manage more systems than needed but ASSP has 1000x more settings than hmail for spam detection & response (I mean that's WHAT it's made for so nothing negative toward hmail) plus since it's open source & perl I've been able to mod it to my liking or as needs arose. It was a bit of a bear to learn & get setup but once I got the hang of it I'd definitely recommend it to anyone running a mail server especially if you get pounded hard by spam cuz it can really hold it's own & take a huge load of your server. Btw I didn't do the full-blown ASSP in/out setup where all mail goes through it so it can auto-learn but instead it just sits in front of incoming SMTP and Spam Assassin handles bayes. That alone made it a ton easier to get setup but it definitely has a huge learning curve to it and is an example of where 1000's of options could be a bad thing. lol
DeanoX wrote: While I agree bind is efficient as a caching server, queries are still made, and quite a lot of them when you are getting hammered. That's when the offending IP gets blocked at the gateway routers. :twisted:
My goal is to save as much bandwidth everywhere I can. Effective bandwidth control saves $$ in the long run here.
Yeah I'd say BIND CPU usage even budges during heavy load of incoming email with tons of RBL, MX, RDNS lookups so it's gotta be pretty efficient. SQL queries on the other hand sure can. I am confused by what you mean by queries are still made though.. Do you mean from hmail to your caching dns server (which uses no internet except for the initial query or once it expires) or the initial/expired queries themselves? Because if the records are indeed cached & not expired there should be no outside lookups to cost you usage.. :? I mean if hmail caches them in a database the initial lookup still needs to occur (same as DNS caching) and I assume some sort of expiry system would be implemented so later more outside lookups would occur (same as DNS) but the big difference could be YOU set the expiry for DB cleanup that could be a lot longer than DNS expiry which is set by the primary DNS server which in turn could reduce your outside lookups but could create headaches when valid IP's get listed & then de-listed & you're still blocking them. Anyhow, sorry if I'm dense & misunderstanding what you're saying but as I read it the outside queries would be about the same besides an extended expiry possibility.
DeanoX wrote: I would recommend a local, caching only name server to anyone running a mail server locally. And the more rbl's in use locally, I would imagine that you would see a pretty good improvement in all dns response times, not just rbl lookups.
Oh yeah it's a no brainer for sure. Cached lookups help a ton, my DNS server is magnitudes more reliable than my ISP's and generally even non-cached lookups are a lot faster too since my BIND load is lower & most ISP NS's are a few hops away which adds times to queries.
DeanoX wrote: Since greylisting works rather nicely in hMail, I believe that a large benefit could be derived by handling rbl positives in a similar fashion. And since the greylisting framework is already in hMail, I am 'assuming', it could be integrated nicely for use with the rbl. Another anti-spam fighting tool at the least.
Have you seen spammers getting more clever & learning to bypass greylisting? I recall when I 1st setup greylisting spam dropped considerably but as time has gone on seems more & more spam started to come in. Of course that could be attributed to other things like spammers creating SPF records or finding new/better means but logs show they are often retrying later which reduces greylisting effectiveness. Still definitely useful but not as great as it used to be and at some point might not be too useful at all. But yeah you'd think reusing code should be easier than starting from scratch & of course the more tools the better either way!
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

^DooM^
Site Admin
Posts: 13862
Joined: 2005-07-29 16:18
Location: UK

Re: Cache results of RBL's for user defined period of time

Post by ^DooM^ » 2010-04-25 12:06

The reason for my flippant comment was because of this.
Bill48105 wrote:Yeah some sort of caching would be nice but doesn't your DNS server cache lookups?
When I had already made that very same point in my post.
^DooM^ wrote:Now I know DNS servers cache results but you still have the overhead of contacting the DNS servers.
I wasn't offended it just came across as you not reading the post because you repeated something I had already considered.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Cache results of RBL's for user defined period of time

Post by Bill48105 » 2010-04-25 16:02

Sorry again ^DooM^. After I re-read what I wrote it could appear like I hadn't read what you said but like I said, what was going on in my head wasn't exactly what came out that's for sure. I take full responsibility as I should have written it when I wasn't distracted & proof-read it but I was in a hurry. In the end I still think my point should have come across without getting hung up on misguided wording so I'll retry:

I understand you know some DNS servers cache based on your original post but you did not specify if the DNS server YOU use are actually caching the results, if you ran your own or used your routers, all of which could possibly not cache & result in extra bandwidth usage & slower lookups & create a desire for hmail to perform the caching.

Then the rest of what I meant to say about server work load of SQL lookups & management of such a database vs relying on DNS caching was in my next post & should have been in my 1st post to help clarify further my thoughts.

Maybe my other comments to DeanoX helped clarify where I intended to go with my original post where I was trying to open a discussion about it rather than spew out every possible thought regarding the topic.

Hopefully if you read what I said above & re-read my original post you can better see my intended comment instead of focusing on some underlying tone or inference possibility. Since your original post did not specify if your DNS server was caching (just that you were aware DNS servers did) or make reference to pros/cons of utilizing DNS caching vs SQL I had to pick one possibility & it seems you were annoyed thus your "flippant comment".

Anyway I am on here to learn about hmail, help others where I can, and hopefully participate in making hmail better, not make enemies because of poorly written or misread posts. I'll do my best to word my comments better to avoid confusion but please understand I mean no offense or malice so that shouldn't be the focus of my comments but rather the potential for contribution to the community.
Thx
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

^DooM^
Site Admin
Posts: 13862
Joined: 2005-07-29 16:18
Location: UK

Re: Cache results of RBL's for user defined period of time

Post by ^DooM^ » 2010-04-25 21:01

Like I said, No offence taken I appreciate it when people take the effort to help out here.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

rolaids0
Normal user
Normal user
Posts: 150
Joined: 2010-04-27 02:03
Location: Florida
Contact:

Re: Cache results of RBL's for user defined period of time

Post by rolaids0 » 2010-07-08 17:37

I think putting RBL results into the DB would be bad for a number of reasons. First, the mail server would have to deal with expiration. If that's done with an external script/process then that would work, however, then you run into locking while the delete is underway. If you use a transactional DB that is somewhat resolved, but then you run into having those records there for eternity (if you use MySQL with InnoDB, with a single file format) eating up space.

Bind can be changed to override the TTL in order to let results live in the cache longer, but is generally a no-no to do so:
http://serverfault.com/questions/113954 ... et-address

^DooM^
Site Admin
Posts: 13862
Joined: 2005-07-29 16:18
Location: UK

Re: Cache results of RBL's for user defined period of time

Post by ^DooM^ » 2010-07-08 20:58

I assume you don't know how the greylist table works in hmail, if you did your post would have been redundant. Greylisting removes unused records after X days depending on your setting, Cache results would be no different.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Cache results of RBL's for user defined period of time

Post by Bill48105 » 2010-07-08 22:35

Hate to seem argumentative & stir the pot but seems although rolaids0 didn't mention anything about how hmail greylisting works, his comment tells me he DOES & that's his concern.. I read it as he figured some work would need to be done to maintain the database which is true, be it by hmail or an external script (not sure if you read it as he thought hmail wouldn't do the work or if that was where your focus was & source of your response but I read it as he assumed hmail would likely do it otherwise if done externally there would be more potential for issues..)

Truth is creating those entries & managing them takes server work/space/memory either way and as I've mentioned before IMO relying on caching DNS seems best & most efficient approach. (Without testing I can't prove it but I suspect hmail doing DNS lookup to local caching DNS server uses a lot less resources than database queries let alone adding in work involved in managing that data..)

And rolaids0 pointed out fix for the shortcoming of short TTL's to have them expire later if you like although to me I'd rather waste a little bandwidth & have more accurate results than cache them too long and end up with too many false positives & negatives because the cache hasn't expired yet..

I will give you that having them in a database would make life easier in managing them though as you could see which are in there & delete them as needed much easier than dealing with the DNS server but that just brings us back to the whitelist/blacklist stuff discussed elsewhere.. Having those would make that argument almost moot. Have a FP? Add them to whitelist (or IP range with spam testing disabled I suppose). Have a FN? Add them to blacklist (or IP range with allow connections would work) Anyhow guess I'm rambling. lol

I guess one of us needs to setup some testing & prove one way or another once & for all. NOT IT! :D
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

rolaids0
Normal user
Normal user
Posts: 150
Joined: 2010-04-27 02:03
Location: Florida
Contact:

Re: Cache results of RBL's for user defined period of time

Post by rolaids0 » 2010-07-08 23:07

If you knew how DNS worked, you'd realize this feature is also redundant ;-)

Anyway, yes, I know full well how it works. I simply pointed out the shortcomings of such a feature request. Secondly, what you want is available in Bind, with a minor tweak. Another question for you, how many graylist entries does your server add/remove every day? I can see by my DNS server, sending it one e-mail with a single URL in it yields about 20 queries to various RBLs/Whitelists/URLBLS. So each message does about 20 lookups, Bind caches those 20. Then another comes in, different mail server, 20 more queries. On and on. Now lets put in the one time spammers. 1000's of records could be added daily on a low volume mail server. That was my point. It could be added, but there should be a config to turn it off.

^DooM^
Site Admin
Posts: 13862
Joined: 2005-07-29 16:18
Location: UK

Re: Cache results of RBL's for user defined period of time

Post by ^DooM^ » 2010-07-08 23:27

rolaids0 wrote:If you knew how DNS worked, you'd realize this feature is also redundant ;-)
Again if you had read my first post you would see I say exactly
^DooM^ wrote:Now I know DNS servers cache results but you still have the overhead of contacting the DNS servers.
And way to go bill, you just can't help yourself can you.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Cache results of RBL's for user defined period of time

Post by Bill48105 » 2010-07-08 23:41

^DooM^ wrote:Now I know DNS servers cache results but you still have the overhead of contacting the DNS servers.
...
And way to go bill, you just can't help yourself can you.
You must think I'm one evil person or you're just a cynic.. News flash: Not everyone has to agree with you DooM and those who don't are not necessarily your enemies.. I certainly am not nor do I wish to be. I'd hope that these forums were a place of open discussion on hmail but over & over I'm battling petty arguments and accusations like I'm the bad guy and just want to argue. Trust me that is not the case, I have tons of more important things do to. But I WILL express my opinion on any topic I feel I can contribute. ;)

What planet would you be on if you believed I thought DNS had no overhead or that SQL had less overhead than DNS? Certainly wouldn't be Earth because neither is true and that is the reason for my posts not to argue with you. ;)
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

rolaids0
Normal user
Normal user
Posts: 150
Joined: 2010-04-27 02:03
Location: Florida
Contact:

Re: Cache results of RBL's for user defined period of time

Post by rolaids0 » 2010-07-08 23:48

Bill?

What overhead in contacting DNS servers? Do you not run your own local caching server? Even failing that, the Windows DNS client caches anyway.

Here's the flaw in having the DB do it:
*You have to do a look up anyway, which won't be in the DB on first query. Still requires a lookup.
*Spammers monitor BLs of all kinds. Once they are listed they move on to greener pastures.
*The overhead in indexes alone will negate any speed up of not using a DNS server.

So you want to save 512bytes+UDP+IP headers on your WAN connection per lookup, which has to occur anyway. If those packets are causing bandwidth issues, then you have far more serious issues than the overhead of a DNS lookup.

So, to address your original post:
^Doom^ wrote: Whilst reading a users logs earlier from an open relay test looking at TCPIP it showed that hMail contacted 4 blacklists with the same IP for each attempt (around 20 attempts) so that was 80 lookups for one test over the space of about 5 minutes. This to me seems like a lot of wasted bandwidth. Now I know DNS servers cache results but you still have the overhead of contacting the DNS servers.
What did the DNS server show? Did these lookups return anything? 80 lookups over 5mins is roughly 16 lookups a minute, or 8,192bytes over a minute or 136.5bytes per second. (plus overhead) Not really all that much, I've got roughly 20~100 lookups per second going on with hardly any bandwidth being used in comparison to other traffic. The load is really minimal. If the connection is bogged down by DNS queries its time for an upgrade or...
^Doom^ wrote: I propose having a new table for RBL Lookups, somewhat like the greylist table. IP's that have been checked are added to DB and expired after a few minutes/hours/days/however long the user wants.
Whilst reading a users logs earlier from an open relay test looking at TCPIP it showed that hMail contacted 4 blacklists with the same IP for each attempt (around 20 attempts) so that was 80 lookups for one test over the space of about 5 minutes. This to me seems like a lot of wasted bandwidth. Now I know DNS servers cache results but you still have the overhead of contacting the DNS servers.
Understood, you want to override the TTL as returned by the DNS server. What about not found results? Cache those as well? If you just want to cache the hits, mod bind, its more efficient than any DB will be. If you want to cache the not found results, then the DB may be the answer.
^Doom^ wrote: I propose having a new table for RBL Lookups, somewhat like the greylist table. IP's that have been checked are added to DB and expired after a few minutes/hours/days/however long the user wants.
I'd say do it purely in memory with a temp table (realizing that over a certain size, any decent mail server will flush to disk.) I would add to your options to Not do it at all/minutes/hours/days/however long the user wants. Some RBLS are set for weeks or months. Do you want to over ride that for a shorter period or respect those times?

I know Bill, and I can assure you I am NOT Bill ;-D

^DooM^
Site Admin
Posts: 13862
Joined: 2005-07-29 16:18
Location: UK

Re: Cache results of RBL's for user defined period of time

Post by ^DooM^ » 2010-07-09 00:03

Rolaids, makes sense, i'll archive this request.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

Post Reply