Thread: [mod-security-users] RBL queries: filtering possible?
Brought to you by:
victorhora,
zimmerletw
From: Michael R. <mre...@ot...> - 2007-04-24 12:32:13
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi all. Some IP blacklists have special zones that combine some (or all) of their other zones in a single, bitmasked list. The advantage is that one can check with a single query (and by applying a bitmask to the result) whether an IP is listed in more than one zone. See http://www.surbl.org/lists.html#multi for more details. I wonder if mod-security supports such bit-masked lookups already, or if that could be implemented in a future version. Bye, Mike -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iEYEARECAAYFAkYt+LYACgkQa3V7dXg8JKudUgCfSOATjQ4VXKXxQC1f5Hfh5NmA 1bgAoNWOcMeDPPGyJ5f6oLnayPay7mym =ItFK -----END PGP SIGNATURE----- |
From: Ryan B. <Ryan.Barnett@Breach.com> - 2007-04-24 12:46:25
|
Interesting... I see the resource/time advantage of being able to just do one @rbl looking to multi.surbl.org and it will then query the other list. As for inspecting the bitmask portion, does it really matter which list it was on? Normally, if the IP address is listed on any of the individual lists, people would want to deny access. The @rbl operator will just return true or false whether the IP was listed or not. Is there something specific that you are looking to do depending on which list had the client IP address listed? --=20 Ryan C. Barnett ModSecurity Community Manager Breach Security: Director of Application Security Training Web Application Security Consortium (WASC) Member Author: Preventing Web Attacks with Apache =20 -------------- Web Security Threat Report Webinar on May 9, 2007 (12 pm EST) Learn More About the Breach Webinar Series: http://www.breach.com/webinars.asp -------------- =20 > -----Original Message----- > From: mod...@li... [mailto:mod- > sec...@li...] On Behalf Of Michael > Renzmann > Sent: Tuesday, April 24, 2007 8:32 AM > To: mod...@li... > Subject: [mod-security-users] RBL queries: filtering possible? >=20 > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 >=20 > Hi all. >=20 > Some IP blacklists have special zones that combine some (or all) of their > other zones in a single, bitmasked list. The advantage is that one can > check with a single query (and by applying a bitmask to the result) > whether an IP is listed in more than one zone. >=20 > See http://www.surbl.org/lists.html#multi for more details. >=20 > I wonder if mod-security supports such bit-masked lookups already, or if > that could be implemented in a future version. >=20 > Bye, Mike > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.1 (GNU/Linux) >=20 > iEYEARECAAYFAkYt+LYACgkQa3V7dXg8JKudUgCfSOATjQ4VXKXxQC1f5Hfh5NmA > 1bgAoNWOcMeDPPGyJ5f6oLnayPay7mym > =3DItFK > -----END PGP SIGNATURE----- >=20 > ------------------------------------------------------------------------ - > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > mod-security-users mailing list > mod...@li... > https://lists.sourceforge.net/lists/listinfo/mod-security-users |
From: Michael R. <mre...@ot...> - 2007-04-24 13:33:00
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Ryan. > Interesting... I see the resource/time advantage of being able to just > do one @rbl looking to multi.surbl.org and it will then query the other > list. >From what I understand the dnsbl-server does not perform "local sub-queries to other zones". Instead, the database used for that zone is fed by a appropriate zonefile (which, in return, most probably will be built by some script or whatever that combines the current contents of the zones in question). > As for inspecting the bitmask portion, does it really matter > which list it was on? [...] Is there something specific that you are > looking to do depending on which list had the client IP address listed? Yes, but I have to explain a bit. A friend of mine just started a dnsbl that lists known spider IPs. This dnsbl is fed with data from iplists.com, which in return allows pretty good destinction between various "big" spiders such as Googlebot or the MS live.com spider. One of the websites I'm responsible for makes use of Trac (http://trac.edgewall.org). This has proven to be very useful for our purposes, but it can be pretty demanding when processing some requests. Unfortunately the webserver isn't very powerful (P3-850), and I'm happy for any unnecessary expensive request I can prevent. One alternative would be to upgrade to a more powerful server (but we don't have the money for that at this time), another would be to implement caching for Trac (which is actually on my to-do list, but will take some time). Analysing the access logs reveals that a good share of the server's load seems to be caused by spiders hitting places that have been disallowed in robots.txt. At first I thought that I'm just too dumb to write a proper robots.txt, but after double-checking the file content and reading through the online available information on that topic I came to believe that I'm not at fault. Hence I'm currently considering to use a combination of mod-security rules and queries to the mentioned spider dnsbl to implement some kind of "robots.txt enforcement". GET-requests from IP-addresses that are listed in the spider dnsbl to locations that are disallowed in robots.txt should be answered with an error (403 or 410 or ...). You may wonder: where is the need for bitmasking the result? Well, some of the locations in robots.txt are not disallowed for all spiders. One example is the feed generator which is allowed to be accessed by Google's Feedfetcher, but not by all the other "ordinary" spiders. Without bitmasking this will be pretty hard/expensive to achieve. I'm aware that this is a very special case, and most probably one that looks pretty borked. But think of a case where a user is interested to check an IP against three zones of a dnsbl service. If that service offers a multi zone that combines more than these three zones it's still easier to perform one lookup and mask only the interesting bits in the result, rather than performing three consecutive lookups to get the same result. Bye, Mike -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iEYEARECAAYFAkYuBwIACgkQa3V7dXg8JKuKyQCeII4bUd46D/ekgmMKpBGgOJxN rI8AoI6chV4t8NefGDYRpG5AC/vMKDcm =WOQo -----END PGP SIGNATURE----- |
From: Ivan R. <iva...@gm...> - 2007-04-26 11:40:51
|
Hi Michael, Thank you for the detailed explanation. Adding that functionality to ModSecurity shouldn't be much work. I was thinking implementing it along these lines: # Perform lookup. We are not blocking straight away but by specifying # the "capture" action we instruct the @rbl operator to extract the response # bits and place them into transaction variables (only when successful). SecRule REMOTE_ADDR "@rbl whatever.org" nolog,pass,capture # At this point the TX:0 variable contains the actual response returned. # Variables TX:1, TX:2, ..., TX:9 contain individual bits so you can use them for any purpose. For example: SecRule TX:1 log,deny SecRule TX:2 pass On 4/24/07, Michael Renzmann <mre...@ot...> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi Ryan. > > > Interesting... I see the resource/time advantage of being able to just > > do one @rbl looking to multi.surbl.org and it will then query the other > > list. > > >From what I understand the dnsbl-server does not perform "local > sub-queries to other zones". Instead, the database used for that zone is > fed by a appropriate zonefile (which, in return, most probably will be > built by some script or whatever that combines the current contents of the > zones in question). > > > As for inspecting the bitmask portion, does it really matter > > which list it was on? [...] Is there something specific that you are > > looking to do depending on which list had the client IP address listed? > > Yes, but I have to explain a bit. > > A friend of mine just started a dnsbl that lists known spider IPs. This > dnsbl is fed with data from iplists.com, which in return allows pretty > good destinction between various "big" spiders such as Googlebot or the MS > live.com spider. > > One of the websites I'm responsible for makes use of Trac > (http://trac.edgewall.org). This has proven to be very useful for our > purposes, but it can be pretty demanding when processing some requests. > Unfortunately the webserver isn't very powerful (P3-850), and I'm happy > for any unnecessary expensive request I can prevent. One alternative would > be to upgrade to a more powerful server (but we don't have the money for > that at this time), another would be to implement caching for Trac (which > is actually on my to-do list, but will take some time). > > Analysing the access logs reveals that a good share of the server's load > seems to be caused by spiders hitting places that have been disallowed in > robots.txt. At first I thought that I'm just too dumb to write a proper > robots.txt, but after double-checking the file content and reading through > the online available information on that topic I came to believe that I'm > not at fault. > > Hence I'm currently considering to use a combination of mod-security rules > and queries to the mentioned spider dnsbl to implement some kind of > "robots.txt enforcement". GET-requests from IP-addresses that are listed > in the spider dnsbl to locations that are disallowed in robots.txt should > be answered with an error (403 or 410 or ...). > > You may wonder: where is the need for bitmasking the result? Well, some of > the locations in robots.txt are not disallowed for all spiders. One > example is the feed generator which is allowed to be accessed by Google's > Feedfetcher, but not by all the other "ordinary" spiders. Without > bitmasking this will be pretty hard/expensive to achieve. > > > I'm aware that this is a very special case, and most probably one that > looks pretty borked. > > But think of a case where a user is interested to check an IP against > three zones of a dnsbl service. If that service offers a multi zone that > combines more than these three zones it's still easier to perform one > lookup and mask only the interesting bits in the result, rather than > performing three consecutive lookups to get the same result. > > Bye, Mike > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.1 (GNU/Linux) > > iEYEARECAAYFAkYuBwIACgkQa3V7dXg8JKuKyQCeII4bUd46D/ekgmMKpBGgOJxN > rI8AoI6chV4t8NefGDYRpG5AC/vMKDcm > =WOQo > -----END PGP SIGNATURE----- > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > mod-security-users mailing list > mod...@li... > https://lists.sourceforge.net/lists/listinfo/mod-security-users > -- Ivan Ristic |
From: Michael R. <mre...@ot...> - 2007-04-27 05:06:00
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi. > I was thinking implementing it along these lines: > [...] Sounds good. But one question: Assume that the response was 127.0.0.7. Would TX:0 then contain just the last 8 bit (7 in this case), or the whole response (127.0.0.7)? Bye, Mike -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iEYEARECAAYFAkYxhK4ACgkQa3V7dXg8JKsVWgCgg7MllpKVDMg+WUHIYXSxJeuo 6zYAnimtmSI4/MBRClOQzplsoZ6tUIQt =EwO0 -----END PGP SIGNATURE----- |
From: Brian R. <Bri...@br...> - 2007-04-27 15:04:12
|
Michael Renzmann wrote: > Hi. > >> I was thinking implementing it along these lines: >> [...] > > Sounds good. But one question: > > Assume that the response was 127.0.0.7. Would TX:0 then contain just the > last 8 bit (7 in this case), or the whole response (127.0.0.7)? > > Bye, Mike Good question :) It seems that it should contain the last octet (the 7) to follow the meanings of TX:1-8. So the capture would be on a match for the last octet in binary where a 7 would look like this as a regex, but reversed capture numbers and the captures would be converted back to decimal: 8 7 6 5 4 3 2 1 (0)(0)(0)(0)(0)(1)(1)(1) But, this is getting confusing ;) Let me think about it some more. Ideas welcome. -- Brian Rectanus Breach Security |