Thread: [mod-security-users] compiling against PCRE
Brought to you by:
victorhora,
zimmerletw
From: Justin G. <web...@sw...> - 2005-12-21 15:20:52
|
hi, What is the compile flag for apache1/PCRE? Do I need anything special installed on the server in order to use PCRE? I've seen a script called filter-spamc.pl in the util dir, but not sure what it's used for... thanks, Justin |
From: K. C. L. <li...@la...> - 2005-12-21 17:18:17
|
On Wed, 21 Dec 2005, Justin Grindea wrote: > What is the compile flag for apache1/PCRE? > Do I need anything special installed on the server in order to use PCRE? The relevant information is available in the section "Compiling the Apache 1.x version against PCRE" in doc/modsecurity.txt (and others) file. Regards, Kwong Li London |
From: Justin G. <web...@sw...> - 2005-12-21 17:38:20
|
thanks, found it. locate pcre finds the following, not sure what I need to copy to apache/libexex and what <pcre-source> would be when installing mod_security. Also not sure about loading the library before mod_security in httpd.conf, again, maybe I can use /lib/libpcre.so.0 or /usr/lib/libpcre.so in httpd.conf and compile. Ivan, can you look at this please, I want to install it tonight. thanks, Justin K. C. Li wrote: > On Wed, 21 Dec 2005, Justin Grindea wrote: > > >>What is the compile flag for apache1/PCRE? >>Do I need anything special installed on the server in order to use PCRE? > > > The relevant information is available in the section "Compiling the Apache > 1.x version against PCRE" in doc/modsecurity.txt (and others) file. > > Regards, > > Kwong Li > London > > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > mod-security-users mailing list > mod...@li... > https://lists.sourceforge.net/lists/listinfo/mod-security-users |
From: Justin G. <web...@sw...> - 2005-12-21 18:33:32
|
hmm, forgot to paste the output... here it is: /lib/libpcre.so.0 /lib/libpcre.so.0.0.1 /usr/share/man/man1/pcregrep.1.gz /usr/share/man/man1/pcretest.1.gz /usr/share/man/man3/clnt_pcreateerror.3.gz /usr/share/man/man3/clnt_spcreateerror.3.gz /usr/share/man/man3/pcreposix.3.gz /usr/share/man/man3/pcre.3.gz /usr/bin/pcregrep /usr/bin/pcretest /usr/bin/pcre-config /usr/include/pcre /usr/include/pcre/pcreposix.h /usr/include/pcre/pcre.h /usr/lib/python2.2/lib-dynload/pcre.so /usr/lib/libpcreposix.so.0 /usr/lib/libpcreposix.so.0.0.0 /usr/lib/libpcreposix.a /usr/lib/libpcre.a /usr/lib/libpcre.so /usr/lib/libpcreposix.so Justin Grindea wrote: > > thanks, > found it. > > locate pcre finds the following, not sure what I need to copy to > apache/libexex > and what <pcre-source> would be when installing mod_security. Also not > sure about loading > the library before mod_security in httpd.conf, again, maybe I can use > /lib/libpcre.so.0 > or /usr/lib/libpcre.so in httpd.conf and compile. > > Ivan, can you look at this please, I want to install it tonight. > > > thanks, > Justin > > > > K. C. Li wrote: > >> On Wed, 21 Dec 2005, Justin Grindea wrote: >> >> >>> What is the compile flag for apache1/PCRE? >>> Do I need anything special installed on the server in order to use PCRE? >> >> >> >> The relevant information is available in the section "Compiling the >> Apache >> 1.x version against PCRE" in doc/modsecurity.txt (and others) file. >> >> Regards, >> >> Kwong Li >> London >> >> >> >> >> ------------------------------------------------------- >> This SF.net email is sponsored by: Splunk Inc. Do you grep through log >> files >> for problems? Stop! Download the new AJAX search engine that makes >> searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! >> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click >> _______________________________________________ >> mod-security-users mailing list >> mod...@li... >> https://lists.sourceforge.net/lists/listinfo/mod-security-users > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > mod-security-users mailing list > mod...@li... > https://lists.sourceforge.net/lists/listinfo/mod-security-users |
From: Ivan R. <iv...@we...> - 2005-12-21 19:23:36
|
Justin Grindea wrote: > hmm, forgot to paste the output... > here it is: Try this first: <apache1-home>/bin/apxs -DUSE_PCRE -cia mod_security.c If that works but you still need to use LoadFile use: LoadFile /usr/lib/libpcre.so Otherwise just download the source from pcre.org and install it exactly as described in the manual. -- Ivan Ristic Apache Security (O'Reilly) - http://www.apachesecurity.net Open source web application firewall - http://www.modsecurity.org |
From: Zach R. <ad...@li...> - 2005-12-22 00:40:20
|
In my more updated tests it appears as if the PCRE does help quite a bit but, it still isn't enough. Mod_security cannot handle the thousands of rules necessary to secure against all the security threats there seem to be. Since gotroot.com's ruleset seems to be standard for mod_security installations I did tests with those rules. To start off I loaded the rules into the configuration in no particular order except exclude.conf being first and watched as the server became unstable then crashed. After rebooting I reordered them where the less intensive rules were first (badips.conf) and others were last but, no ordering seemed to have a very noticeable effect. The server's load went back up and it crashed again. By removing badips.conf, several thousand rules from rules.conf, and reordering them again I did get the server stable enough with "SecFilterEngine On" with low to medium traffic. When traffic picked up at 5PM the server load started to rise and the server crashed again. Any further improvements would definitely be welcomed. ;) Zach Ivan Ristic wrote: >Justin Grindea wrote: > > >>hmm, forgot to paste the output... >>here it is: >> >> > > Try this first: > <apache1-home>/bin/apxs -DUSE_PCRE -cia mod_security.c > > If that works but you still need to use LoadFile > use: > > LoadFile /usr/lib/libpcre.so > > Otherwise just download the source from pcre.org > and install it exactly as described in the manual. > > > |
From: Justin G. <web...@sw...> - 2005-12-22 00:58:19
|
badips is good maybe on a quad opteron box :) while it's impossible to use them in mod_security, we found that iptables can handle them without much pain, well, depending on the amount of traffic. Our servers do less than 5Mb in average so it's fine. One drawback is blacklist.conf which I also drop. It should be broken down to few files and sorted out by relevance/priority. rules.conf should also definatelly be edited, tons of junk and also duplicates in there. Looks like author starts to use IDs for the rules so I hope it will be easier to categorize per/server rules and make the update process easier. Also, try using DynamicOnly. How PCRE would speed up processing a PDF/SWF/JPG? Justin Zach Roberts wrote: > In my more updated tests it appears as if the PCRE does help quite a bit > but, it still isn't enough. > > Mod_security cannot handle the thousands of rules necessary to secure > against all the security threats there seem to be. > > Since gotroot.com's ruleset seems to be standard for mod_security > installations I did tests with those rules. > > To start off I loaded the rules into the configuration in no particular > order except exclude.conf being first and watched as the server became > unstable then crashed. > > After rebooting I reordered them where the less intensive rules were > first (badips.conf) and others were last but, no ordering seemed to have > a very noticeable effect. The server's load went back up and it crashed > again. > > By removing badips.conf, several thousand rules from rules.conf, and > reordering them again I did get the server stable enough with > "SecFilterEngine On" with low to medium traffic. When traffic picked up > at 5PM the server load started to rise and the server crashed again. > > Any further improvements would definitely be welcomed. ;) > > Zach > > Ivan Ristic wrote: > >> Justin Grindea wrote: >> >> >>> hmm, forgot to paste the output... >>> here it is: >>> >> >> >> Try this first: >> <apache1-home>/bin/apxs -DUSE_PCRE -cia mod_security.c >> >> If that works but you still need to use LoadFile >> use: >> >> LoadFile /usr/lib/libpcre.so >> >> Otherwise just download the source from pcre.org >> and install it exactly as described in the manual. >> >> >> > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > mod-security-users mailing list > mod...@li... > https://lists.sourceforge.net/lists/listinfo/mod-security-users |
From: Zach R. <ad...@li...> - 2005-12-22 02:47:59
|
I also removed the badips and converted it into firewall rules. ipfw2 seems to handle them just fine. blacklist.conf has quite a bit of potential to solve a rather annoying problem but, 6900+ lines of rules is just too much to effectively run for all requests. The rules.conf file I have now was cut down only to include applications that I see used. Quite a few of those seem to be for applications that aren't run quite as much. Standard: 4239 Cut down:1975 After switching it back to DynamicOnly and the cutdown rules.conf everything seems to work just fine. I didn't mean to say that mod_security alone was responsible but, its use with the complete ruleset is problematic. Optimizing and being selective is definitely necessary even with PCRE but, PCRE is a huge boost to speed any way you look at it. Zach Justin Grindea wrote: > badips is good maybe on a quad opteron box :) > while it's impossible to use them in mod_security, we found that > iptables can > handle them without much pain, well, depending on the amount of > traffic. Our > servers do less than 5Mb in average so it's fine. > > One drawback is blacklist.conf which I also drop. It should be broken > down to > few files and sorted out by relevance/priority. > > rules.conf should also definatelly be edited, tons of junk and also > duplicates in > there. Looks like author starts to use IDs for the rules so I hope it > will be easier > to categorize per/server rules and make the update process easier. > > Also, try using DynamicOnly. How PCRE would speed up processing a > PDF/SWF/JPG? > > Justin > > > Zach Roberts wrote: > >> In my more updated tests it appears as if the PCRE does help quite a >> bit but, it still isn't enough. >> >> Mod_security cannot handle the thousands of rules necessary to secure >> against all the security threats there seem to be. >> >> Since gotroot.com's ruleset seems to be standard for mod_security >> installations I did tests with those rules. >> >> To start off I loaded the rules into the configuration in no >> particular order except exclude.conf being first and watched as the >> server became unstable then crashed. >> >> After rebooting I reordered them where the less intensive rules were >> first (badips.conf) and others were last but, no ordering seemed to >> have a very noticeable effect. The server's load went back up and it >> crashed again. >> >> By removing badips.conf, several thousand rules from rules.conf, and >> reordering them again I did get the server stable enough with >> "SecFilterEngine On" with low to medium traffic. When traffic picked >> up at 5PM the server load started to rise and the server crashed again. >> >> Any further improvements would definitely be welcomed. ;) >> >> Zach >> >> Ivan Ristic wrote: >> >>> Justin Grindea wrote: >>> >>> >>>> hmm, forgot to paste the output... >>>> here it is: >>>> >>> >>> >>> >>> Try this first: >>> <apache1-home>/bin/apxs -DUSE_PCRE -cia mod_security.c >>> >>> If that works but you still need to use LoadFile >>> use: >>> >>> LoadFile /usr/lib/libpcre.so >>> >>> Otherwise just download the source from pcre.org >>> and install it exactly as described in the manual. >>> >>> >>> >> >> >> >> ------------------------------------------------------- >> This SF.net email is sponsored by: Splunk Inc. Do you grep through >> log files >> for problems? Stop! Download the new AJAX search engine that makes >> searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! >> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click >> _______________________________________________ >> mod-security-users mailing list >> mod...@li... >> https://lists.sourceforge.net/lists/listinfo/mod-security-users > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > mod-security-users mailing list > mod...@li... > https://lists.sourceforge.net/lists/listinfo/mod-security-users |
From: Ivan R. <iv...@we...> - 2005-12-22 12:11:00
|
On 12/22/05, Zach Roberts <ad...@li...> wrote: > In my more updated tests it appears as if the PCRE does help quite a bit > but, it still isn't enough. Hi Zach, Thanks for the update. > Since gotroot.com's ruleset seems to be standard for mod_security > installations I did tests with those rules. I have to disagree slightly. I don't think there is such thing as standard rules for ModSecurity. That's why I don't include any with the distribution. ModSecurity is a versatile tool. It can be applied to many different scenarios and there can not be a single rule set that fits them all. Having thousands of rules in the configuration is clearly wrong, even if there were no performance problems. Take the bad IP addresses list, for example. It makes no sense (at least to me) to watch for them on the Apache level. As you have noted, a much better approach is to restrict access at the firewall. That way Apache would not even have to bother. ModSecurity should, in my opinion, be configured with a couple of hundred of rules at the most. Processing of such a rule set takes only a millisecond or two on a reasonably fast box. Processing thousands of regular expressions for every request, where 99% of them do not apply, is a tremendous waste of resources. For that one would clearly need to either use specialised hardware (making the regexes much faster) or front the web servers with a cluster of reverse proxies with ModSecurity. People running shared hosting facilities are clearly in a very difficult position. (I used to do just that in my previous job, BTW.) I think a completely different approach is needed to solve the problem: 1) completely isolate customers from one another (so that an intrusion in one account can not affect the others), 2) give customers option to run certified applications (applications which can be patched automatically), and 3) explain to those that do not choose option 2 that they are responsible for maintaining security. Ivan |
From: Tom A. <tan...@oa...> - 2005-12-22 15:30:35
|
Zach Roberts wrote: > Mod_security cannot handle the thousands of rules necessary to secure > against all the security threats there seem to be. Thousands of rules? Are you mad? No wonder your server is screeching to a halt. Rather than starting with thousands of rules and slowly removing them as they are unneeded, how about starting with the bare minimum and adding rules as needed. I'd be surprised if I run with more than two dozen rules! I can understand the desire to protect against any possible entry point, but you should really customize your rule set to the applications actually running on your server. For instance, if you don't have PHP, then you shouldn't need any rules which address PHP vulnerabilities. If you don't have a blog, then you don't need blog spamming rules. And I think blacklisting rules are probably unnecessary overhead for anyone unless you can do something like a DNS block list where it is a simple query instead of a huge list of sequential rules. Finally, I would imagine that condensing rules would help significantly, but maybe someone who has time can test it for certain. But intuitively, I'd think that defining a new rule would require more instructions than tacking on elements to another rule. For instance, if you want to block these IPs: SecFilterSelective REMOTE_ADDR 192\.168\.123\.456 SecFilterSelective REMOTE_ADDR 10\.2\.3\.4 SecFilterSelective REMOTE_ADDR 172\.16\.17\.18 SecFilterSelective REMOTE_ADDR 192\.168\.123\.789 SecFilterSelective REMOTE_ADDR 192\.168\.456\.789 SecFilterSelective REMOTE_ADDR 10\.3\.4\.5 SecFilterSelective REMOTE_ADDR 10\.2\.3\.10 SecFilterSelective REMOTE_ADDR 172\.16\.17\.20 Then you should be able to write this as one rule: SecFilterSelective REMOTE_ADDR (192\.(168\.(123\.(456|789)|456\.789))|(10\.(2\.3\.(4|10)|3\.4\.5))|(172\.(16\.17\.(18|20)))) Or if you wanted to block these URLs: SecFilterSelective THE_REQUEST www\.hackers\.com SecFilterSelective THE_REQUEST www\.hackerz\.com SecFilterSelective THE_REQUEST www\.hax0rz\.com SecFilterSelective THE_REQUEST www\.h4ckers\.com SecFilterSelective THE_REQUEST www\.hack3rs\.net SecFilterSelective THE_REQUEST www\.hackerz\.net SecFilterSelective THE_REQUEST www\.h4x0rz\.net SecFilterSelective THE_REQUEST www\.h4ckerz\.net Seems like quite a cacophony of spellings, but you could use this one rule: SecFilterSelective THE_REQUEST www\.h(a|4)(ck|x)(e|3|0)r(s|z)\.(com|net) Let's say that the actual URL coming to the site is "www.h4ckerz.net"... then in the first case (multiple rules), it would begin each rule, start matching the "www\.h" and then fail, over and over again, until it finally matched the whole string. In the second case, it matches on the very first rule and only fails in part, such as on the "a" instead of "4", but then picks up where it left off without having to start all over again with a new rule. But like I said, I don't know the interal processing of these things, so it may not make a huge difference. The best thing to do would be to test it. Tom |
From: Justin G. <web...@sw...> - 2005-12-22 22:02:48
|
ok, my finding are not so good. load didn't go down drastically on a quite busy server loaded with quite a lot rules from gotroot. Trying to load gotroot's blacklist.conf immediatelly raised the load way above normal use and I had it off in 10 seconds. thanks, Justin Ivan Ristic wrote: > Justin Grindea wrote: > >>hmm, forgot to paste the output... >>here it is: > > > Try this first: > <apache1-home>/bin/apxs -DUSE_PCRE -cia mod_security.c > > If that works but you still need to use LoadFile > use: > > LoadFile /usr/lib/libpcre.so > > Otherwise just download the source from pcre.org > and install it exactly as described in the manual. > |
From: Zach R. <ad...@li...> - 2005-12-22 22:13:28
|
You need to abandon the blacklist.conf completely. You just cannot load that ruleset under a heavy load and keep your server stable. Also change from On to DynamicOnly. Load the badips.conf IPs into your firewall, keep the blacklist2.conf, and load these rules from the blacklist.conf into your rules.conf: SecFilterSelective REQUEST_URI "!(horde/imp/compose\.php\?)" chain SecFilterSelective THE_REQUEST "Subject\:" chain SecFilterSelective ARG_Bcc ".*\@" SecFilterSelective REQUEST_URI "!(horde/imp/compose\.php\?)" chain SecFilterSelective POST_PAYLOAD "Subject\:" chain SecFilterSelective POST_PAYLOAD "\s*bcc\:" SecFilterSelective REQUEST_URI "!(horde/imp/compose\.php\?)" chain SecFilterSelective POST_PAYLOAD "\s*bcc\:\s*[a-z0-9._%-]+@[A-Z0-9.-]+\.[a-z]{2,}" SecFilterSelective REQUEST_URI "!(horde/imp/compose\.php\?)" chain SecFilterSelective ARGS_VALUES "\s*bcc\:\s*[a-z0-9._%-]+\@.*\.[a-z]{2,}" SecFilterSelective HTTP_x-aaaaaaaaa|HTTP_XAAAAAAAAA ".+$" SecFilterSelective HTTP_x-aaaaaaaaaaa|HTTP_XAAAAAAAAAAA ".+$" SecFilterSelective HTTP_x-aaaaaaaaaaaa|HTTP_X_AAAAAAAAAAAA ".+$" That will help quite a bit. :) Zach Justin Grindea wrote: > ok, my finding are not so good. load didn't go down drastically on a > quite busy server loaded with quite a lot > rules from gotroot. > Trying to load gotroot's blacklist.conf immediatelly raised the load > way above normal use and I had it off in 10 > seconds. > > thanks, > Justin > > > Ivan Ristic wrote: > >> Justin Grindea wrote: >> >>> hmm, forgot to paste the output... >>> here it is: >> >> >> >> Try this first: >> <apache1-home>/bin/apxs -DUSE_PCRE -cia mod_security.c >> >> If that works but you still need to use LoadFile >> use: >> >> LoadFile /usr/lib/libpcre.so >> >> Otherwise just download the source from pcre.org >> and install it exactly as described in the manual. >> > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > mod-security-users mailing list > mod...@li... > https://lists.sourceforge.net/lists/listinfo/mod-security-users |