Re: [mod-security-users] Disappointed with the Google Safe List implementation
Brought to you by:
victorhora,
zimmerletw
Menu
▾
▴
Re: [mod-security-users] Disappointed with the Google Safe List implementation
|
From: Breno S. <bre...@gm...> - 2011-05-25 20:42:19
|
Hello, I'd like to register some informations about some of tests we did (me and Phoenix). Right now the faster way to lookup into Gsb database is loading the data into memory during the apache startup, however it will increase +10Mb the apache tasks. So you probably will need to re-tuning your apache settings. We notice that on Linux Kernel 2.6 the forking/kill became more expensive, so you should consider change apache setting to reduce this operations. The Gsblookup operator is fast, spending some small usecs to process the data. The problem seems to be the overhead caused on Linux systems for forking/kill tasks. I also changed the data structure inside modsecurity to make add/search operations faster, will be available in 2.6.1. However on heavy systems a tunning process is a must, specially on Linux systems. I spent some time tunning my Linux box and got some good results, apache+worker remained stable, a little bit slow after the tunning... but responding well. Phoenix did some changes on his env but without success. Also i did tests on FreeBSD 8 and the behavior was too different. FreeBSD remained very stable and apache worked fine. I will continue work to find a way to make better for future versions. Any suggestions in the code, please send to devel list. Thanks Breno On Sun, May 22, 2011 at 6:17 PM, Breno Silva <bre...@gm...> wrote: > Please follow the MODSEC-245 created today. I want to implement it next > week. So if really want it quickly for testing, you can get the lastest sn > branch 2.6 version .. or i can send you a tarball for testing. > > thanks > > Breno > > > On Sun, May 22, 2011 at 6:05 PM, Breno Silva <bre...@gm...>wrote: > >> Phoenix, >> >> This is a info when the rules match: >> >> [22/May/2011:15:28:41 --0700] [ >> 192.168.0.101/sid#2276e6c0][rid#2278c698][/index.html][9<http://192.168.0.101/sid#2276e6c0][rid%232278c698%5D%5B/index.html%5D%5B9>] >> Target value: "malware.testing.google.test/testing/malware/" >> [22/May/2011:15:28:41 --0700] [ >> 192.168.0.101/sid#2276e6c0][rid#2278c698][/index.html][4<http://192.168.0.101/sid#2276e6c0][rid%232278c698%5D%5B/index.html%5D%5B4>] >> GSB: Successfully extracted url: >> malware.testing.google.test/testing/malware/ >> [22/May/2011:15:28:41 --0700] [ >> 192.168.0.101/sid#2276e6c0][rid#2278c698][/index.html][9<http://192.168.0.101/sid#2276e6c0][rid%232278c698%5D%5B/index.html%5D%5B9>] >> Added phrase match to TX.0: malware.testing.google.test/testing/malware/ >> [22/May/2011:15:28:41 --0700] [ >> 192.168.0.101/sid#2276e6c0][rid#2278c698][/index.html][9<http://192.168.0.101/sid#2276e6c0][rid%232278c698%5D%5B/index.html%5D%5B9>] >> Added phrase match to TX.1: malware.testing.google.test >> [22/May/2011:15:28:41 --0700] [ >> 192.168.0.101/sid#2276e6c0][rid#2278c698][/index.html][4<http://192.168.0.101/sid#2276e6c0][rid%232278c698%5D%5B/index.html%5D%5B4>] >> Operator completed in 86 usec. >> >> Also... i will change (modsec 2.6.1) the data struct (tables - apr) to >> another that is faster with insert/append operation. >> >> thanks >> >> Breno >> >> >> >> On Sun, May 22, 2011 at 4:35 PM, Breno Silva <bre...@gm...>wrote: >> >>> Hi Phoenix, >>> >>> We need to see some debug log from your side to help you. Below i'm >>> sending a test sending ~ 1.500 request per second and inspecting the >>> ARGS:url with and without GSB. >>> >>> You can see that there is a delay but not too much. >>> >>> 22/May/2011:13:18:03 --0700] [ >>> 192.168.0.101/sid#b63ea718][rid#214473f0][/index.html][4<http://192.168.0.101/sid#b63ea718][rid%23214473f0%5D%5B/index.html%5D%5B4>] >>> Executing operator "gsbLookup" with param "\\Whttps?\\:\\/\\/(.*?)(\"|>)" >>> against ARGS:url. >>> [22/May/2011:13:18:03 --0700] [ >>> 192.168.0.101/sid#b63ea718][rid#214473f0][/index.html][9<http://192.168.0.101/sid#b63ea718][rid%23214473f0%5D%5B/index.html%5D%5B9>] >>> Target value: "http://www.terra.com.br/b/c" >>> [22/May/2011:13:18:03 --0700] [ >>> 192.168.0.101/sid#b63ea718][rid#214473f0][/index.html][4<http://192.168.0.101/sid#b63ea718][rid%23214473f0%5D%5B/index.html%5D%5B4>] >>> Operator completed in 22 usec. >>> >>> The operator spent 22usec. >>> >>> You will see that in my tests the apache spent 2 seconds more (with GSB) >>> to answer all my requests.... so around 30ms more per requests. >>> >>> You also have new PERF* variables to help you get some new stats. >>> >>> Also i suggest you to review your CRS... i dont know if you are using the >>> full CRS... but you can consider tunning it for your env. You also should >>> leave apache running for some time with GSB feature enabled... because it >>> spent some time loading the GSB database. >>> >>> Please follow the Ryan's instructions... it will give us some info to >>> help you. >>> >>> thanks >>> >>> Breno >>> >>> >>> RESULTS: >>> >>> Without GSB: >>> >>> sh-3.2# ab -n 20000 -c 500 " >>> http://192.168.0.101/index.html?url=www.hacker.com/malware/" >>> This is ApacheBench, Version 2.3 <$Revision: 655654 $> >>> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ >>> Licensed to The Apache Software Foundation, http://www.apache.org/ >>> >>> Benchmarking 192.168.0.101 (be patient) >>> Completed 2000 requests >>> Completed 4000 requests >>> Completed 6000 requests >>> Completed 8000 requests >>> Completed 10000 requests >>> Completed 12000 requests >>> Completed 14000 requests >>> Completed 16000 requests >>> Completed 18000 requests >>> Completed 20000 requests >>> Finished 20000 requests >>> >>> >>> Server Software: Apache/2.2.14 >>> Server Hostname: 192.168.0.101 >>> Server Port: 80 >>> >>> Document Path: /index.html?url=www.hacker.com/malware/ >>> Document Length: 4101 bytes >>> >>> Concurrency Level: 500 >>> Time taken for tests: 12.878 seconds >>> Complete requests: 20000 >>> Failed requests: 0 >>> Write errors: 0 >>> Total transferred: 87922896 bytes >>> HTML transferred: 82022601 bytes >>> Requests per second: 1553.06 [#/sec] (mean) >>> Time per request: 321.945 [ms] (mean) >>> Time per request: 0.644 [ms] (mean, across all concurrent requests) >>> Transfer rate: 6667.47 [Kbytes/sec] received >>> >>> Connection Times (ms) >>> min mean[+/-sd] median max >>> Connect: 0 89 427.2 3 5036 >>> Processing: 2 161 546.6 81 12855 >>> Waiting: 2 139 544.6 74 12855 >>> Total: 4 251 746.3 85 12872 >>> >>> Percentage of the requests served within a certain time (ms) >>> 50% 85 >>> 66% 95 >>> 75% 125 >>> 80% 150 >>> 90% 283 >>> 95% 1053 >>> 98% 2138 >>> 99% 3379 >>> 100% 12872 (longest request) >>> sh-3.2# >>> >>> >>> With GSB: >>> >>> sh-3.2# ab -n 20000 -c 500 " >>> http://192.168.0.101/index.html?url=www.hacker.com/malware/" >>> This is ApacheBench, Version 2.3 <$Revision: 655654 $> >>> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ >>> Licensed to The Apache Software Foundation, http://www.apache.org/ >>> >>> Benchmarking 192.168.0.101 (be patient) >>> Completed 2000 requests >>> Completed 4000 requests >>> Completed 6000 requests >>> Completed 8000 requests >>> Completed 10000 requests >>> Completed 12000 requests >>> Completed 14000 requests >>> Completed 16000 requests >>> Completed 18000 requests >>> Completed 20000 requests >>> Finished 20000 requests >>> >>> >>> Server Software: Apache/2.2.14 >>> Server Hostname: 192.168.0.101 >>> Server Port: 80 >>> >>> Document Path: /index.html?url=www.hacker.com/malware/ >>> Document Length: 4101 bytes >>> >>> Concurrency Level: 500 >>> Time taken for tests: 14.246 seconds >>> Complete requests: 20000 >>> Failed requests: 0 >>> Write errors: 0 >>> Total transferred: 87924396 bytes >>> HTML transferred: 82024101 bytes >>> Requests per second: 1403.91 [#/sec] (mean) >>> Time per request: 356.147 [ms] (mean) >>> Time per request: 0.712 [ms] (mean, across all concurrent requests) >>> Transfer rate: 6027.26 [Kbytes/sec] received >>> >>> Connection Times (ms) >>> min mean[+/-sd] median max >>> Connect: 0 79 387.3 3 5035 >>> Processing: 3 214 803.7 96 12945 >>> Waiting: 1 186 800.3 83 12904 >>> Total: 10 293 919.4 101 12967 >>> >>> Percentage of the requests served within a certain time (ms) >>> 50% 101 >>> 66% 123 >>> 75% 154 >>> 80% 179 >>> 90% 327 >>> 95% 1061 >>> 98% 3040 >>> 99% 4336 >>> 100% 12967 (longest request) >>> >>> >>> >>> >>> On Sun, May 22, 2011 at 12:08 PM, Ryan Barnett <RBa...@tr...>wrote: >>> >>>> Yeah that won't work because your rule isn't matching. What you need to >>>> do is create a separate rule that inspects REMOTE_ADDR and specify your >>>> client IP and then use the ctl action to enable debug logging only fir your >>>> traffic and not for all visitors to your site. >>>> >>>> When this is in place you then interact with that page and review the >>>> debug log data. >>>> >>>> Ryan >>>> >>>> On May 22, 2011, at 12:21 PM, Phoenix Kiula <pho...@gm...> >>>> wrote: >>>> >>>> > On Sun, May 22, 2011 at 11:47 PM, Ryan Barnett < >>>> RBa...@tr...> wrote: >>>> >> I sent you reference info yesterday about conditional debug logging. >>>> >> >>>> > >>>> > >>>> > >>>> > The ctl thing? It doesn't work. My debug log is still 0 bytes. >>>> > >>>> > I have this: >>>> > >>>> > >>>> > SecGsbLookupDB /usr/local/apache/conf/gsb-bl.dat >>>> > SecRule ARGS:url|ARGS:link "@gsbLookup (http|https)\:\/\/(.*)" >>>> > "phase:2,ctl:debugLogLevel=9,log,deny,msg:'Google Malware Check'" >>>> > >>>> > >>>> > However, earlier on in the modsec2.user.conf I have >>>> > >>>> > SecDebugLogLevel 0 >>>> > >>>> > My understanding is that the rule-specific debugloglevel will >>>> > basically override this broader rule? >>>> > >>>> > Thanks. >>>> > >>>> > >>>> ------------------------------------------------------------------------------ >>>> > What Every C/C++ and Fortran developer Should Know! >>>> > Read this article and learn how Intel has extended the reach of its >>>> > next-generation tools to help Windows* and Linux* C/C++ and Fortran >>>> > developers boost performance applications - including clusters. >>>> > http://p.sf.net/sfu/intel-dev2devmay >>>> > _______________________________________________ >>>> > mod-security-users mailing list >>>> > mod...@li... >>>> > https://lists.sourceforge.net/lists/listinfo/mod-security-users >>>> > ModSecurity Services from Trustwave's SpiderLabs: >>>> > https://www.trustwave.com/spiderLabs.php >>>> >>>> This transmission may contain information that is privileged, >>>> confidential, and/or exempt from disclosure under applicable law. If you are >>>> not the intended recipient, you are hereby notified that any disclosure, >>>> copying, distribution, or use of the information contained herein (including >>>> any reliance thereon) is STRICTLY PROHIBITED. If you received this >>>> transmission in error, please immediately contact the sender and destroy the >>>> material in its entirety, whether in electronic or hard copy format. >>>> >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> What Every C/C++ and Fortran developer Should Know! >>>> Read this article and learn how Intel has extended the reach of its >>>> next-generation tools to help Windows* and Linux* C/C++ and Fortran >>>> developers boost performance applications - including clusters. >>>> http://p.sf.net/sfu/intel-dev2devmay >>>> _______________________________________________ >>>> mod-security-users mailing list >>>> mod...@li... >>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users >>>> ModSecurity Services from Trustwave's SpiderLabs: >>>> https://www.trustwave.com/spiderLabs.php >>>> >>> >>> >> > |