Re: [mod-security-users] rule performance

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Hi Bruno,

are you still using nginx? Or did you moved to Apache?

Felipe ³Zimmerle² Costa
Security Researcher, SpiderLabs

Trustwave | SMART SECURITY ON DEMAND
www.trustwave.com <http://www.trustwave.com/>

On 3/25/15, 1:48 PM, "Christian Folini" <chr...@ti...>
wrote:

>Hey Bruno,
>
>I had hoped firing a broadside at your problem might solve it,
>but I see a more selective approach is needed.
>
>Unfortunately, I am not that much acquainted with the
>Stopwatch2 info to tell you what it really represents.
>The audit-log format is underdocumented in my eyes.
>
>What I would do in your case would be to work with the
>output in the DebugLog. If you raise the loglevel to
>a sufficiently high value (5? 9?), the performance of
>every rule executed is reported as it's being executed.
>This should help to nail down the culprit.
>
>Your perf problem is not unheard off, but usually there
>is a solution.
>
>Ahoj,
>
>Christian
>
>
>On Wed, Mar 25, 2015 at 12:35:49PM -0400, Bruno de Almeida wrote:
>> Hi Christian,
>>
>> Thanks for the link, but it looks like all the information generated
>>from
>> that tutorial is available in the Stopwatch2 field.
>>
>> The problem I am having are with rules that are not triggering, but
>>still
>> being executed.
>>
>> >From what I understand, the information on Stopwatch2 includes the
>>combined
>> performance of ALL the rules that analysed that specific request and not
>> only the triggered rule. I'd like to know the Phase2 times for each
>>rule,
>> so I can tell which ones are causing the times to go up.
>>
>> Basically, I am upgrading my owasp crs rules from a very old version to
>>the
>> latest and I found that the new ones are a LOT slower  and apache uses a
>> lot more cpu.
>>
>> Using the example I gave above of one of my custom authentication rule,
>> with the old owasp crs installed, the avg Phase 2 time is 4100usec, with
>> owasp crs 2.2.9, the avg goes up to 12000usec.
>>
>> Hope I am making sense..
>>
>>
>> Bruno
>>
>> On 25 March 2015 at 00:32, Christian Folini <
>> chr...@ti...> wrote:
>>
>> > Hi Bruno,
>> >
>> > There is a (German) tutorial at
>> >
>>http://scanmail.trustwave.com/?c=4062&d=vOeS1Z1rnm8cYx9uRXjjrHn2EQREOKjs7
>>Q2bjxUfyQ&s=5&u=http%3a%2f%2fwww%2enetnea%2ecom%2fcms%2fapache-tutorial-6
>>-modsecurity-einbinden%2f
>> > which brings a complete apache/modsec configuration with an
>> > extensive performance log that covers all the phases seperately
>> > and which can be switched on and off based on a SecRule->Env-Variable.
>> >
>> > Even if you do not read German, you should be able to get it working
>> > for your.
>> >
>> > Let me know, if you need any help.
>> >
>> > Ahoj,
>> >
>> > Christian
>> >
>> >
>> > On Tue, Mar 24, 2015 at 06:03:33PM -0400, Bruno de Almeida wrote:
>> > > Hi All,
>> > >
>> > > I am trying to find out the most expensive rules I have on my setup
>>using
>> > > SecRulePerfTime and PERF_RULES but the data is not making much
>>sense.
>> > >
>> > >
>> > > For example, I have this one custom rule that is tracking
>>authentication
>> > > events, it basically inspects POST to a specific URL and grabs some
>>info.
>> > >
>> > > SecRule REQUEST_FILENAME "@strmatch j_spring_security_check"
>> > >
>> >
>>"chain,phase:3,id:'7002',t:none,pass,nolog,auditlog,severity:6,msg:'Succe
>>ssful
>> > > Authentication',logdata:'email=%{args.j_username}'"
>> > > SecRule REQUEST_METHOD "@streq POST" "chain,t:none"
>> > > SecRule RESPONSE_STATUS "@streq 302" "chain,t:none"
>> > > SecRule RESPONSE_HEADERS:Location "!@strmatch signin" "chain,t:none"
>> > > SecRule WEBAPPID "@strmatch www" "chain,t:none"
>> > > SecRule ARGS:j_username ".*" "t:none"
>> > >
>> > >
>> > > The combined time for this rule varies between 10000 and 40000, 99%
>>of
>> > this
>> > > time is in Phase 2, as you can see below
>> > >
>> > > Stopwatch2: 1427233612156631 1027209; combined=30177, p1=420,
>>p2=29315,
>> > > p3=329, p4=51, p5=60, sr=49, sw=2, l=0, gc=0
>> > >
>> > >
>> > > I have added SecRulePerfTime 1 to my config so I can see all rules
>>taking
>> > > 1+ usec.
>> > >
>> > > The problem is, according to Rules-Performance-Info, the sum of all
>>rules
>> > > taking more than 1 usec is nowhere near p2=29315, and, still
>>acording
>> > > to Rules-Performance-Info, the Authentication Tracking rule is only
>> > taking
>> > > "2 usecs" ("7002=2")
>> > >
>> > > The sum of all the processing times below is *850*, a lot less than
>>the
>> > > combined=30177
>> > >
>> > > Rules-Performance-Info: "900012=1", "900018=2", "900019=4",
>>"1000=4",
>> > > "900020=1", "900021=2", "5001=2", "5006=1", "5009=1", "5010=1",
>>"4001=1",
>> > > "4002=1", "960911=6", "960016=1", "960012=1", "960342=1",
>>"960032=3",
>> > > "950012=1", "10001=6", "
>> > > 10002=1", "11003=1", "4003=3", "900040=1", "960912=2", "960914=1",
>> > > "960915=1", "958295=1", "950108=4", "950116=1", "960901=15",
>>"960008=1",
>> > > "960006=1", "960017=1", "960209=1", "960208=1", "960335=1",
>>"960341=1",
>> > > "981078=3", "960034=4", "9
>> > > 60035=1", "960038=36", "990002=7", "990901=7", "990902=1",
>>"990012=7",
>> > > "950907=4", "950018=1", "950019=2", "950910=39", "950911=43",
>>"950117=3",
>> > > "950118=2", "950119=2", "950120=1", "981133=20", "981134=1",
>>"950009=36",
>> > > "950003=3", "950000
>> > > =3", "950005=4", "950002=5", "950006=6", "981231=3", "981260=5",
>> > > "981318=6", "981319=9", "950901=7", "981320=3", "981300=10",
>>"981303=1",
>> > > "981304=1", "981306=1", "981307=1", "981308=1", "981309=1",
>>"981311=1",
>> > > "981312=1", "981313=1", "981
>> > > 314=1", "950007=6", "950001=16", "959070=8", "959071=2", "959072=1",
>> > > "950908=2", "959073=23", "981272=2", "981244=8", "981255=7",
>>"981257=6",
>> > > "981248=10", "981277=3", "981250=4", "981241=3", "981252=4",
>>"981256=4",
>> > > "981245=9", "981276=2",
>> > >  "981254=2", "981270=1", "981240=6", "981249=7", "981253=4",
>>"981242=11",
>> > > "981246=7", "981251=4", "981247=8", "981243=6", "973336=1",
>>"973337=3",
>> > > "973338=6", "981136=14", "981018=1", "973300=3", "973301=1",
>>"973302=29",
>> > > "973303=5", "97330
>> > > 4=4", "973305=2", "973306=3", "973307=2", "973308=2", "973309=3",
>> > > "973310=1", "973311=3", "973312=2", "973313=3", "973314=3",
>>"973331=2",
>> > > "973315=1", "973330=4", "973327=4", "973326=5", "973346=8",
>>"973345=4",
>> > > "973324=2", "973323=3", "973
>> > > 322=2", "973348=2", "973321=3", "973320=1", "973318=1", "973317=2",
>> > > "973347=2", "973335=1", "973334=3", "973333=3", "973332=3",
>>"973329=1",
>> > > "973328=1", "973316=1", "973325=1", "973319=2", "950103=22",
>>"950110=1",
>> > > "981020=1", "981022=2", "
>> > > 981175=1", "1001=1", "2010=1", "2011=1", "3001=12", "9001=1",
>>"200003=1",
>> > > "200004=1", "7002=2", "7003=1", "7004=1", "7006=2", "7009=1",
>>"8002=2",
>> > > "8003=1", "1234=66", "981080=3", "970118=1", "981177=1", "981004=1",
>> > > "981007=1", "981200=2",
>> > >  "981201=2", "981204=3", "981205=13", "7001=1".
>> > >
>> > > Does anyone know if Rules-Performance-Info takes into account all
>>phases?
>> > > If not, which phase is it reporting?
>> > >
>> > > Thanks,
>> > >
>> > >
>> > > --
>> > > - Bruno
>> >
>> > >
>> >
>>-------------------------------------------------------------------------
>>-----
>> > > Dive into the World of Parallel Programming The Go Parallel Website,
>> > sponsored
>> > > by Intel and developed in partnership with Slashdot Media, is your
>>hub
>> > for all
>> > > things parallel software development, from weekly thought leadership
>> > blogs to
>> > > news, videos, case studies, tutorials and more. Take a look and
>>join the
>> > > conversation now.
>>http://scanmail.trustwave.com/?c=4062&d=vOeS1Z1rnm8cYx9uRXjjrHn2EQREOKjs7
>>VXL30ZBmg&s=5&u=http%3a%2f%2fgoparallel%2esourceforge%2enet%2f
>> >
>> > > _______________________________________________
>> > > mod-security-users mailing list
>> > > mod...@li...
>> > >
>>http://scanmail.trustwave.com/?c=4062&d=vOeS1Z1rnm8cYx9uRXjjrHn2EQREOKjs7
>>QnJiRIdyA&s=5&u=https%3a%2f%2flists%2esourceforge%2enet%2flists%2flistinf
>>o%2fmod-security-users
>> > > Commercial ModSecurity Rules and Support from Trustwave's
>>SpiderLabs:
>> > >
>>http://scanmail.trustwave.com/?c=4062&d=vOeS1Z1rnm8cYx9uRXjjrHn2EQREOKjs7
>>Vub3UtJzA&s=5&u=http%3a%2f%2fwww%2emodsecurity%2eorg%2fprojects%2fcommerc
>>ial%2frules%2f
>> > >
>>http://scanmail.trustwave.com/?c=4062&d=vOeS1Z1rnm8cYx9uRXjjrHn2EQREOKjs7
>>Vqfj0cdmQ&s=5&u=http%3a%2f%2fwww%2emodsecurity%2eorg%2fprojects%2fcommerc
>>ial%2fsupport%2f
>> >
>> >
>> >
>> >
>>-------------------------------------------------------------------------
>>-----
>> > Dive into the World of Parallel Programming The Go Parallel Website,
>> > sponsored
>> > by Intel and developed in partnership with Slashdot Media, is your
>>hub for
>> > all
>> > things parallel software development, from weekly thought leadership
>>blogs
>> > to
>> > news, videos, case studies, tutorials and more. Take a look and join
>>the
>> > conversation now.
>>http://scanmail.trustwave.com/?c=4062&d=vOeS1Z1rnm8cYx9uRXjjrHn2EQREOKjs7
>>VXL30ZBmg&s=5&u=http%3a%2f%2fgoparallel%2esourceforge%2enet%2f
>> > _______________________________________________
>> > mod-security-users mailing list
>> > mod...@li...
>> >
>>http://scanmail.trustwave.com/?c=4062&d=vOeS1Z1rnm8cYx9uRXjjrHn2EQREOKjs7
>>QnJiRIdyA&s=5&u=https%3a%2f%2flists%2esourceforge%2enet%2flists%2flistinf
>>o%2fmod-security-users
>> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> >
>>http://scanmail.trustwave.com/?c=4062&d=vOeS1Z1rnm8cYx9uRXjjrHn2EQREOKjs7
>>Vub3UtJzA&s=5&u=http%3a%2f%2fwww%2emodsecurity%2eorg%2fprojects%2fcommerc
>>ial%2frules%2f
>> >
>>http://scanmail.trustwave.com/?c=4062&d=vOeS1Z1rnm8cYx9uRXjjrHn2EQREOKjs7
>>Vqfj0cdmQ&s=5&u=http%3a%2f%2fwww%2emodsecurity%2eorg%2fprojects%2fcommerc
>>ial%2fsupport%2f
>> >
>>
>>
>>
>> --
>> - Bruno
>
>>
>>-------------------------------------------------------------------------
>>-----
>> Dive into the World of Parallel Programming The Go Parallel Website,
>>sponsored
>> by Intel and developed in partnership with Slashdot Media, is your hub
>>for all
>> things parallel software development, from weekly thought leadership
>>blogs to
>> news, videos, case studies, tutorials and more. Take a look and join
>>the
>> conversation now.
>>http://scanmail.trustwave.com/?c=4062&d=vOeS1Z1rnm8cYx9uRXjjrHn2EQREOKjs7
>>VXL30ZBmg&s=5&u=http%3a%2f%2fgoparallel%2esourceforge%2enet%2f
>
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>>
>>http://scanmail.trustwave.com/?c=4062&d=vOeS1Z1rnm8cYx9uRXjjrHn2EQREOKjs7
>>QnJiRIdyA&s=5&u=https%3a%2f%2flists%2esourceforge%2enet%2flists%2flistinf
>>o%2fmod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>
>>http://scanmail.trustwave.com/?c=4062&d=vOeS1Z1rnm8cYx9uRXjjrHn2EQREOKjs7
>>Vub3UtJzA&s=5&u=http%3a%2f%2fwww%2emodsecurity%2eorg%2fprojects%2fcommerc
>>ial%2frules%2f
>>
>>http://scanmail.trustwave.com/?c=4062&d=vOeS1Z1rnm8cYx9uRXjjrHn2EQREOKjs7
>>Vqfj0cdmQ&s=5&u=http%3a%2f%2fwww%2emodsecurity%2eorg%2fprojects%2fcommerc
>>ial%2fsupport%2f
>
>
>--------------------------------------------------------------------------
>----
>Dive into the World of Parallel Programming The Go Parallel Website,
>sponsored
>by Intel and developed in partnership with Slashdot Media, is your hub
>for all
>things parallel software development, from weekly thought leadership
>blogs to
>news, videos, case studies, tutorials and more. Take a look and join the
>conversation now.
>http://scanmail.trustwave.com/?c=4062&d=vOeS1Z1rnm8cYx9uRXjjrHn2EQREOKjs7V
>XL30ZBmg&s=5&u=http%3a%2f%2fgoparallel%2esourceforge%2enet%2f
>_______________________________________________
>mod-security-users mailing list
>mod...@li...
>http://scanmail.trustwave.com/?c=4062&d=vOeS1Z1rnm8cYx9uRXjjrHn2EQREOKjs7Q
>nJiRIdyA&s=5&u=https%3a%2f%2flists%2esourceforge%2enet%2flists%2flistinfo%
>2fmod-security-users
>Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>http://scanmail.trustwave.com/?c=4062&d=vOeS1Z1rnm8cYx9uRXjjrHn2EQREOKjs7V
>ub3UtJzA&s=5&u=http%3a%2f%2fwww%2emodsecurity%2eorg%2fprojects%2fcommercia
>l%2frules%2f
>http://scanmail.trustwave.com/?c=4062&d=vOeS1Z1rnm8cYx9uRXjjrHn2EQREOKjs7V
>qfj0cdmQ&s=5&u=http%3a%2f%2fwww%2emodsecurity%2eorg%2fprojects%2fcommercia
>l%2fsupport%2f

________________________________

This transmission may contain information that is privileged, confidential, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is strictly prohibited. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format.