mod-security-users — General discussion about mod_security

Re: [mod-security-users] [Mod-security-developers] ModSecurity version 3.0.1 announcement

[Robert P. <rpa...@fe...>]

Hi Felipe,

On Thu, Apr 5, 2018 at 6:33 AM, Felipe Zimmerle <fe...@zi...> wrote:

> Hi,
>
> There are few things in the code that can be improved. There are even
> "TODO:" marking. But at certain point you may want to look at the rules.
> That is very important.
>

I'm not sure if I emphasized my last point well enough. Yes, the design of
the rules impacts performance (that's a given in a data-driven model), but
the engine itself suffers from some severe limitations at this point.

I created a very dumb, sizeable set of rules: https://gist.github.com
/p0pr0ck5/e0c73606f0be8ab93edb729e6cb56c5d

Using the simple harness I mentioned earlier (https://gist.github.com/p0pr0
ck5/9b2c414641c9b03d527679d0c8cb7d86), we see a sizeable decrease in
performance as the number of dumb rules increases. With 200 rules, a single
process can evaluate around 800 full transactions per second. When we
double the number of rules processed (remaining equally distributed
throughout phases 1-5), around 450 transactions are processed, and another
doubling in rule size (to 800 rules) results in a throughput of about 200
transactions/sec.

Of course, this simple example does a lot of unnecessary memory
transactions with ModSecurity objects; the Nginx connector is a bit more
performant. With 800 rules, a single worker process reports still only
triple-digit RPS:

$ wrk -t 5 -c 50 -d 5s http://localhost:8080/index.html
Running 5s test @ http://localhost:8080/index.html
  5 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   133.28ms  230.40ms   1.92s    92.92%
    Req/Sec   133.53     53.89   410.00     77.39%
  3085 requests in 5.03s, 2.50MB read
Requests/sec:    612.94
Transfer/sec:    508.79KB

Yes, this is better than the CRS performance, but it's still orders of
magnitude from what Nginx is capable of processing on its own. So we cannot
say that such poor performance with the CRS is related to the design of the
ruleset itself. There is a clear relationship between the *number* of rules
that lobmodsecurity has to process, and performance. This further confirms
my original assertions, that there are fundamental limitations in core
ModSecurity code that hinders performance. Furthermore there are no TODO
comments regarding the performance of Rule object evaluation:

poprocks@mini-vm:~/src/ModSecurity/src$ git grep TODO
parser/seclang-scanner.cc:/* TODO: this is always defined, so inline it */
parser/seclang-scanner.cc:            /** TODO: Implement the server
logging mechanism. */
parser/seclang-scanner.cc:      /* TODO. We should be able to replace this
entire function body
parser/seclang-scanner.ll:            /** TODO: Implement the server
logging mechanism. */
request_body_processor/json.cc:     * TODO: make UTF8 validation optional,
as it depends on Content-Encoding
transaction.cc:    /** TODO: Check variable */
transaction.cc:    /** TODO: Check variable */
transaction.cc:    /** TODO: Check variable */
transaction.cc:    /** TODO: Check variable */
transaction.cc:        /** TODO: write audit_log D part. */
transaction.cc:        /** TODO: write audit_log G part. */
transaction.cc:        /** TODO: write audit_log H part. */
transaction.cc:        /** TODO: write audit_log I part. */
transaction.cc:        /** TODO: write audit_log J part. */
transaction.cc:        /** TODO: write audit_log K part. */
utils/acmp.cc: * TODO: This code comes from ModSecurity 2.9.0 there are two
memory leaks here
utils/msc_tree.h: * TODO: This is an improved copy of the ModSecurity 2.9
file, this may need

There is a lot of needlessly repeated work done in Rule::evaluate.
Collection values, exemptions, transformations, etc., these can all be
cached based on some key relevant to the rule. I implemented a lot of these
design patterns for lua-resty-waf, and it's really helped with performance;
I'd love to share some detailed thoughts on this in a development
discussion setting. And, as flamegraphs show, there's a lot of std
container allocation/management that's done that I suspect could be
optimized away. But these are fundamental design changes that would have a
major impact, and would require careful study and planning. This is
something I doubt could be handled by a community contribution. Ultimately
we're talking about a refactor of some significant hot path code. And
Felipe, please understand that I and the community deeply respect the work
that's been done on this project, but frankly this doesn't seem to be a
priority for you, based on the responses in this thread. And if in your
view I'm way off base in my assumptions, I'd love to hear it, and review
some data and example execution that contradicts what I, Christian, Jai,
and Andrei have shown here, and if not, at least an acknowledgement from
either Trustwave or Nginx that the inclusion of Modsecurity into Nginx
results in a substantial, meaningful variance in throughput capacity.

Re: [mod-security-users] [Mod-security-developers] ModSecurity version 3.0.1 announcement

[Jai H. <jai...@mu...>]

Also have results for XML body which I added to attachment below.

On Thu, Apr 5, 2018 at 11:01 AM, Jai Harpalani <jai...@mu...>
wrote:

> Appears that the image I attached is not viewable. I've attached pdf,
> instead.
>
> On Thu, Apr 5, 2018 at 10:55 AM, Jai Harpalani <jai...@mu...
> > wrote:
>
>> I ran some tests with a) ModSecurity disabled, b) ModSecurity v3.0.0, and
>> c) ModSecurity v3.0.1.
>> I saw a speedup of about 3.5x with v3.0.1 vs v3.0.0.
>> Details are below.
>>
>>
>>
>>
>>
>> On Thu, Apr 5, 2018 at 8:33 AM, Felipe Zimmerle <fe...@zi...>
>> wrote:
>>
>>> Hi,
>>>
>>> On Wed, Apr 4, 2018 at 2:44 PM Robert Paprocki <
>>> rpa...@fe...> wrote:
>>>
>>>> Hi,
>>>>
>>>> On Wed, Apr 4, 2018 at 5:03 AM, Christian Folini <
>>>> chr...@ne...> wrote:
>>>>>
>>>>>
>>>>> > However, I have limited knowledge on the following:
>>>>> >  - is ModSec 3.x has been ever targeted to support CRS < 3,
>>>>>
>>>>> See above.
>>>>>
>>>>
>>>> It would be great to here an official stance from the development team
>>>> on this. @Felipe can you comment?
>>>>
>>>>
>>> Feature wise it should support both. ModSecurity was not written to
>>> support OWASP CRS, but the SecRule Language.
>>> Consequently it supports OWASP CRS. There is a milestone for that here:
>>> https://github.com/SpiderLabs/ModSecurity/milestone/11
>>>
>>>
>>>>
>>>>> > Also I'm not sure whether ModSec 2.x has its own benchmarks (not
>>>>> related to any connector).
>>>>> > If it does, then perhaps it would be good to compare "generic"
>>>>> ModSec 2.x
>>>>> > vs "generic" ModSec 3.x as well.
>>>>>
>>>>> Yes, that would be cool. But from what I understand, ModSec 2.9.x is
>>>>> deeply
>>>>> integrated into the webserver.
>>>>>
>>>>
>>>>
>>>>
>>> I don't think it is possible to split v2 from Apache.
>>>
>>>
>>>> Yeah, this is a bit of a pain to test. I modified one of the example
>>>> programs that comes with the v3/master ModSecurity source code as follows:
>>>>
>>>> https://gist.github.com/p0pr0ck5/9b2c414641c9b03d527679d0c8cb7d86
>>>>
>>>>
>>> There is a benchmark utility in the repository:
>>> https://github.com/SpiderLabs/ModSecurity/tree/v3/master/test/benchmark
>>>
>>>
>>>> Note that this doesn't add any headers or body data (request or
>>>> response) to the transaction, and the included "basic_rules.conf" is
>>>> unchanged from what's in the example repo (and ignore the memory leak in
>>>> not cleaning up the transaction). So running this very light example:
>>>>
>>>> $ ./test
>>>> Rules:
>>>> Phase: 0 (0 rules)
>>>> Phase: 1 (0 rules)
>>>> Phase: 2 (2 rules)
>>>>     Rule ID: 200000--0x1e2b6c0
>>>>     Rule ID: 200001--0x1e2bc90
>>>> Phase: 3 (4 rules)
>>>>     Rule ID: 200002--0x1e2c720
>>>>     Rule ID: 200003--0x1e2ea00
>>>>     Rule ID: 200004--0x1e2f200
>>>>     Rule ID: 200005--0x1e2fc10
>>>> Phase: 4 (0 rules)
>>>> Phase: 5 (0 rules)
>>>> Phase: 6 (0 rules)
>>>> Phase: 7 (0 rules)
>>>> Did 9907
>>>> Done!
>>>>
>>>> We see around 10k processes per second. Essentially all of our time is
>>>> spent waiting on memory allocations: https://s3.amazon
>>>> aws.com/p0pr0ck5-data/modsec-simple.svg
>>>>
>>>> Now consider the case where we include the full 3.0.0 CRS:
>>>>
>>>> $ ./test
>>>> Rules:
>>>> Phase: 0 (32 rules)
>>>>     Rule ID: 0--0x22ca500
>>>>     Rule ID: 0--0x22d9480
>>>>     Rule ID: 0--0x22f9530
>>>>     Rule ID: 0--0x22f9f20
>>>> [...snip several hundred lines...]
>>>> Did 319
>>>> Done!
>>>>
>>>> So this is about the same throughput we saw from Nginx + libmodsec
>>>> integration. A flamegraph also highlights hot paths, particularly in
>>>> Rule::evaluate (also largely spent on allocation wait time at this
>>>> point, since each evaluation instantiates many new objects):
>>>> https://s3.amazonaws.com/p0pr0ck5-data/modsec-simple-crs.svg
>>>>
>>>> (I also performed the same tests on 3.0.0; flamegraphs are also in this
>>>> s3 bucket but I will avoid commentary here for now as this thread is gotten
>>>> long in the tooth as it is).
>>>>
>>>> A few initial takeaways:
>>>>
>>>> - There is a clearly definable minimum overhead needed to execute
>>>> libmodsecurity, based on its current architecture
>>>> - I noted the same memory leak that defanator noted in #1729. I have
>>>> yet to apply the patches noted.
>>>> - HTTP integrations will induce some overhead, which may or may not be
>>>> substantial. With smart memory pooling and good design, though, this should
>>>> be kept to a minimum.
>>>> - Leveraging the full CRS completely tanks throughput. It would take a
>>>> while more to dig into specifics but I suspect that Rule::evaluate
>>>> probably needs an overhaul.
>>>> - I ran one more test again a mock set of dumb rules (generated via
>>>> https://gist.github.com/p0pr0ck5/c99bca54734af7546d910db8d7c97ab3).
>>>> Saw about 1000 transactions processed per second, indicating (as should be
>>>> assumed) that linear growth in the size of the included ruleset results in
>>>> similar performance reduction. Flamegraphs again point to
>>>> Rule::evaluate, with Rule::getFinalVars taking up half those traces. I
>>>> suspect there could be some smarter behaviors about avoiding object
>>>> creation if its unneeded on these hot paths.
>>>>
>>>>
>>> There are few things in the code that can be improved. There are even
>>> "TODO:" marking. But at certain point you may want to look at the rules.
>>> That is very important.
>>>
>>> Hopefully this is of use to some folks. I'd be interested in writing
>>>> some automated scaffolding to execute this type of testing against every
>>>> commit, to maintain a continuous audit trail of performance impacts; if
>>>> this sounds interesting to the maintainers I'd be happy to chat further.
>>>>
>>>
>>> Andrei does that already. Not automagically, but frequently updated.
>>> Some of the results are posted here:
>>> https://github.com/defanator/modsecurity-performance/wiki
>>>
>>>
>>>> ------------------------------------------------------------
>>>> ------------------
>>>> Check out the vibrant tech community on one of the world's most
>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot______
>>>> _________________________________________
>>>> mod-security-users mailing list
>>>> mod...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>>> http://www.modsecurity.org/projects/commercial/rules/
>>>> http://www.modsecurity.org/projects/commercial/support/
>>>>
>>>
>>> ------------------------------------------------------------
>>> ------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>> _______________________________________________
>>> mod-security-users mailing list
>>> mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> http://www.modsecurity.org/projects/commercial/rules/
>>> http://www.modsecurity.org/projects/commercial/support/
>>>
>>>
>>
>

Re: [mod-security-users] [Mod-security-developers] ModSecurity version 3.0.1 announcement

[Jai H. <jai...@mu...>]

Appears that the image I attached is not viewable. I've attached pdf,
instead.

On Thu, Apr 5, 2018 at 10:55 AM, Jai Harpalani <jai...@mu...>
wrote:

> I ran some tests with a) ModSecurity disabled, b) ModSecurity v3.0.0, and
> c) ModSecurity v3.0.1.
> I saw a speedup of about 3.5x with v3.0.1 vs v3.0.0.
> Details are below.
>
>
>
>
>
> On Thu, Apr 5, 2018 at 8:33 AM, Felipe Zimmerle <fe...@zi...>
> wrote:
>
>> Hi,
>>
>> On Wed, Apr 4, 2018 at 2:44 PM Robert Paprocki <
>> rpa...@fe...> wrote:
>>
>>> Hi,
>>>
>>> On Wed, Apr 4, 2018 at 5:03 AM, Christian Folini <
>>> chr...@ne...> wrote:
>>>>
>>>>
>>>> > However, I have limited knowledge on the following:
>>>> >  - is ModSec 3.x has been ever targeted to support CRS < 3,
>>>>
>>>> See above.
>>>>
>>>
>>> It would be great to here an official stance from the development team
>>> on this. @Felipe can you comment?
>>>
>>>
>> Feature wise it should support both. ModSecurity was not written to
>> support OWASP CRS, but the SecRule Language.
>> Consequently it supports OWASP CRS. There is a milestone for that here:
>> https://github.com/SpiderLabs/ModSecurity/milestone/11
>>
>>
>>>
>>>> > Also I'm not sure whether ModSec 2.x has its own benchmarks (not
>>>> related to any connector).
>>>> > If it does, then perhaps it would be good to compare "generic" ModSec
>>>> 2.x
>>>> > vs "generic" ModSec 3.x as well.
>>>>
>>>> Yes, that would be cool. But from what I understand, ModSec 2.9.x is
>>>> deeply
>>>> integrated into the webserver.
>>>>
>>>
>>>
>>>
>> I don't think it is possible to split v2 from Apache.
>>
>>
>>> Yeah, this is a bit of a pain to test. I modified one of the example
>>> programs that comes with the v3/master ModSecurity source code as follows:
>>>
>>> https://gist.github.com/p0pr0ck5/9b2c414641c9b03d527679d0c8cb7d86
>>>
>>>
>> There is a benchmark utility in the repository:
>> https://github.com/SpiderLabs/ModSecurity/tree/v3/master/test/benchmark
>>
>>
>>> Note that this doesn't add any headers or body data (request or
>>> response) to the transaction, and the included "basic_rules.conf" is
>>> unchanged from what's in the example repo (and ignore the memory leak in
>>> not cleaning up the transaction). So running this very light example:
>>>
>>> $ ./test
>>> Rules:
>>> Phase: 0 (0 rules)
>>> Phase: 1 (0 rules)
>>> Phase: 2 (2 rules)
>>>     Rule ID: 200000--0x1e2b6c0
>>>     Rule ID: 200001--0x1e2bc90
>>> Phase: 3 (4 rules)
>>>     Rule ID: 200002--0x1e2c720
>>>     Rule ID: 200003--0x1e2ea00
>>>     Rule ID: 200004--0x1e2f200
>>>     Rule ID: 200005--0x1e2fc10
>>> Phase: 4 (0 rules)
>>> Phase: 5 (0 rules)
>>> Phase: 6 (0 rules)
>>> Phase: 7 (0 rules)
>>> Did 9907
>>> Done!
>>>
>>> We see around 10k processes per second. Essentially all of our time is
>>> spent waiting on memory allocations: https://s3.amazon
>>> aws.com/p0pr0ck5-data/modsec-simple.svg
>>>
>>> Now consider the case where we include the full 3.0.0 CRS:
>>>
>>> $ ./test
>>> Rules:
>>> Phase: 0 (32 rules)
>>>     Rule ID: 0--0x22ca500
>>>     Rule ID: 0--0x22d9480
>>>     Rule ID: 0--0x22f9530
>>>     Rule ID: 0--0x22f9f20
>>> [...snip several hundred lines...]
>>> Did 319
>>> Done!
>>>
>>> So this is about the same throughput we saw from Nginx + libmodsec
>>> integration. A flamegraph also highlights hot paths, particularly in
>>> Rule::evaluate (also largely spent on allocation wait time at this
>>> point, since each evaluation instantiates many new objects):
>>> https://s3.amazonaws.com/p0pr0ck5-data/modsec-simple-crs.svg
>>>
>>> (I also performed the same tests on 3.0.0; flamegraphs are also in this
>>> s3 bucket but I will avoid commentary here for now as this thread is gotten
>>> long in the tooth as it is).
>>>
>>> A few initial takeaways:
>>>
>>> - There is a clearly definable minimum overhead needed to execute
>>> libmodsecurity, based on its current architecture
>>> - I noted the same memory leak that defanator noted in #1729. I have yet
>>> to apply the patches noted.
>>> - HTTP integrations will induce some overhead, which may or may not be
>>> substantial. With smart memory pooling and good design, though, this should
>>> be kept to a minimum.
>>> - Leveraging the full CRS completely tanks throughput. It would take a
>>> while more to dig into specifics but I suspect that Rule::evaluate
>>> probably needs an overhaul.
>>> - I ran one more test again a mock set of dumb rules (generated via
>>> https://gist.github.com/p0pr0ck5/c99bca54734af7546d910db8d7c97ab3). Saw
>>> about 1000 transactions processed per second, indicating (as should be
>>> assumed) that linear growth in the size of the included ruleset results in
>>> similar performance reduction. Flamegraphs again point to Rule::evaluate,
>>> with Rule::getFinalVars taking up half those traces. I suspect there
>>> could be some smarter behaviors about avoiding object creation if its
>>> unneeded on these hot paths.
>>>
>>>
>> There are few things in the code that can be improved. There are even
>> "TODO:" marking. But at certain point you may want to look at the rules.
>> That is very important.
>>
>> Hopefully this is of use to some folks. I'd be interested in writing some
>>> automated scaffolding to execute this type of testing against every commit,
>>> to maintain a continuous audit trail of performance impacts; if this sounds
>>> interesting to the maintainers I'd be happy to chat further.
>>>
>>
>> Andrei does that already. Not automagically, but frequently updated. Some
>> of the results are posted here:
>> https://github.com/defanator/modsecurity-performance/wiki
>>
>>
>>> ------------------------------------------------------------
>>> ------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot______
>>> _________________________________________
>>> mod-security-users mailing list
>>> mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> http://www.modsecurity.org/projects/commercial/rules/
>>> http://www.modsecurity.org/projects/commercial/support/
>>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
>>
>>
>

Re: [mod-security-users] [Mod-security-developers] ModSecurity version 3.0.1 announcement

[Jai H. <jai...@mu...>]

I ran some tests with a) ModSecurity disabled, b) ModSecurity v3.0.0, and
c) ModSecurity v3.0.1.
I saw a speedup of about 3.5x with v3.0.1 vs v3.0.0.
Details are below.





On Thu, Apr 5, 2018 at 8:33 AM, Felipe Zimmerle <fe...@zi...> wrote:

> Hi,
>
> On Wed, Apr 4, 2018 at 2:44 PM Robert Paprocki <rpaprocki@
> fearnothingproductions.net> wrote:
>
>> Hi,
>>
>> On Wed, Apr 4, 2018 at 5:03 AM, Christian Folini <
>> chr...@ne...> wrote:
>>>
>>>
>>> > However, I have limited knowledge on the following:
>>> >  - is ModSec 3.x has been ever targeted to support CRS < 3,
>>>
>>> See above.
>>>
>>
>> It would be great to here an official stance from the development team on
>> this. @Felipe can you comment?
>>
>>
> Feature wise it should support both. ModSecurity was not written to
> support OWASP CRS, but the SecRule Language.
> Consequently it supports OWASP CRS. There is a milestone for that here:
> https://github.com/SpiderLabs/ModSecurity/milestone/11
>
>
>>
>>> > Also I'm not sure whether ModSec 2.x has its own benchmarks (not
>>> related to any connector).
>>> > If it does, then perhaps it would be good to compare "generic" ModSec
>>> 2.x
>>> > vs "generic" ModSec 3.x as well.
>>>
>>> Yes, that would be cool. But from what I understand, ModSec 2.9.x is
>>> deeply
>>> integrated into the webserver.
>>>
>>
>>
>>
> I don't think it is possible to split v2 from Apache.
>
>
>> Yeah, this is a bit of a pain to test. I modified one of the example
>> programs that comes with the v3/master ModSecurity source code as follows:
>>
>> https://gist.github.com/p0pr0ck5/9b2c414641c9b03d527679d0c8cb7d86
>>
>>
> There is a benchmark utility in the repository:
> https://github.com/SpiderLabs/ModSecurity/tree/v3/master/test/benchmark
>
>
>> Note that this doesn't add any headers or body data (request or response)
>> to the transaction, and the included "basic_rules.conf" is unchanged from
>> what's in the example repo (and ignore the memory leak in not cleaning up
>> the transaction). So running this very light example:
>>
>> $ ./test
>> Rules:
>> Phase: 0 (0 rules)
>> Phase: 1 (0 rules)
>> Phase: 2 (2 rules)
>>     Rule ID: 200000--0x1e2b6c0
>>     Rule ID: 200001--0x1e2bc90
>> Phase: 3 (4 rules)
>>     Rule ID: 200002--0x1e2c720
>>     Rule ID: 200003--0x1e2ea00
>>     Rule ID: 200004--0x1e2f200
>>     Rule ID: 200005--0x1e2fc10
>> Phase: 4 (0 rules)
>> Phase: 5 (0 rules)
>> Phase: 6 (0 rules)
>> Phase: 7 (0 rules)
>> Did 9907
>> Done!
>>
>> We see around 10k processes per second. Essentially all of our time is
>> spent waiting on memory allocations: https://s3.
>> amazonaws.com/p0pr0ck5-data/modsec-simple.svg
>>
>> Now consider the case where we include the full 3.0.0 CRS:
>>
>> $ ./test
>> Rules:
>> Phase: 0 (32 rules)
>>     Rule ID: 0--0x22ca500
>>     Rule ID: 0--0x22d9480
>>     Rule ID: 0--0x22f9530
>>     Rule ID: 0--0x22f9f20
>> [...snip several hundred lines...]
>> Did 319
>> Done!
>>
>> So this is about the same throughput we saw from Nginx + libmodsec
>> integration. A flamegraph also highlights hot paths, particularly in
>> Rule::evaluate (also largely spent on allocation wait time at this
>> point, since each evaluation instantiates many new objects): https://s3.
>> amazonaws.com/p0pr0ck5-data/modsec-simple-crs.svg
>>
>> (I also performed the same tests on 3.0.0; flamegraphs are also in this
>> s3 bucket but I will avoid commentary here for now as this thread is gotten
>> long in the tooth as it is).
>>
>> A few initial takeaways:
>>
>> - There is a clearly definable minimum overhead needed to execute
>> libmodsecurity, based on its current architecture
>> - I noted the same memory leak that defanator noted in #1729. I have yet
>> to apply the patches noted.
>> - HTTP integrations will induce some overhead, which may or may not be
>> substantial. With smart memory pooling and good design, though, this should
>> be kept to a minimum.
>> - Leveraging the full CRS completely tanks throughput. It would take a
>> while more to dig into specifics but I suspect that Rule::evaluate
>> probably needs an overhaul.
>> - I ran one more test again a mock set of dumb rules (generated via
>> https://gist.github.com/p0pr0ck5/c99bca54734af7546d910db8d7c97ab3). Saw
>> about 1000 transactions processed per second, indicating (as should be
>> assumed) that linear growth in the size of the included ruleset results in
>> similar performance reduction. Flamegraphs again point to Rule::evaluate,
>> with Rule::getFinalVars taking up half those traces. I suspect there
>> could be some smarter behaviors about avoiding object creation if its
>> unneeded on these hot paths.
>>
>>
> There are few things in the code that can be improved. There are even
> "TODO:" marking. But at certain point you may want to look at the rules.
> That is very important.
>
> Hopefully this is of use to some folks. I'd be interested in writing some
>> automated scaffolding to execute this type of testing against every commit,
>> to maintain a continuous audit trail of performance impacts; if this sounds
>> interesting to the maintainers I'd be happy to chat further.
>>
>
> Andrei does that already. Not automagically, but frequently updated. Some
> of the results are posted here:
> https://github.com/defanator/modsecurity-performance/wiki
>
>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot______
>> _________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
>>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>
>

[mod-security-users] ruleRemoveTargetByTag - am I misunderstanding how it works?

[Eirik Ø. - M. <ltn...@an...>]

Hi all,

I'm new at modsec rules, but I'm fairly certain I've tried all I can on this one. Upgraded to 3.0.2 to get the various fixes there, and have created the rule chain below. The intention is to ignore all instances of the parameter TermURL (whether GET or POST at the moment) for all other rules with the attack-slqi/rce/xss tag set.

The below is listed above all the CRS rules in the configuration.
I suspect the tags should be quoted, as I see tags are quoted in other examples - but that gives an error.

SecRule REQUEST_URI "@beginsWith /mdpayacs/pareq" \
    "phase:1,id:1003,t:none,pass,nolog,chain,\
        ctl:ruleRemoveTargetByTag=attack-sqli;ARGS:TermURL,\
        ctl:ruleRemoveTargetByTag=attack-rce;ARGS:TermURL,\
        ctl:ruleRemoveTargetByTag=attack-xss;ARGS:TermURL"
            SecRule ARGS:TermURL "@beginsWith http" "t:none"

Can anyone help me determine what is wrong here? I'm still being flooded with notifications despite the above..

Wbr
/Eirik

Re: [mod-security-users] [Mod-security-developers] ModSecurity version 3.0.1 announcement

[Felipe Z. <fe...@zi...>]

Hi,

On Wed, Apr 4, 2018 at 2:44 PM Robert Paprocki <
rpa...@fe...> wrote:

> Hi,
>
> On Wed, Apr 4, 2018 at 5:03 AM, Christian Folini <
> chr...@ne...> wrote:
>>
>>
>> > However, I have limited knowledge on the following:
>> >  - is ModSec 3.x has been ever targeted to support CRS < 3,
>>
>> See above.
>>
>
> It would be great to here an official stance from the development team on
> this. @Felipe can you comment?
>
>
Feature wise it should support both. ModSecurity was not written to support
OWASP CRS, but the SecRule Language.
Consequently it supports OWASP CRS. There is a milestone for that here:
https://github.com/SpiderLabs/ModSecurity/milestone/11


>
>> > Also I'm not sure whether ModSec 2.x has its own benchmarks (not
>> related to any connector).
>> > If it does, then perhaps it would be good to compare "generic" ModSec
>> 2.x
>> > vs "generic" ModSec 3.x as well.
>>
>> Yes, that would be cool. But from what I understand, ModSec 2.9.x is
>> deeply
>> integrated into the webserver.
>>
>
>
>
I don't think it is possible to split v2 from Apache.


> Yeah, this is a bit of a pain to test. I modified one of the example
> programs that comes with the v3/master ModSecurity source code as follows:
>
> https://gist.github.com/p0pr0ck5/9b2c414641c9b03d527679d0c8cb7d86
>
>
There is a benchmark utility in the repository:
https://github.com/SpiderLabs/ModSecurity/tree/v3/master/test/benchmark


> Note that this doesn't add any headers or body data (request or response)
> to the transaction, and the included "basic_rules.conf" is unchanged from
> what's in the example repo (and ignore the memory leak in not cleaning up
> the transaction). So running this very light example:
>
> $ ./test
> Rules:
> Phase: 0 (0 rules)
> Phase: 1 (0 rules)
> Phase: 2 (2 rules)
>     Rule ID: 200000--0x1e2b6c0
>     Rule ID: 200001--0x1e2bc90
> Phase: 3 (4 rules)
>     Rule ID: 200002--0x1e2c720
>     Rule ID: 200003--0x1e2ea00
>     Rule ID: 200004--0x1e2f200
>     Rule ID: 200005--0x1e2fc10
> Phase: 4 (0 rules)
> Phase: 5 (0 rules)
> Phase: 6 (0 rules)
> Phase: 7 (0 rules)
> Did 9907
> Done!
>
> We see around 10k processes per second. Essentially all of our time is
> spent waiting on memory allocations:
> https://s3.amazonaws.com/p0pr0ck5-data/modsec-simple.svg
>
> Now consider the case where we include the full 3.0.0 CRS:
>
> $ ./test
> Rules:
> Phase: 0 (32 rules)
>     Rule ID: 0--0x22ca500
>     Rule ID: 0--0x22d9480
>     Rule ID: 0--0x22f9530
>     Rule ID: 0--0x22f9f20
> [...snip several hundred lines...]
> Did 319
> Done!
>
> So this is about the same throughput we saw from Nginx + libmodsec
> integration. A flamegraph also highlights hot paths, particularly in
> Rule::evaluate (also largely spent on allocation wait time at this point,
> since each evaluation instantiates many new objects):
> https://s3.amazonaws.com/p0pr0ck5-data/modsec-simple-crs.svg
>
> (I also performed the same tests on 3.0.0; flamegraphs are also in this s3
> bucket but I will avoid commentary here for now as this thread is gotten
> long in the tooth as it is).
>
> A few initial takeaways:
>
> - There is a clearly definable minimum overhead needed to execute
> libmodsecurity, based on its current architecture
> - I noted the same memory leak that defanator noted in #1729. I have yet
> to apply the patches noted.
> - HTTP integrations will induce some overhead, which may or may not be
> substantial. With smart memory pooling and good design, though, this should
> be kept to a minimum.
> - Leveraging the full CRS completely tanks throughput. It would take a
> while more to dig into specifics but I suspect that Rule::evaluate
> probably needs an overhaul.
> - I ran one more test again a mock set of dumb rules (generated via
> https://gist.github.com/p0pr0ck5/c99bca54734af7546d910db8d7c97ab3). Saw
> about 1000 transactions processed per second, indicating (as should be
> assumed) that linear growth in the size of the included ruleset results in
> similar performance reduction. Flamegraphs again point to Rule::evaluate,
> with Rule::getFinalVars taking up half those traces. I suspect there
> could be some smarter behaviors about avoiding object creation if its
> unneeded on these hot paths.
>
>
There are few things in the code that can be improved. There are even
"TODO:" marking. But at certain point you may want to look at the rules.
That is very important.

Hopefully this is of use to some folks. I'd be interested in writing some
> automated scaffolding to execute this type of testing against every commit,
> to maintain a continuous audit trail of performance impacts; if this sounds
> interesting to the maintainers I'd be happy to chat further.
>

Andrei does that already. Not automagically, but frequently updated. Some
of the results are posted here:
https://github.com/defanator/modsecurity-performance/wiki


>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>

Re: [mod-security-users] [Mod-security-developers] ModSecurity version 3.0.1 announcement

[Robert P. <rpa...@fe...>]

Hi,

On Wed, Apr 4, 2018 at 1:20 PM, Felipe Costa <FC...@tr...> wrote:
>
> I like the idea of having multiple rules set and versions. My rationale is
> that
> ModSecurity was not designed to run a single rule set, nor a rule set
> version.
> It may be optimized for a specific rule set, but still, what are the
> consequences of
> the optimizations for the others.... At this point it makes sense to focus
> only the public
> rules set, but we might need to take into consideration other rule sets as
> well.
>

I wholeheartedly agree. This is one of the largest drawbacks of
ModSecurity's design, IMO. Trying to separate *engine* performance from
*rule* performance was a big focus while developing lua-resty-waf, and we
still don't have it down right. Trying to write a highly optimized engine
for an arbitrary DSL is a tall order.

At this point, I don't think that there needs to be specific
optimization/tuning done that is geared specifically toward any particular
ruleset. Indeed, "ruleset" is a nebulous topic in its own right. I think
we've clearly shown in this thread that there are hot paths within the
engine itself that can use improvement. I do not mean to criticize any of
the development team by this; I simply wish to highlight the nature of what
we've found through some basic benchmarking and profiling. I suspect
further investigative efforts will shed more light on areas where both
community-back rulesets, and the ModSecurity rule engine, can be improved.

Re: [mod-security-users] [Mod-security-developers] ModSecurity version 3.0.1 announcement

[Felipe C. <FC...@tr...>]

Hi,
 
I really like to see those performance experiments going on. There are too many
perspectives in this subject.
 
I like the idea of having multiple rules set and versions. My rationale is that
ModSecurity was not designed to run a single rule set, nor a rule set version.
It may be optimized for a specific rule set, but still, what are the consequences of
the optimizations for the others.... At this point it makes sense to focus only the public
rules set, but we might need to take into consideration other rule sets as well.

Felipe “Zimmerle” Costa
Security Researcher, Lead Developer ModSecurity.
 
Trustwave | SMART SECURITY ON DEMAND
www.trustwave.com <http://www.trustwave.com/>
 
On 4/4/18, 8:15 AM, "Andrei Belov" <de...@ng...> wrote:

    
    > On 04 Apr 2018, at 11:58, Christian Folini <chr...@ne...> wrote:
    > 
    > Hello Andrei,
    > 
    > On Wed, Apr 04, 2018 at 11:29:18AM +0300, Andrei Belov wrote:
    >> I think that environment could be [relatively easily] extended to support
    >> Apache + ModSec 2.x, in addition to nginx + ModSec 3.x, in order to simplify
    >> "direct" comparison and provide reproducible, statistically significant results.
    > 
    > Very cool. Thank you for sharing - and thanks for your contributions to
    > ModSecurity, namely 3.0.1.
    > 
    > The conceptual problem is see is that it's more than one variable here.
    > Apache/ModSec2 vs. NGINX/ModSec3. I'm an Apache person, but when I stripped
    > the two of Modsec and let the bare minimum installations serve static
    > files, NGINX blew me away.
    > 
    > So I kind of think that one would have to slow down NGINX to reach an Apache
    > level and then in a 2nd step add ModSec again to be able to measure ModSec2 vs
    > ModSec3.
    > 
    > What is your take on this?
    
    Well, ideally it would be awesome to have the following combos in [perf] tests:
    
     a) Apache + ModSec 2.x + CRS 2.x
     b) Apache + ModSec 3.x + CRS 3.x
     c) nginx + ModSec 2.x + CRS 2.x
     d) nginx + ModSec 3.x + CRS 3.x
    
    (obviously, CRS component could be optional when one is going to measure
    "generic overhead")
    
    However, I have limited knowledge on the following:
     - is ModSec 3.x has been ever targeted to support CRS < 3,
     - is there a working Apache connector for ModSec 3.x.
    
    Also I'm not sure whether ModSec 2.x has its own benchmarks (not related to any connector).
    If it does, then perhaps it would be good to compare "generic" ModSec 2.x
    vs "generic" ModSec 3.x as well.
    
    BTW, for those who are familiar with tools like gdb / perf / systemtap etc,
    there's the "debugenv" state in vagrant env:
    
        https://scanmail.trustwave.com/?c=4062&d=37PE2hgmrBoCfPZ1xNp9TVfwnlZlypKJZfBOSVkaZQ&s=5&u=https%3a%2f%2fgithub%2ecom%2fdefanator%2fmodsecurity-performance%2fblob%2fmaster%2fstates%2fdebugenv%2esls
    
    It could be useful for some deeper investigations.
    
    
    ------------------------------------------------------------------------------
    Check out the vibrant tech community on one of the world's most
    engaging tech sites, http://scanmail.trustwave.com/?c=4062&d=37PE2hgmrBoCfPZ1xNp9TVfwnlZlypKJZaVOHg4cZQ&s=5&u=http%3a%2f%2fSlashdot%2eorg%21 http://scanmail.trustwave.com/?c=4062&d=37PE2hgmrBoCfPZ1xNp9TVfwnlZlypKJZfMdGAoaYQ&s=5&u=http%3a%2f%2fsdm%2elink%2fslashdot
    _______________________________________________
    mod-security-users mailing list
    mod...@li...
    https://scanmail.trustwave.com/?c=4062&d=37PE2hgmrBoCfPZ1xNp9TVfwnlZlypKJZfAaH1kcYA&s=5&u=https%3a%2f%2flists%2esourceforge%2enet%2flists%2flistinfo%2fmod-security-users
    Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
    http://scanmail.trustwave.com/?c=4062&d=37PE2hgmrBoCfPZ1xNp9TVfwnlZlypKJZaJISwBIZA&s=5&u=http%3a%2f%2fwww%2emodsecurity%2eorg%2fprojects%2fcommercial%2frules%2f
    http://scanmail.trustwave.com/?c=4062&d=37PE2hgmrBoCfPZ1xNp9TVfwnlZlypKJZaNMGQwcMQ&s=5&u=http%3a%2f%2fwww%2emodsecurity%2eorg%2fprojects%2fcommercial%2fsupport%2f
    

Re: [mod-security-users] Sanitize JSON Request / Response

[Robert P. <rpa...@fe...>]

Hi,

On Wed, Apr 4, 2018 at 3:48 AM, Osama Elnaggar <oel...@gm...>
wrote:

> Hi Robert,
>
> Sorry for the late reply.  I was a little busy the past 2 weeks.
>
> I just retested it (on 2.9.2 + manually patching the two files mentioned
> here - https://github.com/SpiderLabs/ModSecurity/pull/1714/files)  It
> looks like:
>
> - it works if the output format (SecAuditLogFormat) is JSON
> - it doesn't work if the output is Native
>
> For example, with the following rule:
>
> SecAction "phase:5,id:22,nolog,pass,sanitiseArg:cvv"
>
> and the following request:
>
> curl -H "Content-Type: application/json" -X POST -d '{"cvv":"123"}'
> http://localhost/?id=/bin/bash
>
> and SecAuditLogFormat JSON, I get the following:
>
> .. "body":["{\"cvv\":\"***\"}"]} ..
>
> while with normal native logging, I get the following:
>
> --c2397953-C--
> {"cvv":"123"}
>
> In my previous email below, I was looking at native logging so it appeared
> that it wasn’t sanitizing the output.
>
> Thanks.
>


Are you sure you applied the patch and rebuilt correctly? I have modsec
built against commit 8d4124eee26cc018f6ed306e0d404737ce82c849 and loaded
into Apache 2.4. JSON body sanitization does indeed work for me with native
audit logging with this:


root@mini-vm:~# curl -H "Content-Type: application/json" -X POST -d
'{"cvv":"123"}' http://localhost/?id=/bin/bash
root@mini-vm:~# cat /var/log/apache2/modsec_audit.log
--0c9caa71-A--
[04/Apr/2018:11:46:49 --0700] WsUdmX8AAQEAAbGcdh4AAAAA 127.0.0.1 39072
127.0.0.1 80
--0c9caa71-B--
POST /?id=/bin/bash HTTP/1.1
Host: localhost
User-Agent: curl/7.47.0
Accept: */*
Content-Type: application/json
Content-Length: 13

--0c9caa71-C--
{"cvv":"***"}
--0c9caa71-F--
HTTP/1.1 200 OK
Last-Modified: Mon, 12 Mar 2018 20:28:52 GMT
ETag: "2c39-5673cfd7c5884"
Accept-Ranges: bytes
Content-Length: 11321
Vary: Accept-Encoding
Content-Type: text/html

--0c9caa71-H--
Apache-Error: [file "mod_authz_core.c"] [line 809] [level 7] AH01626:
authorization result of Require all granted: granted
Apache-Error: [file "mod_authz_core.c"] [line 809] [level 7] AH01626:
authorization result of <RequireAny>: granted
Stopwatch: 1522867609080274 935 (- - -)
Stopwatch2: 1522867609080274 935; combined=202, p1=93, p2=69, p3=3, p4=3,
p5=34, sr=0, sw=0, l=0, gc=0
Producer: ModSecurity for Apache/2.9.2 (http://www.modsecurity.org/).
Server: Apache/2.4.18 (Ubuntu)
Sanitised-Args: "cvv".
Engine-Mode: "ENABLED"

--0c9caa71-Z--

Re: [mod-security-users] [Mod-security-developers] ModSecurity version 3.0.1 announcement

[Andrei B. <de...@ng...>]

> On 04 Apr 2018, at 20:44, Robert Paprocki <rpa...@fe...> wrote:
> 

[..]

> A few initial takeaways:
> 
> - There is a clearly definable minimum overhead needed to execute libmodsecurity, based on its current architecture
> - I noted the same memory leak that defanator noted in #1729. I have yet to apply the patches noted.

Let me know if those work for you.


> - HTTP integrations will induce some overhead, which may or may not be substantial. With smart memory pooling and good design, though, this should be kept to a minimum.
> - Leveraging the full CRS completely tanks throughput. It would take a while more to dig into specifics but I suspect that Rule::evaluate probably needs an overhaul.
> - I ran one more test again a mock set of dumb rules (generated via https://gist.github.com/p0pr0ck5/c99bca54734af7546d910db8d7c97ab3 <https://gist.github.com/p0pr0ck5/c99bca54734af7546d910db8d7c97ab3>). Saw about 1000 transactions processed per second, indicating (as should be assumed) that linear growth in the size of the included ruleset results in similar performance reduction. Flamegraphs again point to Rule::evaluate, with Rule::getFinalVars taking up half those traces. I suspect there could be some smarter behaviors about avoiding object creation if its unneeded on these hot paths.

I observed the same peak places in my perf experiments earlier.


> Hopefully this is of use to some folks. I'd be interested in writing some automated scaffolding to execute this type of testing against every commit, to maintain a continuous audit trail of performance impacts; if this sounds interesting to the maintainers I'd be happy to chat further.

We used to maintain automated benchmark system powered by codespeed [1] for such kind of task.

The main issue here is that you have to rely on underlying hardware and OS in order to keep benchmark results in consistent state. If anything is changed (e.g. server upgrade / OS upgrade / kernel upgrade), it's worth to re-run the entire series of benchmark subsets, which would (and probably will) take a lot of time.

We've been running those on bare-metal servers, and there still were some issues with inconsistency growing over time.


[1] https://github.com/tobami/codespeed

Re: [mod-security-users] REQUEST_COOKIES_NAMES:/regex/ does not work

[Eric W. <mod...@li...>]

On Mon, 2 Apr 2018, Robert Paprocki wrote:

> Right, so the reason SecRuleUpdateTargetById wasn't working was because it was listed in the config file before
> my rule definition. It's gotta live after.(ref: https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/v3.0/master/rules/REQUEST-900-EXCLUSION-RULES-BEF
> ORE-CRS.conf.example#L43)
> So for example, the following works:
> 
> SecRuleUpdateTargetById 1234123413 "!REQUEST_HEADERS:/Coo.*/"
> 
> Because the Cookie request header matches that regex. Of course, as mentioned above you cannot whitelist specific
> cookie in this manner, it would need to be done like so:
> 
> SecRule REQUEST_HEADERS|XML:/*|!REQUEST_HEADERS:Referer|!REQUEST_HEADERS:Cookie "\b(\d+) ?=
> ?\1\b|[\'\"](\w+)[\'\"] ?= ?[\'\"]\2\b" \
>        "phase:2,capture,t:none,t:urlDecodeUni,t:htmlEntityDecode,t:replaceComments,t:compressWhiteSpace,t:lowercase,ctl
> :auditLogParts=+E,log,auditlog,msg:'SQL Injection
> Attack',id:1234123413,tag:'WEB_ATTACK/SQL_INJECTION',logdata:'%{TX.0}',severity:'2',deny"
> 
> SecRule "REQUEST_COOKIES|!REQUEST_COOKIES:/_gac_UA[\d-]+$/" "\b(\d+) ?= ?\1\b|[\'\"](\w+)[\'\"] ?= ?[\'\"]\2\b" \
>        "phase:2,capture,t:none,t:urlDecodeUni,t:htmlEntityDecode,t:replaceComments,t:compressWhiteSpace,t:lowercase,ctl
> :auditLogParts=+E,log,auditlog,msg:'SQL Injection
> Attack',id:1234123414,tag:'WEB_ATTACK/SQL_INJECTION',logdata:'%{TX.0}',severity:'2',deny"
> 
> But again, using REQUEST_COOKIES as a target is not meaningful because the regex in question doesn't match the
> cookie value, only the name+value.

Hello Robert and Christian,

Thank you for your help figuring this out!

This is a default rule from cPanel that tries to match content like "1=1" 
for SQL injections. Ultimately matching a cookie on an equals sign in the 
header is a false positive because the header always contains 
<cookie_name>=<cookie_value>; an application is only ever going to use the 
cookie name or cookie value independently, but not at it appears in the 
header (I hope!).

To solve this, we have created a rule based on your suggestions that 
specifically checks this SQL injection attack in cookie names and cookie 
values independently and excluded cookie processing from the header 
section to prevent the false positive.

This solves the issue for Google cookies and is more correct in terms of 
addressing this type of attack.

These are the new rules:


SecRuleUpdateTargetById 1234123413 "!REQUEST_HEADERS:Cookie"

SecRule REQUEST_COOKIES_NAMES|REQUEST_COOKIES "\b(\d+) ?= ?\1\b|[\'\"](\w+)[\'\"] ?= ?[\'\"]\2\b" \
        "phase:2,capture,t:none,t:urlDecodeUni,t:htmlEntityDecode,t:replaceComments,t:compressWhiteSpace,t:lowercase,ctl:auditLogParts=+E,log,auditlog,msg:'SQL Injection Attack',id:'1334123413',tag:'WEB_ATTACK/SQL_INJECTION',logdata:'%{TX.0}',severity:'2'"


Thank you again!

--
Eric Wheeler

> 
> 
> 
> On Mon, Apr 2, 2018 at 2:20 PM, Robert Paprocki <rpa...@fe...> wrote:
>       Okay, so I got the following to work (as expected), ignoring SecRuleUpdateTargetById which I've never
>       had much luck with:
> SecRule REQUEST_HEADERS|XML:/*|!REQUEST_HEADERS:Referer "\b(\d+) ?= ?\1\b|[\'\"](\w+)[\'\"] ?= ?[\'\"]\2\b"
> \
>        "phase:2,capture,t:none,t:urlDecodeUni,t:htmlEntityDecode,t:replaceComments,t:compressWhiteSpace,t:lowercase,ctl
> :auditLogParts=+E,log,auditlog,msg:'SQL Injection
> Attack',id:1234123413,tag:'WEB_ATTACK/SQL_INJECTION',logdata:'%{TX.0}',severity:'2',deny"
> 
> SecRule ARGS:test "@streq foo" "id:12345,deny,phase:2,msg:'lolnope',sanitiseArg:yomama"
> 
> SecAction "ctl:ruleRemoveTargetById=1234123413;REQUEST_HEADERS:Cookie,id:12346,phase:1"
> 
> Relevent debug logs:
> 
> https://gist.github.com/p0pr0ck5/3d7b38c3c182604242449ff0b04c444d
> 
> @Eric, because your rule targets REQUEST_HEADERS and not COOKIES, you cannot remove a specific cookie from
> this target. Headers and cookies are treated as separated tables. You will either need to use an ignore or
> craft a second rule that looks specifically at cookies.
> 
> BTW, you may want to evaluate the regex in question here; what specifically are you trying to catch? It
> matches on a cookie only because of how cookies are formed.
> 
> On Mon, Apr 2, 2018 at 1:41 PM, Robert Paprocki <rpa...@fe...> wrote:
>       Hey Christian,
> 
>       On Mon, Apr 2, 2018 at 1:38 PM, Christian Folini <chr...@ne...> wrote:
>             Hello Eric,
> 
>             On Mon, Apr 02, 2018 at 07:31:14PM +0000, Eric Wheeler wrote:
>             > We have tried the following, but none have worked:
>             >
>             >       SecRuleUpdateTargetById 1234123413
>             "!REQUEST_COOKIES_NAMES:/_gac_UA-5521579-1/"
>             >       SecRuleUpdateTargetById 1234123413 "!REQUEST_COOKIES:/_gac_UA-5521579-1/"
>             >
>             >       SecRuleUpdateTargetById 1234123413 "!REQUEST_HEADERS:/_gac_UA-5521579-1/"
>             >
>             >       SecRuleUpdateTargetById 1234123413
>             "!REQUEST_COOKIES_NAMES:_gac_UA-5521579-1"
>             >       SecRuleUpdateTargetById 1234123413 "!REQUEST_COOKIES:_gac_UA-5521579-1"
>             >
>             >
>             > Interestingly, these two work, but are of course too permissive:
>             >
>             >       SecRuleUpdateTargetById 1234123413 "!REQUEST_HEADERS:/./"
>             >       SecRuleUpdateTargetById 1234123413 "!REQUEST_HEADERS:Cookie"
> 
>             If
>             SecRuleUpdateTargetById 1234123413 "!REQUEST_HEADERS:/./"
>             works, it's undocumented behaviour. This does not really support regexes.
> 
> 
> From
> https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-%28v2.x%29#SecRuleUpdateTargetById:
> 
> Note that is is also possible to use regular expressions in the target specification:
> 
> SecRuleUpdateTargetById 981172 "!REQUEST_COOKIES:/^appl1_.*/"
> 
> Interestingly, neither of the following work for me:
> 
> SecRuleUpdateTargetById 1234123413 "!REQUEST_HEADERS:/./"
> SecRuleUpdateTargetById 1234123413 "!REQUEST_HEADERS:Cookie"  
> 
> And there is no meaningful log-level 9 debug information to indicate that SecRuleUpdateTargetById did
> anything (im still walkingthrough https://github.com/SpiderLabs/ModSecurity/blob/72f632e9b6b2e63677cfba7e62a47efb87c90b48/apache2/re.c#L19
> 8 at this point- busy watching the Falcon 9 launch atm ;) ).
> 
> 
> 
> 
> 

Re: [mod-security-users] [Mod-security-developers] ModSecurity version 3.0.1 announcement

[Robert P. <rpa...@fe...>]

Hi,

On Wed, Apr 4, 2018 at 5:03 AM, Christian Folini <
chr...@ne...> wrote:
>
>
> > However, I have limited knowledge on the following:
> >  - is ModSec 3.x has been ever targeted to support CRS < 3,
>
> See above.
>

It would be great to here an official stance from the development team on
this. @Felipe can you comment?



> > Also I'm not sure whether ModSec 2.x has its own benchmarks (not related
> to any connector).
> > If it does, then perhaps it would be good to compare "generic" ModSec 2.x
> > vs "generic" ModSec 3.x as well.
>
> Yes, that would be cool. But from what I understand, ModSec 2.9.x is deeply
> integrated into the webserver.
>


Yeah, this is a bit of a pain to test. I modified one of the example
programs that comes with the v3/master ModSecurity source code as follows:

https://gist.github.com/p0pr0ck5/9b2c414641c9b03d527679d0c8cb7d86

Note that this doesn't add any headers or body data (request or response)
to the transaction, and the included "basic_rules.conf" is unchanged from
what's in the example repo (and ignore the memory leak in not cleaning up
the transaction). So running this very light example:

$ ./test
Rules:
Phase: 0 (0 rules)
Phase: 1 (0 rules)
Phase: 2 (2 rules)
    Rule ID: 200000--0x1e2b6c0
    Rule ID: 200001--0x1e2bc90
Phase: 3 (4 rules)
    Rule ID: 200002--0x1e2c720
    Rule ID: 200003--0x1e2ea00
    Rule ID: 200004--0x1e2f200
    Rule ID: 200005--0x1e2fc10
Phase: 4 (0 rules)
Phase: 5 (0 rules)
Phase: 6 (0 rules)
Phase: 7 (0 rules)
Did 9907
Done!

We see around 10k processes per second. Essentially all of our time is
spent waiting on memory allocations:
https://s3.amazonaws.com/p0pr0ck5-data/modsec-simple.svg

Now consider the case where we include the full 3.0.0 CRS:

$ ./test
Rules:
Phase: 0 (32 rules)
    Rule ID: 0--0x22ca500
    Rule ID: 0--0x22d9480
    Rule ID: 0--0x22f9530
    Rule ID: 0--0x22f9f20
[...snip several hundred lines...]
Did 319
Done!

So this is about the same throughput we saw from Nginx + libmodsec
integration. A flamegraph also highlights hot paths, particularly in
Rule::evaluate (also largely spent on allocation wait time at this point,
since each evaluation instantiates many new objects):
https://s3.amazonaws.com/p0pr0ck5-data/modsec-simple-crs.svg

(I also performed the same tests on 3.0.0; flamegraphs are also in this s3
bucket but I will avoid commentary here for now as this thread is gotten
long in the tooth as it is).

A few initial takeaways:

- There is a clearly definable minimum overhead needed to execute
libmodsecurity, based on its current architecture
- I noted the same memory leak that defanator noted in #1729. I have yet to
apply the patches noted.
- HTTP integrations will induce some overhead, which may or may not be
substantial. With smart memory pooling and good design, though, this should
be kept to a minimum.
- Leveraging the full CRS completely tanks throughput. It would take a
while more to dig into specifics but I suspect that Rule::evaluate probably
needs an overhaul.
- I ran one more test again a mock set of dumb rules (generated via
https://gist.github.com/p0pr0ck5/c99bca54734af7546d910db8d7c97ab3). Saw
about 1000 transactions processed per second, indicating (as should be
assumed) that linear growth in the size of the included ruleset results in
similar performance reduction. Flamegraphs again point to Rule::evaluate,
with Rule::getFinalVars taking up half those traces. I suspect there could
be some smarter behaviors about avoiding object creation if its unneeded on
these hot paths.

Hopefully this is of use to some folks. I'd be interested in writing some
automated scaffolding to execute this type of testing against every commit,
to maintain a continuous audit trail of performance impacts; if this sounds
interesting to the maintainers I'd be happy to chat further.

Re: [mod-security-users] [Mod-security-developers] ModSecurity version 3.0.1 announcement

[Christian F. <chr...@ne...>]

Hello Andrei,

On Wed, Apr 04, 2018 at 02:14:12PM +0300, Andrei Belov wrote:
> Well, ideally it would be awesome to have the following combos in [perf] tests:
> 
>  a) Apache + ModSec 2.x + CRS 2.x
>  b) Apache + ModSec 3.x + CRS 3.x
>  c) nginx + ModSec 2.x + CRS 2.x
>  d) nginx + ModSec 3.x + CRS 3.x
> 
> (obviously, CRS component could be optional when one is going to measure
> "generic overhead")

I think CRS3 can serve as a general baseline to get a standard rule base.

As a CRS project lead, I hope people abandon CRS2 and move to CRS3 not the
least because the performance is better due to the smaller rule set in the
default installation. The ModSecurity Handbook has the numbers on Apache /
ModSec 2.9.x.

Personally, I would not test CRS2 anymore.

> However, I have limited knowledge on the following:
>  - is ModSec 3.x has been ever targeted to support CRS < 3,

See above.

>  - is there a working Apache connector for ModSec 3.x.

According to Felipe it is not ready for production.

> Also I'm not sure whether ModSec 2.x has its own benchmarks (not related to any connector).
> If it does, then perhaps it would be good to compare "generic" ModSec 2.x
> vs "generic" ModSec 3.x as well.

Yes, that would be cool. But from what I understand, ModSec 2.9.x is deeply
integrated into the webserver.

But I read from your proposal above that the real base to gauge 
ModSec 2.9.x vs 3.0 would be to test on NGINX.

> BTW, for those who are familiar with tools like gdb / perf / systemtap etc,
> there's the "debugenv" state in vagrant env:
> 
> https://github.com/defanator/modsecurity-performance/blob/master/states/debugenv.sls

Thanks.

Ahoj,

Christian


-- 
Money is always to be found when men are to be sent to the frontiers to be
destroyed: when the object is to preserve them, it is no longer so.
-- Voltaire

Re: [mod-security-users] [Mod-security-developers] ModSecurity version 3.0.1 announcement

[Andrei B. <de...@ng...>]

> On 04 Apr 2018, at 11:58, Christian Folini <chr...@ne...> wrote:
> 
> Hello Andrei,
> 
> On Wed, Apr 04, 2018 at 11:29:18AM +0300, Andrei Belov wrote:
>> I think that environment could be [relatively easily] extended to support
>> Apache + ModSec 2.x, in addition to nginx + ModSec 3.x, in order to simplify
>> "direct" comparison and provide reproducible, statistically significant results.
> 
> Very cool. Thank you for sharing - and thanks for your contributions to
> ModSecurity, namely 3.0.1.
> 
> The conceptual problem is see is that it's more than one variable here.
> Apache/ModSec2 vs. NGINX/ModSec3. I'm an Apache person, but when I stripped
> the two of Modsec and let the bare minimum installations serve static
> files, NGINX blew me away.
> 
> So I kind of think that one would have to slow down NGINX to reach an Apache
> level and then in a 2nd step add ModSec again to be able to measure ModSec2 vs
> ModSec3.
> 
> What is your take on this?

Well, ideally it would be awesome to have the following combos in [perf] tests:

 a) Apache + ModSec 2.x + CRS 2.x
 b) Apache + ModSec 3.x + CRS 3.x
 c) nginx + ModSec 2.x + CRS 2.x
 d) nginx + ModSec 3.x + CRS 3.x

(obviously, CRS component could be optional when one is going to measure
"generic overhead")

However, I have limited knowledge on the following:
 - is ModSec 3.x has been ever targeted to support CRS < 3,
 - is there a working Apache connector for ModSec 3.x.

Also I'm not sure whether ModSec 2.x has its own benchmarks (not related to any connector).
If it does, then perhaps it would be good to compare "generic" ModSec 2.x
vs "generic" ModSec 3.x as well.

BTW, for those who are familiar with tools like gdb / perf / systemtap etc,
there's the "debugenv" state in vagrant env:

    https://github.com/defanator/modsecurity-performance/blob/master/states/debugenv.sls

It could be useful for some deeper investigations.

Re: [mod-security-users] Sanitize JSON Request / Response

[Osama E. <oel...@gm...>]

Hi Robert,

Sorry for the late reply.  I was a little busy the past 2 weeks.

I just retested it (on 2.9.2 + manually patching the two files mentioned
here - https://github.com/SpiderLabs/ModSecurity/pull/1714/files)  It looks
like:

- it works if the output format (SecAuditLogFormat) is JSON
- it doesn't work if the output is Native

For example, with the following rule:

SecAction "phase:5,id:22,nolog,pass,sanitiseArg:cvv"

and the following request:

curl -H "Content-Type: application/json" -X POST -d '{"cvv":"123"}'
http://localhost/?id=/bin/bash

and SecAuditLogFormat JSON, I get the following:

.. "body":["{\"cvv\":\"***\"}"]} ..

while with normal native logging, I get the following:

--c2397953-C--
{"cvv":"123"}

In my previous email below, I was looking at native logging so it appeared
that it wasn’t sanitizing the output.

Thanks.

-- 
Osama Elnaggar

On March 21, 2018 at 11:19:51 AM, Robert Paprocki (
rpa...@fe...) wrote:

Can you post a reproducible example (and what git commit you built against)
to verify? Im afk and don't have access to my dev env but I did see that
the current v2/master does sanitize args in json bodied noted with the
"sanitizearg" rule action (assuming the body processing engine is
configured appropriately of course)

Sent from my iPhone

On Mar 20, 2018, at 14:25, Osama Elnaggar <oel...@gm...> wrote:

Are you sure about this?  I tried it with the format set to JSON and it
still didn’t work.

If you are interesting in tackling this, below are some pointers Felipe
sent a few months ago:

The general idea is that the content to be sanitized is placed under an
apt_trable that is further checked while the logs are being generated.

For instance: regarding the request headers, we have the log generation
detailed here:

https://github.com/SpiderLabs/ModSecurity/blob/bb577950bf983811ff1892e87d815a1909c0b96b/apache2/msc_logging.c#L1211-L1236


As of the "sanitization selection”, we have:

https://github.com/SpiderLabs/ModSecurity/blob/923c3c67938da4de4f7f147816b6d2d6ffff5e6f/apache2/re_actions.c#L1425-L1431


The sanitizeMatch is a little bit more complex, but still uses the same
logic:

https://github.com/SpiderLabs/ModSecurity/blob/923c3c67938da4de4f7f147816b6d2d6ffff5e6f/apache2/re_actions.c#L1350-L1423

All the sanitize actions are bound here -

https://github.com/SpiderLabs/ModSecurity/blob/923c3c67938da4de4f7f147816b6d2d6ffff5e6f/apache2/re_actions.c#L2663-L2791

Both JSON and XML have their own structures to hold their data, as you can
see:

- JSON:
https://github.com/SpiderLabs/ModSecurity/blob/923c3c67938da4de4f7f147816b6d2d6ffff5e6f/apache2/modsecurity.h#L391

- XML:
https://github.com/SpiderLabs/ModSecurity/blob/923c3c67938da4de4f7f147816b6d2d6ffff5e6f/apache2/modsecurity.h#L389


-- 
Osama Elnaggar

On March 21, 2018 at 5:41:12 AM, Robert Paprocki (
rpa...@fe...) wrote:

Whups, wow, I really need to open my eyes :p

The patch above allows for sanitizing JSON request bodies only when
SecAuditLogFormat is *also* set to JSON. I've pushed up
https://github.com/SpiderLabs/ModSecurity/pull/1714 which enables
sanitization of JSON request bodies in native audit log formats.

On Thu, Mar 15, 2018 at 1:07 PM, Osama Elnaggar <oel...@gm...>
wrote:

> I don't think the proposed patch actually works.  I tried patching v2.9.2
> with it and even using v2 master but with no success.  Have you been able
> to get the patch working Robert?
>
> --
> Osama Elnaggar
>
> On March 15, 2018 at 11:06:37 AM, Robert Paprocki (rpaprocki@
> fearnothingproductions.net) wrote:
>
> Have a look at
>
> https://github.com/SpiderLabs/ModSecurity/commit/
> f86de566d18dda6351ecba52d5e5f1d29ad02a12
>
> JSON body audit log sanitization was only very recently introduced, it's
> not yet made its way to a formal release. (I need to check sources before
> opening my mouth :p).
>
> So you can rebuild ModSecurity off `v2/master` if you want to test this
> functionality. :)
>
> On Wed, Mar 14, 2018 at 4:47 PM, Cristiano Galdino <cri...@ga...>
> wrote:
>
>> Hello there!
>>
>> If modsecurity can parse the values ​​of JSON payloads, why can not it
>> sanitize?
>>
>> This is non-sense for me.
>>
>> Look this request:
>> $> curl -H "Content-Type: application/json" -X POST -d
>> '{"CVV":"123","blah":"/bin/bash"}' localhost/Authenticate
>>
>> and this is audit-log:
>>
>> --9eb5dc70-A--
>>
>> [14/Mar/2018:20:37:35 --0300] WqmyP6wfJasAAFQJf@AAAAAS 127.0.0.1 53230
>> 127.0.0.1 80
>>
>> --9eb5dc70-B--
>>
>> POST /Authenticate HTTP/1.1
>>
>> Host: localhost
>>
>> User-Agent: curl/7.47.0
>>
>> Accept: */*
>>
>> Content-Type: application/json
>>
>> Content-Length: 36
>>
>>
>> --9eb5dc70-C--
>>
>> {"CVV":"123","blah":"/bin/bash"}
>>
>> --9eb5dc70-E--
>>
>> {"message":"Failed"}
>>
>> --9eb5dc70-F--
>>
>> HTTP/1.1 400 Bad Request
>>
>> Access-Control-Allow-Origin: *
>>
>> Content-Type: application/json
>>
>> Content-Length: 190
>>
>> X-Content-Type-Options: nosniff
>>
>> X-Frame-Options: sameorigin
>>
>> Connection: close
>>
>>
>> --9eb5dc70-H--
>>
>> Message: Warning. Matched phrase "bin/bash" at ARGS:blah. [file
>> "/usr/share/modsecurity-crs/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf"]
>> [line "448"] [id "932160"] [rev "1"] [msg "Remote Command Execution: Unix
>> Shell Code Found"] [data "Matched Data: bin/bash found within ARGS:blah:
>> /bin/bash"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "1"]
>> [accuracy "8"] [tag "application-multi"] [tag "language-shell"] [tag
>> "platform-unix"] [tag "attack-rce"] [tag "OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION"]
>> [tag "WASCTC/WASC-31"] [tag "OWASP_TOP_10/A1"] [tag "PCI/6.5.2"]
>>
>> Message: Warning. Operator GE matched 5 at TX:anomaly_score. [file
>> "/usr/share/modsecurity-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"]
>> [line "57"] [id "949110"] [msg "Inbound Anomaly Score Exceeded (Total
>> Score: 5)"] [severity "CRITICAL"] [tag "application-multi"] [tag
>> "language-multi"] [tag "platform-multi"] [tag "attack-generic"]
>>
>> Message: Warning. Operator GE matched 5 at TX:inbound_anomaly_score.
>> [file "/usr/share/modsecurity-crs/rules/RESPONSE-980-CORRELATION.conf"]
>> [line "73"] [id "980130"] [msg "Inbound Anomaly Score Exceeded (Total
>> Inbound Score: 5 - SQLI=0,XSS=0,RFI=0,LFI=0,RCE=5,PHPI=0,HTTP=0,SESS=0):
>> Remote Command Execution: Unix Shell Code Found"] [tag "event-correlation"]
>>
>> Apache-Handler: proxy-server
>>
>> Stopwatch: 1521070655519139 8420 (- - -)
>>
>> Stopwatch2: 1521070655519139 8420; combined=1400, p1=343, p2=801, p3=40,
>> p4=129, p5=86, sr=35, sw=1, l=0, gc=0
>>
>> Response-Body-Transformed: Dechunked
>>
>> Producer: ModSecurity for Apache/2.9.0 (http://www.modsecurity.org/);
>> OWASP_CRS/3.0.0.
>>
>> Server: Apache/2.4.18
>>
>> Sanitised-Args: "CVV".
>>
>> Engine-Mode: "DETECTION_ONLY"
>>
>>
>> --9eb5dc70-Z--
>>
>>
>>
>> Cristiano Galdino
>> (61) 9860 1 9860
>> cri...@ga...
>>
>> On 14 Mar 2018 19:05 -0300, Christian Folini <chr...@ne...>,
>> wrote:
>>
>> Sorry, I was a bit quick to jump to that conclusion. Overlooked your
>> remark
>> on JSON.
>>
>> I confirm this does not work.
>>
>> Sanitation is generally an issue as there is no sanitation in the alerts
>> written into the error-log. Even it is less severe as the audit log.
>>
>> Best,
>>
>> Christian
>>
>>
>> On Wed, Mar 14, 2018 at 06:41:25PM -0300, Cristiano Galdino wrote:
>>
>> Yep! My application use JSON payloads.
>> Christian, please try it:
>> $> curl -H "Content-Type: application/json" -X POST -d '{"cvv”:"123"}'
>> [1]http://localhost/?id=/bin/bash
>>
>> Cristiano Galdino
>> (61) 9860 1 9860
>> cri...@ga...
>>
>> On 14 Mar 2018 18:38 -0300, Robert Paprocki
>> <rpa...@fe...>, wrote:
>>
>> Christian, you tested with a application/x-www-form-urlencoded
>> request; Christiano's use case involves JSON-encoded bodies.
>> I do not believe JSON request bodies can be translated into data
>> collections that can have sanitize actions applied on them at this
>> point.
>>
>> On Wed, Mar 14, 2018 at 2:34 PM, Christian Folini
>> <[2]chr...@ne...> wrote:
>>
>> Hello Cristiano,
>> I did the following request:
>> $> curl localhost -d "CVV=0000-0000-0000-0000" -d "exec=/bin/bash"
>> and got the following audit-log when using CRS3 (parameter exec
>> triggering
>> the writing of the audit log):
>> --a7997f3d-A--
>> [14/Mar/2018:22:29:03 +0100] WqmUH6r6pkVX9OUmJm3aggAAAAM 127.0.0.1
>> 50058 127.0.0.1 40080
>> --a7997f3d-B--
>> POST / HTTP/1.1
>> Host: localhost
>> User-Agent: curl/7.50.1
>> Accept: */*
>> Content-Length: 38
>> Content-Type: application/x-www-form-urlencoded
>> --a7997f3d-C--
>> CVV=*******************&exec=/bin/bash
>> --a7997f3d-F--
>> HTTP/1.1 200 OK
>> Last-Modified: Sun, 17 Dec 2017 11:08:45 GMT
>> ETag: "2d-5608741dac6fd"
>> Accept-Ranges: bytes
>> Content-Length: 45
>> Content-Type: text/html
>> ...
>> I'm running ModSec 2.9.2 on Apache 2.4.29, both self compiled
>> according to
>> the tutorials on [3]netnea.com.
>> My ModSec Configuration:
>> ------------------------------------------------------------
>> ------------------
>> SecRuleEngine On
>> SecRequestBodyAccess On
>> SecRequestBodyLimit 10000000
>> SecRequestBodyNoFilesLimit 64000
>> SecResponseBodyAccess On
>> SecResponseBodyLimit 10000000
>> SecTmpDir /tmp/
>> SecDataDir /tmp/
>> SecUploadDir /tmp/
>> SecDebugLog /apache/logs/modsec_debug.log
>> SecDebugLogLevel 3
>> SecAuditEngine RelevantOnly
>> SecAuditLogRelevantStatus "^(?:5|4(?!04))"
>> SecAuditLogParts ABEFHIJZ
>> SecAuditLogType Concurrent
>> SecAuditLog /apache/logs/modsec_audit.log
>> SecAuditLogStorageDir /apache/logs/audit/
>> SecPcreMatchLimit 500000
>> SecPcreMatchLimitRecursion 500000
>> SecDefaultAction "phase:2,pass,log"
>> # == ModSec Rule ID Namespace Definition
>> # Service-specific before Core-Rules: 10000 - 49999
>> # Service-specific after Core-Rules: 50000 - 79999
>> # Locally shared rules: 80000 - 99999
>> # - Performance: 90000 - 90199
>> # Recommended ModSec Rules (few): 200000 - 200010
>> # OWASP Core-Rules: 900000 - 999999
>> # === ModSec timestamps at the start of each phase (ids: 90000 -
>> 90009)
>> SecAction "id:'90000',phase:1,nolog,pass,setvar:TX.
>> ModSecTimestamp1start=%{DURATION}"
>> SecAction "id:'90001',phase:2,nolog,pass,setvar:TX.
>> ModSecTimestamp2start=%{DURATION}"
>> SecAction "id:'90002',phase:3,nolog,pass,setvar:TX.
>> ModSecTimestamp3start=%{DURATION}"
>> SecAction "id:'90003',phase:4,nolog,pass,setvar:TX.
>> ModSecTimestamp4start=%{DURATION}"
>> SecAction "id:'90004',phase:5,nolog,pass,setvar:TX.
>> ModSecTimestamp5start=%{DURATION}"
>> # SecRule REQUEST_FILENAME "@beginsWith /"
>> "id:'90005',phase:5,t:none,nolog,noauditlog,pass,setenv:
>> write_perflog"
>> # === ModSec Recommended Rules (in modsec src package) (ids:
>> 200000-200010)
>> SecRule REQUEST_HEADERS:Content-Type "text/xml"
>> "id:'200000',phase:1,t:none,t:lowercase,pass,nolog,ctl:
>> requestBodyProcessor=XML"
>> SecRule REQBODY_ERROR "!@eq 0" "id:'200001',phase:2,t:none,
>> deny,status:400,log,msg:'Failed to parse request body.',\
>> logdata:'%{reqbody_error_msg}',severity:2"
>> SecRule MULTIPART_STRICT_ERROR "!@eq 0" \
>> "id:'200002',phase:2,t:none,log,deny,status:403, \
>> msg:'Multipart request body failed strict validation: \
>> PE %{REQBODY_PROCESSOR_ERROR}, \
>> BQ %{MULTIPART_BOUNDARY_QUOTED}, \
>> BW %{MULTIPART_BOUNDARY_WHITESPACE}, \
>> DB %{MULTIPART_DATA_BEFORE}, \
>> DA %{MULTIPART_DATA_AFTER}, \
>> HF %{MULTIPART_HEADER_FOLDING}, \
>> LF %{MULTIPART_LF_LINE}, \
>> SM %{MULTIPART_MISSING_SEMICOLON}, \
>> IQ %{MULTIPART_INVALID_QUOTING}, \
>> IP %{MULTIPART_INVALID_PART}, \
>> IH %{MULTIPART_INVALID_HEADER_FOLDING}, \
>> FL %{MULTIPART_FILE_LIMIT_EXCEEDED}'"
>> SecRule TX:/^MSC_/ "!@streq 0" "id:'200004',phase:2,t:none,
>> deny,status:500,msg:'ModSecurity internal error flagged:
>> %{MATCHED_VAR_NAME}'"
>> # === ModSecurity Rules (ids: 900000-999999)
>> # === ModSec Core Rules Base Configuration (ids: 900001-900021)
>> Include /home/dune73/data/git/crs-official/crs-setup.conf.
>> example
>> SecAction "id:900111,phase:1,nolog,pass,t:none,setvar:tx.inbound_
>> anomaly_score_threshold=500,setvar:tx.outbound_anomaly_
>> score_threshold=500"
>> SecAction "id:'900000',phase:1,nolog,pass,t:none,setvar:tx.
>> paranoia_level=4"
>> # === ModSecurity Ignore Rules Before Core Rules Inclusion; order by
>> id of ignored rule (ids: 10000-49999)
>> # SecRule ARGS:a "."
>> "id:1001,phase:2,pass,log,msg:'XXX1: %{MATCHED_VAR}'"
>> # SecRule ARGS_GET:a "."
>> "id:1002,phase:2,pass,log,msg:'XXX2: %{MATCHED_VAR}'"
>> # SecRule ARGS_POST:a "."
>> "id:1003,phase:2,pass,log,msg:'XXX3: %{MATCHED_VAR}'"
>> # SecRule REQUEST_URI "."
>> "id:1004,phase:2,pass,log,msg:'XXX4: %{MATCHED_VAR}'"
>> # SecRule REQUEST_HEADERS:User-Agent "."
>> "id:1005,phase:2,pass,log,msg:'XXX5: %{MATCHED_VAR}'"
>> SecRule ARGS:b "." "id:1006,phase:2,pass,log,
>> auditlog,msg:'XXX6: %{MATCHED_VAR}'"
>> SecAction "nolog,phase:2,id:101,sanitiseArg:CVV"
>> SecAction "nolog,phase:4,id:102,sanitiseArg:CVV_Reponse"
>> # === ModSecurity Core Rules Inclusion
>> Include /home/dune73/data/git/crs-official/rules/*.conf
>> # === ModSec Core Rules: Startup Time Rules Exclusions
>> # === ModSec timestamps at the end of each phase (ids: 90010 -
>> 90019)
>> SecAction "id:'90010',phase:1,pass,nolog,setvar:TX.
>> ModSecTimestamp1end=%{DURATION}"
>> SecAction "id:'90011',phase:2,pass,nolog,setvar:TX.
>> ModSecTimestamp2end=%{DURATION}"
>> SecAction "id:'90012',phase:3,pass,nolog,setvar:TX.
>> ModSecTimestamp3end=%{DURATION}"
>> SecAction "id:'90013',phase:4,pass,nolog,setvar:TX.
>> ModSecTimestamp4end=%{DURATION}"
>> SecAction "id:'90014',phase:5,pass,nolog,setvar:TX.
>> ModSecTimestamp5end=%{DURATION}"
>> # === ModSec performance calculations and variable export (ids:
>> 90100 - 90199)
>> SecAction "id:'90100',phase:5,pass,nolog,setvar:TX.perf_
>> modsecinbound=%{PERF_PHASE1}"
>> SecAction "id:'90101',phase:5,pass,nolog,setvar:TX.perf_
>> modsecinbound=+%{PERF_PHASE2}"
>> SecAction "id:'90102',phase:5,pass,nolog,setvar:TX.perf_
>> application=%{TX.ModSecTimestamp3start}"
>> SecAction "id:'90103',phase:5,pass,nolog,setvar:TX.perf_
>> application=-%{TX.ModSecTimestamp2end}"
>> SecAction "id:'90104',phase:5,pass,nolog,setvar:TX.perf_
>> modsecoutbound=%{PERF_PHASE3}"
>> SecAction "id:'90105',phase:5,pass,nolog,setvar:TX.perf_
>> modsecoutbound=+%{PERF_PHASE4}"
>> SecAction "id:'90106',phase:5,pass,nolog,setenv:ModSecTimeIn=%{
>> TX.perf_modsecinbound}"
>> SecAction "id:'90107',phase:5,pass,nolog,setenv:ApplicationTime=%
>> {TX.perf_application}"
>> SecAction "id:'90108',phase:5,pass,nolog,setenv:ModSecTimeOut=%{
>> TX.perf_modsecoutbound}"
>> SecAction "id:'90109',phase:5,pass,nolog,setenv:
>> ModSecAnomalyScoreIn=%{TX.anomaly_score}"
>> SecAction "id:'90110',phase:5,pass,nolog,setenv:
>> ModSecAnomalyScoreOut=%{TX.outbound_anomaly_score}"
>> # === End ModSec Configuration
>> ------------------------------------------------------------
>> ------------------
>> So I think this generally works. If it does not for you, then please
>> try and
>> reproduce the behaviour on the latest ModSec version of the 2.9
>> series and
>> open a bug report in case.
>> Ahoj,
>> Christian
>> On Wed, Mar 14, 2018 at 06:13:04PM -0300, Cristiano Galdino wrote:
>>
>> Hi Christian!
>> Modsecurity: 2.9.0-1 (from Ubuntu repository)
>> Apache 2.4.18-2ubuntu3.5
>> Tks!
>>
>> Cristiano Galdino
>> [4]cri...@ga...
>>
>> On 14 Mar 2018 17:55 -0300, Christian Folini
>> <[5]chr...@ne...>, wrote:
>>
>> Hello Christiano,
>> What platform are you using? (-> ModSec version, Apache /
>>
>> NGINX /
>>
>> IIS?)
>> Ahoj,
>> Christian
>> On Wed, Mar 14, 2018 at 05:06:28PM -0300, Cristiano Galdino
>>
>> wrote:
>>
>>
>> Hello!
>> I created a rule in ModSecurity to sanitize param CVV (credit
>>
>> card)
>>
>> but
>> it is not working.
>> Samples:
>> SecAction "nolog,phase:2,id:101,sanitiseArg:CVV”
>> SecAction "nolog,phase:4,id:102,sanitiseArg:CVV_Reponse"
>> This prevents me from using modsecurity because PCI does not
>>
>> allow
>>
>> CVV
>> to be stored.
>> I found this issue without response.
>> [1][6]https://github.com/SpiderLabs/ModSecurity/issues/715
>> What can I do?
>> Cristiano Galdino
>> [7]cri...@ga...
>> References
>> 1. [8]https://github.com/SpiderLabs/ModSecurity/issues/715
>>
>> ------------------------------------------------------------
>>
>> --------
>>
>> ----------
>> Check out the vibrant tech community on one of the world's
>>
>> most
>>
>> engaging tech sites, Slashdot.org!
>>
>> [9]http://sdm.link/slashdot
>>
>>
>> _______________________________________________
>> mod-security-users mailing list
>> [10]mod...@li...
>> [11]https://lists.sourceforge.net/
>>
>> lists/listinfo/mod-security-users
>>
>> Commercial ModSecurity Rules and Support from Trustwave's
>> SpiderLabs:
>> [12]http://www.modsecurity.org/projects/commercial/rules/
>> [13]http://www.modsecurity.org/projects/commercial/support/
>>
>> --
>> [14]https://www.feistyduck.com/training/modsecurity-training-
>>
>> course
>>
>> [15]https://www.feistyduck.com/books/modsecurity-handbook/
>> mailto:[16]chr...@ne...
>> twitter: @ChrFolini
>> ------------------------------------------------------------
>>
>> --------
>>
>> ----------
>> Check out the vibrant tech community on one of the world's
>>
>> most
>>
>> engaging tech sites, Slashdot.org!
>>
>> [17]http://sdm.link/slashdot
>>
>> _______________________________________________
>> mod-security-users mailing list
>> [18]mod...@li...
>> [19]https://lists.sourceforge.net/
>>
>> lists/listinfo/mod-security-users
>>
>> Commercial ModSecurity Rules and Support from Trustwave's
>> SpiderLabs:
>> [20]http://www.modsecurity.org/projects/commercial/rules/
>> [21]http://www.modsecurity.org/projects/commercial/support/
>> ------------------------------------------------------------
>>
>> ------------------
>>
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! [22]http://sdm.link/slashdot
>> _______________________________________________
>> mod-security-users mailing list
>> [23]mod...@li...
>> [24]https://lists.sourceforge.net/lists/listinfo/mod-security-
>>
>> users
>>
>> Commercial ModSecurity Rules and Support from Trustwave's
>>
>> SpiderLabs:
>>
>> [25]http://www.modsecurity.org/projects/commercial/rules/
>> [26]http://www.modsecurity.org/projects/commercial/support/
>>
>> --
>> [27]https://www.feistyduck.com/training/modsecurity-training-course
>> [28]https://www.feistyduck.com/books/modsecurity-handbook/
>> mailto:[29]chr...@ne...
>> twitter: @ChrFolini
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! [30]http://sdm.link/slashdot
>> _______________________________________________
>> mod-security-users mailing list
>> [31]mod...@li...
>> [32]https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's
>> SpiderLabs:
>> [33]http://www.modsecurity.org/projects/commercial/rules/
>> [34]http://www.modsecurity.org/projects/commercial/support/
>>
>> --------------------------------------------------------------------
>> ----------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's
>> SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
>>
>> References
>>
>> 1. http://localhost:3000/api/login
>> 2. mailto:chr...@ne...
>> 3. http://netnea.com/
>> 4. mailto:cri...@ga...
>> 5. mailto:chr...@ne...
>> 6. https://github.com/SpiderLabs/ModSecurity/issues/715
>> 7. mailto:cri...@ga...
>> 8. https://github.com/SpiderLabs/ModSecurity/issues/715
>> 9. http://sdm.link/slashdot
>> 10. mailto:mod...@li...
>> 11. https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> 12. http://www.modsecurity.org/projects/commercial/rules/
>> 13. http://www.modsecurity.org/projects/commercial/support/
>> 14. https://www.feistyduck.com/training/modsecurity-training-course
>> 15. https://www.feistyduck.com/books/modsecurity-handbook/
>> 16. mailto:chr...@ne...
>> 17. http://sdm.link/slashdot
>> 18. mailto:mod...@li...
>> 19. https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> 20. http://www.modsecurity.org/projects/commercial/rules/
>> 21. http://www.modsecurity.org/projects/commercial/support/
>> 22. http://sdm.link/slashdot
>> 23. mailto:mod...@li...
>> 24. https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> 25. http://www.modsecurity.org/projects/commercial/rules/
>> 26. http://www.modsecurity.org/projects/commercial/support/
>> 27. https://www.feistyduck.com/training/modsecurity-training-course
>> 28. https://www.feistyduck.com/books/modsecurity-handbook/
>> 29. mailto:chr...@ne...
>> 30. http://sdm.link/slashdot
>> 31. mailto:mod...@li...
>> 32. https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> 33. http://www.modsecurity.org/projects/commercial/rules/
>> 34. http://www.modsecurity.org/projects/commercial/support/
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>
>>
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
>>
>>
>>
>> --
>> https://www.feistyduck.com/training/modsecurity-training-course
>> https://www.feistyduck.com/books/modsecurity-handbook/
>> mailto:chr...@ne...
>> twitter: @ChrFolini
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
>>
>>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot______
> _________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>
>

Re: [mod-security-users] [Mod-security-developers] ModSecurity version 3.0.1 announcement

[Robert P. <rpa...@fe...>]

Hi,

> On Apr 4, 2018, at 02:12, Osama Elnaggar <oel...@gm...> wrote:

I think wrk is the best tool for now. ModSecurity is (and has been) a cpu-bounded workload. Benchmarks and profiling should focus on things like hot code path analysis, cache coherency, etc. unit/integration tests might make better use of more complex HTTP test harnesses, but I think in this case we can pretty much just throw a bunch of traffic in and burn the processor. 

Frankly, even with the 3.0.1 improvements, the performance deltas in nginx are discouraging to see. Tomorrow I will be doing some profiling of the sample C application to see how it compares, but at the moment I can't imagine that a provider at any meaningful scale would be okay deploying a WAF that can do at most a few thousand RPS with only a minimal ruleset. @Andrei do you have any comments about the specific numbers (RPS and latency) that have been noted here, and that your repo seems to corroborate?

Re: [mod-security-users] [Mod-security-developers] ModSecurity version 3.0.1 announcement

[Christian F. <chr...@ne...>]

Osama,

You mean ab is like a tool from the stone age? How dare you! :)

I'll investigate. Appreciated.

A word of caution, though: As long as we are talking of performance boost of 
several hundred percents between releases, I doubt that the benchmark tool
is of much concern.

Wish I had known / used wrk when I wrote the performance chapter for the
ModSecurity Handbook, though. I used a variety of tools there and they all
had their issues.

Ahoj,

Christian






On Wed, Apr 04, 2018 at 04:12:48AM -0500, Osama Elnaggar wrote:
> Some interesting ideas here.  I think using a single tool (+ a specific set
> of queries) for our benchmarks would be useful + be more standardized.
> 
> wrk is currently one of the most promising benchmarking tools and supports
> Lua plugins so it is a lot more flexible than ab.  I noticed that both
> Robert and Andrei used it (Christian: time to ditch ab :)).  I also
> recently used it recently (with the below script) to benchmark an API
> endpoint (ModSecurity wasn’t part of the solution so I don’t have any
> ModSecurity benchmarks using wrk).  It is a lot more flexible than ab.
> 
> You might find the multiplepaths.lua script / plugin useful.
>  multiplepaths.lua (https://github.com/timotta/wrk-scripts) is a Lua script
> that allows you to provide wrk with a file with different queries you want
> to perform in your benchmark so you can cover different areas such as OS
> command injection, SQL injection, XSS, etc.
> 
> Since it is written in Lua, we can probably extend it to provide additional
> customized payloads so they aren’t all in the GET request + send customized
> cookies, headers, etc.
> 
> Another option would be to write something up with Python + asyncio +
> requests although that will need a little more effort
> 
> -- 
> Osama Elnaggar
> 
> On April 4, 2018 at 6:46:57 PM, Andrei Belov (de...@ng...) wrote:
> 
> Hi folks,
> 
> > On 04 Apr 2018, at 07:48, Christian Folini <chr...@ne...>
> wrote:
> >
> > Hey Robert,
> >
> > On Tue, Apr 03, 2018 at 05:50:07PM -0700, Robert Paprocki wrote:
> >> Can you share the specifics of your evaluation? Performance in modsec +
> crs
> >> will vary greatly depending on the request payload. Soon I would like to
> do
> >> some before and after trace profiling of these releases to better
> illustrate
> >> how libmodsec performs in various conditions.
> >
> > I did a minimal self-compiled NGINX with a basic ModSecurity and CRS
> > as documented on https://www.netnea.com/cms/nginx-modsecurity-tutorials/
> .
> > (These new tutorials are in a draft state, the quality is not yet there.
> Use
> > with caution.)
> 
> [..]
> 
> > Felipe tagged a 3.0.2 yesterday and made it available at
> > https://github.com/SpiderLabs/ModSecurity/releases
> > I took that one for my tests. I reckon the performance is the same as
> with
> > the 3.0.1 that has been announced.
> >
> > This perf test is obviously very superficial. A thing to note is that
> even
> > testrun 2 would write the error-log (to gather statistical data).
> >
> > But whatever the specifics, I think this big performance boost will show
> in
> > any setup even if the factor might not be that high.
> >
> > Having real perf tests done regularly would be very welcome, Robert.
> 
> JFYI, I have created vagrant-based tools to run performance tests with
> nginx and libmodsecurity some time ago:
> 
> https://github.com/defanator/modsecurity-performance
> 
> It creates pre-configured environment suitable for wide range of
> investigations,
> related both to performance and functionality. I tried to include
> meaningful
> configurations, e.g.:
> 
> https://github.com/defanator/modsecurity-performance#what-is-being-tested
> 
> I think that environment could be [relatively easily] extended to support
> Apache + ModSec 2.x, in addition to nginx + ModSec 3.x, in order to
> simplify
> "direct" comparison and provide reproducible, statistically significant
> results.
> 
> (PRs are welcome of course.)
> 
> 
> -- 
> Andrei Belov
> Product Engineer
> NGINX
> ------------------------------------------------------------------------------
> 
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot

> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/


-- 
https://www.feistyduck.com/training/modsecurity-training-course
https://www.feistyduck.com/books/modsecurity-handbook/
mailto:chr...@ne...
twitter: @ChrFolini

Re: [mod-security-users] [Mod-security-developers] ModSecurity version 3.0.1 announcement

[Osama E. <oel...@gm...>]

Some interesting ideas here.  I think using a single tool (+ a specific set
of queries) for our benchmarks would be useful + be more standardized.

wrk is currently one of the most promising benchmarking tools and supports
Lua plugins so it is a lot more flexible than ab.  I noticed that both
Robert and Andrei used it (Christian: time to ditch ab :)).  I also
recently used it recently (with the below script) to benchmark an API
endpoint (ModSecurity wasn’t part of the solution so I don’t have any
ModSecurity benchmarks using wrk).  It is a lot more flexible than ab.

You might find the multiplepaths.lua script / plugin useful.
 multiplepaths.lua (https://github.com/timotta/wrk-scripts) is a Lua script
that allows you to provide wrk with a file with different queries you want
to perform in your benchmark so you can cover different areas such as OS
command injection, SQL injection, XSS, etc.

Since it is written in Lua, we can probably extend it to provide additional
customized payloads so they aren’t all in the GET request + send customized
cookies, headers, etc.

Another option would be to write something up with Python + asyncio +
requests although that will need a little more effort

-- 
Osama Elnaggar

On April 4, 2018 at 6:46:57 PM, Andrei Belov (de...@ng...) wrote:

Hi folks,

> On 04 Apr 2018, at 07:48, Christian Folini <chr...@ne...>
wrote:
>
> Hey Robert,
>
> On Tue, Apr 03, 2018 at 05:50:07PM -0700, Robert Paprocki wrote:
>> Can you share the specifics of your evaluation? Performance in modsec +
crs
>> will vary greatly depending on the request payload. Soon I would like to
do
>> some before and after trace profiling of these releases to better
illustrate
>> how libmodsec performs in various conditions.
>
> I did a minimal self-compiled NGINX with a basic ModSecurity and CRS
> as documented on https://www.netnea.com/cms/nginx-modsecurity-tutorials/
.
> (These new tutorials are in a draft state, the quality is not yet there.
Use
> with caution.)

[..]

> Felipe tagged a 3.0.2 yesterday and made it available at
> https://github.com/SpiderLabs/ModSecurity/releases
> I took that one for my tests. I reckon the performance is the same as
with
> the 3.0.1 that has been announced.
>
> This perf test is obviously very superficial. A thing to note is that
even
> testrun 2 would write the error-log (to gather statistical data).
>
> But whatever the specifics, I think this big performance boost will show
in
> any setup even if the factor might not be that high.
>
> Having real perf tests done regularly would be very welcome, Robert.

JFYI, I have created vagrant-based tools to run performance tests with
nginx and libmodsecurity some time ago:

https://github.com/defanator/modsecurity-performance

It creates pre-configured environment suitable for wide range of
investigations,
related both to performance and functionality. I tried to include
meaningful
configurations, e.g.:

https://github.com/defanator/modsecurity-performance#what-is-being-tested

I think that environment could be [relatively easily] extended to support
Apache + ModSec 2.x, in addition to nginx + ModSec 3.x, in order to
simplify
"direct" comparison and provide reproducible, statistically significant
results.

(PRs are welcome of course.)


-- 
Andrei Belov
Product Engineer
NGINX
------------------------------------------------------------------------------

Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
mod-security-users mailing list
mod...@li...
https://lists.sourceforge.net/lists/listinfo/mod-security-users
Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
http://www.modsecurity.org/projects/commercial/rules/
http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] [Mod-security-developers] ModSecurity version 3.0.1 announcement

[Christian F. <chr...@ne...>]

Hello Andrei,

On Wed, Apr 04, 2018 at 11:29:18AM +0300, Andrei Belov wrote:
> I think that environment could be [relatively easily] extended to support
> Apache + ModSec 2.x, in addition to nginx + ModSec 3.x, in order to simplify
> "direct" comparison and provide reproducible, statistically significant results.

Very cool. Thank you for sharing - and thanks for your contributions to
ModSecurity, namely 3.0.1.

The conceptual problem is see is that it's more than one variable here.
Apache/ModSec2 vs. NGINX/ModSec3. I'm an Apache person, but when I stripped
the two of Modsec and let the bare minimum installations serve static
files, NGINX blew me away.

So I kind of think that one would have to slow down NGINX to reach an Apache
level and then in a 2nd step add ModSec again to be able to measure ModSec2 vs
ModSec3.

What is your take on this?

Best,

Christian

-- 
The Universe is made of stories, not of atoms.
-- Muriel Rukeyser

Re: [mod-security-users] [Mod-security-developers] ModSecurity version 3.0.1 announcement

[Andrei B. <de...@ng...>]

Hi folks,

> On 04 Apr 2018, at 07:48, Christian Folini <chr...@ne...> wrote:
> 
> Hey Robert,
> 
> On Tue, Apr 03, 2018 at 05:50:07PM -0700, Robert Paprocki wrote:
>> Can you share the specifics of your evaluation? Performance in modsec + crs
>> will vary greatly depending on the request payload. Soon I would like to do
>> some before and after trace profiling of these releases to better illustrate
>> how libmodsec performs in various conditions.
> 
> I did a minimal self-compiled NGINX with a basic ModSecurity and CRS
> as documented on https://www.netnea.com/cms/nginx-modsecurity-tutorials/ .
> (These new tutorials are in a draft state, the quality is not yet there. Use
> with caution.)

[..]

> Felipe tagged a 3.0.2 yesterday and made it available at
> https://github.com/SpiderLabs/ModSecurity/releases
> I took that one for my tests. I reckon the performance is the same as with
> the 3.0.1 that has been announced.
> 
> This perf test is obviously very superficial. A thing to note is that even
> testrun 2 would write the error-log (to gather statistical data).
> 
> But whatever the specifics, I think this big performance boost will show in
> any setup even if the factor might not be that high.
> 
> Having real perf tests done regularly would be very welcome, Robert.

JFYI, I have created vagrant-based tools to run performance tests with
nginx and libmodsecurity some time ago:

    https://github.com/defanator/modsecurity-performance

It creates pre-configured environment suitable for wide range of investigations,
related both to performance and functionality. I tried to include meaningful
configurations, e.g.:

    https://github.com/defanator/modsecurity-performance#what-is-being-tested

I think that environment could be [relatively easily] extended to support
Apache + ModSec 2.x, in addition to nginx + ModSec 3.x, in order to simplify
"direct" comparison and provide reproducible, statistically significant results.

(PRs are welcome of course.)


--
Andrei Belov
Product Engineer
NGINX

Re: [mod-security-users] [Mod-security-developers] ModSecurity version 3.0.1 announcement

[Robert P. <rpa...@fe...>]

 Whups, sorry for the x-post.

Christian, thanks for the numbers.

I made a very very quick test with libmodsec 3.0.2
(commit 8d0f51beda5c031e38741c27f29b67f0266352bb) and Nginx 1.13.9, built
against Modsecurity-nginx/master (commit 995f631767c24de8fabf82
8b8f44d27b316d1395).

Testing with wrk against a vanilla Nginx config running a single worker:

poprocks@mini-vm:~/nginx$ wrk -c 5 -d 5 -t 5 'http://localhost:8080/index.h
tml?exec=/bin/bash'
Running 5s test @ http://localhost:8080/index.html?exec=/bin/bash
  5 threads and 5 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   291.81us  665.96us  19.62ms   99.59%
    Req/Sec     3.89k   335.52     6.64k    92.89%
  97899 requests in 5.10s, 79.35MB read
Requests/sec:  19196.82
Transfer/sec:     15.56MB

Adding the following configs in the server block:

 modsecurity_rules_file /home/poprocks/src/ModSecurity/modsecurity.conf;
 modsecurity_rules_file /home/poprocks/src/owasp-modse
curity-crs/crs-setup.conf;

Where these are vanilla configs with the following notables:

SecAuditEngine Off
Include /home/poprocks/src/owasp-modsecurity-crs/rules/*.conf
[rule 910100 commented out b/c lack of GeoIP support]

RPS is... substantially different:

poprocks@mini-vm:~/nginx$ wrk -c 5 -d 5 -t 5 'http://localhost:8080/index.h
tml?exec=/bin/bash'
Running 5s test @ http://localhost:8080/index.html?exec=/bin/bash
  5 threads and 5 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    14.92ms    1.39ms  36.23ms   89.95%
    Req/Sec    67.02      5.03    90.00     70.80%
  1680 requests in 5.03s, 531.49KB read
  Non-2xx or 3xx responses: 1680
Requests/sec:    334.25
Transfer/sec:    105.74KB

Of course, this short-circuits request processing. A request pattern that
is not dropped by the CRS sees worse performance:

poprocks@mini-vm:~/nginx$ wrk -c 5 -d 5 -t 5 '
http://localhost:8080/index.html?test=foo'
Running 5s test @ http://localhost:8080/index.html?test=foo
  5 threads and 5 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    21.41ms    2.52ms  51.67ms   81.28%
    Req/Sec    46.60      5.73    70.00     60.80%
  1169 requests in 5.02s, 0.95MB read
Requests/sec:    232.65
Transfer/sec:    193.11KB

And audit log entries do indeed look correct, when enabled:

---E0qxI3Fx---A--
[03/Apr/2018:21:23:46 -0700] 152281582640.577439 127.0.0.1 48956 127.0.0.1
8080
---E0qxI3Fx---B--
GET /index.html?exec=/bin/bash HTTP/1.1
Host: localhost:8080

---E0qxI3Fx---D--

---E0qxI3Fx---F--
HTTP/1.1 403
Server: nginx/1.13.9
Date: Wed, 04 Apr 2018 04:23:46 GMT
Content-Length: 169
Content-Type: text/html
Connection: keep-alive

---E0qxI3Fx---H--
ModSecurity: Access denied with code 403 (phase 2). Matched "Operator
`PmFromFile' with parameter `unix-shell.data' against variable `ARGS:exec'
(Value: `/bin/bash' ) [file "/home/poprocks/src/owasp-mods
ecurity-crs/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf"] [line "404"]
[id "932160"] [rev "1"] [msg "Remote Command Execution: Unix Shell Code
Found"] [data "Matched Data: bin/bash found within ARGS:exec: /bin/bash"]
[severity "2"] [ver "OWASP_CRS/3.0.0"] [maturity "1"] [accuracy "8"] [tag
"application-multi"] [tag "language-shell"] [tag "platform-unix"] [tag
"attack-rce"] [tag "OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION"] [tag
"WASCTC/WASC-31"] [tag "OWASP_TOP_10/A1"] [tag "PCI/6.5.2"] [hostname
"127.0.0.1"] [uri "/index.html"] [unique_id "152281582640.577439"] [ref
"o1,8v21,9t:urlDecodeUni,t:cmdLine,t:normalizePath,t:lowercase"]

---E0qxI3Fx---I--

---E0qxI3Fx---J--

---E0qxI3Fx---Z--


So I suspected something was wrong with my setup, but after looking at
Christian's numbers I'm perhaps thinking otherwise. I also made a quick
test with libmodsecurity 3.0.0, and saw the same order of magnitude drop
that Christian reported.

I also tested removing the CRS entirely and mocking against a single
SecRule:

poprocks@mini-vm:~$ tail -1 /home/poprocks/src/ModSecurity/modsecurity.conf
SecRules ARGS:test "@streq foo" "id:12345,phase:1,deny"
poprocks@mini-vm:~/nginx$ wrk -c 5 -d 5 -t 5 'http://localhost:8080/index.
html?test=fo'
Running 5s test @ http://localhost:8080/index.html?test=fo
  5 threads and 5 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.18ms  286.19us   9.79ms   89.05%
    Req/Sec     0.86k    28.30     0.93k    70.40%
  21322 requests in 5.01s, 17.28MB read
Requests/sec:   4258.45
Transfer/sec:      3.45MB
poprocks@mini-vm:~/nginx$ wrk -c 5 -d 5 -t 5 'http://localhost:8080/index.
html?test=foo'
Running 5s test @ http://localhost:8080/index.html?test=foo
  5 threads and 5 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.03ms  192.30us   2.88ms   81.58%
    Req/Sec     0.97k    54.99     1.08k    67.20%
  24130 requests in 5.00s, 7.45MB read
  Non-2xx or 3xx responses: 24130
Requests/sec:   4822.16
Transfer/sec:      1.49MB


So the CRS clearly has a substantial impact on RPS, but the mere presence
of Nginx and a single rule (with an inexpensive operator and no
transformations) still results in a drop in performance of an order of
magnitude, and *another* order of magnitude with the full CRS in play. A
more comprehensive set of tests with a meaningful, real-world corpus of
data is definitely on my radar at this point. Perhaps good material for a
blog post ;)

BTW, I made a userspace flamegraph of the running Nginx worker process
while under wrk load with ModSecurity enabled, running the CRS:
https://s3.amazonaws.com/p0pr0ck5-data/ngx-modsec.svg


On Tue, Apr 3, 2018 at 9:48 PM, Christian Folini <christian.folini@
netnea.com> wrote:

> Hey Robert,
>
> On Tue, Apr 03, 2018 at 05:50:07PM -0700, Robert Paprocki wrote:
> > Can you share the specifics of your evaluation? Performance in modsec +
> crs
> > will vary greatly depending on the request payload. Soon I would like to
> do
> > some before and after trace profiling of these releases to better
> illustrate
> > how libmodsec performs in various conditions.
>
> I did a minimal self-compiled NGINX with a basic ModSecurity and CRS
> as documented on https://www.netnea.com/cms/nginx-modsecurity-tutorials/ .
> (These new tutorials are in a draft state, the quality is not yet there.
> Use
> with caution.)
>
> Testrun 1:
> ----------
>
> /apache/bin/ab -n 1000 -c 1 "http://localhost/index.html?test=/etc/passwd"
>
> 3.0.0
>         Concurrency Level:      1
>         Time taken for tests:   26.172 seconds
>         Complete requests:      1000
>         Failed requests:        0
>         Non-2xx responses:      1000
>         Total transferred:      320000 bytes
>         HTML transferred:       162000 bytes
>         Requests per second:    38.21 [#/sec] (mean)
>         Time per request:       26.172 [ms] (mean)
>         Time per request:       26.172 [ms] (mean, across all concurrent
> reqs)
>         Transfer rate:          11.94 [Kbytes/sec] received
>
> 3.0.2
>         Concurrency Level:      1
>         Time taken for tests:   4.585 seconds
>         Complete requests:      1000
>         Failed requests:        0
>         Non-2xx responses:      1000
>         Total transferred:      320000 bytes
>         HTML transferred:       162000 bytes
>         Requests per second:    218.12 [#/sec] (mean)
>         Time per request:       4.585 [ms] (mean)
>         Time per request:       4.585 [ms] (mean, across all concurrent
> reqs)
>         Transfer rate:          68.16 [Kbytes/sec] received
>
>
> Testrun 2:
> ----------
>
> /apache/bin/ab -n 1000 -c 1 "http://localhost/index.html?test=innocent"
>
> 3.0.0
>         Concurrency Level:      1
>         Time taken for tests:   26.168 seconds
>         Complete requests:      1000
>         Failed requests:        0
>         Total transferred:      853000 bytes
>         HTML transferred:       612000 bytes
>         Requests per second:    38.21 [#/sec] (mean)
>         Time per request:       26.168 [ms] (mean)
>         Time per request:       26.168 [ms] (mean, across all concurrent
> reqs)
>         Transfer rate:          31.83 [Kbytes/sec] received
>
>
> 3.0.2
>         Concurrency Level:      1
>         Time taken for tests:   3.996 seconds
>         Complete requests:      1000
>         Failed requests:        0
>         Total transferred:      853000 bytes
>         HTML transferred:       612000 bytes
>         Requests per second:    250.25 [#/sec] (mean)
>         Time per request:       3.996 [ms] (mean)
>         Time per request:       3.996 [ms] (mean, across all concurrent
> reqs)
>         Transfer rate:          208.46 [Kbytes/sec] received
>
> Felipe tagged a 3.0.2 yesterday and made it available at
> https://github.com/SpiderLabs/ModSecurity/releases
> I took that one for my tests. I reckon the performance is the same as with
> the 3.0.1 that has been announced.
>
> This perf test is obviously very superficial. A thing to note is that even
> testrun 2 would write the error-log (to gather statistical data).
>
> But whatever the specifics, I think this big performance boost will show in
> any setup even if the factor might not be that high.
>
> Having real perf tests done regularly would be very welcome, Robert.
>
> Best,
>
> Christian
>
>
> --
> I don't believe that we have come to the end of the democratic experiment.
> -- Bruce Schneier
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>



On Tue, Apr 3, 2018 at 9:48 PM, Christian Folini <
chr...@ne...> wrote:

> Hey Robert,
>
> On Tue, Apr 03, 2018 at 05:50:07PM -0700, Robert Paprocki wrote:
> > Can you share the specifics of your evaluation? Performance in modsec +
> crs
> > will vary greatly depending on the request payload. Soon I would like to
> do
> > some before and after trace profiling of these releases to better
> illustrate
> > how libmodsec performs in various conditions.
>
> I did a minimal self-compiled NGINX with a basic ModSecurity and CRS
> as documented on https://www.netnea.com/cms/nginx-modsecurity-tutorials/ .
> (These new tutorials are in a draft state, the quality is not yet there.
> Use
> with caution.)
>
> Testrun 1:
> ----------
>
> /apache/bin/ab -n 1000 -c 1 "http://localhost/index.html?test=/etc/passwd"
>
> 3.0.0
>         Concurrency Level:      1
>         Time taken for tests:   26.172 seconds
>         Complete requests:      1000
>         Failed requests:        0
>         Non-2xx responses:      1000
>         Total transferred:      320000 bytes
>         HTML transferred:       162000 bytes
>         Requests per second:    38.21 [#/sec] (mean)
>         Time per request:       26.172 [ms] (mean)
>         Time per request:       26.172 [ms] (mean, across all concurrent
> reqs)
>         Transfer rate:          11.94 [Kbytes/sec] received
>
> 3.0.2
>         Concurrency Level:      1
>         Time taken for tests:   4.585 seconds
>         Complete requests:      1000
>         Failed requests:        0
>         Non-2xx responses:      1000
>         Total transferred:      320000 bytes
>         HTML transferred:       162000 bytes
>         Requests per second:    218.12 [#/sec] (mean)
>         Time per request:       4.585 [ms] (mean)
>         Time per request:       4.585 [ms] (mean, across all concurrent
> reqs)
>         Transfer rate:          68.16 [Kbytes/sec] received
>
>
> Testrun 2:
> ----------
>
> /apache/bin/ab -n 1000 -c 1 "http://localhost/index.html?test=innocent"
>
> 3.0.0
>         Concurrency Level:      1
>         Time taken for tests:   26.168 seconds
>         Complete requests:      1000
>         Failed requests:        0
>         Total transferred:      853000 bytes
>         HTML transferred:       612000 bytes
>         Requests per second:    38.21 [#/sec] (mean)
>         Time per request:       26.168 [ms] (mean)
>         Time per request:       26.168 [ms] (mean, across all concurrent
> reqs)
>         Transfer rate:          31.83 [Kbytes/sec] received
>
>
> 3.0.2
>         Concurrency Level:      1
>         Time taken for tests:   3.996 seconds
>         Complete requests:      1000
>         Failed requests:        0
>         Total transferred:      853000 bytes
>         HTML transferred:       612000 bytes
>         Requests per second:    250.25 [#/sec] (mean)
>         Time per request:       3.996 [ms] (mean)
>         Time per request:       3.996 [ms] (mean, across all concurrent
> reqs)
>         Transfer rate:          208.46 [Kbytes/sec] received
>
> Felipe tagged a 3.0.2 yesterday and made it available at
> https://github.com/SpiderLabs/ModSecurity/releases
> I took that one for my tests. I reckon the performance is the same as with
> the 3.0.1 that has been announced.
>
> This perf test is obviously very superficial. A thing to note is that even
> testrun 2 would write the error-log (to gather statistical data).
>
> But whatever the specifics, I think this big performance boost will show in
> any setup even if the factor might not be that high.
>
> Having real perf tests done regularly would be very welcome, Robert.
>
> Best,
>
> Christian
>
>
> --
> I don't believe that we have come to the end of the democratic experiment.
> -- Bruce Schneier
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>


On Tue, Apr 3, 2018 at 9:48 PM, Christian Folini <
chr...@ne...> wrote:

> Hey Robert,
>
> On Tue, Apr 03, 2018 at 05:50:07PM -0700, Robert Paprocki wrote:
> > Can you share the specifics of your evaluation? Performance in modsec +
> crs
> > will vary greatly depending on the request payload. Soon I would like to
> do
> > some before and after trace profiling of these releases to better
> illustrate
> > how libmodsec performs in various conditions.
>
> I did a minimal self-compiled NGINX with a basic ModSecurity and CRS
> as documented on https://www.netnea.com/cms/nginx-modsecurity-tutorials/ .
> (These new tutorials are in a draft state, the quality is not yet there.
> Use
> with caution.)
>
> Testrun 1:
> ----------
>
> /apache/bin/ab -n 1000 -c 1 "http://localhost/index.html?test=/etc/passwd"
>
> 3.0.0
>         Concurrency Level:      1
>         Time taken for tests:   26.172 seconds
>         Complete requests:      1000
>         Failed requests:        0
>         Non-2xx responses:      1000
>         Total transferred:      320000 bytes
>         HTML transferred:       162000 bytes
>         Requests per second:    38.21 [#/sec] (mean)
>         Time per request:       26.172 [ms] (mean)
>         Time per request:       26.172 [ms] (mean, across all concurrent
> reqs)
>         Transfer rate:          11.94 [Kbytes/sec] received
>
> 3.0.2
>         Concurrency Level:      1
>         Time taken for tests:   4.585 seconds
>         Complete requests:      1000
>         Failed requests:        0
>         Non-2xx responses:      1000
>         Total transferred:      320000 bytes
>         HTML transferred:       162000 bytes
>         Requests per second:    218.12 [#/sec] (mean)
>         Time per request:       4.585 [ms] (mean)
>         Time per request:       4.585 [ms] (mean, across all concurrent
> reqs)
>         Transfer rate:          68.16 [Kbytes/sec] received
>
>
> Testrun 2:
> ----------
>
> /apache/bin/ab -n 1000 -c 1 "http://localhost/index.html?test=innocent"
>
> 3.0.0
>         Concurrency Level:      1
>         Time taken for tests:   26.168 seconds
>         Complete requests:      1000
>         Failed requests:        0
>         Total transferred:      853000 bytes
>         HTML transferred:       612000 bytes
>         Requests per second:    38.21 [#/sec] (mean)
>         Time per request:       26.168 [ms] (mean)
>         Time per request:       26.168 [ms] (mean, across all concurrent
> reqs)
>         Transfer rate:          31.83 [Kbytes/sec] received
>
>
> 3.0.2
>         Concurrency Level:      1
>         Time taken for tests:   3.996 seconds
>         Complete requests:      1000
>         Failed requests:        0
>         Total transferred:      853000 bytes
>         HTML transferred:       612000 bytes
>         Requests per second:    250.25 [#/sec] (mean)
>         Time per request:       3.996 [ms] (mean)
>         Time per request:       3.996 [ms] (mean, across all concurrent
> reqs)
>         Transfer rate:          208.46 [Kbytes/sec] received
>
> Felipe tagged a 3.0.2 yesterday and made it available at
> https://github.com/SpiderLabs/ModSecurity/releases
> I took that one for my tests. I reckon the performance is the same as with
> the 3.0.1 that has been announced.
>
> This perf test is obviously very superficial. A thing to note is that even
> testrun 2 would write the error-log (to gather statistical data).
>
> But whatever the specifics, I think this big performance boost will show in
> any setup even if the factor might not be that high.
>
> Having real perf tests done regularly would be very welcome, Robert.
>
> Best,
>
> Christian
>
>
> --
> I don't believe that we have come to the end of the democratic experiment.
> -- Bruce Schneier
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>

Re: [mod-security-users] [Mod-security-developers] ModSecurity version 3.0.1 announcement

[Christian F. <chr...@ne...>]

Hey Robert,

On Tue, Apr 03, 2018 at 05:50:07PM -0700, Robert Paprocki wrote:
> Can you share the specifics of your evaluation? Performance in modsec + crs
> will vary greatly depending on the request payload. Soon I would like to do
> some before and after trace profiling of these releases to better illustrate
> how libmodsec performs in various conditions.

I did a minimal self-compiled NGINX with a basic ModSecurity and CRS
as documented on https://www.netnea.com/cms/nginx-modsecurity-tutorials/ .
(These new tutorials are in a draft state, the quality is not yet there. Use
with caution.)

Testrun 1:
----------

/apache/bin/ab -n 1000 -c 1 "http://localhost/index.html?test=/etc/passwd"

3.0.0
	Concurrency Level:      1
	Time taken for tests:   26.172 seconds
	Complete requests:      1000
	Failed requests:        0
	Non-2xx responses:      1000
	Total transferred:      320000 bytes
	HTML transferred:       162000 bytes
	Requests per second:    38.21 [#/sec] (mean)
	Time per request:       26.172 [ms] (mean)
	Time per request:       26.172 [ms] (mean, across all concurrent reqs)
	Transfer rate:          11.94 [Kbytes/sec] received

3.0.2
	Concurrency Level:      1
	Time taken for tests:   4.585 seconds
	Complete requests:      1000
	Failed requests:        0
	Non-2xx responses:      1000
	Total transferred:      320000 bytes
	HTML transferred:       162000 bytes
	Requests per second:    218.12 [#/sec] (mean)
	Time per request:       4.585 [ms] (mean)
	Time per request:       4.585 [ms] (mean, across all concurrent reqs)
	Transfer rate:          68.16 [Kbytes/sec] received


Testrun 2:
----------

/apache/bin/ab -n 1000 -c 1 "http://localhost/index.html?test=innocent"

3.0.0
	Concurrency Level:      1
	Time taken for tests:   26.168 seconds
	Complete requests:      1000
	Failed requests:        0
	Total transferred:      853000 bytes
	HTML transferred:       612000 bytes
	Requests per second:    38.21 [#/sec] (mean)
	Time per request:       26.168 [ms] (mean)
	Time per request:       26.168 [ms] (mean, across all concurrent reqs)
	Transfer rate:          31.83 [Kbytes/sec] received
				

3.0.2
	Concurrency Level:      1
	Time taken for tests:   3.996 seconds
	Complete requests:      1000
	Failed requests:        0
	Total transferred:      853000 bytes
	HTML transferred:       612000 bytes
	Requests per second:    250.25 [#/sec] (mean)
	Time per request:       3.996 [ms] (mean)
	Time per request:       3.996 [ms] (mean, across all concurrent reqs)
	Transfer rate:          208.46 [Kbytes/sec] received

Felipe tagged a 3.0.2 yesterday and made it available at
https://github.com/SpiderLabs/ModSecurity/releases
I took that one for my tests. I reckon the performance is the same as with
the 3.0.1 that has been announced.

This perf test is obviously very superficial. A thing to note is that even
testrun 2 would write the error-log (to gather statistical data).

But whatever the specifics, I think this big performance boost will show in
any setup even if the factor might not be that high.

Having real perf tests done regularly would be very welcome, Robert.

Best,

Christian


-- 
I don't believe that we have come to the end of the democratic experiment.
-- Bruce Schneier

Re: [mod-security-users] ModSecurity version 3.0.1 announcement

[Robert P. <rpa...@fe...>]

Christian, 

> On Apr 3, 2018, at 13:18, Christian Folini <chr...@ne...> wrote:
> 
> Congratulations on this release Felipe.
> 
> I confirm the great improvement in terms of requests per second. A brief
> test against nginx/modsecurity running on localhost brought me a speedup of
> factor 5 with a CRS 3.0.2 default installation.

Can you share the specifics of your evaluation? Performance in modsec + crs will vary greatly depending on the request payload. Soon I would like to do some before and after trace profiling of these releases to better illustrate how libmodsec performs in various conditions.

Re: [mod-security-users] ModSecurity version 3.0.1 announcement

[Christian F. <chr...@ne...>]

Congratulations on this release Felipe.

I confirm the great improvement in terms of requests per second. A brief
test against nginx/modsecurity running on localhost brought me a speedup of
factor 5 with a CRS 3.0.2 default installation.

The changelog mentions three performance improvements. Which one had this
dramatic effect? 

And finally: Apache/ModSec 2.9.2 still trumps Nginx/ModSec 3.0.1 big time in
my lab setup. What is your projection for future performance improvements on 
the new 3.0 release line? When will this be ready to replace existing
installations with a similar performance?

Cheers,

Christian



On Mon, Apr 02, 2018 at 12:30:07PM +0000, Felipe Costa wrote:
> 
> 
> It is a pleasure to announce the release of ModSecurity version 3.0.1
> (libModSecurity). This version contains improvements, fixes and new features.
> 
> The most important new feature is the support for the libMaxMinddb,
> popularly kown as the new version of the GeoIP library. 
> 
> There is a splendid performance upgrade on v3.0.1. A significant amount of
> work was placed on how to handle the memory usage more efficiently, which
> leaded to great improvements in terms of latency and requests per second.
> 
> The list with the full changes can be found on the project CHANGES
> file, available here:
> - https://github.com/SpiderLabs/ModSecurity/releases/tag/v3.0.1/CHANGES
> 
> The list of open issues is available on GitHub:
> - https://github.com/SpiderLabs/ModSecurity/labels/3.x
> 
> Thanks to everybody who helped in this process: reporting issues, making
> comments and suggestions, sending patches and so on.
> 
> Further details on the compilation process for ModSecurity v3, can be found on
> the project README:
>  - https://github.com/SpiderLabs/ModSecurity/tree/v3/master#compilation
> 
> Complementary documentation for the connectors are available here:
>  - nginx: https://github.com/SpiderLabs/ModSecurity-nginx/#compilation
>  - Apache: https://github.com/SpiderLabs/ModSecurity-apache/#compilation
> 
> 
> IMPORTANT: ModSecurity version 2 will be available and maintained parallel
> to version 3. There is no ETA to deprecate the version 2.x. New features
> and major improvements will be implemented on version 3.x. Security or major
> bugs are planned to be back ported. Version 2 and version 3 has a completely
> independent development/release cycle.
> 
> 
> Br.,
> Felipe “Zimmerle” Costa
> Security Researcher, Lead Developer ModSecurity.
> 
> Trustwave | SMART SECURITY ON DEMAND
> www.trustwave.com
> 
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

-- 
https://www.feistyduck.com/training/modsecurity-training-course
https://www.feistyduck.com/books/modsecurity-handbook/
mailto:chr...@ne...
twitter: @ChrFolini

Re: [mod-security-users] libmodsecurity3.0.1 shared object version number regression?

[Gregory L. <gr...@cl...>]

Hi Felipe,

Thank you for the fix. My GitHub login is gl1f1v21.

Gregory


On Mon, Apr 2, 2018 at 9:57 PM, Felipe Zimmerle <fe...@zi...> wrote:

> Hi Gregory,
>
> Thanks for the report. Fixed here:
> https://github.com/SpiderLabs/ModSecurity/commit/
> 2e87c4e751b191619c0de68e3f0567bdac58ed16
>
>
> The problem was here:
> https://github.com/SpiderLabs/ModSecurity/commit/
> 95cb4c56ab90ec19fea4a7beea8ed070e1e61e23#diff-
> 67e997bcfdac55191033d57a16d1408aR14
>
> Can you share your GitHub info, so I can update the CHANGES properly?
>
> Br.,
> Felipe.
>
>
> On Mon, Apr 2, 2018 at 11:52 PM Felipe Zimmerle <fe...@zi...>
> wrote:
>
>> Hi Gregory,
>>
>> The correct object is the one out of compilation process, which indeed
>> seems to have the wrong version. I will investigate. In the meanwhile it
>> will be very good if you manage to report this on GitHub.
>>
>> Br.,
>> F.
>>
>> On Mon, Apr 2, 2018 at 11:04 PM Gregory LeFevre <gr...@cl...>
>> wrote:
>>
>>>
>>> Hi,
>>>
>>> I am compiling libmodsecurity3.0.1 from https://github.com/SpiderLabs/
>>> ModSecurity/releases/download/v3.0.1/modsecurity-v3.0.1.tar.gz on
>>> CentOS7 using:
>>>
>>>
>>> $ ./build.sh
>>> $ ./configure
>>> $ make
>>>
>>>
>>> I see libmodsecurity.so.*2.1.0* afterward in the build artifacts:
>>>
>>>
>>> [modsecurity-v3.0.1]$ ls -l src/.libs
>>> total 172892
>>> -rw-rw-r-- 1 build build 127289914 Apr  3 00:51 libmodsecurity.a
>>> lrwxrwxrwx 1 build build        20 Apr  3 00:51 libmodsecurity.la -> ../
>>> libmodsecurity.la
>>> -rw-rw-r-- 1 build build    819584 Apr  3 00:48
>>> libmodsecurity_la-anchored_set_variable.o
>>> -rw-rw-r-- 1 build build    601432 Apr  3 00:48
>>> libmodsecurity_la-anchored_variable.o
>>> -rw-rw-r-- 1 build build      1050 Apr  3 00:51 libmodsecurity.lai
>>> -rw-rw-r-- 1 build build    742312 Apr  3 00:48
>>> libmodsecurity_la-modsecurity.o
>>> -rw-rw-r-- 1 build build   1027624 Apr  3 00:48 libmodsecurity_la-rule_
>>> message.o
>>> -rw-rw-r-- 1 build build   2442416 Apr  3 00:48 libmodsecurity_la-rule.o
>>> -rw-rw-r-- 1 build build    936496 Apr  3 00:48
>>> libmodsecurity_la-rule_script.o
>>> -rw-rw-r-- 1 build build   1205592 Apr  3 00:48 libmodsecurity_la-rules_
>>> exceptions.o
>>> -rw-rw-r-- 1 build build   1413432 Apr  3 00:48 libmodsecurity_la-rules.o
>>> -rw-rw-r-- 1 build build    814008 Apr  3 00:48
>>> libmodsecurity_la-run_time_string.o
>>> -rw-rw-r-- 1 build build   2373024 Apr  3 00:48
>>> libmodsecurity_la-transaction.o
>>> -rw-rw-r-- 1 build build    126280 Apr  3 00:48
>>> libmodsecurity_la-unique_id.o
>>> lrwxrwxrwx 1 build build        23 Apr  3 00:51 libmodsecurity.so ->
>>> libmodsecurity.so.2.1.0
>>> lrwxrwxrwx 1 build build        23 Apr  3 00:51 libmodsecurity.so.2 ->
>>> libmodsecurity.so.2.1.0
>>> -rwxrwxr-x 1 build build  37216744 Apr  3 00:51 libmodsecurity.so.2.1.0
>>>
>>>
>>> and a previous libmodsecurity3.0.0 build (on the same machine, using
>>> the same build steps) has libmodsecurity.so.3.0.0:
>>>
>>>
>>> [modsecurity-v3.0.0]$ ls -l src/.libs
>>> total 150292
>>> -rw-rw-r-- 1 build build 108625334 Apr  3 00:45 libmodsecurity.a
>>> lrwxrwxrwx 1 build build        20 Apr  3 00:45 libmodsecurity.la -> ../
>>> libmodsecurity.la
>>> -rw-rw-r-- 1 build build    754752 Apr  3 00:42
>>> libmodsecurity_la-anchored_set_variable.o
>>> -rw-rw-r-- 1 build build    572888 Apr  3 00:42
>>> libmodsecurity_la-anchored_variable.o
>>> -rw-rw-r-- 1 build build      1050 Apr  3 00:45 libmodsecurity.lai
>>> -rw-rw-r-- 1 build build   1133712 Apr  3 00:42 libmodsecurity_la-macro_
>>> expansion.o
>>> -rw-rw-r-- 1 build build    753320 Apr  3 00:42
>>> libmodsecurity_la-modsecurity.o
>>> -rw-rw-r-- 1 build build   1074248 Apr  3 00:42 libmodsecurity_la-rule_
>>> message.o
>>> -rw-rw-r-- 1 build build   2713344 Apr  3 00:42 libmodsecurity_la-rule.o
>>> -rw-rw-r-- 1 build build    927592 Apr  3 00:42
>>> libmodsecurity_la-rule_script.o
>>> -rw-rw-r-- 1 build build   1171128 Apr  3 00:42 libmodsecurity_la-rules_
>>> exceptions.o
>>> -rw-rw-r-- 1 build build   1364000 Apr  3 00:42 libmodsecurity_la-rules.o
>>> -rw-rw-r-- 1 build build   2442848 Apr  3 00:42
>>> libmodsecurity_la-transaction.o
>>> -rw-rw-r-- 1 build build    126280 Apr  3 00:42
>>> libmodsecurity_la-unique_id.o
>>> lrwxrwxrwx 1 build build        23 Apr  3 00:45 libmodsecurity.so ->
>>> libmodsecurity.so.3.0.0
>>> lrwxrwxrwx 1 build build        23 Apr  3 00:45 libmodsecurity.so.3 ->
>>> libmodsecurity.so.3.0.0
>>> -rwxrwxr-x 1 build build  32211128 Apr  3 00:45 libmodsecurity.so.3.0.0
>>>
>>>
>>> Does anyone else see this? How can I be certain that I am getting the
>>> correct shared object?
>>>
>>> Thank you,
>>>
>>> Gregory
>>>
>>> ------------------------------------------------------------
>>> ------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot______
>>> _________________________________________
>>> mod-security-users mailing list
>>> mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> http://www.modsecurity.org/projects/commercial/rules/
>>> http://www.modsecurity.org/projects/commercial/support/
>>>
>>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>
>