|
From: Robert P. <rpa...@fe...> - 2018-04-04 17:44:20
|
Hi, On Wed, Apr 4, 2018 at 5:03 AM, Christian Folini < chr...@ne...> wrote: > > > > However, I have limited knowledge on the following: > > - is ModSec 3.x has been ever targeted to support CRS < 3, > > See above. > It would be great to here an official stance from the development team on this. @Felipe can you comment? > > Also I'm not sure whether ModSec 2.x has its own benchmarks (not related > to any connector). > > If it does, then perhaps it would be good to compare "generic" ModSec 2.x > > vs "generic" ModSec 3.x as well. > > Yes, that would be cool. But from what I understand, ModSec 2.9.x is deeply > integrated into the webserver. > Yeah, this is a bit of a pain to test. I modified one of the example programs that comes with the v3/master ModSecurity source code as follows: https://gist.github.com/p0pr0ck5/9b2c414641c9b03d527679d0c8cb7d86 Note that this doesn't add any headers or body data (request or response) to the transaction, and the included "basic_rules.conf" is unchanged from what's in the example repo (and ignore the memory leak in not cleaning up the transaction). So running this very light example: $ ./test Rules: Phase: 0 (0 rules) Phase: 1 (0 rules) Phase: 2 (2 rules) Rule ID: 200000--0x1e2b6c0 Rule ID: 200001--0x1e2bc90 Phase: 3 (4 rules) Rule ID: 200002--0x1e2c720 Rule ID: 200003--0x1e2ea00 Rule ID: 200004--0x1e2f200 Rule ID: 200005--0x1e2fc10 Phase: 4 (0 rules) Phase: 5 (0 rules) Phase: 6 (0 rules) Phase: 7 (0 rules) Did 9907 Done! We see around 10k processes per second. Essentially all of our time is spent waiting on memory allocations: https://s3.amazonaws.com/p0pr0ck5-data/modsec-simple.svg Now consider the case where we include the full 3.0.0 CRS: $ ./test Rules: Phase: 0 (32 rules) Rule ID: 0--0x22ca500 Rule ID: 0--0x22d9480 Rule ID: 0--0x22f9530 Rule ID: 0--0x22f9f20 [...snip several hundred lines...] Did 319 Done! So this is about the same throughput we saw from Nginx + libmodsec integration. A flamegraph also highlights hot paths, particularly in Rule::evaluate (also largely spent on allocation wait time at this point, since each evaluation instantiates many new objects): https://s3.amazonaws.com/p0pr0ck5-data/modsec-simple-crs.svg (I also performed the same tests on 3.0.0; flamegraphs are also in this s3 bucket but I will avoid commentary here for now as this thread is gotten long in the tooth as it is). A few initial takeaways: - There is a clearly definable minimum overhead needed to execute libmodsecurity, based on its current architecture - I noted the same memory leak that defanator noted in #1729. I have yet to apply the patches noted. - HTTP integrations will induce some overhead, which may or may not be substantial. With smart memory pooling and good design, though, this should be kept to a minimum. - Leveraging the full CRS completely tanks throughput. It would take a while more to dig into specifics but I suspect that Rule::evaluate probably needs an overhaul. - I ran one more test again a mock set of dumb rules (generated via https://gist.github.com/p0pr0ck5/c99bca54734af7546d910db8d7c97ab3). Saw about 1000 transactions processed per second, indicating (as should be assumed) that linear growth in the size of the included ruleset results in similar performance reduction. Flamegraphs again point to Rule::evaluate, with Rule::getFinalVars taking up half those traces. I suspect there could be some smarter behaviors about avoiding object creation if its unneeded on these hot paths. Hopefully this is of use to some folks. I'd be interested in writing some automated scaffolding to execute this type of testing against every commit, to maintain a continuous audit trail of performance impacts; if this sounds interesting to the maintainers I'd be happy to chat further. |
