|
From: Robert P. <rpa...@fe...> - 2018-04-04 05:12:14
|
Whups, sorry for the x-post.
Christian, thanks for the numbers.
I made a very very quick test with libmodsec 3.0.2
(commit 8d0f51beda5c031e38741c27f29b67f0266352bb) and Nginx 1.13.9, built
against Modsecurity-nginx/master (commit 995f631767c24de8fabf82
8b8f44d27b316d1395).
Testing with wrk against a vanilla Nginx config running a single worker:
poprocks@mini-vm:~/nginx$ wrk -c 5 -d 5 -t 5 'http://localhost:8080/index.h
tml?exec=/bin/bash'
Running 5s test @ http://localhost:8080/index.html?exec=/bin/bash
5 threads and 5 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 291.81us 665.96us 19.62ms 99.59%
Req/Sec 3.89k 335.52 6.64k 92.89%
97899 requests in 5.10s, 79.35MB read
Requests/sec: 19196.82
Transfer/sec: 15.56MB
Adding the following configs in the server block:
modsecurity_rules_file /home/poprocks/src/ModSecurity/modsecurity.conf;
modsecurity_rules_file /home/poprocks/src/owasp-modse
curity-crs/crs-setup.conf;
Where these are vanilla configs with the following notables:
SecAuditEngine Off
Include /home/poprocks/src/owasp-modsecurity-crs/rules/*.conf
[rule 910100 commented out b/c lack of GeoIP support]
RPS is... substantially different:
poprocks@mini-vm:~/nginx$ wrk -c 5 -d 5 -t 5 'http://localhost:8080/index.h
tml?exec=/bin/bash'
Running 5s test @ http://localhost:8080/index.html?exec=/bin/bash
5 threads and 5 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 14.92ms 1.39ms 36.23ms 89.95%
Req/Sec 67.02 5.03 90.00 70.80%
1680 requests in 5.03s, 531.49KB read
Non-2xx or 3xx responses: 1680
Requests/sec: 334.25
Transfer/sec: 105.74KB
Of course, this short-circuits request processing. A request pattern that
is not dropped by the CRS sees worse performance:
poprocks@mini-vm:~/nginx$ wrk -c 5 -d 5 -t 5 '
http://localhost:8080/index.html?test=foo'
Running 5s test @ http://localhost:8080/index.html?test=foo
5 threads and 5 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 21.41ms 2.52ms 51.67ms 81.28%
Req/Sec 46.60 5.73 70.00 60.80%
1169 requests in 5.02s, 0.95MB read
Requests/sec: 232.65
Transfer/sec: 193.11KB
And audit log entries do indeed look correct, when enabled:
---E0qxI3Fx---A--
[03/Apr/2018:21:23:46 -0700] 152281582640.577439 127.0.0.1 48956 127.0.0.1
8080
---E0qxI3Fx---B--
GET /index.html?exec=/bin/bash HTTP/1.1
Host: localhost:8080
---E0qxI3Fx---D--
---E0qxI3Fx---F--
HTTP/1.1 403
Server: nginx/1.13.9
Date: Wed, 04 Apr 2018 04:23:46 GMT
Content-Length: 169
Content-Type: text/html
Connection: keep-alive
---E0qxI3Fx---H--
ModSecurity: Access denied with code 403 (phase 2). Matched "Operator
`PmFromFile' with parameter `unix-shell.data' against variable `ARGS:exec'
(Value: `/bin/bash' ) [file "/home/poprocks/src/owasp-mods
ecurity-crs/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf"] [line "404"]
[id "932160"] [rev "1"] [msg "Remote Command Execution: Unix Shell Code
Found"] [data "Matched Data: bin/bash found within ARGS:exec: /bin/bash"]
[severity "2"] [ver "OWASP_CRS/3.0.0"] [maturity "1"] [accuracy "8"] [tag
"application-multi"] [tag "language-shell"] [tag "platform-unix"] [tag
"attack-rce"] [tag "OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION"] [tag
"WASCTC/WASC-31"] [tag "OWASP_TOP_10/A1"] [tag "PCI/6.5.2"] [hostname
"127.0.0.1"] [uri "/index.html"] [unique_id "152281582640.577439"] [ref
"o1,8v21,9t:urlDecodeUni,t:cmdLine,t:normalizePath,t:lowercase"]
---E0qxI3Fx---I--
---E0qxI3Fx---J--
---E0qxI3Fx---Z--
So I suspected something was wrong with my setup, but after looking at
Christian's numbers I'm perhaps thinking otherwise. I also made a quick
test with libmodsecurity 3.0.0, and saw the same order of magnitude drop
that Christian reported.
I also tested removing the CRS entirely and mocking against a single
SecRule:
poprocks@mini-vm:~$ tail -1 /home/poprocks/src/ModSecurity/modsecurity.conf
SecRules ARGS:test "@streq foo" "id:12345,phase:1,deny"
poprocks@mini-vm:~/nginx$ wrk -c 5 -d 5 -t 5 'http://localhost:8080/index.
html?test=fo'
Running 5s test @ http://localhost:8080/index.html?test=fo
5 threads and 5 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.18ms 286.19us 9.79ms 89.05%
Req/Sec 0.86k 28.30 0.93k 70.40%
21322 requests in 5.01s, 17.28MB read
Requests/sec: 4258.45
Transfer/sec: 3.45MB
poprocks@mini-vm:~/nginx$ wrk -c 5 -d 5 -t 5 'http://localhost:8080/index.
html?test=foo'
Running 5s test @ http://localhost:8080/index.html?test=foo
5 threads and 5 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.03ms 192.30us 2.88ms 81.58%
Req/Sec 0.97k 54.99 1.08k 67.20%
24130 requests in 5.00s, 7.45MB read
Non-2xx or 3xx responses: 24130
Requests/sec: 4822.16
Transfer/sec: 1.49MB
So the CRS clearly has a substantial impact on RPS, but the mere presence
of Nginx and a single rule (with an inexpensive operator and no
transformations) still results in a drop in performance of an order of
magnitude, and *another* order of magnitude with the full CRS in play. A
more comprehensive set of tests with a meaningful, real-world corpus of
data is definitely on my radar at this point. Perhaps good material for a
blog post ;)
BTW, I made a userspace flamegraph of the running Nginx worker process
while under wrk load with ModSecurity enabled, running the CRS:
https://s3.amazonaws.com/p0pr0ck5-data/ngx-modsec.svg
On Tue, Apr 3, 2018 at 9:48 PM, Christian Folini <christian.folini@
netnea.com> wrote:
> Hey Robert,
>
> On Tue, Apr 03, 2018 at 05:50:07PM -0700, Robert Paprocki wrote:
> > Can you share the specifics of your evaluation? Performance in modsec +
> crs
> > will vary greatly depending on the request payload. Soon I would like to
> do
> > some before and after trace profiling of these releases to better
> illustrate
> > how libmodsec performs in various conditions.
>
> I did a minimal self-compiled NGINX with a basic ModSecurity and CRS
> as documented on https://www.netnea.com/cms/nginx-modsecurity-tutorials/ .
> (These new tutorials are in a draft state, the quality is not yet there.
> Use
> with caution.)
>
> Testrun 1:
> ----------
>
> /apache/bin/ab -n 1000 -c 1 "http://localhost/index.html?test=/etc/passwd"
>
> 3.0.0
> Concurrency Level: 1
> Time taken for tests: 26.172 seconds
> Complete requests: 1000
> Failed requests: 0
> Non-2xx responses: 1000
> Total transferred: 320000 bytes
> HTML transferred: 162000 bytes
> Requests per second: 38.21 [#/sec] (mean)
> Time per request: 26.172 [ms] (mean)
> Time per request: 26.172 [ms] (mean, across all concurrent
> reqs)
> Transfer rate: 11.94 [Kbytes/sec] received
>
> 3.0.2
> Concurrency Level: 1
> Time taken for tests: 4.585 seconds
> Complete requests: 1000
> Failed requests: 0
> Non-2xx responses: 1000
> Total transferred: 320000 bytes
> HTML transferred: 162000 bytes
> Requests per second: 218.12 [#/sec] (mean)
> Time per request: 4.585 [ms] (mean)
> Time per request: 4.585 [ms] (mean, across all concurrent
> reqs)
> Transfer rate: 68.16 [Kbytes/sec] received
>
>
> Testrun 2:
> ----------
>
> /apache/bin/ab -n 1000 -c 1 "http://localhost/index.html?test=innocent"
>
> 3.0.0
> Concurrency Level: 1
> Time taken for tests: 26.168 seconds
> Complete requests: 1000
> Failed requests: 0
> Total transferred: 853000 bytes
> HTML transferred: 612000 bytes
> Requests per second: 38.21 [#/sec] (mean)
> Time per request: 26.168 [ms] (mean)
> Time per request: 26.168 [ms] (mean, across all concurrent
> reqs)
> Transfer rate: 31.83 [Kbytes/sec] received
>
>
> 3.0.2
> Concurrency Level: 1
> Time taken for tests: 3.996 seconds
> Complete requests: 1000
> Failed requests: 0
> Total transferred: 853000 bytes
> HTML transferred: 612000 bytes
> Requests per second: 250.25 [#/sec] (mean)
> Time per request: 3.996 [ms] (mean)
> Time per request: 3.996 [ms] (mean, across all concurrent
> reqs)
> Transfer rate: 208.46 [Kbytes/sec] received
>
> Felipe tagged a 3.0.2 yesterday and made it available at
> https://github.com/SpiderLabs/ModSecurity/releases
> I took that one for my tests. I reckon the performance is the same as with
> the 3.0.1 that has been announced.
>
> This perf test is obviously very superficial. A thing to note is that even
> testrun 2 would write the error-log (to gather statistical data).
>
> But whatever the specifics, I think this big performance boost will show in
> any setup even if the factor might not be that high.
>
> Having real perf tests done regularly would be very welcome, Robert.
>
> Best,
>
> Christian
>
>
> --
> I don't believe that we have come to the end of the democratic experiment.
> -- Bruce Schneier
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>
On Tue, Apr 3, 2018 at 9:48 PM, Christian Folini <
chr...@ne...> wrote:
> Hey Robert,
>
> On Tue, Apr 03, 2018 at 05:50:07PM -0700, Robert Paprocki wrote:
> > Can you share the specifics of your evaluation? Performance in modsec +
> crs
> > will vary greatly depending on the request payload. Soon I would like to
> do
> > some before and after trace profiling of these releases to better
> illustrate
> > how libmodsec performs in various conditions.
>
> I did a minimal self-compiled NGINX with a basic ModSecurity and CRS
> as documented on https://www.netnea.com/cms/nginx-modsecurity-tutorials/ .
> (These new tutorials are in a draft state, the quality is not yet there.
> Use
> with caution.)
>
> Testrun 1:
> ----------
>
> /apache/bin/ab -n 1000 -c 1 "http://localhost/index.html?test=/etc/passwd"
>
> 3.0.0
> Concurrency Level: 1
> Time taken for tests: 26.172 seconds
> Complete requests: 1000
> Failed requests: 0
> Non-2xx responses: 1000
> Total transferred: 320000 bytes
> HTML transferred: 162000 bytes
> Requests per second: 38.21 [#/sec] (mean)
> Time per request: 26.172 [ms] (mean)
> Time per request: 26.172 [ms] (mean, across all concurrent
> reqs)
> Transfer rate: 11.94 [Kbytes/sec] received
>
> 3.0.2
> Concurrency Level: 1
> Time taken for tests: 4.585 seconds
> Complete requests: 1000
> Failed requests: 0
> Non-2xx responses: 1000
> Total transferred: 320000 bytes
> HTML transferred: 162000 bytes
> Requests per second: 218.12 [#/sec] (mean)
> Time per request: 4.585 [ms] (mean)
> Time per request: 4.585 [ms] (mean, across all concurrent
> reqs)
> Transfer rate: 68.16 [Kbytes/sec] received
>
>
> Testrun 2:
> ----------
>
> /apache/bin/ab -n 1000 -c 1 "http://localhost/index.html?test=innocent"
>
> 3.0.0
> Concurrency Level: 1
> Time taken for tests: 26.168 seconds
> Complete requests: 1000
> Failed requests: 0
> Total transferred: 853000 bytes
> HTML transferred: 612000 bytes
> Requests per second: 38.21 [#/sec] (mean)
> Time per request: 26.168 [ms] (mean)
> Time per request: 26.168 [ms] (mean, across all concurrent
> reqs)
> Transfer rate: 31.83 [Kbytes/sec] received
>
>
> 3.0.2
> Concurrency Level: 1
> Time taken for tests: 3.996 seconds
> Complete requests: 1000
> Failed requests: 0
> Total transferred: 853000 bytes
> HTML transferred: 612000 bytes
> Requests per second: 250.25 [#/sec] (mean)
> Time per request: 3.996 [ms] (mean)
> Time per request: 3.996 [ms] (mean, across all concurrent
> reqs)
> Transfer rate: 208.46 [Kbytes/sec] received
>
> Felipe tagged a 3.0.2 yesterday and made it available at
> https://github.com/SpiderLabs/ModSecurity/releases
> I took that one for my tests. I reckon the performance is the same as with
> the 3.0.1 that has been announced.
>
> This perf test is obviously very superficial. A thing to note is that even
> testrun 2 would write the error-log (to gather statistical data).
>
> But whatever the specifics, I think this big performance boost will show in
> any setup even if the factor might not be that high.
>
> Having real perf tests done regularly would be very welcome, Robert.
>
> Best,
>
> Christian
>
>
> --
> I don't believe that we have come to the end of the democratic experiment.
> -- Bruce Schneier
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>
On Tue, Apr 3, 2018 at 9:48 PM, Christian Folini <
chr...@ne...> wrote:
> Hey Robert,
>
> On Tue, Apr 03, 2018 at 05:50:07PM -0700, Robert Paprocki wrote:
> > Can you share the specifics of your evaluation? Performance in modsec +
> crs
> > will vary greatly depending on the request payload. Soon I would like to
> do
> > some before and after trace profiling of these releases to better
> illustrate
> > how libmodsec performs in various conditions.
>
> I did a minimal self-compiled NGINX with a basic ModSecurity and CRS
> as documented on https://www.netnea.com/cms/nginx-modsecurity-tutorials/ .
> (These new tutorials are in a draft state, the quality is not yet there.
> Use
> with caution.)
>
> Testrun 1:
> ----------
>
> /apache/bin/ab -n 1000 -c 1 "http://localhost/index.html?test=/etc/passwd"
>
> 3.0.0
> Concurrency Level: 1
> Time taken for tests: 26.172 seconds
> Complete requests: 1000
> Failed requests: 0
> Non-2xx responses: 1000
> Total transferred: 320000 bytes
> HTML transferred: 162000 bytes
> Requests per second: 38.21 [#/sec] (mean)
> Time per request: 26.172 [ms] (mean)
> Time per request: 26.172 [ms] (mean, across all concurrent
> reqs)
> Transfer rate: 11.94 [Kbytes/sec] received
>
> 3.0.2
> Concurrency Level: 1
> Time taken for tests: 4.585 seconds
> Complete requests: 1000
> Failed requests: 0
> Non-2xx responses: 1000
> Total transferred: 320000 bytes
> HTML transferred: 162000 bytes
> Requests per second: 218.12 [#/sec] (mean)
> Time per request: 4.585 [ms] (mean)
> Time per request: 4.585 [ms] (mean, across all concurrent
> reqs)
> Transfer rate: 68.16 [Kbytes/sec] received
>
>
> Testrun 2:
> ----------
>
> /apache/bin/ab -n 1000 -c 1 "http://localhost/index.html?test=innocent"
>
> 3.0.0
> Concurrency Level: 1
> Time taken for tests: 26.168 seconds
> Complete requests: 1000
> Failed requests: 0
> Total transferred: 853000 bytes
> HTML transferred: 612000 bytes
> Requests per second: 38.21 [#/sec] (mean)
> Time per request: 26.168 [ms] (mean)
> Time per request: 26.168 [ms] (mean, across all concurrent
> reqs)
> Transfer rate: 31.83 [Kbytes/sec] received
>
>
> 3.0.2
> Concurrency Level: 1
> Time taken for tests: 3.996 seconds
> Complete requests: 1000
> Failed requests: 0
> Total transferred: 853000 bytes
> HTML transferred: 612000 bytes
> Requests per second: 250.25 [#/sec] (mean)
> Time per request: 3.996 [ms] (mean)
> Time per request: 3.996 [ms] (mean, across all concurrent
> reqs)
> Transfer rate: 208.46 [Kbytes/sec] received
>
> Felipe tagged a 3.0.2 yesterday and made it available at
> https://github.com/SpiderLabs/ModSecurity/releases
> I took that one for my tests. I reckon the performance is the same as with
> the 3.0.1 that has been announced.
>
> This perf test is obviously very superficial. A thing to note is that even
> testrun 2 would write the error-log (to gather statistical data).
>
> But whatever the specifics, I think this big performance boost will show in
> any setup even if the factor might not be that high.
>
> Having real perf tests done regularly would be very welcome, Robert.
>
> Best,
>
> Christian
>
>
> --
> I don't believe that we have come to the end of the democratic experiment.
> -- Bruce Schneier
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>
|
