[mod-security-users] Race Conditions and mod_security Variables

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hi all,

Does mod_security have a race condition problem/issue with its variables, or am I doing something incorrect with my configuration ?  (See details below.)

I’m wanting to rate limit connections to a resource to no more than 20 requests in a minute.  What I’m finding is that if (from a test client) I spawn connection requests as fast as possible, i.e.:

#!/bin/bash
for ((i=1;i<=3000;i++));
do
                nohup /usr/bin/curl -b /path/to/cookies https://myserver.url/fragile/script.php &
done

Then I was surprised to find that I could far more than 20 requests – I got not 20 but more than 350 requests through via my test.

When I set:

    SecDebugLog /var/log/httpd/modsec_debug.log

    SecDebugLogLevel 9

I was able to see what was happening.  This is modsec_debug grepped for “Set variable” and you can see that there’s an apparent race condition involving the variables as follows (note the short time period of 13 seconds):

[13/Mar/2022:16:50:39 +1300] [myserver.url/sid#56481359f968][rid#564813818810][/fragile/script.php][9] Set variable "ip.access_count" to "1".
[13/Mar/2022:16:50:39 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "1".
[13/Mar/2022:16:50:39 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "1".
[13/Mar/2022:16:50:39 +1300] [myserver.url/sid#56481359f968][rid#564813826880][/fragile/script.php][9] Set variable "ip.access_count" to "1".
[13/Mar/2022:16:50:40 +1300] [myserver.url/sid#56481359f968][rid#564813818880][/fragile/script.php][9] Set variable "ip.access_count" to "2".
[13/Mar/2022:16:50:40 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "2".
[13/Mar/2022:16:50:40 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "2".
[13/Mar/2022:16:50:40 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "2".
[13/Mar/2022:16:50:40 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "2".
[13/Mar/2022:16:50:40 +1300] [myserver.url/sid#56481359f968][rid#564813818810][/fragile/script.php][9] Set variable "ip.access_count" to "1".
[13/Mar/2022:16:50:41 +1300] [myserver.url/sid#56481359f968][rid#564813818880][/fragile/script.php][9] Set variable "ip.access_count" to "2".
[13/Mar/2022:16:50:41 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "2".
[13/Mar/2022:16:50:41 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "2".
[13/Mar/2022:16:50:41 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "2".
[13/Mar/2022:16:50:41 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "2".
[13/Mar/2022:16:50:41 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "2".
[13/Mar/2022:16:50:41 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "2".
[13/Mar/2022:16:50:41 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "2".
[13/Mar/2022:16:50:41 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "2".
[13/Mar/2022:16:50:41 +1300] [myserver.url/sid#56481359f968][rid#564813828890][/fragile/script.php][9] Set variable "ip.access_count" to "1".
[13/Mar/2022:16:50:41 +1300] [myserver.url/sid#56481359f968][rid#5648138e3f70][/fragile/script.php][9] Set variable "ip.access_count" to "2".
[13/Mar/2022:16:50:42 +1300] [myserver.url/sid#56481359f968][rid#564813818880][/fragile/script.php][9] Set variable "ip.access_count" to "3".
[13/Mar/2022:16:50:42 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "3".
[13/Mar/2022:16:50:42 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "3".
[13/Mar/2022:16:50:42 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "3".
[13/Mar/2022:16:50:42 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "3".
[13/Mar/2022:16:50:42 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "3".
[13/Mar/2022:16:50:42 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "3".
[13/Mar/2022:16:50:42 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "3".
[13/Mar/2022:16:50:42 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "3".
[13/Mar/2022:16:50:42 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "3".
[13/Mar/2022:16:50:42 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "3".
[13/Mar/2022:16:50:42 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "3".
[13/Mar/2022:16:50:42 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "3".
[13/Mar/2022:16:50:42 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "3".
[13/Mar/2022:16:50:42 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "3".
[13/Mar/2022:16:50:42 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "3".
[13/Mar/2022:16:50:42 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "3".

<< More than 300 lines with the same pattern as the above >>

[13/Mar/2022:16:50:51 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "19".

[13/Mar/2022:16:50:51 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "19".

[13/Mar/2022:16:50:51 +1300] [myserver.url/sid#56481359f968][rid#564813818880][/fragile/script.php][9] Set variable "ip.access_count" to "20".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813818880][/fragile/script.php][9] Set variable "ip.access_count" to "20".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

[13/Mar/2022:16:50:52 +1300] [myserver.url/sid#56481359f968][rid#564813816870][/fragile/script.php][9] Set variable "ip.access_count" to "21".

For “well behaved clients” there is no problem (i.e. if I send a request with a second pause etc between requests) – but the “fast as possible / brute force” incurs the above.

Looking at the Apache logs – for my 3,000 brute force connection attempts, I got 358 with HTTP Code 200, and 2642 with HTTP Code 429 (not 20 and 2950 respectively as expected).

Is this a bug with mod_security ?  Is there any way to avoid/workaround the above – even if it incurs some performance overhead ?

My complete mod_security config is:

<IfModule mod_security2.c>
    SecRuleEngine On
    SecRequestBodyAccess Off
    SecResponseBodyAccess Off
    SecDebugLog /var/log/httpd/modsec_debug.log
    SecDebugLogLevel 9
    SecAuditEngine RelevantOnly
    SecAuditLogRelevantStatus "^(?:5|4(?!04))"
    SecAuditLogParts ABIJDEFHZ
    SecAuditLogType Serial
    SecAuditLog /var/log/httpd/modsec_audit.log
    SecArgumentSeparator &
    SecCookieFormat 0
    SecTmpDir /var/lib/mod_security
    SecDataDir /var/lib/mod_security

<LocationMatch "^/fragile/script.php.*">
  # initialise the state based on X-Forwarded-For ip address
  SecRule REQUEST_HEADERS:X-Forwarded-For "@unconditionalMatch" "phase:2,initcol:ip=%{MATCHED_VAR},pass,nolog,id:100"
  SecRule IP:ACCESS_COUNT "@gt 20" "phase:2,deny,status:429,setenv:RATELIMITED,skip:1,nolog,id:102"
  SecAction "phase:2,setvar:ip.access_count=+1,pass,nolog,id:103"
  SecAction "phase:5,ctl:auditEngine=On,deprecatevar:ip.access_count=20/60,pass,nolog,id:104"
  Header always set Retry-After "10" env=RATELIMITED
</LocationMatch>

ErrorDocument 429 "Rate Limited"

Thanks in Advance,

Patrick