[mod-security-users] Working Around Race Conditions in Persistent Storage

[Riemann . <rie...@gm...>]

Hello,

I'm trying to write rules to track and log (not block) brute force login
attempts. I've initialized the Global and IP collection in CRS_10, and have
created a separate rule/config file for horizontal brute force attempts.
I've included my attempt at a horizontal brute force rule below ("rule
one"). The rule works exactly as I'd expect if I use Burp Intruder to
simulate a brute force attack, sending POST requests to login.jsp, with
"username" and "password" parameters in the request body, and changing only
the username on each request.

If I run the exact same 'attack,' but this time with five threads, I run
into problems. My counter may equal '1' for five consecutive requests, and
then jump to '6', but only show two usernames are stored in IP.username
(for example). I've also tried using rules similar to the horizontal brute
force rules in The Web Application Defender's Cookbook (recipe 7-2, pg
269-271), and found I had similar issues with multi-threaded attacks for
those rules too.

I know SDBM allows for concurrent access, and I've read (and re-read) about
how collections work in Chapter 8 of the ModSecurity Handbook (including
the parts about storing non-numeric values in SDBM, and race conditions).
In light of that I've attempted to use only numeric values by eliminating
all stored variables except the counter, as seen in "rule two" below (and
completely eliminating the tracking of unique usernames). Still, I had
issues where multi-threading a brute force attack resulted in the counter
value not being correct for several requests in a row.

So my questions:

   1. Am I missing something, or doing something fundamentally wrong in how
   I'm using persistent storage?
   2. I realize the potential for race conditions with SDBM, especially
   with non-numeric data, but is what I'm describing entirely expected
   behavior? Numeric values in persistent storage seem to be fairly unreliable
   on a per-request basis (in logs, my counter is wrong more than it is right).
   3. Is there a better approach to accomplish what I'm trying to do, which
   is track horizontal (by IP addr and username), and eventually vertical (by
   IP addr and password hash) brute force attempts? For example, using Lua, or
   some other persistent storage mechanism (memcached isn't an option).

If it matters, I'm currently testing with ModSec 2.9.1 on Apache 2.4.23,
running on Windows.

Thanks,
Riemann

########################################
# 'Rule' One #
###########

SecRule REQUEST_FILENAME "!@pm login.jsp" \
"phase:2,t:none,pass,nolog, \
id:999321, \
skipAfter:END_BRUTE_FORCE"
SecRule REQUEST_METHOD "!@streq POST" \
"phase:2,t:none,pass,nolog, \
id:999322, \
skipAfter:END_BRUTE_FORCE"

SecRule ARGS:username|ARGS:user ".+" \
"chain,phase:2,t:none,t:lowercase,pass,nolog, \
id:999323, \
skipAfter:LOGIN_TEST"
SecRule ARGS:password|ARGS:pass ".+" \
"t:none,t:lowercase"

SecAction \
"phase:2,pass,nolog, \
skipAfter:END_BRUTE_FORCE, \
id:'1000950'"

SecMarker LOGIN_TEST
SecRule IP:username_ct "@ge 5" \
"phase:5,t:none,log,pass, \
id:'1000951', \
msg:'Multiple Username Violation - Too Many Usernames Submitted from
%{REMOTE_ADDR}.' \
logdata:'Usernames Attempted Count: %{IP.username_ct}, Attempted Usernames:
%{IP.username}.', \
setvar:IP.clear_username=1"
SecRule &IP:CLEAR_USERNAME "@eq 1" \
"phase:5,pass,nolog, \
id:'1000952', \
setvar:!IP.KEY"
SecRule &IP:username_ct "@eq 0" \
"chain,phase:2,pass,nolog, \
id:'1000953'"
SecRule ARGS:username|ARGS:user ".+" \
"t:none,t:lowercase, \
setvar:IP.username_ct=+1, \
expirevar:IP.username_ct=300, \
setvar:IP.username=%{matched_var}, \
expirevar:IP.username=300"

SecRule &IP:username_ct "@ge 1" \
"chain,phase:2,pass,nolog, \
id:'1000954'"
SecRule ARGS:username|ARGS:user "!@within %{IP.username}" \
"t:none,t:lowercase, \
setvar:IP.username_ct=+1, \
expirevar:IP.username_ct=300, \
setvar:'IP.username=%{IP.username},%{matched_var}', \
expirevar:IP.username=300"
SecMarker END_BRUTE_FORCE

########################################
# 'Rule' Two #
###########

SecRule REQUEST_FILENAME "!@pm login.jsp" \
"phase:2,t:none,pass,nolog, \
id:999321, \
skipAfter:END_BRUTE_FORCE"
SecRule REQUEST_METHOD "!@streq POST" \
"phase:2,t:none,pass,nolog, \
id:999322, \
skipAfter:END_BRUTE_FORCE"

SecRule ARGS:username|ARGS:user ".+" \
"chain,phase:2,t:none,t:lowercase,pass,nolog, \
id:999323, \
skipAfter:LOGIN_TEST"
SecRule ARGS:password|ARGS:pass ".+" \
"t:none,t:lowercase"

SecAction \
"phase:2,pass,nolog, \
skipAfter:END_BRUTE_FORCE, \
id:'1000950'"

SecAction \
"phase:2,pass,log, \
id:'1000959', \
logdata:'Usernames Attempted Count: %{IP.username_ct}.'"
SecRule IP:username_ct "@ge 7" \
"phase:2,t:none,log,pass, \
id:'1000951', \
msg:'Multiple Username Violation - Too Many Usernames Submitted from
%{REMOTE_ADDR}.' \
logdata:'Usernames Attempted Count: %{IP.username_ct}.', \
setvar:IP.clear_username=1"
SecRule &IP:CLEAR_USERNAME "@eq 1" \
"phase:2,pass,nolog, \
id:'1000952', \
setvar:!IP.KEY"
SecRule ARGS:username|ARGS:user ".+" \
"t:none,t:lowercase,pass,nolog, \
id:'1000953', \
setvar:IP.username_ct=+1, \
expirevar:IP.username_ct=300"
SecMarker END_BRUTE_FORCE