mod-security-users — General discussion about mod_security

Re: [mod-security-users] Problem executing PHP script as filter action

[David B. Jr. <db...@gm...>]

On 4/26/06, Alex V. <ale...@ss...> wrote:
>
> On Mer 26 avril 2006 17:32, David Brieck Jr. a =E9crit :
> [...]
> > mod_security-message: Access denied with code 403. Pattern match
> > "/bin/davetest" at REQUEST_URI [severity "EMERGENCY"]
> > mod_security-action: 403
> > mod_security-executed: /usr/local/mod_sec/report- attack.sh
> [...]
> Why do you have a space betwwen report- and attack ??? try avoid dashes i=
n
> name and see if that's problem source.
>
> Hope this help
>
> Alex



Not sure why that came through with a space,  there are no spaces in the
filename.

Removing the dash doesn't seem to do anything either:

mod_security-message: Access denied with code 403. Pattern match
"/bin/davetest" at REQUEST_URI [severity "EMERGENCY"]
mod_security-action: 403
mod_security-executed: /usr/local/mod_sec/reportattack.sh

and still no email. :(

Re: [mod-security-users] Problem executing PHP script as filter action

[David B. Jr. <db...@gm...>]

On 4/26/06, Ivan Ristic <iva...@gm...> wrote:
>
> >
> > I increased the dubug level to 9 and there were no error messages, just
> it's
> > normal stuff.
>
> There should be a line that begins with "sec_exec_child: First line
> from script output". Can you find it? What does it say?
>
> > Ultimately the script will do much more than send an email, but I figur=
e
> > that's a good place to start.
>
> I am not sure that is such a good idea. What will happen when you get
> 100 attacks per second? Even if you build a throttling mechanism your
> box is going to have difficulties starting 100 binaries per second. :)
>
> A safer approach is to observe the audit log entries from a single
> process.
>

Here are the results from the debug log if the level is set to 9.

[26/Apr/2006:13:09:05 -0400] [
xx.xx.com/sid#8234520][rid#8288628][/index.php][2] Detection phase starting
(request 8288628): "GET /index.php?act=3Drssout&amp;id=3D1&/bin/davetest
HTTP/1.1"
[26/Apr/2006:13:09:05 -0400] [
xx.xx.com/sid#8234520][rid#8288628][/index.php][4] Normalised REQUEST_URI:
"/index.php?act=3Drssout&amp;id=3D1&/bin/davetest"
[26/Apr/2006:13:09:05 -0400] [
xx.xx.com/sid#8234520][rid#8288628][/index.php][9] Checking against
"/index.php?act=3Drssout&amp;id=3D1&/bin/davetest"
[26/Apr/2006:13:09:05 -0400] [
xx.xx.com/sid#8234520][rid#8288628][/index.php][9] Checking against
"/index.php?act=3Drssout&amp;id=3D1&/bin/davetest"
[26/Apr/2006:13:09:05 -0400] [
xx.xx.com/sid#8234520][rid#8288628][/index.php][9] Checking against "GET
/index.php?act=3Drssout&amp;id=3D1&/bin/davetest HTTP/1.1"
[26/Apr/2006:13:09:05 -0400] [
xx.xx.com/sid#8234520][rid#8288628][/index.php][9] Checking against
"/index.php?act=3Drssout&amp;id=3D1&/bin/davetest"
[26/Apr/2006:13:09:05 -0400] [
xx.xx.com/sid#8234520][rid#8288628][/index.php][9] Checking against
"/index.php?act=3Drssout&amp;id=3D1&/bin/davetest"
[26/Apr/2006:13:09:05 -0400] [
xx.xx.com/sid#8234520][rid#8288628][/index.php][9] Checking against
"/index.php?act=3Drssout&amp;id=3D1&/bin/davetest"
[26/Apr/2006:13:09:05 -0400] [
xx.xx.com/sid#8234520][rid#8288628][/index.php][9] Checking against
"/index.php?act=3Drssout&amp;id=3D1&/bin/davetest"
[26/Apr/2006:13:09:05 -0400] [
xx.xx.com/sid#8234520][rid#8288628][/index.php][9] Checking against
"/index.php?act=3Drssout&amp;id=3D1&/bin/davetest"
[26/Apr/2006:13:09:05 -0400] [
xx.xx.com/sid#8234520][rid#8288628][/index.php][9] Checking against
"/index.php?act=3Drssout&amp;id=3D1&/bin/davetest"
[26/Apr/2006:13:09:05 -0400] [
xx.xx.com/sid#8234520][rid#8288628][/index.php][9] Checking against
"/index.php?act=3Drssout&amp;id=3D1&/bin/davetest"


If I "grep "sec_exec_child" modsec_debug_log" there are no results.

My reasoning is that we were recently broken into and after going back over
all the logs it became very clear that if we just had something running to
block the offenders based on mod security's filters we would probably not
have been hacked.

What would you suggest using instead to monitor the logs?

Re: [mod-security-users] Problem executing PHP script as filter action

[Alex V. <ale...@ss...>]

On Mer 26 avril 2006 17:32, David Brieck Jr. a =E9crit :
[...]
> mod_security-message: Access denied with code 403. Pattern match
> "/bin/davetest" at REQUEST_URI [severity "EMERGENCY"]
> mod_security-action: 403
> mod_security-executed: /usr/local/mod_sec/report- attack.sh
[...]
Why do you have a space betwwen report- and attack ??? try avoid dashes i=
n
name and see if that's problem source.

Hope this help

Alex

Re: [mod-security-users] Problem executing PHP script as filter action

[Ivan R. <iva...@gm...>]

>
> I increased the dubug level to 9 and there were no error messages, just i=
t's
> normal stuff.

There should be a line that begins with "sec_exec_child: First line
from script output". Can you find it? What does it say?

> Ultimately the script will do much more than send an email, but I figure
> that's a good place to start.

I am not sure that is such a good idea. What will happen when you get
100 attacks per second? Even if you build a throttling mechanism your
box is going to have difficulties starting 100 binaries per second. :)

A safer approach is to observe the audit log entries from a single process.

--
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall

Re: [mod-security-users] Problem executing PHP script as filter action

[David B. Jr. <db...@gm...>]

On 4/25/06, Tom Anderson <tan...@oa...> wrote:

> Does it execute correctly from the command line if run as user "apache"?
>   That would be the first thing I would check.  Likely a permissions
> issue.
>
> Tom


Our apache runs as httpd. If I su httpd then run the file I both get an
email and have the variables written out to the text file.

Since my original email I also tried to send an email with a perl script
with the same results: an email sent from the command line and nothing when
executed with mod security. I also tried to run the perl script both throug=
h
mod sec and as the httpd user with the same results as the php file.

We don't run sendmail, we run qmail with the sendmail replacement, not sure
if this matters.

Ultimately the script will do much more than send an email, but I figure
that's a good place to start.

(resending this because i didn't hit reply to all)

Re: [mod-security-users] Problem executing PHP script as filter action

[David B. Jr. <db...@gm...>]

On 4/26/06, Ivan Ristic <iva...@gm...> wrote:
>
>
> Also, can you try executing some other script that is not PHP? PHP has
> some built-in security "logic" (need I say that it's faulty?) that
> attempts to detect if it's run as a CGI script (and then stops
> executing if it does).
>
> If you increase the debug log level you might get more information
> about the execution.
>


Thanks. I just finished trying a bash script to send me an email. It looks
like this:

#!/bin/bash


/bin/mail -s "My subject" db...@xx... <<EOF
This is a test email.

EOF

It's permissions are:

[root@cp mod_sec]# ls -l report-attack.sh
-rwxr-xr-x    1 root     root           93 Apr 26 10:34 report-attack.sh

The permissions on /bin/mail are:

[root@cp mod_sec]# ls -l /bin/mail
-rwxr-xr-x    1 root     mail        66492 Jun 24  2001 /bin/mail


Again, I have no problems doing this from the command line, it's just when
mod_sec tries to do it. Our apache is not chrooted nor are we using the
mod_sec chroot path.

I increased the dubug level to 9 and there were no error messages, just it'=
s
normal stuff. Another interesting thing I noticed was that the error code
returned is 403, but it should be 500 as the default is set:

# By default log and deny suspicious requests
# with HTTP status 500
SecFilterDefaultAction "deny,log,status:500"

Any ideas why it would be giving a different error code for this rule with
an exec on it as well? Here is the entire entry from the audit log:


=3D=3D4124522d=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Request: REMOVED xx.xx.xx.xx - - [26/Apr/2006:10:33:55 -0400] "GET
/index.php?act=3Drssout&amp;id=3D1&/
bin/davetest HTTP/1.1" 403 219 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1=
;
en-US; rv:1.8.0.2) Gecko/20060308 Firefox/1.
5.0.2" RE@E0woBlkYAAEUAj4k "-"
Handler: mod_gzip_handler
----------------------------------------
GET /index.php?act=3Drssout&amp;id=3D1&/bin/davetest HTTP/1.1
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=3D0.9
,text/plain;q=3D0.8,image/png,*/*;q=3D0.5
Accept-Charset: ISO-8859-1,utf-8;q=3D0.7,*;q=3D0.7
Accept-Encoding: gzip,deflate
Accept-Language: en-us,en;q=3D 0.5
Cache-Control: max-age=3D0
Connection: keep-alive
Cookie: REMOVED
Host: REMOVED
Keep-Alive: 300
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.2)
Gecko/20060308 Firefox/1.5.0.2

mod_security-message: Access denied with code 403. Pattern match
"/bin/davetest" at REQUEST_URI [severity "EMERGENCY"]
mod_security-action: 403
mod_security-executed: /usr/local/mod_sec/report- attack.sh

HTTP/1.1 403 Forbidden
Keep-Alive: timeout=3D10, max=3D99
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=3Diso-8859-1
--4124522d--

Thanks for your help, I'm really at a loss to the problem.

(resending this because i didn't hit reply to all)

Re: [mod-security-users] Problem executing PHP script as filter action

[Ivan R. <iva...@gm...>]

On 4/25/06, Tom Anderson <tan...@oa...> wrote:
> David Brieck Jr. wrote:
> > The file will execute from the command line, and it looks like it's
> > processed in the audit log:
>
> > But I never get an email and the file is never written. Am I doing
> > something wrong?
>
> Does it execute correctly from the command line if run as user "apache"?
>   That would be the first thing I would check.  Likely a permissions issu=
e.

Also, can you try executing some other script that is not PHP? PHP has
some built-in security "logic" (need I say that it's faulty?) that
attempts to detect if it's run as a CGI script (and then stops
executing if it does).

If you increase the debug log level you might get more information
about the execution.

--
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall

Re: [mod-security-users] (no subject)

[Ivan R. <iva...@gm...>]

On 4/26/06, BassPlayer <bas...@an...> wrote:
> Hi All,
> How much at risk am I with this off? With it on, it really makes it hard
> to talk about anything interesting, using squirrelmail, without it 403ing
> my mail message.
>
> POST scanning

You are experiencing problems not because you have request body
buffering enabled, but because you have incorrect rules setup for
SquirerelMail. You should really focus on the latter (for example, you
might decide to turn ModSecurity off for that part of the web server).

As for the risk measurement - it's not a purely technical issue. It
depends on how likely is that someone is going to attack you.

--
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall

[mod-security-users] (no subject)

[BassPlayer <bas...@an...>]

Hi All,
How much at risk am I with this off? With it on, it really makes it hard
to talk about anything interesting, using squirrelmail, without it 403ing
my mail message.

POST scanning

Request body payload (or POST payload) scanning is disabled by default. To
use it, you need to turn it on:

SecFilterScanPOST On

BP

Re: [mod-security-users] Problem executing PHP script as filter action

[Tom A. <tan...@oa...>]

David Brieck Jr. wrote:
> The file will execute from the command line, and it looks like it's 
> processed in the audit log:

> But I never get an email and the file is never written. Am I doing 
> something wrong?

Does it execute correctly from the command line if run as user "apache"? 
  That would be the first thing I would check.  Likely a permissions issue.

Tom

[mod-security-users] Problem executing PHP script as filter action

[David B. Jr. <db...@gm...>]

I'm having a problem with the following rule:

SecFilter "/bin/davetest" "exec:/usr/local/mod_sec/report-attack.sh"

where the contents of /usr/local/mod_sec/report-attack.sh are

#!/usr/local/bin/php -q
<?php
ob_start();
print_r($_SERVER);
$data =3D ob_get_contents();//save it in a variable for later use
ob_end_clean();//stop buffering

mail("db...@xx...","Environment Vars",$data);
echo "Done! \n";
$file =3D "/tmp/davetest.txt";
$open =3D @fopen($file, "w");
fwrite($open, $data);
fclose($open);

?>

The file will execute from the command line, and it looks like it's
processed in the audit log:

mod_security-message: Access denied with code 403. Pattern match
"/bin/davetest" at REQUEST_URI [severity "EMERGENCY"]
mod_security-action: 403
mod_security-executed: /usr/local/mod_sec/report-attack.sh

But I never get an email and the file is never written. Am I doing somethin=
g
wrong?

mod_security 1.9.3
apache 1.3.33
php version 4.4.0

Thanks for any help

David Brieck

Re: [mod-security-users] SecFilterScanOutput and mod_include (SSIs) not behaving together

[Ivan R. <iva...@gm...>]

That problem was fixed in 1.9.3. A list of known and fixed issues is
available in Thinking Stone Network:
https://www.thinkingstone.com/tsn/ (free registration, no spam).

On 4/25/06, Ryan Boyd <rb...@ri...> wrote:
>
>
> modsecurity: 1.9.2
>  apache: 2.2.0
>
>  I have some sites that use SSIs for doing virtual and file includes.  Th=
ese
> work fine under mod_security so long as SecFilterScanOutput is not enable=
d.
> As soon as SecFilterScanOutput is 'on' (even with no OUTPUT filters), the=
se
> SSIs fail to include the appropriate data. I can tell from the error logs
> that the includes are being processed and the virtual includes are actual=
ly
> being requested through the server (can see debug output from mod_rewrite=
),
> but the final results of those includes are never actually included in th=
e
> output from the server when SecFilterScanOutput is on.  Looking at the
> modsecurity debug output doesnt seem to reveal anything exciting that hel=
ps
> towards solving this problem.
>
>  Has anyone experienced this problem?  Or have further hints as to what I
> can look at to resolve this or at least track it down further?
>
>  Thanks,
>
>  -Ryan
>


--
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall

[mod-security-users] SecFilterScanOutput and mod_include (SSIs) not behaving together

[Ryan B. <rb...@ri...>]

modsecurity: 1.9.2
apache: 2.2.0

I have some sites that use SSIs for doing virtual and file includes.  =
These work fine under mod_security so long as SecFilterScanOutput is not =
enabled.  As soon as SecFilterScanOutput is 'on' (even with no OUTPUT =
filters), these SSIs fail to include the appropriate data. I can tell =
from the error logs that the includes are being processed and the =
virtual includes are actually being requested through the server (can =
see debug output from mod_rewrite), but the final results of those =
includes are never actually included in the output from the server when =
SecFilterScanOutput is on.  Looking at the modsecurity debug output =
doesnt seem to reveal anything exciting that helps towards solving this =
problem.

Has anyone experienced this problem?  Or have further hints as to what I =
can look at to resolve this or at least track it down further?

Thanks,

-Ryan

Re: [mod-security-users] mod_security or deny, allow to protect a specail directory

[Jochen K. <fvg...@wl...>]

Am Montag, den 24.04.2006, 20:25 +0100 schrieb Ivan Ristic:
> On 4/24/06, Jochen Kaechelin <fvg...@wl...> wrote:
> > <Directory /var/www/pics.gissmoh.de/test>
> >   SecFilterEngine On
> >   SecFilterSelective REMOTE_ADDR "!^xxx\.xxx\.xxx\.xxx$" deny
> > </Directory>
> >
> > or
> >
> > <Directory /var/www/pics.gissmoh.de/test>
> >   Order deny,allow
> >   Deny from all
> >   Allow from xxx.xxx.xxx.xxx
> > </Directory>
> 
> This one. Why use ModSecurity for things possible with stock Apache.

OK, you right! I'am very impressed by mod_security..so I thought maybe
the technique mod_security uses is more stable/secure than the features
implemented in stock apache. 

And I love to add the redirect-feature to
select different actions according to different remote ips.


-- 
wlanhacking.de
http://mail.wlanhacking.de/cgi-bin/mailman/listinfo

Frauen sind die einzigsten Opfer die auf ihre Jäger lauern!

Re: [mod-security-users] mod_security or deny, allow to protect a specail directory

[Ivan R. <iva...@gm...>]

On 4/24/06, Jochen Kaechelin <fvg...@wl...> wrote:
> <Directory /var/www/pics.gissmoh.de/test>
>   SecFilterEngine On
>   SecFilterSelective REMOTE_ADDR "!^xxx\.xxx\.xxx\.xxx$" deny
> </Directory>
>
> or
>
> <Directory /var/www/pics.gissmoh.de/test>
>   Order deny,allow
>   Deny from all
>   Allow from xxx.xxx.xxx.xxx
> </Directory>

This one. Why use ModSecurity for things possible with stock Apache.

--
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall

[mod-security-users] mod_security or deny, allow to protect a specail directory

[Jochen K. <fvg...@wl...>]

<Directory /var/www/pics.gissmoh.de/test>
  SecFilterEngine On
  SecFilterSelective REMOTE_ADDR "!^xxx\.xxx\.xxx\.xxx$" deny
</Directory>

or

<Directory /var/www/pics.gissmoh.de/test>
  Order deny,allow
  Deny from all
  Allow from xxx.xxx.xxx.xxx
</Directory>           


-- 
wlanhacking.de
http://mail.wlanhacking.de/cgi-bin/mailman/listinfo

Frauen sind die einzigsten Opfer die auf ihre Jäger lauern!

Re: [mod-security-users] v1.9 memory usage problem

[Ivan R. <iva...@gm...>]

On 4/23/06, Jim <st...@cl...> wrote:
> Hello Ivan,
>
> I had planned to begin troubleshooting properly tomorrow but while I was =
here I started some things and thought I'd post back as the early results a=
re interesting.

Thanks a lot for the detailed information.

> ...
> Finally, I rolled back to version 1.8.7 (this is a live server and I can'=
t leave it on 1.9) and put our full ruleset back into place (including SecF=
ilterScanPOST On).  As it has been doing for months on this version, everyt=
hing is still fine.

I think we should focus on the fact 1.9.x with no rules creates
problems for you where 1.8.7 does not. I will create a special version
of mod_security for you to test and hopefully we'll locate the
problem.

--
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall

Re: [mod-security-users] v1.9 memory usage problem

[Jim <st...@cl...>]

Hello Ivan,

I had planned to begin troubleshooting properly tomorrow but while I was here I started some things and thought I'd post back as the early results are interesting.

I upgraded to mod_sec 1.9.3 again and first tried the suggestion of adding the 'DynamicOnly' and no post buffer settings and got the same problem.

Next, I started with the removal of all rules and adding them back block by block to see if this turns anything up.  I removed everything and got the same problem within 10 minutes of the apache restart.

mod_sec config:

root@xxxx [~]# cat /etc/httpd/conf/mod_security.conf
# Turn the filtering engine On or Off
SecFilterEngine DynamicOnly

# Make sure that URL encoding is valid
SecFilterCheckURLEncoding Off

# Unicode encoding check
SecFilterCheckUnicodeEncoding Off

# Only allow bytes from this range
SecFilterForceByteRange 0 255

# Only log suspicious requests
SecAuditEngine RelevantOnly

# The name of the audit log file
SecAuditLog /var/log/httpd/audit_log

# Debug level set to a minimum
SecFilterDebugLog /var/log/httpd/modsec_debug_log
SecFilterDebugLevel 0

# Should mod_security inspect POST payloads
SecFilterScanPOST On

# By default log and deny suspicious requests
# with HTTP status 500
SecFilterDefaultAction "deny,log,status:406"

# no ban to localhost
SecFilterSelective REMOTE_ADDR "^127.0.0.1$" nolog,allow

# suggestion by mod_sec devs
SetEnvIfNoCase Content-Type "^multipart/form-data;" \
"MODSEC_NOPOSTBUFFERING=Do not buffer file uploads"

SecFilterSelective THE_REQUEST "_vti_bin" allow,nolog
SecFilterSelective THE_REQUEST "/fpsrvadm\.exe" pass
SecFilterSelective THE_REQUEST "/fpremadm\.exe" pass
SecFilterSelective THE_REQUEST "/admisapi/fpadmin\.htm" pass
SecFilterSelective THE_REQUEST "/scripts/Fpadmcgi\.exe" pass
SecFilterSelective THE_REQUEST "/_private/orders\.txt" pass
SecFilterSelective THE_REQUEST "/_private/form_results\.txt" pass
SecFilterSelective THE_REQUEST "/_private/registrations\.htm" pass
SecFilterSelective THE_REQUEST "/cfgwiz\.exe" pass
SecFilterSelective THE_REQUEST "/authors\.pwd" pass
SecFilterSelective THE_REQUEST "/_vti_bin/_vti_aut/author\.exe" pass
SecFilterSelective THE_REQUEST "/administrators\.pwd" pass
SecFilterSelective THE_REQUEST "/_private/form_results\.htm" pass
SecFilterSelective THE_REQUEST "/_vti_pvt/access\.cnf" pass
SecFilterSelective THE_REQUEST "/_private/register\.txt" pass
SecFilterSelective THE_REQUEST "/_private/registrations\.txt" pass
SecFilterSelective THE_REQUEST "/_vti_pvt/service\.cnf" pass
SecFilterSelective THE_REQUEST "/service\.pwd" pass
SecFilterSelective THE_REQUEST "/_vti_pvt/service\.stp" pass
SecFilterSelective THE_REQUEST "/_vti_pvt/services\.cnf" pass
SecFilterSelective THE_REQUEST "/_vti_bin/shtml\.exe" pass
SecFilterSelective THE_REQUEST "/_vti_pvt/svcacl\.cnf" pass
SecFilterSelective THE_REQUEST "/users\.pwd" pass
SecFilterSelective THE_REQUEST "/_vti_pvt/writeto\.cnf" pass
SecFilterSelective THE_REQUEST "/dvwssr\.dll" pass
SecFilterSelective THE_REQUEST "/_private/register\.htm" pass
SecFilterSelective THE_REQUEST "/_vti_bin/" pass


httpd processes in top when problem starts (see highlighted process which is the parent httpd process previously mentioned):

root@xxxx [~]# top -b | grep httpd
 2088 nobody    15   0 24756  24M  5352 S     0.4  1.2   0:00   0 httpd
 1479 root      15   0 24072  23M  4944 S     0.0  1.1   0:01   0 httpd
 1497 nobody    15   0 28128  27M  5488 S     0.0  1.3   0:01   1 httpd
 1498 nobody    15   0 31420  30M  5664 S     0.0  1.5   0:02   0 httpd
 1499 nobody    15   0 28688  28M  5668 S     0.0  1.3   0:01   0 httpd
 1500 nobody    15   0 29768  29M  5652 S     0.0  1.4   0:01   0 httpd
 1501 nobody    15   0 34308  33M  5676 S     0.0  1.6   0:01   1 httpd
 1502 nobody    15   0 29076  28M  5676 S     0.0  1.4   0:02   0 httpd
 1503 nobody    15   0 31460  30M  5664 S     0.0  1.5   0:02   1 httpd
********
 1504 nobody    15   0 1052M 1.0G  5492 S     0.0 52.4   0:03   1 httpd
********
 1505 nobody    15   0 31636  30M  5668 S     0.0  1.5   0:02   0 httpd
 1506 nobody    15   0 37040  36M  5464 S     0.0  1.8   0:01   0 httpd
 2086 nobody    15   0 27888  27M  5412 S     0.0  1.3   0:00   0 httpd
 2087 nobody    15   0 24684  24M  5320 S     0.0  1.2   0:00   0 httpd
 2104 nobody    16   0 24628  24M  5316 S     0.0  1.1   0:00   1 httpd


Output of free:

root@xxxx [~]# free -m
             total       used       free     shared    buffers     cached
Mem:          2006       1983         23          0         61        515
-/+ buffers/cache:       1406        600
Swap:         4094        148       3945

================================================
================================================

Next, I kept things *exactly* the same except I changed one line of the mod_sec config:


root@xxxx [~]# grep SecFilterScanPOST /etc/httpd/conf/mod_security.conf
SecFilterScanPOST Off


I sat and watched top for 30-40 minutes (the big httpd process has usually reared its head by this time) and all is fine:

root@xxxx [~]# top -b | grep httpd
 2322 nobody    15   0 35492  34M  5496 S     1.4  1.7   0:02   1 httpd
 2325 nobody    15   0 36964  36M  5496 S     0.9  1.7   0:01   0 httpd
 2330 nobody    15   0 36112  35M  5504 S     0.9  1.7   0:03   0 httpd
 2327 nobody    15   0 37252  36M  5672 S     0.4  1.8   0:02   1 httpd
 2329 nobody    15   0 29296  28M  5504 S     0.4  1.4   0:02   1 httpd
 2311 root      15   0 24076  23M  4944 S     0.0  1.1   0:01   0 httpd
 2321 nobody    15   0 31444  30M  5496 S     0.0  1.5   0:02   0 httpd
 2323 nobody    15   0 31592  30M  5664 S     0.0  1.5   0:02   0 httpd
 2324 nobody    15   0 30424  29M  5508 S     0.0  1.4   0:02   1 httpd
 2326 nobody    15   0 35908  35M  5504 S     0.0  1.7   0:03   1 httpd
 2328 nobody    15   0 37556  36M  5492 S     0.0  1.8   0:03   1 httpd


root@xxxx [~]# free -m
             total       used       free     shared    buffers     cached
Mem:          2006       1125        881          0         92        603
-/+ buffers/cache:        428       1577
Swap:         4094        151       3943


================================================
================================================

Finally, I rolled back to version 1.8.7 (this is a live server and I can't leave it on 1.9) and put our full ruleset back into place (including SecFilterScanPOST On).  As it has been doing for months on this version, everything is still fine.

root@xxxx [~]# top -b | grep httpd
 4321 nobody    15   0 36732  35M  5620 S     0.9  1.7   0:02   0 httpd
 4325 nobody    15   0 30792  30M  5636 S     0.9  1.4   0:03   0 httpd
 4327 nobody    15   0 54432  53M  5628 S     0.9  2.6   0:10   1 httpd
 4328 nobody    16   0 26240  25M  5428 S     0.9  1.2   0:01   1 httpd
 4315 nobody    15   0 29888  29M  5464 S     0.4  1.4   0:01   1 httpd
 4317 nobody    15   0 27272  26M  5444 S     0.4  1.3   0:01   1 httpd
 4308 root      15   0 24552  23M  4928 S     0.0  1.1   0:01   0 httpd
 4318 nobody    15   0 30404  29M  5460 S     0.0  1.4   0:02   1 httpd
 4319 nobody    15   0 29752  29M  5480 S     0.0  1.4   0:02   0 httpd
 4322 nobody    15   0 36556  35M  5444 S     0.0  1.7   0:02   0 httpd
 4326 nobody    15   0 28120  27M  5448 S     0.0  1.3   0:01   0 httpd
 4357 nobody    15   0 30624  29M  5444 S     0.0  1.4   0:02   1 httpd
 4537 nobody    15   0 27656  27M  5364 S     0.0  1.3   0:00   0 httpd
 4538 nobody    15   0 25100  24M  5196 S     0.0  1.2   0:00   0 httpd
 4539 nobody    15   0 25216  24M  5344 S     0.0  1.2   0:00   1 httpd


root@xxxx [/var/log/httpd]# free -m
             total       used       free     shared    buffers     cached
Mem:          2006       1167        838          0         98        654
-/+ buffers/cache:        415       1590
Swap:         4094        151       3943


================================================
================================================

One thing I forgot to mention earlier which I don't think should make a difference but not sure is that we're using the -DDISABLE_HTACCESS_CONFIG flag on our mod_security compiles to disable the addition/alteration of rulesets via user htaccess files.

Re: [mod-security-users] v1.9 memory usage problem

[Ivan R. <iva...@gm...>]

On 4/23/06, Jim <st...@cl...> wrote:
> Hi,
>
> Thank you for looking into this Ivan.  My main intention in posting
> here was to see whether there was a 'quick fix' from others who have
> experienced the same issue (in case it was a known bug or similar) as
> it seemed odd that this issue only happening with the upgrade of
> mod_sec with all other variables (hardware, server activity, software
> config) remaining exactly the same.

My gut feeling is that 1.9.x requires more memory than 1.8.x, so that
might be the problem. FYI my gut feeling also tells me 2.x will
require more memory than 1.9.x.

The only known problem that applies is the one I already mentioned,
documented here:

https://www.thinkingstone.com/tsn/defects/view_defect.php?defect=3DMSA-116


> With this not being the case, I'm happy to put this issue on hold for
> the time being.

That's OK, but be aware that if you don't pursue the issue probably no
one else will.

--
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall

Re: [mod-security-users] v1.9 memory usage problem

[Jim <st...@cl...>]

Hi,

Thank you for looking into this Ivan.  My main intention in posting
here was to see whether there was a 'quick fix' from others who have
experienced the same issue (in case it was a known bug or similar) as
it seemed odd that this issue only happening with the upgrade of
mod_sec with all other variables (hardware, server activity, software
config) remaining exactly the same.

With this not being the case, I'm happy to put this issue on hold for
the time being.  I will follow your advice on adding back the rules in
separate blocks to see if we can get closer to any that could be
causing this.

If this doesn't work I will get all the testing data you mentioned in
your email such as requests per second, mem usage, etc.

I'll post back here with any findings.

Thanks again,

Jim

Re: [mod-security-users] v1.9 memory usage problem

[Ivan R. <iva...@gm...>]

On 4/23/06, Jim <st...@cl...> wrote:
> ...
> The issue I am posting about has previously been discussed here: http://w=
ww.gotroot.com/tiki-view_forum_thread.php?forumId=3D35&comments_threshold=
=3D0&comments_parentId=3D658&comments_offset=3D0&comments_maxComments=3D20&=
comments_style=3DcommentStyle_threaded

A small comment for the sake of list subscribers: I've been in touch
with several people who referred me to this forum thread. All of the
issues so far have been related to having too little memory to handle
the workload. Still, I am approaching every problem report as a new
issue just in case.

--
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall

Re: [mod-security-users] v1.9 memory usage problem

[Jim <st...@cl...>]

>> That;s the thing, the *exact* same ruleset works perfectly on v1.8 but
>> not on v1.9.  Is there any specific changes made to the software
>> between these versions with regard to how 'SecFilterScanPOST' is
>> handled which could cause this?

> Not that I recall. Which 1.8.x version exactly is that?

We're currently running (and having no problems with) mod_security
version 1.8.7.

We've tried versions 1.9.1, 1.9.2 and 1.9.3 and hit the same issue
each time before rolling back to 1.8.7.

Re: [mod-security-users] v1.9 memory usage problem

[Jim <st...@cl...>]

Hello Ivan,

>> This only happens on 2 of our servers out of about 45-50 which is very strange.
> Are all these servers running the same software and the same rule set?

Throughout all servers there is a mixture on Centos v3 and v4 with the
same Apache + PHP versions (though slightly different compile options
on PHP throughout).

All servers run exactly the same mod_sec ruleset.

>> Our ruleset is a custom ruleset comprising of around 500-600 rules.
> Please send me the ruleset (privately) and I'll have a look. To begin
> with I want to rule out a known regex library problem that causes
> problems similar to those you describe.

I'll send this over to you in a sec.

Thanks,

Jim

Re: [mod-security-users] v1.9 memory usage problem

[Ivan R. <iva...@gm...>]

On 4/23/06, Jim <st...@cl...> wrote:
> Hi,
>
> > must be one of the rules I guess. We use 1.9.3 on several servers with =
way more than 500 rules
> > and don't see it. We're on RHEL 3/4 here.
>
> That;s the thing, the *exact* same ruleset works perfectly on v1.8 but
> not on v1.9.  Is there any specific changes made to the software
> between these versions with regard to how 'SecFilterScanPOST' is
> handled which could cause this?

Not that I recall. Which 1.8.x version exactly is that?

--
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall

Re: [mod-security-users] v1.9 memory usage problem

[Ivan R. <iva...@gm...>]

On 4/23/06, Jim <st...@cl...> wrote:
> Hi all,

Hi Jim,


> This only happens on 2 of our servers out of about 45-50 which is very st=
range.

Are all these servers running the same software and the same rule set?


> Our ruleset is a custom ruleset comprising of around 500-600 rules.

Please send me the ruleset (privately) and I'll have a look. To begin
with I want to rule out a known regex library problem that causes
problems similar to those you describe.

--
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall