Re: [mod-security-users] how to get console to collect concurrent logs

[Dan R. <sp...@el...>]

Hi Ryan, we currently have our servers in detection mode atm we have 5 
freebsd boxes to monitor. Is there a way to run the log generators 
manually without going via AuditLog during the period of fixing up false 
positives then turn it on to pipe when everything has stabilised, i feel 
the traffic is going to create a problem as its trigerring mod sec quite 
a bit.

let me know thanks.

Ryan Barnett wrote:
>
> See comments inline below.
>
>  
>
> -- 
> */Ryan C. Barnett
> /*Breach Security: Director of Application Security Training
> Web Application Security Consortium (WASC) Member
> CIS Apache Benchmark Project Lead
> SANS Instructor, GCIA, GCFA, GCIH, GSNA, GCUX, GSEC
> Author: Preventing Web Attacks with Apache
>
>  
>
> ------------------------------------------------------------------------
>
> *From:* Dan Rossi [mailto:sp...@el...]
> *Sent:* Thursday, December 28, 2006 2:41 AM
> *To:* Ryan Barnett
> *Cc:* mod...@li...
> *Subject:* Re: [mod-security-users] how to get console to collect 
> concurrent logs
>
>  
>
> Hi thanks for the clarification however im reading the mod security 
> 2.0 docs as thats what im using.
>
> There is some basic information in the script which is enough to work 
> with, so it posts the data to the single hosted console !
>
> Im a little worried about this comment though, so it will kill apache 
> if its run in production ?
>
> */[Ryan Barnett] The modsec-auditlog-collector script may work fine 
> for some smaller organizations.  The 2 main items that would impact 
> its effectiveness are --/*
>
>    1. */The current amount of client traffic on the Apache web
>       server.  The script can only handle one file at a time and has
>       problems when under heavy load./*
>    2. */How you have the modsecurity SecAuditEngine configured.  If
>       the SecAuditEngine is set to on, it will log all traffic to the
>       auditlog (including requests for downloading images, etc...).
>        This would drastically impact this scripts ability to process
>       the INDEX file and send data to the Console host.  You should
>       use SecAuditEngine RelevantOnly and set the
>       SecAuditEngineRelevantStatus to something like "^(?:4|5)" so
>       that is will only log 4XX and 5XX level status code transactions
>       to the auditlog./*
>
> */ /*
>
> */Also as the script indicates, it doesn't handle errors that well./*
>
> And what do i do about the current logs already in there i have to 
> somehow go through now. Can  it be run on the current logs like this 
> to start it off ?
>
> /path/to/modsec-auditlog-collector.pl /path/to/auditlog/data/  
> /path/to/auditlog/index
>
> */[Ryan Barnett] Just use this command to get all of your past audit 
> log data into the console --/*
>
> */# cat /path/to/apache/logs/index | 
> /path/to/apache/bin/modsec-auditlog-collector.pl/*
>
> */ /*
>
> */This will pipe all of your past audit log data that is held in the 
> INDEX file through the script and it will then submit the logs to the 
> Console./*
>
>
> # This is a proof-of-concept script that listens to the
> # audit log in real time and submits the entries to
> # a remote HTTP server. This code is not suitable for
> # non-trivial production use since it can only submit
> # one audit log entry at a time, plus it does not handle
> # errors gracefully.
> #
> # Usage:
> #
> # 1) Enter the correct parameters $CONSOLE_* below
> #
> # 2) Configure ModSecurity to use this script for
> #    concurrent audit logging index:
> #
> #    SecAuditLog "|/path/to/modsec-auditlog-collector.pl \
> #        /path/to/auditlog/data/ \
> #        /path/to/auditlog/index"
>
>
> Where do i put the info here for a particular sesnor for a particular 
> server if thats how it works, hopefully the data doesnt get jumbled up 
> together ?
>
> my $CONSOLE_URI = "/rpc/auditLogReceiver";
> my $CONSOLE_HOST = "192.168.2.11";
> my $CONSOLE_PORT = "8886";
> my $CONSOLE_USERNAME = "alpha";
> my $CONSOLE_PASSWORD = "sensor";
>
> */[Ryan Barnett] You need to do the following --/*
>
> */1./**/ Go into your console and create a new sensor profile.  From 
> the main page, go to Sensors -> then click on the "Add Sensor" button./*
>
> */2./**/ Fill out the necessary information for your new sensor.  
> Important -- you must remember the Username and Password that you set 
> for this sensor as you will need this information when setting up the 
> concurrent log forwarding script on your ModSecurity host./*
>
> */3./**/ Edit the modsec-auditlog-collector scrip on your modsecurity 
> host  --/*
>
>           * */CONSOLE_HOST needs to have the correct IP address of the
>             host that is running the Console./*
>           * */CONSOLE_USERNAME is the username you specified in the
>             Console when setting up the Sensor profile./*
>           * */CONSOLE_PASSWORD is the password you specified in the
>             Console when setting up the Sensor profile./*
>
>
> Ryan Barnett wrote:
>
> So you are installing the ModSecurity Console on each host that is 
> running ModSecurity?  The idea behind the console is have a central 
> location for remote ModSecurity hosts to send their logs to. 
>  Regardless, the mechanism to use to actually transfer the logs into 
> the console is to use the modsec-auditlog-collector perl script that 
> comes with the ModSecurity 1.9.4 archive.  Take a look at the logging 
> documentation here - 
> http://www.modsecurity.org/documentation/modsecurity-apache/1.9.3/html-multipage/07-logging.html. 
>  Look under the "New Audit Log Type" section for info.
>
>  
>
> -- 
> */Ryan C. Barnett
> /*Breach Security: Director of Application Security Training
> Web Application Security Consortium (WASC) Member
> CIS Apache Benchmark Project Lead
> SANS Instructor, GCIA, GCFA, GCIH, GSNA, GCUX, GSEC
> Author: Preventing Web Attacks with Apache
>
>  
>
> ------------------------------------------------------------------------
>
> *From:* Dan Rossi [mailto:sp...@el...]
> *Sent:* Thursday, December 28, 2006 1:42 AM
> *To:* Ryan Barnett
> *Cc:* mod...@li... 
> <mailto:mod...@li...>
> *Subject:* Re: [mod-security-users] how to get console to collect 
> concurrent logs
>
>  
>
> Ryan Barnett wrote:
>
> What do you mean by "collect concurrent logs from a given path"?  Are 
> you referring to how to send concurrent audit log data from 
> ModSecurity hosts to the central Console host?
>
>
> Hi Ryan, I dont know if you understood it, the console on the 
> localhost of the server does not collect any of the mod security logs 
> this is on all servers i have tried it on. There is definately logs in 
> there though, tonnes of false positives which is why i need this up 
> and running so i can fix it all up.
>
> So basically console runs fine, but cannot load any transactions or 
> any data at all and there is no documentation of what to do next.
>
> I setup some sensor if thats what it needs and selected apache in the 
> pulldown i use apache 2.0.59 and mod sec 2, the interesting thing is 
> in the server-info section it does not display the set configs for mod 
> security could this be the issue , is that how it knows where to get 
> the logs ie i have them being stored on our development machine 
> /var/log/apache2/modsec/console/
>
> etc
>
>
>  
>
> -- 
> */Ryan C. Barnett
> /*Breach Security: Director of Application Security Training
> Web Application Security Consortium (WASC) Member
> CIS Apache Benchmark Project Lead
> SANS Instructor, GCIA, GCFA, GCIH, GSNA, GCUX, GSEC
> Author: Preventing Web Attacks with Apache
>
>  
>
> ------------------------------------------------------------------------
>
> *From:* mod...@li... 
> <mailto:mod...@li...> 
> [mailto:mod...@li...] *On Behalf 
> Of *Dan Rossi
> *Sent:* Wednesday, December 27, 2006 7:21 PM
> *To:* mod...@li... 
> <mailto:mod...@li...>
> *Subject:* [mod-security-users] how to get console to collect 
> concurrent logs
>
>  
>
> Hi ive asked here quite a few times already, i cant work out how to 
> get the console to collect the concurrent logs from a given path. The 
> console is blank its not collecting and transactions at all, any ideas 
> what do i need to do as there is no log path setting.
>
> Let me know thanks.
>
>  
>
>  
>