mod-security-users

[mod-security-users] DDOS on the appl level, timeouts and blacklisting

[Christian F. <chr...@ti...>]

Hi there,

There have been a couple of meetings regarding ddos threats
against our ssl sites. So far no attack occurred, but we trying
to be prepared.

It seems simple to initiate a valid connection
with one of our reverse proxies and then do basically nothing
until the timeout occurs. It's 300 seconds by default.
Let's say we swallow a couple of thousands of these connections,
but we are no challenge for one of the bigger botnets out there.
The network guys are constantly building up their defense lines
against ddos attacks on the network layers, but our domain
OSI layer 7, looks quite undefended in this aspect.

We can't seem to find a way to the tell the legitimate
from ddos requests. Ideally we would be able to lower the
"total amount of time it takes to receive a GET request"
(and POST headers without file uploads accordingly)
seperately from the rest of the timeouts. But even apache 2.2
knows but a general timeout parameter.
If we would be able to tell the bad requests and get their
IP address, we could forward them to our front firewall
or even the ISP and have their SYNs dropped.

I have browsed the mod_security 2.0 reference, but i did not
find the silver bullet. So the question is, can mod_security 
do anything in this regard or do you fellow readers have any 
other smart ideas?

It might be helpful to have a variable in mod_security, that
would tell me something about the time it took from the
initialisation of the request to the processing in mod_security.
Obviously in mod_security, there is TIME_EPOCH, but how to
tell when the connection started?

However, this idea is built on the premise, that the request 
arrives in mod_security at last. This means the header has 
to be sent. How about something like a watchdog timer that
activates some special mod_security processing phase for
requests, that never make it to Phase 1 - request headers.
Even those that do not terminate the the ssl-handshake
should be considered.

Any thoughts are appreciated,

Christian Folini

-- 
The fool doth think he is wise, but the wise man knows himself to be
a fool.
-- William Shakespeare in As You Like It

Re: [mod-security-users] DDOS on the appl level, timeouts and blacklisting

[Ryan B. <rcb...@gm...>]

Christian,
My comments are inline below.  You are correct in that web-based DDoS/DoS
attacks are tough to handle, especially at layer 7.

On 8/21/06, Christian Folini <chr...@ti... > wrote:
>
> Hi there,
>
> There have been a couple of meetings regarding ddos threats
> against our ssl sites. So far no attack occurred, but we trying
> to be prepared.
>
> It seems simple to initiate a valid connection
> with one of our reverse proxies and then do basically nothing
> until the timeout occurs. It's 300 seconds by default.
> Let's say we swallow a couple of thousands of these connections,
> but we are no challenge for one of the bigger botnets out there.
> The network guys are constantly building up their defense lines
> against ddos attacks on the network layers, but our domain
> OSI layer 7, looks quite undefended in this aspect.


There are a few OS TCP stack settings that you should tweak to help defend
against the effects of DoS attacks.  Here are some issues to address -

   - *Buy more RAM.* Each inbound SYN packet is intended to establish a
   TCP circuit, which requires resources on the server. The TCP buffers require
   memory to be allocated.
   - *Use TCP SYN Cookies (Linux and BSD only).* With TCP Syn Cookies,
   the kernel does not really allocate the TCP buffers unless the server's
   ACK/SYN packet gets an ACK back, meaning that it was a legitimate request.
   - *Reduce the allowed number of HALF_OPEN TCP circuits.* Further
   requests are refused, a denial of service, but at least the server hasn't
   run out of memory.
   - *Reduce the amount of time an opening TCP circuit can stay in the
   HALF_OPEN state.* The server is made less patient -- if the TCP
   circuit is not fully established quickly, it is dropped and the client, if
   legitimate but very slow, must start again.
   - *Reduce the amount of time a closing TCP circuit can stay in the
   TIME_WAIT state.* Some clients are very rude, apparently Microsoft
   Explorer is particularly bad. They establish a connection, get their data,
   but then refuse to participate in cleanly shutting down the TCP circuit. At
   least for busy web servers, make them very impatient with such nonsense,
   dropping these no longer active connections and freeing resources.

Now in Apache itself, there are a number of config settings you can updated
to help.  This data is taken from the Center for Internet Security (CIS)
Apache Benchmark Document -
http://www.cisecurity.org/bench_apache.html

*L1 1.       Denial of Service (DoS) Protective General Directives*

*Description*
It is almost impossible to fully protect against denial of service attacks
since they are attacks against the fundamental underpinnings of the
Internet.   Denial of service attacks often try to exhaust the system's
resources.  For instance, since the system can only wait for a limited
number of connection requests to complete, an attacker will flood a system
with a bunch of connection requests that will never be completed.   The
server will keep the connection request open until either the connection has
been made or a specified length of time has gone by.  Once the attacker has
filled the pending connection queue, a small trickle of new requests will
keep the server unavailable to legitimate requests.   Apache has several
directives available that provide some protective capabilities against
denial of services attacks.



We will be verifying/updating the following Apache directives:

   - *Timeout***

One way of attacking systems on the Internet is to try to prevent the target
system from operating correctly by overloading it. This is called a 'denial
of service' attack. One method of doing this is to open multiple connections
to a server and never close them. The more connections the server has open
at once, the more resources are tied up holding details of those connection,
which can lead to increased load and eventually to the server running out of
resources.  The Timeout directive tells the server how long to wait to
receive a GET request, the amount of time between receipt of TCP packets on
a POST or PUT request, or the amount of time between ACKs on transmissions
of TCP packets in responses.

In order to prevent a denial of service attack from shutting down our web
server, we need to change the default setting of 300 (which is 5 minutes) to
*60* (which is 1 minute).  You may even adjust this setting to be lower than
60.



   - *KeepAlive ** *

In order to make our web server more efficient, and therefore more resistant
to DoS attacks, we want to allow TCP KeepAlives.  This setting will allow
for multiple HTTP requests to be serviced across one connection.  Without
this setting, a new Apache server would have to be spawned for every
subsequent HTTP request for gifs, etc. Recommended value is *On*.





   - *KeepAliveTimeout***

This directive will limit the time the Apache web server will wait in
seconds for a KeepAlive request to be completed.  Setting KeepAliveTimeout
to a high value may cause performance problems in heavily loaded servers.
The higher the timeout, the more server processes will be kept occupied
waiting on connections with idle clients.   If this number is set too high,
then a DoS attack could be more successful. Recommend value is *15* which is
also the default value.

* *

* **  *

* *

   - *Other Performance and DoS Related Modules and Directives*



Taking together the directives StartServers, StartServers, MaxSpareServers
and MaxClients, along with the MPM (MultiProcessing Module) work to
establish a reasonable yet dynamic number of child processes and/or threads.
Starting with Apache 1.3 the process creation process was dramatically
improved so that tweaking the StartServers, StartServers, MaxSpareServers
and MaxClients settings is unnecessary for most situations.   Therefore the
default settings are fine for most usages. For very high traffic servers,
and optimal performance consult the Apache performance recommendations at
http://httpd.apache.org/docs/2.0/misc/perf-tuning.html  or for Apache
1.3refer to http://httpd.apache.org/docs/1.3/misc/perf-tuning.html




We can't seem to find a way to the tell the legitimate
> from ddos requests. Ideally we would be able to lower the
> "total amount of time it takes to receive a GET request"
> (and POST headers without file uploads accordingly)
> seperately from the rest of the timeouts. But even apache 2.2
> knows but a general timeout parameter.
> If we would be able to tell the bad requests and get their
> IP address, we could forward them to our front firewall
> or even the ISP and have their SYNs dropped.


Unfortunately, with HTTP DoS attacks, there is no way to tell if a request
is an attack by looking at the individual request itself.  They can either
open the connection and then send nothing, as you mentioned previously, or
they can go ahead and send a legit request.  The problem is the shear number
of requests being recieved.  There are some Apache configs that can be
updated, but you really need to have a separate module/program to monitor
these requests over time.

Mod_Evasive is a good module for this -
http://www.zdziarski.com/projects/mod_evasive/.  The only real short-coming
with Mod_Evasive is that it doesn't use shared memory.  This means that it
can only inspect one child/thread httpd process.  So, it a client sends too
many requests in the threshold time interval specified, it can identifiy it
a block it.  This is effective at catching HTTP DoS/Brute Force attacks that
levarage KeepAlives to pipeline their requests over on chile process.  If,
on the other hand, a client doesn't use keepalives and instead spawns a new
child process for each and every request, then Mod_Evasive will not catch
this.



I have browsed the mod_security 2.0 reference, but i did not
> find the silver bullet. So the question is, can mod_security
> do anything in this regard or do you fellow readers have any
> other smart ideas?


 Ivan Ristic has some tools over at www.apachesecurity.net/tools/ that will
help with identifying DoS attacks.  A script called httpd-guardian can
help.  Here is the info from the modsecurity documentation page -
http://www.modsecurity.org/documentation/modsecurity-apache/1.9.3/modsecurity-manual.html#N10B1C

   Guardian log

Since 1.9 ModSecurity supports a new directive, SecGuardianLog, that is
designed to send all access data to another program using the piped logging
feature. Since Apache is typically deployed in a multi-process fashion,
making information sharing difficult, the idea is to deploy a single
external process to observe all requests in a stateful manner, providing
additional protection.

Development of a state of the art external protection tool will be a focus
of subsequent ModSecurity releases. However, a fully functional tool is
already available as part of the Apache httpd tools project (
http://www.apachesecurity.net/tools/). The tool is called httpd-guardian and
can be used to defend against Denial of Service attacks. It uses the
blacklist tool (from the same project) to interact with an iptables-based
(Linux) or pf-based (*BSD) firewall, dynamically blacklisting the offending
IP addresses. It can also interact with SnortSam (http://www.snortsam.net).
Assuming httpd-guardian is already configured (look into the source code for
the detailed instructions) you only need to add one line to your Apache
configuration to deploy it:

SecGuardianLog |/path/to/httpd-guardian

By default httpd-guardian will defend against clients that send more 120
requests in a minute, or more than 360 requests in five minutes.

It might be helpful to have a variable in mod_security, that
> would tell me something about the time it took from the
> initialisation of the request to the processing in mod_security.
> Obviously in mod_security, there is TIME_EPOCH, but how to
> tell when the connection started?


Take a look at the CustomLog Format options -
http://httpd.apache.org/docs/2.2/mod/mod_log_config.html#formats

There are a few format strings that deal with time intervals between when a
request was received and how long it takes to fulfill the request -

  %t Time the request was received (standard english format)
  %T The time taken to serve the request, in seconds.

These could potentially be correlated to try and identify DoS attacks.


However, this idea is built on the premise, that the request
> arrives in mod_security at last. This means the header has
> to be sent. How about something like a watchdog timer that
> activates some special mod_security processing phase for
> requests, that never make it to Phase 1 - request headers.
> Even those that do not terminate the the ssl-handshake
> should be considered.


This would probably need to be handled by back-end log monitoring of the SSL
logs.  Keep in mind that SSL happens right between the OSI layers 6 & 7.
ModSecurity doesn't jump in until the low level SSL encryption stuff
finishes.

Hope this info helps.

-- 
Ryan C. Barnett
Web Application Security Consortium (WASC) Member
CIS Apache Benchmark Project Lead
SANS Instructor, GCIA, GCFA, GCIH, GSNA, GCUX, GSEC
Author: Preventing Web Attacks with Apache


Any thoughts are appreciated,
>
> Christian Folini
>
> --
> The fool doth think he is wise, but the wise man knows himself to be
> a fool.
> -- William Shakespeare in As You Like It
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job
> easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>

Re: [mod-security-users] DDOS on the appl level, timeouts and blacklisting

[Ivan R. <iva...@gm...>]

On 8/21/06, Christian Folini <chr...@ti...> wrote:
> Hi there,
>
> There have been a couple of meetings regarding ddos threats
> against our ssl sites. So far no attack occurred, but we trying
> to be prepared.

Hi Christian,

I have thought a lot about this subject. Unfortunately I have little
time to respond in depth right now but I promise to follow up when I
get back in two weeks' time.

Ryan has already provided a very good answer. httpd-guardian should
work but you need to put some effort to install it.

ModSecurity 2 can actually track request rate per, well, anything, but
you are likely to want to look at the request rate per IP address. It
works something like this (not tested):

# initialise IP tracking, then update variable to force the collection
# to be updated on disk (ModSecurity only updates data when it changes.)
SecAction initcol:ip=%{REMOTE_ADDR},setvar:ip.dummy=1,nolog,pass

Once the above line executes the variable IP.UPDATE_RATE should
contain the number of updates (request) per minute.

The only aspect of the above configuration I haven't tested is speed.
At the moment ModSecurity uses a SDBM database to track persistent
data. It is probably going to be fast but not as fast as an in-memory
solution. But that's just an implementation detail, I expect an
improved (faster) persistent storage mechanism to be added reasonably
soon.

Also, I am happy to add a few variables to track the time elapsed
since the connection was created, and also to track the speed at which
the data is being received. E.g. if someone is sending data very
slowly he's probably trying to DoS the server.

-- 
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall

Re: [mod-security-users] DDOS on the appl level, timeouts and blacklisting

[De V. R. <Ric...@bm...>]

What is everyone's opinion on mod_evasive, found here:
http://www.zdziarski.com/projects/mod_evasive/ ?

-----Original Message-----
From: mod...@li...
[mailto:mod...@li...] On Behalf Of
Ivan Ristic
Sent: Wednesday, August 23, 2006 3:56 PM
To: Christian Folini
Cc: mod...@li...
Subject: Re: [mod-security-users] DDOS on the appl level,timeouts and
blacklisting

On 8/21/06, Christian Folini <chr...@ti...> wrote:
> Hi there,
>
> There have been a couple of meetings regarding ddos threats against=20
> our ssl sites. So far no attack occurred, but we trying to be=20
> prepared.

Hi Christian,

I have thought a lot about this subject. Unfortunately I have little
time to respond in depth right now but I promise to follow up when I get
back in two weeks' time.

Ryan has already provided a very good answer. httpd-guardian should work
but you need to put some effort to install it.

ModSecurity 2 can actually track request rate per, well, anything, but
you are likely to want to look at the request rate per IP address. It
works something like this (not tested):

# initialise IP tracking, then update variable to force the collection #
to be updated on disk (ModSecurity only updates data when it changes.)
SecAction initcol:ip=3D%{REMOTE_ADDR},setvar:ip.dummy=3D1,nolog,pass

Once the above line executes the variable IP.UPDATE_RATE should contain
the number of updates (request) per minute.

The only aspect of the above configuration I haven't tested is speed.
At the moment ModSecurity uses a SDBM database to track persistent data.
It is probably going to be fast but not as fast as an in-memory
solution. But that's just an implementation detail, I expect an improved
(faster) persistent storage mechanism to be added reasonably soon.

Also, I am happy to add a few variables to track the time elapsed since
the connection was created, and also to track the speed at which the
data is being received. E.g. if someone is sending data very slowly he's
probably trying to DoS the server.

--
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall

------------------------------------------------------------------------
-
Using Tomcat but need to do more? Need to support web services,
security?
Get stuff done quickly with pre-integrated technology to make your job
easier Download IBM WebSphere Application Server v.1.0.1 based on Apache
Geronimo
http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=3D=
121642
_______________________________________________
mod-security-users mailing list
mod...@li...
https://lists.sourceforge.net/lists/listinfo/mod-security-users

Re: [mod-security-users] DDOS on the appl level, timeouts and blacklisting

[Christian F. <chr...@ti...>]

On Wed, Aug 23, 2006 at 04:17:21PM -0500, De Vries, Richard wrote:
> What is everyone's opinion on mod_evasive, found here:
> http://www.zdziarski.com/projects/mod_evasive/ ?

Ryan has mentioned it in his message. It looks like an interesting
tool when an attacker wants to eat your bandwidth and hammers
you using http keepalive. But if he just tries to block your
threads/processes, when mod_evasive can't help you. This
is due to the lack of shared memory. As a consequence every
process/thread remains isolated from the other ones and
mod_evasive can't tell what's going on.

regs,

Christian

> 
> -----Original Message-----
> From: mod...@li...
> [mailto:mod...@li...] On Behalf Of
> Ivan Ristic
> Sent: Wednesday, August 23, 2006 3:56 PM
> To: Christian Folini
> Cc: mod...@li...
> Subject: Re: [mod-security-users] DDOS on the appl level,timeouts and
> blacklisting
> 
> On 8/21/06, Christian Folini <chr...@ti...> wrote:
> > Hi there,
> >
> > There have been a couple of meetings regarding ddos threats against 
> > our ssl sites. So far no attack occurred, but we trying to be 
> > prepared.
> 
> Hi Christian,
> 
> I have thought a lot about this subject. Unfortunately I have little
> time to respond in depth right now but I promise to follow up when I get
> back in two weeks' time.
> 
> Ryan has already provided a very good answer. httpd-guardian should work
> but you need to put some effort to install it.
> 
> ModSecurity 2 can actually track request rate per, well, anything, but
> you are likely to want to look at the request rate per IP address. It
> works something like this (not tested):
> 
> # initialise IP tracking, then update variable to force the collection #
> to be updated on disk (ModSecurity only updates data when it changes.)
> SecAction initcol:ip=%{REMOTE_ADDR},setvar:ip.dummy=1,nolog,pass
> 
> Once the above line executes the variable IP.UPDATE_RATE should contain
> the number of updates (request) per minute.
> 
> The only aspect of the above configuration I haven't tested is speed.
> At the moment ModSecurity uses a SDBM database to track persistent data.
> It is probably going to be fast but not as fast as an in-memory
> solution. But that's just an implementation detail, I expect an improved
> (faster) persistent storage mechanism to be added reasonably soon.
> 
> Also, I am happy to add a few variables to track the time elapsed since
> the connection was created, and also to track the speed at which the
> data is being received. E.g. if someone is sending data very slowly he's
> probably trying to DoS the server.
> 
> --
> Ivan Ristic, Technical Director
> Thinking Stone, http://www.thinkingstone.com
> ModSecurity: Open source Web Application Firewall
> 
> ------------------------------------------------------------------------
> -
> Using Tomcat but need to do more? Need to support web services,
> security?
> Get stuff done quickly with pre-integrated technology to make your job
> easier Download IBM WebSphere Application Server v.1.0.1 based on Apache
> Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> 
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users

Re: [mod-security-users] DDOS on the appl level, timeouts and blacklisting

[Christian F. <chr...@ti...>]

Hello,

A couple of weeks ago i asked for ideas on a defense against
a type of DoS attacks that could be described as
request delaying on the application level (I do not know the
proper technical term so far): Basically, you just open
a connection to a webserver and then you do everything
to keep the connection open while keeping the webserver in the read
phase. You can spend 300 seconds in the read request
header phase by default on a default apache and there won't be 
a single line in one of the logfiles.

When addressing the problem first, i thought mod_security
may be able to help me. I proposed kind of a watchdog timer
to handle the situation, but this would not be generic enough.

A private exchange with Ryan and Ivan evolved. This directed me
away from mod_security. I have done my homework now and i
return to this list. My message is somewhat offtopic, but i
think the general level of knowhow is very high here, so i would
appreciate your imput.

Here we go:

Apache knows a timeout value that is described at
http://httpd.apache.org/docs/2.0/mod/core.html#timeout

> 1. The total amount of time it takes to receive a GET request.
> 2. The amount of time between receipt of TCP packets on a POST or PUT request.
> 3. The amount of time between ACKs on transmissions of TCP packets in responses.

(I tested this, and it seems that the [1] is not the total time, but
also the time between TCP packets too.)

This value is 300 seconds by default and this is very high. It is safe
to lower it to below 60 seconds, maybe even 5 but it does not mitigate
the problem, as there are countless ways to escape the timeout.
Among these is the possibility to slow down the ssl handshake, the
read request header phase, the read request body phase, the file upload
phase and especially sly: do a proper first request and slow down the
second when keepalive is active.

As mentioned previously, apache does not even notice. The apache logfile
is written at the very end of a transaction and even the debug error
loglevel does not reveal anything. There are a few indications in the
ssl-log, but only when slowing down that handshake. The rest
goes all unnoticed. Still the logfiles are the key, as a single apache
thread or process has a very limited scope. See mod_evasive that
protects from DoS attacks, but only as long as the attacker uses
keepalive.
-> A single apache thread or process lacks the "whole picture" and
it can not do a proper assessment of the probability of an attack.

Therefore i propose to defend in 3 seperate steps:
 - reconnaissance (apache log)
 - analysis (external process scanning logfiles in realtime)
 - defense (ip blacklisting or anything else)

It can be argued, that 2 and 3 should be within a single
mechanism, but the important thing is to seperate reconnaissance
and analysis. 

What do we need to notice the request delaying attack when
it is happening? We need information about the state of
a request. This is what mod_status is about, but mod_status
does not tell us the ip address of a client host during the
read phase and furthermore, accessing http://localhost/status
may be a hopeless task during a DoS attack.
So i opt for writing more log messages than apache does
today.

To be more precise, i want to write the following state messages:
 - open connection (pre-connection)
 - finished read request header (post-read-header)
 - finished read request body (access-checker)
 - finished request (log-transaction)
 - close connection (hook missing)

I have listed the respective apache 2.0 hooks in brackets. I have looked
at existing modules as mod_evasive, mod_log_forensic, mod_security
to find the hook suiteable to the task.

The problem is, that there is nothing like a close connection hook.
However, in http://hypermail.linklord.com/new-httpd/2004/Feb/3597.html
i have found a patch from Joe Orton, that brings a finish_connection hook.

At cut down version of the patch looks as follows:

--- httpd-2.0.59/server/connection.c    2006-07-12 09:40:55.000000000 +0200
+++ httpd-2.0.59.patched/server/connection.c    2006-08-31 16:49:37.880730953 +0200
@@ -36,12 +36,15 @@
	     APR_HOOK_LINK(create_connection)
	    APR_HOOK_LINK(process_connection)
	    APR_HOOK_LINK(pre_connection)
+           APR_HOOK_LINK(finish_connection)
 )
 AP_IMPLEMENT_HOOK_RUN_FIRST(conn_rec *,create_connection,
			     (apr_pool_t *p, server_rec *server, apr_socket_t *csd, long conn_id, void *sbh, apr_bucket_alloc_t *alloc),
			     (p, server, csd, conn_id, sbh, alloc), NULL)
 AP_IMPLEMENT_HOOK_RUN_FIRST(int,process_connection,(conn_rec *c),(c),DECLINED)
 AP_IMPLEMENT_HOOK_RUN_ALL(int,pre_connection,(conn_rec *c, void *csd),(c, csd),OK,DECLINED)
+AP_IMPLEMENT_HOOK_RUN_ALL(int,finish_connection,(conn_rec *c),(c),OK,DECLINED)
+
 /*
  * More machine-dependent networking gooo... on some systems,
  * you've got to be *really* sure that all the packets are acknowledged
@@ -174,6 +177,7 @@
     
     if (!c->aborted) {
	 ap_run_process_connection(c);
+       ap_run_finish_connection(c);
     }
 }
 
--- httpd-2.0.59/include/http_connection.h      2006-07-12 09:40:55.000000000 +0200
+++ httpd-2.0.59.patched/include/http_connection.h      2006-08-31 16:50:25.069198657 +0200
@@ -104,6 +104,19 @@
  */
 AP_DECLARE_HOOK(int,process_connection,(conn_rec *c))
 
+/**
+ * This hook allows connection-level filters to perform any necessary
+ * processing before a connection is closed.
+ * @param c The connection which is about to be closed
+ * @return OK or DECLINED
+ * @deffunc int ap_run_finish_connection(conn_rec *c)
+ */
+AP_DECLARE_HOOK(int,finish_connection,(conn_rec *c))
+
+#ifdef __cplusplus
+}
+#endif
+
 /* End Of Connection (EOC) bucket */
 
 AP_DECLARE_DATA extern const apr_bucket_type_t ap_bucket_type_eoc;






With this, we get all the necessary hooks to write a new logging
module that writes all necessary data to the error_log (as a starting
point).

I have called this module mod_log_reqphase as a working title.
Below is the main part of the module:


static int log_reqphase_pre_conn(conn_rec *c, void *csd)
{
    ap_log_error(APLOG_MARK, APLOG_INFO, 0, c->base_server, "mod_log_reqphase: init connection from client %s, port %u", c->remote_ip, c->remote_addr->port);
    
    return OK;
}

static int log_reqphase_post_read_request(request_rec *r)
{
    ap_log_error(APLOG_MARK, APLOG_INFO, 0, r->connection->base_server, "mod_log_reqphase: finished reading headers from client %s, port %u", r->connection->remote_ip, r->connection->remote_addr->port);
    
    return OK;
}

static int log_reqphase_log_transaction(request_rec *r)
{
    ap_log_error(APLOG_MARK, APLOG_INFO, 0, r->connection->base_server, "mod_log_reqphase: finished request from client %s, port %u", r->connection->remote_ip, r->connection->remote_addr->port);
    
    return OK;
}

static int log_reqphase_finish_connection(conn_rec *c)
{
    ap_log_error(APLOG_MARK, APLOG_INFO, 0, c->base_server, "mod_log_reqphase: closed connection from client %s, port %u", c->remote_ip, c->remote_addr->port);
    
    return OK;
}

static void log_reqphase_register_hooks(apr_pool_t *p)
{

  ap_hook_pre_connection(log_reqphase_pre_conn, NULL, NULL, APR_HOOK_MIDDLE);
  ap_hook_post_read_request(log_reqphase_post_read_request,NULL,NULL,APR_HOOK_MIDDLE);
  ap_hook_log_transaction(log_reqphase_log_transaction,NULL,NULL,APR_HOOK_MIDDLE);
  ap_hook_finish_connection(log_reqphase_finish_connection,NULL,NULL,APR_HOOK_MIDDLE);

}

The access checker is not yet present in the code piece above. That's
because i just did a proof of concept so far.


Now that we get the necessary infos, how can we analyse them to detect
the attack?

- attack #1: open a connection, remain idle
  detection: the analyzer notices a read-request-header phase not being
   finished n seconds after the opening of the connection
- attack #2: delaying the ssl handshake
  detection: the analyzer notices a read-request-header phase not being
   finished n seconds after the opening of the connection
- attack #3: delaying the request header read phase by sending in one (bogus-)
   header after other, each one in a single tcp packet
  detection: the analyzer notices a read-request-header phase not being
   finished n seconds after the opening of the connection
- attack #4: delaying the request body read phase
  detection: the analyzer notices the phase is not done within n
   seconds. Optionally an additional mod_security check
   after the read header phase to make sure a post is feasible to the
   URL in question at all. Actually in a standard application, an non-authorized 
   client should only be able to make a post to the login-form. 
   However, it is a bit tricky to detect the client being unauthorised 
   in mod_security after the read request header phase already.
   All i can see is a check against a list of client ip addresses with
   open sessions, provided by the authentication layer of the
   application running.
- attack #5: Delaying a file upload
  detection: File uploads can be big. I would try to check with
  mod_security wether the client is authorised to upload at all.
  Otherwise this may be quite tricky for the external analyzer.
  How to tell a slow modem over a weak connection from a DoS attacker...
- attack #6: Using keepalive to delay a subsequent request
  detection: After a finished request, the connection has to be closed
  after k seconds or a new read header phase has to be finished after
  k + n seconds.

I do not want to dive into the actual defense once a client is
identified as an attacker. However, i would have this log analysis
running during normal operation and try to tune it to minimize
the false positives. Then once a serious attack is starting, the
defense action can be triggered either automatically or by hand.

I do not know, if this makes any sense at all. I wonder what you guys 
think and welcome any feedback.
The point is, that network layer DoS attacks are getting harder and
harder and i see the application level getting more attention.
And i think the application level is not yet prepared in the case
of apache.

best regards,

Christian Folini

-- 
First you make it, then it works, then you invite people to 
make it better. 
-- Eben Moglen, Free Software Foundation

Re: [mod-security-users] DDOS on the appl level, timeouts and blacklisting

[Ivan R. <iva...@gm...>]

On 9/16/06, Christian Folini <chr...@ti...> wrote:
> Hello,
>
> ...
>
> A private exchange with Ryan and Ivan evolved. This directed me
> away from mod_security.

No, no, no, we have to find a way to get you back now :)


> A single apache thread or process lacks the "whole picture" and
> it can not do a proper assessment of the probability of an attack.

With ModSecurity 2.0 you actually get a persistent data store that is
shared across processes and threads. But even that will not solve a
problem if you have a cluster of web servers. (At least not until I
release a cluster-ready version of ModSecurity, that is.)


> Therefore i propose to defend in 3 seperate steps:
>  - reconnaissance (apache log)
>  - analysis (external process scanning logfiles in realtime)
>  - defense (ip blacklisting or anything else)

I believe it was Ryan who mentioned httpd_guardian in response to your
recent email. On a single box httpd_guardian will connect to Apache
via the piped logging mechanism. If you need to support a cluster it
will work with the Spread toolkit to receive the required information
in real time. This utility is at this point capable of detecting brute
force attacks, but it can (and should) be extended to handle DoS
attacks you describe.


> It can be argued, that 2 and 3 should be within a single
> mechanism, but the important thing is to seperate reconnaissance
> and analysis.

I fully agree.

>
> What do we need to notice the request delaying attack when
> it is happening? We need information about the state of
> a request. This is what mod_status is about, but mod_status
> does not tell us the ip address of a client host during the
> read phase and furthermore, accessing http://localhost/status
> may be a hopeless task during a DoS attack.

Slightly off-topic but potentially useful to some: mod_backdoor
(http://people.apache.org/~trawick/) keeps a separate process ready
for access for cases like that.


> The problem is, that there is nothing like a close connection hook.
> However, in http://hypermail.linklord.com/new-httpd/2004/Feb/3597.html
> i have found a patch from Joe Orton, that brings a finish_connection hook.

But such hook might not be necessary. Each connection comes with its
own memory pool, which is destroyed after the connection is closed.
You can simply register a call-back with this memory pool and it will
be invoked just before the pool is destroyed.


> I do not know, if this makes any sense at all. I wonder what you guys
> think and welcome any feedback.

I have some concerns (some of which you identified already):

1. A large number of messages may be needed per connection. This could
be a performance bottleneck for large installations.

2. I don't think the Apache error log is the right place to write
these messages. The sheer volume of messages will hide other
potentially interesting log entries.

3. It is not possible for such external checker to take care of large
request bodies and file uploads.

How about this:

1. We keep the log-oriented approach.

2. In addition to working with Apache hooks as you did, we write an
Apache input filter that looks at raw data. We look at the number of
bytes received on every invocation and the time it took for the data
to arrive. This could allow us to enforce:

2.1 A time limit for the request headers to arrive.

2.2 Minimum communication speed.

A log entry is made only when a problem is detected.

3. Because log entries now appear only on attacks we can use the error
log for this.

One drawback of the filter approach, though, is that it is passive -
it can only do its thing when the next batch of data arrives. This
shouldn't be a big problem provided the Apache timeout value is low
enough.

-- 
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall

Re: [mod-security-users] DDOS on the appl level, timeouts and blacklisting

[Christian F. <chr...@ti...>]

Hi there,

On Mon, Sep 18, 2006 at 03:18:51PM +0100, Ivan Ristic wrote:
> >A private exchange with Ryan and Ivan evolved. This directed me
> >away from mod_security.
> 
> No, no, no, we have to find a way to get you back now :)

Oh, you are on good tracks.
 
> >A single apache thread or process lacks the "whole picture" and
> >it can not do a proper assessment of the probability of an attack.
> 
> With ModSecurity 2.0 you actually get a persistent data store that is
> shared across processes and threads. But even that will not solve a
> problem if you have a cluster of web servers. (At least not until I
> release a cluster-ready version of ModSecurity, that is.)

This ability makes things a lot easier with modsecurity. I have
played with the ability and it is really cool. Too many ideas spring to
mind...

Still the basic problem with the attack i have described is for
me, that we never get into a position where modsecurity is able
to check something, as the attacker never reaches the post_read_request 
hook with his request. But you addressing this below.

> I believe it was Ryan who mentioned httpd_guardian in response to your
> recent email. On a single box httpd_guardian will connect to Apache
> via the piped logging mechanism. If you need to support a cluster it
> will work with the Spread toolkit to receive the required information
> in real time. This utility is at this point capable of detecting brute
> force attacks, but it can (and should) be extended to handle DoS
> attacks you describe.

I was planning to start with httpd_guardian as a base and extend
it in the direction described. Spread makes it a valuable tool.
It is good you included it from the start.

> Slightly off-topic but potentially useful to some: mod_backdoor
> (http://people.apache.org/~trawick/) keeps a separate process ready
> for access for cases like that.

good hint. thanks. One more apache module i did not know. :)

> >The problem is, that there is nothing like a close connection hook.
> >However, in http://hypermail.linklord.com/new-httpd/2004/Feb/3597.html
> >i have found a patch from Joe Orton, that brings a finish_connection hook.
> 
> But such hook might not be necessary. Each connection comes with its
> own memory pool, which is destroyed after the connection is closed.
> You can simply register a call-back with this memory pool and it will
> be invoked just before the pool is destroyed.

Thanks for pointing this out. I am not really a coder. :)

> I have some concerns (some of which you identified already):
> 
> 1. A large number of messages may be needed per connection. This could
> be a performance bottleneck for large installations.

Yes. I see this too.

> 2. I don't think the Apache error log is the right place to write
> these messages. The sheer volume of messages will hide other
> potentially interesting log entries.

In my testcode i have individual switches for the different
messages so they need not be running all the time in full mode. 
error_log is a starting point as i do not want to have different 
logfile for every possible threat... Still thinking about that.

> 3. It is not possible for such external checker to take care of large
> request bodies and file uploads.

? What do you mean by that? I do not want it to check the body.
I just want it to keep an eye on timeouts. It's modsecurity's
job to check that. And i would use modsecurity to authorise
the client to send a request body at all to this URL.

> How about this:
> 
> 1. We keep the log-oriented approach.
> 
> 2. In addition to working with Apache hooks as you did, we write an
> Apache input filter that looks at raw data. We look at the number of
> bytes received on every invocation and the time it took for the data
> to arrive. This could allow us to enforce:
> 
> 2.1 A time limit for the request headers to arrive.
> 
> 2.2 Minimum communication speed.
> 
> A log entry is made only when a problem is detected.
> 
> 3. Because log entries now appear only on attacks we can use the error
> log for this.

This reduces the logfile a lot. So this is clearly a positive
aspect. The analysis is within apache now, which is much more
efficient, but we can not tweak the rules so easily anymore in
case of an attack - it demands a graceful reload of the apache
now. Maybe even a not so graceful one when we want to free
the threads under attack.
This is modsecurity's problem anyways, but i think it is worth
noting, even if it is no big deal in real life.

Would you add such an input filter to modsecurity?

> One drawback of the filter approach, though, is that it is passive -
> it can only do its thing when the next batch of data arrives. This
> shouldn't be a big problem provided the Apache timeout value is low
> enough.

Hmm. We are thinking about setting up defcon scenarios where we would
set the timeout to 5 seconds or so in case of an attack.

What is if the attacker sends a TCP packet every second. Will this
trigger the input filter? As far as i have tested, the timeout is
never reached.

Thanks for your thoughts. We are advancing.

Christian


-- 
Given the choice between two theories, take the one which is funnier.
--- Blore's Razor, Author unknown