Christian,
My comments are inline below.  You are correct in that web-based DDoS/DoS attacks are tough to handle, especially at layer 7.
 
On 8/21/06, Christian Folini <christian.folini@time-machine.ch > wrote:
Hi there,

There have been a couple of meetings regarding ddos threats
against our ssl sites. So far no attack occurred, but we trying
to be prepared.

It seems simple to initiate a valid connection
with one of our reverse proxies and then do basically nothing
until the timeout occurs. It's 300 seconds by default.
Let's say we swallow a couple of thousands of these connections,
but we are no challenge for one of the bigger botnets out there.
The network guys are constantly building up their defense lines
against ddos attacks on the network layers, but our domain
OSI layer 7, looks quite undefended in this aspect.
 
There are a few OS TCP stack settings that you should tweak to help defend against the effects of DoS attacks.  Here are some issues to address -
Now in Apache itself, there are a number of config settings you can updated to help.  This data is taken from the Center for Internet Security (CIS) Apache Benchmark Document -
http://www.cisecurity.org/bench_apache.html
 
L1 1.       Denial of Service (DoS) Protective General Directives

Description
It is almost impossible to fully protect against denial of service attacks since they are attacks against the fundamental underpinnings of the Internet.   Denial of service attacks often try to exhaust the system's resources.  For instance, since the system can only wait for a limited number of connection requests to complete, an attacker will flood a system with a bunch of connection requests that will never be completed.   The server will keep the connection request open until either the connection has been made or a specified length of time has gone by.  Once the attacker has filled the pending connection queue, a small trickle of new requests will keep the server unavailable to legitimate requests.   Apache has several directives available that provide some protective capabilities against denial of services attacks.
 

 

We will be verifying/updating the following Apache directives:

One way of attacking systems on the Internet is to try to prevent the target system from operating correctly by overloading it. This is called a 'denial of service' attack. One method of doing this is to open multiple connections to a server and never close them. The more connections the server has open at once, the more resources are tied up holding details of those connection, which can lead to increased load and eventually to the server running out of resources.  The Timeout directive tells the server how long to wait to receive a GET request, the amount of time between receipt of TCP packets on a POST or PUT request, or the amount of time between ACKs on transmissions of TCP packets in responses.  

In order to prevent a denial of service attack from shutting down our web server, we need to change the default setting of 300 (which is 5 minutes) to 60 (which is 1 minute).  You may even adjust this setting to be lower than 60.

 

In order to make our web server more efficient, and therefore more resistant to DoS attacks, we want to allow TCP KeepAlives.  This setting will allow for multiple HTTP requests to be serviced across one connection.  Without this setting, a new Apache server would have to be spawned for every subsequent HTTP request for gifs, etc. Recommended value is On.

 

 

This directive will limit the time the Apache web server will wait in seconds for a KeepAlive request to be completed.  Setting KeepAliveTimeout to a high value may cause performance problems in heavily loaded servers. The higher the timeout, the more server processes will be kept occupied waiting on connections with idle clients.   If this number is set too high, then a DoS attack could be more successful. Recommend value is 15 which is also the default value.

 

  

 

 

Taking together the directives StartServers, StartServers, MaxSpareServers and MaxClients, along with the MPM (MultiProcessing Module) work to establish a reasonable yet dynamic number of child processes and/or threads. Starting with Apache 1.3 the process creation process was dramatically improved so that tweaking the StartServers, StartServers, MaxSpareServers and MaxClients settings is unnecessary for most situations.   Therefore the default settings are fine for most usages. For very high traffic servers, and optimal performance consult the Apache performance recommendations at http://httpd.apache.org/docs/2.0/misc/perf-tuning.html  or for Apache 1.3 refer to http://httpd.apache.org/docs/1.3/misc/perf-tuning.html  

 
 

We can't seem to find a way to the tell the legitimate
from ddos requests. Ideally we would be able to lower the
"total amount of time it takes to receive a GET request"
(and POST headers without file uploads accordingly)
seperately from the rest of the timeouts. But even apache 2.2
knows but a general timeout parameter.
If we would be able to tell the bad requests and get their
IP address, we could forward them to our front firewall
or even the ISP and have their SYNs dropped.
 
Unfortunately, with HTTP DoS attacks, there is no way to tell if a request is an attack by looking at the individual request itself.  They can either open the connection and then send nothing, as you mentioned previously, or they can go ahead and send a legit request.  The problem is the shear number of requests being recieved.  There are some Apache configs that can be updated, but you really need to have a separate module/program to monitor these requests over time. 
 
Mod_Evasive is a good module for this - http://www.zdziarski.com/projects/mod_evasive/ .  The only real short-coming with Mod_Evasive is that it doesn't use shared memory.  This means that it can only inspect one child/thread httpd process.  So, it a client sends too many requests in the threshold time interval specified, it can identifiy it a block it.  This is effective at catching HTTP DoS/Brute Force attacks that levarage KeepAlives to pipeline their requests over on chile process.  If, on the other hand, a client doesn't use keepalives and instead spawns a new child process for each and every request, then Mod_Evasive will not catch this.
 
 

I have browsed the mod_security 2.0 reference, but i did not
find the silver bullet. So the question is, can mod_security
do anything in this regard or do you fellow readers have any
other smart ideas?
 
Ivan Ristic has some tools over at www.apachesecurity.net/tools/ that will help with identifying DoS attacks.  A script called httpd-guardian can help.  Here is the info from the modsecurity documentation page - http://www.modsecurity.org/documentation/modsecurity-apache/1.9.3/modsecurity-manual.html#N10B1C
 

Guardian log

Since 1.9 ModSecurity supports a new directive, SecGuardianLog, that is designed to send all access data to another program using the piped logging feature. Since Apache is typically deployed in a multi-process fashion, making information sharing difficult, the idea is to deploy a single external process to observe all requests in a stateful manner, providing additional protection.

Development of a state of the art external protection tool will be a focus of subsequent ModSecurity releases. However, a fully functional tool is already available as part of the Apache httpd tools project ( http://www.apachesecurity.net/tools/). The tool is called httpd-guardian and can be used to defend against Denial of Service attacks. It uses the blacklist tool (from the same project) to interact with an iptables-based (Linux) or pf-based (*BSD) firewall, dynamically blacklisting the offending IP addresses. It can also interact with SnortSam ( http://www.snortsam.net). Assuming httpd-guardian is already configured (look into the source code for the detailed instructions) you only need to add one line to your Apache configuration to deploy it:

SecGuardianLog |/path/to/httpd-guardian

By default httpd-guardian will defend against clients that send more 120 requests in a minute, or more than 360 requests in five minutes.


It might be helpful to have a variable in mod_security, that
would tell me something about the time it took from the
initialisation of the request to the processing in mod_security.
Obviously in mod_security, there is TIME_EPOCH, but how to
tell when the connection started?
 
Take a look at the CustomLog Format options -
http://httpd.apache.org/docs/2.2/mod/mod_log_config.html#formats
 
There are a few format strings that deal with time intervals between when a request was received and how long it takes to fulfill the request -
 
%t Time the request was received (standard english format)
%T The time taken to serve the request, in seconds.
 
These could potentially be correlated to try and identify DoS attacks.
 

However, this idea is built on the premise, that the request
arrives in mod_security at last. This means the header has
to be sent. How about something like a watchdog timer that
activates some special mod_security processing phase for
requests, that never make it to Phase 1 - request headers.
Even those that do not terminate the the ssl-handshake
should be considered.
 
This would probably need to be handled by back-end log monitoring of the SSL logs.  Keep in mind that SSL happens right between the OSI layers 6 & 7.  ModSecurity doesn't jump in until the low level SSL encryption stuff finishes.
 
Hope this info helps.
 
--
Ryan C. Barnett
Web Application Security Consortium (WASC) Member
CIS Apache Benchmark Project Lead
SANS Instructor, GCIA, GCFA, GCIH, GSNA, GCUX, GSEC
Author: Preventing Web Attacks with Apache
 

Any thoughts are appreciated,

Christian Folini

--
The fool doth think he is wise, but the wise man knows himself to be
a fool.
-- William Shakespeare in As You Like It

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
mod-security-users mailing list
mod-security-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users