Hi Breno,

 

> BTW...  what are u thinking about the SSL stuff ?

As mentioned, I do not know enough about the internals of the SSL protocol. However, I see various ways to
delay a HTTPS request. More and more thoughts and tools are being published. But AFAIK, so far there was nothing
regarding the delaying of an SSL handshake. I mentioned this a few years back on the ModSec mailinglist but
got no response.

So either it is not really possible - or it is just complicated to achieve and thus potentially difficult to defend against.
Maybe Ivan has something to say.

Cheers,

Christian

 

 

 

Von: Breno Silva [mailto:breno.silva@gmail.com]
Gesendet: Dienstag, 30. November 2010 21:21
An: Folini Christian, IT222 extern
Cc: mod-security-users@lists.sourceforge.net
Betreff: Re: [mod-security-users] [Owasp-modsecurity-core-rule-set] Advanced Topic of the Week: Mitigating Slow HTTP DoS Attacks

 

The other msg was to another mail :)

BTW...  what are u thinking about the SSL stuff ?

On Tue, Nov 30, 2010 at 12:59 AM, <christian.folini@post.ch> wrote:

Hi Breno,

 

Thank you for your extensive efforts and research on real-world behavior of your new feature.

 

> As you can see we have the two side of the coin… nothing is perfect in the life :-).

> If you don't turn on the SecReadStateLImit the attacker may cause a problem from all the world that is

> trying to contact your server and nobody will be able to connect to your site.In another hand if you enable

> the defense you will lost some connections from the attacker IP address (lets suppose he is behind a NAT for example),

> but the rest of the world still will be able to connect to your site.

 

That is more or less what I expected from this feature. I suggest you make this clear in the documentation.

As such I believe it is a defense utility which you enable in case of attack, but not in the base config,

since the negative side-effect comes into play immediately, also in absence of any ongoing attack.

 

> During my tests with slowloris it required hundreds (sometimes thousands) of sockets available on the attacker

> side to have success, which means that a single ip will generate a lot of concurrent connections. So i understand

> that a number of 100 to SecReadStateLImit will be good for many environments, however people may spending

> some time tuning it to their environment.

> Also this feature can be a way to detect the ip address of an attacker, because without this apache will only

> print a lot of request headers error msgs without any source. And as it is a low bandwidth attack if very difficult

> to detect using a network device.

 

You can get the info from netstat as long as it is DoS. If you face a DDoS attack, then the defense gets

tricky – and SecReadStateLImit won’t help you anymore.

 

> Keep it in your sleeve… maybe it can help you sometime.

 

Exactly. Thanks for another tool in that case. They are desperately needed.

 

And from the other message:

> An improvement for it will be to find a way to identify the larger upload connections and do not count it.

 

I do not understand you here. What do you mean exactly?

 

 

A sidenote: Has anybody ever thought of a request delaying / slowloris attack targeting the SSL handshake?

Is that technically feasible – and how would one defend against it? Sorry if this leads away from the ModSec

discussion, but I think it’s an intriguing idea and I never saw it discussed anywhere.

Fjodor used to have a blog post online touching on other very evil request delaying techniques. Unfortunately

he has pulled it back. Maybe there is a copy somewhere in the web.

 

Regs,

 

Christian

 

 

 

 

 

Von: Breno Silva [mailto:breno.silva@gmail.com]
Gesendet: Donnerstag, 25. November 2010 01:39
An: Folini Christian, IT2
22 extern


Cc: RBarnett@trustwave.com; mod-security-users@lists.sourceforge.net
Betreff: Re: [mod-security-users] [Owasp-modsecurity-core-rule-set] Advanced Topic of the Week: Mitigating Slow HTTP DoS Attacks

 

Hi Christian,

I spent some time doing some tests to share with you. I setup a webserver 192.168.0.102 and two clients (192.168.0.101 and 192.168.0.100)

- First, let's see what happens when we send some requests. The server is under attack (192.168.0.100) without the defense:


< From the client 192.168.0.101 >

Server Software:        Apache
Server Hostname:        192.168.0.102
Server Port:            80

Document Path:          /www/discr/disc.zip
Document Length:        696374 bytes

Concurrency Level:      10
Time taken for tests:   405.342 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      696623000 bytes
HTML transferred:       696374000 bytes
Requests per second:    2.47 [#/sec] (mean)
Time per request:       4053.420 [ms] (mean)
Time per request:       405.342 [ms] (mean, across all concurrent requests)
Transfer rate:          1678.33 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   25 249.8      3    3000
Processing:    69 4025 4196.3   1576   12466
Waiting:        0 3762 4191.1   1340   11985
Total:         71 4050 4232.2   1578   13559

Percentage of the requests served within a certain time (ms)
  50%   1578
  66%   5268
  75%   8896
  80%   9902
  90%  10716
  95%  11127
  98%  12028
  99%  12193
 100%  13559 (longest request)


<From the attacker 192.168.0.100 >

Server Software:        Apache
Server Hostname:        192.168.0.102
Server Port:            80

Document Path:          /www/disc/disc.zip
Document Length:        696374 bytes

Concurrency Level:      10
Time taken for tests:   438.080 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      696623000 bytes
HTML transferred:       696374000 bytes
Requests per second:    2.28 [#/sec] (mean)
Time per request:       4380.795 [ms] (mean)
Time per request:       438.080 [ms] (mean, across all concurrent requests)
Transfer rate:          1552.90 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    3  30.2      1     955
Processing:    35 4366 4277.3   2039   12568
Waiting:        2 4023 4289.1   1642   12064
Total:         35 4369 4276.6   2044   12570

Percentage of the requests served within a certain time (ms)
  50%   2044
  66%   6185
  75%   9538
  80%  10040
  90%  10755
  95%  11171
  98%  11642
  99%  12103
 100%  12570 (longest request)

- Now, i set SecReadStateLimit to 100. Let's see the results:

<From the client 192.168.0.101>

Server Software:        Apache
Server Hostname:        192.168.0.102
Server Port:            80

Document Path:          /www.limao.com.br/discador/discador_limao.zip
Document Length:        696374 bytes

Concurrency Level:      10
Time taken for tests:   56.275 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      696623000 bytes
HTML transferred:       696374000 bytes
Requests per second:    17.77 [#/sec] (mean)
Time per request:       562.745 [ms] (mean)
Time per request:       56.275 [ms] (mean, across all concurrent requests)
Transfer rate:          12088.88 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   31 283.2      3    3005
Processing:    40  521 491.5    386    3822
Waiting:        2  161 388.7     17    2989
Total:         41  552 572.8    391    4122

Percentage of the requests served within a certain time (ms)
  50%    391
  66%    534
  75%    654
  80%    756
  90%   1142
  95%   1582
  98%   2413
  99%   3467
 100%   4122 (longest request)


<From the attacker 192.168.0.100>

Server Software:        Apache
Server Hostname:        192.168.0.102
Server Port:            80

Document Path:          /www.limao.com.br/discador/discador_limao.zip
Document Length:        696374 bytes

Concurrency Level:      10
Time taken for tests:   13.877 seconds
Complete requests:      1000
Failed requests:        572
   (Connect: 0, Receive: 0, Length: 872, Exceptions: 0)
Write errors:           0
Total transferred:      89167744 bytes
HTML transferred:       89135872 bytes
Requests per second:    72.06 [#/sec] (mean)
Time per request:       138.772 [ms] (mean)
Time per request:       13.877 [ms] (mean, across all concurrent requests)
Transfer rate:          6274.90 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    3  28.8      1     910
Processing:     1  136 379.7      3    2404
Waiting:        0   28 188.1      0    2205
Total:          1  139 384.3      5    2433

Percentage of the requests served within a certain time (ms)
  50%      5
  66%      7
  75%     22
  80%     75
  90%    451
  95%   1061
  98%   1672
  99%   1899
 100%   2433 (longest request)


As you can see we have the two side of the coin… nothing is perfect in the life :-). If you don't turn on the SecReadStateLImit the attacker may cause a problem from all the world that is trying to contact your server and nobody will be able to connect to your site.In another hand if you enable the defense you will lost some connections from the attacker IP address (lets suppose he is behind a NAT for example), but the rest of the world still will be able to connect to your site.

The trick is to set a good value to SecReadStateLimit which is the number of threads that can remain in READ_BUSY_STATE (before request_rec is valid for apache) per ip address. I suggest for almost all environments to understand it as the number of concurrent connections per IP address.

During my tests with slowloris it required hundreds (sometimes thousands) of sockets available on the attacker side to have success, which means that a single ip will generate a lot of concurrent connections. So i understand that a number of 100 to SecReadStateLImit will be good for many environments, however people may spending some time tuning it to their environment.

Also this feature can be a way to detect the ip address of an attacker, because without this apache will only print a lot of request headers error msgs without any source. And as it is a low bandwidth attack if very difficult to detect using a network device.

So… want to say that it is an option. Not a mandatory feature… and have pros and cons… in my point of view as it can save some heads in a company … there are more pros :-)

Keep it in your sleeve… maybe it can help you sometime.

Also if you have an high speed environment and want to try and share with us good values to SecReadStateLimit

Cheers and thanks for the discussion

Breno

On Wed, Nov 24, 2010 at 9:22 AM, Breno Silva <breno.silva@gmail.com> wrote:

If you are using the apache-prefork and receive this kind of attack ... it can kill all your operation system forking too many process. So  a good reason to enable this feature :)

Note that this is not a definitive solution, but can help you a lot during this kind of attack saving a big part of legit connections.

Thanks

Breno

 

On Wed, Nov 24, 2010 at 7:47 AM, Breno Silva <breno.silva@gmail.com> wrote:

Christian,

Yes and No :)

Keep it mind that with small number of threads can handle many requests ... so you can also have a large number of long (BUSY_STATE) legit connections with low threads per ipaddress.

However you u have behind your proxy a lot of user doing a LOT of big uploads to a specific server... and the SecReadStateLimit is very small ... it can reject the connection.

 

On Wed, Nov 24, 2010 at 7:22 AM, <christian.folini@post.ch> wrote:

Thanks Breno,

 

I think I got it.

 

So clients will be save for GET requests, but for long POST requests the mileage may vary.

 

 

Best,

 

Christian

 

 

Von: Breno Silva [mailto:breno.silva@gmail.com]
Gesendet: Mittwoch, 24. November 2010 14:18

Betreff: Re: [mod-security-users] [Owasp-modsecurity-core-rule-set] Advanced Topic of the Week: Mitigating Slow HTTP DoS Attacks

 

This an example

I sent an attack from my PC to my apache server in another machine. And without the protection i sent in the same time using ab ligit connections:

Concurrency Level:      100
Time taken for tests:   32.842 seconds
Complete requests:      500

As you can see it spent too much time to answer my connections


Now i did the same test, but turn on the defense:

Concurrency Level:      100
Time taken for tests:   0.122 seconds
Complete requests:      500

As you can see the legit connections was accepted normally during the attack from the same ip address.

Breno

On Wed, Nov 24, 2010 at 7:10 AM, Breno Silva <breno.silva@gmail.com> wrote:

Hi Christian,

Internally into the modsec you are checking how many threads are in SERVER_BUSY_STATE and limiting the number of this threads per ip address using the SecReadStateLimit. The large part of the proxied connections will leave this state very quickly and the connection will be not counted to be rejected. So a few number of threads should be enough to handle your legit SERVER_BUSY_STATE connections.

Thanks

Breno

 

On Wed, Nov 24, 2010 at 6:31 AM, <christian.folini@post.ch> wrote:

Hi Breno,

 

Could explain the term „bad connection“ a bit? Ryan’s blog post implies a client IP is considered bad

when it has too many connections in read state. Your entry in the CHANGES document reads,

“Add SecReadStateLimit to limit the number of BUSY connections”.

 

I can’t see why a proxy can’t have a lot of legitimate connections

in read state. AFAIK Request Body reading is also considered “read”.

So uploads can remain in READ for a certain time – depending on service.

 

I do not want to pester you too much, but I just want to make sure I

get this correctly – and people are aware that telling good from bad

connections is very tricky. Especially when it comes to request delaying and

you want to make sure you are not locking legitimate users.

 

Best Regs,

 

Christian

 

 

 

 

Von: Breno Silva [mailto:breno.silva@gmail.com]
Gesendet: Mittwoch, 24. November 2010 13:20
An: Folini Christian, IT222 extern
Cc: RBarnett@trustwave.com; mod-security-users@lists.sourceforge.net; owasp-modsecurity-core-rule-set@lists.owasp.org
Betreff: Re: [mod-security-users] [Owasp-modsecurity-core-rule-set] Advanced Topic of the Week: Mitigating Slow HTTP DoS Attacks

 

Hi Christian,

The SecReadStateLimit is not only a threshold for ip address. It is looking for an "anomaly" in connection process. So if you are behind a proxy or a NAT only the bad connections will be dropped. The good ones will pass normally. So legit connections behind the proxy will works fine.

Thanks

Breno

On Wed, Nov 24, 2010 at 1:17 AM, <christian.folini@post.ch> wrote:

Hi Ryan,

Nice post. Thanks. Especially the combination of mod_reqtimeout and ModS
is very elegant in my eyes.

I am not so happy with SecReadStateLimit looking only at the IP address.
How do protect proxies from your countermeasures? A proxy might share multiple
hundred legitimate connections with your server for multiple hundred legitimate
clients, all appearing to come from the same IP address.

Regs,

Christian


-----Ursprüngliche Nachricht-----
Von: owasp-modsecurity-core-rule-set-bounces@lists.owasp.org [mailto:owasp-modsecurity-core-rule-set-bounces@lists.owasp.org] Im Auftrag von Ryan Barnett
Gesendet: Mittwoch, 24. November 2010 02:45
An: mod-security-users@lists.sourceforge.net; owasp-modsecurity-core-rule-set@lists.owasp.org
Betreff: [Owasp-modsecurity-core-rule-set] Advanced Topic of the Week: Mitigating Slow HTTP DoS Attacks


This week's blog post -

http://blog.spiderlabs.com/2010/11/advanced-topic-of-the-week-mitigating-slow-http-dos-attacks.html

--
Ryan Barnett
Senior Security Researcher
Trustwave - SpiderLabs

_______________________________________________
Owasp-modsecurity-core-rule-set mailing list
Owasp-modsecurity-core-rule-set@lists.owasp.org
https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set


------------------------------------------------------------------------------
Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
Tap into the largest installed PC base & get more eyes on your game by
optimizing for Intel(R) Graphics Technology. Get started today with the
Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
http://p.sf.net/sfu/intelisp-dev2dev
_______________________________________________
mod-security-users mailing list
mod-security-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
Commercial ModSecurity Appliances, Rule Sets and Support:
http://www.modsecurity.org/breach/index.html