Thread: [Mod-security-developers] 2.9.0-RC1 test results

Brought to you by: victorhora, zimmerletw

mod-security-developers

[Mod-security-developers] 2.9.0-RC1 test results

From: Walter H. <mo...@sp...> - 2014-11-24 17:09:51

Hi Felipe and others,

Thanks again for the hard work on the release.
Here are my preliminary experiences with 2.9.0-RC1 on FreeBSD.

My overall impression is the normal ModSecurity features and earlier ones introduced in 2.8.0 seem mostly stable, with one weird exception. The remote resources didn't work for me at all.

I've updated the FreeBSD port to pull in yajl, curl (ModSec would not load without), lua51 (see below), and optionally ssdeep. I have to read about fuzzy hashing before testing it, but it builds and the syntax seems to be accepted.

Okay, on to the problems I've found. All tests were on FreeBSD 10.0-p12 with stack smashing protection, amd64, Apache 2.4.10 prefork, OpenSSL 1.0.1j, clang 3.3.

1) High prio: Remote resources fail with segfaults and other problems in @pmFromFile, @ipMatchFromFile and SecRemoteRules. https://gist.github.com/lifeforms/102f66246de8bd33a2ca

2) High prio: Undiagnosed persistent crash. https://gist.github.com/lifeforms/4356643edfe8f39c2991

3) Medium prio: httpd crash on every request when using Lua 5.2. Working fine with Lua 5.1. https://gist.github.com/lifeforms/3ecc60c67012a053d060

4) Low prio: Apache log messages not prefixed with name. (Also present in earlier version) https://gist.github.com/lifeforms/4b41ae6464073ced39f5

Since I don't know if it's a useful workflow to create github issues, I’ve put the long descriptions in gists for now, but of course I can submit them wherever you like.

If I can submit more info, let me know. Except for issue 2) it’s easy to reproduce.

Thanks!
WH

--
Walter Hop | PGP key: https://lifeforms.nl/pgp

Re: [Mod-security-developers] 2.9.0-RC1 test results

From: Walter H. <mo...@sp...> - 2014-11-25 22:43:24

> 2) High prio: Undiagnosed persistent crash. https://gist.github.com/lifeforms/4356643edfe8f39c2991 <https://gist.github.com/lifeforms/4356643edfe8f39c2991>

Got the same crash on a second test box today.
I have updated the gist with information from a debug build: https://gist.github.com/lifeforms/4356643edfe8f39c2991 <https://gist.github.com/lifeforms/4356643edfe8f39c2991>

This crash appears to be serious. I don’t think I’ve ever seen ModSecurity segfault while parsing a request before. Since it starts happening on a random moment of the day, I’m a bit concerned this might be a remote DoS vuln, so I’m reverting to 2.7.7 on the public boxes.

I have kept some core files, but it’s been a long time since I worked with gdb so let me know if I should extract more info out of them.

Is there a way to enable asserts in the code so we can find out why/when node is unset?

5) I have tested @fuzzyHash and could not get it to work. My experiences are in this gist: https://gist.github.com/lifeforms/660e995254aba740856e <https://gist.github.com/lifeforms/660e995254aba740856e>

Sorry I couldn’t bring more positive news!

Cheers,
WH

-- 
Walter Hop | PGP key: https://lifeforms.nl/pgp

Re: [Mod-security-developers] 2.9.0-RC1 test results

From: Ryan B. <RBa...@tr...> - 2014-11-26 03:42:53

Walter,
Please see this blog post I did about using @fuzzyHash operator -
http://blog.spiderlabs.com/2014/11/modsecurity-advanced-topic-of-the-week-detecting-malware-with-fuzzy-hashing.html

Hopefully this will help with some testing.

Ryan Barnett
Senior Lead Security Researcher, SpiderLabs

Trustwave | SMART SECURITY ON DEMAND
www.trustwave.com<http://www.trustwave.com/>

From: Walter Hop <mo...@sp...<mailto:mo...@sp...>>
Reply-To: "mod...@li...<mailto:mod...@li...>" <mod...@li...<mailto:mod...@li...>>
Date: Tuesday, November 25, 2014 5:43 PM
To: "mod...@li...<mailto:mod...@li...>" <mod...@li...<mailto:mod...@li...>>
Subject: Re: [Mod-security-developers] 2.9.0-RC1 test results

2) High prio: Undiagnosed persistent crash. https://gist.github.com/lifeforms/4356643edfe8f39c2991<http://scanmail.trustwave.com/?c=4062&d=mYb11GfxNXS5cIjJ4hTdkjqKeLDfsoZGAe3WPGO1uw&s=5&u=https%3a%2f%2fgist%2egithub%2ecom%2flifeforms%2f4356643edfe8f39c2991>

Got the same crash on a second test box today.
I have updated the gist with information from a debug build: https://gist.github.com/lifeforms/4356643edfe8f39c2991<http://scanmail.trustwave.com/?c=4062&d=mYb11GfxNXS5cIjJ4hTdkjqKeLDfsoZGAe3WPGO1uw&s=5&u=https%3a%2f%2fgist%2egithub%2ecom%2flifeforms%2f4356643edfe8f39c2991>

This crash appears to be serious. I don’t think I’ve ever seen ModSecurity segfault while parsing a request before. Since it starts happening on a random moment of the day, I’m a bit concerned this might be a remote DoS vuln, so I’m reverting to 2.7.7 on the public boxes.

I have kept some core files, but it’s been a long time since I worked with gdb so let me know if I should extract more info out of them.

Is there a way to enable asserts in the code so we can find out why/when node is unset?

5) I have tested @fuzzyHash and could not get it to work. My experiences are in this gist: https://gist.github.com/lifeforms/660e995254aba740856e<http://scanmail.trustwave.com/?c=4062&d=mYb11GfxNXS5cIjJ4hTdkjqKeLDfsoZGAbnQODC06A&s=5&u=https%3a%2f%2fgist%2egithub%2ecom%2flifeforms%2f660e995254aba740856e>

Sorry I couldn’t bring more positive news!

Cheers,
WH

--
Walter Hop | PGP key: https://lifeforms.nl/pgp<http://scanmail.trustwave.com/?c=4062&d=mYb11GfxNXS5cIjJ4hTdkjqKeLDfsoZGAe2HPTbi7w&s=5&u=https%3a%2f%2flifeforms%2enl%2fpgp>


________________________________

This transmission may contain information that is privileged, confidential, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is strictly prohibited. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format.

Re: [Mod-security-developers] 2.9.0-RC1 test results

From: Walter H. <mo...@sp...> - 2014-12-04 17:01:56

An update about this crash. Last week, I got very similar repeated segfaults on three boxes running ModSecurity 2.7.7! Of course I don’t have debug builds running everywhere, but it seemed to be in the same function.

Interestingly, more out of luck than anything else, two of these boxes were slated for upgrading to FreeBSD 10.1, and I noticed the segfaults completely went away on them for a week (knock on wood) while I was having them almost daily.

So I am now thinking this is *not* a regression in 2.9.0.

My working theory now is, either the interaction of some library update (pcre? libxml2?) with the FreeBSD 10.0 (clang?) runtime leads to memory corruption.


> 2) High prio: Undiagnosed persistent crash. https://gist.github.com/lifeforms/4356643edfe8f39c2991 <https://gist.github.com/lifeforms/4356643edfe8f39c2991>
> 
> Got the same crash on a second test box today.
> I have updated the gist with information from a debug build: https://gist.github.com/lifeforms/4356643edfe8f39c2991 <https://gist.github.com/lifeforms/4356643edfe8f39c2991>
> 
> This crash appears to be serious. I don’t think I’ve ever seen ModSecurity segfault while parsing a request before. Since it starts happening on a random moment of the day, I’m a bit concerned this might be a remote DoS vuln, so I’m reverting to 2.7.7 on the public boxes.
> 
> I have kept some core files, but it’s been a long time since I worked with gdb so let me know if I should extract more info out of them.
> 
> Is there a way to enable asserts in the code so we can find out why/when node is unset?


-- 
Walter Hop | PGP key: https://lifeforms.nl/pgp

Re: [Mod-security-developers] 2.9.0-RC1 test results

From: Felipe C. <FC...@tr...> - 2014-12-04 18:16:31

Hi Walter,

Thank you again to let as know.

As this specific part of the code was not being updated for a while, that was my suspicion.

By looking at the GDB output that you have provided, I identified the reason of the segfault. 2.9.0[-RC2|] will not segfault because of that. Anyhow, we still have a problem as we have understand what circumstances are leading us to that NULL pointer.

I was not able to reproduce the problem, so I am sending to your email a GDB script that will avoid the crash in your server and take a snapshot of important pieces of the memory that may be useful to understand what is going on. This snapshot may contains sensitive information of your server, double check before send it back.

It will be very helpful if you can test that in your server.

Br.,
Felipe "Zimmerle" Costa
Security Researcher, SpiderLabs

Trustwave | SMART SECURITY ON DEMAND
www.trustwave.com <http://www.trustwave.com/>



________________________________
From: Walter Hop [mo...@sp...]
Sent: Thursday, December 04, 2014 3:01 PM
To: mod...@li...
Subject: Re: [Mod-security-developers] 2.9.0-RC1 test results

An update about this crash. Last week, I got very similar repeated segfaults on three boxes running ModSecurity 2.7.7! Of course I don’t have debug builds running everywhere, but it seemed to be in the same function.

Interestingly, more out of luck than anything else, two of these boxes were slated for upgrading to FreeBSD 10.1, and I noticed the segfaults completely went away on them for a week (knock on wood) while I was having them almost daily.

So I am now thinking this is *not* a regression in 2.9.0.

My working theory now is, either the interaction of some library update (pcre? libxml2?) with the FreeBSD 10.0 (clang?) runtime leads to memory corruption.


2) High prio: Undiagnosed persistent crash. https://gist.github.com/lifeforms/4356643edfe8f39c2991<http://scanmail.trustwave.com/?c=4062&d=opOA1B1Ql0cbsNHR1AF9Pmo8cR78lITwOVPqKoj65g&s=5&u=https%3a%2f%2fgist%2egithub%2ecom%2flifeforms%2f4356643edfe8f39c2991>

Got the same crash on a second test box today.
I have updated the gist with information from a debug build: https://gist.github.com/lifeforms/4356643edfe8f39c2991<http://scanmail.trustwave.com/?c=4062&d=opOA1B1Ql0cbsNHR1AF9Pmo8cR78lITwOVPqKoj65g&s=5&u=https%3a%2f%2fgist%2egithub%2ecom%2flifeforms%2f4356643edfe8f39c2991>

This crash appears to be serious. I don’t think I’ve ever seen ModSecurity segfault while parsing a request before. Since it starts happening on a random moment of the day, I’m a bit concerned this might be a remote DoS vuln, so I’m reverting to 2.7.7 on the public boxes.

I have kept some core files, but it’s been a long time since I worked with gdb so let me know if I should extract more info out of them.

Is there a way to enable asserts in the code so we can find out why/when node is unset?


--
Walter Hop | PGP key: https://lifeforms.nl/pgp<http://scanmail.trustwave.com/?c=4062&d=opOA1B1Ql0cbsNHR1AF9Pmo8cR78lITwOVO7K92tsg&s=5&u=https%3a%2f%2flifeforms%2enl%2fpgp>


________________________________

This transmission may contain information that is privileged, confidential, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is strictly prohibited. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format.

Re: [Mod-security-developers] 2.9.0-RC1 test results

From: Felipe C. <FC...@tr...> - 2014-12-15 19:24:36

Hi Walter,

Comments below...

>

> Okay, on to the problems I've found. All tests were on FreeBSD 10.0-p12
> with stack smashing protection, amd64, Apache 2.4.10 prefork, OpenSSL
> 1.0.1j, clang 3.3.
>
>

> 1) High prio: Remote resources fail with segfaults and other problems in
> @pmFromFile, @ipMatchFromFile and SecRemoteRules.
>

> https://gist.github.com/lifeforms/102f66246de8bd33a2ca
>

1.a) Crash

Fixed. That was a consequence of mod_ssl utilization. Now we are doing the
OpenSSL Initialization globally, instead of a initialization and cleanup
for every request.

1.b) Nonexisting files

Fixed. Now the HTTP error code is being taken into consideration.

>
> 2) High prio: Undiagnosed persistent crash.
>
> https://gist.github.com/lifeforms/4356643edfe8f39c2991
>

After upgrade the box, Walter was not able to reproduce the problem.

>
> 3) Medium prio: httpd crash on every request when using Lua 5.2. Working
> fine with Lua 5.1.
>
>
> https://gist.github.com/lifeforms/3ecc60c67012a053d060

>

That is something that also happens with oldest versions of ModSecurity. We
have an issue opened regarding that
(https://github.com/SpiderLabs/ModSecurity/issues/762), actually it is more
about make the installer smart enough to check the Lua versions, however, I
believe that we have to support the newest version as well, thus, I have
just
opened this new issue: https://github.com/SpiderLabs/ModSecurity/issues/814

>
> 4) Low prio: Apache log messages not prefixed with name. (Also present
>in earlier version)
>
> https://gist.github.com/lifeforms/4b41ae6464073ced39f5
>


"ModSecurity:" prefix was added.

>
> Since I don't know if it's a useful workflow to create github issues,
>I¹ve put the long
> descriptions in gists for now, but of course I can submit them wherever
>you like.
>

That was just fine.

Thank you again for your work. Very valuable report.

The fixes are already on top of our master, there will be a -RC2 soon.

Br.,
Felipe "Zimmerle" Costa
Security Researcher, SpiderLabs

Trustwave | SMART SECURITY ON DEMAND
www.trustwave.com <http://www.trustwave.com/>





________________________________

This transmission may contain information that is privileged, confidential, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is strictly prohibited. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format.

Re: [Mod-security-developers] 2.9.0-RC1 test results

From: Christian F. <chr...@ti...> - 2014-12-16 04:48:34

Hi there,

On Mon, Dec 15, 2014 at 07:22:04PM +0000, Felipe Costa wrote:
> 1.a) Crash
> 
> Fixed. That was a consequence of mod_ssl utilization. Now we are doing the
> OpenSSL Initialization globally, instead of a initialization and cleanup
> for every request.

I got the impression the crash I reported might have been the same as
the one by Walter. Is that correct?

I can run another test tomorrow, if that matters.

Ahoj,

Christian

-- 
Human history becomes more and more a race between 
education and catastrophe.
--- H. G. Wells

Re: [Mod-security-developers] 2.9.0-RC1 test results

From: Walter H. <mo...@sp...> - 2014-12-16 15:27:43

> The fixes are already on top of our master, there will be a -RC2 soon.

Awesome, thanks for the fixes. It’s a busy period but I will give the new RC some exercise at the beginning of next week.

I’ll also check ssdeep some more then. (I figured that maybe my problems getting a match might be related to SecChrootDir which I always use; so I’ll try again without it)

Again thanks for the nice work!
WH