Re: [Mod-security-developers] Finding triggered RuleIds

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hey Ervin,

I know you are working on fixies on CRS test bed and corelated stuff in order to make it compatible/consistent with v3, congrats a lot of work there. I know that you also have a patch to change v3 in order to make it match the expected data on the test beds. I will have a look at it again on the next week. Sorry on hurry here.

It may be confusing but I am: Felipe "Zimmerle" Costa or any permutation ( I am the guy that you are talking about ModSec on hangout sessions, the very same on GitHub and Slack and Email (

If I remember correctly, You and Victor Hora had being discussing on our Slack channel, among other things, about the semantic of block being or not a disruptive action, in fact it is a poser as a linker that may call a disruptive action -- not talking about the manual but how it is currently implemented. Notice also that proxy is not supported on v3. The reason why I am saying that, is the fact that we do have a lot of information on the example that you have posted. The amount of information could: (a) be wrongly implemented at any point; (b) be misinterpreted and lead us to false conclusions. My suggestion is to reduce the tests to a very little amount of rules/actions until it demonstrated without a doubt where the problem is. In fact, those could be implemented as v3 test cases. There is an error_log entry on the test case that was built for that.  Not sure if it is handling multiple entries, guess so. Put a break on regression.cc:90 it will tell you [sorry, not on a computer].
- https://github.com/SpiderLabs/ModSecurity/blob/b392a1ca36181a786a6d68b6eab57a8bb0bd558e/test/test-cases/regression/offset-variable.json#L54

You may found a bug; it demands further investigation. We have to narrow down where is the bug in order for us to fix. It could be even related to the v3 state [off, on, detection] that is only trigged in certain conditions. If you want we can do a hangout session to do it together, but I am not available before 1 April. I am with Victor on BlackHat Asia to present the new stuff on v3.1.

If there is a bug, it needs to be fixed in our side [and will]; Jai does not need to fix our bug on what he is coding (At least, I hope not.

Br.,
Felipe “Zimmerle” Costa
Security Researcher, Lead Developer ModSecurity.

Trustwave | SMART SECURITY ON DEMAND
www.trustwave.com <http://www.trustwave.com/>

On 3/23/19, 6:07 AM, "Ervin Hegedüs" <ai...@gm...> wrote:

    Hi Jai, Felipe,

    On Fri, Mar 22, 2019 at 10:05:29PM -0500, Jai Harpalani via mod-security-developers wrote:

    pre-note: I'm working on libmodsecurity3, fix bugs/different
    behaviours than ModSecurity2. The reference ruleset is owasp CRS
    3.1.

    To understand the libmodsecurity3 API, I've created a small (and
    really ugly :)) http server, which uses API calls.

    Of course, then I confronted with your logging issue.

    > In 3.0.2, I believe the callback was always invoked when a rule was
    > triggered. With 3.0.3, it is not invoked for a disruptive action.

    Disruptive actions are those that causes ModSecurity to do
    something. They are: block, deny, drop, allow, proxy and
    redirect.

    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JPoZnsCL3Q&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fModSecurity%2fwiki%2fReference-Manual-%28v2%2ex%29%23secruleengine
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JKIeyMPbjw&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fModSecurity%2fwiki%2fReference-Manual-%28v2%2ex%29%23actions

    I've checked your statement with my ugly httpd with this request:

    GET /?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d HTTP/1.1
    Host: localhost

    and in the my log (error.log!) I got these rules:

    id "920280" 
    id "920320" 
    id "920273"
    id "941330"
    id "942110"
    id "942432"
    id "949110" 
    id "980130" 

    Let's see, which are disputive and which isn't:

    920280:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JPUSz82H2Q&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fREQUEST-920-PROTOCOL-ENFORCEMENT%2econf%23L593-L595
    non disruptiv

    920320:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JPtPyJTa3w&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fREQUEST-920-PROTOCOL-ENFORCEMENT%2econf%23L1328-L1330
    non disruptiv

    920273:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JPAfyM2I3A&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fREQUEST-920-PROTOCOL-ENFORCEMENT%2econf%23L1472-L1474
    disruptiv

    941330:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JPIdl8Pdiw&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fREQUEST-941-APPLICATION-ATTACK-XSS%2econf%23L876-L878
    disruptiv

    942110:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JPMSm8zc1g&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fREQUEST-942-APPLICATION-ATTACK-SQLI%2econf%23L505-L508
    disruptive

    942432:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JPESnsyH3Q&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fREQUEST-942-APPLICATION-ATTACK-SQLI%2econf%23L1580-L1582
    disruptive

    949110:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JKcbnczfjQ&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fREQUEST-949-BLOCKING-EVALUATION%2econf%23L81-L83
    disruptive

    980130:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JKISz5Ta3g&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fRESPONSE-980-CORRELATION%2econf%23L79-L81
    non-disputive

    so, looks like libmodsecurity3, version 3.0.3 logs mixed
    disruptive and non-disruptive triggered rules.

    > My
    > calling code was expecting the callback to always be invoked because that
    > is the only way we know that a rule has triggered.

    yes, you're right, this is a new feature request.

    > You state that for a
    > disruptive action, "the logging message is delivered via the disruptive
    > structure in text format". Where is this done?

    I'm not sure that understand you, but here is my snippet:

    ...
        modsec = msc_init();
        msc_set_log_cb(modsec, msc_logdata);
    ...

    upper...

    void msc_logdata(void *log, const void *data) {
        FILE *fp;
        char logline[4096];

        struct timeval tv;

        wrote_log_line(...); 
        fclose(fp);
    }

    and that's it. Not need to call this function explicitly, but I
    guess you know that - that's why I asked you above, what do you
    think about whit this question.

    > How is it delivered? How can
    > the calling code inspect this disruptive structure?

    there are some examples in examples/ subdirectory.

    > And, does the
    > disruptive structure contain all the details which were provided by the
    > callback (eg ip address, api, ruleId, rule tags, etc)?

    hmm... here is a logged line:

    1553327402.117947 ModSecurity: Warning. Matched "Operator `Ge' with parameter `5' against variable `TX:ANOMALY_SCORE' (Value: `21' ) [file "/usr/share/modsecurity-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "80"] [id "949110"] [rev ""] [msg "Inbound Anomaly Score Exceeded (Total Score: 21)"] [data ""] [severity "2"] [ver ""] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "127.0.0.1"] [uri "http://localhost/"] [unique_id "1553327401.929417"] [ref ""]

    ip address: [hostname "127.0.0.1"]
    api: what do you think about?
    ruleId: [id: "949110"]
    rule tags: [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"]
    etc: what do you think about it more?

    note, that I'm working with "SecRuleEngine DetectOnly" mode.

    > > There are three logs in v3: Debug, Msg/Error, Audit. I am talking about
    > > the “error log” one. It could be “displayed” as o consequence of the rules
    > > execution flow (log, auditlog action, etc) or consequence of a disruptive
    > > action. The first you retrieve via callback, the second via disruptive
    > > structure.
    > >
    > 
    > The "error log" one is the one I am talking about, as well.

    sure, in the debug log, I think I got the all triggered rule:

    grep Rule: modsec_debug.log  | wc -l
    541

    eg:
    [1553327401.929417] [http://localhost/?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d] [4] (Rule: 200000) Executing operator "Rx" with param "(?:application(?:/soap\+|/)|text/)xml" against REQUEST_HEADERS:Content-Type.
    [1553327401.929417] [http://localhost/?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d] [4] (Rule: 200001) Executing operator "Rx" with param "application/json" against REQUEST_HEADERS:Content-Type.
    [1553327401.929417] [http://localhost/?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d] [4] (Rule: 900000) Executing unconditional rule...
    ...

    > > Jai, from your initial email I had the feeling that you would like to have
    > > the id in a data structure at the disruptive message or having the logging
    > > callback to replace the logging data inside the disruptive structure. It
    > > will be a pleasure to make the library more flexible, but, can you give us
    > > a brief explanation of your requirements?
    > >
    > 
    > The requirement is for the calling code to be able to determine if a rule
    > was triggered, regardless of whether the rule was disruptive or not. And,
    > if triggered, the calling code should be provided details about the
    > trigger.

    for this I supposed we can use a new configuration variable,
    which can control that all triggered rule will logged or not.

    > I may have found a workaround for 3.0.3. Details below.

    I'll see that later, now I just wanted to reflect first part of
    your mail :).

    Hope this helped to clear some things.

    a.