Re: [Mod-security-developers] Finding triggered RuleIds

[Felipe C. <FC...@tr...>]

Hi Jai,

Currently, the disruptive data does not contain the details into separated structures, instead of in the format of the logging message. I propose to expand that to may (depending on a library configuration flag) contains the logging in the data structure.

In the released version this data structure is a “shared pointer” but it is suitable to be improved, and it is likely to be transformed on a && for version 3.1. Regardless, let me show you the pieces of code:
https://github.com/SpiderLabs/ModSecurity/blob/1ecd9713061c3534626bf6a5f59d1c928c0c52bb/headers/modsecurity/modsecurity.h#L236-L267

Notice the property RuleMessageLogProperty; it can be used to receive the rule message structure on the logging callback:
https://github.com/SpiderLabs/ModSecurity/blob/1ecd9713061c3534626bf6a5f59d1c928c0c52bb/headers/modsecurity/rule_message.h#L37-L117

For the disruptive actions we have:
https://github.com/SpiderLabs/ModSecurity/blob/v3/master/headers/modsecurity/intervention.h#L23-L29

The *log is the one that contains the information that you are looking for, but it is not respecting the configuration flag and always returning the log in the format of a text message. I want to change that also to support the callback. My advice is for you to call the intervention method to proceed with the evaluation of the rules. The intervention is the one who fulfills this structure among other things:

https://github.com/SpiderLabs/ModSecurity-nginx/blob/f1a7ab6e3c3e010dbbf2cfc37ae7f81fa410dc57/src/ngx_http_modsecurity_module.c#L132-L212

 

You may ask…

Why in this specific log in not on the callback? Well, the callback may be threated asynchronies by the consumer, leading to a possibility where the disruptive action takes place before the log the be processed/delivered, making it very hard the couple this event into a web server log, for instance.

Why not send it twice? The avoid spending resources in vain. After all, the disruptive is conditioned to the logging and vice-versa.

Change this behavior via a Rule, may not a good idea. It put the data to be shaped into memory in the format desired by the user, but, may not in the format expected by the consumer, leading to an invalid memory access; That most certainly will lead to crashes and instabilities.


I see two possible problems in your code snippet:

a) You are accessing the library raw data/internals. One of the beauties of the ModSecurity library (<3) is the fact that it contains two set of headers: public [https://github.com/SpiderLabs/ModSecurity/tree/v3/master/headers/modsecurity] and privates [all others].

The public ones are the ones that you should use. The others you should not. Due to a simple reason: we can change the internals without further notice, and that will break your implementation. You can achieve the same level of functionality by using the public headers. If not, ask, and the library will be expanded to support the missing feature (of course it has to make sense to other users as well).

Just to give you an example, that is what the nginx connector does to threat the disruptive actions and all the messages:
https://github.com/SpiderLabs/ModSecurity-nginx/blob/f1a7ab6e3c3e010dbbf2cfc37ae7f81fa410dc57/src/ngx_http_modsecurity_header_filter.c#L523-L528
(please do not use the Apache connector as a guide as it still under development. The nginx is fine.)

The version 3.1, where the first pieces of rules reload [reload the rules upon file update -- without restart] are already implemented, may cause you a little trouble. I still finishing the last touches: https://github.com/SpiderLabs/ModSecurity/tree/v3/dev/3.1-experimental


b) You may have a conceptual problem while handling an "escalation": A trigged rule does not mean that an action should be taken. You may find it workable in a given rule set (or rule set version), but no guarantee won't become a bug in a <near> future. Better to rely on the library for that. For instance: a rule can be trigged to set a variable.

In this example you can check how to read the RuleMessage without access the library internals:
https://github.com/SpiderLabs/ModSecurity/tree/v3/master/examples/reading_logs_via_rule_message

The question is: Having the *log in the ModSecurityIntervention_t in the format of a RuleMessage will solve your initial request? I guess so, but I am not sure if that fits your current implementation. Let's discuss; I don't know if you have other motivations to use the library in such fashion.


Br.,
Felipe “Zimmerle” Costa
Security Researcher, Lead Developer ModSecurity.

Trustwave | SMART SECURITY ON DEMAND
http://www.trustwave.com/


From: Jai Harpalani <jai...@mu...>
Date: Saturday, March 23, 2019 at 12:05 AM
To: Felipe Costa <FC...@tr...>
Cc: "mod...@li..." <mod...@li...>
Subject: Re: [Mod-security-developers] Finding triggered RuleIds

Responses interspersed below:

On Fri, Mar 22, 2019 at 8:34 PM Felipe Costa <mailto:FC...@tr...> wrote:
Just to make sure that we are talking about the same thing… when there Is a log to be shown the API is calling the callback. The only circumstance where the callback is not called is when the logging is a consequence of a disruptive action. In this last case the logging message is delivered via the disruptive structure in text format. So you guys want me to disable the logging for certain cases? If that is your request, there is a feature for that among the compilation flags to make ModSecurtiy mute.

In 3.0.2, I believe the callback was always invoked when a rule was triggered. With 3.0.3, it is not invoked for a disruptive action. My calling code was expecting the callback to always be invoked because that is the only way we know that a rule has triggered. You state that for a disruptive action, "the logging message is delivered via the disruptive structure in text format". Where is this done? How is it delivered? How can the calling code inspect this disruptive structure? And, does the disruptive structure contain all the details which were provided by the callback (eg ip address, api, ruleId, rule tags, etc)?
 
 
There are three logs in v3: Debug, Msg/Error, Audit. I am talking about the “error log” one. It could be “displayed” as o consequence of the rules execution flow (log, auditlog action, etc) or consequence of a disruptive action. The first you retrieve via callback, the second via disruptive structure.

The "error log" one is the one I am talking about, as well.
 
 
Jai, from your initial email I had the feeling that you would like to have the id in a data structure at the disruptive message or having the logging callback to replace the logging data inside the disruptive structure. It will be a pleasure to make the library more flexible, but, can you give us a brief explanation of your requirements?

The requirement is for the calling code to be able to determine if a rule was triggered, regardless of whether the rule was disruptive or not. And, if triggered, the calling code should be provided details about the trigger. 



I may have found a workaround for 3.0.3. Details below.

For 3.0.2, I have code such as the following inside of logCallback():

    const modsecurity::RuleMessage * lRuleMessage =
        reinterpret_cast<const modsecurity::RuleMessage *>(aRuleMessage);
    ruleId = lRuleMessage->m_ruleId;
    logMsg << "clientIp:" << m_clientIpAddress 
    << "clientPort:" << m_clientPort

    . . .

We have a class, ModSecTransaction, which inherits from modsecurity::Transaction. In 3.0.3, I have moved the above code into the ModSecTransaction dtor:

ModSecTransaction::~ModSecTransaction() noexcept
{
    for (modsecurity::RuleMessage& lRuleMessage : m_rulesMessages)
    {
      ruleId = lRuleMessage->m_ruleId;
      logMsg << . . .
      . . .
    }
    . . .
}

With 3.0.3, I also iterate through m_ruleMessages to determine if any rules have triggered after each call to processRequestHeaders(), processRequestBody().

bool
ModSecTransaction::shouldEscalateAnyTriggeredRule() noexcept
{
    bool lRetVal = false;

    // Iterate through all triggered rules
    for (modsecurity::RuleMessage& lRuleMessage : m_rulesMessages)
    {
        // Check if triggered rule should be escalated
        if (m_Origins.shouldEscalate(lRuleMessage.m_ruleId))
        {
            lRetVal = true;
            break;
        }
    }

    return lRetVal;
}

So far, this approach in 3.0.3 is working. Is it okay to use this approach in ModSec 3.0.3? Will it always work? Are there cases in which modsecurity::Transaction.m_rulesMessages will not be populated?

Thanks,
Jai