mod-security-developers

[Mod-security-developers] Finding triggered RuleIds

[Jai H. <jai...@mu...>]

We are integrating ModSecurity into our product as a library, and using it
to evaluate owasp crs rules.

For anyone else doing this, can you explain how your calling code is
determining which ruleId(s) were triggered as a result of
calling processRequestHeaders(), processRequestBody(),
processResponseHeaders(), processResponseBody()?

Curious how this is being done in ModSec 3.0.2 and if it is done
differently with version 3.0.3.

Re: [Mod-security-developers] Finding triggered RuleIds

[Ervin H. <ai...@gm...>]

Hi Jai,

once upon I've discussed about this with @zimmerle, and he helped me with
this links:

https://github.com/SpiderLabs/ModSecurity/blob/1ecd9713061c3534626bf6a5f59d1c928c0c52bb/examples/reading_logs_via_rule_message/reading_logs_via_rule_message.h#L141-L142
https://github.com/SpiderLabs/ModSecurity/blob/f77db2cc2eff4808ad59118f1a11baea8f849b04/headers/modsecurity/modsecurity.h#L242-L267
https://github.com/SpiderLabs/ModSecurity/blob/ad28de4f14e47d3c6b479a1d043f2bd0b7a17706/headers/modsecurity/rule_message.h

You can set up a log callbck function, which will got a structure, and you
don't need to parse the logfile.

Try this and let me know what you got.


a.


On Thu, Mar 21, 2019 at 3:29 PM Jai Harpalani via mod-security-developers <
mod...@li...> wrote:

> We are integrating ModSecurity into our product as a library, and using it
> to evaluate owasp crs rules.
>
> For anyone else doing this, can you explain how your calling code is
> determining which ruleId(s) were triggered as a result of
> calling processRequestHeaders(), processRequestBody(),
> processResponseHeaders(), processResponseBody()?
>
> Curious how this is being done in ModSec 3.0.2 and if it is done
> differently with version 3.0.3.
> _______________________________________________
> mod-security-developers mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
> ModSecurity Services from Trustwave's SpiderLabs:
> https://www.trustwave.com/spiderLabs.php

Re: [Mod-security-developers] Finding triggered RuleIds

[Jai H. <jai...@mu...>]

Ervin,

The log callback function approach is what we used with ModSec 3.0.2 and it
worked well. Unfortunately, that approach no longer works for ModSec 3.0.3
because not all rule triggers invoke the log callback. Reason for this was
provided by Felipe:

Sometimes logging is a consequence of a disruptive action; sometimes the
> logging is just a warning. On 3.0.2 the logging for disruptive (aka error
> on 2.x) was being generated as a warning as well. To avoid creating the
> same message twice, we have changed 3.0.3 to produce only warnings, and
> give access to error message along with the disruptive structure.


So, I'm trying to determine the recommended approach for acquiring the
triggered ruleId(s) in ModSec 3.0.3.

Thanks,

Jai


On Thu, Mar 21, 2019 at 10:29 AM Ervin Hegedüs <ai...@gm...> wrote:

> Hi Jai,
>
> once upon I've discussed about this with @zimmerle, and he helped me with
> this links:
>
>
> https://github.com/SpiderLabs/ModSecurity/blob/1ecd9713061c3534626bf6a5f59d1c928c0c52bb/examples/reading_logs_via_rule_message/reading_logs_via_rule_message.h#L141-L142
>
> https://github.com/SpiderLabs/ModSecurity/blob/f77db2cc2eff4808ad59118f1a11baea8f849b04/headers/modsecurity/modsecurity.h#L242-L267
>
> https://github.com/SpiderLabs/ModSecurity/blob/ad28de4f14e47d3c6b479a1d043f2bd0b7a17706/headers/modsecurity/rule_message.h
>
> You can set up a log callbck function, which will got a structure, and you
> don't need to parse the logfile.
>
> Try this and let me know what you got.
>
>
> a.
>
>
> On Thu, Mar 21, 2019 at 3:29 PM Jai Harpalani via mod-security-developers <
> mod...@li...> wrote:
>
>> We are integrating ModSecurity into our product as a library, and using
>> it to evaluate owasp crs rules.
>>
>> For anyone else doing this, can you explain how your calling code is
>> determining which ruleId(s) were triggered as a result of
>> calling processRequestHeaders(), processRequestBody(),
>> processResponseHeaders(), processResponseBody()?
>>
>> Curious how this is being done in ModSec 3.0.2 and if it is done
>> differently with version 3.0.3.
>> _______________________________________________
>> mod-security-developers mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
>> ModSecurity Services from Trustwave's SpiderLabs:
>> https://www.trustwave.com/spiderLabs.php
>
> _______________________________________________
> mod-security-developers mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
> ModSecurity Services from Trustwave's SpiderLabs:
> https://www.trustwave.com/spiderLabs.php

Re: [Mod-security-developers] Finding triggered RuleIds

[Felipe C. <FC...@tr...>]

Hi Jai,

For the current public supported connectors, the rule id altogether with logging text is enough. There is no data structure except for char pointer that point towards the logging string [for the logging attached to the disruptive events]. Is my understanding that it may be useful for your application,  to have an specific field that held to rule id (may other information regarding the rule as well). Having that in mind, we can change the API the make it more useful to your application. Sorry for the inconvenience. Lets discuss out-of-band the specific characteristics of your use case, so we can make the API suits you better.

Br.,
Felipe “Zimmerle” Costa
Security Researcher, Lead Developer ModSecurity.

Trustwave | SMART SECURITY ON DEMAND
www.trustwave.com<http://www.trustwave.com/>


From: Jai Harpalani via mod-security-developers <mod...@li...>
Reply-To: "mod...@li..." <mod...@li...>
Date: Thursday, March 21, 2019 at 5:38 PM
To: "mod-security-d." <mod...@li...>
Cc: Jai Harpalani <jai...@mu...>
Subject: Re: [Mod-security-developers] Finding triggered RuleIds

Ervin,

The log callback function approach is what we used with ModSec 3.0.2 and it worked well. Unfortunately, that approach no longer works for ModSec 3.0.3 because not all rule triggers invoke the log callback. Reason for this was provided by Felipe:
Sometimes logging is a consequence of a disruptive action; sometimes the logging is just a warning. On 3.0.2 the logging for disruptive (aka error on 2.x) was being generated as a warning as well. To avoid creating the same message twice, we have changed 3.0.3 to produce only warnings, and give access to error message along with the disruptive structure.

So, I'm trying to determine the recommended approach for acquiring the triggered ruleId(s) in ModSec 3.0.3.

Thanks,

Jai


On Thu, Mar 21, 2019 at 10:29 AM Ervin Hegedüs <ai...@gm...<mailto:ai...@gm...>> wrote:
Hi Jai,

once upon I've discussed about this with @zimmerle, and he helped me with this links:

https://github.com/SpiderLabs/ModSecurity/blob/1ecd9713061c3534626bf6a5f59d1c928c0c52bb/examples/reading_logs_via_rule_message/reading_logs_via_rule_message.h#L141-L142<https://scanmail.trustwave.com/?c=4062&d=4PaT3Ib8KCgqsgIHe59jxLZMn7cvRuyxwHopdLzlMA&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fModSecurity%2fblob%2f1ecd9713061c3534626bf6a5f59d1c928c0c52bb%2fexamples%2freading%5flogs%5fvia%5frule%5fmessage%2freading%5flogs%5fvia%5frule%5fmessage%2eh%23L141-L142>
https://github.com/SpiderLabs/ModSecurity/blob/f77db2cc2eff4808ad59118f1a11baea8f849b04/headers/modsecurity/modsecurity.h#L242-L267<https://scanmail.trustwave.com/?c=4062&d=4PaT3Ib8KCgqsgIHe59jxLZMn7cvRuyxwCooce-yMw&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fModSecurity%2fblob%2ff77db2cc2eff4808ad59118f1a11baea8f849b04%2fheaders%2fmodsecurity%2fmodsecurity%2eh%23L242-L267>
https://github.com/SpiderLabs/ModSecurity/blob/ad28de4f14e47d3c6b479a1d043f2bd0b7a17706/headers/modsecurity/rule_message.h<https://scanmail.trustwave.com/?c=4062&d=4PaT3Ib8KCgqsgIHe59jxLZMn7cvRuyxwHgrde_mMw&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fModSecurity%2fblob%2fad28de4f14e47d3c6b479a1d043f2bd0b7a17706%2fheaders%2fmodsecurity%2frule%5fmessage%2eh>

You can set up a log callbck function, which will got a structure, and you don't need to parse the logfile.

Try this and let me know what you got.


a.


On Thu, Mar 21, 2019 at 3:29 PM Jai Harpalani via mod-security-developers <mod...@li...<mailto:mod...@li...>> wrote:
We are integrating ModSecurity into our product as a library, and using it to evaluate owasp crs rules.

For anyone else doing this, can you explain how your calling code is determining which ruleId(s) were triggered as a result of calling processRequestHeaders(), processRequestBody(), processResponseHeaders(), processResponseBody()?

Curious how this is being done in ModSec 3.0.2 and if it is done differently with version 3.0.3.
_______________________________________________
mod-security-developers mailing list
mod...@li...<mailto:mod...@li...>
https://lists.sourceforge.net/lists/listinfo/mod-security-developers<https://scanmail.trustwave.com/?c=4062&d=4PaT3Ib8KCgqsgIHe59jxLZMn7cvRuyxwH4tKOzmYQ&s=5&u=https%3a%2f%2flists%2esourceforge%2enet%2flists%2flistinfo%2fmod-security-developers>
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/spiderLabs.php
_______________________________________________
mod-security-developers mailing list
mod...@li...<mailto:mod...@li...>
https://lists.sourceforge.net/lists/listinfo/mod-security-developers<https://scanmail.trustwave.com/?c=4062&d=4PaT3Ib8KCgqsgIHe59jxLZMn7cvRuyxwH4tKOzmYQ&s=5&u=https%3a%2f%2flists%2esourceforge%2enet%2flists%2flistinfo%2fmod-security-developers>
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/spiderLabs.php

Re: [Mod-security-developers] Finding triggered RuleIds

[Ervin H. <ai...@gm...>]

hi,

On Fri, Mar 22, 2019 at 12:49:18PM +0000, Felipe Costa wrote:
> Hi Jai,
> 
> For the current public supported connectors, the rule id altogether with logging text is enough. There is no data structure except for char pointer that point towards the logging string [for the logging attached to the disruptive events]. Is my understanding that it may be useful for your application,  to have an specific field that held to rule id (may other information regarding the rule as well). Having that in mind, we can change the API the make it more useful to your application. Sorry for the inconvenience. Lets discuss out-of-band the specific characteristics of your use case, so we can make the API suits you better.

I can imagine that there should be a new configuration directive,
which allows to log every triggered rule, not just when it
intervents.

The default value should be disable to do this, but if the
end-user wants to see that, then it can be use that.

eg.

SecLogAllRule 0|1



a.

Re: [Mod-security-developers] Finding triggered RuleIds

[Jai H. <jai...@mu...>]

A 'SecLogAllRule' such as Ervin suggests would work well. Ideally, the log
message would contain all of the information currently contained in the
ruleMessage.

I may have found a workaround for this. Will send another email with the
details.

On Fri, Mar 22, 2019 at 3:10 PM Ervin Hegedüs <ai...@gm...> wrote:

> hi,
>
> On Fri, Mar 22, 2019 at 12:49:18PM +0000, Felipe Costa wrote:
> > Hi Jai,
> >
> > For the current public supported connectors, the rule id altogether with
> logging text is enough. There is no data structure except for char pointer
> that point towards the logging string [for the logging attached to the
> disruptive events]. Is my understanding that it may be useful for your
> application,  to have an specific field that held to rule id (may other
> information regarding the rule as well). Having that in mind, we can change
> the API the make it more useful to your application. Sorry for the
> inconvenience. Lets discuss out-of-band the specific characteristics of
> your use case, so we can make the API suits you better.
>
> I can imagine that there should be a new configuration directive,
> which allows to log every triggered rule, not just when it
> intervents.
>
> The default value should be disable to do this, but if the
> end-user wants to see that, then it can be use that.
>
> eg.
>
> SecLogAllRule 0|1
>
>
>
> a.
>
>
>
> _______________________________________________
> mod-security-developers mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
> ModSecurity Services from Trustwave's SpiderLabs:
> https://www.trustwave.com/spiderLabs.php
>

Re: [Mod-security-developers] Finding triggered RuleIds

[Felipe C. <FC...@tr...>]

Just to make sure that we are talking about the same thing… when there Is a log to be shown the API is calling the callback. The only circumstance where the callback is not called is when the logging is a consequence of a disruptive action. In this last case the logging message is delivered via the disruptive structure in text format. So you guys want me to disable the logging for certain cases? If that is your request, there is a feature for that among the compilation flags to make ModSecurtiy mute.

There are three logs in v3: Debug, Msg/Error, Audit. I am talking about the “error log” one. It could be “displayed” as o consequence of the rules execution flow (log, auditlog action, etc) or consequence of a disruptive action. The first you retrieve via callback, the second via disruptive structure.

Jai, from your initial email I had the feeling that you would like to have the id in a data structure at the disruptive message or having the logging callback to replace the logging data inside the disruptive structure. It will be a pleasure to make the library more flexible, but, can you give us a brief explanation of your requirements?

Br.,
Felipe “Zimmerle” Costa
Security Researcher, Lead Developer ModSecurity.

Trustwave | SMART SECURITY ON DEMAND
www.trustwave.com<http://www.trustwave.com/>


From: Jai Harpalani via mod-security-developers <mod...@li...>
Reply-To: "mod...@li..." <mod...@li...>
Date: Friday, March 22, 2019 at 5:16 PM
To: "mod-security-d." <mod...@li...>
Cc: Jai Harpalani <jai...@mu...>
Subject: Re: [Mod-security-developers] Finding triggered RuleIds

A 'SecLogAllRule' such as Ervin suggests would work well. Ideally, the log message would contain all of the information currently contained in the ruleMessage.

I may have found a workaround for this. Will send another email with the details.

On Fri, Mar 22, 2019 at 3:10 PM Ervin Hegedüs <ai...@gm...<mailto:ai...@gm...>> wrote:
hi,

On Fri, Mar 22, 2019 at 12:49:18PM +0000, Felipe Costa wrote:
> Hi Jai,
>
> For the current public supported connectors, the rule id altogether with logging text is enough. There is no data structure except for char pointer that point towards the logging string [for the logging attached to the disruptive events]. Is my understanding that it may be useful for your application,  to have an specific field that held to rule id (may other information regarding the rule as well). Having that in mind, we can change the API the make it more useful to your application. Sorry for the inconvenience. Lets discuss out-of-band the specific characteristics of your use case, so we can make the API suits you better.

I can imagine that there should be a new configuration directive,
which allows to log every triggered rule, not just when it
intervents.

The default value should be disable to do this, but if the
end-user wants to see that, then it can be use that.

eg.

SecLogAllRule 0|1



a.



_______________________________________________
mod-security-developers mailing list
mod...@li...<mailto:mod...@li...>
https://lists.sourceforge.net/lists/listinfo/mod-security-developers<https://scanmail.trustwave.com/?c=4062&d=qsKV3OL797AuHt0YIpZtf8wkheEsZv18XuZPrY4e8A&s=5&u=https%3a%2f%2flists%2esourceforge%2enet%2flists%2flistinfo%2fmod-security-developers>
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/spiderLabs.php

Re: [Mod-security-developers] Finding triggered RuleIds

[Jai H. <jai...@mu...>]

Responses interspersed below:

On Fri, Mar 22, 2019 at 8:34 PM Felipe Costa <FC...@tr...> wrote:

> Just to make sure that we are talking about the same thing… when there Is
> a log to be shown the API is calling the callback. The only circumstance
> where the callback is not called is when the logging is a consequence of a
> disruptive action. In this last case the logging message is delivered via
> the disruptive structure in text format. So you guys want me to disable the
> logging for certain cases? If that is your request, there is a feature for
> that among the compilation flags to make ModSecurtiy mute.
>

In 3.0.2, I believe the callback was always invoked when a rule was
triggered. With 3.0.3, it is not invoked for a disruptive action. My
calling code was expecting the callback to always be invoked because that
is the only way we know that a rule has triggered. You state that for a
disruptive action, "the logging message is delivered via the disruptive
structure in text format". Where is this done? How is it delivered? How can
the calling code inspect this disruptive structure? And, does the
disruptive structure contain all the details which were provided by the
callback (eg ip address, api, ruleId, rule tags, etc)?


>
>
> There are three logs in v3: Debug, Msg/Error, Audit. I am talking about
> the “error log” one. It could be “displayed” as o consequence of the rules
> execution flow (log, auditlog action, etc) or consequence of a disruptive
> action. The first you retrieve via callback, the second via disruptive
> structure.
>

The "error log" one is the one I am talking about, as well.


>
>
> Jai, from your initial email I had the feeling that you would like to have
> the id in a data structure at the disruptive message or having the logging
> callback to replace the logging data inside the disruptive structure. It
> will be a pleasure to make the library more flexible, but, can you give us
> a brief explanation of your requirements?
>

The requirement is for the calling code to be able to determine if a rule
was triggered, regardless of whether the rule was disruptive or not. And,
if triggered, the calling code should be provided details about the
trigger.



I may have found a workaround for 3.0.3. Details below.

For 3.0.2, I have code such as the following inside of logCallback():

   * const modsecurity::RuleMessage * lRuleMessage =*
*        reinterpret_cast<const modsecurity::RuleMessage *>(aRuleMessage);*
*    ruleId = lRuleMessage->m_ruleId;*
*    logMsg << "clientIp:" << m_clientIpAddress *
*    << "clientPort:" << m_clientPort*

    . . .

We have a class, ModSecTransaction, which inherits from
modsecurity::Transaction. In 3.0.3, I have moved the above code into the
ModSecTransaction dtor:

*ModSecTransaction::~ModSecTransaction() noexcept*
*{*
*    for (modsecurity::RuleMessage& lRuleMessage : m_rulesMessages)*
*    {*
*      ruleId = lRuleMessage->m_ruleId;*
*      logMsg << . . .*
*      . . .*
*    }*
*    . . .*
*}*

With 3.0.3, I also iterate through m_ruleMessages to determine if any rules
have triggered after each call to processRequestHeaders(),
processRequestBody().

*bool*
*ModSecTransaction::shouldEscalateAnyTriggeredRule() noexcept*
*{*
*    bool lRetVal = false;*

*    // Iterate through all triggered rules*
*    for (modsecurity::RuleMessage& lRuleMessage : m_rulesMessages)*
*    {*
*        // Check if triggered rule should be escalated*
*        if (m_Origins.shouldEscalate(lRuleMessage.m_ruleId))*
*        {*
*            lRetVal = true;*
*            break;*
*        }*
*    }*

*    return lRetVal;*
*}*

So far, this approach in 3.0.3 is working. *Is it okay to use this approach
in ModSec 3.0.3? Will it always work? Are there cases in which
modsecurity::Transaction.m_rulesMessages will not be populated?*

Thanks,
Jai

>

Re: [Mod-security-developers] Finding triggered RuleIds

[Felipe C. <FC...@tr...>]

Hi Jai,

Currently, the disruptive data does not contain the details into separated structures, instead of in the format of the logging message. I propose to expand that to may (depending on a library configuration flag) contains the logging in the data structure.

In the released version this data structure is a “shared pointer” but it is suitable to be improved, and it is likely to be transformed on a && for version 3.1. Regardless, let me show you the pieces of code:
https://github.com/SpiderLabs/ModSecurity/blob/1ecd9713061c3534626bf6a5f59d1c928c0c52bb/headers/modsecurity/modsecurity.h#L236-L267

Notice the property RuleMessageLogProperty; it can be used to receive the rule message structure on the logging callback:
https://github.com/SpiderLabs/ModSecurity/blob/1ecd9713061c3534626bf6a5f59d1c928c0c52bb/headers/modsecurity/rule_message.h#L37-L117

For the disruptive actions we have:
https://github.com/SpiderLabs/ModSecurity/blob/v3/master/headers/modsecurity/intervention.h#L23-L29

The *log is the one that contains the information that you are looking for, but it is not respecting the configuration flag and always returning the log in the format of a text message. I want to change that also to support the callback. My advice is for you to call the intervention method to proceed with the evaluation of the rules. The intervention is the one who fulfills this structure among other things:

https://github.com/SpiderLabs/ModSecurity-nginx/blob/f1a7ab6e3c3e010dbbf2cfc37ae7f81fa410dc57/src/ngx_http_modsecurity_module.c#L132-L212

 

You may ask…

Why in this specific log in not on the callback? Well, the callback may be threated asynchronies by the consumer, leading to a possibility where the disruptive action takes place before the log the be processed/delivered, making it very hard the couple this event into a web server log, for instance.

Why not send it twice? The avoid spending resources in vain. After all, the disruptive is conditioned to the logging and vice-versa.

Change this behavior via a Rule, may not a good idea. It put the data to be shaped into memory in the format desired by the user, but, may not in the format expected by the consumer, leading to an invalid memory access; That most certainly will lead to crashes and instabilities.


I see two possible problems in your code snippet:

a) You are accessing the library raw data/internals. One of the beauties of the ModSecurity library (<3) is the fact that it contains two set of headers: public [https://github.com/SpiderLabs/ModSecurity/tree/v3/master/headers/modsecurity] and privates [all others].

The public ones are the ones that you should use. The others you should not. Due to a simple reason: we can change the internals without further notice, and that will break your implementation. You can achieve the same level of functionality by using the public headers. If not, ask, and the library will be expanded to support the missing feature (of course it has to make sense to other users as well).

Just to give you an example, that is what the nginx connector does to threat the disruptive actions and all the messages:
https://github.com/SpiderLabs/ModSecurity-nginx/blob/f1a7ab6e3c3e010dbbf2cfc37ae7f81fa410dc57/src/ngx_http_modsecurity_header_filter.c#L523-L528
(please do not use the Apache connector as a guide as it still under development. The nginx is fine.)

The version 3.1, where the first pieces of rules reload [reload the rules upon file update -- without restart] are already implemented, may cause you a little trouble. I still finishing the last touches: https://github.com/SpiderLabs/ModSecurity/tree/v3/dev/3.1-experimental


b) You may have a conceptual problem while handling an "escalation": A trigged rule does not mean that an action should be taken. You may find it workable in a given rule set (or rule set version), but no guarantee won't become a bug in a <near> future. Better to rely on the library for that. For instance: a rule can be trigged to set a variable.

In this example you can check how to read the RuleMessage without access the library internals:
https://github.com/SpiderLabs/ModSecurity/tree/v3/master/examples/reading_logs_via_rule_message

The question is: Having the *log in the ModSecurityIntervention_t in the format of a RuleMessage will solve your initial request? I guess so, but I am not sure if that fits your current implementation. Let's discuss; I don't know if you have other motivations to use the library in such fashion.


Br.,
Felipe “Zimmerle” Costa
Security Researcher, Lead Developer ModSecurity.

Trustwave | SMART SECURITY ON DEMAND
http://www.trustwave.com/


From: Jai Harpalani <jai...@mu...>
Date: Saturday, March 23, 2019 at 12:05 AM
To: Felipe Costa <FC...@tr...>
Cc: "mod...@li..." <mod...@li...>
Subject: Re: [Mod-security-developers] Finding triggered RuleIds

Responses interspersed below:

On Fri, Mar 22, 2019 at 8:34 PM Felipe Costa <mailto:FC...@tr...> wrote:
Just to make sure that we are talking about the same thing… when there Is a log to be shown the API is calling the callback. The only circumstance where the callback is not called is when the logging is a consequence of a disruptive action. In this last case the logging message is delivered via the disruptive structure in text format. So you guys want me to disable the logging for certain cases? If that is your request, there is a feature for that among the compilation flags to make ModSecurtiy mute.

In 3.0.2, I believe the callback was always invoked when a rule was triggered. With 3.0.3, it is not invoked for a disruptive action. My calling code was expecting the callback to always be invoked because that is the only way we know that a rule has triggered. You state that for a disruptive action, "the logging message is delivered via the disruptive structure in text format". Where is this done? How is it delivered? How can the calling code inspect this disruptive structure? And, does the disruptive structure contain all the details which were provided by the callback (eg ip address, api, ruleId, rule tags, etc)?
 
 
There are three logs in v3: Debug, Msg/Error, Audit. I am talking about the “error log” one. It could be “displayed” as o consequence of the rules execution flow (log, auditlog action, etc) or consequence of a disruptive action. The first you retrieve via callback, the second via disruptive structure.

The "error log" one is the one I am talking about, as well.
 
 
Jai, from your initial email I had the feeling that you would like to have the id in a data structure at the disruptive message or having the logging callback to replace the logging data inside the disruptive structure. It will be a pleasure to make the library more flexible, but, can you give us a brief explanation of your requirements?

The requirement is for the calling code to be able to determine if a rule was triggered, regardless of whether the rule was disruptive or not. And, if triggered, the calling code should be provided details about the trigger. 



I may have found a workaround for 3.0.3. Details below.

For 3.0.2, I have code such as the following inside of logCallback():

    const modsecurity::RuleMessage * lRuleMessage =
        reinterpret_cast<const modsecurity::RuleMessage *>(aRuleMessage);
    ruleId = lRuleMessage->m_ruleId;
    logMsg << "clientIp:" << m_clientIpAddress 
    << "clientPort:" << m_clientPort

    . . .

We have a class, ModSecTransaction, which inherits from modsecurity::Transaction. In 3.0.3, I have moved the above code into the ModSecTransaction dtor:

ModSecTransaction::~ModSecTransaction() noexcept
{
    for (modsecurity::RuleMessage& lRuleMessage : m_rulesMessages)
    {
      ruleId = lRuleMessage->m_ruleId;
      logMsg << . . .
      . . .
    }
    . . .
}

With 3.0.3, I also iterate through m_ruleMessages to determine if any rules have triggered after each call to processRequestHeaders(), processRequestBody().

bool
ModSecTransaction::shouldEscalateAnyTriggeredRule() noexcept
{
    bool lRetVal = false;

    // Iterate through all triggered rules
    for (modsecurity::RuleMessage& lRuleMessage : m_rulesMessages)
    {
        // Check if triggered rule should be escalated
        if (m_Origins.shouldEscalate(lRuleMessage.m_ruleId))
        {
            lRetVal = true;
            break;
        }
    }

    return lRetVal;
}

So far, this approach in 3.0.3 is working. Is it okay to use this approach in ModSec 3.0.3? Will it always work? Are there cases in which modsecurity::Transaction.m_rulesMessages will not be populated?

Thanks,
Jai

Re: [Mod-security-developers] Finding triggered RuleIds

[Ervin H. <ai...@gm...>]

Hi Jai, Felipe,

On Fri, Mar 22, 2019 at 10:05:29PM -0500, Jai Harpalani via mod-security-developers wrote:

pre-note: I'm working on libmodsecurity3, fix bugs/different
behaviours than ModSecurity2. The reference ruleset is owasp CRS
3.1.

To understand the libmodsecurity3 API, I've created a small (and
really ugly :)) http server, which uses API calls.

Of course, then I confronted with your logging issue.
 
> In 3.0.2, I believe the callback was always invoked when a rule was
> triggered. With 3.0.3, it is not invoked for a disruptive action.

Disruptive actions are those that causes ModSecurity to do
something. They are: block, deny, drop, allow, proxy and
redirect.

https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#secruleengine
https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#actions

I've checked your statement with my ugly httpd with this request:

GET /?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d HTTP/1.1
Host: localhost

and in the my log (error.log!) I got these rules:

id "920280" 
id "920320" 
id "920273"
id "941330"
id "942110"
id "942432"
id "949110" 
id "980130" 

Let's see, which are disputive and which isn't:

920280:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf#L593-L595
non disruptiv

920320:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf#L1328-L1330
non disruptiv

920273:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf#L1472-L1474
disruptiv

941330:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf#L876-L878
disruptiv

942110:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf#L505-L508
disruptive

942432:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf#L1580-L1582
disruptive

949110:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/REQUEST-949-BLOCKING-EVALUATION.conf#L81-L83
disruptive

980130:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/RESPONSE-980-CORRELATION.conf#L79-L81
non-disputive

so, looks like libmodsecurity3, version 3.0.3 logs mixed
disruptive and non-disruptive triggered rules.


> My
> calling code was expecting the callback to always be invoked because that
> is the only way we know that a rule has triggered.

yes, you're right, this is a new feature request.

> You state that for a
> disruptive action, "the logging message is delivered via the disruptive
> structure in text format". Where is this done?

I'm not sure that understand you, but here is my snippet:

...
    modsec = msc_init();
    msc_set_log_cb(modsec, msc_logdata);
...

upper...

void msc_logdata(void *log, const void *data) {
    FILE *fp;
    char logline[4096];

    struct timeval tv;

    wrote_log_line(...); 
    fclose(fp);
}

and that's it. Not need to call this function explicitly, but I
guess you know that - that's why I asked you above, what do you
think about whit this question.

> How is it delivered? How can
> the calling code inspect this disruptive structure?

there are some examples in examples/ subdirectory.

> And, does the
> disruptive structure contain all the details which were provided by the
> callback (eg ip address, api, ruleId, rule tags, etc)?

hmm... here is a logged line:

1553327402.117947 ModSecurity: Warning. Matched "Operator `Ge' with parameter `5' against variable `TX:ANOMALY_SCORE' (Value: `21' ) [file "/usr/share/modsecurity-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "80"] [id "949110"] [rev ""] [msg "Inbound Anomaly Score Exceeded (Total Score: 21)"] [data ""] [severity "2"] [ver ""] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "127.0.0.1"] [uri "http://localhost/"] [unique_id "1553327401.929417"] [ref ""]

ip address: [hostname "127.0.0.1"]
api: what do you think about?
ruleId: [id: "949110"]
rule tags: [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"]
etc: what do you think about it more?

note, that I'm working with "SecRuleEngine DetectOnly" mode.

> > There are three logs in v3: Debug, Msg/Error, Audit. I am talking about
> > the “error log” one. It could be “displayed” as o consequence of the rules
> > execution flow (log, auditlog action, etc) or consequence of a disruptive
> > action. The first you retrieve via callback, the second via disruptive
> > structure.
> >
> 
> The "error log" one is the one I am talking about, as well.

sure, in the debug log, I think I got the all triggered rule:

grep Rule: modsec_debug.log  | wc -l
541

eg:
[1553327401.929417] [http://localhost/?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d] [4] (Rule: 200000) Executing operator "Rx" with param "(?:application(?:/soap\+|/)|text/)xml" against REQUEST_HEADERS:Content-Type.
[1553327401.929417] [http://localhost/?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d] [4] (Rule: 200001) Executing operator "Rx" with param "application/json" against REQUEST_HEADERS:Content-Type.
[1553327401.929417] [http://localhost/?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d] [4] (Rule: 900000) Executing unconditional rule...
...

> > Jai, from your initial email I had the feeling that you would like to have
> > the id in a data structure at the disruptive message or having the logging
> > callback to replace the logging data inside the disruptive structure. It
> > will be a pleasure to make the library more flexible, but, can you give us
> > a brief explanation of your requirements?
> >
> 
> The requirement is for the calling code to be able to determine if a rule
> was triggered, regardless of whether the rule was disruptive or not. And,
> if triggered, the calling code should be provided details about the
> trigger.

for this I supposed we can use a new configuration variable,
which can control that all triggered rule will logged or not.
 
> I may have found a workaround for 3.0.3. Details below.

I'll see that later, now I just wanted to reflect first part of
your mail :).

Hope this helped to clear some things.


a.
 

Re: [Mod-security-developers] Finding triggered RuleIds

[Felipe C. <FC...@tr...>]

Hey Ervin,

I know you are working on fixies on CRS test bed and corelated stuff in order to make it compatible/consistent with v3, congrats a lot of work there. I know that you also have a patch to change v3 in order to make it match the expected data on the test beds. I will have a look at it again on the next week. Sorry on hurry here.

It may be confusing but I am: Felipe "Zimmerle" Costa or any permutation ( I am the guy that you are talking about ModSec on hangout sessions, the very same on GitHub and Slack and Email (

If I remember correctly, You and Victor Hora had being discussing on our Slack channel, among other things, about the semantic of block being or not a disruptive action, in fact it is a poser as a linker that may call a disruptive action -- not talking about the manual but how it is currently implemented. Notice also that proxy is not supported on v3. The reason why I am saying that, is the fact that we do have a lot of information on the example that you have posted. The amount of information could: (a) be wrongly implemented at any point; (b) be misinterpreted and lead us to false conclusions. My suggestion is to reduce the tests to a very little amount of rules/actions until it demonstrated without a doubt where the problem is. In fact, those could be implemented as v3 test cases. There is an error_log entry on the test case that was built for that.  Not sure if it is handling multiple entries, guess so. Put a break on regression.cc:90 it will tell you [sorry, not on a computer].
- https://github.com/SpiderLabs/ModSecurity/blob/b392a1ca36181a786a6d68b6eab57a8bb0bd558e/test/test-cases/regression/offset-variable.json#L54


You may found a bug; it demands further investigation. We have to narrow down where is the bug in order for us to fix. It could be even related to the v3 state [off, on, detection] that is only trigged in certain conditions. If you want we can do a hangout session to do it together, but I am not available before 1 April. I am with Victor on BlackHat Asia to present the new stuff on v3.1.

If there is a bug, it needs to be fixed in our side [and will]; Jai does not need to fix our bug on what he is coding (At least, I hope not.

Br.,
Felipe “Zimmerle” Costa
Security Researcher, Lead Developer ModSecurity.
 
Trustwave | SMART SECURITY ON DEMAND
www.trustwave.com <http://www.trustwave.com/>
 

On 3/23/19, 6:07 AM, "Ervin Hegedüs" <ai...@gm...> wrote:

    Hi Jai, Felipe,
    
    On Fri, Mar 22, 2019 at 10:05:29PM -0500, Jai Harpalani via mod-security-developers wrote:
    
    pre-note: I'm working on libmodsecurity3, fix bugs/different
    behaviours than ModSecurity2. The reference ruleset is owasp CRS
    3.1.
    
    To understand the libmodsecurity3 API, I've created a small (and
    really ugly :)) http server, which uses API calls.
    
    Of course, then I confronted with your logging issue.
     
    > In 3.0.2, I believe the callback was always invoked when a rule was
    > triggered. With 3.0.3, it is not invoked for a disruptive action.
    
    Disruptive actions are those that causes ModSecurity to do
    something. They are: block, deny, drop, allow, proxy and
    redirect.
    
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JPoZnsCL3Q&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fModSecurity%2fwiki%2fReference-Manual-%28v2%2ex%29%23secruleengine
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JKIeyMPbjw&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fModSecurity%2fwiki%2fReference-Manual-%28v2%2ex%29%23actions
    
    I've checked your statement with my ugly httpd with this request:
    
    GET /?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d HTTP/1.1
    Host: localhost
    
    and in the my log (error.log!) I got these rules:
    
    id "920280" 
    id "920320" 
    id "920273"
    id "941330"
    id "942110"
    id "942432"
    id "949110" 
    id "980130" 
    
    Let's see, which are disputive and which isn't:
    
    920280:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JPUSz82H2Q&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fREQUEST-920-PROTOCOL-ENFORCEMENT%2econf%23L593-L595
    non disruptiv
    
    920320:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JPtPyJTa3w&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fREQUEST-920-PROTOCOL-ENFORCEMENT%2econf%23L1328-L1330
    non disruptiv
    
    920273:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JPAfyM2I3A&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fREQUEST-920-PROTOCOL-ENFORCEMENT%2econf%23L1472-L1474
    disruptiv
    
    941330:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JPIdl8Pdiw&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fREQUEST-941-APPLICATION-ATTACK-XSS%2econf%23L876-L878
    disruptiv
    
    942110:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JPMSm8zc1g&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fREQUEST-942-APPLICATION-ATTACK-SQLI%2econf%23L505-L508
    disruptive
    
    942432:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JPESnsyH3Q&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fREQUEST-942-APPLICATION-ATTACK-SQLI%2econf%23L1580-L1582
    disruptive
    
    949110:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JKcbnczfjQ&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fREQUEST-949-BLOCKING-EVALUATION%2econf%23L81-L83
    disruptive
    
    980130:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JKISz5Ta3g&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fRESPONSE-980-CORRELATION%2econf%23L79-L81
    non-disputive
    
    so, looks like libmodsecurity3, version 3.0.3 logs mixed
    disruptive and non-disruptive triggered rules.
    
    
    > My
    > calling code was expecting the callback to always be invoked because that
    > is the only way we know that a rule has triggered.
    
    yes, you're right, this is a new feature request.
    
    > You state that for a
    > disruptive action, "the logging message is delivered via the disruptive
    > structure in text format". Where is this done?
    
    I'm not sure that understand you, but here is my snippet:
    
    ...
        modsec = msc_init();
        msc_set_log_cb(modsec, msc_logdata);
    ...
    
    upper...
    
    void msc_logdata(void *log, const void *data) {
        FILE *fp;
        char logline[4096];
    
        struct timeval tv;
    
        wrote_log_line(...); 
        fclose(fp);
    }
    
    and that's it. Not need to call this function explicitly, but I
    guess you know that - that's why I asked you above, what do you
    think about whit this question.
    
    > How is it delivered? How can
    > the calling code inspect this disruptive structure?
    
    there are some examples in examples/ subdirectory.
    
    > And, does the
    > disruptive structure contain all the details which were provided by the
    > callback (eg ip address, api, ruleId, rule tags, etc)?
    
    hmm... here is a logged line:
    
    1553327402.117947 ModSecurity: Warning. Matched "Operator `Ge' with parameter `5' against variable `TX:ANOMALY_SCORE' (Value: `21' ) [file "/usr/share/modsecurity-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "80"] [id "949110"] [rev ""] [msg "Inbound Anomaly Score Exceeded (Total Score: 21)"] [data ""] [severity "2"] [ver ""] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "127.0.0.1"] [uri "http://localhost/"] [unique_id "1553327401.929417"] [ref ""]
    
    ip address: [hostname "127.0.0.1"]
    api: what do you think about?
    ruleId: [id: "949110"]
    rule tags: [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"]
    etc: what do you think about it more?
    
    note, that I'm working with "SecRuleEngine DetectOnly" mode.
    
    > > There are three logs in v3: Debug, Msg/Error, Audit. I am talking about
    > > the “error log” one. It could be “displayed” as o consequence of the rules
    > > execution flow (log, auditlog action, etc) or consequence of a disruptive
    > > action. The first you retrieve via callback, the second via disruptive
    > > structure.
    > >
    > 
    > The "error log" one is the one I am talking about, as well.
    
    sure, in the debug log, I think I got the all triggered rule:
    
    grep Rule: modsec_debug.log  | wc -l
    541
    
    eg:
    [1553327401.929417] [http://localhost/?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d] [4] (Rule: 200000) Executing operator "Rx" with param "(?:application(?:/soap\+|/)|text/)xml" against REQUEST_HEADERS:Content-Type.
    [1553327401.929417] [http://localhost/?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d] [4] (Rule: 200001) Executing operator "Rx" with param "application/json" against REQUEST_HEADERS:Content-Type.
    [1553327401.929417] [http://localhost/?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d] [4] (Rule: 900000) Executing unconditional rule...
    ...
    
    > > Jai, from your initial email I had the feeling that you would like to have
    > > the id in a data structure at the disruptive message or having the logging
    > > callback to replace the logging data inside the disruptive structure. It
    > > will be a pleasure to make the library more flexible, but, can you give us
    > > a brief explanation of your requirements?
    > >
    > 
    > The requirement is for the calling code to be able to determine if a rule
    > was triggered, regardless of whether the rule was disruptive or not. And,
    > if triggered, the calling code should be provided details about the
    > trigger.
    
    for this I supposed we can use a new configuration variable,
    which can control that all triggered rule will logged or not.
     
    > I may have found a workaround for 3.0.3. Details below.
    
    I'll see that later, now I just wanted to reflect first part of
    your mail :).
    
    Hope this helped to clear some things.
    
    
    a.