Re: [Mod-security-developers] Finding triggered RuleIds

[Ervin H. <ai...@gm...>]

Hi Jai, Felipe,

On Fri, Mar 22, 2019 at 10:05:29PM -0500, Jai Harpalani via mod-security-developers wrote:

pre-note: I'm working on libmodsecurity3, fix bugs/different
behaviours than ModSecurity2. The reference ruleset is owasp CRS
3.1.

To understand the libmodsecurity3 API, I've created a small (and
really ugly :)) http server, which uses API calls.

Of course, then I confronted with your logging issue.
 
> In 3.0.2, I believe the callback was always invoked when a rule was
> triggered. With 3.0.3, it is not invoked for a disruptive action.

Disruptive actions are those that causes ModSecurity to do
something. They are: block, deny, drop, allow, proxy and
redirect.

https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#secruleengine
https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#actions

I've checked your statement with my ugly httpd with this request:

GET /?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d HTTP/1.1
Host: localhost

and in the my log (error.log!) I got these rules:

id "920280" 
id "920320" 
id "920273"
id "941330"
id "942110"
id "942432"
id "949110" 
id "980130" 

Let's see, which are disputive and which isn't:

920280:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf#L593-L595
non disruptiv

920320:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf#L1328-L1330
non disruptiv

920273:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf#L1472-L1474
disruptiv

941330:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf#L876-L878
disruptiv

942110:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf#L505-L508
disruptive

942432:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf#L1580-L1582
disruptive

949110:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/REQUEST-949-BLOCKING-EVALUATION.conf#L81-L83
disruptive

980130:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/RESPONSE-980-CORRELATION.conf#L79-L81
non-disputive

so, looks like libmodsecurity3, version 3.0.3 logs mixed
disruptive and non-disruptive triggered rules.


> My
> calling code was expecting the callback to always be invoked because that
> is the only way we know that a rule has triggered.

yes, you're right, this is a new feature request.

> You state that for a
> disruptive action, "the logging message is delivered via the disruptive
> structure in text format". Where is this done?

I'm not sure that understand you, but here is my snippet:

...
    modsec = msc_init();
    msc_set_log_cb(modsec, msc_logdata);
...

upper...

void msc_logdata(void *log, const void *data) {
    FILE *fp;
    char logline[4096];

    struct timeval tv;

    wrote_log_line(...); 
    fclose(fp);
}

and that's it. Not need to call this function explicitly, but I
guess you know that - that's why I asked you above, what do you
think about whit this question.

> How is it delivered? How can
> the calling code inspect this disruptive structure?

there are some examples in examples/ subdirectory.

> And, does the
> disruptive structure contain all the details which were provided by the
> callback (eg ip address, api, ruleId, rule tags, etc)?

hmm... here is a logged line:

1553327402.117947 ModSecurity: Warning. Matched "Operator `Ge' with parameter `5' against variable `TX:ANOMALY_SCORE' (Value: `21' ) [file "/usr/share/modsecurity-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "80"] [id "949110"] [rev ""] [msg "Inbound Anomaly Score Exceeded (Total Score: 21)"] [data ""] [severity "2"] [ver ""] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "127.0.0.1"] [uri "http://localhost/"] [unique_id "1553327401.929417"] [ref ""]

ip address: [hostname "127.0.0.1"]
api: what do you think about?
ruleId: [id: "949110"]
rule tags: [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"]
etc: what do you think about it more?

note, that I'm working with "SecRuleEngine DetectOnly" mode.

> > There are three logs in v3: Debug, Msg/Error, Audit. I am talking about
> > the “error log” one. It could be “displayed” as o consequence of the rules
> > execution flow (log, auditlog action, etc) or consequence of a disruptive
> > action. The first you retrieve via callback, the second via disruptive
> > structure.
> >
> 
> The "error log" one is the one I am talking about, as well.

sure, in the debug log, I think I got the all triggered rule:

grep Rule: modsec_debug.log  | wc -l
541

eg:
[1553327401.929417] [http://localhost/?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d] [4] (Rule: 200000) Executing operator "Rx" with param "(?:application(?:/soap\+|/)|text/)xml" against REQUEST_HEADERS:Content-Type.
[1553327401.929417] [http://localhost/?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d] [4] (Rule: 200001) Executing operator "Rx" with param "application/json" against REQUEST_HEADERS:Content-Type.
[1553327401.929417] [http://localhost/?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d] [4] (Rule: 900000) Executing unconditional rule...
...

> > Jai, from your initial email I had the feeling that you would like to have
> > the id in a data structure at the disruptive message or having the logging
> > callback to replace the logging data inside the disruptive structure. It
> > will be a pleasure to make the library more flexible, but, can you give us
> > a brief explanation of your requirements?
> >
> 
> The requirement is for the calling code to be able to determine if a rule
> was triggered, regardless of whether the rule was disruptive or not. And,
> if triggered, the calling code should be provided details about the
> trigger.

for this I supposed we can use a new configuration variable,
which can control that all triggered rule will logged or not.
 
> I may have found a workaround for 3.0.3. Details below.

I'll see that later, now I just wanted to reflect first part of
your mail :).

Hope this helped to clear some things.


a.