mod-security-developers — Development discussion about mod_security

[Mod-security-developers] ModSecurity version 2.9.4

[Felipe Z. <fe...@zi...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hi,

It is a pleasure to announce ModSecurity version 2.9.4. Release 2.9.4
contains fixes and enhancements atop version 2.9.3.

The details on this release are available on GitHub:
 - https://github.com/SpiderLabs/ModSecurity/releases/tag/v2.9.4

Further details on ModSecurity v2 can be found on the project README:
 - https://github.com/SpiderLabs/ModSecurity/tree/v2/master

The list of open issues is available on GitHub:
 - https://github.com/SpiderLabs/ModSecurity/labels/2.x

IMPORTANT:
- - Windows installer no longer includes OWASP CRS.
- - Release files are no longer available on modsecurity.org, only on GitHub.
- - ModSecurity version 2 will be available and maintained parallel
to version 3. There is no ETA to deprecate version 2.x. ModSecurity versions
2 and 3 have a completely independent development/release cycle.

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iFwEARECAB0WIQQZDvrMoen6RmqOzZzm37CM6LESdwUCYNEVfgAKCRDm37CM6LES
dxUkAJ4naJ9ysl5HZPSEhmO0wxrLduDI4wCYzRhp8EuuSp/TW5uVNdF+eDl8UA==
=yA1i
-----END PGP SIGNATURE-----

[Mod-security-developers] ModSecurity version 2.9.4

[Felipe Z. <fe...@zi...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hi,

It is a pleasure to announce ModSecurity version 2.9.4. Release 2.9.4
contains fixes and enhancements atop version 2.9.3.

The details on this release are available on GitHub:
 - https://github.com/SpiderLabs/ModSecurity/releases/tag/v2.9.4

Further details on ModSecurity v2 can be found on the project README:
 - https://github.com/SpiderLabs/ModSecurity/tree/v2/master

The list of open issues is available on GitHub:
 - https://github.com/SpiderLabs/ModSecurity/labels/2.x

IMPORTANT:
- - Windows installer no longer includes OWASP CRS.
- - Release files are no longer available on modsecurity.org, only on GitHub.
- - ModSecurity version 2 will be available and maintained parallel
to version 3. There is no ETA to deprecate version 2.x. ModSecurity versions
2 and 3 have a completely independent development/release cycle.

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iFwEARECAB0WIQQZDvrMoen6RmqOzZzm37CM6LESdwUCYNEVfgAKCRDm37CM6LES
dxUkAJ4naJ9ysl5HZPSEhmO0wxrLduDI4wCYzRhp8EuuSp/TW5uVNdF+eDl8UA==
=yA1i
-----END PGP SIGNATURE-----

[Mod-security-developers] CVE-2020-15598 – ModSecurity v3 Affected By DoS (Severity HIGH)

[Christian F. <chr...@ne...>]

Dear all,

ModSecurity v3.0.x is affected by a Denial of Service vulnerability due to the
global matching of regular expressions. The combination of a non-anchored
regular expression and the ModSecurity “capture” action can be exploited via a
specially crafted payload.

While ModSecurity v2.x used to quit the execution of a regular expression
after the first match. ModSecurity v3.0.x silently changed the behavior to
global matching. This results in a DoS for existing non-anchored regexes in
rules containing the “capture” action. It also fills the TX variable space
beyond the documented limit of 10 instances. The defense is handicapped due to
the absence of the SecRequestBodyNoFilesLimit directive. The vendor Trustwave
Spiderlabs dropped this functionality for ModSecurity v3.

The vendor did not publish a new release, but there is a patch that brings
back the former behavior.

CVSSv3: 7.5 HIGH

https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator?vector=AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H&version=3.1

Exploitability Metrics:
* Attack Vector: Network
* Attack Complexity: Low
* Priviledges Required: None
* User Interaction: None
* Scope: Unchanged
Impact Metrics:
* Confidentiality Impact: None
* Integrity Impact: None
* Availability Impact: High

Weakness Enumeration: CWE-400: Uncontrolled Resource Consumption

Known Affected Software Configurations:

    ModSecurity v3.0.0
    ModSecurity v3.0.1
    ModSecurity v3.0.2
    ModSecurity v3.0.3
    ModSecurity v3.0.4 (patch for this version available)

More Information and patch: https://coreruleset.org/20200914/cve-2020-15598/



Best regards,

Christian Folini and Ervin Hegedüs on behalf of the CRS team

-- 
OWASP ModSecurity Core Rule Set - https://coreruleset.org

[Mod-security-developers] ModSecurity v3.0.4 announcement

[Felipe Z. <fe...@zi...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hi

It is a pleasure to announce the release of ModSecurity version 3.0.4
(libModSecurity). This version contains a number of improvements in
different areas. These include cleanups, better practices for improved
code readability, resilience and overall performance and security
fixes.

A huge refactoring was placed on the Regex engine, which is now more
performant. The Logging was polished and hex-encoded strings are now
pretty printed. An operator to detect Australian social security
number was added. The audit log is now working with section H and
better dealing with logs, nologs and auditlogs combinations.

POTENTIAL SECURITY ISSUES:

- - Cookie parser problems
 [@theMiddleBlue, @airween, @martinhsv]

The list with the full changes can be found on the project CHANGES file,
available here:
- - https://github.com/SpiderLabs/ModSecurity/releases/tag/v3.0.4/CHANGES

The list of open issues is available on GitHub:
- - https://github.com/SpiderLabs/ModSecurity/labels/3.x

As with every new release, a milestone was created to host all the issues
that will be fixed till we reach the given milestone. With that, we not
only give the community the full transparency of the work that is being
doing on ModSec, but also even more chances to participate. Milestones
give the chance to anyone from the community to deduce when and what will
be released.

Thanks to everybody who helped in this process: reporting issues, making
comments and suggestions, sending patches and so on.

Further details on the compilation process for ModSecurity v3, can be
found on the project README:
 - https://github.com/SpiderLabs/ModSecurity/tree/v3/master#compilation

Complementary documentation for the connectors are available here:
 - nginx: https://github.com/SpiderLabs/ModSecurity-nginx/#compilation
 - Apache: https://github.com/SpiderLabs/ModSecurity-apache/#compilation

IMPORTANT: ModSecurity version 2 will be available and maintained parallel
to version 3. There is no ETA to deprecate the version 2.x. New features
and major improvements will be implemented on version 3.x. Security or
major bugs are planned to be back ported. Version 2 and version 3 has a
completely independent development/release cycle.


Br.,
Felipe "Zimmerle" Costa

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iF0EARECAB0WIQQZDvrMoen6RmqOzZzm37CM6LESdwUCXhxx8QAKCRDm37CM6LES
dy8jAJ4l6Goa0qn+RyxwrFPa8Zjl9t8HagCeJeHULU8EsT2M2S0Ho6ROgOdQstM=
=GeNp
-----END PGP SIGNATURE-----

[Mod-security-developers] Problem in retrieving collections from Persistant Storage

[Marc S. <mar...@ap...>]

I had problems when loading the IP collection - values were not correct.

In collection_retrieve_ex(), we check if a key exist with the name "KEY":
if (apr_table_get(col, "KEY") == NULL) ...

In collection_store(), we store the key with the name "__KEY":
var_key = (msc_string *)apr_table_get(col, "__key");

By using "__KEY" at both places, everything works smoothly.

Can somebody explain the logic?

Thanks

[Mod-security-developers] modsecurity_request_body_to_stream() logic

[Marc S. <mar...@ap...>]

Hello,

While looking at memory leaks inside 
modsecurity_request_body_to_stream(), I found some strange logic I 
couldn't really explain but by mistakes.
I've been looking at the code when MSC_LARGE_STREAM_INPUT is NOT defined.
Can somebody validate a few points:

1. we always allocate (msr->stream_input_length + 1) bytes (potentitally 
-buflen when relevant)
Do we really need this extra-byte (used for EOS I think)?

2. we have some memset(msr->stream_input_data, 0, ...) at several places
For me, these are only used to set a 0 at EOS (thus last byte).
We shouldn't loose time to set all bytes to 0, but I think that the last 
one is also useless.
Example, when in first packet, we have that code:
memset(msr->stream_input_data, 0, msr->stream_input_length+1);
if(first_pkt)   {
    memcpy(msr->stream_input_data, buffer, msr->stream_input_length);
}
Remark: potentially a EOS could be added at the very end of the whole 
buffer processing, if really needed - or we can add it every time if simpler

3. when not in first packet, we have that (simplified) code:
// copy beginning of msr->stream_input_data in a new buffer
data = (char *)malloc(msr->stream_input_length + 1 - buflen);
memset(data, 0, msr->stream_input_length + 1 - buflen); // only EOS 
should be enough
memcpy(data, msr->stream_input_data, msr->stream_input_length - buflen);
=> useless because msr->stream_input_data content is preserved in realloc()

// realloc msr->stream_input_data
stream_input_body = (char *)realloc(msr->stream_input_data, 
msr->stream_input_length + 1);
msr->stream_input_data = (char *)stream_input_body;

memset(msr->stream_input_data, 0, msr->stream_input_length+1); // only 
EOS should be enough
memcpy(msr->stream_input_data, data, msr->stream_input_length - buflen); 
// already done in realloc()
// append buffer (buflen)
memcpy(msr->stream_input_data+(msr->stream_input_length - buflen), 
buffer, buflen);

I think this whole code could be simplified:
stream_input_body = (char *)realloc(msr->stream_input_data, 
msr->stream_input_length + 1);
msr->stream_input_data = stream_input_body;
memcpy(msr->stream_input_data+(msr->stream_input_length - buflen), 
buffer, buflen);
=> did I miss something?

[Mod-security-developers] Announcement: msc_pyparser

[Ervin H. <ai...@gm...>]

Hi all,

(sorry for the cross posting)

let me announce the msc_pyparser tool, a ModSecurity ruleset parser.
msc_pyparser can made a full lexical and syntactic analisys on the rules
(currently it tested only on CRS and some custom rules). Note, that
msc_pyparser supports only four MSC keyword (and the comments) - see the
docs.
Beyond these abilities, msc_pyparser builds an AST (abstract syntax tree),
and made an own structure in memory - especially a list of dictionaries.
Every dict item stores several datas about the recognized token, including
number of line.

You can dump this structure (eg. JSON, YAML - or SQL), or can make any
contextual depend modification. After this, you can save back the modified
version to the original ModSecurity format.

There are some usefull examples in source.

I think this tool can helps you to maintain your rulesets, eg. merge with
custom modifications, formatting, or - as above - make any context
dependent changes. Please review the examples directory.

If you have any question, issue, bugreport or other feedback, please
contact me at the given e-mail address in README, or open an issue on
Github (do not disturb this list).

The tool available here:
https://github.com/digitalwave/msc_pyparser

a.

Re: [Mod-security-developers] Announce of ftwrunner

[Christian F. <chr...@ne...>]

Congratulations, Ervin.

This is a very welcome addition to our toolbox. Not the least for CRS
development.

Ervin is also presenting the tool at the OWASP ModSecurity Core Rule Set
community summit next Wednesday in Amsterdam. If anybody has not hooked up
so far and wants to join, please get in touch.

Best,

Christian

On Thu, Sep 19, 2019 at 08:45:00AM +0200, Ervin Hegedüs wrote:
> Hi all,
> 
> (sorry for the cross posting)
> 
> let me announce the ftwrunner tool, a libmodsecurity3 testing helper.
> Ftwrunner can use the existing CRS tests (which made for ftw), but of
> course you can write your own rules. The tool uses libmodsecurity3 API,
> instead of sending HTTP requests to the configured webserver. You can check
> your whole config (base modsecurity.conf, your rulesets) as ftw.
> 
> I think this tool also can helps you to check the library - similar to
> existing regression test framework.
> 
> If you have any question, issue, bugreport or other feedback, please
> contact me at the given e-mail address in README, or open an issue on
> Github (do not disturb this list).
> 
> The tool available here:
> https://github.com/digitalwave/ftwrunner
> 
> 
> a.


> _______________________________________________
> mod-security-developers mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
> ModSecurity Services from Trustwave's SpiderLabs:
> https://www.trustwave.com/spiderLabs.php

[Mod-security-developers] Announce of ftwrunner

[Ervin H. <ai...@gm...>]

Hi all,

(sorry for the cross posting)

let me announce the ftwrunner tool, a libmodsecurity3 testing helper.
Ftwrunner can use the existing CRS tests (which made for ftw), but of
course you can write your own rules. The tool uses libmodsecurity3 API,
instead of sending HTTP requests to the configured webserver. You can check
your whole config (base modsecurity.conf, your rulesets) as ftw.

I think this tool also can helps you to check the library - similar to
existing regression test framework.

If you have any question, issue, bugreport or other feedback, please
contact me at the given e-mail address in README, or open an issue on
Github (do not disturb this list).

The tool available here:
https://github.com/digitalwave/ftwrunner


a.

[Mod-security-developers] Using ModSecurity 3 with SIP instead of HTTP.

[Michel P. <mi...@vo...>]

Hi,

Congrats for ModSecurity 3.  I've been waiting for a while for a version
where the core engine would be completely decoupled from its Apache
implementation.  I work in VoIP and I can assure you there are just as many
threats to fight off as there are with HTTP.  I wonder if it would be
possible to use ModSecurity with SIP instead of HTTP.  The two protocols
share the same core specification.  Would it be just a matter of writing
the proper connector?

Best regards,

Michel.

[Mod-security-developers] Mod-security-Issue

[KAMLAPURI P. <pra...@so...>]

Hello,

The issue is regarding for implementation of ModSecurity (both Version 2.8 and 2.9) over IIS 7.5 in Window server 2008 R2. I have followed below mentioned 2 link and have tried to implemented it one of our test server using below 2 link:

https://admin-ahead.com/forum/server-security-hardening-21/installing-and-configuring-mod_security-on-windows-server/

https://jesscoburn.com/archives/2013/05/14/installing-modsecurity-on-iis7-x/

Post installation of all steps and configuration in application web.config file, to ensure whether ModSecurity is working or not, I have taken the reference of above link and created a test rule:

SecRule ARGS, "zzz" phase:1,log,deny,status:503,id:1 in the modsecurity.conf file and set SecRuleEngine On

Then I have browsed application: http://localhost/Sitename/default.aspx?a=zzz. An error (503) should be expected but I am not getting any 503 error.

Setting in config:
<system.webServer> <ModSecurity enabled="true" configFile="C:\Program Files\ModSecurity IIS\modsecurity_iis.conf" /> </system.webServer>

We can see below logs in Event Viewer in given order. Below information are being logged in Event viewer in first hit of application.

ModSccurity | 28-03-2019 12:57:00 | The description for Event ID 0 from source ModSecurity cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.   If the event originated on another computer, the display information had to be saved with the event.   The following information was included with the event:   ModSecurity for IIS (STABLE)/2.8.0 (http://www.modsecurity.org/) configured.

ModSccurity | 28-03-2019 12:57:00 | The description for Event ID 0 from source ModSecurity cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.   If the event originated on another computer, the display information had to be saved with the event.   The following information was included with the event:   ModSecurity: APR compiled version="1.4.8"; loaded version="1.4.8"

ModSccurity | 28-03-2019 12:57:00 | The description for Event ID 0 from source ModSecurity cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.   If the event originated on another computer, the display information had to be saved with the event.   The following information was included with the event:   ModSecurity: PCRE compiled version="8.33 "; loaded version="8.33 2013-05-28"

ModSccurity | 28-03-2019 12:57:00 | The description for Event ID 0 from source ModSecurity cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.   If the event originated on another computer, the display information had to be saved with the event.   The following information was included with the event:   ModSecurity: LUA compiled version="Lua 5.1"

ModSccurity | 28-03-2019 12:57:00 | The description for Event ID 0 from source ModSecurity cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.   If the event originated on another computer, the display information had to be saved with the event.   The following information was included with the event:   ModSecurity: LIBXML compiled version="2.9.1"

Please help me to sort out this issue as i have already invested 3 days but unable to fix this.

Thank you in advance!!

[Mod-security-developers] Rule evaluation question

[Jai H. <jai...@mu...>]

Any way to stop rule evaluation on the request side after a request rule
"triggers", but then continue with response rule evaluation?

>From my crs-setup.conf file:

SecDefaultAction "phase:1,log,auditlog,deny,status:403"
SecDefaultAction "phase:2,log,auditlog,deny,status:403"

# For response (phase 3/4), continue processing after first rule is
# triggered.
#SecDefaultAction "phase:3,log,auditlog,deny,status:403"
#SecDefaultAction "phase:4,log,auditlog,deny,status:403"

With above settings, if a rule is triggered during phase 1 or phase 2, the
remaining phases don't evaluate rules. I would like just the rules within
the phase which triggered the rule to be skipped, but not subsequent ones.

Any way to achieve this?

Re: [Mod-security-developers] ModSecurity with afl - fuzz

[Ervin H. <ai...@gm...>]

Hi Jerald,

On Mon, Apr 22, 2019 at 06:33:12PM +0800, Jerald Cheong wrote:
> I've been trying to compile ModSecurity with afl-fuzz but been having no
> luck after my nginx got upgraded.
> 
> Any guidance would be most appreciated.
> 
> I wanted to see what is the difference with afl-fuzz.

what do you want to see, I mean what's your expected result as
"difference"?

As I know, the afl-fuzz is "just" a test method, but as I see,
the implementation in libmodsecurity3 is not final.

> Tried both ModSecurity, tag 3.0.3 and ModSecurity Master from github.
> 
> Environment:
> CentOS Linux release 7.6.1810 (Core)
> SCL devtoolset-7 llvm-toolset-7 -->> For clang 5.0.1
> afl compiled from source: https://github.com/mirrorer/afl/
> 
> Configure options: ./configure --with-lmdb --enable-parser-generation
> --enable-afl-fuzz

I think you should skip the --enable-afl-fuzz switch to get a
worling instance.

And please note, that if you use lmdb as collection backend, the
expire var does not work:

https://github.com/SpiderLabs/ModSecurity/issues/1803



Regards,

a.
 

[Mod-security-developers] ModSecurity with afl - fuzz

[Jerald C. <jer...@gm...>]

I've been trying to compile ModSecurity with afl-fuzz but been having no
luck after my nginx got upgraded.

Any guidance would be most appreciated.

I wanted to see what is the difference with afl-fuzz.

Tried both ModSecurity, tag 3.0.3 and ModSecurity Master from github.

Environment:
CentOS Linux release 7.6.1810 (Core)
SCL devtoolset-7 llvm-toolset-7 -->> For clang 5.0.1
afl compiled from source: https://github.com/mirrorer/afl/

Configure options: ./configure --with-lmdb --enable-parser-generation
--enable-afl-fuzz

This is the final error:
afl-clang-fast 2.52b by <lszekeres@******.com>
clang-5.0: warning: argument '-fsanitize-coverage=4' is deprecated, use
'-fsanitize-coverage=trace-pc-guard' instead [-Wdeprecated]
afl_fuzzer.cc:24:48: warning: '/*' within block comment [-Wcomment]
 * for i in $(ls -l src/actions/transformations/*.h | awk {'print $9'})...
                                               ^
afl_fuzzer.cc:67:34: warning: '/*' within block comment [-Wcomment]
 * for i in $(ls -l src/operators/*.h | awk {'print $9'}); do echo "#inc...
                                 ^
afl_fuzzer.cc:147:67: warning: '/*' within block comment [-Wcomment]
        * for i in $(grep "class " -Ri src/actions/transformations/* | grep
" :...
                                                                  ^
afl_fuzzer.cc:192:53: warning: '/*' within block comment [-Wcomment]
        * for i in $(grep "class " -Ri src/operators/* | grep " :" | aw...
                                                    ^
afl_fuzzer.cc:195:30: error: no matching constructor for initialization of
      'modsecurity::operators::BeginsWith'
BeginsWith *beginswith = new BeginsWith("BeginsWith", z, false); beginsw...
                             ^          ~~~~~~~~~~~~~~~~~~~~~~
../../src/operators/begins_with.h:32:14: note: candidate constructor not
viable:
      requires single argument 'param', but 3 arguments were provided
    explicit BeginsWith(std::unique_ptr<RunTimeString> param)
             ^
../../src/operators/begins_with.h:29:7: note: candidate constructor (the
      implicit copy constructor) not viable: requires 1 argument, but 3 were
      provided
class BeginsWith : public Operator {
      ^
afl_fuzzer.cc:195:91: error: too few arguments to function call, expected 4,
      have 2
  ...new BeginsWith("BeginsWith", z, false); beginswith->evaluate(t, s);
dele...
                                             ~~~~~~~~~~~~~~~~~~~~     ^
../../src/operators/begins_with.h:35:5: note: 'evaluate' declared here
    bool evaluate(Transaction *transaction, Rule *rule, const std::string
&str,
    ^
afl_fuzzer.cc:196:26: error: no matching constructor for initialization of
      'modsecurity::operators::Contains'
Contains *contains = new Contains("Contains", z, false); contains->evalu...
                         ^        ~~~~~~~~~~~~~~~~~~~~
../../src/operators/contains.h:35:14: note: candidate constructor not
viable:
      requires single argument 'param', but 3 arguments were provided
    explicit Contains(std::unique_ptr<RunTimeString> param)
             ^
../../src/operators/contains.h:32:7: note: candidate constructor (the
implicit
      copy constructor) not viable: requires 1 argument, but 3 were provided
class Contains : public Operator {
      ^
afl_fuzzer.cc:196:81: error: too few arguments to function call, expected 4,
      have 2
  ...= new Contains("Contains", z, false); contains->evaluate(t, s); delete
c...
                                           ~~~~~~~~~~~~~~~~~~     ^
../../src/operators/contains.h:37:5: note: 'evaluate' declared here
    bool evaluate(Transaction *transaction, Rule *rule,
    ^
afl_fuzzer.cc:197:34: error: no matching constructor for initialization of
      'modsecurity::operators::ContainsWord'
  ...*containsword = new ContainsWord("ContainsWord", z, false);
containsword...
                         ^            ~~~~~~~~~~~~~~~~~~~~~~~~
../../src/operators/contains_word.h:32:14: note: candidate constructor not
      viable: requires single argument 'param', but 3 arguments were
provided
    explicit ContainsWord(std::unique_ptr<RunTimeString> param)
             ^
../../src/operators/contains_word.h:29:7: note: candidate constructor (the
      implicit copy constructor) not viable: requires 1 argument, but 3 were
      provided
class ContainsWord : public Operator {
      ^
afl_fuzzer.cc:197:101: error: too few arguments to function call, expected
4,
      have 2
  ...ContainsWord("ContainsWord", z, false); containsword->evaluate(t, s);
de...
                                             ~~~~~~~~~~~~~~~~~~~~~~     ^
../../src/operators/contains_word.h:35:5: note: 'evaluate' declared here
    bool evaluate(Transaction *transaction, Rule *rule,
    ^
afl_fuzzer.cc:198:30: error: no matching constructor for initialization of
      'modsecurity::operators::DetectSQLi'
DetectSQLi *detectsqli = new DetectSQLi("DetectSQLi", z, false); detects...
                             ^          ~~~~~~~~~~~~~~~~~~~~~~
../../src/operators/detect_sqli.h:27:7: note: candidate constructor (the
      implicit copy constructor) not viable: requires 1 argument, but 3 were
      provided
class DetectSQLi : public Operator {
      ^
../../src/operators/detect_sqli.h:30:5: note: candidate constructor not
viable:
      requires 0 arguments, but 3 were provided
    DetectSQLi()
    ^
afl_fuzzer.cc:198:91: error: too few arguments to function call, expected 4,
      have 2
  ...new DetectSQLi("DetectSQLi", z, false); detectsqli->evaluate(t, s);
dele...
                                             ~~~~~~~~~~~~~~~~~~~~     ^
../../src/operators/detect_sqli.h:35:5: note: 'evaluate' declared here
    bool evaluate(Transaction *t, Rule *rule,
    ^
afl_fuzzer.cc:199:28: error: no matching constructor for initialization of
      'modsecurity::operators::DetectXSS'
DetectXSS *detectxss = new DetectXSS("DetectXSS", z, false); detectxss->...
                           ^         ~~~~~~~~~~~~~~~~~~~~~
../../src/operators/detect_xss.h:26:7: note: candidate constructor (the
implicit
      copy constructor) not viable: requires 1 argument, but 3 were provided
class DetectXSS : public Operator {
      ^
../../src/operators/detect_xss.h:29:5: note: candidate constructor not
viable:
      requires 0 arguments, but 3 were provided
    DetectXSS()
    ^
afl_fuzzer.cc:199:86: error: too few arguments to function call, expected 4,
      have 2
  ...= new DetectXSS("DetectXSS", z, false); detectxss->evaluate(t, s);
delet...
                                             ~~~~~~~~~~~~~~~~~~~     ^
../../src/operators/detect_xss.h:34:5: note: 'evaluate' declared here
    bool evaluate(Transaction *t, Rule *rule,
    ^
afl_fuzzer.cc:200:26: error: no matching constructor for initialization of
      'modsecurity::operators::EndsWith'
EndsWith *endswith = new EndsWith("EndsWith", z, false); endswith->evalu...
                         ^        ~~~~~~~~~~~~~~~~~~~~
../../src/operators/ends_with.h:32:14: note: candidate constructor not
viable:
      requires single argument 'param', but 3 arguments were provided
    explicit EndsWith(std::unique_ptr<RunTimeString> param)
             ^
../../src/operators/ends_with.h:29:7: note: candidate constructor (the
implicit
      copy constructor) not viable: requires 1 argument, but 3 were provided
class EndsWith : public Operator {
      ^
afl_fuzzer.cc:200:81: error: too few arguments to function call, expected 4,
      have 2
  ...= new EndsWith("EndsWith", z, false); endswith->evaluate(t, s); delete
e...
                                           ~~~~~~~~~~~~~~~~~~     ^
../../src/operators/ends_with.h:36:5: note: 'evaluate' declared here
    bool evaluate(Transaction *transaction, Rule *rule,
    ^
afl_fuzzer.cc:201:14: error: no matching constructor for initialization of
      'modsecurity::operators::Eq'
Eq *eq = new Eq("Eq", z, false); eq->evaluate(t, s); delete eq;
             ^  ~~~~~~~~~~~~~~
../../src/operators/eq.h:32:14: note: candidate constructor not viable:
requires
      single argument 'param', but 3 arguments were provided
    explicit Eq(std::unique_ptr<RunTimeString> param)
             ^
../../src/operators/eq.h:29:7: note: candidate constructor (the implicit
copy
      constructor) not viable: requires 1 argument, but 3 were provided
class Eq : public Operator {
      ^
afl_fuzzer.cc:202:28: error: no matching constructor for initialization of
      'modsecurity::operators::FuzzyHash'
FuzzyHash *fuzzyhash = new FuzzyHash("FuzzyHash", z, false); fuzzyhash->...
                           ^         ~~~~~~~~~~~~~~~~~~~~~
../../src/operators/fuzzy_hash.h:41:14: note: candidate constructor not
viable:
      requires single argument 'param', but 3 arguments were provided
    explicit FuzzyHash(std::unique_ptr<RunTimeString> param)
             ^
../../src/operators/fuzzy_hash.h:38:7: note: candidate constructor (the
implicit
      copy constructor) not viable: requires 1 argument, but 3 were provided
class FuzzyHash : public Operator {
      ^
afl_fuzzer.cc:203:14: error: no matching constructor for initialization of
      'modsecurity::operators::Ge'
Ge *ge = new Ge("Ge", z, false); ge->evaluate(t, s); delete ge;
             ^  ~~~~~~~~~~~~~~
../../src/operators/ge.h:31:14: note: candidate constructor not viable:
requires
      single argument 'param', but 3 arguments were provided
    explicit Ge(std::unique_ptr<RunTimeString> param)
             ^
../../src/operators/ge.h:28:7: note: candidate constructor (the implicit
copy
      constructor) not viable: requires 1 argument, but 3 were provided
class Ge : public Operator {
      ^
afl_fuzzer.cc:204:28: error: no matching constructor for initialization of
      'modsecurity::operators::GeoLookup'
GeoLookup *geolookup = new GeoLookup("GeoLookup", z, false); geolookup->...
                           ^         ~~~~~~~~~~~~~~~~~~~~~
../../src/operators/geo_lookup.h:27:7: note: candidate constructor (the
implicit
      copy constructor) not viable: requires 1 argument, but 3 were provided
class GeoLookup : public Operator {
      ^
../../src/operators/geo_lookup.h:30:5: note: candidate constructor not
viable:
      requires 0 arguments, but 3 were provided
    GeoLookup()
    ^
afl_fuzzer.cc:205:28: error: no matching constructor for initialization of
      'modsecurity::operators::GsbLookup'
GsbLookup *gsblookup = new GsbLookup("GsbLookup", z, false); gsblookup->...
                           ^         ~~~~~~~~~~~~~~~~~~~~~
../../src/operators/gsblookup.h:31:14: note: candidate constructor not
viable:
      requires single argument 'param', but 3 arguments were provided
    explicit GsbLookup(std::unique_ptr<RunTimeString> param)
             ^
../../src/operators/gsblookup.h:28:7: note: candidate constructor (the
implicit
      copy constructor) not viable: requires 1 argument, but 3 were provided
class GsbLookup : public Operator {
      ^
afl_fuzzer.cc:206:14: error: no matching constructor for initialization of
      'modsecurity::operators::Gt'
Gt *gt = new Gt("Gt", z, false); gt->evaluate(t, s); delete gt;
             ^  ~~~~~~~~~~~~~~
../../src/operators/gt.h:32:14: note: candidate constructor not viable:
requires
      single argument 'param', but 3 arguments were provided
    explicit Gt(std::unique_ptr<RunTimeString> param)
             ^
../../src/operators/gt.h:29:7: note: candidate constructor (the implicit
copy
      constructor) not viable: requires 1 argument, but 3 were provided
class Gt : public Operator {
      ^
afl_fuzzer.cc:207:32: error: no matching constructor for initialization of
      'modsecurity::operators::InspectFile'
InspectFile *inspectfile = new InspectFile("InspectFile", z, false); ins...
                               ^           ~~~~~~~~~~~~~~~~~~~~~~~
../../src/operators/inspect_file.h:33:14: note: candidate constructor not
      viable: requires single argument 'param', but 3 arguments were
provided
    explicit InspectFile(std::unique_ptr<RunTimeString> param)
             ^
../../src/operators/inspect_file.h:30:7: note: candidate constructor (the
      implicit copy constructor) not viable: requires 1 argument, but 3 were
      provided
class InspectFile : public Operator {
      ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
4 warnings and 20 errors generated.

Re: [Mod-security-developers] Fwd: Adding new operator

[Ervin H. <ai...@gm...>]

Hi Rufus,

On Sat, Apr 06, 2019 at 06:01:03PM +0200, Rufus Pwner wrote:
> Hi Ervin!
> 
> Thanks a lot!
> 
> ./configure --enable-parser-generation did the trick!

you're welcome :),

> Thanks to your support I was able to finish the work on the operator and
> created a PR:
> 
> https://github.com/SpiderLabs/ModSecurity/issues/2062
> https://github.com/SpiderLabs/ModSecurity/pull/2063
> https://github.com/SpiderLabs/secrules-language-tests/pull/5

note, that all Travis CI test had failed:

https://github.com/SpiderLabs/ModSecurity/pull/2063
"All checks have failed"

details:
https://travis-ci.org/SpiderLabs/ModSecurity/builds/516607693

looks like your regression test didn't pass, eg.:

https://travis-ci.org/SpiderLabs/ModSecurity/jobs/516607694

3012. (   0/  1/   1): test/test-cases/regression/operator-verifysvnr.json


Anyway, strictly my private opinion, but - despite of your work
is nice - this feature request wouldn't deserve an own new
operator (without any argument, I mean if somebody wants to use
an another algorithm, how can it use _this_?).

This "problem" what you solve as very elegant way, is tipically
can be solved with Lua - see the examples in libmodsecurity3
source dir.

But as I wrote, this is strictly IMHO. :)



Regards,


a.

Re: [Mod-security-developers] Fwd: Adding new operator

[Rufus P. <ruf...@gm...>]

Hi Ervin!

Thanks a lot!

./configure --enable-parser-generation did the trick!

Thanks to your support I was able to finish the work on the operator and
created a PR:

https://github.com/SpiderLabs/ModSecurity/issues/2062
https://github.com/SpiderLabs/ModSecurity/pull/2063
https://github.com/SpiderLabs/secrules-language-tests/pull/5

BR
Rufus


On Sun, Mar 31, 2019 at 8:57 PM Ervin Hegedüs <ai...@gm...> wrote:

> Hi Rufus,
>
> On Sun, Mar 31, 2019 at 03:12:58PM +0200, Rufus Pwner wrote:
> > Hello!
>
> [...]
>
> > "src/parser/seclang-scanner.cc" in the same way as verifyssn.
> >
> > I drew a new number 595 for the operator, incremented const unsigned
> > user_token_number_max_ by one
> > Added the new number 595 to yytoken_number_[]
> > Added a condition in src/parser/seclang-parser.yy
> >
> > Can you please assist me on what is missing or what I am doing wrong?
>
> how do you run the configure script? Could you show us your
> first few lines of config,log?
>
> $ head -n 8 config.log
> This file contains any messages produced by compilers while
> running configure, to aid debugging if configure makes a mistake.
>
> It was created by modsecurity configure 3.0, which was
> generated by GNU Autoconf 2.69.  Invocation command line was
>
>   $ ./configure --without-lmdb --enable-parser-generation
> --enable-mutex-on-pm --prefix=/usr
>
> may be you have to pass the "--enable-parser-generation" option.
>
>
> a.
>
>
>
> _______________________________________________
> mod-security-developers mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
> ModSecurity Services from Trustwave's SpiderLabs:
> https://www.trustwave.com/spiderLabs.php
>

Re: [Mod-security-developers] Fwd: Adding new operator

[Ervin H. <ai...@gm...>]

Hi Rufus,

On Sun, Mar 31, 2019 at 03:12:58PM +0200, Rufus Pwner wrote:
> Hello!

[...]
 
> "src/parser/seclang-scanner.cc" in the same way as verifyssn.
> 
> I drew a new number 595 for the operator, incremented const unsigned
> user_token_number_max_ by one
> Added the new number 595 to yytoken_number_[]
> Added a condition in src/parser/seclang-parser.yy
> 
> Can you please assist me on what is missing or what I am doing wrong?

how do you run the configure script? Could you show us your
first few lines of config,log?

$ head -n 8 config.log 
This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.

It was created by modsecurity configure 3.0, which was
generated by GNU Autoconf 2.69.  Invocation command line was

  $ ./configure --without-lmdb --enable-parser-generation --enable-mutex-on-pm --prefix=/usr

may be you have to pass the "--enable-parser-generation" option.


a.

[Mod-security-developers] Fwd: Adding new operator

[Rufus P. <ruf...@gm...>]

Hello!

I am trying to add a new operator "verifysvnr" to ModSecurity.

I have already written a unit-test for the operator which passes, however
the regression test fails, because ModSecurity does not detect
@verifysvnr as an operator:

[/] [4] (Rule: 1) Executing operator "Rx" with param "@verifysvnr \d{4}
?\d{6}" against ARGS.

compare with verifyssn:

[/] [4] (Rule: 1) Executing operator "VerifySSN" with param "\d{4} ?\d{6}"
against ARGS.


I added the new operator to  "src/parser/seclang-parser.hh",
"src/parser/seclang-parser.yy", "src/parser/seclang-scanner.ll" and
"src/parser/seclang-scanner.cc" in the same way as verifyssn.

I drew a new number 595 for the operator, incremented const unsigned
user_token_number_max_ by one
Added the new number 595 to yytoken_number_[]
Added a condition in src/parser/seclang-parser.yy

Can you please assist me on what is missing or what I am doing wrong?


For the actual work I have done see the commits on my fork of ModSecurity
on Github: https://github.com/Rufus125/ModSecurity/commits/v3/master

Thanks and have a nice day!
Rufus

Re: [Mod-security-developers] Finding triggered RuleIds

[Felipe C. <FC...@tr...>]

Hey Ervin,

I know you are working on fixies on CRS test bed and corelated stuff in order to make it compatible/consistent with v3, congrats a lot of work there. I know that you also have a patch to change v3 in order to make it match the expected data on the test beds. I will have a look at it again on the next week. Sorry on hurry here.

It may be confusing but I am: Felipe "Zimmerle" Costa or any permutation ( I am the guy that you are talking about ModSec on hangout sessions, the very same on GitHub and Slack and Email (

If I remember correctly, You and Victor Hora had being discussing on our Slack channel, among other things, about the semantic of block being or not a disruptive action, in fact it is a poser as a linker that may call a disruptive action -- not talking about the manual but how it is currently implemented. Notice also that proxy is not supported on v3. The reason why I am saying that, is the fact that we do have a lot of information on the example that you have posted. The amount of information could: (a) be wrongly implemented at any point; (b) be misinterpreted and lead us to false conclusions. My suggestion is to reduce the tests to a very little amount of rules/actions until it demonstrated without a doubt where the problem is. In fact, those could be implemented as v3 test cases. There is an error_log entry on the test case that was built for that.  Not sure if it is handling multiple entries, guess so. Put a break on regression.cc:90 it will tell you [sorry, not on a computer].
- https://github.com/SpiderLabs/ModSecurity/blob/b392a1ca36181a786a6d68b6eab57a8bb0bd558e/test/test-cases/regression/offset-variable.json#L54


You may found a bug; it demands further investigation. We have to narrow down where is the bug in order for us to fix. It could be even related to the v3 state [off, on, detection] that is only trigged in certain conditions. If you want we can do a hangout session to do it together, but I am not available before 1 April. I am with Victor on BlackHat Asia to present the new stuff on v3.1.

If there is a bug, it needs to be fixed in our side [and will]; Jai does not need to fix our bug on what he is coding (At least, I hope not.

Br.,
Felipe “Zimmerle” Costa
Security Researcher, Lead Developer ModSecurity.
 
Trustwave | SMART SECURITY ON DEMAND
www.trustwave.com <http://www.trustwave.com/>
 

On 3/23/19, 6:07 AM, "Ervin Hegedüs" <ai...@gm...> wrote:

    Hi Jai, Felipe,
    
    On Fri, Mar 22, 2019 at 10:05:29PM -0500, Jai Harpalani via mod-security-developers wrote:
    
    pre-note: I'm working on libmodsecurity3, fix bugs/different
    behaviours than ModSecurity2. The reference ruleset is owasp CRS
    3.1.
    
    To understand the libmodsecurity3 API, I've created a small (and
    really ugly :)) http server, which uses API calls.
    
    Of course, then I confronted with your logging issue.
     
    > In 3.0.2, I believe the callback was always invoked when a rule was
    > triggered. With 3.0.3, it is not invoked for a disruptive action.
    
    Disruptive actions are those that causes ModSecurity to do
    something. They are: block, deny, drop, allow, proxy and
    redirect.
    
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JPoZnsCL3Q&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fModSecurity%2fwiki%2fReference-Manual-%28v2%2ex%29%23secruleengine
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JKIeyMPbjw&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fModSecurity%2fwiki%2fReference-Manual-%28v2%2ex%29%23actions
    
    I've checked your statement with my ugly httpd with this request:
    
    GET /?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d HTTP/1.1
    Host: localhost
    
    and in the my log (error.log!) I got these rules:
    
    id "920280" 
    id "920320" 
    id "920273"
    id "941330"
    id "942110"
    id "942432"
    id "949110" 
    id "980130" 
    
    Let's see, which are disputive and which isn't:
    
    920280:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JPUSz82H2Q&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fREQUEST-920-PROTOCOL-ENFORCEMENT%2econf%23L593-L595
    non disruptiv
    
    920320:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JPtPyJTa3w&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fREQUEST-920-PROTOCOL-ENFORCEMENT%2econf%23L1328-L1330
    non disruptiv
    
    920273:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JPAfyM2I3A&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fREQUEST-920-PROTOCOL-ENFORCEMENT%2econf%23L1472-L1474
    disruptiv
    
    941330:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JPIdl8Pdiw&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fREQUEST-941-APPLICATION-ATTACK-XSS%2econf%23L876-L878
    disruptiv
    
    942110:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JPMSm8zc1g&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fREQUEST-942-APPLICATION-ATTACK-SQLI%2econf%23L505-L508
    disruptive
    
    942432:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JPESnsyH3Q&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fREQUEST-942-APPLICATION-ATTACK-SQLI%2econf%23L1580-L1582
    disruptive
    
    949110:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JKcbnczfjQ&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fREQUEST-949-BLOCKING-EVALUATION%2econf%23L81-L83
    disruptive
    
    980130:
    https://scanmail.trustwave.com/?c=4062&d=2_eV3PTMtLKF5f8G20tyCNrtg22G3Cm6JKISz5Ta3g&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fowasp-modsecurity-crs%2fblob%2f46171c0ef335f92b26787ce269e397c480286155%2frules%2fRESPONSE-980-CORRELATION%2econf%23L79-L81
    non-disputive
    
    so, looks like libmodsecurity3, version 3.0.3 logs mixed
    disruptive and non-disruptive triggered rules.
    
    
    > My
    > calling code was expecting the callback to always be invoked because that
    > is the only way we know that a rule has triggered.
    
    yes, you're right, this is a new feature request.
    
    > You state that for a
    > disruptive action, "the logging message is delivered via the disruptive
    > structure in text format". Where is this done?
    
    I'm not sure that understand you, but here is my snippet:
    
    ...
        modsec = msc_init();
        msc_set_log_cb(modsec, msc_logdata);
    ...
    
    upper...
    
    void msc_logdata(void *log, const void *data) {
        FILE *fp;
        char logline[4096];
    
        struct timeval tv;
    
        wrote_log_line(...); 
        fclose(fp);
    }
    
    and that's it. Not need to call this function explicitly, but I
    guess you know that - that's why I asked you above, what do you
    think about whit this question.
    
    > How is it delivered? How can
    > the calling code inspect this disruptive structure?
    
    there are some examples in examples/ subdirectory.
    
    > And, does the
    > disruptive structure contain all the details which were provided by the
    > callback (eg ip address, api, ruleId, rule tags, etc)?
    
    hmm... here is a logged line:
    
    1553327402.117947 ModSecurity: Warning. Matched "Operator `Ge' with parameter `5' against variable `TX:ANOMALY_SCORE' (Value: `21' ) [file "/usr/share/modsecurity-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "80"] [id "949110"] [rev ""] [msg "Inbound Anomaly Score Exceeded (Total Score: 21)"] [data ""] [severity "2"] [ver ""] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "127.0.0.1"] [uri "http://localhost/"] [unique_id "1553327401.929417"] [ref ""]
    
    ip address: [hostname "127.0.0.1"]
    api: what do you think about?
    ruleId: [id: "949110"]
    rule tags: [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"]
    etc: what do you think about it more?
    
    note, that I'm working with "SecRuleEngine DetectOnly" mode.
    
    > > There are three logs in v3: Debug, Msg/Error, Audit. I am talking about
    > > the “error log” one. It could be “displayed” as o consequence of the rules
    > > execution flow (log, auditlog action, etc) or consequence of a disruptive
    > > action. The first you retrieve via callback, the second via disruptive
    > > structure.
    > >
    > 
    > The "error log" one is the one I am talking about, as well.
    
    sure, in the debug log, I think I got the all triggered rule:
    
    grep Rule: modsec_debug.log  | wc -l
    541
    
    eg:
    [1553327401.929417] [http://localhost/?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d] [4] (Rule: 200000) Executing operator "Rx" with param "(?:application(?:/soap\+|/)|text/)xml" against REQUEST_HEADERS:Content-Type.
    [1553327401.929417] [http://localhost/?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d] [4] (Rule: 200001) Executing operator "Rx" with param "application/json" against REQUEST_HEADERS:Content-Type.
    [1553327401.929417] [http://localhost/?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d] [4] (Rule: 900000) Executing unconditional rule...
    ...
    
    > > Jai, from your initial email I had the feeling that you would like to have
    > > the id in a data structure at the disruptive message or having the logging
    > > callback to replace the logging data inside the disruptive structure. It
    > > will be a pleasure to make the library more flexible, but, can you give us
    > > a brief explanation of your requirements?
    > >
    > 
    > The requirement is for the calling code to be able to determine if a rule
    > was triggered, regardless of whether the rule was disruptive or not. And,
    > if triggered, the calling code should be provided details about the
    > trigger.
    
    for this I supposed we can use a new configuration variable,
    which can control that all triggered rule will logged or not.
     
    > I may have found a workaround for 3.0.3. Details below.
    
    I'll see that later, now I just wanted to reflect first part of
    your mail :).
    
    Hope this helped to clear some things.
    
    
    a.
     
    

Re: [Mod-security-developers] Finding triggered RuleIds

[Felipe C. <FC...@tr...>]

Hi Jai,

Currently, the disruptive data does not contain the details into separated structures, instead of in the format of the logging message. I propose to expand that to may (depending on a library configuration flag) contains the logging in the data structure.

In the released version this data structure is a “shared pointer” but it is suitable to be improved, and it is likely to be transformed on a && for version 3.1. Regardless, let me show you the pieces of code:
https://github.com/SpiderLabs/ModSecurity/blob/1ecd9713061c3534626bf6a5f59d1c928c0c52bb/headers/modsecurity/modsecurity.h#L236-L267

Notice the property RuleMessageLogProperty; it can be used to receive the rule message structure on the logging callback:
https://github.com/SpiderLabs/ModSecurity/blob/1ecd9713061c3534626bf6a5f59d1c928c0c52bb/headers/modsecurity/rule_message.h#L37-L117

For the disruptive actions we have:
https://github.com/SpiderLabs/ModSecurity/blob/v3/master/headers/modsecurity/intervention.h#L23-L29

The *log is the one that contains the information that you are looking for, but it is not respecting the configuration flag and always returning the log in the format of a text message. I want to change that also to support the callback. My advice is for you to call the intervention method to proceed with the evaluation of the rules. The intervention is the one who fulfills this structure among other things:

https://github.com/SpiderLabs/ModSecurity-nginx/blob/f1a7ab6e3c3e010dbbf2cfc37ae7f81fa410dc57/src/ngx_http_modsecurity_module.c#L132-L212

 

You may ask…

Why in this specific log in not on the callback? Well, the callback may be threated asynchronies by the consumer, leading to a possibility where the disruptive action takes place before the log the be processed/delivered, making it very hard the couple this event into a web server log, for instance.

Why not send it twice? The avoid spending resources in vain. After all, the disruptive is conditioned to the logging and vice-versa.

Change this behavior via a Rule, may not a good idea. It put the data to be shaped into memory in the format desired by the user, but, may not in the format expected by the consumer, leading to an invalid memory access; That most certainly will lead to crashes and instabilities.


I see two possible problems in your code snippet:

a) You are accessing the library raw data/internals. One of the beauties of the ModSecurity library (<3) is the fact that it contains two set of headers: public [https://github.com/SpiderLabs/ModSecurity/tree/v3/master/headers/modsecurity] and privates [all others].

The public ones are the ones that you should use. The others you should not. Due to a simple reason: we can change the internals without further notice, and that will break your implementation. You can achieve the same level of functionality by using the public headers. If not, ask, and the library will be expanded to support the missing feature (of course it has to make sense to other users as well).

Just to give you an example, that is what the nginx connector does to threat the disruptive actions and all the messages:
https://github.com/SpiderLabs/ModSecurity-nginx/blob/f1a7ab6e3c3e010dbbf2cfc37ae7f81fa410dc57/src/ngx_http_modsecurity_header_filter.c#L523-L528
(please do not use the Apache connector as a guide as it still under development. The nginx is fine.)

The version 3.1, where the first pieces of rules reload [reload the rules upon file update -- without restart] are already implemented, may cause you a little trouble. I still finishing the last touches: https://github.com/SpiderLabs/ModSecurity/tree/v3/dev/3.1-experimental


b) You may have a conceptual problem while handling an "escalation": A trigged rule does not mean that an action should be taken. You may find it workable in a given rule set (or rule set version), but no guarantee won't become a bug in a <near> future. Better to rely on the library for that. For instance: a rule can be trigged to set a variable.

In this example you can check how to read the RuleMessage without access the library internals:
https://github.com/SpiderLabs/ModSecurity/tree/v3/master/examples/reading_logs_via_rule_message

The question is: Having the *log in the ModSecurityIntervention_t in the format of a RuleMessage will solve your initial request? I guess so, but I am not sure if that fits your current implementation. Let's discuss; I don't know if you have other motivations to use the library in such fashion.


Br.,
Felipe “Zimmerle” Costa
Security Researcher, Lead Developer ModSecurity.

Trustwave | SMART SECURITY ON DEMAND
http://www.trustwave.com/


From: Jai Harpalani <jai...@mu...>
Date: Saturday, March 23, 2019 at 12:05 AM
To: Felipe Costa <FC...@tr...>
Cc: "mod...@li..." <mod...@li...>
Subject: Re: [Mod-security-developers] Finding triggered RuleIds

Responses interspersed below:

On Fri, Mar 22, 2019 at 8:34 PM Felipe Costa <mailto:FC...@tr...> wrote:
Just to make sure that we are talking about the same thing… when there Is a log to be shown the API is calling the callback. The only circumstance where the callback is not called is when the logging is a consequence of a disruptive action. In this last case the logging message is delivered via the disruptive structure in text format. So you guys want me to disable the logging for certain cases? If that is your request, there is a feature for that among the compilation flags to make ModSecurtiy mute.

In 3.0.2, I believe the callback was always invoked when a rule was triggered. With 3.0.3, it is not invoked for a disruptive action. My calling code was expecting the callback to always be invoked because that is the only way we know that a rule has triggered. You state that for a disruptive action, "the logging message is delivered via the disruptive structure in text format". Where is this done? How is it delivered? How can the calling code inspect this disruptive structure? And, does the disruptive structure contain all the details which were provided by the callback (eg ip address, api, ruleId, rule tags, etc)?
 
 
There are three logs in v3: Debug, Msg/Error, Audit. I am talking about the “error log” one. It could be “displayed” as o consequence of the rules execution flow (log, auditlog action, etc) or consequence of a disruptive action. The first you retrieve via callback, the second via disruptive structure.

The "error log" one is the one I am talking about, as well.
 
 
Jai, from your initial email I had the feeling that you would like to have the id in a data structure at the disruptive message or having the logging callback to replace the logging data inside the disruptive structure. It will be a pleasure to make the library more flexible, but, can you give us a brief explanation of your requirements?

The requirement is for the calling code to be able to determine if a rule was triggered, regardless of whether the rule was disruptive or not. And, if triggered, the calling code should be provided details about the trigger. 



I may have found a workaround for 3.0.3. Details below.

For 3.0.2, I have code such as the following inside of logCallback():

    const modsecurity::RuleMessage * lRuleMessage =
        reinterpret_cast<const modsecurity::RuleMessage *>(aRuleMessage);
    ruleId = lRuleMessage->m_ruleId;
    logMsg << "clientIp:" << m_clientIpAddress 
    << "clientPort:" << m_clientPort

    . . .

We have a class, ModSecTransaction, which inherits from modsecurity::Transaction. In 3.0.3, I have moved the above code into the ModSecTransaction dtor:

ModSecTransaction::~ModSecTransaction() noexcept
{
    for (modsecurity::RuleMessage& lRuleMessage : m_rulesMessages)
    {
      ruleId = lRuleMessage->m_ruleId;
      logMsg << . . .
      . . .
    }
    . . .
}

With 3.0.3, I also iterate through m_ruleMessages to determine if any rules have triggered after each call to processRequestHeaders(), processRequestBody().

bool
ModSecTransaction::shouldEscalateAnyTriggeredRule() noexcept
{
    bool lRetVal = false;

    // Iterate through all triggered rules
    for (modsecurity::RuleMessage& lRuleMessage : m_rulesMessages)
    {
        // Check if triggered rule should be escalated
        if (m_Origins.shouldEscalate(lRuleMessage.m_ruleId))
        {
            lRetVal = true;
            break;
        }
    }

    return lRetVal;
}

So far, this approach in 3.0.3 is working. Is it okay to use this approach in ModSec 3.0.3? Will it always work? Are there cases in which modsecurity::Transaction.m_rulesMessages will not be populated?

Thanks,
Jai

Re: [Mod-security-developers] Finding triggered RuleIds

[Ervin H. <ai...@gm...>]

Hi Jai, Felipe,

On Fri, Mar 22, 2019 at 10:05:29PM -0500, Jai Harpalani via mod-security-developers wrote:

pre-note: I'm working on libmodsecurity3, fix bugs/different
behaviours than ModSecurity2. The reference ruleset is owasp CRS
3.1.

To understand the libmodsecurity3 API, I've created a small (and
really ugly :)) http server, which uses API calls.

Of course, then I confronted with your logging issue.
 
> In 3.0.2, I believe the callback was always invoked when a rule was
> triggered. With 3.0.3, it is not invoked for a disruptive action.

Disruptive actions are those that causes ModSecurity to do
something. They are: block, deny, drop, allow, proxy and
redirect.

https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#secruleengine
https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#actions

I've checked your statement with my ugly httpd with this request:

GET /?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d HTTP/1.1
Host: localhost

and in the my log (error.log!) I got these rules:

id "920280" 
id "920320" 
id "920273"
id "941330"
id "942110"
id "942432"
id "949110" 
id "980130" 

Let's see, which are disputive and which isn't:

920280:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf#L593-L595
non disruptiv

920320:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf#L1328-L1330
non disruptiv

920273:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf#L1472-L1474
disruptiv

941330:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf#L876-L878
disruptiv

942110:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf#L505-L508
disruptive

942432:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf#L1580-L1582
disruptive

949110:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/REQUEST-949-BLOCKING-EVALUATION.conf#L81-L83
disruptive

980130:
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/46171c0ef335f92b26787ce269e397c480286155/rules/RESPONSE-980-CORRELATION.conf#L79-L81
non-disputive

so, looks like libmodsecurity3, version 3.0.3 logs mixed
disruptive and non-disruptive triggered rules.


> My
> calling code was expecting the callback to always be invoked because that
> is the only way we know that a rule has triggered.

yes, you're right, this is a new feature request.

> You state that for a
> disruptive action, "the logging message is delivered via the disruptive
> structure in text format". Where is this done?

I'm not sure that understand you, but here is my snippet:

...
    modsec = msc_init();
    msc_set_log_cb(modsec, msc_logdata);
...

upper...

void msc_logdata(void *log, const void *data) {
    FILE *fp;
    char logline[4096];

    struct timeval tv;

    wrote_log_line(...); 
    fclose(fp);
}

and that's it. Not need to call this function explicitly, but I
guess you know that - that's why I asked you above, what do you
think about whit this question.

> How is it delivered? How can
> the calling code inspect this disruptive structure?

there are some examples in examples/ subdirectory.

> And, does the
> disruptive structure contain all the details which were provided by the
> callback (eg ip address, api, ruleId, rule tags, etc)?

hmm... here is a logged line:

1553327402.117947 ModSecurity: Warning. Matched "Operator `Ge' with parameter `5' against variable `TX:ANOMALY_SCORE' (Value: `21' ) [file "/usr/share/modsecurity-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "80"] [id "949110"] [rev ""] [msg "Inbound Anomaly Score Exceeded (Total Score: 21)"] [data ""] [severity "2"] [ver ""] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "127.0.0.1"] [uri "http://localhost/"] [unique_id "1553327401.929417"] [ref ""]

ip address: [hostname "127.0.0.1"]
api: what do you think about?
ruleId: [id: "949110"]
rule tags: [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"]
etc: what do you think about it more?

note, that I'm working with "SecRuleEngine DetectOnly" mode.

> > There are three logs in v3: Debug, Msg/Error, Audit. I am talking about
> > the “error log” one. It could be “displayed” as o consequence of the rules
> > execution flow (log, auditlog action, etc) or consequence of a disruptive
> > action. The first you retrieve via callback, the second via disruptive
> > structure.
> >
> 
> The "error log" one is the one I am talking about, as well.

sure, in the debug log, I think I got the all triggered rule:

grep Rule: modsec_debug.log  | wc -l
541

eg:
[1553327401.929417] [http://localhost/?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d] [4] (Rule: 200000) Executing operator "Rx" with param "(?:application(?:/soap\+|/)|text/)xml" against REQUEST_HEADERS:Content-Type.
[1553327401.929417] [http://localhost/?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d] [4] (Rule: 200001) Executing operator "Rx" with param "application/json" against REQUEST_HEADERS:Content-Type.
[1553327401.929417] [http://localhost/?var=%22in%20\\u0076\\u0061l\\u0075e\\u004F\\u0066%3d] [4] (Rule: 900000) Executing unconditional rule...
...

> > Jai, from your initial email I had the feeling that you would like to have
> > the id in a data structure at the disruptive message or having the logging
> > callback to replace the logging data inside the disruptive structure. It
> > will be a pleasure to make the library more flexible, but, can you give us
> > a brief explanation of your requirements?
> >
> 
> The requirement is for the calling code to be able to determine if a rule
> was triggered, regardless of whether the rule was disruptive or not. And,
> if triggered, the calling code should be provided details about the
> trigger.

for this I supposed we can use a new configuration variable,
which can control that all triggered rule will logged or not.
 
> I may have found a workaround for 3.0.3. Details below.

I'll see that later, now I just wanted to reflect first part of
your mail :).

Hope this helped to clear some things.


a.
 

Re: [Mod-security-developers] Finding triggered RuleIds

[Jai H. <jai...@mu...>]

Responses interspersed below:

On Fri, Mar 22, 2019 at 8:34 PM Felipe Costa <FC...@tr...> wrote:

> Just to make sure that we are talking about the same thing… when there Is
> a log to be shown the API is calling the callback. The only circumstance
> where the callback is not called is when the logging is a consequence of a
> disruptive action. In this last case the logging message is delivered via
> the disruptive structure in text format. So you guys want me to disable the
> logging for certain cases? If that is your request, there is a feature for
> that among the compilation flags to make ModSecurtiy mute.
>

In 3.0.2, I believe the callback was always invoked when a rule was
triggered. With 3.0.3, it is not invoked for a disruptive action. My
calling code was expecting the callback to always be invoked because that
is the only way we know that a rule has triggered. You state that for a
disruptive action, "the logging message is delivered via the disruptive
structure in text format". Where is this done? How is it delivered? How can
the calling code inspect this disruptive structure? And, does the
disruptive structure contain all the details which were provided by the
callback (eg ip address, api, ruleId, rule tags, etc)?


>
>
> There are three logs in v3: Debug, Msg/Error, Audit. I am talking about
> the “error log” one. It could be “displayed” as o consequence of the rules
> execution flow (log, auditlog action, etc) or consequence of a disruptive
> action. The first you retrieve via callback, the second via disruptive
> structure.
>

The "error log" one is the one I am talking about, as well.


>
>
> Jai, from your initial email I had the feeling that you would like to have
> the id in a data structure at the disruptive message or having the logging
> callback to replace the logging data inside the disruptive structure. It
> will be a pleasure to make the library more flexible, but, can you give us
> a brief explanation of your requirements?
>

The requirement is for the calling code to be able to determine if a rule
was triggered, regardless of whether the rule was disruptive or not. And,
if triggered, the calling code should be provided details about the
trigger.



I may have found a workaround for 3.0.3. Details below.

For 3.0.2, I have code such as the following inside of logCallback():

   * const modsecurity::RuleMessage * lRuleMessage =*
*        reinterpret_cast<const modsecurity::RuleMessage *>(aRuleMessage);*
*    ruleId = lRuleMessage->m_ruleId;*
*    logMsg << "clientIp:" << m_clientIpAddress *
*    << "clientPort:" << m_clientPort*

    . . .

We have a class, ModSecTransaction, which inherits from
modsecurity::Transaction. In 3.0.3, I have moved the above code into the
ModSecTransaction dtor:

*ModSecTransaction::~ModSecTransaction() noexcept*
*{*
*    for (modsecurity::RuleMessage& lRuleMessage : m_rulesMessages)*
*    {*
*      ruleId = lRuleMessage->m_ruleId;*
*      logMsg << . . .*
*      . . .*
*    }*
*    . . .*
*}*

With 3.0.3, I also iterate through m_ruleMessages to determine if any rules
have triggered after each call to processRequestHeaders(),
processRequestBody().

*bool*
*ModSecTransaction::shouldEscalateAnyTriggeredRule() noexcept*
*{*
*    bool lRetVal = false;*

*    // Iterate through all triggered rules*
*    for (modsecurity::RuleMessage& lRuleMessage : m_rulesMessages)*
*    {*
*        // Check if triggered rule should be escalated*
*        if (m_Origins.shouldEscalate(lRuleMessage.m_ruleId))*
*        {*
*            lRetVal = true;*
*            break;*
*        }*
*    }*

*    return lRetVal;*
*}*

So far, this approach in 3.0.3 is working. *Is it okay to use this approach
in ModSec 3.0.3? Will it always work? Are there cases in which
modsecurity::Transaction.m_rulesMessages will not be populated?*

Thanks,
Jai

>

Re: [Mod-security-developers] Finding triggered RuleIds

[Felipe C. <FC...@tr...>]

Just to make sure that we are talking about the same thing… when there Is a log to be shown the API is calling the callback. The only circumstance where the callback is not called is when the logging is a consequence of a disruptive action. In this last case the logging message is delivered via the disruptive structure in text format. So you guys want me to disable the logging for certain cases? If that is your request, there is a feature for that among the compilation flags to make ModSecurtiy mute.

There are three logs in v3: Debug, Msg/Error, Audit. I am talking about the “error log” one. It could be “displayed” as o consequence of the rules execution flow (log, auditlog action, etc) or consequence of a disruptive action. The first you retrieve via callback, the second via disruptive structure.

Jai, from your initial email I had the feeling that you would like to have the id in a data structure at the disruptive message or having the logging callback to replace the logging data inside the disruptive structure. It will be a pleasure to make the library more flexible, but, can you give us a brief explanation of your requirements?

Br.,
Felipe “Zimmerle” Costa
Security Researcher, Lead Developer ModSecurity.

Trustwave | SMART SECURITY ON DEMAND
www.trustwave.com<http://www.trustwave.com/>


From: Jai Harpalani via mod-security-developers <mod...@li...>
Reply-To: "mod...@li..." <mod...@li...>
Date: Friday, March 22, 2019 at 5:16 PM
To: "mod-security-d." <mod...@li...>
Cc: Jai Harpalani <jai...@mu...>
Subject: Re: [Mod-security-developers] Finding triggered RuleIds

A 'SecLogAllRule' such as Ervin suggests would work well. Ideally, the log message would contain all of the information currently contained in the ruleMessage.

I may have found a workaround for this. Will send another email with the details.

On Fri, Mar 22, 2019 at 3:10 PM Ervin Hegedüs <ai...@gm...<mailto:ai...@gm...>> wrote:
hi,

On Fri, Mar 22, 2019 at 12:49:18PM +0000, Felipe Costa wrote:
> Hi Jai,
>
> For the current public supported connectors, the rule id altogether with logging text is enough. There is no data structure except for char pointer that point towards the logging string [for the logging attached to the disruptive events]. Is my understanding that it may be useful for your application,  to have an specific field that held to rule id (may other information regarding the rule as well). Having that in mind, we can change the API the make it more useful to your application. Sorry for the inconvenience. Lets discuss out-of-band the specific characteristics of your use case, so we can make the API suits you better.

I can imagine that there should be a new configuration directive,
which allows to log every triggered rule, not just when it
intervents.

The default value should be disable to do this, but if the
end-user wants to see that, then it can be use that.

eg.

SecLogAllRule 0|1



a.



_______________________________________________
mod-security-developers mailing list
mod...@li...<mailto:mod...@li...>
https://lists.sourceforge.net/lists/listinfo/mod-security-developers<https://scanmail.trustwave.com/?c=4062&d=qsKV3OL797AuHt0YIpZtf8wkheEsZv18XuZPrY4e8A&s=5&u=https%3a%2f%2flists%2esourceforge%2enet%2flists%2flistinfo%2fmod-security-developers>
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/spiderLabs.php

Re: [Mod-security-developers] Finding triggered RuleIds

[Jai H. <jai...@mu...>]

A 'SecLogAllRule' such as Ervin suggests would work well. Ideally, the log
message would contain all of the information currently contained in the
ruleMessage.

I may have found a workaround for this. Will send another email with the
details.

On Fri, Mar 22, 2019 at 3:10 PM Ervin Hegedüs <ai...@gm...> wrote:

> hi,
>
> On Fri, Mar 22, 2019 at 12:49:18PM +0000, Felipe Costa wrote:
> > Hi Jai,
> >
> > For the current public supported connectors, the rule id altogether with
> logging text is enough. There is no data structure except for char pointer
> that point towards the logging string [for the logging attached to the
> disruptive events]. Is my understanding that it may be useful for your
> application,  to have an specific field that held to rule id (may other
> information regarding the rule as well). Having that in mind, we can change
> the API the make it more useful to your application. Sorry for the
> inconvenience. Lets discuss out-of-band the specific characteristics of
> your use case, so we can make the API suits you better.
>
> I can imagine that there should be a new configuration directive,
> which allows to log every triggered rule, not just when it
> intervents.
>
> The default value should be disable to do this, but if the
> end-user wants to see that, then it can be use that.
>
> eg.
>
> SecLogAllRule 0|1
>
>
>
> a.
>
>
>
> _______________________________________________
> mod-security-developers mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
> ModSecurity Services from Trustwave's SpiderLabs:
> https://www.trustwave.com/spiderLabs.php
>

Re: [Mod-security-developers] Finding triggered RuleIds

[Ervin H. <ai...@gm...>]

hi,

On Fri, Mar 22, 2019 at 12:49:18PM +0000, Felipe Costa wrote:
> Hi Jai,
> 
> For the current public supported connectors, the rule id altogether with logging text is enough. There is no data structure except for char pointer that point towards the logging string [for the logging attached to the disruptive events]. Is my understanding that it may be useful for your application,  to have an specific field that held to rule id (may other information regarding the rule as well). Having that in mind, we can change the API the make it more useful to your application. Sorry for the inconvenience. Lets discuss out-of-band the specific characteristics of your use case, so we can make the API suits you better.

I can imagine that there should be a new configuration directive,
which allows to log every triggered rule, not just when it
intervents.

The default value should be disable to do this, but if the
end-user wants to see that, then it can be use that.

eg.

SecLogAllRule 0|1



a.