mod-security-users — General discussion about mod_security

[mod-security-users] string compare

[<877...@qq...>]

hi all, 


Who can help me understand why the highlighted is not  toupper(aa) == toupper(bb)  or tolower(aa)==tolower(bb)  ?


in file ModSecurity/src/variables/variable.h


class VariableMonkeyResolution {
 public:
    VariableMonkeyResolution () { } 
    static inline bool comp(const std::string &a, const std::string &b) {
        return a.size() == b.size()
             && std::equal(a.begin(), a.end(), b.begin(),
            [](char aa, char bb) {
            return toupper(aa) == bb; 
        }); 
    }   



thanks
huiming

Re: [mod-security-users] Standard testing methodology for virtual patching

[Christian F. <chr...@ne...>]

Hey Kyle,

This sounds like an intriguing project. A research project, I presume?

There is a problem I see where you trust your security scanner to discover
every vulnerability. You effective limit CRS to the depth of the security
scanner. But that's probably not what you are interested in as feedback.

I get the impression that security scanners have a small intersection with
their findings beyond the most obvious ones. So when you discover something
with scanner A, patch it virtually and then scanner B is no longer able to
discover it, that can give you a false sense of security. You absolutely
need to run scanner A again too.

It might make sense to give scanner A the original vulnerability and have it
try to fuzz more weaknesses in that area after your virtual patch.

Other than that, I agree there is little conceptual work in this regard and it
feels like you are stepping on new ground. Which can be exciting for a piece
of research. :)

Good luck!

Christian


On Thu, Jul 01, 2021 at 04:42:28PM +0000, Kyle Richard Orlando wrote:
> Hi,
> 
> Does anyone have any tips of recommendations for a standard way of testing/evaluating the effectiveness of a set of virtual patches? I've written a little python script that conditionally includes CRS rules for a given location and parameter(s) based on a vulnerability report generated by scanners like ZAP. Really, I can't think of much more beyond a before/after active scan with ZAP, and then a before/after scan with another tool that wasn't used in virtual patch creation. I know these virtual patches reduce the rate of false positives compared to just setting up CRS out-of-the-box, since they won't block requests for locations/parameters that aren't associated with some vulnerability. However, I can't think of a particularly useful way of showing this beyond picking a random vulnerable location and parameter and "attacking" it with words from a dictionary or book.
> 
> I've been to a couple of OWASP pages on the topic of virtual patching, but the testing methodology seems to be fairly manual and ad-hoc.
> 
> Thanks,
> Kyle


> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

[mod-security-users] Standard testing methodology for virtual patching

[Kyle R. O. <ky...@st...>]

Hi,

Does anyone have any tips of recommendations for a standard way of testing/evaluating the effectiveness of a set of virtual patches? I've written a little python script that conditionally includes CRS rules for a given location and parameter(s) based on a vulnerability report generated by scanners like ZAP. Really, I can't think of much more beyond a before/after active scan with ZAP, and then a before/after scan with another tool that wasn't used in virtual patch creation. I know these virtual patches reduce the rate of false positives compared to just setting up CRS out-of-the-box, since they won't block requests for locations/parameters that aren't associated with some vulnerability. However, I can't think of a particularly useful way of showing this beyond picking a random vulnerable location and parameter and "attacking" it with words from a dictionary or book.

I've been to a couple of OWASP pages on the topic of virtual patching, but the testing methodology seems to be fairly manual and ad-hoc.

Thanks,
Kyle

[mod-security-users] CVE-2021-35368: OWASP ModSecurity Core Rule Set security release

[Christian F. <chr...@ne...>]

Dear all,

The OWASP ModSecurity Core Rule Set team has published the following releases:

* v3.3.2 (supported)
* v3.2.1 (supported)
* v3.1.2 (EOL)

All these releases are meant to fight CVE-2021-35368, a CRS request body
bypass vulnerability.

Details about the vulnerability as well as links to release files and
changelog can be found here:

https://coreruleset.org/20210630/cve-2021-35368-crs-request-body-bypass/

Please note that this is a CRS problem and has nothing to do with the engine
ModSecurity. The changeset is minimal, so an update should be smooth.

Best regards,

Christian Folini, CRS Co-Lead

-- 
Had I been present at the creation, I would have given some useful
hints for the better ordering of the universe.
-- Alfonso the Wise, 1221 - 1284

Re: [mod-security-users] about ip/IP

[<877...@qq...>]

I found it , ip is initialized in configation file REQUEST-901-INITIALIZATION.conf file




------------------ Original ------------------
From:                                                                                                                        "mod-security-users"                                                                                    <mod...@li...&gt;;
Date:&nbsp;Wed, Jun 30, 2021 10:33 AM
To:&nbsp;"mod-security-users"<mod...@li...&gt;;
Cc:&nbsp;"huiming"<877...@qq...&gt;;
Subject:&nbsp;Re: [mod-security-users] about ip/IP



but I DO NOT find any document that&nbsp; ip (low case) will be translated to brower ip.




------------------ Original ------------------
From:                                                                                                                        "mod-security-users"                                                                                    <mod...@li...&gt;;
Date:&nbsp;Wed, Jun 30, 2021 10:17 AM
To:&nbsp;"mod-security-users"<mod...@li...&gt;;
Cc:&nbsp;"huiming"<877...@qq...&gt;;
Subject:&nbsp;[mod-security-users] about ip/IP



hi, all:


For below rule, two high lighted IP/ip will be translated to client IP address ? only in this case, it is reasonable.


SecRule IP:DOS_BURST_COUNTER "@ge 1" \
&nbsp; &nbsp; "id:912171,\
&nbsp; &nbsp; phase:5,\
&nbsp; &nbsp; pass,\
&nbsp; &nbsp; t:none,\
&nbsp; &nbsp; log,\
&nbsp; &nbsp; msg:'Potential Denial of Service (DoS) Attack from %{tx.real_ip} - # of Request Bursts: %{ip.dos_burst_counter}',\
&nbsp; &nbsp; tag:'application-multi',\
&nbsp; &nbsp; tag:'language-multi',\
&nbsp; &nbsp; tag:'platform-multi',\
&nbsp; &nbsp; tag:'attack-dos',\
&nbsp; &nbsp; tag:'paranoia-level/2',\
&nbsp; &nbsp; ver:'OWASP_CRS/3.2.0',\
&nbsp; &nbsp; setvar:'ip.dos_block=1',\
&nbsp; &nbsp; expirevar:'ip.dos_block=%{tx.dos_block_timeout}'"





Thanks
huiming

Re: [mod-security-users] about ip/IP

[<877...@qq...>]

but I DO NOT find any document that  ip (low case) will be translated to brower ip.




------------------ Original ------------------
From:                                                                                                                        "mod-security-users"                                                                                    <mod...@li...&gt;;
Date:&nbsp;Wed, Jun 30, 2021 10:17 AM
To:&nbsp;"mod-security-users"<mod...@li...&gt;;
Cc:&nbsp;"huiming"<877...@qq...&gt;;
Subject:&nbsp;[mod-security-users] about ip/IP



hi, all:


For below rule, two high lighted IP/ip will be translated to client IP address ? only in this case, it is reasonable.


SecRule IP:DOS_BURST_COUNTER "@ge 1" \
&nbsp; &nbsp; "id:912171,\
&nbsp; &nbsp; phase:5,\
&nbsp; &nbsp; pass,\
&nbsp; &nbsp; t:none,\
&nbsp; &nbsp; log,\
&nbsp; &nbsp; msg:'Potential Denial of Service (DoS) Attack from %{tx.real_ip} - # of Request Bursts: %{ip.dos_burst_counter}',\
&nbsp; &nbsp; tag:'application-multi',\
&nbsp; &nbsp; tag:'language-multi',\
&nbsp; &nbsp; tag:'platform-multi',\
&nbsp; &nbsp; tag:'attack-dos',\
&nbsp; &nbsp; tag:'paranoia-level/2',\
&nbsp; &nbsp; ver:'OWASP_CRS/3.2.0',\
&nbsp; &nbsp; setvar:'ip.dos_block=1',\
&nbsp; &nbsp; expirevar:'ip.dos_block=%{tx.dos_block_timeout}'"





Thanks
huiming

[mod-security-users] about ip/IP

[<877...@qq...>]

hi, all:


For below rule, two high lighted IP/ip will be translated to client IP address ? only in this case, it is reasonable.


SecRule IP:DOS_BURST_COUNTER "@ge 1" \
    "id:912171,\
    phase:5,\
    pass,\
    t:none,\
    log,\
    msg:'Potential Denial of Service (DoS) Attack from %{tx.real_ip} - # of Request Bursts: %{ip.dos_burst_counter}',\
    tag:'application-multi',\
    tag:'language-multi',\
    tag:'platform-multi',\
    tag:'attack-dos',\
    tag:'paranoia-level/2',\
    ver:'OWASP_CRS/3.2.0',\
    setvar:'ip.dos_block=1',\
    expirevar:'ip.dos_block=%{tx.dos_block_timeout}'"





Thanks
huiming

[mod-security-users] Upcoming OWASP ModSecurity Core Rule Set Security Releases

[Christian F. <chr...@ne...>]

Dear all,

A security problem with the OWASP ModSecurity Core Rule Set has been brought
to our attention recently. The CRS team is now working on a fix that will be
released on Wednesday as 3.1.2, 3.2.1 and 3.3.2.

More infos at 

https://coreruleset.org/20210628/upcoming-crs-security-releases/

Best regards,

Christian Folini, CRS Co-Lead


-- 
The intersection of all majorities is the empty set - The 
union of even the smallest minorities is the universal set.
--- Linus Thorvalds

[mod-security-users] ModSecurity version 2.9.4

[Felipe Z. <fe...@zi...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hi,

It is a pleasure to announce ModSecurity version 2.9.4. Release 2.9.4
contains fixes and enhancements atop version 2.9.3.

The details on this release are available on GitHub:
 - https://github.com/SpiderLabs/ModSecurity/releases/tag/v2.9.4

Further details on ModSecurity v2 can be found on the project README:
 - https://github.com/SpiderLabs/ModSecurity/tree/v2/master

The list of open issues is available on GitHub:
 - https://github.com/SpiderLabs/ModSecurity/labels/2.x

IMPORTANT:
- - Windows installer no longer includes OWASP CRS.
- - Release files are no longer available on modsecurity.org, only on GitHub.
- - ModSecurity version 2 will be available and maintained parallel
to version 3. There is no ETA to deprecate version 2.x. ModSecurity versions
2 and 3 have a completely independent development/release cycle.

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iFwEARECAB0WIQQZDvrMoen6RmqOzZzm37CM6LESdwUCYNEVfgAKCRDm37CM6LES
dxUkAJ4naJ9ysl5HZPSEhmO0wxrLduDI4wCYzRhp8EuuSp/TW5uVNdF+eDl8UA==
=yA1i
-----END PGP SIGNATURE-----

[mod-security-users] ModSecurity version 2.9.4

[Felipe Z. <fe...@zi...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hi,

It is a pleasure to announce ModSecurity version 2.9.4. Release 2.9.4
contains fixes and enhancements atop version 2.9.3.

The details on this release are available on GitHub:
 - https://github.com/SpiderLabs/ModSecurity/releases/tag/v2.9.4

Further details on ModSecurity v2 can be found on the project README:
 - https://github.com/SpiderLabs/ModSecurity/tree/v2/master

The list of open issues is available on GitHub:
 - https://github.com/SpiderLabs/ModSecurity/labels/2.x

IMPORTANT:
- - Windows installer no longer includes OWASP CRS.
- - Release files are no longer available on modsecurity.org, only on GitHub.
- - ModSecurity version 2 will be available and maintained parallel
to version 3. There is no ETA to deprecate version 2.x. ModSecurity versions
2 and 3 have a completely independent development/release cycle.

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iFwEARECAB0WIQQZDvrMoen6RmqOzZzm37CM6LESdwUCYNEVfgAKCRDm37CM6LES
dxUkAJ4naJ9ysl5HZPSEhmO0wxrLduDI4wCYzRhp8EuuSp/TW5uVNdF+eDl8UA==
=yA1i
-----END PGP SIGNATURE-----

Re: [mod-security-users] Version 2.9.4 released?

[Felipe C. <FC...@tr...>]

There is a release that went out few seconds ago. I am not sure about this typo. Do you happens to have the commit hash?

Br.,
Felipe “Zimmerle” Costa
Security Researcher, Lead Developer ModSecurity.

Trustwave | SMART SECURITY ON DEMAND
www.trustwave.com<http://www.trustwave.com/>


From: Rainer Jung <rai...@ki...>
Date: Friday, June 18, 2021 at 7:52 AM
To: mod...@li... <mod...@li...>
Subject: [mod-security-users] Version 2.9.4 released?
There's a 2.9.4 tag in GitHub (a few days old), but CHANGES says July 14
2021 as release date.

So is that a typo in CHANGES and it was already released (what date
please?), or is the tag just a release candidate or similar and you
expect to release on July 14th? I couldn't find a release announcement.

See also:

https://scanmail.trustwave.com/?c=4062&d=2vrM4GqgRrW32aJ1JP2YasC_YlWI9mrHUQIsaO-uiA&s=5&u=https%3a%2f%2fgithub%2ecom%2fSpiderLabs%2fModSecurity%2fissues%2f2579

Tanks and regards,

Rainer


_______________________________________________
mod-security-users mailing list
mod...@li...
https://scanmail.trustwave.com/?c=4062&d=2vrM4GqgRrW32aJ1JP2YasC_YlWI9mrHUVd6buis3A&s=5&u=https%3a%2f%2flists%2esourceforge%2enet%2flists%2flistinfo%2fmod-security-users
Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
http://scanmail.trustwave.com/?c=4062&d=2vrM4GqgRrW32aJ1JP2YasC_YlWI9mrHUQUoOrH42A&s=5&u=http%3a%2f%2fwww%2emodsecurity%2eorg%2fprojects%2fcommercial%2frules%2f
http://scanmail.trustwave.com/?c=4062&d=2vrM4GqgRrW32aJ1JP2YasC_YlWI9mrHUQQsaL2sjQ&s=5&u=http%3a%2f%2fwww%2emodsecurity%2eorg%2fprojects%2fcommercial%2fsupport%2f
This transmission may contain information that is privileged, confidential, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format.

Re: [mod-security-users] reference link issue

[Christian F. <chr...@ne...>]

Because nobody has written it so far.

See https://github.com/SpiderLabs/ModSecurity/issues/2563

For 90% of the functionality, you can go and read the v2.x manual. For the
remaining 10% - which are hard to identify - you are a bit left on your own
devices.

Best,

Christian

On Mon, Jun 21, 2021 at 07:15:45PM +0800, huiming via mod-security-users wrote:
> hi, 
> 
> 
> Why link below is empty？
> https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v3.x)
> 
> 
> 
> 
> but 
> 
> 
> https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x) 
> 
> 
> NOT
> 
> 
> huiming


> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

[mod-security-users] reference link issue

[<877...@qq...>]

hi, 


Why link below is empty？
https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v3.x)




but 


https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x) 


NOT


huiming

[mod-security-users] Version 2.9.4 released?

[Rainer J. <rai...@ki...>]

There's a 2.9.4 tag in GitHub (a few days old), but CHANGES says July 14 
2021 as release date.

So is that a typo in CHANGES and it was already released (what date 
please?), or is the tag just a release candidate or similar and you 
expect to release on July 14th? I couldn't find a release announcement.

See also:

https://github.com/SpiderLabs/ModSecurity/issues/2579

Tanks and regards,

Rainer

[mod-security-users] 回复： Is it Possible to apply different ruleset for different url.

[<877...@qq...>]

Thanks, seems OK for my usage scenarios， because I have a dispatch nginx which is responsible receiving packet from internet and transferring packet to nginx+modsecurity server.








------------------ 原始邮件 ------------------
发件人:                                                                                                                        "mod-security-users"                                                                                    <ehs...@gm...&gt;;
发送时间:&nbsp;2021年6月18日(星期五) 下午2:32
收件人:&nbsp;"mod-security-users"<mod...@li...&gt;;

主题:&nbsp;Re: [mod-security-users] Is it Possible to apply different ruleset for different url.



Hi

Yes, u can serve different sites via distinct "server " directive in the same Nginx instance, each containing a custom modsecurity rule set.


On Fri, Jun 18, 2021 at 6:41 AM 成会明 via mod-security-users <mod...@li...&gt; wrote:

hi all,
&nbsp; &nbsp; &nbsp;Is it Possible to apply different ruleset for different domain name with one nginx+modsecurity entity? that is, to support different protect for different web sit.&nbsp;


Thanks in advance.


&nbsp;
_______________________________________________
 mod-security-users mailing list
 mod...@li...
 https://lists.sourceforge.net/lists/listinfo/mod-security-users
 Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
 http://www.modsecurity.org/projects/commercial/rules/
 http://www.modsecurity.org/projects/commercial/support/
 



-- 
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; regards
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Ehsan.Mahdavi
PhD candidated for&nbsp;Computer Engineering
&nbsp; &nbsp; by Isfahan University of Technology
&nbsp; &nbsp; &nbsp; &nbsp; http://emahdavi.ece.iut.ac.ir/

Re: [mod-security-users] Is it Possible to apply different ruleset for different url.

[Ehsan M. <ehs...@gm...>]

Hi

Yes, u can serve different sites via distinct "server " directive in the
same Nginx instance, each containing a custom modsecurity rule set.

On Fri, Jun 18, 2021 at 6:41 AM 成会明 via mod-security-users <
mod...@li...> wrote:

> hi all,
>      Is it Possible to apply different ruleset for different domain name
> with one nginx+modsecurity entity? that is, to support different protect
> for different web sit.
>
> Thanks in advance.
>
>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>


-- 
                    regards
                 Ehsan.Mahdavi
PhD candidated for Computer Engineering
    by Isfahan University of Technology
        http://emahdavi.ece.iut.ac.ir/

[mod-security-users] Is it Possible to apply different ruleset for different url.

[<877...@qq...>]

hi all,
     Is it Possible to apply different ruleset for different domain name with one nginx+modsecurity entity? that is, to support different protect for different web sit. 


Thanks in advance.


 

Re: [mod-security-users] Problem with pipe and @pmFromFile

[<az...@po...>]

So, when using @pmFromFile, at most 1 pipe per pattern can be matched?




Citát Marc Stern <mar...@ap...>:

> Hi Azur,
>
> With the @pm operator, the pipe character is used to encode bytes in  
> hexa, like |090a|.
> When using only 1 pipe, it's not seen as this encoding character,  
> but with 2 well.
>
> You would need to encode the pipe to have it "escaped": aaa|7c|bbb|7c|ccc
> That's the theory because the implementation is buggy and this doesn't work.
> The only solution is to use a regex: "@rx aaa[|]bbb[|]ccc"
> ... but you have to add another rule on top of your @pmFromFile one :-(
>
> On 03-06-2021 09:25, az...@po... wrote:
>> Hi,
>>
>> i'm having problems with matching strings containing multiple pipe  
>> characters using @pmFromFile. For example, i'm able to match this  
>> pattern (with only 1 pipe):
>> aaa|bbb
>>
>>
>> But i'm unable to match this pattern (with multiple pipes):
>> aaa|bbb|ccc
>>
>> Any hints what's wrong? Thanks. modsecurity 2.9.3, Apache 2.4.
>>
>> azur
>>
>>
>>
>>
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
>
>
>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] Problem with pipe and @pmFromFile

[Marc S. <mar...@ap...>]

Hi Azur,

With the @pm operator, the pipe character is used to encode bytes in 
hexa, like |090a|.
When using only 1 pipe, it's not seen as this encoding character, but 
with 2 well.

You would need to encode the pipe to have it "escaped": aaa|7c|bbb|7c|ccc
That's the theory because the implementation is buggy and this doesn't work.
The only solution is to use a regex: "@rx aaa[|]bbb[|]ccc"
... but you have to add another rule on top of your @pmFromFile one :-(

On 03-06-2021 09:25, az...@po... wrote:
> Hi,
>
> i'm having problems with matching strings containing multiple pipe 
> characters using @pmFromFile. For example, i'm able to match this 
> pattern (with only 1 pipe):
> aaa|bbb
>
>
> But i'm unable to match this pattern (with multiple pipes):
> aaa|bbb|ccc
>
> Any hints what's wrong? Thanks. modsecurity 2.9.3, Apache 2.4.
>
> azur
>
>
>
>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

[mod-security-users] Problem with pipe and @pmFromFile

[<az...@po...>]

Hi,

i'm having problems with matching strings containing multiple pipe  
characters using @pmFromFile. For example, i'm able to match this  
pattern (with only 1 pipe):
aaa|bbb


But i'm unable to match this pattern (with multiple pipes):
aaa|bbb|ccc

Any hints what's wrong? Thanks. modsecurity 2.9.3, Apache 2.4.

azur

Re: [mod-security-users] Performance woes - larger JSON payloads with CRS

[Henri C. <he...@pr...>]

This is really great, and works like a charm - thanks so much

On Mon, 10 May 2021 at 15:35, <az...@po...> wrote:

> Another, maybe better, solution, as you are already going to modify
> modsecurity.conf, would be to modify rule 200001 directly. Try this:
>
>
> SecRule REQUEST_HEADERS:Content-Type "application/json" \
>       "id:'200001',phase:1,t:none,t:lowercase,pass,nolog,chain"
>       SecRule REQUEST_HEADERS:Content-Length "@lt 131072" \
>            "t:none,ctl:requestBodyProcessor=JSON"
>
>
>
>
>
> Citát az...@po...:
>
> > Sorry, now I see the problem: Rule which is doing JSON processing
> > (200001) is running in phase 1 and my rule below is running in phase
> > 2. Problem is, as i stated below, REQUEST_BODY_LENGTH is available
> > from phase 2 so my rule cannot be run in phase 1. Also, i noticed
> > another problem - we cannot do 'ruleRemoveTargetByTag=OWASP_CRS' as
> > rule 200001 is not part of the CRS and it's running before CRS (and
> > also before rules in file
> > REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf).
> >
> > In this case, we can use Content-Length header, which won't be so
> > realibe but should be sufficient. Try this:
> >
> > SecRule REQUEST_HEADERS:Content-Type "application/json" \
> >     "id:9999999,\
> >     phase:1,\
> >     pass,\
> >     t:none,\
> >     nolog,\
> >     chain"
> >     SecRule REQUEST_HEADERS:Content-Length "@gt 131072" \
> >         "t:none,\
> >         ctl:ruleRemoveTargetById=200001;REQUEST_BODY"
> >
> >
> > This rule should be run before rule 200001 so you will, probably,
> > need to put it inside modsecurity.conf before rule 200001 (but i
> > didn't tested it at all).
> >
> >
> >
> >
> >
> > Citát Henri Cook <he...@pr...>:
> >
> >> Thanks for this azurit but neither of these worked for me. I changed
> the id
> >> to 1001 and put it in my REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf - I
> >> still see a delay processing a large json payload - do you have any idea
> >> where I might have gone wrong?
> >>
> >> I think my goal is to stop the JSON request body being parsed if it's
> over
> >> 128kb in size. An advance goal would be to have it scanned as text to
> >> provide *some* level of protection - I think this is what cloudflare
> does
> >> (from a very cursory glance)
> >>
> >> Best Regards,
> >>
> >> Henri
> >>
> >> On Thu, 29 Apr 2021 at 19:02, <az...@po...> wrote:
> >>
> >>> I just noticed you are using CRS, so this one would be probably better:
> >>>
> >>>
> >>> SecRule REQUEST_BODY_LENGTH "@gt 131072" \
> >>>     "id:9999999,\
> >>>     phase:2,\
> >>>     pass,\
> >>>     t:none,\
> >>>     nolog,\
> >>>     ctl:ruleRemoveTargetByTag=OWASP_CRS;REQUEST_BODY"
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Citát az...@po...:
> >>>
> >>>> Hi,
> >>>>
> >>>> you can do this using exclusive rule, something like this (set the
> >>>> rule ID to something 'better'), length is in bytes:
> >>>>
> >>>>
> >>>> SecRule REQUEST_BODY_LENGTH "@gt 131072" \
> >>>>     "id:9999999,\
> >>>>     phase:2,\
> >>>>     pass,\
> >>>>     t:none,\
> >>>>     nolog,\
> >>>>     ctl:ruleEngine=Off"
> >>>>
> >>>>
> >>>>
> >>>> REQUEST_BODY_LENGTH is available from phase 2 so you cannot put this
> >>>> into phase 1.
> >>>>
> >>>>
> >>>>
> >>>> Citát Henri Cook <he...@pr...>:
> >>>>
> >>>>> I think what I need here is a way to exempt the request body from any
> >>>>> scanning when it's over 128kb (my chosen upper limit)...
> >>>>>
> >>>>> I'll probably try implementing this as a phase 1 rule on
> Content-Length
> >>>>> unless anyone shouts there's a better way. `ProcessPartial` sounded
> like
> >>>>> the holy grail, but you can't partial process JSON (because it
> doesn't
> >>>>> parse!).
> >>>>>
> >>>>> It'd be cool to be able to fallback to a fulltext scan of the partial
> >>> JSON
> >>>>> if it's >128kb, I might have a try at looking into that also (again
> any
> >>>>> guidance appreciated)
> >>>>>
> >>>>> Best Regards,
> >>>>>
> >>>>> Henri
> >>>>>
> >>>>> On Mon, 26 Apr 2021 at 15:24, Henri Cook <he...@pr...> wrote:
> >>>>>
> >>>>>> The follow up problem to this is:
> >>>>>>
> >>>>>> Now i'm set to `SecRequestBodyLimit` 31457280 and
> >>>>>> `SecRequestBodyNoFilesLimit` 65536 and `SecRequestBodyLimitAction
> >>>>>> PartialProcess`
> >>>>>>
> >>>>>> In my mind this will process the first part of a request if it can,
> and
> >>>>>> ignore the rest. But:
> >>>>>>
> >>>>>> - Rule 200002 is triggered, which is saying the JSON can't be
> parsed.
> >>>>>> Presumably because in a large request it tries to process the
> >>> beginning of
> >>>>>> the JSON and can't (because it won't parse, because the JSON is cut
> >>> off so
> >>>>>> doesn't end)
> >>>>>>
> >>>>>> So I think I need to find a way to skip JSON parsing entirely when
> the
> >>>>>> payload is over 64kb (65536)? Does that sound right? Assuming 64kb
> is
> >>> the
> >>>>>> limit I want to stick with. I hadn't really considered before this
> >>> point
> >>>>>> that 'partial processing' of JSON was likely to be hairy but of
> course
> >>> it
> >>>>>> makes sense.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Mon, 26 Apr 2021 at 07:17, Christian Folini <
> >>>>>> chr...@ne...> wrote:
> >>>>>>
> >>>>>>> Hey Henri,
> >>>>>>>
> >>>>>>> From a security practice, this is obviously lacking, but in wider
> >>>>>>> perspective,
> >>>>>>> I see it meet "industry standard", yes.
> >>>>>>>
> >>>>>>> When I teach, I tell my student, that the worst WAF is the one
> that is
> >>>>>>> switched off. So if you need to compromise and you can only apply
> 20%
> >>> of
> >>>>>>> the rules because you run the risk of business demanding it's
> switched
> >>>>>>> off,
> >>>>>>> then that 20% WAF is still better than no WAF.
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>>
> >>>>>>> Christian
> >>>>>>>
> >>>>>>>
> >>>>>>> On Mon, Apr 26, 2021 at 06:57:54AM +0100, Henri Cook wrote:
> >>>>>>>> Thanks Christian, taking this in combination with Osama's point
> >>> earlier
> >>>>>>> in
> >>>>>>>> the thread that most 'big four' (AWS/GCP/Cloudflare/Azure) WAFs
> seem
> >>> to
> >>>>>>>> limit the payload they'll scan. From my reading to 128kb
> >>>>>>> (cloudflare+azure)
> >>>>>>>> or 8kb (aws+gcp) I think I'll be able to resolve our particular
> >>> issue.
> >>>>>>>>
> >>>>>>>> I believe my modern application is already very robust in terms of
> >>>>>>> defence
> >>>>>>>> against sql injection as well as other OWASP top 10 attack vectors
> >>> and
> >>>>>>> that
> >>>>>>>> a WAF primarily adds reassurance (for the business and clients who
> >>> ask
> >>>>>>> if I
> >>>>>>>> have one) and minor frustration (for any potential attacker)
> layer.
> >>> The
> >>>>>>>> spec is to add a WAF that meets (but notably does not necessarily
> >>> have
> >>>>>>> to
> >>>>>>>> exceed) industry standards. I believe this means that I can switch
> >>>>>>> modsec
> >>>>>>>> to 128kb or 8kb partial parsing ('SecResponseBodyLimitAction
> >>>>>>>> ProcessPartial' - allowing through unscanned any payloads over
> those
> >>>>>>> sizes)
> >>>>>>>> and be able to say I've got scan-size-policy-parity with an AWS
> or a
> >>>>>>>> Cloudflare which means it is "industry standard".
> >>>>>>>>
> >>>>>>>> Please let me know if you think that's mad and thanks again
> >>>>>>>>
> >>>>>>>> Best Regards,
> >>>>>>>>
> >>>>>>>> Henri
> >>>>>>>>
> >>>>>>>> On Sun, 25 Apr 2021 at 21:39, Christian Folini <
> >>>>>>> chr...@ne...>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> > Hey Henri,
> >>>>>>>> >
> >>>>>>>> > You are in a bad situation and as far as I can see you are
> right,
> >>> you
> >>>>>>> might
> >>>>>>>> > have to drop modsec/CRS in this situation.
> >>>>>>>> >
> >>>>>>>> > I've had a customer with a similar problem and we did a deep
> dive
> >>>>>>>> > investigation and I had to strike colors in the end.
> >>>>>>>> >
> >>>>>>>> > The point is not the JSON parser. That has shown to be really
> fast.
> >>>>>>> The
> >>>>>>>> > point
> >>>>>>>> > is several hundred variables that go into CRS afterwards. If you
> >>> run
> >>>>>>> CRS
> >>>>>>>> > on a
> >>>>>>>> > standard web application you get forms with a few parameters and
> >>>>>>> that's
> >>>>>>>> > easy.
> >>>>>>>> > But several megabytes of JSON means hundreds of arguments and
> CRS
> >>>>>>> parses
> >>>>>>>> > them
> >>>>>>>> > all.
> >>>>>>>> >
> >>>>>>>> > So we tried to work with rule exclusions and skip the
> parameters we
> >>>>>>> did not
> >>>>>>>> > think dangerous, but here comes the bummer: ModSec 2.9 grew
> >>>>>>> substantially
> >>>>>>>> > slower the longer the ignore-lists of parameters became. This
> and a
> >>>>>>> few
> >>>>>>>> > very
> >>>>>>>> > odd behaviors.
> >>>>>>>> >
> >>>>>>>> > Given the customer wanted a generic WAF without tuning of
> >>> individual
> >>>>>>> APIs
> >>>>>>>> > we
> >>>>>>>> > got to a dead end.
> >>>>>>>> >
> >>>>>>>> > However, if tuning was an option, then I would probably edit-CRS
> >>> with
> >>>>>>>> > msc_pyparser and replace the target lists with arguments I was
> >>>>>>> interested
> >>>>>>>> > in.
> >>>>>>>> >
> >>>>>>>> > https://coreruleset.org/20200901/introducing-msc_pyparser/
> >>>>>>>> >
> >>>>>>>> > As a complementary practice, one could think of performing
> >>> allowlist
> >>>>>>>> > checks on
> >>>>>>>> > some / most of the JSON. Say you have a huge JSON payload with
> 500
> >>>>>>>> > parameters.
> >>>>>>>> > You examine it and discover that 300 of them actually contain
> >>> simple
> >>>>>>> digits
> >>>>>>>> > and asciii characters and neither special chars nor escape
> >>> sequences.
> >>>>>>>> > So you do a regex allowlist and apply it to these 300
> parameters of
> >>>>>>> said
> >>>>>>>> > API. And the rest you can push into CRS. Or a subset of CRS.
> >>>>>>>> >
> >>>>>>>> > I have not done this and the problem is if ModSec is able to
> handle
> >>>>>>> the
> >>>>>>>> > large
> >>>>>>>> > target lists in a speedy manner.
> >>>>>>>> >
> >>>>>>>> >
> >>>>>>>> > Now you can turn to a CDN or alternative WAF. I would do an
> >>> extensive
> >>>>>>>> > security
> >>>>>>>> > tests of such a system. As I said, the JSON parser can be really
> >>>>>>> fast. The
> >>>>>>>> > difficult thing is to check several hundred parameters without
> >>> losing
> >>>>>>>> > performance.
> >>>>>>>> >
> >>>>>>>> > Good luck!
> >>>>>>>> >
> >>>>>>>> > Christian
> >>>>>>>> >
> >>>>>>>> >
> >>>>>>>> > On Sun, Apr 25, 2021 at 08:47:06PM +0100, Henri Cook wrote:
> >>>>>>>> > > Hi all,
> >>>>>>>> > >
> >>>>>>>> > > I'm in a situation where the only solution seems to be to drop
> >>>>>>> modsec/CRS
> >>>>>>>> > > and look at something like Cloudflare's WAF (and change our
> >>> security
> >>>>>>>> > model
> >>>>>>>> > > out of necessity). I'm hoping the esteemed membership of this
> >>> list
> >>>>>>> might
> >>>>>>>> > > have some thoughts.
> >>>>>>>> > >
> >>>>>>>> > > I've got about 1MB of JSON, payloads in our app might run to
> 20
> >>> or
> >>>>>>> even
> >>>>>>>> > > 30MB ultimately.
> >>>>>>>> > > This 1MB of somewhat nested JSON (7 or 8 levels deep) can
> take 40
> >>>>>>> seconds
> >>>>>>>> > > to process in mod sec 3.0.4 with CRS 3.2.0
> >>>>>>>> > >
> >>>>>>>> > > It takes 1 second to process in our API so the WAF element is
> a
> >>> 39x
> >>>>>>> slow
> >>>>>>>> > > down. I appreciate there'll be some delays in WAF.
> Cloudflare's
> >>> WAF
> >>>>>>>> > takes 5
> >>>>>>>> > > seconds to scan this payload - and that's my target.
> >>>>>>>> > >
> >>>>>>>> > > Has anyone got any idea how to improve performance? Reading
> blog
> >>>>>>> posts
> >>>>>>>> > > about the development of cloudflare's waf I see that
> memoization
> >>> of
> >>>>>>>> > common
> >>>>>>>> > > function calls was one of their absolute best performance
> >>>>>>> improvements
> >>>>>>>> > over
> >>>>>>>> > > their modsec implementation (e.g. strlen(response_body) so
> it's
> >>> only
> >>>>>>>> > > calculated once instead of once per rule OR
> >>> contains('somestring',
> >>>>>>>> > > response_body)... you get the drift). Do we have anything like
> >>> this
> >>>>>>> in
> >>>>>>>> > > modsec today? Is that already in place and my 39 seconds is
> after
> >>>>>>> that?
> >>>>>>>> > >
> >>>>>>>> > > I appreciate that mod sec is fast on its own and adding
> complex
> >>>>>>> rules can
> >>>>>>>> > > be said to slow it down. With CRS being by far the most common
> >>> use
> >>>>>>> case
> >>>>>>>> > for
> >>>>>>>> > > mod sec (based on my googling) I'm surprised it's this slow,
> do
> >>> you
> >>>>>>> think
> >>>>>>>> > > i've missed something?
> >>>>>>>> > >
> >>>>>>>> > > To note: I'm only scanning JSON payloads, typically much less
> >>> than
> >>>>>>> 0.5MB
> >>>>>>>> > > but new, irregular ones that we need scanned in ideally <10
> >>> seconds
> >>>>>>> that
> >>>>>>>> > > can range from 1MB-30MB
> >>>>>>>> > >
> >>>>>>>> > > Best regards,
> >>>>>>>> > >
> >>>>>>>> > > Henri Cook
> >>>>>>>> >
> >>>>>>>> >
> >>>>>>>> > > _______________________________________________
> >>>>>>>> > > mod-security-users mailing list
> >>>>>>>> > > mod...@li...
> >>>>>>>> > >
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >>>>>>>> > > Commercial ModSecurity Rules and Support from Trustwave's
> >>>>>>> SpiderLabs:
> >>>>>>>> > > http://www.modsecurity.org/projects/commercial/rules/
> >>>>>>>> > > http://www.modsecurity.org/projects/commercial/support/
> >>>>>>>> >
> >>>>>>>> >
> >>>>>>>> >
> >>>>>>>> > _______________________________________________
> >>>>>>>> > mod-security-users mailing list
> >>>>>>>> > mod...@li...
> >>>>>>>> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >>>>>>>> > Commercial ModSecurity Rules and Support from Trustwave's
> >>> SpiderLabs:
> >>>>>>>> > http://www.modsecurity.org/projects/commercial/rules/
> >>>>>>>> > http://www.modsecurity.org/projects/commercial/support/
> >>>>>>>> >
> >>>>>>>
> >>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> mod-security-users mailing list
> >>>>>>>> mod...@li...
> >>>>>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >>>>>>>> Commercial ModSecurity Rules and Support from Trustwave's
> SpiderLabs:
> >>>>>>>> http://www.modsecurity.org/projects/commercial/rules/
> >>>>>>>> http://www.modsecurity.org/projects/commercial/support/
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> mod-security-users mailing list
> >>>>>>> mod...@li...
> >>>>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >>>>>>> Commercial ModSecurity Rules and Support from Trustwave's
> SpiderLabs:
> >>>>>>> http://www.modsecurity.org/projects/commercial/rules/
> >>>>>>> http://www.modsecurity.org/projects/commercial/support/
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> mod-security-users mailing list
> >>>> mod...@li...
> >>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> >>>> http://www.modsecurity.org/projects/commercial/rules/
> >>>> http://www.modsecurity.org/projects/commercial/support/
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> mod-security-users mailing list
> >>> mod...@li...
> >>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> >>> http://www.modsecurity.org/projects/commercial/rules/
> >>> http://www.modsecurity.org/projects/commercial/support/
> >>>
> >
> >
> >
> >
> >
> > _______________________________________________
> > mod-security-users mailing list
> > mod...@li...
> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> > http://www.modsecurity.org/projects/commercial/rules/
> > http://www.modsecurity.org/projects/commercial/support/
>
>
>
>
>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>

Re: [mod-security-users] Performance woes - larger JSON payloads with CRS

[<az...@po...>]

Another, maybe better, solution, as you are already going to modify  
modsecurity.conf, would be to modify rule 200001 directly. Try this:


SecRule REQUEST_HEADERS:Content-Type "application/json" \
      "id:'200001',phase:1,t:none,t:lowercase,pass,nolog,chain"
      SecRule REQUEST_HEADERS:Content-Length "@lt 131072" \
           "t:none,ctl:requestBodyProcessor=JSON"





Citát az...@po...:

> Sorry, now I see the problem: Rule which is doing JSON processing  
> (200001) is running in phase 1 and my rule below is running in phase  
> 2. Problem is, as i stated below, REQUEST_BODY_LENGTH is available  
> from phase 2 so my rule cannot be run in phase 1. Also, i noticed  
> another problem - we cannot do 'ruleRemoveTargetByTag=OWASP_CRS' as  
> rule 200001 is not part of the CRS and it's running before CRS (and  
> also before rules in file  
> REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf).
>
> In this case, we can use Content-Length header, which won't be so  
> realibe but should be sufficient. Try this:
>
> SecRule REQUEST_HEADERS:Content-Type "application/json" \
>     "id:9999999,\
>     phase:1,\
>     pass,\
>     t:none,\
>     nolog,\
>     chain"
>     SecRule REQUEST_HEADERS:Content-Length "@gt 131072" \
>         "t:none,\
>         ctl:ruleRemoveTargetById=200001;REQUEST_BODY"
>
>
> This rule should be run before rule 200001 so you will, probably,  
> need to put it inside modsecurity.conf before rule 200001 (but i  
> didn't tested it at all).
>
>
>
>
>
> Citát Henri Cook <he...@pr...>:
>
>> Thanks for this azurit but neither of these worked for me. I changed the id
>> to 1001 and put it in my REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf - I
>> still see a delay processing a large json payload - do you have any idea
>> where I might have gone wrong?
>>
>> I think my goal is to stop the JSON request body being parsed if it's over
>> 128kb in size. An advance goal would be to have it scanned as text to
>> provide *some* level of protection - I think this is what cloudflare does
>> (from a very cursory glance)
>>
>> Best Regards,
>>
>> Henri
>>
>> On Thu, 29 Apr 2021 at 19:02, <az...@po...> wrote:
>>
>>> I just noticed you are using CRS, so this one would be probably better:
>>>
>>>
>>> SecRule REQUEST_BODY_LENGTH "@gt 131072" \
>>>     "id:9999999,\
>>>     phase:2,\
>>>     pass,\
>>>     t:none,\
>>>     nolog,\
>>>     ctl:ruleRemoveTargetByTag=OWASP_CRS;REQUEST_BODY"
>>>
>>>
>>>
>>>
>>>
>>> Citát az...@po...:
>>>
>>>> Hi,
>>>>
>>>> you can do this using exclusive rule, something like this (set the
>>>> rule ID to something 'better'), length is in bytes:
>>>>
>>>>
>>>> SecRule REQUEST_BODY_LENGTH "@gt 131072" \
>>>>     "id:9999999,\
>>>>     phase:2,\
>>>>     pass,\
>>>>     t:none,\
>>>>     nolog,\
>>>>     ctl:ruleEngine=Off"
>>>>
>>>>
>>>>
>>>> REQUEST_BODY_LENGTH is available from phase 2 so you cannot put this
>>>> into phase 1.
>>>>
>>>>
>>>>
>>>> Citát Henri Cook <he...@pr...>:
>>>>
>>>>> I think what I need here is a way to exempt the request body from any
>>>>> scanning when it's over 128kb (my chosen upper limit)...
>>>>>
>>>>> I'll probably try implementing this as a phase 1 rule on Content-Length
>>>>> unless anyone shouts there's a better way. `ProcessPartial` sounded like
>>>>> the holy grail, but you can't partial process JSON (because it doesn't
>>>>> parse!).
>>>>>
>>>>> It'd be cool to be able to fallback to a fulltext scan of the partial
>>> JSON
>>>>> if it's >128kb, I might have a try at looking into that also (again any
>>>>> guidance appreciated)
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Henri
>>>>>
>>>>> On Mon, 26 Apr 2021 at 15:24, Henri Cook <he...@pr...> wrote:
>>>>>
>>>>>> The follow up problem to this is:
>>>>>>
>>>>>> Now i'm set to `SecRequestBodyLimit` 31457280 and
>>>>>> `SecRequestBodyNoFilesLimit` 65536 and `SecRequestBodyLimitAction
>>>>>> PartialProcess`
>>>>>>
>>>>>> In my mind this will process the first part of a request if it can, and
>>>>>> ignore the rest. But:
>>>>>>
>>>>>> - Rule 200002 is triggered, which is saying the JSON can't be parsed.
>>>>>> Presumably because in a large request it tries to process the
>>> beginning of
>>>>>> the JSON and can't (because it won't parse, because the JSON is cut
>>> off so
>>>>>> doesn't end)
>>>>>>
>>>>>> So I think I need to find a way to skip JSON parsing entirely when the
>>>>>> payload is over 64kb (65536)? Does that sound right? Assuming 64kb is
>>> the
>>>>>> limit I want to stick with. I hadn't really considered before this
>>> point
>>>>>> that 'partial processing' of JSON was likely to be hairy but of course
>>> it
>>>>>> makes sense.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, 26 Apr 2021 at 07:17, Christian Folini <
>>>>>> chr...@ne...> wrote:
>>>>>>
>>>>>>> Hey Henri,
>>>>>>>
>>>>>>> From a security practice, this is obviously lacking, but in wider
>>>>>>> perspective,
>>>>>>> I see it meet "industry standard", yes.
>>>>>>>
>>>>>>> When I teach, I tell my student, that the worst WAF is the one that is
>>>>>>> switched off. So if you need to compromise and you can only apply 20%
>>> of
>>>>>>> the rules because you run the risk of business demanding it's switched
>>>>>>> off,
>>>>>>> then that 20% WAF is still better than no WAF.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Christian
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Apr 26, 2021 at 06:57:54AM +0100, Henri Cook wrote:
>>>>>>>> Thanks Christian, taking this in combination with Osama's point
>>> earlier
>>>>>>> in
>>>>>>>> the thread that most 'big four' (AWS/GCP/Cloudflare/Azure) WAFs seem
>>> to
>>>>>>>> limit the payload they'll scan. From my reading to 128kb
>>>>>>> (cloudflare+azure)
>>>>>>>> or 8kb (aws+gcp) I think I'll be able to resolve our particular
>>> issue.
>>>>>>>>
>>>>>>>> I believe my modern application is already very robust in terms of
>>>>>>> defence
>>>>>>>> against sql injection as well as other OWASP top 10 attack vectors
>>> and
>>>>>>> that
>>>>>>>> a WAF primarily adds reassurance (for the business and clients who
>>> ask
>>>>>>> if I
>>>>>>>> have one) and minor frustration (for any potential attacker) layer.
>>> The
>>>>>>>> spec is to add a WAF that meets (but notably does not necessarily
>>> have
>>>>>>> to
>>>>>>>> exceed) industry standards. I believe this means that I can switch
>>>>>>> modsec
>>>>>>>> to 128kb or 8kb partial parsing ('SecResponseBodyLimitAction
>>>>>>>> ProcessPartial' - allowing through unscanned any payloads over those
>>>>>>> sizes)
>>>>>>>> and be able to say I've got scan-size-policy-parity with an AWS or a
>>>>>>>> Cloudflare which means it is "industry standard".
>>>>>>>>
>>>>>>>> Please let me know if you think that's mad and thanks again
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>>
>>>>>>>> Henri
>>>>>>>>
>>>>>>>> On Sun, 25 Apr 2021 at 21:39, Christian Folini <
>>>>>>> chr...@ne...>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> > Hey Henri,
>>>>>>>> >
>>>>>>>> > You are in a bad situation and as far as I can see you are right,
>>> you
>>>>>>> might
>>>>>>>> > have to drop modsec/CRS in this situation.
>>>>>>>> >
>>>>>>>> > I've had a customer with a similar problem and we did a deep dive
>>>>>>>> > investigation and I had to strike colors in the end.
>>>>>>>> >
>>>>>>>> > The point is not the JSON parser. That has shown to be really fast.
>>>>>>> The
>>>>>>>> > point
>>>>>>>> > is several hundred variables that go into CRS afterwards. If you
>>> run
>>>>>>> CRS
>>>>>>>> > on a
>>>>>>>> > standard web application you get forms with a few parameters and
>>>>>>> that's
>>>>>>>> > easy.
>>>>>>>> > But several megabytes of JSON means hundreds of arguments and CRS
>>>>>>> parses
>>>>>>>> > them
>>>>>>>> > all.
>>>>>>>> >
>>>>>>>> > So we tried to work with rule exclusions and skip the parameters we
>>>>>>> did not
>>>>>>>> > think dangerous, but here comes the bummer: ModSec 2.9 grew
>>>>>>> substantially
>>>>>>>> > slower the longer the ignore-lists of parameters became. This and a
>>>>>>> few
>>>>>>>> > very
>>>>>>>> > odd behaviors.
>>>>>>>> >
>>>>>>>> > Given the customer wanted a generic WAF without tuning of
>>> individual
>>>>>>> APIs
>>>>>>>> > we
>>>>>>>> > got to a dead end.
>>>>>>>> >
>>>>>>>> > However, if tuning was an option, then I would probably edit-CRS
>>> with
>>>>>>>> > msc_pyparser and replace the target lists with arguments I was
>>>>>>> interested
>>>>>>>> > in.
>>>>>>>> >
>>>>>>>> > https://coreruleset.org/20200901/introducing-msc_pyparser/
>>>>>>>> >
>>>>>>>> > As a complementary practice, one could think of performing
>>> allowlist
>>>>>>>> > checks on
>>>>>>>> > some / most of the JSON. Say you have a huge JSON payload with 500
>>>>>>>> > parameters.
>>>>>>>> > You examine it and discover that 300 of them actually contain
>>> simple
>>>>>>> digits
>>>>>>>> > and asciii characters and neither special chars nor escape
>>> sequences.
>>>>>>>> > So you do a regex allowlist and apply it to these 300 parameters of
>>>>>>> said
>>>>>>>> > API. And the rest you can push into CRS. Or a subset of CRS.
>>>>>>>> >
>>>>>>>> > I have not done this and the problem is if ModSec is able to handle
>>>>>>> the
>>>>>>>> > large
>>>>>>>> > target lists in a speedy manner.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > Now you can turn to a CDN or alternative WAF. I would do an
>>> extensive
>>>>>>>> > security
>>>>>>>> > tests of such a system. As I said, the JSON parser can be really
>>>>>>> fast. The
>>>>>>>> > difficult thing is to check several hundred parameters without
>>> losing
>>>>>>>> > performance.
>>>>>>>> >
>>>>>>>> > Good luck!
>>>>>>>> >
>>>>>>>> > Christian
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Sun, Apr 25, 2021 at 08:47:06PM +0100, Henri Cook wrote:
>>>>>>>> > > Hi all,
>>>>>>>> > >
>>>>>>>> > > I'm in a situation where the only solution seems to be to drop
>>>>>>> modsec/CRS
>>>>>>>> > > and look at something like Cloudflare's WAF (and change our
>>> security
>>>>>>>> > model
>>>>>>>> > > out of necessity). I'm hoping the esteemed membership of this
>>> list
>>>>>>> might
>>>>>>>> > > have some thoughts.
>>>>>>>> > >
>>>>>>>> > > I've got about 1MB of JSON, payloads in our app might run to 20
>>> or
>>>>>>> even
>>>>>>>> > > 30MB ultimately.
>>>>>>>> > > This 1MB of somewhat nested JSON (7 or 8 levels deep) can take 40
>>>>>>> seconds
>>>>>>>> > > to process in mod sec 3.0.4 with CRS 3.2.0
>>>>>>>> > >
>>>>>>>> > > It takes 1 second to process in our API so the WAF element is a
>>> 39x
>>>>>>> slow
>>>>>>>> > > down. I appreciate there'll be some delays in WAF. Cloudflare's
>>> WAF
>>>>>>>> > takes 5
>>>>>>>> > > seconds to scan this payload - and that's my target.
>>>>>>>> > >
>>>>>>>> > > Has anyone got any idea how to improve performance? Reading blog
>>>>>>> posts
>>>>>>>> > > about the development of cloudflare's waf I see that memoization
>>> of
>>>>>>>> > common
>>>>>>>> > > function calls was one of their absolute best performance
>>>>>>> improvements
>>>>>>>> > over
>>>>>>>> > > their modsec implementation (e.g. strlen(response_body) so it's
>>> only
>>>>>>>> > > calculated once instead of once per rule OR
>>> contains('somestring',
>>>>>>>> > > response_body)... you get the drift). Do we have anything like
>>> this
>>>>>>> in
>>>>>>>> > > modsec today? Is that already in place and my 39 seconds is after
>>>>>>> that?
>>>>>>>> > >
>>>>>>>> > > I appreciate that mod sec is fast on its own and adding complex
>>>>>>> rules can
>>>>>>>> > > be said to slow it down. With CRS being by far the most common
>>> use
>>>>>>> case
>>>>>>>> > for
>>>>>>>> > > mod sec (based on my googling) I'm surprised it's this slow, do
>>> you
>>>>>>> think
>>>>>>>> > > i've missed something?
>>>>>>>> > >
>>>>>>>> > > To note: I'm only scanning JSON payloads, typically much less
>>> than
>>>>>>> 0.5MB
>>>>>>>> > > but new, irregular ones that we need scanned in ideally <10
>>> seconds
>>>>>>> that
>>>>>>>> > > can range from 1MB-30MB
>>>>>>>> > >
>>>>>>>> > > Best regards,
>>>>>>>> > >
>>>>>>>> > > Henri Cook
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > > _______________________________________________
>>>>>>>> > > mod-security-users mailing list
>>>>>>>> > > mod...@li...
>>>>>>>> > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>>>>>> > > Commercial ModSecurity Rules and Support from Trustwave's
>>>>>>> SpiderLabs:
>>>>>>>> > > http://www.modsecurity.org/projects/commercial/rules/
>>>>>>>> > > http://www.modsecurity.org/projects/commercial/support/
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > _______________________________________________
>>>>>>>> > mod-security-users mailing list
>>>>>>>> > mod...@li...
>>>>>>>> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>>>>>> > Commercial ModSecurity Rules and Support from Trustwave's
>>> SpiderLabs:
>>>>>>>> > http://www.modsecurity.org/projects/commercial/rules/
>>>>>>>> > http://www.modsecurity.org/projects/commercial/support/
>>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> mod-security-users mailing list
>>>>>>>> mod...@li...
>>>>>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>>>>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>>>>>>> http://www.modsecurity.org/projects/commercial/rules/
>>>>>>>> http://www.modsecurity.org/projects/commercial/support/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> mod-security-users mailing list
>>>>>>> mod...@li...
>>>>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>>>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>>>>>> http://www.modsecurity.org/projects/commercial/rules/
>>>>>>> http://www.modsecurity.org/projects/commercial/support/
>>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> mod-security-users mailing list
>>>> mod...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>>> http://www.modsecurity.org/projects/commercial/rules/
>>>> http://www.modsecurity.org/projects/commercial/support/
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> mod-security-users mailing list
>>> mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> http://www.modsecurity.org/projects/commercial/rules/
>>> http://www.modsecurity.org/projects/commercial/support/
>>>
>
>
>
>
>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] Performance woes - larger JSON payloads with CRS

[<az...@po...>]

Sorry, now I see the problem: Rule which is doing JSON processing  
(200001) is running in phase 1 and my rule below is running in phase  
2. Problem is, as i stated below, REQUEST_BODY_LENGTH is available  
from phase 2 so my rule cannot be run in phase 1. Also, i noticed  
another problem - we cannot do 'ruleRemoveTargetByTag=OWASP_CRS' as  
rule 200001 is not part of the CRS and it's running before CRS (and  
also before rules in file REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf).

In this case, we can use Content-Length header, which won't be so  
realibe but should be sufficient. Try this:

SecRule REQUEST_HEADERS:Content-Type "application/json" \
     "id:9999999,\
     phase:1,\
     pass,\
     t:none,\
     nolog,\
     chain"
     SecRule REQUEST_HEADERS:Content-Length "@gt 131072" \
         "t:none,\
         ctl:ruleRemoveTargetById=200001;REQUEST_BODY"


This rule should be run before rule 200001 so you will, probably, need  
to put it inside modsecurity.conf before rule 200001 (but i didn't  
tested it at all).





Citát Henri Cook <he...@pr...>:

> Thanks for this azurit but neither of these worked for me. I changed the id
> to 1001 and put it in my REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf - I
> still see a delay processing a large json payload - do you have any idea
> where I might have gone wrong?
>
> I think my goal is to stop the JSON request body being parsed if it's over
> 128kb in size. An advance goal would be to have it scanned as text to
> provide *some* level of protection - I think this is what cloudflare does
> (from a very cursory glance)
>
> Best Regards,
>
> Henri
>
> On Thu, 29 Apr 2021 at 19:02, <az...@po...> wrote:
>
>> I just noticed you are using CRS, so this one would be probably better:
>>
>>
>> SecRule REQUEST_BODY_LENGTH "@gt 131072" \
>>      "id:9999999,\
>>      phase:2,\
>>      pass,\
>>      t:none,\
>>      nolog,\
>>      ctl:ruleRemoveTargetByTag=OWASP_CRS;REQUEST_BODY"
>>
>>
>>
>>
>>
>> Citát az...@po...:
>>
>> > Hi,
>> >
>> > you can do this using exclusive rule, something like this (set the
>> > rule ID to something 'better'), length is in bytes:
>> >
>> >
>> > SecRule REQUEST_BODY_LENGTH "@gt 131072" \
>> >     "id:9999999,\
>> >     phase:2,\
>> >     pass,\
>> >     t:none,\
>> >     nolog,\
>> >     ctl:ruleEngine=Off"
>> >
>> >
>> >
>> > REQUEST_BODY_LENGTH is available from phase 2 so you cannot put this
>> > into phase 1.
>> >
>> >
>> >
>> > Citát Henri Cook <he...@pr...>:
>> >
>> >> I think what I need here is a way to exempt the request body from any
>> >> scanning when it's over 128kb (my chosen upper limit)...
>> >>
>> >> I'll probably try implementing this as a phase 1 rule on Content-Length
>> >> unless anyone shouts there's a better way. `ProcessPartial` sounded like
>> >> the holy grail, but you can't partial process JSON (because it doesn't
>> >> parse!).
>> >>
>> >> It'd be cool to be able to fallback to a fulltext scan of the partial
>> JSON
>> >> if it's >128kb, I might have a try at looking into that also (again any
>> >> guidance appreciated)
>> >>
>> >> Best Regards,
>> >>
>> >> Henri
>> >>
>> >> On Mon, 26 Apr 2021 at 15:24, Henri Cook <he...@pr...> wrote:
>> >>
>> >>> The follow up problem to this is:
>> >>>
>> >>> Now i'm set to `SecRequestBodyLimit` 31457280 and
>> >>> `SecRequestBodyNoFilesLimit` 65536 and `SecRequestBodyLimitAction
>> >>> PartialProcess`
>> >>>
>> >>> In my mind this will process the first part of a request if it can, and
>> >>> ignore the rest. But:
>> >>>
>> >>> - Rule 200002 is triggered, which is saying the JSON can't be parsed.
>> >>> Presumably because in a large request it tries to process the
>> beginning of
>> >>> the JSON and can't (because it won't parse, because the JSON is cut
>> off so
>> >>> doesn't end)
>> >>>
>> >>> So I think I need to find a way to skip JSON parsing entirely when the
>> >>> payload is over 64kb (65536)? Does that sound right? Assuming 64kb is
>> the
>> >>> limit I want to stick with. I hadn't really considered before this
>> point
>> >>> that 'partial processing' of JSON was likely to be hairy but of course
>> it
>> >>> makes sense.
>> >>>
>> >>>
>> >>>
>> >>> On Mon, 26 Apr 2021 at 07:17, Christian Folini <
>> >>> chr...@ne...> wrote:
>> >>>
>> >>>> Hey Henri,
>> >>>>
>> >>>> From a security practice, this is obviously lacking, but in wider
>> >>>> perspective,
>> >>>> I see it meet "industry standard", yes.
>> >>>>
>> >>>> When I teach, I tell my student, that the worst WAF is the one that is
>> >>>> switched off. So if you need to compromise and you can only apply 20%
>> of
>> >>>> the rules because you run the risk of business demanding it's switched
>> >>>> off,
>> >>>> then that 20% WAF is still better than no WAF.
>> >>>>
>> >>>> Cheers,
>> >>>>
>> >>>> Christian
>> >>>>
>> >>>>
>> >>>> On Mon, Apr 26, 2021 at 06:57:54AM +0100, Henri Cook wrote:
>> >>>>> Thanks Christian, taking this in combination with Osama's point
>> earlier
>> >>>> in
>> >>>>> the thread that most 'big four' (AWS/GCP/Cloudflare/Azure) WAFs seem
>> to
>> >>>>> limit the payload they'll scan. From my reading to 128kb
>> >>>> (cloudflare+azure)
>> >>>>> or 8kb (aws+gcp) I think I'll be able to resolve our particular
>> issue.
>> >>>>>
>> >>>>> I believe my modern application is already very robust in terms of
>> >>>> defence
>> >>>>> against sql injection as well as other OWASP top 10 attack vectors
>> and
>> >>>> that
>> >>>>> a WAF primarily adds reassurance (for the business and clients who
>> ask
>> >>>> if I
>> >>>>> have one) and minor frustration (for any potential attacker) layer.
>> The
>> >>>>> spec is to add a WAF that meets (but notably does not necessarily
>> have
>> >>>> to
>> >>>>> exceed) industry standards. I believe this means that I can switch
>> >>>> modsec
>> >>>>> to 128kb or 8kb partial parsing ('SecResponseBodyLimitAction
>> >>>>> ProcessPartial' - allowing through unscanned any payloads over those
>> >>>> sizes)
>> >>>>> and be able to say I've got scan-size-policy-parity with an AWS or a
>> >>>>> Cloudflare which means it is "industry standard".
>> >>>>>
>> >>>>> Please let me know if you think that's mad and thanks again
>> >>>>>
>> >>>>> Best Regards,
>> >>>>>
>> >>>>> Henri
>> >>>>>
>> >>>>> On Sun, 25 Apr 2021 at 21:39, Christian Folini <
>> >>>> chr...@ne...>
>> >>>>> wrote:
>> >>>>>
>> >>>>> > Hey Henri,
>> >>>>> >
>> >>>>> > You are in a bad situation and as far as I can see you are right,
>> you
>> >>>> might
>> >>>>> > have to drop modsec/CRS in this situation.
>> >>>>> >
>> >>>>> > I've had a customer with a similar problem and we did a deep dive
>> >>>>> > investigation and I had to strike colors in the end.
>> >>>>> >
>> >>>>> > The point is not the JSON parser. That has shown to be really fast.
>> >>>> The
>> >>>>> > point
>> >>>>> > is several hundred variables that go into CRS afterwards. If you
>> run
>> >>>> CRS
>> >>>>> > on a
>> >>>>> > standard web application you get forms with a few parameters and
>> >>>> that's
>> >>>>> > easy.
>> >>>>> > But several megabytes of JSON means hundreds of arguments and CRS
>> >>>> parses
>> >>>>> > them
>> >>>>> > all.
>> >>>>> >
>> >>>>> > So we tried to work with rule exclusions and skip the parameters we
>> >>>> did not
>> >>>>> > think dangerous, but here comes the bummer: ModSec 2.9 grew
>> >>>> substantially
>> >>>>> > slower the longer the ignore-lists of parameters became. This and a
>> >>>> few
>> >>>>> > very
>> >>>>> > odd behaviors.
>> >>>>> >
>> >>>>> > Given the customer wanted a generic WAF without tuning of
>> individual
>> >>>> APIs
>> >>>>> > we
>> >>>>> > got to a dead end.
>> >>>>> >
>> >>>>> > However, if tuning was an option, then I would probably edit-CRS
>> with
>> >>>>> > msc_pyparser and replace the target lists with arguments I was
>> >>>> interested
>> >>>>> > in.
>> >>>>> >
>> >>>>> > https://coreruleset.org/20200901/introducing-msc_pyparser/
>> >>>>> >
>> >>>>> > As a complementary practice, one could think of performing
>> allowlist
>> >>>>> > checks on
>> >>>>> > some / most of the JSON. Say you have a huge JSON payload with 500
>> >>>>> > parameters.
>> >>>>> > You examine it and discover that 300 of them actually contain
>> simple
>> >>>> digits
>> >>>>> > and asciii characters and neither special chars nor escape
>> sequences.
>> >>>>> > So you do a regex allowlist and apply it to these 300 parameters of
>> >>>> said
>> >>>>> > API. And the rest you can push into CRS. Or a subset of CRS.
>> >>>>> >
>> >>>>> > I have not done this and the problem is if ModSec is able to handle
>> >>>> the
>> >>>>> > large
>> >>>>> > target lists in a speedy manner.
>> >>>>> >
>> >>>>> >
>> >>>>> > Now you can turn to a CDN or alternative WAF. I would do an
>> extensive
>> >>>>> > security
>> >>>>> > tests of such a system. As I said, the JSON parser can be really
>> >>>> fast. The
>> >>>>> > difficult thing is to check several hundred parameters without
>> losing
>> >>>>> > performance.
>> >>>>> >
>> >>>>> > Good luck!
>> >>>>> >
>> >>>>> > Christian
>> >>>>> >
>> >>>>> >
>> >>>>> > On Sun, Apr 25, 2021 at 08:47:06PM +0100, Henri Cook wrote:
>> >>>>> > > Hi all,
>> >>>>> > >
>> >>>>> > > I'm in a situation where the only solution seems to be to drop
>> >>>> modsec/CRS
>> >>>>> > > and look at something like Cloudflare's WAF (and change our
>> security
>> >>>>> > model
>> >>>>> > > out of necessity). I'm hoping the esteemed membership of this
>> list
>> >>>> might
>> >>>>> > > have some thoughts.
>> >>>>> > >
>> >>>>> > > I've got about 1MB of JSON, payloads in our app might run to 20
>> or
>> >>>> even
>> >>>>> > > 30MB ultimately.
>> >>>>> > > This 1MB of somewhat nested JSON (7 or 8 levels deep) can take 40
>> >>>> seconds
>> >>>>> > > to process in mod sec 3.0.4 with CRS 3.2.0
>> >>>>> > >
>> >>>>> > > It takes 1 second to process in our API so the WAF element is a
>> 39x
>> >>>> slow
>> >>>>> > > down. I appreciate there'll be some delays in WAF. Cloudflare's
>> WAF
>> >>>>> > takes 5
>> >>>>> > > seconds to scan this payload - and that's my target.
>> >>>>> > >
>> >>>>> > > Has anyone got any idea how to improve performance? Reading blog
>> >>>> posts
>> >>>>> > > about the development of cloudflare's waf I see that memoization
>> of
>> >>>>> > common
>> >>>>> > > function calls was one of their absolute best performance
>> >>>> improvements
>> >>>>> > over
>> >>>>> > > their modsec implementation (e.g. strlen(response_body) so it's
>> only
>> >>>>> > > calculated once instead of once per rule OR
>> contains('somestring',
>> >>>>> > > response_body)... you get the drift). Do we have anything like
>> this
>> >>>> in
>> >>>>> > > modsec today? Is that already in place and my 39 seconds is after
>> >>>> that?
>> >>>>> > >
>> >>>>> > > I appreciate that mod sec is fast on its own and adding complex
>> >>>> rules can
>> >>>>> > > be said to slow it down. With CRS being by far the most common
>> use
>> >>>> case
>> >>>>> > for
>> >>>>> > > mod sec (based on my googling) I'm surprised it's this slow, do
>> you
>> >>>> think
>> >>>>> > > i've missed something?
>> >>>>> > >
>> >>>>> > > To note: I'm only scanning JSON payloads, typically much less
>> than
>> >>>> 0.5MB
>> >>>>> > > but new, irregular ones that we need scanned in ideally <10
>> seconds
>> >>>> that
>> >>>>> > > can range from 1MB-30MB
>> >>>>> > >
>> >>>>> > > Best regards,
>> >>>>> > >
>> >>>>> > > Henri Cook
>> >>>>> >
>> >>>>> >
>> >>>>> > > _______________________________________________
>> >>>>> > > mod-security-users mailing list
>> >>>>> > > mod...@li...
>> >>>>> > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> >>>>> > > Commercial ModSecurity Rules and Support from Trustwave's
>> >>>> SpiderLabs:
>> >>>>> > > http://www.modsecurity.org/projects/commercial/rules/
>> >>>>> > > http://www.modsecurity.org/projects/commercial/support/
>> >>>>> >
>> >>>>> >
>> >>>>> >
>> >>>>> > _______________________________________________
>> >>>>> > mod-security-users mailing list
>> >>>>> > mod...@li...
>> >>>>> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> >>>>> > Commercial ModSecurity Rules and Support from Trustwave's
>> SpiderLabs:
>> >>>>> > http://www.modsecurity.org/projects/commercial/rules/
>> >>>>> > http://www.modsecurity.org/projects/commercial/support/
>> >>>>> >
>> >>>>
>> >>>>
>> >>>>> _______________________________________________
>> >>>>> mod-security-users mailing list
>> >>>>> mod...@li...
>> >>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> >>>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> >>>>> http://www.modsecurity.org/projects/commercial/rules/
>> >>>>> http://www.modsecurity.org/projects/commercial/support/
>> >>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> mod-security-users mailing list
>> >>>> mod...@li...
>> >>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> >>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> >>>> http://www.modsecurity.org/projects/commercial/rules/
>> >>>> http://www.modsecurity.org/projects/commercial/support/
>> >>>>
>> >>>
>> >
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > mod-security-users mailing list
>> > mod...@li...
>> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> > http://www.modsecurity.org/projects/commercial/rules/
>> > http://www.modsecurity.org/projects/commercial/support/
>>
>>
>>
>>
>>
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
>>

Re: [mod-security-users] Performance woes - larger JSON payloads with CRS

[Henri C. <he...@pr...>]

Thanks Srikanth. In my case I want to allow large payloads through
unscanned (or partially scanned) rather than rejecting them outright. My
issue is that scanning my large JSON payloads takes some time. Since
they're rare I'm willing to not scan them in the name of balancing security
and performance, but I'd still rather scan payloads that are <128k in size

On Mon, 10 May 2021 at 14:30, Srikanth Arunachalam via mod-security-users <
mod...@li...> wrote:

> Hi Henri
>
> It defintly worked for me.
> SecRequestBodyLimit 13107200
> SecRequestBodyNoFileaLimit 131072
>
> Restricts to 12.5 mb of large payload.
>
> The only thing is if i had injections through the end, it passes through.
> In your case i think you are fine with it.
>
> And of course SecRequestBodyAccess On
> Should be set
>
> Thanks
> Srikanth Arunachalam
>
> On Mon, 10 May 2021, 14:17 Henri Cook, <he...@pr...> wrote:
>
>> Thanks for this azurit but neither of these worked for me. I changed the
>> id to 1001 and put it in my REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf - I
>> still see a delay processing a large json payload - do you have any idea
>> where I might have gone wrong?
>>
>> I think my goal is to stop the JSON request body being parsed if it's
>> over 128kb in size. An advance goal would be to have it scanned as text to
>> provide *some* level of protection - I think this is what cloudflare
>> does (from a very cursory glance)
>>
>> Best Regards,
>>
>> Henri
>>
>> On Thu, 29 Apr 2021 at 19:02, <az...@po...> wrote:
>>
>>> I just noticed you are using CRS, so this one would be probably better:
>>>
>>>
>>> SecRule REQUEST_BODY_LENGTH "@gt 131072" \
>>>      "id:9999999,\
>>>      phase:2,\
>>>      pass,\
>>>      t:none,\
>>>      nolog,\
>>>      ctl:ruleRemoveTargetByTag=OWASP_CRS;REQUEST_BODY"
>>>
>>>
>>>
>>>
>>>
>>> Citát az...@po...:
>>>
>>> > Hi,
>>> >
>>> > you can do this using exclusive rule, something like this (set the
>>> > rule ID to something 'better'), length is in bytes:
>>> >
>>> >
>>> > SecRule REQUEST_BODY_LENGTH "@gt 131072" \
>>> >     "id:9999999,\
>>> >     phase:2,\
>>> >     pass,\
>>> >     t:none,\
>>> >     nolog,\
>>> >     ctl:ruleEngine=Off"
>>> >
>>> >
>>> >
>>> > REQUEST_BODY_LENGTH is available from phase 2 so you cannot put this
>>> > into phase 1.
>>> >
>>> >
>>> >
>>> > Citát Henri Cook <he...@pr...>:
>>> >
>>> >> I think what I need here is a way to exempt the request body from any
>>> >> scanning when it's over 128kb (my chosen upper limit)...
>>> >>
>>> >> I'll probably try implementing this as a phase 1 rule on
>>> Content-Length
>>> >> unless anyone shouts there's a better way. `ProcessPartial` sounded
>>> like
>>> >> the holy grail, but you can't partial process JSON (because it doesn't
>>> >> parse!).
>>> >>
>>> >> It'd be cool to be able to fallback to a fulltext scan of the partial
>>> JSON
>>> >> if it's >128kb, I might have a try at looking into that also (again
>>> any
>>> >> guidance appreciated)
>>> >>
>>> >> Best Regards,
>>> >>
>>> >> Henri
>>> >>
>>> >> On Mon, 26 Apr 2021 at 15:24, Henri Cook <he...@pr...> wrote:
>>> >>
>>> >>> The follow up problem to this is:
>>> >>>
>>> >>> Now i'm set to `SecRequestBodyLimit` 31457280 and
>>> >>> `SecRequestBodyNoFilesLimit` 65536 and `SecRequestBodyLimitAction
>>> >>> PartialProcess`
>>> >>>
>>> >>> In my mind this will process the first part of a request if it can,
>>> and
>>> >>> ignore the rest. But:
>>> >>>
>>> >>> - Rule 200002 is triggered, which is saying the JSON can't be parsed.
>>> >>> Presumably because in a large request it tries to process the
>>> beginning of
>>> >>> the JSON and can't (because it won't parse, because the JSON is cut
>>> off so
>>> >>> doesn't end)
>>> >>>
>>> >>> So I think I need to find a way to skip JSON parsing entirely when
>>> the
>>> >>> payload is over 64kb (65536)? Does that sound right? Assuming 64kb
>>> is the
>>> >>> limit I want to stick with. I hadn't really considered before this
>>> point
>>> >>> that 'partial processing' of JSON was likely to be hairy but of
>>> course it
>>> >>> makes sense.
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Mon, 26 Apr 2021 at 07:17, Christian Folini <
>>> >>> chr...@ne...> wrote:
>>> >>>
>>> >>>> Hey Henri,
>>> >>>>
>>> >>>> From a security practice, this is obviously lacking, but in wider
>>> >>>> perspective,
>>> >>>> I see it meet "industry standard", yes.
>>> >>>>
>>> >>>> When I teach, I tell my student, that the worst WAF is the one that
>>> is
>>> >>>> switched off. So if you need to compromise and you can only apply
>>> 20% of
>>> >>>> the rules because you run the risk of business demanding it's
>>> switched
>>> >>>> off,
>>> >>>> then that 20% WAF is still better than no WAF.
>>> >>>>
>>> >>>> Cheers,
>>> >>>>
>>> >>>> Christian
>>> >>>>
>>> >>>>
>>> >>>> On Mon, Apr 26, 2021 at 06:57:54AM +0100, Henri Cook wrote:
>>> >>>>> Thanks Christian, taking this in combination with Osama's point
>>> earlier
>>> >>>> in
>>> >>>>> the thread that most 'big four' (AWS/GCP/Cloudflare/Azure) WAFs
>>> seem to
>>> >>>>> limit the payload they'll scan. From my reading to 128kb
>>> >>>> (cloudflare+azure)
>>> >>>>> or 8kb (aws+gcp) I think I'll be able to resolve our particular
>>> issue.
>>> >>>>>
>>> >>>>> I believe my modern application is already very robust in terms of
>>> >>>> defence
>>> >>>>> against sql injection as well as other OWASP top 10 attack vectors
>>> and
>>> >>>> that
>>> >>>>> a WAF primarily adds reassurance (for the business and clients who
>>> ask
>>> >>>> if I
>>> >>>>> have one) and minor frustration (for any potential attacker)
>>> layer. The
>>> >>>>> spec is to add a WAF that meets (but notably does not necessarily
>>> have
>>> >>>> to
>>> >>>>> exceed) industry standards. I believe this means that I can switch
>>> >>>> modsec
>>> >>>>> to 128kb or 8kb partial parsing ('SecResponseBodyLimitAction
>>> >>>>> ProcessPartial' - allowing through unscanned any payloads over
>>> those
>>> >>>> sizes)
>>> >>>>> and be able to say I've got scan-size-policy-parity with an AWS or
>>> a
>>> >>>>> Cloudflare which means it is "industry standard".
>>> >>>>>
>>> >>>>> Please let me know if you think that's mad and thanks again
>>> >>>>>
>>> >>>>> Best Regards,
>>> >>>>>
>>> >>>>> Henri
>>> >>>>>
>>> >>>>> On Sun, 25 Apr 2021 at 21:39, Christian Folini <
>>> >>>> chr...@ne...>
>>> >>>>> wrote:
>>> >>>>>
>>> >>>>> > Hey Henri,
>>> >>>>> >
>>> >>>>> > You are in a bad situation and as far as I can see you are
>>> right, you
>>> >>>> might
>>> >>>>> > have to drop modsec/CRS in this situation.
>>> >>>>> >
>>> >>>>> > I've had a customer with a similar problem and we did a deep dive
>>> >>>>> > investigation and I had to strike colors in the end.
>>> >>>>> >
>>> >>>>> > The point is not the JSON parser. That has shown to be really
>>> fast.
>>> >>>> The
>>> >>>>> > point
>>> >>>>> > is several hundred variables that go into CRS afterwards. If you
>>> run
>>> >>>> CRS
>>> >>>>> > on a
>>> >>>>> > standard web application you get forms with a few parameters and
>>> >>>> that's
>>> >>>>> > easy.
>>> >>>>> > But several megabytes of JSON means hundreds of arguments and CRS
>>> >>>> parses
>>> >>>>> > them
>>> >>>>> > all.
>>> >>>>> >
>>> >>>>> > So we tried to work with rule exclusions and skip the parameters
>>> we
>>> >>>> did not
>>> >>>>> > think dangerous, but here comes the bummer: ModSec 2.9 grew
>>> >>>> substantially
>>> >>>>> > slower the longer the ignore-lists of parameters became. This
>>> and a
>>> >>>> few
>>> >>>>> > very
>>> >>>>> > odd behaviors.
>>> >>>>> >
>>> >>>>> > Given the customer wanted a generic WAF without tuning of
>>> individual
>>> >>>> APIs
>>> >>>>> > we
>>> >>>>> > got to a dead end.
>>> >>>>> >
>>> >>>>> > However, if tuning was an option, then I would probably edit-CRS
>>> with
>>> >>>>> > msc_pyparser and replace the target lists with arguments I was
>>> >>>> interested
>>> >>>>> > in.
>>> >>>>> >
>>> >>>>> > https://coreruleset.org/20200901/introducing-msc_pyparser/
>>> >>>>> >
>>> >>>>> > As a complementary practice, one could think of performing
>>> allowlist
>>> >>>>> > checks on
>>> >>>>> > some / most of the JSON. Say you have a huge JSON payload with
>>> 500
>>> >>>>> > parameters.
>>> >>>>> > You examine it and discover that 300 of them actually contain
>>> simple
>>> >>>> digits
>>> >>>>> > and asciii characters and neither special chars nor escape
>>> sequences.
>>> >>>>> > So you do a regex allowlist and apply it to these 300 parameters
>>> of
>>> >>>> said
>>> >>>>> > API. And the rest you can push into CRS. Or a subset of CRS.
>>> >>>>> >
>>> >>>>> > I have not done this and the problem is if ModSec is able to
>>> handle
>>> >>>> the
>>> >>>>> > large
>>> >>>>> > target lists in a speedy manner.
>>> >>>>> >
>>> >>>>> >
>>> >>>>> > Now you can turn to a CDN or alternative WAF. I would do an
>>> extensive
>>> >>>>> > security
>>> >>>>> > tests of such a system. As I said, the JSON parser can be really
>>> >>>> fast. The
>>> >>>>> > difficult thing is to check several hundred parameters without
>>> losing
>>> >>>>> > performance.
>>> >>>>> >
>>> >>>>> > Good luck!
>>> >>>>> >
>>> >>>>> > Christian
>>> >>>>> >
>>> >>>>> >
>>> >>>>> > On Sun, Apr 25, 2021 at 08:47:06PM +0100, Henri Cook wrote:
>>> >>>>> > > Hi all,
>>> >>>>> > >
>>> >>>>> > > I'm in a situation where the only solution seems to be to drop
>>> >>>> modsec/CRS
>>> >>>>> > > and look at something like Cloudflare's WAF (and change our
>>> security
>>> >>>>> > model
>>> >>>>> > > out of necessity). I'm hoping the esteemed membership of this
>>> list
>>> >>>> might
>>> >>>>> > > have some thoughts.
>>> >>>>> > >
>>> >>>>> > > I've got about 1MB of JSON, payloads in our app might run to
>>> 20 or
>>> >>>> even
>>> >>>>> > > 30MB ultimately.
>>> >>>>> > > This 1MB of somewhat nested JSON (7 or 8 levels deep) can take
>>> 40
>>> >>>> seconds
>>> >>>>> > > to process in mod sec 3.0.4 with CRS 3.2.0
>>> >>>>> > >
>>> >>>>> > > It takes 1 second to process in our API so the WAF element is
>>> a 39x
>>> >>>> slow
>>> >>>>> > > down. I appreciate there'll be some delays in WAF.
>>> Cloudflare's WAF
>>> >>>>> > takes 5
>>> >>>>> > > seconds to scan this payload - and that's my target.
>>> >>>>> > >
>>> >>>>> > > Has anyone got any idea how to improve performance? Reading
>>> blog
>>> >>>> posts
>>> >>>>> > > about the development of cloudflare's waf I see that
>>> memoization of
>>> >>>>> > common
>>> >>>>> > > function calls was one of their absolute best performance
>>> >>>> improvements
>>> >>>>> > over
>>> >>>>> > > their modsec implementation (e.g. strlen(response_body) so
>>> it's only
>>> >>>>> > > calculated once instead of once per rule OR
>>> contains('somestring',
>>> >>>>> > > response_body)... you get the drift). Do we have anything like
>>> this
>>> >>>> in
>>> >>>>> > > modsec today? Is that already in place and my 39 seconds is
>>> after
>>> >>>> that?
>>> >>>>> > >
>>> >>>>> > > I appreciate that mod sec is fast on its own and adding complex
>>> >>>> rules can
>>> >>>>> > > be said to slow it down. With CRS being by far the most common
>>> use
>>> >>>> case
>>> >>>>> > for
>>> >>>>> > > mod sec (based on my googling) I'm surprised it's this slow,
>>> do you
>>> >>>> think
>>> >>>>> > > i've missed something?
>>> >>>>> > >
>>> >>>>> > > To note: I'm only scanning JSON payloads, typically much less
>>> than
>>> >>>> 0.5MB
>>> >>>>> > > but new, irregular ones that we need scanned in ideally <10
>>> seconds
>>> >>>> that
>>> >>>>> > > can range from 1MB-30MB
>>> >>>>> > >
>>> >>>>> > > Best regards,
>>> >>>>> > >
>>> >>>>> > > Henri Cook
>>> >>>>> >
>>> >>>>> >
>>> >>>>> > > _______________________________________________
>>> >>>>> > > mod-security-users mailing list
>>> >>>>> > > mod...@li...
>>> >>>>> > >
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> >>>>> > > Commercial ModSecurity Rules and Support from Trustwave's
>>> >>>> SpiderLabs:
>>> >>>>> > > http://www.modsecurity.org/projects/commercial/rules/
>>> >>>>> > > http://www.modsecurity.org/projects/commercial/support/
>>> >>>>> >
>>> >>>>> >
>>> >>>>> >
>>> >>>>> > _______________________________________________
>>> >>>>> > mod-security-users mailing list
>>> >>>>> > mod...@li...
>>> >>>>> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> >>>>> > Commercial ModSecurity Rules and Support from Trustwave's
>>> SpiderLabs:
>>> >>>>> > http://www.modsecurity.org/projects/commercial/rules/
>>> >>>>> > http://www.modsecurity.org/projects/commercial/support/
>>> >>>>> >
>>> >>>>
>>> >>>>
>>> >>>>> _______________________________________________
>>> >>>>> mod-security-users mailing list
>>> >>>>> mod...@li...
>>> >>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> >>>>> Commercial ModSecurity Rules and Support from Trustwave's
>>> SpiderLabs:
>>> >>>>> http://www.modsecurity.org/projects/commercial/rules/
>>> >>>>> http://www.modsecurity.org/projects/commercial/support/
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> _______________________________________________
>>> >>>> mod-security-users mailing list
>>> >>>> mod...@li...
>>> >>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> >>>> Commercial ModSecurity Rules and Support from Trustwave's
>>> SpiderLabs:
>>> >>>> http://www.modsecurity.org/projects/commercial/rules/
>>> >>>> http://www.modsecurity.org/projects/commercial/support/
>>> >>>>
>>> >>>
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > mod-security-users mailing list
>>> > mod...@li...
>>> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> > http://www.modsecurity.org/projects/commercial/rules/
>>> > http://www.modsecurity.org/projects/commercial/support/
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> mod-security-users mailing list
>>> mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> http://www.modsecurity.org/projects/commercial/rules/
>>> http://www.modsecurity.org/projects/commercial/support/
>>>
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
>>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>

Re: [mod-security-users] Performance woes - larger JSON payloads with CRS

[Srikanth A. <sri...@go...>]

Hi Henri

It defintly worked for me.
SecRequestBodyLimit 13107200
SecRequestBodyNoFileaLimit 131072

Restricts to 12.5 mb of large payload.

The only thing is if i had injections through the end, it passes through.
In your case i think you are fine with it.

And of course SecRequestBodyAccess On
Should be set

Thanks
Srikanth Arunachalam

On Mon, 10 May 2021, 14:17 Henri Cook, <he...@pr...> wrote:

> Thanks for this azurit but neither of these worked for me. I changed the
> id to 1001 and put it in my REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf - I
> still see a delay processing a large json payload - do you have any idea
> where I might have gone wrong?
>
> I think my goal is to stop the JSON request body being parsed if it's over
> 128kb in size. An advance goal would be to have it scanned as text to
> provide *some* level of protection - I think this is what cloudflare does
> (from a very cursory glance)
>
> Best Regards,
>
> Henri
>
> On Thu, 29 Apr 2021 at 19:02, <az...@po...> wrote:
>
>> I just noticed you are using CRS, so this one would be probably better:
>>
>>
>> SecRule REQUEST_BODY_LENGTH "@gt 131072" \
>>      "id:9999999,\
>>      phase:2,\
>>      pass,\
>>      t:none,\
>>      nolog,\
>>      ctl:ruleRemoveTargetByTag=OWASP_CRS;REQUEST_BODY"
>>
>>
>>
>>
>>
>> Citát az...@po...:
>>
>> > Hi,
>> >
>> > you can do this using exclusive rule, something like this (set the
>> > rule ID to something 'better'), length is in bytes:
>> >
>> >
>> > SecRule REQUEST_BODY_LENGTH "@gt 131072" \
>> >     "id:9999999,\
>> >     phase:2,\
>> >     pass,\
>> >     t:none,\
>> >     nolog,\
>> >     ctl:ruleEngine=Off"
>> >
>> >
>> >
>> > REQUEST_BODY_LENGTH is available from phase 2 so you cannot put this
>> > into phase 1.
>> >
>> >
>> >
>> > Citát Henri Cook <he...@pr...>:
>> >
>> >> I think what I need here is a way to exempt the request body from any
>> >> scanning when it's over 128kb (my chosen upper limit)...
>> >>
>> >> I'll probably try implementing this as a phase 1 rule on Content-Length
>> >> unless anyone shouts there's a better way. `ProcessPartial` sounded
>> like
>> >> the holy grail, but you can't partial process JSON (because it doesn't
>> >> parse!).
>> >>
>> >> It'd be cool to be able to fallback to a fulltext scan of the partial
>> JSON
>> >> if it's >128kb, I might have a try at looking into that also (again any
>> >> guidance appreciated)
>> >>
>> >> Best Regards,
>> >>
>> >> Henri
>> >>
>> >> On Mon, 26 Apr 2021 at 15:24, Henri Cook <he...@pr...> wrote:
>> >>
>> >>> The follow up problem to this is:
>> >>>
>> >>> Now i'm set to `SecRequestBodyLimit` 31457280 and
>> >>> `SecRequestBodyNoFilesLimit` 65536 and `SecRequestBodyLimitAction
>> >>> PartialProcess`
>> >>>
>> >>> In my mind this will process the first part of a request if it can,
>> and
>> >>> ignore the rest. But:
>> >>>
>> >>> - Rule 200002 is triggered, which is saying the JSON can't be parsed.
>> >>> Presumably because in a large request it tries to process the
>> beginning of
>> >>> the JSON and can't (because it won't parse, because the JSON is cut
>> off so
>> >>> doesn't end)
>> >>>
>> >>> So I think I need to find a way to skip JSON parsing entirely when the
>> >>> payload is over 64kb (65536)? Does that sound right? Assuming 64kb is
>> the
>> >>> limit I want to stick with. I hadn't really considered before this
>> point
>> >>> that 'partial processing' of JSON was likely to be hairy but of
>> course it
>> >>> makes sense.
>> >>>
>> >>>
>> >>>
>> >>> On Mon, 26 Apr 2021 at 07:17, Christian Folini <
>> >>> chr...@ne...> wrote:
>> >>>
>> >>>> Hey Henri,
>> >>>>
>> >>>> From a security practice, this is obviously lacking, but in wider
>> >>>> perspective,
>> >>>> I see it meet "industry standard", yes.
>> >>>>
>> >>>> When I teach, I tell my student, that the worst WAF is the one that
>> is
>> >>>> switched off. So if you need to compromise and you can only apply
>> 20% of
>> >>>> the rules because you run the risk of business demanding it's
>> switched
>> >>>> off,
>> >>>> then that 20% WAF is still better than no WAF.
>> >>>>
>> >>>> Cheers,
>> >>>>
>> >>>> Christian
>> >>>>
>> >>>>
>> >>>> On Mon, Apr 26, 2021 at 06:57:54AM +0100, Henri Cook wrote:
>> >>>>> Thanks Christian, taking this in combination with Osama's point
>> earlier
>> >>>> in
>> >>>>> the thread that most 'big four' (AWS/GCP/Cloudflare/Azure) WAFs
>> seem to
>> >>>>> limit the payload they'll scan. From my reading to 128kb
>> >>>> (cloudflare+azure)
>> >>>>> or 8kb (aws+gcp) I think I'll be able to resolve our particular
>> issue.
>> >>>>>
>> >>>>> I believe my modern application is already very robust in terms of
>> >>>> defence
>> >>>>> against sql injection as well as other OWASP top 10 attack vectors
>> and
>> >>>> that
>> >>>>> a WAF primarily adds reassurance (for the business and clients who
>> ask
>> >>>> if I
>> >>>>> have one) and minor frustration (for any potential attacker) layer.
>> The
>> >>>>> spec is to add a WAF that meets (but notably does not necessarily
>> have
>> >>>> to
>> >>>>> exceed) industry standards. I believe this means that I can switch
>> >>>> modsec
>> >>>>> to 128kb or 8kb partial parsing ('SecResponseBodyLimitAction
>> >>>>> ProcessPartial' - allowing through unscanned any payloads over those
>> >>>> sizes)
>> >>>>> and be able to say I've got scan-size-policy-parity with an AWS or a
>> >>>>> Cloudflare which means it is "industry standard".
>> >>>>>
>> >>>>> Please let me know if you think that's mad and thanks again
>> >>>>>
>> >>>>> Best Regards,
>> >>>>>
>> >>>>> Henri
>> >>>>>
>> >>>>> On Sun, 25 Apr 2021 at 21:39, Christian Folini <
>> >>>> chr...@ne...>
>> >>>>> wrote:
>> >>>>>
>> >>>>> > Hey Henri,
>> >>>>> >
>> >>>>> > You are in a bad situation and as far as I can see you are right,
>> you
>> >>>> might
>> >>>>> > have to drop modsec/CRS in this situation.
>> >>>>> >
>> >>>>> > I've had a customer with a similar problem and we did a deep dive
>> >>>>> > investigation and I had to strike colors in the end.
>> >>>>> >
>> >>>>> > The point is not the JSON parser. That has shown to be really
>> fast.
>> >>>> The
>> >>>>> > point
>> >>>>> > is several hundred variables that go into CRS afterwards. If you
>> run
>> >>>> CRS
>> >>>>> > on a
>> >>>>> > standard web application you get forms with a few parameters and
>> >>>> that's
>> >>>>> > easy.
>> >>>>> > But several megabytes of JSON means hundreds of arguments and CRS
>> >>>> parses
>> >>>>> > them
>> >>>>> > all.
>> >>>>> >
>> >>>>> > So we tried to work with rule exclusions and skip the parameters
>> we
>> >>>> did not
>> >>>>> > think dangerous, but here comes the bummer: ModSec 2.9 grew
>> >>>> substantially
>> >>>>> > slower the longer the ignore-lists of parameters became. This and
>> a
>> >>>> few
>> >>>>> > very
>> >>>>> > odd behaviors.
>> >>>>> >
>> >>>>> > Given the customer wanted a generic WAF without tuning of
>> individual
>> >>>> APIs
>> >>>>> > we
>> >>>>> > got to a dead end.
>> >>>>> >
>> >>>>> > However, if tuning was an option, then I would probably edit-CRS
>> with
>> >>>>> > msc_pyparser and replace the target lists with arguments I was
>> >>>> interested
>> >>>>> > in.
>> >>>>> >
>> >>>>> > https://coreruleset.org/20200901/introducing-msc_pyparser/
>> >>>>> >
>> >>>>> > As a complementary practice, one could think of performing
>> allowlist
>> >>>>> > checks on
>> >>>>> > some / most of the JSON. Say you have a huge JSON payload with 500
>> >>>>> > parameters.
>> >>>>> > You examine it and discover that 300 of them actually contain
>> simple
>> >>>> digits
>> >>>>> > and asciii characters and neither special chars nor escape
>> sequences.
>> >>>>> > So you do a regex allowlist and apply it to these 300 parameters
>> of
>> >>>> said
>> >>>>> > API. And the rest you can push into CRS. Or a subset of CRS.
>> >>>>> >
>> >>>>> > I have not done this and the problem is if ModSec is able to
>> handle
>> >>>> the
>> >>>>> > large
>> >>>>> > target lists in a speedy manner.
>> >>>>> >
>> >>>>> >
>> >>>>> > Now you can turn to a CDN or alternative WAF. I would do an
>> extensive
>> >>>>> > security
>> >>>>> > tests of such a system. As I said, the JSON parser can be really
>> >>>> fast. The
>> >>>>> > difficult thing is to check several hundred parameters without
>> losing
>> >>>>> > performance.
>> >>>>> >
>> >>>>> > Good luck!
>> >>>>> >
>> >>>>> > Christian
>> >>>>> >
>> >>>>> >
>> >>>>> > On Sun, Apr 25, 2021 at 08:47:06PM +0100, Henri Cook wrote:
>> >>>>> > > Hi all,
>> >>>>> > >
>> >>>>> > > I'm in a situation where the only solution seems to be to drop
>> >>>> modsec/CRS
>> >>>>> > > and look at something like Cloudflare's WAF (and change our
>> security
>> >>>>> > model
>> >>>>> > > out of necessity). I'm hoping the esteemed membership of this
>> list
>> >>>> might
>> >>>>> > > have some thoughts.
>> >>>>> > >
>> >>>>> > > I've got about 1MB of JSON, payloads in our app might run to 20
>> or
>> >>>> even
>> >>>>> > > 30MB ultimately.
>> >>>>> > > This 1MB of somewhat nested JSON (7 or 8 levels deep) can take
>> 40
>> >>>> seconds
>> >>>>> > > to process in mod sec 3.0.4 with CRS 3.2.0
>> >>>>> > >
>> >>>>> > > It takes 1 second to process in our API so the WAF element is a
>> 39x
>> >>>> slow
>> >>>>> > > down. I appreciate there'll be some delays in WAF. Cloudflare's
>> WAF
>> >>>>> > takes 5
>> >>>>> > > seconds to scan this payload - and that's my target.
>> >>>>> > >
>> >>>>> > > Has anyone got any idea how to improve performance? Reading blog
>> >>>> posts
>> >>>>> > > about the development of cloudflare's waf I see that
>> memoization of
>> >>>>> > common
>> >>>>> > > function calls was one of their absolute best performance
>> >>>> improvements
>> >>>>> > over
>> >>>>> > > their modsec implementation (e.g. strlen(response_body) so it's
>> only
>> >>>>> > > calculated once instead of once per rule OR
>> contains('somestring',
>> >>>>> > > response_body)... you get the drift). Do we have anything like
>> this
>> >>>> in
>> >>>>> > > modsec today? Is that already in place and my 39 seconds is
>> after
>> >>>> that?
>> >>>>> > >
>> >>>>> > > I appreciate that mod sec is fast on its own and adding complex
>> >>>> rules can
>> >>>>> > > be said to slow it down. With CRS being by far the most common
>> use
>> >>>> case
>> >>>>> > for
>> >>>>> > > mod sec (based on my googling) I'm surprised it's this slow, do
>> you
>> >>>> think
>> >>>>> > > i've missed something?
>> >>>>> > >
>> >>>>> > > To note: I'm only scanning JSON payloads, typically much less
>> than
>> >>>> 0.5MB
>> >>>>> > > but new, irregular ones that we need scanned in ideally <10
>> seconds
>> >>>> that
>> >>>>> > > can range from 1MB-30MB
>> >>>>> > >
>> >>>>> > > Best regards,
>> >>>>> > >
>> >>>>> > > Henri Cook
>> >>>>> >
>> >>>>> >
>> >>>>> > > _______________________________________________
>> >>>>> > > mod-security-users mailing list
>> >>>>> > > mod...@li...
>> >>>>> > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> >>>>> > > Commercial ModSecurity Rules and Support from Trustwave's
>> >>>> SpiderLabs:
>> >>>>> > > http://www.modsecurity.org/projects/commercial/rules/
>> >>>>> > > http://www.modsecurity.org/projects/commercial/support/
>> >>>>> >
>> >>>>> >
>> >>>>> >
>> >>>>> > _______________________________________________
>> >>>>> > mod-security-users mailing list
>> >>>>> > mod...@li...
>> >>>>> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> >>>>> > Commercial ModSecurity Rules and Support from Trustwave's
>> SpiderLabs:
>> >>>>> > http://www.modsecurity.org/projects/commercial/rules/
>> >>>>> > http://www.modsecurity.org/projects/commercial/support/
>> >>>>> >
>> >>>>
>> >>>>
>> >>>>> _______________________________________________
>> >>>>> mod-security-users mailing list
>> >>>>> mod...@li...
>> >>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> >>>>> Commercial ModSecurity Rules and Support from Trustwave's
>> SpiderLabs:
>> >>>>> http://www.modsecurity.org/projects/commercial/rules/
>> >>>>> http://www.modsecurity.org/projects/commercial/support/
>> >>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> mod-security-users mailing list
>> >>>> mod...@li...
>> >>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> >>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> >>>> http://www.modsecurity.org/projects/commercial/rules/
>> >>>> http://www.modsecurity.org/projects/commercial/support/
>> >>>>
>> >>>
>> >
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > mod-security-users mailing list
>> > mod...@li...
>> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> > http://www.modsecurity.org/projects/commercial/rules/
>> > http://www.modsecurity.org/projects/commercial/support/
>>
>>
>>
>>
>>
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
>>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>