Re: [mod-security-users] Performance woes - larger JSON payloads with CRS
Brought to you by:
victorhora,
zimmerletw
Menu
▾
▴
Re: [mod-security-users] Performance woes - larger JSON payloads with CRS
|
From: Henri C. <he...@pr...> - 2021-05-10 13:38:41
|
Thanks Srikanth. In my case I want to allow large payloads through
unscanned (or partially scanned) rather than rejecting them outright. My
issue is that scanning my large JSON payloads takes some time. Since
they're rare I'm willing to not scan them in the name of balancing security
and performance, but I'd still rather scan payloads that are <128k in size
On Mon, 10 May 2021 at 14:30, Srikanth Arunachalam via mod-security-users <
mod...@li...> wrote:
> Hi Henri
>
> It defintly worked for me.
> SecRequestBodyLimit 13107200
> SecRequestBodyNoFileaLimit 131072
>
> Restricts to 12.5 mb of large payload.
>
> The only thing is if i had injections through the end, it passes through.
> In your case i think you are fine with it.
>
> And of course SecRequestBodyAccess On
> Should be set
>
> Thanks
> Srikanth Arunachalam
>
> On Mon, 10 May 2021, 14:17 Henri Cook, <henri@proteus.tech> wrote:
>
>> Thanks for this azurit but neither of these worked for me. I changed the
>> id to 1001 and put it in my REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf - I
>> still see a delay processing a large json payload - do you have any idea
>> where I might have gone wrong?
>>
>> I think my goal is to stop the JSON request body being parsed if it's
>> over 128kb in size. An advance goal would be to have it scanned as text to
>> provide *some* level of protection - I think this is what cloudflare
>> does (from a very cursory glance)
>>
>> Best Regards,
>>
>> Henri
>>
>> On Thu, 29 Apr 2021 at 19:02, <az...@po...> wrote:
>>
>>> I just noticed you are using CRS, so this one would be probably better:
>>>
>>>
>>> SecRule REQUEST_BODY_LENGTH "@gt 131072" \
>>> "id:9999999,\
>>> phase:2,\
>>> pass,\
>>> t:none,\
>>> nolog,\
>>> ctl:ruleRemoveTargetByTag=OWASP_CRS;REQUEST_BODY"
>>>
>>>
>>>
>>>
>>>
>>> Citát az...@po...:
>>>
>>> > Hi,
>>> >
>>> > you can do this using exclusive rule, something like this (set the
>>> > rule ID to something 'better'), length is in bytes:
>>> >
>>> >
>>> > SecRule REQUEST_BODY_LENGTH "@gt 131072" \
>>> > "id:9999999,\
>>> > phase:2,\
>>> > pass,\
>>> > t:none,\
>>> > nolog,\
>>> > ctl:ruleEngine=Off"
>>> >
>>> >
>>> >
>>> > REQUEST_BODY_LENGTH is available from phase 2 so you cannot put this
>>> > into phase 1.
>>> >
>>> >
>>> >
>>> > Citát Henri Cook <henri@proteus.tech>:
>>> >
>>> >> I think what I need here is a way to exempt the request body from any
>>> >> scanning when it's over 128kb (my chosen upper limit)...
>>> >>
>>> >> I'll probably try implementing this as a phase 1 rule on
>>> Content-Length
>>> >> unless anyone shouts there's a better way. `ProcessPartial` sounded
>>> like
>>> >> the holy grail, but you can't partial process JSON (because it doesn't
>>> >> parse!).
>>> >>
>>> >> It'd be cool to be able to fallback to a fulltext scan of the partial
>>> JSON
>>> >> if it's >128kb, I might have a try at looking into that also (again
>>> any
>>> >> guidance appreciated)
>>> >>
>>> >> Best Regards,
>>> >>
>>> >> Henri
>>> >>
>>> >> On Mon, 26 Apr 2021 at 15:24, Henri Cook <henri@proteus.tech> wrote:
>>> >>
>>> >>> The follow up problem to this is:
>>> >>>
>>> >>> Now i'm set to `SecRequestBodyLimit` 31457280 and
>>> >>> `SecRequestBodyNoFilesLimit` 65536 and `SecRequestBodyLimitAction
>>> >>> PartialProcess`
>>> >>>
>>> >>> In my mind this will process the first part of a request if it can,
>>> and
>>> >>> ignore the rest. But:
>>> >>>
>>> >>> - Rule 200002 is triggered, which is saying the JSON can't be parsed.
>>> >>> Presumably because in a large request it tries to process the
>>> beginning of
>>> >>> the JSON and can't (because it won't parse, because the JSON is cut
>>> off so
>>> >>> doesn't end)
>>> >>>
>>> >>> So I think I need to find a way to skip JSON parsing entirely when
>>> the
>>> >>> payload is over 64kb (65536)? Does that sound right? Assuming 64kb
>>> is the
>>> >>> limit I want to stick with. I hadn't really considered before this
>>> point
>>> >>> that 'partial processing' of JSON was likely to be hairy but of
>>> course it
>>> >>> makes sense.
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Mon, 26 Apr 2021 at 07:17, Christian Folini <
>>> >>> chr...@ne...> wrote:
>>> >>>
>>> >>>> Hey Henri,
>>> >>>>
>>> >>>> From a security practice, this is obviously lacking, but in wider
>>> >>>> perspective,
>>> >>>> I see it meet "industry standard", yes.
>>> >>>>
>>> >>>> When I teach, I tell my student, that the worst WAF is the one that
>>> is
>>> >>>> switched off. So if you need to compromise and you can only apply
>>> 20% of
>>> >>>> the rules because you run the risk of business demanding it's
>>> switched
>>> >>>> off,
>>> >>>> then that 20% WAF is still better than no WAF.
>>> >>>>
>>> >>>> Cheers,
>>> >>>>
>>> >>>> Christian
>>> >>>>
>>> >>>>
>>> >>>> On Mon, Apr 26, 2021 at 06:57:54AM +0100, Henri Cook wrote:
>>> >>>>> Thanks Christian, taking this in combination with Osama's point
>>> earlier
>>> >>>> in
>>> >>>>> the thread that most 'big four' (AWS/GCP/Cloudflare/Azure) WAFs
>>> seem to
>>> >>>>> limit the payload they'll scan. From my reading to 128kb
>>> >>>> (cloudflare+azure)
>>> >>>>> or 8kb (aws+gcp) I think I'll be able to resolve our particular
>>> issue.
>>> >>>>>
>>> >>>>> I believe my modern application is already very robust in terms of
>>> >>>> defence
>>> >>>>> against sql injection as well as other OWASP top 10 attack vectors
>>> and
>>> >>>> that
>>> >>>>> a WAF primarily adds reassurance (for the business and clients who
>>> ask
>>> >>>> if I
>>> >>>>> have one) and minor frustration (for any potential attacker)
>>> layer. The
>>> >>>>> spec is to add a WAF that meets (but notably does not necessarily
>>> have
>>> >>>> to
>>> >>>>> exceed) industry standards. I believe this means that I can switch
>>> >>>> modsec
>>> >>>>> to 128kb or 8kb partial parsing ('SecResponseBodyLimitAction
>>> >>>>> ProcessPartial' - allowing through unscanned any payloads over
>>> those
>>> >>>> sizes)
>>> >>>>> and be able to say I've got scan-size-policy-parity with an AWS or
>>> a
>>> >>>>> Cloudflare which means it is "industry standard".
>>> >>>>>
>>> >>>>> Please let me know if you think that's mad and thanks again
>>> >>>>>
>>> >>>>> Best Regards,
>>> >>>>>
>>> >>>>> Henri
>>> >>>>>
>>> >>>>> On Sun, 25 Apr 2021 at 21:39, Christian Folini <
>>> >>>> chr...@ne...>
>>> >>>>> wrote:
>>> >>>>>
>>> >>>>> > Hey Henri,
>>> >>>>> >
>>> >>>>> > You are in a bad situation and as far as I can see you are
>>> right, you
>>> >>>> might
>>> >>>>> > have to drop modsec/CRS in this situation.
>>> >>>>> >
>>> >>>>> > I've had a customer with a similar problem and we did a deep dive
>>> >>>>> > investigation and I had to strike colors in the end.
>>> >>>>> >
>>> >>>>> > The point is not the JSON parser. That has shown to be really
>>> fast.
>>> >>>> The
>>> >>>>> > point
>>> >>>>> > is several hundred variables that go into CRS afterwards. If you
>>> run
>>> >>>> CRS
>>> >>>>> > on a
>>> >>>>> > standard web application you get forms with a few parameters and
>>> >>>> that's
>>> >>>>> > easy.
>>> >>>>> > But several megabytes of JSON means hundreds of arguments and CRS
>>> >>>> parses
>>> >>>>> > them
>>> >>>>> > all.
>>> >>>>> >
>>> >>>>> > So we tried to work with rule exclusions and skip the parameters
>>> we
>>> >>>> did not
>>> >>>>> > think dangerous, but here comes the bummer: ModSec 2.9 grew
>>> >>>> substantially
>>> >>>>> > slower the longer the ignore-lists of parameters became. This
>>> and a
>>> >>>> few
>>> >>>>> > very
>>> >>>>> > odd behaviors.
>>> >>>>> >
>>> >>>>> > Given the customer wanted a generic WAF without tuning of
>>> individual
>>> >>>> APIs
>>> >>>>> > we
>>> >>>>> > got to a dead end.
>>> >>>>> >
>>> >>>>> > However, if tuning was an option, then I would probably edit-CRS
>>> with
>>> >>>>> > msc_pyparser and replace the target lists with arguments I was
>>> >>>> interested
>>> >>>>> > in.
>>> >>>>> >
>>> >>>>> > https://coreruleset.org/20200901/introducing-msc_pyparser/
>>> >>>>> >
>>> >>>>> > As a complementary practice, one could think of performing
>>> allowlist
>>> >>>>> > checks on
>>> >>>>> > some / most of the JSON. Say you have a huge JSON payload with
>>> 500
>>> >>>>> > parameters.
>>> >>>>> > You examine it and discover that 300 of them actually contain
>>> simple
>>> >>>> digits
>>> >>>>> > and asciii characters and neither special chars nor escape
>>> sequences.
>>> >>>>> > So you do a regex allowlist and apply it to these 300 parameters
>>> of
>>> >>>> said
>>> >>>>> > API. And the rest you can push into CRS. Or a subset of CRS.
>>> >>>>> >
>>> >>>>> > I have not done this and the problem is if ModSec is able to
>>> handle
>>> >>>> the
>>> >>>>> > large
>>> >>>>> > target lists in a speedy manner.
>>> >>>>> >
>>> >>>>> >
>>> >>>>> > Now you can turn to a CDN or alternative WAF. I would do an
>>> extensive
>>> >>>>> > security
>>> >>>>> > tests of such a system. As I said, the JSON parser can be really
>>> >>>> fast. The
>>> >>>>> > difficult thing is to check several hundred parameters without
>>> losing
>>> >>>>> > performance.
>>> >>>>> >
>>> >>>>> > Good luck!
>>> >>>>> >
>>> >>>>> > Christian
>>> >>>>> >
>>> >>>>> >
>>> >>>>> > On Sun, Apr 25, 2021 at 08:47:06PM +0100, Henri Cook wrote:
>>> >>>>> > > Hi all,
>>> >>>>> > >
>>> >>>>> > > I'm in a situation where the only solution seems to be to drop
>>> >>>> modsec/CRS
>>> >>>>> > > and look at something like Cloudflare's WAF (and change our
>>> security
>>> >>>>> > model
>>> >>>>> > > out of necessity). I'm hoping the esteemed membership of this
>>> list
>>> >>>> might
>>> >>>>> > > have some thoughts.
>>> >>>>> > >
>>> >>>>> > > I've got about 1MB of JSON, payloads in our app might run to
>>> 20 or
>>> >>>> even
>>> >>>>> > > 30MB ultimately.
>>> >>>>> > > This 1MB of somewhat nested JSON (7 or 8 levels deep) can take
>>> 40
>>> >>>> seconds
>>> >>>>> > > to process in mod sec 3.0.4 with CRS 3.2.0
>>> >>>>> > >
>>> >>>>> > > It takes 1 second to process in our API so the WAF element is
>>> a 39x
>>> >>>> slow
>>> >>>>> > > down. I appreciate there'll be some delays in WAF.
>>> Cloudflare's WAF
>>> >>>>> > takes 5
>>> >>>>> > > seconds to scan this payload - and that's my target.
>>> >>>>> > >
>>> >>>>> > > Has anyone got any idea how to improve performance? Reading
>>> blog
>>> >>>> posts
>>> >>>>> > > about the development of cloudflare's waf I see that
>>> memoization of
>>> >>>>> > common
>>> >>>>> > > function calls was one of their absolute best performance
>>> >>>> improvements
>>> >>>>> > over
>>> >>>>> > > their modsec implementation (e.g. strlen(response_body) so
>>> it's only
>>> >>>>> > > calculated once instead of once per rule OR
>>> contains('somestring',
>>> >>>>> > > response_body)... you get the drift). Do we have anything like
>>> this
>>> >>>> in
>>> >>>>> > > modsec today? Is that already in place and my 39 seconds is
>>> after
>>> >>>> that?
>>> >>>>> > >
>>> >>>>> > > I appreciate that mod sec is fast on its own and adding complex
>>> >>>> rules can
>>> >>>>> > > be said to slow it down. With CRS being by far the most common
>>> use
>>> >>>> case
>>> >>>>> > for
>>> >>>>> > > mod sec (based on my googling) I'm surprised it's this slow,
>>> do you
>>> >>>> think
>>> >>>>> > > i've missed something?
>>> >>>>> > >
>>> >>>>> > > To note: I'm only scanning JSON payloads, typically much less
>>> than
>>> >>>> 0.5MB
>>> >>>>> > > but new, irregular ones that we need scanned in ideally <10
>>> seconds
>>> >>>> that
>>> >>>>> > > can range from 1MB-30MB
>>> >>>>> > >
>>> >>>>> > > Best regards,
>>> >>>>> > >
>>> >>>>> > > Henri Cook
>>> >>>>> >
>>> >>>>> >
>>> >>>>> > > _______________________________________________
>>> >>>>> > > mod-security-users mailing list
>>> >>>>> > > mod...@li...
>>> >>>>> > >
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> >>>>> > > Commercial ModSecurity Rules and Support from Trustwave's
>>> >>>> SpiderLabs:
>>> >>>>> > > http://www.modsecurity.org/projects/commercial/rules/
>>> >>>>> > > http://www.modsecurity.org/projects/commercial/support/
>>> >>>>> >
>>> >>>>> >
>>> >>>>> >
>>> >>>>> > _______________________________________________
>>> >>>>> > mod-security-users mailing list
>>> >>>>> > mod...@li...
>>> >>>>> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> >>>>> > Commercial ModSecurity Rules and Support from Trustwave's
>>> SpiderLabs:
>>> >>>>> > http://www.modsecurity.org/projects/commercial/rules/
>>> >>>>> > http://www.modsecurity.org/projects/commercial/support/
>>> >>>>> >
>>> >>>>
>>> >>>>
>>> >>>>> _______________________________________________
>>> >>>>> mod-security-users mailing list
>>> >>>>> mod...@li...
>>> >>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> >>>>> Commercial ModSecurity Rules and Support from Trustwave's
>>> SpiderLabs:
>>> >>>>> http://www.modsecurity.org/projects/commercial/rules/
>>> >>>>> http://www.modsecurity.org/projects/commercial/support/
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> _______________________________________________
>>> >>>> mod-security-users mailing list
>>> >>>> mod...@li...
>>> >>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> >>>> Commercial ModSecurity Rules and Support from Trustwave's
>>> SpiderLabs:
>>> >>>> http://www.modsecurity.org/projects/commercial/rules/
>>> >>>> http://www.modsecurity.org/projects/commercial/support/
>>> >>>>
>>> >>>
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > mod-security-users mailing list
>>> > mod...@li...
>>> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> > http://www.modsecurity.org/projects/commercial/rules/
>>> > http://www.modsecurity.org/projects/commercial/support/
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> mod-security-users mailing list
>>> mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> http://www.modsecurity.org/projects/commercial/rules/
>>> http://www.modsecurity.org/projects/commercial/support/
>>>
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
>>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>
|
