Re: [mod-security-users] Performance woes - larger JSON payloads with CRS
Brought to you by:
victorhora,
zimmerletw
Menu
▾
▴
Re: [mod-security-users] Performance woes - larger JSON payloads with CRS
|
From: <az...@po...> - 2021-05-10 14:32:45
|
Another, maybe better, solution, as you are already going to modify
modsecurity.conf, would be to modify rule 200001 directly. Try this:
SecRule REQUEST_HEADERS:Content-Type "application/json" \
"id:'200001',phase:1,t:none,t:lowercase,pass,nolog,chain"
SecRule REQUEST_HEADERS:Content-Length "@lt 131072" \
"t:none,ctl:requestBodyProcessor=JSON"
Citát az...@po...:
> Sorry, now I see the problem: Rule which is doing JSON processing
> (200001) is running in phase 1 and my rule below is running in phase
> 2. Problem is, as i stated below, REQUEST_BODY_LENGTH is available
> from phase 2 so my rule cannot be run in phase 1. Also, i noticed
> another problem - we cannot do 'ruleRemoveTargetByTag=OWASP_CRS' as
> rule 200001 is not part of the CRS and it's running before CRS (and
> also before rules in file
> REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf).
>
> In this case, we can use Content-Length header, which won't be so
> realibe but should be sufficient. Try this:
>
> SecRule REQUEST_HEADERS:Content-Type "application/json" \
> "id:9999999,\
> phase:1,\
> pass,\
> t:none,\
> nolog,\
> chain"
> SecRule REQUEST_HEADERS:Content-Length "@gt 131072" \
> "t:none,\
> ctl:ruleRemoveTargetById=200001;REQUEST_BODY"
>
>
> This rule should be run before rule 200001 so you will, probably,
> need to put it inside modsecurity.conf before rule 200001 (but i
> didn't tested it at all).
>
>
>
>
>
> Citát Henri Cook <henri@proteus.tech>:
>
>> Thanks for this azurit but neither of these worked for me. I changed the id
>> to 1001 and put it in my REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf - I
>> still see a delay processing a large json payload - do you have any idea
>> where I might have gone wrong?
>>
>> I think my goal is to stop the JSON request body being parsed if it's over
>> 128kb in size. An advance goal would be to have it scanned as text to
>> provide *some* level of protection - I think this is what cloudflare does
>> (from a very cursory glance)
>>
>> Best Regards,
>>
>> Henri
>>
>> On Thu, 29 Apr 2021 at 19:02, <az...@po...> wrote:
>>
>>> I just noticed you are using CRS, so this one would be probably better:
>>>
>>>
>>> SecRule REQUEST_BODY_LENGTH "@gt 131072" \
>>> "id:9999999,\
>>> phase:2,\
>>> pass,\
>>> t:none,\
>>> nolog,\
>>> ctl:ruleRemoveTargetByTag=OWASP_CRS;REQUEST_BODY"
>>>
>>>
>>>
>>>
>>>
>>> Citát az...@po...:
>>>
>>>> Hi,
>>>>
>>>> you can do this using exclusive rule, something like this (set the
>>>> rule ID to something 'better'), length is in bytes:
>>>>
>>>>
>>>> SecRule REQUEST_BODY_LENGTH "@gt 131072" \
>>>> "id:9999999,\
>>>> phase:2,\
>>>> pass,\
>>>> t:none,\
>>>> nolog,\
>>>> ctl:ruleEngine=Off"
>>>>
>>>>
>>>>
>>>> REQUEST_BODY_LENGTH is available from phase 2 so you cannot put this
>>>> into phase 1.
>>>>
>>>>
>>>>
>>>> Citát Henri Cook <henri@proteus.tech>:
>>>>
>>>>> I think what I need here is a way to exempt the request body from any
>>>>> scanning when it's over 128kb (my chosen upper limit)...
>>>>>
>>>>> I'll probably try implementing this as a phase 1 rule on Content-Length
>>>>> unless anyone shouts there's a better way. `ProcessPartial` sounded like
>>>>> the holy grail, but you can't partial process JSON (because it doesn't
>>>>> parse!).
>>>>>
>>>>> It'd be cool to be able to fallback to a fulltext scan of the partial
>>> JSON
>>>>> if it's >128kb, I might have a try at looking into that also (again any
>>>>> guidance appreciated)
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Henri
>>>>>
>>>>> On Mon, 26 Apr 2021 at 15:24, Henri Cook <henri@proteus.tech> wrote:
>>>>>
>>>>>> The follow up problem to this is:
>>>>>>
>>>>>> Now i'm set to `SecRequestBodyLimit` 31457280 and
>>>>>> `SecRequestBodyNoFilesLimit` 65536 and `SecRequestBodyLimitAction
>>>>>> PartialProcess`
>>>>>>
>>>>>> In my mind this will process the first part of a request if it can, and
>>>>>> ignore the rest. But:
>>>>>>
>>>>>> - Rule 200002 is triggered, which is saying the JSON can't be parsed.
>>>>>> Presumably because in a large request it tries to process the
>>> beginning of
>>>>>> the JSON and can't (because it won't parse, because the JSON is cut
>>> off so
>>>>>> doesn't end)
>>>>>>
>>>>>> So I think I need to find a way to skip JSON parsing entirely when the
>>>>>> payload is over 64kb (65536)? Does that sound right? Assuming 64kb is
>>> the
>>>>>> limit I want to stick with. I hadn't really considered before this
>>> point
>>>>>> that 'partial processing' of JSON was likely to be hairy but of course
>>> it
>>>>>> makes sense.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, 26 Apr 2021 at 07:17, Christian Folini <
>>>>>> chr...@ne...> wrote:
>>>>>>
>>>>>>> Hey Henri,
>>>>>>>
>>>>>>> From a security practice, this is obviously lacking, but in wider
>>>>>>> perspective,
>>>>>>> I see it meet "industry standard", yes.
>>>>>>>
>>>>>>> When I teach, I tell my student, that the worst WAF is the one that is
>>>>>>> switched off. So if you need to compromise and you can only apply 20%
>>> of
>>>>>>> the rules because you run the risk of business demanding it's switched
>>>>>>> off,
>>>>>>> then that 20% WAF is still better than no WAF.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Christian
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Apr 26, 2021 at 06:57:54AM +0100, Henri Cook wrote:
>>>>>>>> Thanks Christian, taking this in combination with Osama's point
>>> earlier
>>>>>>> in
>>>>>>>> the thread that most 'big four' (AWS/GCP/Cloudflare/Azure) WAFs seem
>>> to
>>>>>>>> limit the payload they'll scan. From my reading to 128kb
>>>>>>> (cloudflare+azure)
>>>>>>>> or 8kb (aws+gcp) I think I'll be able to resolve our particular
>>> issue.
>>>>>>>>
>>>>>>>> I believe my modern application is already very robust in terms of
>>>>>>> defence
>>>>>>>> against sql injection as well as other OWASP top 10 attack vectors
>>> and
>>>>>>> that
>>>>>>>> a WAF primarily adds reassurance (for the business and clients who
>>> ask
>>>>>>> if I
>>>>>>>> have one) and minor frustration (for any potential attacker) layer.
>>> The
>>>>>>>> spec is to add a WAF that meets (but notably does not necessarily
>>> have
>>>>>>> to
>>>>>>>> exceed) industry standards. I believe this means that I can switch
>>>>>>> modsec
>>>>>>>> to 128kb or 8kb partial parsing ('SecResponseBodyLimitAction
>>>>>>>> ProcessPartial' - allowing through unscanned any payloads over those
>>>>>>> sizes)
>>>>>>>> and be able to say I've got scan-size-policy-parity with an AWS or a
>>>>>>>> Cloudflare which means it is "industry standard".
>>>>>>>>
>>>>>>>> Please let me know if you think that's mad and thanks again
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>>
>>>>>>>> Henri
>>>>>>>>
>>>>>>>> On Sun, 25 Apr 2021 at 21:39, Christian Folini <
>>>>>>> chr...@ne...>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> > Hey Henri,
>>>>>>>> >
>>>>>>>> > You are in a bad situation and as far as I can see you are right,
>>> you
>>>>>>> might
>>>>>>>> > have to drop modsec/CRS in this situation.
>>>>>>>> >
>>>>>>>> > I've had a customer with a similar problem and we did a deep dive
>>>>>>>> > investigation and I had to strike colors in the end.
>>>>>>>> >
>>>>>>>> > The point is not the JSON parser. That has shown to be really fast.
>>>>>>> The
>>>>>>>> > point
>>>>>>>> > is several hundred variables that go into CRS afterwards. If you
>>> run
>>>>>>> CRS
>>>>>>>> > on a
>>>>>>>> > standard web application you get forms with a few parameters and
>>>>>>> that's
>>>>>>>> > easy.
>>>>>>>> > But several megabytes of JSON means hundreds of arguments and CRS
>>>>>>> parses
>>>>>>>> > them
>>>>>>>> > all.
>>>>>>>> >
>>>>>>>> > So we tried to work with rule exclusions and skip the parameters we
>>>>>>> did not
>>>>>>>> > think dangerous, but here comes the bummer: ModSec 2.9 grew
>>>>>>> substantially
>>>>>>>> > slower the longer the ignore-lists of parameters became. This and a
>>>>>>> few
>>>>>>>> > very
>>>>>>>> > odd behaviors.
>>>>>>>> >
>>>>>>>> > Given the customer wanted a generic WAF without tuning of
>>> individual
>>>>>>> APIs
>>>>>>>> > we
>>>>>>>> > got to a dead end.
>>>>>>>> >
>>>>>>>> > However, if tuning was an option, then I would probably edit-CRS
>>> with
>>>>>>>> > msc_pyparser and replace the target lists with arguments I was
>>>>>>> interested
>>>>>>>> > in.
>>>>>>>> >
>>>>>>>> > https://coreruleset.org/20200901/introducing-msc_pyparser/
>>>>>>>> >
>>>>>>>> > As a complementary practice, one could think of performing
>>> allowlist
>>>>>>>> > checks on
>>>>>>>> > some / most of the JSON. Say you have a huge JSON payload with 500
>>>>>>>> > parameters.
>>>>>>>> > You examine it and discover that 300 of them actually contain
>>> simple
>>>>>>> digits
>>>>>>>> > and asciii characters and neither special chars nor escape
>>> sequences.
>>>>>>>> > So you do a regex allowlist and apply it to these 300 parameters of
>>>>>>> said
>>>>>>>> > API. And the rest you can push into CRS. Or a subset of CRS.
>>>>>>>> >
>>>>>>>> > I have not done this and the problem is if ModSec is able to handle
>>>>>>> the
>>>>>>>> > large
>>>>>>>> > target lists in a speedy manner.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > Now you can turn to a CDN or alternative WAF. I would do an
>>> extensive
>>>>>>>> > security
>>>>>>>> > tests of such a system. As I said, the JSON parser can be really
>>>>>>> fast. The
>>>>>>>> > difficult thing is to check several hundred parameters without
>>> losing
>>>>>>>> > performance.
>>>>>>>> >
>>>>>>>> > Good luck!
>>>>>>>> >
>>>>>>>> > Christian
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Sun, Apr 25, 2021 at 08:47:06PM +0100, Henri Cook wrote:
>>>>>>>> > > Hi all,
>>>>>>>> > >
>>>>>>>> > > I'm in a situation where the only solution seems to be to drop
>>>>>>> modsec/CRS
>>>>>>>> > > and look at something like Cloudflare's WAF (and change our
>>> security
>>>>>>>> > model
>>>>>>>> > > out of necessity). I'm hoping the esteemed membership of this
>>> list
>>>>>>> might
>>>>>>>> > > have some thoughts.
>>>>>>>> > >
>>>>>>>> > > I've got about 1MB of JSON, payloads in our app might run to 20
>>> or
>>>>>>> even
>>>>>>>> > > 30MB ultimately.
>>>>>>>> > > This 1MB of somewhat nested JSON (7 or 8 levels deep) can take 40
>>>>>>> seconds
>>>>>>>> > > to process in mod sec 3.0.4 with CRS 3.2.0
>>>>>>>> > >
>>>>>>>> > > It takes 1 second to process in our API so the WAF element is a
>>> 39x
>>>>>>> slow
>>>>>>>> > > down. I appreciate there'll be some delays in WAF. Cloudflare's
>>> WAF
>>>>>>>> > takes 5
>>>>>>>> > > seconds to scan this payload - and that's my target.
>>>>>>>> > >
>>>>>>>> > > Has anyone got any idea how to improve performance? Reading blog
>>>>>>> posts
>>>>>>>> > > about the development of cloudflare's waf I see that memoization
>>> of
>>>>>>>> > common
>>>>>>>> > > function calls was one of their absolute best performance
>>>>>>> improvements
>>>>>>>> > over
>>>>>>>> > > their modsec implementation (e.g. strlen(response_body) so it's
>>> only
>>>>>>>> > > calculated once instead of once per rule OR
>>> contains('somestring',
>>>>>>>> > > response_body)... you get the drift). Do we have anything like
>>> this
>>>>>>> in
>>>>>>>> > > modsec today? Is that already in place and my 39 seconds is after
>>>>>>> that?
>>>>>>>> > >
>>>>>>>> > > I appreciate that mod sec is fast on its own and adding complex
>>>>>>> rules can
>>>>>>>> > > be said to slow it down. With CRS being by far the most common
>>> use
>>>>>>> case
>>>>>>>> > for
>>>>>>>> > > mod sec (based on my googling) I'm surprised it's this slow, do
>>> you
>>>>>>> think
>>>>>>>> > > i've missed something?
>>>>>>>> > >
>>>>>>>> > > To note: I'm only scanning JSON payloads, typically much less
>>> than
>>>>>>> 0.5MB
>>>>>>>> > > but new, irregular ones that we need scanned in ideally <10
>>> seconds
>>>>>>> that
>>>>>>>> > > can range from 1MB-30MB
>>>>>>>> > >
>>>>>>>> > > Best regards,
>>>>>>>> > >
>>>>>>>> > > Henri Cook
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > > _______________________________________________
>>>>>>>> > > mod-security-users mailing list
>>>>>>>> > > mod...@li...
>>>>>>>> > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>>>>>> > > Commercial ModSecurity Rules and Support from Trustwave's
>>>>>>> SpiderLabs:
>>>>>>>> > > http://www.modsecurity.org/projects/commercial/rules/
>>>>>>>> > > http://www.modsecurity.org/projects/commercial/support/
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > _______________________________________________
>>>>>>>> > mod-security-users mailing list
>>>>>>>> > mod...@li...
>>>>>>>> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>>>>>> > Commercial ModSecurity Rules and Support from Trustwave's
>>> SpiderLabs:
>>>>>>>> > http://www.modsecurity.org/projects/commercial/rules/
>>>>>>>> > http://www.modsecurity.org/projects/commercial/support/
>>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> mod-security-users mailing list
>>>>>>>> mod...@li...
>>>>>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>>>>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>>>>>>> http://www.modsecurity.org/projects/commercial/rules/
>>>>>>>> http://www.modsecurity.org/projects/commercial/support/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> mod-security-users mailing list
>>>>>>> mod...@li...
>>>>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>>>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>>>>>> http://www.modsecurity.org/projects/commercial/rules/
>>>>>>> http://www.modsecurity.org/projects/commercial/support/
>>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> mod-security-users mailing list
>>>> mod...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>>> http://www.modsecurity.org/projects/commercial/rules/
>>>> http://www.modsecurity.org/projects/commercial/support/
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> mod-security-users mailing list
>>> mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> http://www.modsecurity.org/projects/commercial/rules/
>>> http://www.modsecurity.org/projects/commercial/support/
>>>
>
>
>
>
>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
|
