Re: [mod-security-users] Performance woes - larger JSON payloads with CRS
Brought to you by:
victorhora,
zimmerletw
Menu
▾
▴
Re: [mod-security-users] Performance woes - larger JSON payloads with CRS
|
From: Henri C. <he...@pr...> - 2021-05-10 14:39:34
|
This is really great, and works like a charm - thanks so much
On Mon, 10 May 2021 at 15:35, <az...@po...> wrote:
> Another, maybe better, solution, as you are already going to modify
> modsecurity.conf, would be to modify rule 200001 directly. Try this:
>
>
> SecRule REQUEST_HEADERS:Content-Type "application/json" \
> "id:'200001',phase:1,t:none,t:lowercase,pass,nolog,chain"
> SecRule REQUEST_HEADERS:Content-Length "@lt 131072" \
> "t:none,ctl:requestBodyProcessor=JSON"
>
>
>
>
>
> Citát az...@po...:
>
> > Sorry, now I see the problem: Rule which is doing JSON processing
> > (200001) is running in phase 1 and my rule below is running in phase
> > 2. Problem is, as i stated below, REQUEST_BODY_LENGTH is available
> > from phase 2 so my rule cannot be run in phase 1. Also, i noticed
> > another problem - we cannot do 'ruleRemoveTargetByTag=OWASP_CRS' as
> > rule 200001 is not part of the CRS and it's running before CRS (and
> > also before rules in file
> > REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf).
> >
> > In this case, we can use Content-Length header, which won't be so
> > realibe but should be sufficient. Try this:
> >
> > SecRule REQUEST_HEADERS:Content-Type "application/json" \
> > "id:9999999,\
> > phase:1,\
> > pass,\
> > t:none,\
> > nolog,\
> > chain"
> > SecRule REQUEST_HEADERS:Content-Length "@gt 131072" \
> > "t:none,\
> > ctl:ruleRemoveTargetById=200001;REQUEST_BODY"
> >
> >
> > This rule should be run before rule 200001 so you will, probably,
> > need to put it inside modsecurity.conf before rule 200001 (but i
> > didn't tested it at all).
> >
> >
> >
> >
> >
> > Citát Henri Cook <henri@proteus.tech>:
> >
> >> Thanks for this azurit but neither of these worked for me. I changed
> the id
> >> to 1001 and put it in my REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf - I
> >> still see a delay processing a large json payload - do you have any idea
> >> where I might have gone wrong?
> >>
> >> I think my goal is to stop the JSON request body being parsed if it's
> over
> >> 128kb in size. An advance goal would be to have it scanned as text to
> >> provide *some* level of protection - I think this is what cloudflare
> does
> >> (from a very cursory glance)
> >>
> >> Best Regards,
> >>
> >> Henri
> >>
> >> On Thu, 29 Apr 2021 at 19:02, <az...@po...> wrote:
> >>
> >>> I just noticed you are using CRS, so this one would be probably better:
> >>>
> >>>
> >>> SecRule REQUEST_BODY_LENGTH "@gt 131072" \
> >>> "id:9999999,\
> >>> phase:2,\
> >>> pass,\
> >>> t:none,\
> >>> nolog,\
> >>> ctl:ruleRemoveTargetByTag=OWASP_CRS;REQUEST_BODY"
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Citát az...@po...:
> >>>
> >>>> Hi,
> >>>>
> >>>> you can do this using exclusive rule, something like this (set the
> >>>> rule ID to something 'better'), length is in bytes:
> >>>>
> >>>>
> >>>> SecRule REQUEST_BODY_LENGTH "@gt 131072" \
> >>>> "id:9999999,\
> >>>> phase:2,\
> >>>> pass,\
> >>>> t:none,\
> >>>> nolog,\
> >>>> ctl:ruleEngine=Off"
> >>>>
> >>>>
> >>>>
> >>>> REQUEST_BODY_LENGTH is available from phase 2 so you cannot put this
> >>>> into phase 1.
> >>>>
> >>>>
> >>>>
> >>>> Citát Henri Cook <henri@proteus.tech>:
> >>>>
> >>>>> I think what I need here is a way to exempt the request body from any
> >>>>> scanning when it's over 128kb (my chosen upper limit)...
> >>>>>
> >>>>> I'll probably try implementing this as a phase 1 rule on
> Content-Length
> >>>>> unless anyone shouts there's a better way. `ProcessPartial` sounded
> like
> >>>>> the holy grail, but you can't partial process JSON (because it
> doesn't
> >>>>> parse!).
> >>>>>
> >>>>> It'd be cool to be able to fallback to a fulltext scan of the partial
> >>> JSON
> >>>>> if it's >128kb, I might have a try at looking into that also (again
> any
> >>>>> guidance appreciated)
> >>>>>
> >>>>> Best Regards,
> >>>>>
> >>>>> Henri
> >>>>>
> >>>>> On Mon, 26 Apr 2021 at 15:24, Henri Cook <henri@proteus.tech> wrote:
> >>>>>
> >>>>>> The follow up problem to this is:
> >>>>>>
> >>>>>> Now i'm set to `SecRequestBodyLimit` 31457280 and
> >>>>>> `SecRequestBodyNoFilesLimit` 65536 and `SecRequestBodyLimitAction
> >>>>>> PartialProcess`
> >>>>>>
> >>>>>> In my mind this will process the first part of a request if it can,
> and
> >>>>>> ignore the rest. But:
> >>>>>>
> >>>>>> - Rule 200002 is triggered, which is saying the JSON can't be
> parsed.
> >>>>>> Presumably because in a large request it tries to process the
> >>> beginning of
> >>>>>> the JSON and can't (because it won't parse, because the JSON is cut
> >>> off so
> >>>>>> doesn't end)
> >>>>>>
> >>>>>> So I think I need to find a way to skip JSON parsing entirely when
> the
> >>>>>> payload is over 64kb (65536)? Does that sound right? Assuming 64kb
> is
> >>> the
> >>>>>> limit I want to stick with. I hadn't really considered before this
> >>> point
> >>>>>> that 'partial processing' of JSON was likely to be hairy but of
> course
> >>> it
> >>>>>> makes sense.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Mon, 26 Apr 2021 at 07:17, Christian Folini <
> >>>>>> chr...@ne...> wrote:
> >>>>>>
> >>>>>>> Hey Henri,
> >>>>>>>
> >>>>>>> From a security practice, this is obviously lacking, but in wider
> >>>>>>> perspective,
> >>>>>>> I see it meet "industry standard", yes.
> >>>>>>>
> >>>>>>> When I teach, I tell my student, that the worst WAF is the one
> that is
> >>>>>>> switched off. So if you need to compromise and you can only apply
> 20%
> >>> of
> >>>>>>> the rules because you run the risk of business demanding it's
> switched
> >>>>>>> off,
> >>>>>>> then that 20% WAF is still better than no WAF.
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>>
> >>>>>>> Christian
> >>>>>>>
> >>>>>>>
> >>>>>>> On Mon, Apr 26, 2021 at 06:57:54AM +0100, Henri Cook wrote:
> >>>>>>>> Thanks Christian, taking this in combination with Osama's point
> >>> earlier
> >>>>>>> in
> >>>>>>>> the thread that most 'big four' (AWS/GCP/Cloudflare/Azure) WAFs
> seem
> >>> to
> >>>>>>>> limit the payload they'll scan. From my reading to 128kb
> >>>>>>> (cloudflare+azure)
> >>>>>>>> or 8kb (aws+gcp) I think I'll be able to resolve our particular
> >>> issue.
> >>>>>>>>
> >>>>>>>> I believe my modern application is already very robust in terms of
> >>>>>>> defence
> >>>>>>>> against sql injection as well as other OWASP top 10 attack vectors
> >>> and
> >>>>>>> that
> >>>>>>>> a WAF primarily adds reassurance (for the business and clients who
> >>> ask
> >>>>>>> if I
> >>>>>>>> have one) and minor frustration (for any potential attacker)
> layer.
> >>> The
> >>>>>>>> spec is to add a WAF that meets (but notably does not necessarily
> >>> have
> >>>>>>> to
> >>>>>>>> exceed) industry standards. I believe this means that I can switch
> >>>>>>> modsec
> >>>>>>>> to 128kb or 8kb partial parsing ('SecResponseBodyLimitAction
> >>>>>>>> ProcessPartial' - allowing through unscanned any payloads over
> those
> >>>>>>> sizes)
> >>>>>>>> and be able to say I've got scan-size-policy-parity with an AWS
> or a
> >>>>>>>> Cloudflare which means it is "industry standard".
> >>>>>>>>
> >>>>>>>> Please let me know if you think that's mad and thanks again
> >>>>>>>>
> >>>>>>>> Best Regards,
> >>>>>>>>
> >>>>>>>> Henri
> >>>>>>>>
> >>>>>>>> On Sun, 25 Apr 2021 at 21:39, Christian Folini <
> >>>>>>> chr...@ne...>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> > Hey Henri,
> >>>>>>>> >
> >>>>>>>> > You are in a bad situation and as far as I can see you are
> right,
> >>> you
> >>>>>>> might
> >>>>>>>> > have to drop modsec/CRS in this situation.
> >>>>>>>> >
> >>>>>>>> > I've had a customer with a similar problem and we did a deep
> dive
> >>>>>>>> > investigation and I had to strike colors in the end.
> >>>>>>>> >
> >>>>>>>> > The point is not the JSON parser. That has shown to be really
> fast.
> >>>>>>> The
> >>>>>>>> > point
> >>>>>>>> > is several hundred variables that go into CRS afterwards. If you
> >>> run
> >>>>>>> CRS
> >>>>>>>> > on a
> >>>>>>>> > standard web application you get forms with a few parameters and
> >>>>>>> that's
> >>>>>>>> > easy.
> >>>>>>>> > But several megabytes of JSON means hundreds of arguments and
> CRS
> >>>>>>> parses
> >>>>>>>> > them
> >>>>>>>> > all.
> >>>>>>>> >
> >>>>>>>> > So we tried to work with rule exclusions and skip the
> parameters we
> >>>>>>> did not
> >>>>>>>> > think dangerous, but here comes the bummer: ModSec 2.9 grew
> >>>>>>> substantially
> >>>>>>>> > slower the longer the ignore-lists of parameters became. This
> and a
> >>>>>>> few
> >>>>>>>> > very
> >>>>>>>> > odd behaviors.
> >>>>>>>> >
> >>>>>>>> > Given the customer wanted a generic WAF without tuning of
> >>> individual
> >>>>>>> APIs
> >>>>>>>> > we
> >>>>>>>> > got to a dead end.
> >>>>>>>> >
> >>>>>>>> > However, if tuning was an option, then I would probably edit-CRS
> >>> with
> >>>>>>>> > msc_pyparser and replace the target lists with arguments I was
> >>>>>>> interested
> >>>>>>>> > in.
> >>>>>>>> >
> >>>>>>>> > https://coreruleset.org/20200901/introducing-msc_pyparser/
> >>>>>>>> >
> >>>>>>>> > As a complementary practice, one could think of performing
> >>> allowlist
> >>>>>>>> > checks on
> >>>>>>>> > some / most of the JSON. Say you have a huge JSON payload with
> 500
> >>>>>>>> > parameters.
> >>>>>>>> > You examine it and discover that 300 of them actually contain
> >>> simple
> >>>>>>> digits
> >>>>>>>> > and asciii characters and neither special chars nor escape
> >>> sequences.
> >>>>>>>> > So you do a regex allowlist and apply it to these 300
> parameters of
> >>>>>>> said
> >>>>>>>> > API. And the rest you can push into CRS. Or a subset of CRS.
> >>>>>>>> >
> >>>>>>>> > I have not done this and the problem is if ModSec is able to
> handle
> >>>>>>> the
> >>>>>>>> > large
> >>>>>>>> > target lists in a speedy manner.
> >>>>>>>> >
> >>>>>>>> >
> >>>>>>>> > Now you can turn to a CDN or alternative WAF. I would do an
> >>> extensive
> >>>>>>>> > security
> >>>>>>>> > tests of such a system. As I said, the JSON parser can be really
> >>>>>>> fast. The
> >>>>>>>> > difficult thing is to check several hundred parameters without
> >>> losing
> >>>>>>>> > performance.
> >>>>>>>> >
> >>>>>>>> > Good luck!
> >>>>>>>> >
> >>>>>>>> > Christian
> >>>>>>>> >
> >>>>>>>> >
> >>>>>>>> > On Sun, Apr 25, 2021 at 08:47:06PM +0100, Henri Cook wrote:
> >>>>>>>> > > Hi all,
> >>>>>>>> > >
> >>>>>>>> > > I'm in a situation where the only solution seems to be to drop
> >>>>>>> modsec/CRS
> >>>>>>>> > > and look at something like Cloudflare's WAF (and change our
> >>> security
> >>>>>>>> > model
> >>>>>>>> > > out of necessity). I'm hoping the esteemed membership of this
> >>> list
> >>>>>>> might
> >>>>>>>> > > have some thoughts.
> >>>>>>>> > >
> >>>>>>>> > > I've got about 1MB of JSON, payloads in our app might run to
> 20
> >>> or
> >>>>>>> even
> >>>>>>>> > > 30MB ultimately.
> >>>>>>>> > > This 1MB of somewhat nested JSON (7 or 8 levels deep) can
> take 40
> >>>>>>> seconds
> >>>>>>>> > > to process in mod sec 3.0.4 with CRS 3.2.0
> >>>>>>>> > >
> >>>>>>>> > > It takes 1 second to process in our API so the WAF element is
> a
> >>> 39x
> >>>>>>> slow
> >>>>>>>> > > down. I appreciate there'll be some delays in WAF.
> Cloudflare's
> >>> WAF
> >>>>>>>> > takes 5
> >>>>>>>> > > seconds to scan this payload - and that's my target.
> >>>>>>>> > >
> >>>>>>>> > > Has anyone got any idea how to improve performance? Reading
> blog
> >>>>>>> posts
> >>>>>>>> > > about the development of cloudflare's waf I see that
> memoization
> >>> of
> >>>>>>>> > common
> >>>>>>>> > > function calls was one of their absolute best performance
> >>>>>>> improvements
> >>>>>>>> > over
> >>>>>>>> > > their modsec implementation (e.g. strlen(response_body) so
> it's
> >>> only
> >>>>>>>> > > calculated once instead of once per rule OR
> >>> contains('somestring',
> >>>>>>>> > > response_body)... you get the drift). Do we have anything like
> >>> this
> >>>>>>> in
> >>>>>>>> > > modsec today? Is that already in place and my 39 seconds is
> after
> >>>>>>> that?
> >>>>>>>> > >
> >>>>>>>> > > I appreciate that mod sec is fast on its own and adding
> complex
> >>>>>>> rules can
> >>>>>>>> > > be said to slow it down. With CRS being by far the most common
> >>> use
> >>>>>>> case
> >>>>>>>> > for
> >>>>>>>> > > mod sec (based on my googling) I'm surprised it's this slow,
> do
> >>> you
> >>>>>>> think
> >>>>>>>> > > i've missed something?
> >>>>>>>> > >
> >>>>>>>> > > To note: I'm only scanning JSON payloads, typically much less
> >>> than
> >>>>>>> 0.5MB
> >>>>>>>> > > but new, irregular ones that we need scanned in ideally <10
> >>> seconds
> >>>>>>> that
> >>>>>>>> > > can range from 1MB-30MB
> >>>>>>>> > >
> >>>>>>>> > > Best regards,
> >>>>>>>> > >
> >>>>>>>> > > Henri Cook
> >>>>>>>> >
> >>>>>>>> >
> >>>>>>>> > > _______________________________________________
> >>>>>>>> > > mod-security-users mailing list
> >>>>>>>> > > mod...@li...
> >>>>>>>> > >
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >>>>>>>> > > Commercial ModSecurity Rules and Support from Trustwave's
> >>>>>>> SpiderLabs:
> >>>>>>>> > > http://www.modsecurity.org/projects/commercial/rules/
> >>>>>>>> > > http://www.modsecurity.org/projects/commercial/support/
> >>>>>>>> >
> >>>>>>>> >
> >>>>>>>> >
> >>>>>>>> > _______________________________________________
> >>>>>>>> > mod-security-users mailing list
> >>>>>>>> > mod...@li...
> >>>>>>>> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >>>>>>>> > Commercial ModSecurity Rules and Support from Trustwave's
> >>> SpiderLabs:
> >>>>>>>> > http://www.modsecurity.org/projects/commercial/rules/
> >>>>>>>> > http://www.modsecurity.org/projects/commercial/support/
> >>>>>>>> >
> >>>>>>>
> >>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> mod-security-users mailing list
> >>>>>>>> mod...@li...
> >>>>>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >>>>>>>> Commercial ModSecurity Rules and Support from Trustwave's
> SpiderLabs:
> >>>>>>>> http://www.modsecurity.org/projects/commercial/rules/
> >>>>>>>> http://www.modsecurity.org/projects/commercial/support/
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> mod-security-users mailing list
> >>>>>>> mod...@li...
> >>>>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >>>>>>> Commercial ModSecurity Rules and Support from Trustwave's
> SpiderLabs:
> >>>>>>> http://www.modsecurity.org/projects/commercial/rules/
> >>>>>>> http://www.modsecurity.org/projects/commercial/support/
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> mod-security-users mailing list
> >>>> mod...@li...
> >>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> >>>> http://www.modsecurity.org/projects/commercial/rules/
> >>>> http://www.modsecurity.org/projects/commercial/support/
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> mod-security-users mailing list
> >>> mod...@li...
> >>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> >>> http://www.modsecurity.org/projects/commercial/rules/
> >>> http://www.modsecurity.org/projects/commercial/support/
> >>>
> >
> >
> >
> >
> >
> > _______________________________________________
> > mod-security-users mailing list
> > mod...@li...
> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> > http://www.modsecurity.org/projects/commercial/rules/
> > http://www.modsecurity.org/projects/commercial/support/
>
>
>
>
>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>
|
