Re: [Mod-security-developers] JSON body processor
Brought to you by:
victorhora,
zimmerletw
Menu
▾
▴
Re: [Mod-security-developers] JSON body processor
|
From: Ulisses M. <uli...@gm...> - 2012-10-03 15:23:40
|
Breno,
Working on it. It turns out there are only 2 streaming JSON parsers
for C available, and neither provide samples of working code for
actual streaming usage -- they use files, which can be
rewound/replayed as needed. I am working on a hackish implementation
first, which buffers the entire JSON data and parses it entirely. I
also need to understand better how this data would be exposed to
mod_security's rules.
I will keep everyone up to date on any relevant progress, and I will
certainly need help testing it.
Thanks!
On Wed, Oct 3, 2012 at 11:25 AM, Breno Silva <bre...@gm...> wrote:
> Hello Ulisses,
>
> Did you have any news about this ?
> Let me know your ETA to have it done, maybe we can include it into 2.7 code.
>
> Thanks
>
> Breno
>
>
> On Sun, Sep 23, 2012 at 2:36 PM, Breno Silva <bre...@gm...> wrote:
>>
>> Ulisses,
>>
>> I think something like:
>>
>> SecRule JSON "@rx test" "..." <- This will loop and execute operation
>> against all JSON variable values
>> SecRule JSON:name "@rx test" "..."
>> SecRule JSON:user/phone "@rx 123456" "..."
>>
>> SecRule JSON_RAW "@rx test" "..." -> an unique string with all json data.
>> SecRule JSON_NAMES "@rx [a-b] "..."-> collection with variable names
>>
>> Let see what Ryan think about it from rule creation point of view.
>>
>> Thanks
>>
>> Breno
>>
>>
>> On Sun, Sep 23, 2012 at 1:54 PM, Ulisses Montenegro
>> <uli...@gm...> wrote:
>>>
>>> Breno,
>>>
>>> Perhaps it would be easier to look at this the order way around --
>>> what would be the most flexible way to write rules for matching JSON
>>> data? From a parsing perspective, most libraries offer a
>>> JSON-string-to-hashtable approach, which would work for all either
>>> scenario.
>>>
>>> Ryan, do you have any real world use cases for rules matching JSON
>>> parameters?
>>>
>>> Thanks!
>>>
>>> On Sun, Sep 23, 2012 at 3:38 PM, Breno Silva <bre...@gm...>
>>> wrote:
>>> > Ulisses,
>>> >
>>> > I never had a change to think more about this issue.
>>> > Looking for this specific case i really don't think would be a good
>>> > idea to
>>> > create a new logic to ARGS* collections. Not sure what Ryan B. think
>>> > about
>>> > it, but from my point of view, if we need a new logic we must create
>>> > specific collections.
>>> >
>>> > ie: JSON, JSON_NAMES ....
>>> >
>>> > Is "." (dot) allowed to create variables names ? I think yes.
>>> > If so, we should go json specification and find a better way to create
>>> > this
>>> > logic. Maybe using "/" ?
>>> >
>>> > ie: user/name, user/manager/name
>>> >
>>> > What do you think ?
>>> >
>>> > Breno
>>> >
>>> > On Sun, Sep 23, 2012 at 12:34 PM, Ulisses Montenegro
>>> > <uli...@gm...> wrote:
>>> >>
>>> >> Breno & Ryan
>>> >>
>>> >> Thanks for the pointers. Ryan, I need to look further into how ARGS
>>> >> could be used to handle nested data structures. Although deeper
>>> >> structures are more common in responses, I've seen some in requests
>>> >> too. If we go deeper then 2 levels, then how would we break that data
>>> >> into ARGS?
>>> >>
>>> >> { 'user': {
>>> >> 'name': 'John Doe',
>>> >> 'email': 'jo...@do...',
>>> >> 'manager': {
>>> >> 'name': 'Manager John',
>>> >> 'email': 'ma...@do...',
>>> >> 'company': {
>>> >> 'name': 'ModSecurity Corp.',
>>> >> (...)
>>> >> },
>>> >> }
>>> >> }
>>> >>
>>> >> I was thinking that maybe using the fully qualified name for the
>>> >> variable might be easier, and would not introduce any artificial
>>> >> limitations on the depth on the data structure in the JSON data:
>>> >>
>>> >> ARGS:user.name = 'John Doe'
>>> >> ARGS:user.email = 'jo...@do...'
>>> >> ARGS:user.manager.name = 'Manager John'
>>> >> ARGS:user.manager.company.name = 'ModSecurity Corp.'
>>> >> (...)
>>> >>
>>> >> Of course, JSON also supports arrays, but since mod_security already
>>> >> handles multiple instances of the same parameter, that would not be an
>>> >> issue for either option.
>>> >>
>>> >> Does that make sense, or am I misunderstanding how ARGS work?
>>> >>
>>> >> Thanks,
>>> >> Ulisses
>>> >>
>>> >> On Sun, Sep 23, 2012 at 1:46 PM, Ryan Barnett <RBa...@tr...>
>>> >> wrote:
>>> >> > Regarding #2 below - we have two options.
>>> >> >
>>> >> > 1) A JSON parse could work like the XML parse and access the request
>>> >> > body
>>> >> > content and simply populate a new collection called JSON. This is
>>> >> > like
>>> >> > the XML collection that is simply a long string of text. The
>>> >> > downside
>>> >> > of
>>> >> > this approach is that here is no context as to what are parameter
>>> >> > names/values. Another option would be to have the JSON parser
>>> >> > simply
>>> >> > populate this string of text into the current REQUEST_BODY variable.
>>> >> > A
>>> >> > rule writer can do this today if they wish using the following
>>> >> > example
>>> >> > pseudo-rule -
>>> >> >
>>> >> > SecRule REQUEST_HEADERS:Content-Type "@contains application/json"
>>> >> > "phase:1,id:1,nolog,pass,ctl:forceRequestBodyVariable"
>>> >> >
>>> >> > 2) I think that the best way to do this is to attempt to parse the
>>> >> > JSON
>>> >> > data into name/value pairs and populate that into ARGS. If it is
>>> >> > parsed
>>> >> > in this way, then we don't need to change anything in the current
>>> >> > rules.
>>> >> >
>>> >> > As just one example, I was reviewing the JSON data sent back to
>>> >> > twitter
>>> >> > in
>>> >> > response to a Content Security Policy (CSP) violation. The
>>> >> > content-type
>>> >> > is application/json and uses the name/value pairs -
>>> >> >
>>> >> > POST /scribes/csp_report HTTP/1.1
>>> >> > Host: twitter.com
>>> >> > User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:15.0)
>>> >> > Gecko/20100101 Firefox/15.0
>>> >> > Accept:
>>> >> > text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
>>> >> > Accept-Language: en-us,en;q=0.5
>>> >> > Accept-Encoding: gzip, deflate
>>> >> > DNT: 1
>>> >> > Connection: keep-alive
>>> >> > Content-Length: 338
>>> >> > Content-Type: application/json
>>> >> >
>>> >> >
>>> >> >
>>> >> > {"csp-report":{"document-uri":"https://mobile.twitter.com/i/templates/m5?re
>>> >> >
>>> >> >
>>> >> > v=1347385509950","referrer":"https://mobile.twitter.com/","blocked-uri":"se
>>> >> > lf","violated-directive":"inline
>>> >> >
>>> >> >
>>> >> > <https://mobile.twitter.com/i/templates/m5?rev=1347385509950%22,%22referrer
>>> >> >
>>> >> >
>>> >> > %22:%22https://mobile.twitter.com/%22,%22blocked-uri%22:%22self%22,%22viola
>>> >> > ted-directive%22:%22inline> script base
>>> >> >
>>> >> >
>>> >> > restriction","source-file":"https://mobile.twitter.com/i/templates/m5?rev=1
>>> >> > 347385509950","script-sample":"onclick
>>> >> >
>>> >> >
>>> >> > <https://mobile.twitter.com/i/templates/m5?rev=1347385509950%22,%22script-s
>>> >> > ample%22:%22onclick> attribute on DIV element"}}
>>> >> >
>>> >> > Based on this you would split the name/value pairs by the "Š":"Š."
>>> >> > format and have parsed ARGS variable data for use in our rules like
>>> >> > -
>>> >> >
>>> >> > ######################
>>> >> > ARGS:csp-report =
>>> >> >
>>> >> >
>>> >> > "document-uri":"https://mobile.twitter.com/i/templates/m5?rev=1347385509950
>>> >> >
>>> >> >
>>> >> > ","referrer":"https://mobile.twitter.com/","blocked-uri":"self","violated-d
>>> >> > irective":"inline
>>> >> >
>>> >> >
>>> >> > <https://mobile.twitter.com/i/templates/m5?rev=1347385509950%22,%22referrer
>>> >> >
>>> >> >
>>> >> > %22:%22https://mobile.twitter.com/%22,%22blocked-uri%22:%22self%22,%22viola
>>> >> > ted-directive%22:%22inline> script base
>>> >> >
>>> >> >
>>> >> > restriction","source-file":"https://mobile.twitter.com/i/templates/m5?rev=1
>>> >> > 347385509950","script-sample":"onclick
>>> >> >
>>> >> >
>>> >> > <https://mobile.twitter.com/i/templates/m5?rev=1347385509950%22,%22script-s
>>> >> > ample%22:%22onclick> attribute on DIV element"
>>> >> >
>>> >> > ARGS:document-uri =
>>> >> > https://mobile.twitter.com/i/templates/m5?rev=1347385509950
>>> >> >
>>> >> >
>>> >> > <https://mobile.twitter.com/i/templates/m5?rev=1347385509950%22,%22referrer
>>> >> >
>>> >> >
>>> >> > %22:%22https://mobile.twitter.com/%22,%22blocked-uri%22:%22self%22,%22viola
>>> >> > ted-directive%22:%22inline>
>>> >> >
>>> >> > ARGS:referrer = https://mobile.twitter.com/
>>> >> >
>>> >> >
>>> >> > <https://mobile.twitter.com/i/templates/m5?rev=1347385509950%22,%22referrer
>>> >> >
>>> >> >
>>> >> > %22:%22https://mobile.twitter.com/%22,%22blocked-uri%22:%22self%22,%22viola
>>> >> > ted-directive%22:%22inline>
>>> >> >
>>> >> > ARGS:blocked-uri = self
>>> >> >
>>> >> > ARGS:violated-directive = inline script base restriction
>>> >> >
>>> >> > ARGS:source-file =
>>> >> > https://mobile.twitter.com/i/templates/m5?rev=1347385509950
>>> >> >
>>> >> >
>>> >> > <https://mobile.twitter.com/i/templates/m5?rev=1347385509950%22,%22script-s
>>> >> > ample%22:%22onclick>
>>> >> >
>>> >> > ARGS:script-sample = onclick attribute on DIV element
>>> >> > #######################
>>> >> >
>>> >> > Hope this helps.
>>> >> >
>>> >> >
>>> >> > --
>>> >> > Ryan Barnett
>>> >> > Trustwave SpiderLabs
>>> >> > ModSecurity Project Leader
>>> >> > OWASP ModSecurity CRS Project Leader
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > On 9/23/12 9:31 AM, "Ulisses Montenegro"
>>> >> > <uli...@gm...>
>>> >> > wrote:
>>> >> >
>>> >> >>Team
>>> >> >>
>>> >> >>As my first attempt in contributing to mod_security I've decided to
>>> >> >>tackle MODSEC-253, a JSON body processor. I've gone through the XML
>>> >> >>and multipart body processors and found them apparently
>>> >> >>straightforward. I would like some pointers on issues which I need
>>> >> >> to
>>> >> >>address before deciding on my solution, though.
>>> >> >>
>>> >> >>1. The XML body processor uses libxml for the actual XML parsing, I
>>> >> >>assume adding a JSON parser library would be acceptable as well. If
>>> >> >>so, what licenses would be acceptable?
>>> >> >>2. XML processor offers a XPath interface for rules to match XML
>>> >> >>contents, which is a standard, but AFAIK there is nothing equivalent
>>> >> >>for JSON (aside from evaluating Javascript object references). What
>>> >> >>interface would work best for the rules to gain access to the JSON
>>> >> >>contents?
>>> >> >>3. Are there any guidelines/rules regarding memory usage and
>>> >> >>performance, i.e., how can if my code or the library I'm using is
>>> >> >>performing acceptably? I know I can always benchmark/profile other
>>> >> >>body processors and compare the results directly, but I'm looking
>>> >> >> more
>>> >> >>towards hard numbers, if they're available.
>>> >> >>4. Finally, do these kind of questions go into JIRA? I decided to
>>> >> >> try
>>> >> >>the mailing list first as I did not want to add possibly irrelevant
>>> >> >>information to the JIRA issue, but I think at least items [1] and
>>> >> >> [2]
>>> >> >>should be registered there -- is that how it usually works?
>>> >> >>
>>> >> >>Thanks a lot for the great work on mod_security
>>> >> >>Ulisses
>>> >> >>
>>> >> >>--
>>> >> >>³If debugging is the process of removing software bugs, then
>>> >> >>programming must be the process of putting them in.² - Edsger
>>> >> >> Dijkstra
>>> >> >>
>>> >>
>>> >> >>
>>> >> >> >> >>--------------------------------------------------------------------------
>>> >> >>----
>>> >> >>Everyone hates slow websites. So do we.
>>> >> >>Make your web apps faster with AppDynamics
>>> >> >>Download AppDynamics Lite for free today:
>>> >> >>http://ad.doubleclick.net/clk;258768047;13503038;j?
>>> >> >>http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>>> >> >>_______________________________________________
>>> >> >>mod-security-developers mailing list
>>> >> >>mod...@li...
>>> >> >>https://lists.sourceforge.net/lists/listinfo/mod-security-developers
>>> >> >>ModSecurity Services from Trustwave's SpiderLabs:
>>> >> >>https://www.trustwave.com/spiderLabs.php
>>> >> >
>>> >> >
>>> >> > ________________________________
>>> >> >
>>> >> > This transmission may contain information that is privileged,
>>> >> > confidential, and/or exempt from disclosure under applicable law. If
>>> >> > you are
>>> >> > not the intended recipient, you are hereby notified that any
>>> >> > disclosure,
>>> >> > copying, distribution, or use of the information contained herein
>>> >> > (including
>>> >> > any reliance thereon) is STRICTLY PROHIBITED. If you received this
>>> >> > transmission in error, please immediately contact the sender and
>>> >> > destroy the
>>> >> > material in its entirety, whether in electronic or hard copy format.
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > ------------------------------------------------------------------------------
>>> >> > Everyone hates slow websites. So do we.
>>> >> > Make your web apps faster with AppDynamics
>>> >> > Download AppDynamics Lite for free today:
>>> >> > http://ad.doubleclick.net/clk;258768047;13503038;j?
>>> >> > http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>>> >> > _______________________________________________
>>> >> > mod-security-developers mailing list
>>> >> > mod...@li...
>>> >> > https://lists.sourceforge.net/lists/listinfo/mod-security-developers
>>> >> > ModSecurity Services from Trustwave's SpiderLabs:
>>> >> > https://www.trustwave.com/spiderLabs.php
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> “If debugging is the process of removing software bugs, then
>>> >> programming must be the process of putting them in.” - Edsger Dijkstra
>>> >>
>>> >>
>>> >>
>>> >> ------------------------------------------------------------------------------
>>> >> Everyone hates slow websites. So do we.
>>> >> Make your web apps faster with AppDynamics
>>> >> Download AppDynamics Lite for free today:
>>> >> http://ad.doubleclick.net/clk;258768047;13503038;j?
>>> >> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>>> >> _______________________________________________
>>> >> mod-security-developers mailing list
>>> >> mod...@li...
>>> >> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
>>> >> ModSecurity Services from Trustwave's SpiderLabs:
>>> >> https://www.trustwave.com/spiderLabs.php
>>> >
>>> >
>>> >
>>> >
>>> > ------------------------------------------------------------------------------
>>> > Everyone hates slow websites. So do we.
>>> > Make your web apps faster with AppDynamics
>>> > Download AppDynamics Lite for free today:
>>> > http://ad.doubleclick.net/clk;258768047;13503038;j?
>>> > http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>>> > _______________________________________________
>>> > mod-security-developers mailing list
>>> > mod...@li...
>>> > https://lists.sourceforge.net/lists/listinfo/mod-security-developers
>>> > ModSecurity Services from Trustwave's SpiderLabs:
>>> > https://www.trustwave.com/spiderLabs.php
>>>
>>>
>>>
>>> --
>>> “If debugging is the process of removing software bugs, then
>>> programming must be the process of putting them in.” - Edsger Dijkstra
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Everyone hates slow websites. So do we.
>>> Make your web apps faster with AppDynamics
>>> Download AppDynamics Lite for free today:
>>> http://ad.doubleclick.net/clk;258768047;13503038;j?
>>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>>> _______________________________________________
>>> mod-security-developers mailing list
>>> mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
>>> ModSecurity Services from Trustwave's SpiderLabs:
>>> https://www.trustwave.com/spiderLabs.php
>>
>>
>
>
> ------------------------------------------------------------------------------
> Don't let slow site performance ruin your business. Deploy New Relic APM
> Deploy New Relic app performance management and know exactly
> what is happening inside your Ruby, Python, PHP, Java, and .NET app
> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> http://p.sf.net/sfu/newrelic-dev2dev
> _______________________________________________
> mod-security-developers mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-developers
> ModSecurity Services from Trustwave's SpiderLabs:
> https://www.trustwave.com/spiderLabs.php
--
“If debugging is the process of removing software bugs, then
programming must be the process of putting them in.” - Edsger Dijkstra
|
