Re: [mod-security-users] Disable body checking for file uploads

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Hi,

for now I worked around the problem by applying a rule very similar to the
one you suggested. However it would be interesting to know why this breaks
by default, especially for small files.

Thanks
Alan

On Wed, Sep 30, 2015 at 11:46 AM, Barry Pollard <bar...@ho...>
wrote:

> Can't you just do something like this?:
>
> SecRule REQUEST_URI /upload/path "id:12345, phase:1, nolog, chain"
>    SecRule REQUEST_TYPE PUT "ctrl:requestBodyAccess=Off"
>
> This rule will allow you to turn off Body Processing for PUT requests to
> /upload/path.
>
> Thanks,
> Barry
>
> On 30 Sep 2015, at 08:34, Alan Cassar <ala...@ri...> wrote:
>
> Pdf file here: https://www.dropbox.com/s/p2itlgokofifxci/A1.pdf?dl=0
>
> Thanks
> Alan
>
> On Wed, Sep 30, 2015 at 10:59 AM, Alan Cassar <ala...@ri...>
> wrote:
>
>> Hi,
>>
>> one more thing, this is not a multipart request. This is one simple
>> request with the body containing the byte stream of a pdf with content-type
>> of application/octet-steam.
>>
>> Debug logs here. Please note at the end there is a wait of 1 minute. No
>> errors show.
>>
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Initialising transaction (txid AcAcAcAcAcDaAcAcAcApAcAc).
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][5]
>> Adding request argument (QUERY_STRING): name "serviceId", value "mpcpma"
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][5]
>> Adding request argument (QUERY_STRING): name "serviceVersion", value
>> "mpcpma"
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][5]
>> Adding request argument (QUERY_STRING): name "token", value
>> "AQIC5wM2LY4Sfcx7gBNkPRHIluoB0RICTWrT7dHcoh0eUro.*AAJTSQACMDUAAlMxAAIwMQ..*"
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][5]
>> Adding request cookie: name "iPlanetDirectoryPro", value
>> "AQIC5wM2LY4Sfcx7gBNkPRHIluoB0RICTWrT7dHcoh0eUro.*AAJTSQACMDUAAlMxAAIwMQ..*"
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Transaction context created (dcfg 1015b68).
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Starting phase REQUEST_HEADERS.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][9]
>> This phase consists of 2 rule(s).
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Recipe: Invoking rule 10586c8; [file
>> "/usr/local/nginx/conf/./modsecurity/modsecurity.conf"] [line "23"] [id
>> "200000"].
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][5]
>> Rule 10586c8: SecRule "REQUEST_HEADERS:Content-Type" "@rx text/xml"
>> "phase:1,auditlog,id:200000,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=XML"
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][9]
>> T (0) lowercase: "application/octet-stream"
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Transformation completed in 47 usec.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Executing operator "rx" with param "text/xml" against
>> REQUEST_HEADERS:Content-Type.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][9]
>> Target value: "application/octet-stream"
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Operator completed in 32 usec.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Rule returned 0.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][9]
>> No match, not chained -> mode NEXT_RULE.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Recipe: Invoking rule 105a270; [file
>> "/usr/local/nginx/conf/./modsecurity/modsecurity.conf"] [line "30"] [id
>> "200001"].
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][5]
>> Rule 105a270: SecRule "REQUEST_HEADERS:Content-Type" "@rx application/json"
>> "phase:1,auditlog,id:200001,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=JSON"
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][9]
>> T (0) lowercase: "application/octet-stream"
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Transformation completed in 21 usec.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Executing operator "rx" with param "application/json" against
>> REQUEST_HEADERS:Content-Type.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][9]
>> Target value: "application/octet-stream"
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Operator completed in 20 usec.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Rule returned 0.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][9]
>> No match, not chained -> mode NEXT_RULE.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Second phase starting (dcfg 1015b68).
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Input filter: Reading request body.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][9]
>> Input filter: Bucket type NGINX contains 8192 bytes.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][9]
>> Input filter: Bucket type NGINX contains 7570 bytes.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][9]
>> Input filter: Bucket type EOS contains 0 bytes.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Request body no files length: 0
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Input filter: Completed receiving request body (length 15762).
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Starting phase REQUEST_BODY.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][9]
>> This phase consists of 2 rule(s).
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Recipe: Invoking rule 105bfb8; [file
>> "/usr/local/nginx/conf/./modsecurity/modsecurity.conf"] [line "61"] [id
>> "200002"].
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][5]
>> Rule 105bfb8: SecRule "REQBODY_ERROR" "!@eq 0"
>> "phase:2,auditlog,id:200002,t:none,log,deny,status:400,msg:'Failed to parse
>> request body.',logdata:%{reqbody_error_msg},severity:2"
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Transformation completed in 4 usec.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Executing operator "!eq" with param "0" against REQBODY_ERROR.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][9]
>> Target value: "0"
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Operator completed in 17 usec.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Rule returned 0.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][9]
>> No match, not chained -> mode NEXT_RULE.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Recipe: Invoking rule 105dd70; [file
>> "/usr/local/nginx/conf/./modsecurity/modsecurity.conf"] [line "101"] [id
>> "200005"].
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][5]
>> Rule 105dd70: SecRule "TX:/^MSC_/" "!@streq 0"
>> "phase:2,log,auditlog,id:200005,t:none,deny,msg:'ModSecurity internal error
>> flagged: %{MATCHED_VAR_NAME}'"
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Rule returned 0.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][9]
>> No match, not chained -> mode NEXT_RULE.
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Hook insert_filter: Adding input forwarding filter (r f9e2c8).
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Hook insert_filter: Adding output filter (r f9e2c8).
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Input filter: Forwarding input: mode=0, block=0, nbytes=-1 (f f9f6e0, r
>> f9e2c8).
>> [30/Sep/2015:09:45:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Input filter: Forwarded 8192 bytes.
>> [30/Sep/2015:09:46:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Initialising logging.
>> [30/Sep/2015:09:46:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Starting phase LOGGING.
>> [30/Sep/2015:09:46:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][9]
>> This phase consists of 0 rule(s).
>> [30/Sep/2015:09:46:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Recording persistent data took 0 microseconds.
>> [30/Sep/2015:09:46:02 +0400]
>> [server1/sid#1015828][rid#f9e2c8][/api/1.5/users/aw/pc/1104478/agentInstructions/2/attachments/A1.pdf][4]
>> Audit log: Ignoring a non-relevant request.
>>
>>
>> On Wed, Sep 30, 2015 at 9:46 AM, Christian Folini <
>> chr...@ti...> wrote:
>>
>>> Hello Alan,
>>>
>>> This sounds tricky.
>>>
>>> Not sure the list is able to help you, but if you want
>>> anybody to take a closer look, please raise your
>>> DebugLogLevel in ModSec and provide
>>> - pdf-file
>>> - audit log
>>> - modsec debug log
>>>
>>> Regs,
>>>
>>> Christian
>>>
>>> On Wed, Sep 30, 2015 at 09:39:29AM +0400, Alan Cassar wrote:
>>> > Hi Christian,
>>> >
>>> > I had tried the SecRequestBodyLimitAction ProcessPartial already but
>>> this
>>> > is to no avail. Currently I am trying with a very small pdf file, 16k
>>> in
>>> > size. I am assuming that 16k can very easily be stored in memory and
>>> there
>>> > should not be any issues with that.
>>> >
>>> > However I am still facing the same issue. The browser simply receives
>>> 504
>>> > Gateway Time-out. So in my opinion this has nothing to do with the
>>> size of
>>> > the file, but probably with the binary data itself.
>>> >
>>> > Thanks
>>> > Alan
>>> >
>>> >
>>> > On Tue, Sep 29, 2015 at 6:46 PM, Christian Folini <
>>> > chr...@ti...> wrote:
>>> >
>>> > > Alan,
>>> > >
>>> > > You may want to look at the following group of directives:
>>> > >
>>> > > SecRequestBodyInMemoryLimit
>>> > > SecRequestBodyLimit
>>> > > SecRequestBodyNoFilesLimit
>>> > > SecRequestBodyLimitAction
>>> > >
>>> > > Especially SecRequestBodyLimitAction ProcessPartial.
>>> > >
>>> > > Ahoj,
>>> > >
>>> > > Christian
>>> > >
>>> > >
>>> > > On Tue, Sep 29, 2015 at 05:46:03PM +0400, Alan Cassar wrote:
>>> > > > Hi all,
>>> > > >
>>> > > > I have a problem with file uploads going through mod security.
>>> > > >
>>> > > > Whenever I try to upload a file through my API, mod security is
>>> trying to
>>> > > > inspect the body (since SecRequestBodyAccess is On) and will take
>>> a very
>>> > > > long time, until client times out.
>>> > > >
>>> > > > Is it possible to switch SecRequestBodyAccess Off only for certain
>>> > > content
>>> > > > types such as application/octet-stream?
>>> > > >
>>> > > > Thanks very much
>>> > > > Alan
>>> > >
>>> > > >
>>> > >
>>> ------------------------------------------------------------------------------
>>> > >
>>> > > > _______________________________________________
>>> > > > mod-security-users mailing list
>>> > > > mod...@li...
>>> > > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> > > > Commercial ModSecurity Rules and Support from Trustwave's
>>> SpiderLabs:
>>> > > > http://www.modsecurity.org/projects/commercial/rules/
>>> > > > http://www.modsecurity.org/projects/commercial/support/
>>> > >
>>> > >
>>> > >
>>> > >
>>> ------------------------------------------------------------------------------
>>> > > _______________________________________________
>>> > > mod-security-users mailing list
>>> > > mod...@li...
>>> > > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> > > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> > > http://www.modsecurity.org/projects/commercial/rules/
>>> > > http://www.modsecurity.org/projects/commercial/support/
>>> > >
>>>
>>> >
>>> ------------------------------------------------------------------------------
>>>
>>> > _______________________________________________
>>> > mod-security-users mailing list
>>> > mod...@li...
>>> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> > http://www.modsecurity.org/projects/commercial/rules/
>>> > http://www.modsecurity.org/projects/commercial/support/
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> _______________________________________________
>>> mod-security-users mailing list
>>> mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> http://www.modsecurity.org/projects/commercial/rules/
>>> http://www.modsecurity.org/projects/commercial/support/
>>>
>>
>>
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>
>
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>
>