Thread: [mod-security-users] Still content compression problems with 2.5?
Brought to you by:
victorhora,
zimmerletw
From: Stefan Müller-W. <ste...@re...> - 2008-03-18 11:16:17
|
Dear all, we're still struggling with the infamous content compression problems with a screened tomcat gzipping its output and flooding our reverse proxy logs with "id 960903" errors. Two questions: a) Why can't I switch of 960903 using "SecRemoveRuleById "960903" as I can with all other CRS rules? b) Does version 2.5 support combining mod_deflate with mod_security (e.g. by defining the order in which the two are executed) to switch off content compression on the app server and relocate it to the reverse proxy? Regards Stef. Resco GmbH Geschäftsführer: Michael Mörchen Amtsgericht Hamburg, HRB 76048 Ust.Ident-Nr.:DE208833022 Haftungsausschluss: Diese Nachricht ist ausschließlich für die Person oder Einheit bestimmt, an die sie gerichtet ist. Sie enthält unter Umständen Informationen, die unter geltendem Recht vertraulich, gesetzlich geschützt oder von der Offenlegung ausgeschlossen sind. Falls Sie nicht der vorgesehene Empfänger oder verantwortlich für die Weiterleitung dieser Nachricht an den vorgesehenen Empfänger sind, ist es Ihnen strengstens untersagt, diese Nachricht offenzulegen, zu verteilen, zu kopieren oder in irgendeiner Art zu benutzen. Sollten Sie diese Nachricht versehentlich erhalten haben, benachrichtigen Sie bitte den Absender und löschen und vernichten Sie jegliche Kopie davon, die Sie möglicherweise erhalten haben. Disclaimer: This message is intended only for the use of the individual or entity to which it is addressed and may contain information which is privileged, confidential, proprietary, or exempt from disclosure under applicable law. If you are not the intended recipient or the person responsible for delivering the message to the intended recipient, you are strictly prohibited from disclosing, distributing, copying, or in any way using this message. If you have received this communication in error, please notify the sender and destroy and delete any copies you may have received. |
From: Stefan Müller-W. <ste...@re...> - 2008-03-20 13:18:11
|
[This is a repost as I didn't see my post arrive on the list - sorry, if it actually now has twice!] Dear all, we're still struggling with the infamous content compression problems with a screened tomcat gzipping its output and flooding our reverse proxy logs with "id 960903" errors. Two questions: a) Why can't I switch of 960903 using "SecRemoveRuleById "960903" as I can with all other CRS rules? b) Does version 2.5 support combining mod_deflate with mod_security (e.g. by defining the order in which the two are executed) to switch off content compression on the app server and relocate it to the reverse proxy? Regards Stef. Resco GmbH Geschäftsführer: Michael Mörchen Amtsgericht Hamburg, HRB 76048 Ust.Ident-Nr.:DE208833022 Haftungsausschluss: Diese Nachricht ist ausschließlich für die Person oder Einheit bestimmt, an die sie gerichtet ist. Sie enthält unter Umständen Informationen, die unter geltendem Recht vertraulich, gesetzlich geschützt oder von der Offenlegung ausgeschlossen sind. Falls Sie nicht der vorgesehene Empfänger oder verantwortlich für die Weiterleitung dieser Nachricht an den vorgesehenen Empfänger sind, ist es Ihnen strengstens untersagt, diese Nachricht offenzulegen, zu verteilen, zu kopieren oder in irgendeiner Art zu benutzen. Sollten Sie diese Nachricht versehentlich erhalten haben, benachrichtigen Sie bitte den Absender und löschen und vernichten Sie jegliche Kopie davon, die Sie möglicherweise erhalten haben. Disclaimer: This message is intended only for the use of the individual or entity to which it is addressed and may contain information which is privileged, confidential, proprietary, or exempt from disclosure under applicable law. If you are not the intended recipient or the person responsible for delivering the message to the intended recipient, you are strictly prohibited from disclosing, distributing, copying, or in any way using this message. If you have received this communication in error, please notify the sender and destroy and delete any copies you may have received. |
From: Avi A. <av...@br...> - 2008-03-20 14:35:14
|
> -----Original Message----- > From: mod...@li... [mailto:mod- > sec...@li...] On Behalf Of Stefan M?ller- > Wilken > Sent: Thursday, March 20, 2008 3:17 PM > To: mod...@li... > Subject: [mod-security-users] Still content compression problems with 2.5? > > [This is a repost as I didn't see my post arrive on the list - sorry, if > it actually now has twice!] > > Dear all, > > we're still struggling with the infamous content compression problems with > a screened tomcat gzipping its output and flooding our reverse proxy logs > with "id 960903" errors. Two questions: > > a) Why can't I switch of 960903 using "SecRemoveRuleById "960903" as I can > with all other CRS rules? [Avi] Actually this should work. Do note that the name is SecRuleRemoveById, and not SecRemoveRuleById, and the directive should appear _after_ the rule itself. > > b) Does version 2.5 support combining mod_deflate with mod_security (e.g. > by defining the order in which the two are executed) to switch off content > compression on the app server and relocate it to the reverse proxy? [Avi] on an embedded system, modSecurity runs before mod_deflate on the reply, so inspection is possible. On a reverse proxy, the content is already compressed, so if you still need the compression to the backend, you can use apache's filter directives to define running mod_deflate to decompress, then mod_security, then mod_deflate again to recompress. > > Regards > Stef. > > > Resco GmbH > Geschäftsführer: Michael Mörchen > Amtsgericht Hamburg, HRB 76048 > Ust.Ident-Nr.:DE208833022 > > Haftungsausschluss: Diese Nachricht ist ausschließlich für die Person oder > Einheit bestimmt, an die sie gerichtet ist. Sie enthält unter Umständen > Informationen, die unter geltendem Recht vertraulich, gesetzlich geschützt > oder von der Offenlegung ausgeschlossen sind. Falls Sie nicht der > vorgesehene Empfänger oder verantwortlich für die Weiterleitung dieser > Nachricht an den vorgesehenen Empfänger sind, ist es Ihnen strengstens > untersagt, diese Nachricht offenzulegen, zu verteilen, zu kopieren oder in > irgendeiner Art zu benutzen. Sollten Sie diese Nachricht versehentlich > erhalten haben, benachrichtigen Sie bitte den Absender und löschen und > vernichten Sie jegliche Kopie davon, die Sie möglicherweise erhalten > haben. > > Disclaimer: This message is intended only for the use of the individual or > entity to which it is addressed and may contain information which is > privileged, confidential, proprietary, or exempt from disclosure under > applicable law. If you are not the intended recipient or the person > responsible for delivering the message to the intended recipient, you are > strictly prohibited from disclosing, distributing, copying, or in any way > using this message. If you have received this communication in error, > please notify the sender and destroy and delete any copies you may have > received. > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > mod-security-users mailing list > mod...@li... > https://lists.sourceforge.net/lists/listinfo/mod-security-users |
From: Ryan B. <Ryan.Barnett@Breach.com> - 2008-03-20 14:48:16
|
> -----Original Message----- > From: mod...@li... [mailto:mod- > sec...@li...] On Behalf Of Avi Aminov > Sent: Thursday, March 20, 2008 10:32 AM > To: Stefan Müller-Wilken; mod...@li... > Subject: Re: [mod-security-users] Still content compression problems > with2.5? > > > > -----Original Message----- > > From: mod...@li... [mailto:mod- > > sec...@li...] On Behalf Of Stefan > M?ller- > > Wilken > > Sent: Thursday, March 20, 2008 3:17 PM > > To: mod...@li... > > Subject: [mod-security-users] Still content compression problems with > 2.5? > > > > [This is a repost as I didn't see my post arrive on the list - sorry, if > > it actually now has twice!] > > > > Dear all, > > > > we're still struggling with the infamous content compression problems > with > > a screened tomcat gzipping its output and flooding our reverse proxy > logs > > with "id 960903" errors. Two questions: > > > > a) Why can't I switch of 960903 using "SecRemoveRuleById "960903" as I > can > > with all other CRS rules? > > [Avi] Actually this should work. Do note that the name is > SecRuleRemoveById, and not SecRemoveRuleById, and the directive should > appear _after_ the rule itself. > [Ryan Barnett] Also note that with 2.5, you have the "ctl:ruleRemoveById" action. Two benefits of the ctl action vs. the directive - 1) You can specify disabling a rule based on any Mod data. Most people either specify SecRuleRemoveById globally or they use the Apache Scope directives such as Location which only allows you to disable a rule based on the URI data. 2) Another nice feature is that you do not have to specify it AFTER the rule you are disabling like you have to do with SecRuleRemoveById. You can specify the ctl action anywhere in the config and Mod will disable the rule appropriately. This makes it a bit easier to avoid issues. > > > > b) Does version 2.5 support combining mod_deflate with mod_security > (e.g. > > by defining the order in which the two are executed) to switch off > content > > compression on the app server and relocate it to the reverse proxy? > > [Avi] on an embedded system, modSecurity runs before mod_deflate on the > reply, so inspection is possible. On a reverse proxy, the content is > already compressed, so if you still need the compression to the backend, > you can use apache's filter directives to define running mod_deflate to > decompress, then mod_security, then mod_deflate again to recompress. > [Ryan Barnett] What Avi outlines certainly would work, however there would be a hit on processing and latency. Ideally the process of compressing outbound content should be handled by the Mod reverse proxy so that the communication between the proxy and the backend web app is normal. Brian Rectanus may have more info along these lines as he is our resident guru for Apache/Mod interactions. |
From: Stefan Müller-W. <ste...@re...> - 2008-03-20 15:39:19
|
Thanks Avi and Ryan for your answers. Just to make sure I got that one right: does that mean that if I switch off content compression on the app server and enable mod_deflate and mod_security on the Apache httpd, I _should_ no longer get content compression blind spots with 2.5 and get rid of the 960903 while getting all benefits of ModSecurity? Cheers Stefan. ________ Von: Ryan Barnett [Ryan.Barnett@Breach.com] Gesendet: Donnerstag, 20. März 2008 15:48 An: Avi Aminov; Stefan Müller-Wilken; mod...@li... Betreff: RE: [mod-security-users] Still content compression problems with2.5? > -----Original Message----- > From: mod...@li... [mailto:mod- > sec...@li...] On Behalf Of Avi Aminov > Sent: Thursday, March 20, 2008 10:32 AM > To: Stefan Müller-Wilken; mod...@li... > Subject: Re: [mod-security-users] Still content compression problems > with2.5? > > > > -----Original Message----- > > From: mod...@li... [mailto:mod- > > sec...@li...] On Behalf Of Stefan > M?ller- > > Wilken > > Sent: Thursday, March 20, 2008 3:17 PM > > To: mod...@li... > > Subject: [mod-security-users] Still content compression problems with > 2.5? > > > > [This is a repost as I didn't see my post arrive on the list - sorry, if > > it actually now has twice!] > > > > Dear all, > > > > we're still struggling with the infamous content compression problems > with > > a screened tomcat gzipping its output and flooding our reverse proxy > logs > > with "id 960903" errors. Two questions: > > > > a) Why can't I switch of 960903 using "SecRemoveRuleById "960903" as I > can > > with all other CRS rules? > > [Avi] Actually this should work. Do note that the name is > SecRuleRemoveById, and not SecRemoveRuleById, and the directive should > appear _after_ the rule itself. > [Ryan Barnett] Also note that with 2.5, you have the "ctl:ruleRemoveById" action. Two benefits of the ctl action vs. the directive - 1) You can specify disabling a rule based on any Mod data. Most people either specify SecRuleRemoveById globally or they use the Apache Scope directives such as Location which only allows you to disable a rule based on the URI data. 2) Another nice feature is that you do not have to specify it AFTER the rule you are disabling like you have to do with SecRuleRemoveById. You can specify the ctl action anywhere in the config and Mod will disable the rule appropriately. This makes it a bit easier to avoid issues. > > > > b) Does version 2.5 support combining mod_deflate with mod_security > (e.g. > > by defining the order in which the two are executed) to switch off > content > > compression on the app server and relocate it to the reverse proxy? > > [Avi] on an embedded system, modSecurity runs before mod_deflate on the > reply, so inspection is possible. On a reverse proxy, the content is > already compressed, so if you still need the compression to the backend, > you can use apache's filter directives to define running mod_deflate to > decompress, then mod_security, then mod_deflate again to recompress. > [Ryan Barnett] What Avi outlines certainly would work, however there would be a hit on processing and latency. Ideally the process of compressing outbound content should be handled by the Mod reverse proxy so that the communication between the proxy and the backend web app is normal. Brian Rectanus may have more info along these lines as he is our resident guru for Apache/Mod interactions. Resco GmbH Geschäftsführer: Michael Mörchen Amtsgericht Hamburg, HRB 76048 Ust.Ident-Nr.:DE208833022 Haftungsausschluss: Diese Nachricht ist ausschließlich für die Person oder Einheit bestimmt, an die sie gerichtet ist. Sie enthält unter Umständen Informationen, die unter geltendem Recht vertraulich, gesetzlich geschützt oder von der Offenlegung ausgeschlossen sind. Falls Sie nicht der vorgesehene Empfänger oder verantwortlich für die Weiterleitung dieser Nachricht an den vorgesehenen Empfänger sind, ist es Ihnen strengstens untersagt, diese Nachricht offenzulegen, zu verteilen, zu kopieren oder in irgendeiner Art zu benutzen. Sollten Sie diese Nachricht versehentlich erhalten haben, benachrichtigen Sie bitte den Absender und löschen und vernichten Sie jegliche Kopie davon, die Sie möglicherweise erhalten haben. Disclaimer: This message is intended only for the use of the individual or entity to which it is addressed and may contain information which is privileged, confidential, proprietary, or exempt from disclosure under applicable law. If you are not the intended recipient or the person responsible for delivering the message to the intended recipient, you are strictly prohibited from disclosing, distributing, copying, or in any way using this message. If you have received this communication in error, please notify the sender and destroy and delete any copies you may have received. |
From: Brian R. <Bri...@br...> - 2008-03-20 17:59:24
|
As the others have mentioned, there are a few things you can do in a reverse proxy mode: 1) Turn off compression on all you backend machines. But you may not have access or the option to do this. Besides, it is a bit more risky to do this as you may rely on other administrators to not re-enable compression and to turn it off for any new backends. 2) Use compression on the reverse proxy for outgoing content to the client, but, for the backend, disable compression by not passing on the header(s) stating that compression is supported. This is what I would recommend and pretty easy to do. You just need to load mod_headers module and then remove a few HTTP request headers before sending to the backend: ... LoadModule headers_module modules/mod_headers.so ... LoadModule security2_module modules/mod_security2.so # Disable compression via mod_headers RequestHeader unset Accept-Encoding RequestHeader unset TE 3) Enable decompression of the content so ModSecurity can see it and then recompress to the client if they can deal with compressed content. This could be expensive and a little tricky. You will probably only want to do this if your bandwidth to the backend is limited or expensive. For this you need to load mod_deflate and mod_filter, set it up to decompress and then recompress only if the client said it could handle compressed content. Additionally, you probably do not want to recompress already compressed content such as images, etc. ... LoadModule filter_module modules/mod_filter.so LoadModule deflate_module modules/mod_deflate.so ... LoadModule security2_module modules/mod_security2.so # Decompress the data as early as possible so ModSecurity sees the # decompressed data. Additionally, set an environment var as a flag # for images and/or other types so we do not compress them and add # a Vary header if we are compressing. SetOutputFilter INFLATE SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary Header append Vary User-Agent env=!dont-vary # Next, setup a filter chain so ModSecurity gets the data first, # then it is re-compressed afterwords, but only if the string # "gzip" in in the request Content-Encoding header. FilterDeclare modsec CONTENT_SET FilterProvider modsec modsecurity_out env=modsec-ignore !=1 FilterDeclare compress CONTENT_SET FilterProvider compress deflate resp=Content-Encoding $gzip FilterProtocol compress "change=yes" # Some debugging to verify it is working, comment out for production FilterTrace modsec 1 FilterTrace compress 1 # The filter chain should go in this order FilterChain modsec compress As you can see that gets a bit complex ;) So I would stick with #2 above :) Hope that helps. Eventually I'll type this up in a nicer format in a blog. But I just have not had the time to do so lately. thanks, -B Stefan Müller-Wilken wrote: > Thanks Avi and Ryan for your answers. > > Just to make sure I got that one right: does that mean that if I switch > off content compression on the app server and enable mod_deflate and > mod_security on the Apache httpd, I _should_ no longer get content > compression blind spots with 2.5 and get rid of the 960903 while getting > all benefits of ModSecurity? > > Cheers > Stefan. > > ________ > Von: Ryan Barnett [Ryan.Barnett@Breach.com] > Gesendet: Donnerstag, 20. März 2008 15:48 > An: Avi Aminov; Stefan Müller-Wilken; > mod...@li... > Betreff: RE: [mod-security-users] Still content compression problems > with2.5? > >> -----Original Message----- >> From: mod...@li... [mailto:mod- >> sec...@li...] On Behalf Of Avi Aminov >> Sent: Thursday, March 20, 2008 10:32 AM >> To: Stefan Müller-Wilken; mod...@li... >> Subject: Re: [mod-security-users] Still content compression problems >> with2.5? >> >> >> > -----Original Message----- >> > From: mod...@li... [mailto:mod- >> > sec...@li...] On Behalf Of Stefan >> M?ller- >> > Wilken >> > Sent: Thursday, March 20, 2008 3:17 PM >> > To: mod...@li... >> > Subject: [mod-security-users] Still content compression problems with >> 2.5? >> > >> > [This is a repost as I didn't see my post arrive on the list - sorry, if >> > it actually now has twice!] >> > >> > Dear all, >> > >> > we're still struggling with the infamous content compression problems >> with >> > a screened tomcat gzipping its output and flooding our reverse proxy >> logs >> > with "id 960903" errors. Two questions: >> > >> > a) Why can't I switch of 960903 using "SecRemoveRuleById "960903" as I >> can >> > with all other CRS rules? >> >> [Avi] Actually this should work. Do note that the name is >> SecRuleRemoveById, and not SecRemoveRuleById, and the directive should >> appear _after_ the rule itself. >> > [Ryan Barnett] Also note that with 2.5, you have the > "ctl:ruleRemoveById" action. Two benefits of the ctl action vs. the > directive - > > 1) You can specify disabling a rule based on any Mod data. Most people > either specify SecRuleRemoveById globally or they use the Apache Scope > directives such as Location which only allows you to disable a rule > based on the URI data. > > 2) Another nice feature is that you do not have to specify it AFTER the > rule you are disabling like you have to do with SecRuleRemoveById. You > can specify the ctl action anywhere in the config and Mod will disable > the rule appropriately. This makes it a bit easier to avoid issues. > >> > >> > b) Does version 2.5 support combining mod_deflate with mod_security >> (e.g. >> > by defining the order in which the two are executed) to switch off >> content >> > compression on the app server and relocate it to the reverse proxy? >> >> [Avi] on an embedded system, modSecurity runs before mod_deflate on the >> reply, so inspection is possible. On a reverse proxy, the content is >> already compressed, so if you still need the compression to the backend, >> you can use apache's filter directives to define running mod_deflate to >> decompress, then mod_security, then mod_deflate again to recompress. >> > [Ryan Barnett] What Avi outlines certainly would work, however there > would be a hit on processing and latency. Ideally the process of > compressing outbound content should be handled by the Mod reverse proxy > so that the communication between the proxy and the backend web app is > normal. > > Brian Rectanus may have more info along these lines as he is our > resident guru for Apache/Mod interactions. > > > Resco GmbH > Geschäftsführer: Michael Mörchen > Amtsgericht Hamburg, HRB 76048 > Ust.Ident-Nr.:DE208833022 > > Haftungsausschluss: Diese Nachricht ist ausschließlich für die Person > oder Einheit bestimmt, an die sie gerichtet ist. Sie enthält unter > Umständen Informationen, die unter geltendem Recht vertraulich, > gesetzlich geschützt oder von der Offenlegung ausgeschlossen sind. Falls > Sie nicht der vorgesehene Empfänger oder verantwortlich für die > Weiterleitung dieser Nachricht an den vorgesehenen Empfänger sind, ist > es Ihnen strengstens untersagt, diese Nachricht offenzulegen, zu > verteilen, zu kopieren oder in irgendeiner Art zu benutzen. Sollten Sie > diese Nachricht versehentlich erhalten haben, benachrichtigen Sie bitte > den Absender und löschen und vernichten Sie jegliche Kopie davon, die > Sie möglicherweise erhalten haben. > > Disclaimer: This message is intended only for the use of the individual > or entity to which it is addressed and may contain information which is > privileged, confidential, proprietary, or exempt from disclosure under > applicable law. If you are not the intended recipient or the person > responsible for delivering the message to the intended recipient, you > are strictly prohibited from disclosing, distributing, copying, or in > any way using this message. If you have received this communication in > error, please notify the sender and destroy and delete any copies you > may have received. > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > mod-security-users mailing list > mod...@li... > https://lists.sourceforge.net/lists/listinfo/mod-security-users > -- Brian Rectanus Breach Security |
From: Stefan Müller-W. <ste...@re...> - 2008-03-20 20:19:25
|
Wow, Brian, that's cool stuff! I'm quite sure you could just copy your reply into the FAQ and make a lot of (potential) ModSecurity users quite happy :-) Just for completeness: with #2, you skipped mod_deflate's directives in your description, right? ... LoadModule deflate_module modules/mod_deflate.so ... SetOutputFilter DEFLATE ... Again, thanks for the quick and competent answer! Cheers, Stefan. ________________________________________ Von: Brian Rectanus [Bri...@br...] Gesendet: Donnerstag, 20. März 2008 18:59 An: Stefan Müller-Wilken Cc: mod...@li... Betreff: Re: [mod-security-users] Still content compression problems with2.5? As the others have mentioned, there are a few things you can do in a reverse proxy mode: 1) Turn off compression on all you backend machines. But you may not have access or the option to do this. Besides, it is a bit more risky to do this as you may rely on other administrators to not re-enable compression and to turn it off for any new backends. 2) Use compression on the reverse proxy for outgoing content to the client, but, for the backend, disable compression by not passing on the header(s) stating that compression is supported. This is what I would recommend and pretty easy to do. You just need to load mod_headers module and then remove a few HTTP request headers before sending to the backend: ... LoadModule headers_module modules/mod_headers.so ... LoadModule security2_module modules/mod_security2.so # Disable compression via mod_headers RequestHeader unset Accept-Encoding RequestHeader unset TE 3) Enable decompression of the content so ModSecurity can see it and then recompress to the client if they can deal with compressed content. This could be expensive and a little tricky. You will probably only want to do this if your bandwidth to the backend is limited or expensive. For this you need to load mod_deflate and mod_filter, set it up to decompress and then recompress only if the client said it could handle compressed content. Additionally, you probably do not want to recompress already compressed content such as images, etc. ... LoadModule filter_module modules/mod_filter.so LoadModule deflate_module modules/mod_deflate.so ... LoadModule security2_module modules/mod_security2.so # Decompress the data as early as possible so ModSecurity sees the # decompressed data. Additionally, set an environment var as a flag # for images and/or other types so we do not compress them and add # a Vary header if we are compressing. SetOutputFilter INFLATE SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary Header append Vary User-Agent env=!dont-vary # Next, setup a filter chain so ModSecurity gets the data first, # then it is re-compressed afterwords, but only if the string # "gzip" in in the request Content-Encoding header. FilterDeclare modsec CONTENT_SET FilterProvider modsec modsecurity_out env=modsec-ignore !=1 FilterDeclare compress CONTENT_SET FilterProvider compress deflate resp=Content-Encoding $gzip FilterProtocol compress "change=yes" # Some debugging to verify it is working, comment out for production FilterTrace modsec 1 FilterTrace compress 1 # The filter chain should go in this order FilterChain modsec compress As you can see that gets a bit complex ;) So I would stick with #2 above :) Hope that helps. Eventually I'll type this up in a nicer format in a blog. But I just have not had the time to do so lately. thanks, -B Stefan Müller-Wilken wrote: > Thanks Avi and Ryan for your answers. > > Just to make sure I got that one right: does that mean that if I switch > off content compression on the app server and enable mod_deflate and > mod_security on the Apache httpd, I _should_ no longer get content > compression blind spots with 2.5 and get rid of the 960903 while getting > all benefits of ModSecurity? > > Cheers > Stefan. > > ________ > Von: Ryan Barnett [Ryan.Barnett@Breach.com] > Gesendet: Donnerstag, 20. März 2008 15:48 > An: Avi Aminov; Stefan Müller-Wilken; > mod...@li... > Betreff: RE: [mod-security-users] Still content compression problems > with2.5? > >> -----Original Message----- >> From: mod...@li... [mailto:mod- >> sec...@li...] On Behalf Of Avi Aminov >> Sent: Thursday, March 20, 2008 10:32 AM >> To: Stefan Müller-Wilken; mod...@li... >> Subject: Re: [mod-security-users] Still content compression problems >> with2.5? >> >> >> > -----Original Message----- >> > From: mod...@li... [mailto:mod- >> > sec...@li...] On Behalf Of Stefan >> M?ller- >> > Wilken >> > Sent: Thursday, March 20, 2008 3:17 PM >> > To: mod...@li... >> > Subject: [mod-security-users] Still content compression problems with >> 2.5? >> > >> > [This is a repost as I didn't see my post arrive on the list - sorry, if >> > it actually now has twice!] >> > >> > Dear all, >> > >> > we're still struggling with the infamous content compression problems >> with >> > a screened tomcat gzipping its output and flooding our reverse proxy >> logs >> > with "id 960903" errors. Two questions: >> > >> > a) Why can't I switch of 960903 using "SecRemoveRuleById "960903" as I >> can >> > with all other CRS rules? >> >> [Avi] Actually this should work. Do note that the name is >> SecRuleRemoveById, and not SecRemoveRuleById, and the directive should >> appear _after_ the rule itself. >> > [Ryan Barnett] Also note that with 2.5, you have the > "ctl:ruleRemoveById" action. Two benefits of the ctl action vs. the > directive - > > 1) You can specify disabling a rule based on any Mod data. Most people > either specify SecRuleRemoveById globally or they use the Apache Scope > directives such as Location which only allows you to disable a rule > based on the URI data. > > 2) Another nice feature is that you do not have to specify it AFTER the > rule you are disabling like you have to do with SecRuleRemoveById. You > can specify the ctl action anywhere in the config and Mod will disable > the rule appropriately. This makes it a bit easier to avoid issues. > >> > >> > b) Does version 2.5 support combining mod_deflate with mod_security >> (e.g. >> > by defining the order in which the two are executed) to switch off >> content >> > compression on the app server and relocate it to the reverse proxy? >> >> [Avi] on an embedded system, modSecurity runs before mod_deflate on the >> reply, so inspection is possible. On a reverse proxy, the content is >> already compressed, so if you still need the compression to the backend, >> you can use apache's filter directives to define running mod_deflate to >> decompress, then mod_security, then mod_deflate again to recompress. >> > [Ryan Barnett] What Avi outlines certainly would work, however there > would be a hit on processing and latency. Ideally the process of > compressing outbound content should be handled by the Mod reverse proxy > so that the communication between the proxy and the backend web app is > normal. > > Brian Rectanus may have more info along these lines as he is our > resident guru for Apache/Mod interactions. Resco GmbH Geschäftsführer: Michael Mörchen Amtsgericht Hamburg, HRB 76048 Ust.Ident-Nr.:DE208833022 Haftungsausschluss: Diese Nachricht ist ausschließlich für die Person oder Einheit bestimmt, an die sie gerichtet ist. Sie enthält unter Umständen Informationen, die unter geltendem Recht vertraulich, gesetzlich geschützt oder von der Offenlegung ausgeschlossen sind. Falls Sie nicht der vorgesehene Empfänger oder verantwortlich für die Weiterleitung dieser Nachricht an den vorgesehenen Empfänger sind, ist es Ihnen strengstens untersagt, diese Nachricht offenzulegen, zu verteilen, zu kopieren oder in irgendeiner Art zu benutzen. Sollten Sie diese Nachricht versehentlich erhalten haben, benachrichtigen Sie bitte den Absender und löschen und vernichten Sie jegliche Kopie davon, die Sie möglicherweise erhalten haben. Disclaimer: This message is intended only for the use of the individual or entity to which it is addressed and may contain information which is privileged, confidential, proprietary, or exempt from disclosure under applicable law. If you are not the intended recipient or the person responsible for delivering the message to the intended recipient, you are strictly prohibited from disclosing, distributing, copying, or in any way using this message. If you have received this communication in error, please notify the sender and destroy and delete any copies you may have received. |
From: Brian R. <Bri...@br...> - 2008-03-20 22:27:59
|
Stefan Müller-Wilken wrote: > Wow, Brian, that's cool stuff! I'm quite sure you could just copy your > reply into the FAQ and make a lot of (potential) ModSecurity users quite > happy :-) > > Just for completeness: with #2, you skipped mod_deflate's directives in > your description, right? Ah, yes, good catch. You still need to configure mod_deflate if you want compression to the client. What you have below looks fine off hand. -B > > ... > LoadModule deflate_module modules/mod_deflate.so > ... > > SetOutputFilter DEFLATE > ... > > Again, thanks for the quick and competent answer! > > Cheers, > Stefan. > > ________________________________________ > Von: Brian Rectanus [Bri...@br...] > Gesendet: Donnerstag, 20. März 2008 18:59 > An: Stefan Müller-Wilken > Cc: mod...@li... > Betreff: Re: [mod-security-users] Still content compression problems > with2.5? > > As the others have mentioned, there are a few things you can do in a > reverse proxy mode: > > 1) Turn off compression on all you backend machines. But you may not > have access or the option to do this. Besides, it is a bit more risky > to do this as you may rely on other administrators to not re-enable > compression and to turn it off for any new backends. > > > 2) Use compression on the reverse proxy for outgoing content to the > client, but, for the backend, disable compression by not passing on the > header(s) stating that compression is supported. This is what I would > recommend and pretty easy to do. You just need to load mod_headers > module and then remove a few HTTP request headers before sending to the > backend: > > ... > LoadModule headers_module modules/mod_headers.so > ... > LoadModule security2_module modules/mod_security2.so > > # Disable compression via mod_headers > RequestHeader unset Accept-Encoding > RequestHeader unset TE > > > 3) Enable decompression of the content so ModSecurity can see it and > then recompress to the client if they can deal with compressed content. > This could be expensive and a little tricky. You will probably only > want to do this if your bandwidth to the backend is limited or expensive. > > For this you need to load mod_deflate and mod_filter, set it up to > decompress and then recompress only if the client said it could handle > compressed content. Additionally, you probably do not want to > recompress already compressed content such as images, etc. > > ... > LoadModule filter_module modules/mod_filter.so > LoadModule deflate_module modules/mod_deflate.so > ... > LoadModule security2_module modules/mod_security2.so > > > # Decompress the data as early as possible so ModSecurity sees the > # decompressed data. Additionally, set an environment var as a flag > # for images and/or other types so we do not compress them and add > # a Vary header if we are compressing. > SetOutputFilter INFLATE > SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary > Header append Vary User-Agent env=!dont-vary > > > # Next, setup a filter chain so ModSecurity gets the data first, > # then it is re-compressed afterwords, but only if the string > # "gzip" in in the request Content-Encoding header. > FilterDeclare modsec CONTENT_SET > FilterProvider modsec modsecurity_out env=modsec-ignore !=1 > FilterDeclare compress CONTENT_SET > FilterProvider compress deflate resp=Content-Encoding $gzip > FilterProtocol compress "change=yes" > # Some debugging to verify it is working, comment out for production > FilterTrace modsec 1 > FilterTrace compress 1 > # The filter chain should go in this order > FilterChain modsec compress > > As you can see that gets a bit complex ;) So I would stick with #2 above :) > > Hope that helps. Eventually I'll type this up in a nicer format in a > blog. But I just have not had the time to do so lately. > > thanks, > -B > > Stefan Müller-Wilken wrote: >> Thanks Avi and Ryan for your answers. >> >> Just to make sure I got that one right: does that mean that if I switch >> off content compression on the app server and enable mod_deflate and >> mod_security on the Apache httpd, I _should_ no longer get content >> compression blind spots with 2.5 and get rid of the 960903 while getting >> all benefits of ModSecurity? >> >> Cheers >> Stefan. >> >> ________ >> Von: Ryan Barnett [Ryan.Barnett@Breach.com] >> Gesendet: Donnerstag, 20. März 2008 15:48 >> An: Avi Aminov; Stefan Müller-Wilken; >> mod...@li... >> Betreff: RE: [mod-security-users] Still content compression problems >> with2.5? >> >>> -----Original Message----- >>> From: mod...@li... [mailto:mod- >>> sec...@li...] On Behalf Of Avi Aminov >>> Sent: Thursday, March 20, 2008 10:32 AM >>> To: Stefan Müller-Wilken; mod...@li... >>> Subject: Re: [mod-security-users] Still content compression problems >>> with2.5? >>> >>> >>> > -----Original Message----- >>> > From: mod...@li... [mailto:mod- >>> > sec...@li...] On Behalf Of Stefan >>> M?ller- >>> > Wilken >>> > Sent: Thursday, March 20, 2008 3:17 PM >>> > To: mod...@li... >>> > Subject: [mod-security-users] Still content compression problems with >>> 2.5? >>> > >>> > [This is a repost as I didn't see my post arrive on the list - > sorry, if >>> > it actually now has twice!] >>> > >>> > Dear all, >>> > >>> > we're still struggling with the infamous content compression problems >>> with >>> > a screened tomcat gzipping its output and flooding our reverse proxy >>> logs >>> > with "id 960903" errors. Two questions: >>> > >>> > a) Why can't I switch of 960903 using "SecRemoveRuleById "960903" as I >>> can >>> > with all other CRS rules? >>> >>> [Avi] Actually this should work. Do note that the name is >>> SecRuleRemoveById, and not SecRemoveRuleById, and the directive should >>> appear _after_ the rule itself. >>> >> [Ryan Barnett] Also note that with 2.5, you have the >> "ctl:ruleRemoveById" action. Two benefits of the ctl action vs. the >> directive - >> >> 1) You can specify disabling a rule based on any Mod data. Most people >> either specify SecRuleRemoveById globally or they use the Apache Scope >> directives such as Location which only allows you to disable a rule >> based on the URI data. >> >> 2) Another nice feature is that you do not have to specify it AFTER the >> rule you are disabling like you have to do with SecRuleRemoveById. You >> can specify the ctl action anywhere in the config and Mod will disable >> the rule appropriately. This makes it a bit easier to avoid issues. >> >>> > >>> > b) Does version 2.5 support combining mod_deflate with mod_security >>> (e.g. >>> > by defining the order in which the two are executed) to switch off >>> content >>> > compression on the app server and relocate it to the reverse proxy? >>> >>> [Avi] on an embedded system, modSecurity runs before mod_deflate on the >>> reply, so inspection is possible. On a reverse proxy, the content is >>> already compressed, so if you still need the compression to the backend, >>> you can use apache's filter directives to define running mod_deflate to >>> decompress, then mod_security, then mod_deflate again to recompress. >>> >> [Ryan Barnett] What Avi outlines certainly would work, however there >> would be a hit on processing and latency. Ideally the process of >> compressing outbound content should be handled by the Mod reverse proxy >> so that the communication between the proxy and the backend web app is >> normal. >> >> Brian Rectanus may have more info along these lines as he is our >> resident guru for Apache/Mod interactions. > > Resco GmbH > Geschäftsführer: Michael Mörchen > Amtsgericht Hamburg, HRB 76048 > Ust.Ident-Nr.:DE208833022 > > Haftungsausschluss: Diese Nachricht ist ausschließlich für die Person > oder Einheit bestimmt, an die sie gerichtet ist. Sie enthält unter > Umständen Informationen, die unter geltendem Recht vertraulich, > gesetzlich geschützt oder von der Offenlegung ausgeschlossen sind. Falls > Sie nicht der vorgesehene Empfänger oder verantwortlich für die > Weiterleitung dieser Nachricht an den vorgesehenen Empfänger sind, ist > es Ihnen strengstens untersagt, diese Nachricht offenzulegen, zu > verteilen, zu kopieren oder in irgendeiner Art zu benutzen. Sollten Sie > diese Nachricht versehentlich erhalten haben, benachrichtigen Sie bitte > den Absender und löschen und vernichten Sie jegliche Kopie davon, die > Sie möglicherweise erhalten haben. > > Disclaimer: This message is intended only for the use of the individual > or entity to which it is addressed and may contain information which is > privileged, confidential, proprietary, or exempt from disclosure under > applicable law. If you are not the intended recipient or the person > responsible for delivering the message to the intended recipient, you > are strictly prohibited from disclosing, distributing, copying, or in > any way using this message. If you have received this communication in > error, please notify the sender and destroy and delete any copies you > may have received. > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > mod-security-users mailing list > mod...@li... > https://lists.sourceforge.net/lists/listinfo/mod-security-users > -- Brian Rectanus Breach Security |