mod-security-users — General discussion about mod_security

Re: [mod-security-users] PCRE limits exceeded

[Jamie B. <ja...@ib...>]

I'm hitting this too and have been gradually increasing from the default. Is this somewhat dependent on CPU speed?

Sent from my iPhone

> On 19 Jun 2020, at 08:12, Christian Folini <chr...@ne...> wrote:
> 
> Mahmood,
> 
> This is a standard problem when using ModSec due to the PCRE library used.
> 
> 500K is near the highest sane value in production. Go higher and you make
> a DoS attack more and more likely.
> 
> If 500K does not solve it, then I would suggest to disable this rule
> for this URI. It is possible that other response-rules show the same
> symptoms. In that situation, disabling ResponseBody access for the
> given URI could be a valid alternative.
> 
> One word of warning: I recommend to disable rules. This may lead to 
> insecurity in this situation. One would need to assess the situation
> if it is worth it.
> 
> Best,
> 
> Christian
> 
> 
> 
>> On Fri, Jun 19, 2020 at 06:16:25AM +0000, Mahmood Naderan via mod-security-users wrote:
>> Hi
>> I see some entries like
>> 
>> [Thu Jun 18 11:22:36.512669 2020] [:error] [pid 129057] [client XXXXXXX:20101] [client XXXXXX] ModSecurity: Rule 7f26def146a0 [id "-"][file "/etc/modsecurity/rules/RESPONSE-951-DATA-LEAKAGES-SQL.conf"][line "433"] - Execution error - PCRE limits exceeded (-8): (null). [hostname "DOMAIN.COM"] [uri "/mod/assign/view.php"] [unique_id "XusPM87nvNAwDeCAa568uQAAABo"], referer: https:// DOMAIN.COM/mod/assign/view.php?id=37801
>> 
>> 
>> 
>> I have checked the documents and some stated to set
>> SecPcreMatchLimit 500000
>> SecPcreMatchLimitRecursion 500000
>> 
>> 
>> In /etc/modsecurity/modsecurity.conf but I am not sure about that. I don't know if a high or low value is recommended.
>> 
>> 
>> Regards,
>> Mahmood
> 
> 
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
> 
> 
> 
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] Large Payload processing time

[Madden, J. <Joe...@mo...>]

Hi Manuel,

I have this set – So I thought would have used the correct processor:

    SecRule REQUEST_HEADERS:Content-Type "text/xml" \
         "id:'200000',phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=XML"

Thanks,

Joe.
Joe Madden
Senior Systems Engineer
D 01412224666
joe...@mo...<mailto:joe...@mo...>

From: Manuel Spartan <spa...@gm...>
Sent: 13 June 2020 00:03
To: mod...@li...
Subject: Re: [mod-security-users] Large Payload processing time

Hi Joe, looks like you’re using the Default body processor on xml payloads which causes a lot of problems. Try setting it to XML based on the request_uri with ctl:requestBodyProcessor=XML
Regards,
Manuel
Sent from my iPhone


On Jun 11, 2020, at 9:47 AM, Madden, Joe via mod-security-users <mod...@li...<mailto:mod...@li...>> wrote:
Hi all,

I've had to disable the following rules in order to get a payload to process in a resonable amount of time.

It a XML payload with up to 20Mb in size, These are the rules which cause the processing from from around 30 seconds to  772 seconds


           # Disables checking for Windows command injection
           SecRuleRemoveById 932110

           #Removes unix command injection filtering
           SecRuleRemoveById 932100

           #Removes unix command injection filtering 2
           #SecRuleRemoveById 932105

           #removes unix remote code exceuction
           #SecRuleRemoveById 932150

           #Disables Oracle WebLogic Remote Command Execution exploit
           #SecRuleRemoveById 932115

           #Disables PHPIDS - Converted SQLI Filters - Not required
           #SecRuleRemoveById 942230

           #Disables PHPIDS - Converted SQLI Filters - Not required
           #SecRuleRemoveById 942190

           #Disables HTTP Response Splitting - Not Required
           #SecRuleRemoveById 921120

           # Disables Sources for SQL ALTER statements
           #SecRuleRemoveById 942360

           #Disables XSS Filters - Category 3 - Not required
           #SecRuleRemoveById 941130

           #Disables XSS [NoScript InjectionChecker] Attributes injection - Not required
           #SecRuleRemoveById 941170

           #Disables XSS vectors making use of event handlers like onerror, onload
           #SecRuleRemoveById 941120

I'll have times by the end of the day which rules take the longest but for example - Does anyone have any recommendations about this? We'd like to leave the uinix RCE and command filters on at this is what our platform is.

Thanks

Joe.

_______________________________________________
mod-security-users mailing list
mod...@li...<mailto:mod...@li...>
https://lists.sourceforge.net/lists/listinfo/mod-security-users
Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
http://www.modsecurity.org/projects/commercial/rules/
http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] PCRE limits exceeded

[Christian F. <chr...@ne...>]

Mahmood,

This is a standard problem when using ModSec due to the PCRE library used.

500K is near the highest sane value in production. Go higher and you make
a DoS attack more and more likely.

If 500K does not solve it, then I would suggest to disable this rule
for this URI. It is possible that other response-rules show the same
symptoms. In that situation, disabling ResponseBody access for the
given URI could be a valid alternative.

One word of warning: I recommend to disable rules. This may lead to 
insecurity in this situation. One would need to assess the situation
if it is worth it.

Best,

Christian



On Fri, Jun 19, 2020 at 06:16:25AM +0000, Mahmood Naderan via mod-security-users wrote:
> Hi
> I see some entries like
> 
> [Thu Jun 18 11:22:36.512669 2020] [:error] [pid 129057] [client XXXXXXX:20101] [client XXXXXX] ModSecurity: Rule 7f26def146a0 [id "-"][file "/etc/modsecurity/rules/RESPONSE-951-DATA-LEAKAGES-SQL.conf"][line "433"] - Execution error - PCRE limits exceeded (-8): (null). [hostname "DOMAIN.COM"] [uri "/mod/assign/view.php"] [unique_id "XusPM87nvNAwDeCAa568uQAAABo"], referer: https:// DOMAIN.COM/mod/assign/view.php?id=37801
> 
> 
> 
> I have checked the documents and some stated to set
> SecPcreMatchLimit 500000
> SecPcreMatchLimitRecursion 500000
> 
> 
> In /etc/modsecurity/modsecurity.conf but I am not sure about that. I don't know if a high or low value is recommended.
> 
> 
> Regards,
> Mahmood


> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

[mod-security-users] PCRE limits exceeded

[Mahmood N. <nt_...@ya...>]

Hi
I see some entries like

[Thu Jun 18 11:22:36.512669 2020] [:error] [pid 129057] [client XXXXXXX:20101] [client XXXXXX] ModSecurity: Rule 7f26def146a0 [id "-"][file "/etc/modsecurity/rules/RESPONSE-951-DATA-LEAKAGES-SQL.conf"][line "433"] - Execution error - PCRE limits exceeded (-8): (null). [hostname "DOMAIN.COM"] [uri "/mod/assign/view.php"] [unique_id "XusPM87nvNAwDeCAa568uQAAABo"], referer: https:// DOMAIN.COM/mod/assign/view.php?id=37801



I have checked the documents and some stated to set
SecPcreMatchLimit 500000
SecPcreMatchLimitRecursion 500000


In /etc/modsecurity/modsecurity.conf but I am not sure about that. I don't know if a high or low value is recommended.


Regards,
Mahmood

Re: [mod-security-users] Possible to remove rules by multiple tags?

[Christian F. <chr...@ne...>]

This is quite neat, Ervin! Thanks for sharing.

Christian

On Wed, Jun 17, 2020 at 11:55:19AM +0200, Ervin Hegedüs wrote:
> Hi Jamie,
> 
> as Christian wrote there isn't any solution to remove a rule by multiple
> tags.
> 
> But there is an indirect solution: you can find all rules where the listed
> tags exists.
> 
> There is a small tool, named msc_pyparser[1]. This Python library can parse
> CRS rules and makes the AST (abstract syntax tree) in YAML or JSON format.
> 
> I attached a Python script which loads these rules and search all id where
> the tags above listed. Before you run, you have to install that Python
> library (it works only with Python3), it's available through PIP. First,
> you have to build the AST files, then run script for each file, like:
> 
> for y in `ls -1 export/*.yaml`; do ./crs_gettags.py ${y}; done
> 
> and you'll see something like this:
> 
> SecRuleRemoveById 942110
> SecRuleRemoveById 942120
> SecRuleRemoveById 942130
> SecRuleRemoveById 942150
> SecRuleRemoveById 942180
> SecRuleRemoveById 942200
> SecRuleRemoveById 942210
> SecRuleRemoveById 942260
> SecRuleRemoveById 942300
> SecRuleRemoveById 942310
> SecRuleRemoveById 942330
> SecRuleRemoveById 942340
> SecRuleRemoveById 942361
> SecRuleRemoveById 942370
> SecRuleRemoveById 942380
> SecRuleRemoveById 942390
> SecRuleRemoveById 942400
> SecRuleRemoveById 942410
> SecRuleRemoveById 942470
> SecRuleRemoveById 942480
> SecRuleRemoveById 942430
> SecRuleRemoveById 942440
> SecRuleRemoveById 942450
> SecRuleRemoveById 942510
> 
> Just paste these lines into your exceptions, and hope that will give you
> what you want.
> 
> 
> Regards,
> 
> 
> a.
> 
> 
> [1]: https://github.com/digitalwave/msc_pyparser
> 
> 
> 
> 
> On Wed, Jun 17, 2020 at 1:01 AM Jamie Burchell <ja...@ib...> wrote:
> 
> > Hi
> >
> >
> >
> > Is it possible to remove rules by more than one tag? For example, remove
> > all “paranoia-level/2” “attack-sqli” CRS rules.
> >
> >
> >
> > This would be useful in situations where PL2 is in use, but certain groups
> > of rules should not be at PL2. I was looking at doing this by ID range
> > instead, but the IDs don’t seem facilitate ranges based on PL.
> >
> >
> >
> > Regards,
> >
> > Jamie
> > _______________________________________________
> > mod-security-users mailing list
> > mod...@li...
> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> > http://www.modsecurity.org/projects/commercial/rules/
> > http://www.modsecurity.org/projects/commercial/support/
> >

> #!/usr/bin/env python3
> 
> import sys
> import yaml
> 
> class Transform(object):
>     def __init__(self, data):
>         self.data = data
>         self.lineno = 1
>         self.current_ruleid = 0
>         self.chained = False
>         self.chainlevel = 0
> 
>     def gettag(self):
>         tags = []
>         for d in self.data:
>             if "actions" in d:
>                 aidx = 0
>                 if self.chained == True:
>                     self.chained = False
>                 while aidx < len(d['actions']):
>                     a = d['actions'][aidx]
> 
>                     if a['act_name'] == "tag":
>                         tags.append(a['act_arg'])
> 
>                     if a['act_name'] == "id":
>                         self.current_ruleid = int(a['act_arg'])
> 
>                     if a['act_name'] == "chain":
>                         self.chained = True
>                         self.chainlevel += 1
>                     aidx += 1
> 
>                 if self.chained == False:
>                     if "paranoia-level/2" in tags and "attack-sqli" in tags:
>                         print("SecRuleRemoveById %d" % (self.current_ruleid))
>                     self.current_ruleid = 0
>                     tags = []
> 
> 
> 
> if __name__ == "__main__":
>     if len(sys.argv) < 2:
>         print("Argument missing!")
>         print("Use: %s input" % (sys.argv[0]))
>         sys.exit(-1)
> 
>     fname = sys.argv[1]
>     try:
>         with open(fname, 'r') as inputfile:
>             if yaml.__version__ >= "5.1":
>                 data = yaml.load(inputfile, Loader=yaml.FullLoader)
>             else:
>                 data = yaml.load(inputfile)
>     except:
>         print("Can't open file: %s" % (fname))
>         sys.exit()
> 
>     t = Transform(data)
>     t.gettag()
> 


> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

[mod-security-users] OWASP ModSecurity Core Rule Set v3.3.0 Release Candidate 2 available

[Walter H. <mo...@sp...>]

The OWASP ModSecurity Core Rule Set team is proud to announce the release candidate 2 for the upcoming CRS v3.3.0 release. The release candidate is available at:

https://github.com/coreruleset/coreruleset/archive/v3.3.0-rc2.tar.gz <https://github.com/coreruleset/coreruleset/archive/v3.3.0-rc2.tar.gz>
https://github.com/coreruleset/coreruleset/archive/v3.3.0-rc2.zip <https://github.com/coreruleset/coreruleset/archive/v3.3.0-rc2.zip>
This release packages many changes, such as:

Block backup files ending with ~ in filename (Andrea Menin)
Detect ffuf vuln scanner (Will Woodson)
Detect SemrushBot crawler (Christian Folini)
Detect WFuzz vuln scanner (azurit)
New LDAP injection rule (Christian Folini)
New HTTP Splitting rule (Andrea Menin)
Add .swp to restricted extensions (Andrea Menin)
Allow CloudEvents content types (Bobby Earl)
Add CAPEC tags for attack classification (Fernando Outeda, Christian Folini)
Detect Unix RCE bypass techniques via uninitialized variables, string concatenations and globbing patterns (Andrea Menin)
Important note: The format of configuration setting allowed_request_content_type has been changed to be more in line with other variables. If you had manually changed this setting, then you need to update this configuration setting. Please see the example rule 900220 in crs-setup.conf.example. If you didn’t change this setting, you don’t need to do anything.

If you had tested RC1, the list of changes between RC1 and RC2 are as follows:

New functionality:

Add CloudEvents content types (Bobby Earl)
Add CAPEC tags for attack classification (Fernando Outeda, Christian Folini)
Detect RCE bypass techniques via uninitialized variables, string concatenations and globbing patterns (Andrea Menin)
Removed functionality:

Removed rule tags WASCTC, OWASP_TOP_10, OWASP_AppSensor/RE1, and OWASP_CRS/FOO/BAR; note that tags ‘OWASP_CRS’ and ‘attack-type’ are kept. (Christian Folini)
Fixes and improvements:

Prevent bypass of rule 921110 (Amit Klein, Franziska Bühler)
fix CVE msg in rules 944120 944240 (Fernando Outeda)
Remove broken or no longer used files (Federico G. Schwindt)
Make content-type case insensitive (franbuehler)
Move /util/docker folder from v3.3/dev branch to dedicated repo (Peter Bittner)
feat(lint): split actions in linting and regression (Felipe Zipitria)
Fix FP in 921120 (Amit Klein, Franziska Bühler)
Add missing OWASP_CRS tags (Christian Folini)
Fix GHA badges (Federico G. Schwindt)
feat(badge): add apache license badge
fix typos found by fossies codespell (Tim Herren)
Decrease processing time of rules (Ervin Hegedüs)
handle multiple directives in 920510 (themiddleblue)
Please see the CHANGES document with around 160 entries for the complete list of new features and improvements: https://github.com/coreruleset/coreruleset/blob/v3.3.0-rc2/CHANGES <https://github.com/coreruleset/coreruleset/blob/v3.3.0-rc2/CHANGES>
Our desire is to see the Core Rule Set project used as a baseline security feature, effectively protecting from OWASP Top 10 risks with few side effects. As such we attempt to cut down on false positives as much as possible in the default install. This RC therefore offers an opportunity for individuals to provide feedback and to report any issue they face with this release. We will then try and fix them for the upcoming full release.

Please use the CRS GitHub (https://github.com/coreruleset/coreruleset <https://github.com/coreruleset/coreruleset>), our slack channel (#coreruleset on owasp.slack.com <http://owasp.slack.com/>), or the Core Rule Set mailing list to tell us about your experiences, including false positives or other issues with this release candidate.

Our current timeline is to seek public feedback for the next 1.5 weeks, followed by a release on June 29th. We look forward to hearing your feedback!

Sincerely,
Walter Hop, release manager, on behalf of the Core Rule Set development team

Re: [mod-security-users] Reporting tool

[Jamie B. <ja...@ib...>]

I noticed this when I was looking: http://reconity.com/

I haven't tried it, I'd much prefer some offline software and not to have to paste megabytes of data. I'm also a bit weary about pasting potential sensitive data in to a random website...

Sent from my iPhone

> On 18 Jun 2020, at 13:32, Christian Folini <chr...@ne...> wrote:
> 
> Hey Mahmood,
> 
> No, you are right. There is not much around and certainly not much in the
> OSS space. Usually people built it themselves without documenting or
> sharing their experience.
> 
> An interesting new player seems to be securely.ai. Their website looks a bit
> Dutch, but they speak English and getting a demo is quite impressive.
> 
> Cheers,
> 
> Christian
> 
>> On Thu, Jun 18, 2020 at 06:52:35AM +0000, Mahmood Naderan via mod-security-users wrote:
>> Hi
>> I have searched a lot to see if there is a any reporting tool for mod-security but didn't find a clear answer. Seems that they are commercial tools.
>> If I am wrong, please let me know.
>> 
>> Regards,
>> Mahmood
> 
> 
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
> 
> 
> 
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] Reporting tool

[Christian F. <chr...@ne...>]

Hey Mahmood,

No, you are right. There is not much around and certainly not much in the
OSS space. Usually people built it themselves without documenting or
sharing their experience.

An interesting new player seems to be securely.ai. Their website looks a bit
Dutch, but they speak English and getting a demo is quite impressive.

Cheers,

Christian

On Thu, Jun 18, 2020 at 06:52:35AM +0000, Mahmood Naderan via mod-security-users wrote:
> Hi
> I have searched a lot to see if there is a any reporting tool for mod-security but didn't find a clear answer. Seems that they are commercial tools.
> If I am wrong, please let me know.
> 
> Regards,
> Mahmood


> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

[mod-security-users] Reporting tool

[Mahmood N. <nt_...@ya...>]

Hi
I have searched a lot to see if there is a any reporting tool for mod-security but didn't find a clear answer. Seems that they are commercial tools.
If I am wrong, please let me know.

Regards,
Mahmood

Re: [mod-security-users] Possible to remove rules by multiple tags?

[Ervin H. <ai...@gm...>]

you're welcome - note, the list has been from CRS 3.3/dev, @c11baa.

Regards,

a.


On Wed, Jun 17, 2020 at 12:07 PM Jamie Burchell <ja...@ib...> wrote:

> Hi
>
> That's really useful, thank you.
>
> Cheers
> Jamie
>
> On 17 Jun 2020, at 10:58, Ervin Hegedüs <ai...@gm...> wrote:
>
> 
> Hi Jamie,
>
> as Christian wrote there isn't any solution to remove a rule by multiple
> tags.
>
> But there is an indirect solution: you can find all rules where the listed
> tags exists.
>
> There is a small tool, named msc_pyparser[1]. This Python library can
> parse CRS rules and makes the AST (abstract syntax tree) in YAML or JSON
> format.
>
> I attached a Python script which loads these rules and search all id where
> the tags above listed. Before you run, you have to install that Python
> library (it works only with Python3), it's available through PIP. First,
> you have to build the AST files, then run script for each file, like:
>
> for y in `ls -1 export/*.yaml`; do ./crs_gettags.py ${y}; done
>
> and you'll see something like this:
>
> SecRuleRemoveById 942110
> SecRuleRemoveById 942120
> SecRuleRemoveById 942130
> SecRuleRemoveById 942150
> SecRuleRemoveById 942180
> SecRuleRemoveById 942200
> SecRuleRemoveById 942210
> SecRuleRemoveById 942260
> SecRuleRemoveById 942300
> SecRuleRemoveById 942310
> SecRuleRemoveById 942330
> SecRuleRemoveById 942340
> SecRuleRemoveById 942361
> SecRuleRemoveById 942370
> SecRuleRemoveById 942380
> SecRuleRemoveById 942390
> SecRuleRemoveById 942400
> SecRuleRemoveById 942410
> SecRuleRemoveById 942470
> SecRuleRemoveById 942480
> SecRuleRemoveById 942430
> SecRuleRemoveById 942440
> SecRuleRemoveById 942450
> SecRuleRemoveById 942510
>
> Just paste these lines into your exceptions, and hope that will give you
> what you want.
>
>
> Regards,
>
>
> a.
>
>
> [1]: https://github.com/digitalwave/msc_pyparser
>
>
>
>
> On Wed, Jun 17, 2020 at 1:01 AM Jamie Burchell <ja...@ib...> wrote:
>
>> Hi
>>
>>
>>
>> Is it possible to remove rules by more than one tag? For example, remove
>> all “paranoia-level/2” “attack-sqli” CRS rules.
>>
>>
>>
>> This would be useful in situations where PL2 is in use, but certain
>> groups of rules should not be at PL2. I was looking at doing this by ID
>> range instead, but the IDs don’t seem facilitate ranges based on PL.
>>
>>
>>
>> Regards,
>>
>> Jamie
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
>>
> <crs_gettags.py>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>

Re: [mod-security-users] Possible to remove rules by multiple tags?

[Jamie B. <ja...@ib...>]

Hi

That's really useful, thank you.

Cheers
Jamie

> On 17 Jun 2020, at 10:58, Ervin Hegedüs <ai...@gm...> wrote:
> 
> 
> Hi Jamie,
> 
> as Christian wrote there isn't any solution to remove a rule by multiple tags.
> 
> But there is an indirect solution: you can find all rules where the listed tags exists.
> 
> There is a small tool, named msc_pyparser[1]. This Python library can parse CRS rules and makes the AST (abstract syntax tree) in YAML or JSON format.
> 
> I attached a Python script which loads these rules and search all id where the tags above listed. Before you run, you have to install that Python library (it works only with Python3), it's available through PIP. First, you have to build the AST files, then run script for each file, like:
> 
> for y in `ls -1 export/*.yaml`; do ./crs_gettags.py ${y}; done
> 
> and you'll see something like this:
> 
> SecRuleRemoveById 942110
> SecRuleRemoveById 942120
> SecRuleRemoveById 942130
> SecRuleRemoveById 942150
> SecRuleRemoveById 942180
> SecRuleRemoveById 942200
> SecRuleRemoveById 942210
> SecRuleRemoveById 942260
> SecRuleRemoveById 942300
> SecRuleRemoveById 942310
> SecRuleRemoveById 942330
> SecRuleRemoveById 942340
> SecRuleRemoveById 942361
> SecRuleRemoveById 942370
> SecRuleRemoveById 942380
> SecRuleRemoveById 942390
> SecRuleRemoveById 942400
> SecRuleRemoveById 942410
> SecRuleRemoveById 942470
> SecRuleRemoveById 942480
> SecRuleRemoveById 942430
> SecRuleRemoveById 942440
> SecRuleRemoveById 942450
> SecRuleRemoveById 942510
> 
> Just paste these lines into your exceptions, and hope that will give you what you want.
> 
> 
> Regards,
> 
> 
> a.
> 
> 
> [1]: https://github.com/digitalwave/msc_pyparser
> 
> 
> 
> 
>> On Wed, Jun 17, 2020 at 1:01 AM Jamie Burchell <ja...@ib...> wrote:
>> Hi
>> 
>>  
>> 
>> Is it possible to remove rules by more than one tag? For example, remove all “paranoia-level/2” “attack-sqli” CRS rules.
>> 
>>  
>> 
>> This would be useful in situations where PL2 is in use, but certain groups of rules should not be at PL2. I was looking at doing this by ID range instead, but the IDs don’t seem facilitate ranges based on PL.
>> 
>>  
>> 
>> Regards,
>> 
>> Jamie
>> 
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
> 
> <crs_gettags.py>
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] Possible to remove rules by multiple tags?

[Ervin H. <ai...@gm...>]

Hi Jamie,

as Christian wrote there isn't any solution to remove a rule by multiple
tags.

But there is an indirect solution: you can find all rules where the listed
tags exists.

There is a small tool, named msc_pyparser[1]. This Python library can parse
CRS rules and makes the AST (abstract syntax tree) in YAML or JSON format.

I attached a Python script which loads these rules and search all id where
the tags above listed. Before you run, you have to install that Python
library (it works only with Python3), it's available through PIP. First,
you have to build the AST files, then run script for each file, like:

for y in `ls -1 export/*.yaml`; do ./crs_gettags.py ${y}; done

and you'll see something like this:

SecRuleRemoveById 942110
SecRuleRemoveById 942120
SecRuleRemoveById 942130
SecRuleRemoveById 942150
SecRuleRemoveById 942180
SecRuleRemoveById 942200
SecRuleRemoveById 942210
SecRuleRemoveById 942260
SecRuleRemoveById 942300
SecRuleRemoveById 942310
SecRuleRemoveById 942330
SecRuleRemoveById 942340
SecRuleRemoveById 942361
SecRuleRemoveById 942370
SecRuleRemoveById 942380
SecRuleRemoveById 942390
SecRuleRemoveById 942400
SecRuleRemoveById 942410
SecRuleRemoveById 942470
SecRuleRemoveById 942480
SecRuleRemoveById 942430
SecRuleRemoveById 942440
SecRuleRemoveById 942450
SecRuleRemoveById 942510

Just paste these lines into your exceptions, and hope that will give you
what you want.


Regards,


a.


[1]: https://github.com/digitalwave/msc_pyparser




On Wed, Jun 17, 2020 at 1:01 AM Jamie Burchell <ja...@ib...> wrote:

> Hi
>
>
>
> Is it possible to remove rules by more than one tag? For example, remove
> all “paranoia-level/2” “attack-sqli” CRS rules.
>
>
>
> This would be useful in situations where PL2 is in use, but certain groups
> of rules should not be at PL2. I was looking at doing this by ID range
> instead, but the IDs don’t seem facilitate ranges based on PL.
>
>
>
> Regards,
>
> Jamie
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/
>

Re: [mod-security-users] Possible to remove rules by multiple tags?

[Jamie B. <ja...@ib...>]

Thanks for your replies and time Christian, I appreciate it.

I'm very new to the world of ModSecurity and CRS, so at this stage I have more questions than answers unfortunately! I've realised that actually this is extremely complicated and that the more you look at it, the more nuanced it is and I'm sporting a semi-permanent expression which is somewhere between wonderment and confusion.

I initially just wanted to create a custom rate limiting rule to protect a few endpoints and also enable a specific PL2 rule for blocking scripting agents. I have come at this a bit backwards probably. Anyway, these rules have worked well, so I've been looking at enabling more of the CRS and reading through the documentation and also your tutorials on dealing with and filtering out false positives.

You are spot on about not wanting to see things that you do. I enabled the rules to detect, for example, requests without an Accept header and without a User agent and these have now filled up the log, so it's almost impossible to "see past them". I want to log them, I want them to count towards the anomaly score, but the log is now unmanageable with them in place. For now I removed those rules. I started looking for some sort of GUI that could somehow sift through and summarise the entries that could then be drilled down in to, but have come up short, at least using the basic serial log. I have seen some of your posts where you summarise on the command line, these are helpful.

The web server I'm experimenting on actually houses a few completely different virtual hosts, some web apps that are written by me and some third-party, so the setup is complex and of course now I'm faced with a huge log of false positives (and some true) from each different app, particularly in respect of XSS and SQLi against the admin area/back-end forms. And there are just too many places this sort of thing can crop up. For now, I started turning off the rule engine completely for the back-ends, which are IP protected, but that's almost certainly not the best way to handle things.

Cheers
Jamie

> On 17 Jun 2020, at 09:50, Christian Folini <chr...@ne...> wrote:
> 
> You have a good way of asking the relevant questions Jamie.
> 
> No, there is no established best practice for this very problem. I've been
> thinking about it quite some time, but I have not yet found an elegant
> solution (like the "executing paranoia level" solving the problem how to take
> a well-tuned system to a higher PL without also diluting your anomaly
> threshold setting).
> 
> If you have something in mind, then please share.
> 
> For me, the problem is related to the paradox, that when you write a rule
> exclusion against a false positive, you go blind on the false positive. And
> neither you, nor Schroedinger's cat can easily tell if wether the rule
> exclusion is still needed. It's like you want to see things that you do
> not want to see.
> 
> Best,
> 
> Christian
> 
> 
>> On Wed, Jun 17, 2020 at 08:58:26AM +0100, Jamie Burchell wrote:
>> OK, thank you. Is there a particular strategy or best practise for dealing
>> with new untested rules that are added to the CRS?
>> 
>> I'm thinking if you have tuned the rules, are running at PL2 and new ones
>> are added in an update. It seems that any new rules would need to be
>> identified and carefully managed in an existing setup, short of putting the
>> whole install back in detection only mode and remonitoring.
>> 
>> Thanks Jamie
>> 
>>> On 17 Jun 2020, at 05:37, Christian Folini <chr...@ne...>
>>> wrote:
>>> 
>>> Hello Jamie,
>>> 
>>> I have never throught about this, but it sounds like a cool idea.
>>> 
>>> Unfortunately, it is not possible.
>>> 
>>>> On Tue, Jun 16, 2020 at 11:34:53PM +0100, Jamie Burchell wrote: of rules
>>>> should not be at PL2. I was looking at doing this by ID range instead,
>>>> but the IDs don’t seem facilitate ranges based on PL.
>>> 
>>> The ids and the PL have no connection, that's correct.
>>> 
>>> (And they can't since we are adding new rules from time to time and the
>>> concept of strict siblings would no longer work the way it does with the
>>> ID namespace)
>>> 
>>> Ahoj,
>>> 
>>> Christian
>>> 
>>>> 
>>>> 
>>>> 
>>>> Regards,
>>>> 
>>>> Jamie
>>> 
>>> 
>>>> _______________________________________________ mod-security-users
>>>> mailing list mod...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>>> http://www.modsecurity.org/projects/commercial/rules/
>>>> http://www.modsecurity.org/projects/commercial/support/
>>> 
>>> 
>>> 
>>> _______________________________________________ mod-security-users mailing
>>> list mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users Commercial
>>> ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> http://www.modsecurity.org/projects/commercial/rules/
>>> http://www.modsecurity.org/projects/commercial/support/
>> 
>> 
>> _______________________________________________ mod-security-users mailing
>> list mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users Commercial
>> ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
> 
>> On Wed, Jun 17, 2020 at 08:58:26AM +0100, Jamie Burchell wrote:
>> OK, thank you. Is there a particular strategy or best practise for dealing with new untested rules that are added to the CRS?
>> 
>> I'm thinking if you have tuned the rules, are running at PL2 and new ones are added in an update. It seems that any new rules would need to be identified and carefully managed in an existing setup, short of putting the whole install back in detection only mode and remonitoring.
>> 
>> Thanks
>> Jamie
>> 
>>>> On 17 Jun 2020, at 05:37, Christian Folini <chr...@ne...> wrote:
>>> 
>>> Hello Jamie,
>>> 
>>> I have never throught about this, but it sounds like a cool idea.
>>> 
>>> Unfortunately, it is not possible.
>>> 
>>>> On Tue, Jun 16, 2020 at 11:34:53PM +0100, Jamie Burchell wrote:
>>>> of rules should not be at PL2. I was looking at doing this by ID range
>>>> instead, but the IDs don’t seem facilitate ranges based on PL.
>>> 
>>> The ids and the PL have no connection, that's correct.
>>> 
>>> (And they can't since we are adding new rules from time to time and the
>>> concept of strict siblings would no longer work the way it does with the
>>> ID namespace)
>>> 
>>> Ahoj,
>>> 
>>> Christian
>>> 
>>>> 
>>>> 
>>>> 
>>>> Regards,
>>>> 
>>>> Jamie
>>> 
>>> 
>>>> _______________________________________________
>>>> mod-security-users mailing list
>>>> mod...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>>> http://www.modsecurity.org/projects/commercial/rules/
>>>> http://www.modsecurity.org/projects/commercial/support/
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> mod-security-users mailing list
>>> mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> http://www.modsecurity.org/projects/commercial/rules/
>>> http://www.modsecurity.org/projects/commercial/support/
>> 
>> 
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
> 
> 
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] Possible to remove rules by multiple tags?

[Christian F. <chr...@ne...>]

You have a good way of asking the relevant questions Jamie.

No, there is no established best practice for this very problem. I've been
thinking about it quite some time, but I have not yet found an elegant
solution (like the "executing paranoia level" solving the problem how to take
a well-tuned system to a higher PL without also diluting your anomaly
threshold setting).

If you have something in mind, then please share.

For me, the problem is related to the paradox, that when you write a rule
exclusion against a false positive, you go blind on the false positive. And
neither you, nor Schroedinger's cat can easily tell if wether the rule
exclusion is still needed. It's like you want to see things that you do
not want to see.

Best,

Christian


On Wed, Jun 17, 2020 at 08:58:26AM +0100, Jamie Burchell wrote:
> OK, thank you. Is there a particular strategy or best practise for dealing
> with new untested rules that are added to the CRS?
> 
> I'm thinking if you have tuned the rules, are running at PL2 and new ones
> are added in an update. It seems that any new rules would need to be
> identified and carefully managed in an existing setup, short of putting the
> whole install back in detection only mode and remonitoring.
> 
> Thanks Jamie
> 
> > On 17 Jun 2020, at 05:37, Christian Folini <chr...@ne...>
> > wrote:
> > 
> > Hello Jamie,
> > 
> > I have never throught about this, but it sounds like a cool idea.
> > 
> > Unfortunately, it is not possible.
> > 
> >> On Tue, Jun 16, 2020 at 11:34:53PM +0100, Jamie Burchell wrote: of rules
> >> should not be at PL2. I was looking at doing this by ID range instead,
> >> but the IDs don’t seem facilitate ranges based on PL.
> > 
> > The ids and the PL have no connection, that's correct.
> > 
> > (And they can't since we are adding new rules from time to time and the
> > concept of strict siblings would no longer work the way it does with the
> > ID namespace)
> > 
> > Ahoj,
> > 
> > Christian
> > 
> >> 
> >> 
> >> 
> >> Regards,
> >> 
> >> Jamie
> > 
> > 
> >> _______________________________________________ mod-security-users
> >> mailing list mod...@li...
> >> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> >> http://www.modsecurity.org/projects/commercial/rules/
> >> http://www.modsecurity.org/projects/commercial/support/
> > 
> > 
> > 
> > _______________________________________________ mod-security-users mailing
> > list mod...@li...
> > https://lists.sourceforge.net/lists/listinfo/mod-security-users Commercial
> > ModSecurity Rules and Support from Trustwave's SpiderLabs:
> > http://www.modsecurity.org/projects/commercial/rules/
> > http://www.modsecurity.org/projects/commercial/support/
> 
> 
> _______________________________________________ mod-security-users mailing
> list mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users Commercial
> ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

On Wed, Jun 17, 2020 at 08:58:26AM +0100, Jamie Burchell wrote:
> OK, thank you. Is there a particular strategy or best practise for dealing with new untested rules that are added to the CRS?
> 
> I'm thinking if you have tuned the rules, are running at PL2 and new ones are added in an update. It seems that any new rules would need to be identified and carefully managed in an existing setup, short of putting the whole install back in detection only mode and remonitoring.
> 
> Thanks
> Jamie
> 
> > On 17 Jun 2020, at 05:37, Christian Folini <chr...@ne...> wrote:
> > 
> > Hello Jamie,
> > 
> > I have never throught about this, but it sounds like a cool idea.
> > 
> > Unfortunately, it is not possible.
> > 
> >> On Tue, Jun 16, 2020 at 11:34:53PM +0100, Jamie Burchell wrote:
> >> of rules should not be at PL2. I was looking at doing this by ID range
> >> instead, but the IDs don’t seem facilitate ranges based on PL.
> > 
> > The ids and the PL have no connection, that's correct.
> > 
> > (And they can't since we are adding new rules from time to time and the
> > concept of strict siblings would no longer work the way it does with the
> > ID namespace)
> > 
> > Ahoj,
> > 
> > Christian
> > 
> >> 
> >> 
> >> 
> >> Regards,
> >> 
> >> Jamie
> > 
> > 
> >> _______________________________________________
> >> mod-security-users mailing list
> >> mod...@li...
> >> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> >> http://www.modsecurity.org/projects/commercial/rules/
> >> http://www.modsecurity.org/projects/commercial/support/
> > 
> > 
> > 
> > _______________________________________________
> > mod-security-users mailing list
> > mod...@li...
> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> > Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> > http://www.modsecurity.org/projects/commercial/rules/
> > http://www.modsecurity.org/projects/commercial/support/
> 
> 
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] Possible to remove rules by multiple tags?

[Jamie B. <ja...@ib...>]

OK, thank you. Is there a particular strategy or best practise for dealing with new untested rules that are added to the CRS?

I'm thinking if you have tuned the rules, are running at PL2 and new ones are added in an update. It seems that any new rules would need to be identified and carefully managed in an existing setup, short of putting the whole install back in detection only mode and remonitoring.

Thanks
Jamie

> On 17 Jun 2020, at 05:37, Christian Folini <chr...@ne...> wrote:
> 
> Hello Jamie,
> 
> I have never throught about this, but it sounds like a cool idea.
> 
> Unfortunately, it is not possible.
> 
>> On Tue, Jun 16, 2020 at 11:34:53PM +0100, Jamie Burchell wrote:
>> of rules should not be at PL2. I was looking at doing this by ID range
>> instead, but the IDs don’t seem facilitate ranges based on PL.
> 
> The ids and the PL have no connection, that's correct.
> 
> (And they can't since we are adding new rules from time to time and the
> concept of strict siblings would no longer work the way it does with the
> ID namespace)
> 
> Ahoj,
> 
> Christian
> 
>> 
>> 
>> 
>> Regards,
>> 
>> Jamie
> 
> 
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
> 
> 
> 
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] Possible to remove rules by multiple tags?

[Christian F. <chr...@ne...>]

Hello Jamie,

I have never throught about this, but it sounds like a cool idea.

Unfortunately, it is not possible.

On Tue, Jun 16, 2020 at 11:34:53PM +0100, Jamie Burchell wrote:
> of rules should not be at PL2. I was looking at doing this by ID range
> instead, but the IDs don’t seem facilitate ranges based on PL.

The ids and the PL have no connection, that's correct.

(And they can't since we are adding new rules from time to time and the
concept of strict siblings would no longer work the way it does with the
ID namespace)

Ahoj,

Christian

> 
> 
> 
> Regards,
> 
> Jamie


> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

[mod-security-users] Possible to remove rules by multiple tags?

[Jamie B. <ja...@ib...>]

Hi



Is it possible to remove rules by more than one tag? For example, remove
all “paranoia-level/2” “attack-sqli” CRS rules.



This would be useful in situations where PL2 is in use, but certain groups
of rules should not be at PL2. I was looking at doing this by ID range
instead, but the IDs don’t seem facilitate ranges based on PL.



Regards,

Jamie

Re: [mod-security-users] Advice for moving forward with more CRS rules

[Jamie B. <ja...@ib...>]

Great idea RE setting the thresholds high. I'll have a think. Although after spending around 2 hours trying to understand the CRS DoS protection and realising it's fundamentally broken for PL2 (https://github.com/coreruleset/coreruleset/issues/1597#issuecomment-643641081), I don't think I'll me taking too much inspiration from it! ;)

Sent from my iPhone

> On 13 Jun 2020, at 12:47, Manuel Spartan <spa...@gm...> wrote:
> 
> That’s why I suggest using envvars and stop/redirect using rewrite rules, other than editing the entire thing you can also play with the defaulactio s set to pass and the thresholds to block set to sky high values
> 
> Sent from my iPhone
> 
>> On Jun 13, 2020, at 5:30 AM, Jamie Burchell <ja...@ib...> wrote:
>> 
>> Hi Manuel
>> 
>> Thanks for the reply. The rate limit rule I have in place already is working well. The question is more how can I now try out and monitor more of the CRS safely without actually blocking requests, now that I'm using my own rules and some of the other CRS rules. I don't actually think I can, I think all I can really do is change the actions of the anomaly detection routines to pass and log which will stop the rules I've already tested from actually blocking.
>> 
>> Sent from my iPhone
>> 
>>>> On 12 Jun 2020, at 23:52, Manuel Spartan <spa...@gm...> wrote:
>>> 
>>> Hi Jamie, what about using setenv action on those connections that require to slow down and do a rewrite rule answering with redirect to a static html saying’slowdown’. As for the logic you will need to initialize the collection to keep track of the request rate over time, take a look to the crs brute force/dos rules for inspiration.
>>> 
>>> You can also use modqos.
>>> 
>>> Regards,
>>> Manuel
>>> 
>>> Sent from my iPhone
>>> 
>>>>> On Jun 12, 2020, at 6:02 AM, Jamie Burchell <ja...@ib...> wrote:
>>>> 
>>>> Hello
>>>> 
>>>> I'm hoping for some general advice on next steps for my EL7 / Apache 2.4 / Mod Security 2.9 / OWASP CRS setup please. This is potentially more of a CRS question, so apologies if this is the wrong place to pose the question and I'd appreciate any pointers as to where I could ask this. Server Fault hasn't yielding any response.
>>>> 
>>>> I had a requirement to rate limit some URL paths and to (attempt to) block some well known scripting user agents and have successfully achieved both of these things using a custom mod_security rule for the rate limit and the OWASP CRS rules pertaining to scripting user agents.
>>>> 
>>>> For the moment, I've disabled pretty much all of the CRS rules except for the scanners. I needed to increase the Paranoia Level (PL) to 2 before those rules I needed were activated.
>>>> 
>>>> I'd now like to look at enabling more of the CRS but in a monitored and controlled way, but this now poses a problem. I have a rate limit rule live and working and CRS rules at PL2 just for script scanner detection. I'm not able to put mod_security in detection only mode, because I have live rules working. I can change the blocking actions for the anomaly detection rules to pass and log, but this will deactivate my scanner rules. And I'm stuck with PL2 in order to get the scanner rules I need.
>>>> 
>>>> I'm unsure how to proceed in a controlled way to monitor further CRS rules without them actually blocking - and whether or not I'm stuck with PL2 for all other rules because of what I need for the scanners.
>>>> 
>>>> Would really appreciate any pointers, this conundrum is driving me mad.
>>>> 
>>>> Thanks in advance
>>>> Jamie
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> mod-security-users mailing list
>>>> mod...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>>> http://www.modsecurity.org/projects/commercial/rules/
>>>> http://www.modsecurity.org/projects/commercial/support/
>>> 
>>> 
>>> _______________________________________________
>>> mod-security-users mailing list
>>> mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> http://www.modsecurity.org/projects/commercial/rules/
>>> http://www.modsecurity.org/projects/commercial/support/
>> 
>> 
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
> 
> 
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] Advice for moving forward with more CRS rules

[Manuel S. <spa...@gm...>]

That’s why I suggest using envvars and stop/redirect using rewrite rules, other than editing the entire thing you can also play with the defaulactio s set to pass and the thresholds to block set to sky high values

Sent from my iPhone

> On Jun 13, 2020, at 5:30 AM, Jamie Burchell <ja...@ib...> wrote:
> 
> Hi Manuel
> 
> Thanks for the reply. The rate limit rule I have in place already is working well. The question is more how can I now try out and monitor more of the CRS safely without actually blocking requests, now that I'm using my own rules and some of the other CRS rules. I don't actually think I can, I think all I can really do is change the actions of the anomaly detection routines to pass and log which will stop the rules I've already tested from actually blocking.
> 
> Sent from my iPhone
> 
>> On 12 Jun 2020, at 23:52, Manuel Spartan <spa...@gm...> wrote:
>> 
>> Hi Jamie, what about using setenv action on those connections that require to slow down and do a rewrite rule answering with redirect to a static html saying’slowdown’. As for the logic you will need to initialize the collection to keep track of the request rate over time, take a look to the crs brute force/dos rules for inspiration.
>> 
>> You can also use modqos.
>> 
>> Regards,
>> Manuel
>> 
>> Sent from my iPhone
>> 
>>>> On Jun 12, 2020, at 6:02 AM, Jamie Burchell <ja...@ib...> wrote:
>>> 
>>> Hello
>>> 
>>> I'm hoping for some general advice on next steps for my EL7 / Apache 2.4 / Mod Security 2.9 / OWASP CRS setup please. This is potentially more of a CRS question, so apologies if this is the wrong place to pose the question and I'd appreciate any pointers as to where I could ask this. Server Fault hasn't yielding any response.
>>> 
>>> I had a requirement to rate limit some URL paths and to (attempt to) block some well known scripting user agents and have successfully achieved both of these things using a custom mod_security rule for the rate limit and the OWASP CRS rules pertaining to scripting user agents.
>>> 
>>> For the moment, I've disabled pretty much all of the CRS rules except for the scanners. I needed to increase the Paranoia Level (PL) to 2 before those rules I needed were activated.
>>> 
>>> I'd now like to look at enabling more of the CRS but in a monitored and controlled way, but this now poses a problem. I have a rate limit rule live and working and CRS rules at PL2 just for script scanner detection. I'm not able to put mod_security in detection only mode, because I have live rules working. I can change the blocking actions for the anomaly detection rules to pass and log, but this will deactivate my scanner rules. And I'm stuck with PL2 in order to get the scanner rules I need.
>>> 
>>> I'm unsure how to proceed in a controlled way to monitor further CRS rules without them actually blocking - and whether or not I'm stuck with PL2 for all other rules because of what I need for the scanners.
>>> 
>>> Would really appreciate any pointers, this conundrum is driving me mad.
>>> 
>>> Thanks in advance
>>> Jamie
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> mod-security-users mailing list
>>> mod...@li...
>>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>>> http://www.modsecurity.org/projects/commercial/rules/
>>> http://www.modsecurity.org/projects/commercial/support/
>> 
>> 
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
> 
> 
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] Advice for moving forward with more CRS rules

[Jamie B. <ja...@ib...>]

Hi Manuel

Thanks for the reply. The rate limit rule I have in place already is working well. The question is more how can I now try out and monitor more of the CRS safely without actually blocking requests, now that I'm using my own rules and some of the other CRS rules. I don't actually think I can, I think all I can really do is change the actions of the anomaly detection routines to pass and log which will stop the rules I've already tested from actually blocking.

Sent from my iPhone

> On 12 Jun 2020, at 23:52, Manuel Spartan <spa...@gm...> wrote:
> 
> Hi Jamie, what about using setenv action on those connections that require to slow down and do a rewrite rule answering with redirect to a static html saying’slowdown’. As for the logic you will need to initialize the collection to keep track of the request rate over time, take a look to the crs brute force/dos rules for inspiration.
> 
> You can also use modqos.
> 
> Regards,
> Manuel
> 
> Sent from my iPhone
> 
>> On Jun 12, 2020, at 6:02 AM, Jamie Burchell <ja...@ib...> wrote:
>> 
>> Hello
>> 
>> I'm hoping for some general advice on next steps for my EL7 / Apache 2.4 / Mod Security 2.9 / OWASP CRS setup please. This is potentially more of a CRS question, so apologies if this is the wrong place to pose the question and I'd appreciate any pointers as to where I could ask this. Server Fault hasn't yielding any response.
>> 
>> I had a requirement to rate limit some URL paths and to (attempt to) block some well known scripting user agents and have successfully achieved both of these things using a custom mod_security rule for the rate limit and the OWASP CRS rules pertaining to scripting user agents.
>> 
>> For the moment, I've disabled pretty much all of the CRS rules except for the scanners. I needed to increase the Paranoia Level (PL) to 2 before those rules I needed were activated.
>> 
>> I'd now like to look at enabling more of the CRS but in a monitored and controlled way, but this now poses a problem. I have a rate limit rule live and working and CRS rules at PL2 just for script scanner detection. I'm not able to put mod_security in detection only mode, because I have live rules working. I can change the blocking actions for the anomaly detection rules to pass and log, but this will deactivate my scanner rules. And I'm stuck with PL2 in order to get the scanner rules I need.
>> 
>> I'm unsure how to proceed in a controlled way to monitor further CRS rules without them actually blocking - and whether or not I'm stuck with PL2 for all other rules because of what I need for the scanners.
>> 
>> Would really appreciate any pointers, this conundrum is driving me mad.
>> 
>> Thanks in advance
>> Jamie
>> 
>> 
>> 
>> _______________________________________________
>> mod-security-users mailing list
>> mod...@li...
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
>> http://www.modsecurity.org/projects/commercial/rules/
>> http://www.modsecurity.org/projects/commercial/support/
> 
> 
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] Large Payload processing time

[Manuel S. <spa...@gm...>]

Hi Joe, looks like you’re using the Default body processor on xml payloads which causes a lot of problems. Try setting it to XML based on the request_uri with ctl:requestBodyProcessor=XML

Regards,
Manuel
Sent from my iPhone

> On Jun 11, 2020, at 9:47 AM, Madden, Joe via mod-security-users <mod...@li...> wrote:
> 
> Hi all,
> 
> I've had to disable the following rules in order to get a payload to process in a resonable amount of time.
> 
> It a XML payload with up to 20Mb in size, These are the rules which cause the processing from from around 30 seconds to  772 seconds
> 
> 
>            # Disables checking for Windows command injection 
>            SecRuleRemoveById 932110
> 
>            #Removes unix command injection filtering
>            SecRuleRemoveById 932100
> 
>            #Removes unix command injection filtering 2
>            #SecRuleRemoveById 932105
> 
>            #removes unix remote code exceuction
>            #SecRuleRemoveById 932150
> 
>            #Disables Oracle WebLogic Remote Command Execution exploit
>            #SecRuleRemoveById 932115
> 
>            #Disables PHPIDS - Converted SQLI Filters - Not required
>            #SecRuleRemoveById 942230
> 
>            #Disables PHPIDS - Converted SQLI Filters - Not required
>            #SecRuleRemoveById 942190
> 
>            #Disables HTTP Response Splitting - Not Required
>            #SecRuleRemoveById 921120
> 
>            # Disables Sources for SQL ALTER statements
>            #SecRuleRemoveById 942360
> 
>            #Disables XSS Filters - Category 3 - Not required
>            #SecRuleRemoveById 941130
> 
>            #Disables XSS [NoScript InjectionChecker] Attributes injection - Not required
>            #SecRuleRemoveById 941170
> 
>            #Disables XSS vectors making use of event handlers like onerror, onload
>            #SecRuleRemoveById 941120
> 
> I'll have times by the end of the day which rules take the longest but for example - Does anyone have any recommendations about this? We'd like to leave the uinix RCE and command filters on at this is what our platform is.
> 
> Thanks
> 
> Joe.
> 
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

Re: [mod-security-users] Advice for moving forward with more CRS rules

[Manuel S. <spa...@gm...>]

Hi Jamie, what about using setenv action on those connections that require to slow down and do a rewrite rule answering with redirect to a static html saying’slowdown’. As for the logic you will need to initialize the collection to keep track of the request rate over time, take a look to the crs brute force/dos rules for inspiration.

You can also use modqos.

Regards,
Manuel

Sent from my iPhone

> On Jun 12, 2020, at 6:02 AM, Jamie Burchell <ja...@ib...> wrote:
> 
> Hello
> 
> I'm hoping for some general advice on next steps for my EL7 / Apache 2.4 / Mod Security 2.9 / OWASP CRS setup please. This is potentially more of a CRS question, so apologies if this is the wrong place to pose the question and I'd appreciate any pointers as to where I could ask this. Server Fault hasn't yielding any response.
> 
> I had a requirement to rate limit some URL paths and to (attempt to) block some well known scripting user agents and have successfully achieved both of these things using a custom mod_security rule for the rate limit and the OWASP CRS rules pertaining to scripting user agents.
> 
> For the moment, I've disabled pretty much all of the CRS rules except for the scanners. I needed to increase the Paranoia Level (PL) to 2 before those rules I needed were activated.
> 
> I'd now like to look at enabling more of the CRS but in a monitored and controlled way, but this now poses a problem. I have a rate limit rule live and working and CRS rules at PL2 just for script scanner detection. I'm not able to put mod_security in detection only mode, because I have live rules working. I can change the blocking actions for the anomaly detection rules to pass and log, but this will deactivate my scanner rules. And I'm stuck with PL2 in order to get the scanner rules I need.
> 
> I'm unsure how to proceed in a controlled way to monitor further CRS rules without them actually blocking - and whether or not I'm stuck with PL2 for all other rules because of what I need for the scanners.
> 
> Would really appreciate any pointers, this conundrum is driving me mad.
> 
> Thanks in advance
> Jamie
> 
> 
> 
> _______________________________________________
> mod-security-users mailing list
> mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
> http://www.modsecurity.org/projects/commercial/rules/
> http://www.modsecurity.org/projects/commercial/support/

[mod-security-users] Advice for moving forward with more CRS rules

[Jamie B. <ja...@ib...>]

Hello

I'm hoping for some general advice on next steps for my EL7 / Apache 2.4 / Mod Security 2.9 / OWASP CRS setup please. This is potentially more of a CRS question, so apologies if this is the wrong place to pose the question and I'd appreciate any pointers as to where I could ask this. Server Fault hasn't yielding any response.

I had a requirement to rate limit some URL paths and to (attempt to) block some well known scripting user agents and have successfully achieved both of these things using a custom mod_security rule for the rate limit and the OWASP CRS rules pertaining to scripting user agents.

For the moment, I've disabled pretty much all of the CRS rules except for the scanners. I needed to increase the Paranoia Level (PL) to 2 before those rules I needed were activated.

I'd now like to look at enabling more of the CRS but in a monitored and controlled way, but this now poses a problem. I have a rate limit rule live and working and CRS rules at PL2 just for script scanner detection. I'm not able to put mod_security in detection only mode, because I have live rules working. I can change the blocking actions for the anomaly detection rules to pass and log, but this will deactivate my scanner rules. And I'm stuck with PL2 in order to get the scanner rules I need.

I'm unsure how to proceed in a controlled way to monitor further CRS rules without them actually blocking - and whether or not I'm stuck with PL2 for all other rules because of what I need for the scanners.

Would really appreciate any pointers, this conundrum is driving me mad.

Thanks in advance
Jamie

[mod-security-users] libmosecurity complains that it was not built with libcurl even though it was.

[John E. B. <joh...@bi...>]

I’ve built libmodsecurity 3.0.4 from github source with libcurl included on Joyent SmartOS 2019-q4 using pkgsrc. The regression and unit tests fail on a couple of items saying that it wasn’t built with libcurl support. I am using the testing tools from within the source/test directory directly and running it  like this: ./regression_tests. Am I running the tests correctly? On a side note I thought maybe they were failing because my curl was using openssl so I rebuilt curl with gnutls to be sure that wasn’t the issue. Both ways the configure script line that attempts to detect if curl is linked to gnutls returns a result of “no” and I know for a fact that it is compiled properly with gnutls support.


Thanks!

John Barfield



curl -V

curl 7.67.0 (x86_64-sun-solaris2.11) libcurl/7.67.0 GnuTLS/3.6.11 zlib/1.2.11 libidn2/2.3.0 libssh2/1.9.0 nghttp2/1.40.0
Release-Date: 2019-11-06
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM NTLM_WB SSL TLS-SRP UnixSockets




Regression_tests Error log:


Test failed. From: test-cases/regression/config-secremoterules.json.
Test name: Include remote rules - failed download (Abort).
Reason:
failed!
Expected a parser error.
Expected: Failed to download: HTTP response code said error
Produced: Rules error. File: https://gist.githubusercontent.com/zimmerle/a4c1ec028999f7df71d0cc80f4f271ca/raw/4c74363bf4eae974180f1a82007196e58729dd16/modsecurity-regression-test-secremoterules-bonga.txt. Line: 1. Column: 0.
 - Failed to download: Not compiled with libcurl support

Test failed. From: test-cases/regression/operator-ipMatchFromFile.json.
Test name: Testing Operator :: @ipMatchFromFile - https.
Reason:
parse failed.
Rules error. File: operator-ipMatchFromFile.json. Line: 2. Column: 19. Not compiled with libcurl support

Ran a total of: 554 regression tests - 3 failed. 11 skipped test(s). 0 disabled test(s).
configure scripts show libcurl support enabled. My question is

Re: [mod-security-users] Large Payload processing time

[Madden, J. <Joe...@mo...>]

Processing time of each rules which take the longest with our payload:

932110 - All Enabled 
Processing Time: 779

 

932100- Removes unix command injection filtering
Processing Time: 13

 

932105 - Removes unix command injection filtering 2 
Processing Time: 14

 

932150- Removes remote code execution 
Processing Time: 2

 

932115 - Oracle WebLogic Remote Command Execution exploit
Processing Time: 700

 

942230 - Disables PHPIDS - Converted SQLI Filters
Processing Time: 7

 

942190 - Disables PHPIDS - Converted SQLI Filters
Processing Time: 1 Seconds

 

921120 - Disables HTTP Response Splitting
Processing Time: 2

 

942360 - Disables Sources for SQL ALTER statements
Processing Time: 2

 

941130 - Disables XSS Filters - Category 3 
Processing Time: 9

 

941170 - Disables XSS [NoScript InjectionChecker] Attributes injection
Processing Time: 7

 

941120 - Disables XSS vectors making use of event handlers like onerror, onload
Processing Time: 3 Seconds

Thanks,

Joe.

Joe Madden
Systems Engineer
D 01412224666      
joe...@mo...


-----Original Message-----
From: Madden, Joe via mod-security-users <mod...@li...> 
Sent: 11 June 2020 14:45
To: Madden, Joe via mod-security-users <mod...@li...>
Cc: Madden, Joe <Joe...@mo...>; mod...@ow...
Subject: [mod-security-users] Large Payload processing time

Hi all,

I've had to disable the following rules in order to get a payload to process in a resonable amount of time.

It a XML payload with up to 20Mb in size, These are the rules which cause the processing from from around 30 seconds to  772 seconds


            # Disables checking for Windows command injection 
            SecRuleRemoveById 932110

            #Removes unix command injection filtering
            SecRuleRemoveById 932100

            #Removes unix command injection filtering 2
            #SecRuleRemoveById 932105

            #removes unix remote code exceuction
            #SecRuleRemoveById 932150

            #Disables Oracle WebLogic Remote Command Execution exploit
            #SecRuleRemoveById 932115

            #Disables PHPIDS - Converted SQLI Filters - Not required
            #SecRuleRemoveById 942230

            #Disables PHPIDS - Converted SQLI Filters - Not required
            #SecRuleRemoveById 942190

            #Disables HTTP Response Splitting - Not Required
            #SecRuleRemoveById 921120

            # Disables Sources for SQL ALTER statements
            #SecRuleRemoveById 942360

            #Disables XSS Filters - Category 3 - Not required
            #SecRuleRemoveById 941130

            #Disables XSS [NoScript InjectionChecker] Attributes injection - Not required
            #SecRuleRemoveById 941170

            #Disables XSS vectors making use of event handlers like onerror, onload
            #SecRuleRemoveById 941120

I'll have times by the end of the day which rules take the longest but for example - Does anyone have any recommendations about this? We'd like to leave the uinix RCE and command filters on at this is what our platform is.

Thanks

Joe.

_______________________________________________
mod-security-users mailing list
mod...@li...
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fmod-security-users&amp;data=01%7C01%7Cjoe.madden%40mottmac.com%7C6934d6dee32b4637135c08d80e0dc659%7Ca2bed0c459574f73b0c2a811407590fb%7C0&amp;sdata=sqnv7riFLHYDzTsLikXqVI%2F9BXk7CZHlpgF0XVcVuek%3D&amp;reserved=0
Commercial ModSecurity Rules and Support from Trustwave's SpiderLabs:
https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.modsecurity.org%2Fprojects%2Fcommercial%2Frules%2F&amp;data=01%7C01%7Cjoe.madden%40mottmac.com%7C6934d6dee32b4637135c08d80e0dc659%7Ca2bed0c459574f73b0c2a811407590fb%7C0&amp;sdata=ilFS%2B%2F7W8rO3yPNKC2XuEO9cX%2FrWZtC2uR21KahDZpc%3D&amp;reserved=0
https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.modsecurity.org%2Fprojects%2Fcommercial%2Fsupport%2F&amp;data=01%7C01%7Cjoe.madden%40mottmac.com%7C6934d6dee32b4637135c08d80e0dc659%7Ca2bed0c459574f73b0c2a811407590fb%7C0&amp;sdata=6rwPrhFpR8gkpfPFdNLV6b099UgDzvcfNcwa5BQPs1I%3D&amp;reserved=0