mod-security-developers — Development discussion about mod_security

Re: [Mod-security-developers] A list of small things to fix in ModSecurity

[Ivan R. <iva...@gm...>]

On Fri, Aug 28, 2009 at 6:32 PM, Brian Rectanus<bre...@gm...> wrote:
> On Fri, Aug 28, 2009 at 2:42 AM, Ivan Ristic<iva...@gm...> wrote:
>> I maintain a list of small things I want to fix in ModSecurity. They
>> are all either bugs or wrong design decisions and I think the next
>> non-patch version should be the version in which all of these are
>> fixed. Please comment.
>
> I agree with all of these (I think).

Excellent!


>> You can find the list here:
>>
>> http://docs.google.com/Doc?docid=0AcC0dgmab8WrZGN4MmZtZ2tfODJqZGpjZmdnNw&hl=en
>>
>> (Anyone should be allowed to edit it, so go ahead. Please only leave comments.)
>
> I made some comments, but I'd rather people used the tracker for this.

No problem, I will add them to the tracker individually.


> In fact many of these items were in the tracker (at least when we had
> the internal "trac" one, but they may not have gotten migrated over
> when we went to Jira).

Yes, they were. We didn't migrate most of them, but it would be nice
to have access to the list from Trac, just to see that we're not
missing something we already discussed and designed.

-- 
Ivan Ristic
Security assessment of your SSL servers
https://www.ssllabs.com/ssldb/

Re: [Mod-security-developers] A list of small things to fix in ModSecurity

[Brian R. <bre...@gm...>]

On Fri, Aug 28, 2009 at 2:42 AM, Ivan Ristic<iva...@gm...> wrote:
> I maintain a list of small things I want to fix in ModSecurity. They
> are all either bugs or wrong design decisions and I think the next
> non-patch version should be the version in which all of these are
> fixed. Please comment.

I agree with all of these (I think).

>
> You can find the list here:
>
> http://docs.google.com/Doc?docid=0AcC0dgmab8WrZGN4MmZtZ2tfODJqZGpjZmdnNw&hl=en
>
> (Anyone should be allowed to edit it, so go ahead. Please only leave comments.)

I made some comments, but I'd rather people used the tracker for this.
 In fact many of these items were in the tracker (at least when we had
the internal "trac" one, but they may not have gotten migrated over
when we went to Jira).

-B

[Mod-security-developers] A list of small things to fix in ModSecurity

[Ivan R. <iva...@gm...>]

I maintain a list of small things I want to fix in ModSecurity. They
are all either bugs or wrong design decisions and I think the next
non-patch version should be the version in which all of these are
fixed. Please comment.

You can find the list here:

http://docs.google.com/Doc?docid=0AcC0dgmab8WrZGN4MmZtZ2tfODJqZGpjZmdnNw&hl=en

(Anyone should be allowed to edit it, so go ahead. Please only leave comments.)

-- 
Ivan Ristic
Security assessment of your SSL servers
https://www.ssllabs.com/ssldb/

Re: [Mod-security-developers] Fwd: Performance improvement plan

[Ivan R. <iva...@gm...>]

On Tue, Aug 25, 2009 at 10:28 PM, Brian Rectanus<bre...@gm...> wrote:
>
> ...
>
>> 2. Change the individual rule processing code to avoid discovering
>> tfns at runtime. At the moment we are re-discovering tfns for every
>> variable, even though the action list never changes. Resolving the
>> tfns at configuration time should result a significant performance
>> improvement.
>
> You will need to re-build the list from SecRuleUpdateActionById at
> config time, which should be fine.  Just a stored list of tfn structs
> would be fine instead of building the list at runtime.  It should have
> been like that to begin with ;)

Sure, but then it would be more difficult to improve the performance ;)


>> 3. Cache entire target lists. Currently a lot of time is spent
>> resolving complex variables, adding and removing variables from the
>> target list, and transforming them in the end. I want to identify the
>> rules that share target lists, cache a reusable target list when it is
>> created for the first time, only to reuse it later. The cost of each
>> rule in the engine should go down a lot. In the worst case scenario we
>> waste a hash lookup (per rule). This should be countered by the
>> removal of the multiMatch lookup ;)
>
> Only issue I see here is with non-cacheable targets

Right. If there's even a single non-cacheable variable in a list, it
will make the list uncacheable.

> and targets that
> are listed in different orders.  We need a good way to "hash" the
> target list so that any rule that uses the same targets (at least the
> same cacheable targets and in any order) resolves to the same hashed
> value.

I am not going to address that in the first version, mostly because
variables are usually listed in the same order and also because
caching would cause us to change the order in which variables are
evaluated. I am guessing it's not important, but still.


> I had thought about using the ptr value of the tfn structure as this
> is unique per tfn and it does not have to survive a restart.

I am sorry, I don't follow what you're saying here.


> So, you have to pull the cached values, then merge in and transform
> any of the non-cacheable targets.  i would prefer if target order is
> maintained, which gives the rule writer some flexibility for
> optimizing the most likely targets first., but I am not sure that is a
> big deal.

Actually I don't want to bother with individual variables. I will only
identify identical variable lists and cache entire lists at a time. It
makes the code much simpler and the performance better.


>> I have a couple of other smaller things I may look into, but I suspect
>> the improvements from the above 3 changes will make me happy enough.
>
> One other *big* improvement that is needed is how SecAction/SecMarker
> work as well as skipping.  This was meant to just skip a few rules,
> but it is now being used to skip entire sections of rules and this is
> *very* slow.

I was looking at that code the other day. I don't think that it is
very slow right now (there's so much happening within one rule!), but
it may be perceived as very slow once the caching I add kicks in. I
don't think it's a big deal, but we'll see.


>> The plan is to benchmark with the available rule sets.
>>
>
> Sounds great.  It may also show some need for optimization in the
> rulesets as well.  Any plan for how you will be doing the benchmark?

Initially just the built-in performance measurement. I may try oprofile as well.


>> The only question is what to branch and when? I am hoping for the
>> improvements to be significant enough to warrant a middle digit bump
>> and I'd like to see them go into 2.6.0.
>
> The trunk is 2.6 (will be).  Work there as nothing but 2.5.x merges
> are getting commited there currently.

OK.


> thanks,
> -B
>



-- 
Ivan Ristic
Security assessment of your SSL servers
https://www.ssllabs.com/ssldb/

Re: [Mod-security-developers] Coverity Scan for ModSecurity?

[Brian R. <Bri...@br...>]

Ivan Ristic wrote:
> Brian,
>
> How do you feel about applying ModSecurity for the Coverity Scan?
>
> http://www.scan.coverity.com/
>

I'd love to, but have not had the time to look into the details.  I'd 
also need to check with Breach.

I've wanted to get demos of what both Coverity's and Fortify's tools can 
do (for comparison).  I have seen demos at conferences, etc. and they 
look quite nice.

-B

-- 
Brian Rectanus
Breach Security

Re: [Mod-security-developers] Fwd: Performance improvement plan

[Brian R. <bre...@gm...>]

On Tue, Aug 25, 2009 at 1:38 PM, Ivan Ristic<iva...@gm...> wrote:
> Forwarding message, as the original did not show in the archives.
>
> Is it getting through?

I do not see any since Jan 2009.

https://sourceforge.net/mailarchive/forum.php?forum_name=mod-security-developers

>
>
> ---------- Forwarded message ----------
> From: Ivan Ristic <iva...@gm...>
> Date: Fri, Aug 21, 2009 at 5:54 PM
> Subject: Performance improvement plan
> To: mod...@li...
>
>
> I am planning to improve the performance of the ModSecurity rule
> engine. I've reviewed the source code and come up with the following
> ideas:
>
> 1. Avoid table lookup currently used to determine if a rule is
> multiMatch. I don't suppose this is going to result with a significant
> improvement, but it's been getting on my nerves for years now and I
> want to get rid of it. I plan to add a variable to the rule structure
> and set it at configuration time. Before I do anything, however, I
> will investigate how SecRuleUpdateActionById interacts with action
> lists.

Agreed.

>
> 2. Change the individual rule processing code to avoid discovering
> tfns at runtime. At the moment we are re-discovering tfns for every
> variable, even though the action list never changes. Resolving the
> tfns at configuration time should result a significant performance
> improvement.

You will need to re-build the list from SecRuleUpdateActionById at
config time, which should be fine.  Just a stored list of tfn structs
would be fine instead of building the list at runtime.  It should have
been like that to begin with ;)

The whole runtime "rule" structure needs to be direct-access instead
of lookups, so you might as well remove all the lookups.

>
> 3. Cache entire target lists. Currently a lot of time is spent
> resolving complex variables, adding and removing variables from the
> target list, and transforming them in the end. I want to identify the
> rules that share target lists, cache a reusable target list when it is
> created for the first time, only to reuse it later. The cost of each
> rule in the engine should go down a lot. In the worst case scenario we
> waste a hash lookup (per rule). This should be countered by the
> removal of the multiMatch lookup ;)

Only issue I see here is with non-cacheable targets and targets that
are listed in different orders.  We need a good way to "hash" the
target list so that any rule that uses the same targets (at least the
same cacheable targets and in any order) resolves to the same hashed
value.

I had thought about using the ptr value of the tfn structure as this
is unique per tfn and it does not have to survive a restart.

So, you have to pull the cached values, then merge in and transform
any of the non-cacheable targets.  i would prefer if target order is
maintained, which gives the rule writer some flexibility for
optimizing the most likely targets first., but I am not sure that is a
big deal.

>
> I have a couple of other smaller things I may look into, but I suspect
> the improvements from the above 3 changes will make me happy enough.

One other *big* improvement that is needed is how SecAction/SecMarker
work as well as skipping.  This was meant to just skip a few rules,
but it is now being used to skip entire sections of rules and this is
*very* slow.

We need a way to do a real skipAfter without nop'ing each rule in
between.  This has to also handle rule removal/updating at config time
*and* runtime, which is where it gets tricky.

Originally I was thinking a linked list with a pointer to the "next
rule" to run on a match success/failure.  But the accounting is tricky
with ctl:ruleRemovebyId.

>
> The plan is to benchmark with the available rule sets.
>

Sounds great.  It may also show some need for optimization in the
rulesets as well.  Any plan for how you will be doing the benchmark?

> The only question is what to branch and when? I am hoping for the
> improvements to be significant enough to warrant a middle digit bump
> and I'd like to see them go into 2.6.0.

The trunk is 2.6 (will be).  Work there as nothing but 2.5.x merges
are getting commited there currently.

thanks,
-B

[Mod-security-developers] Fwd: Performance improvement plan

[Ivan R. <iva...@gm...>]

Forwarding message, as the original did not show in the archives.

Is it getting through?


---------- Forwarded message ----------
From: Ivan Ristic <iva...@gm...>
Date: Fri, Aug 21, 2009 at 5:54 PM
Subject: Performance improvement plan
To: mod...@li...


I am planning to improve the performance of the ModSecurity rule
engine. I've reviewed the source code and come up with the following
ideas:

1. Avoid table lookup currently used to determine if a rule is
multiMatch. I don't suppose this is going to result with a significant
improvement, but it's been getting on my nerves for years now and I
want to get rid of it. I plan to add a variable to the rule structure
and set it at configuration time. Before I do anything, however, I
will investigate how SecRuleUpdateActionById interacts with action
lists.

2. Change the individual rule processing code to avoid discovering
tfns at runtime. At the moment we are re-discovering tfns for every
variable, even though the action list never changes. Resolving the
tfns at configuration time should result a significant performance
improvement.

3. Cache entire target lists. Currently a lot of time is spent
resolving complex variables, adding and removing variables from the
target list, and transforming them in the end. I want to identify the
rules that share target lists, cache a reusable target list when it is
created for the first time, only to reuse it later. The cost of each
rule in the engine should go down a lot. In the worst case scenario we
waste a hash lookup (per rule). This should be countered by the
removal of the multiMatch lookup ;)

I have a couple of other smaller things I may look into, but I suspect
the improvements from the above 3 changes will make me happy enough.

The plan is to benchmark with the available rule sets.

The only question is what to branch and when? I am hoping for the
improvements to be significant enough to warrant a middle digit bump
and I'd like to see them go into 2.6.0.

-- 
Ivan Ristic
Security assessment of your SSL servers
https://www.ssllabs.com/ssldb/

[Mod-security-developers] Performance improvement plan

[Ivan R. <iva...@gm...>]

I am planning to improve the performance of the ModSecurity rule
engine. I've reviewed the source code and come up with the following
ideas:

1. Avoid table lookup currently used to determine if a rule is
multiMatch. I don't suppose this is going to result with a significant
improvement, but it's been getting on my nerves for years now and I
want to get rid of it. I plan to add a variable to the rule structure
and set it at configuration time. Before I do anything, however, I
will investigate how SecRuleUpdateActionById interacts with action
lists.

2. Change the individual rule processing code to avoid discovering
tfns at runtime. At the moment we are re-discovering tfns for every
variable, even though the action list never changes. Resolving the
tfns at configuration time should result a significant performance
improvement.

3. Cache entire target lists. Currently a lot of time is spent
resolving complex variables, adding and removing variables from the
target list, and transforming them in the end. I want to identify the
rules that share target lists, cache a reusable target list when it is
created for the first time, only to reuse it later. The cost of each
rule in the engine should go down a lot. In the worst case scenario we
waste a hash lookup (per rule). This should be countered by the
removal of the multiMatch lookup ;)

I have a couple of other smaller things I may look into, but I suspect
the improvements from the above 3 changes will make me happy enough.

The plan is to benchmark with the available rule sets.

The only question is what to branch and when? I am hoping for the
improvements to be significant enough to warrant a middle digit bump
and I'd like to see them go into 2.6.0.

-- 
Ivan Ristic
Security assessment of your SSL servers
https://www.ssllabs.com/ssldb/

[Mod-security-developers] Coverity Scan for ModSecurity?

[Ivan R. <iva...@gm...>]

Brian,

How do you feel about applying ModSecurity for the Coverity Scan?

http://www.scan.coverity.com/

-- 
Ivan Ristic
Security assessment of your SSL servers
https://www.ssllabs.com/ssldb/

[Mod-security-developers] Function equivalence from modsecurity 1.9x to 2.5x

[Jordi R. <jr...@is...>]

Hi,

In the old modsecurity 1.9x I use the function "perform_action" to force
run of arbitrary ActionSet... ¿what is the equivalence in modsecurity
2.5x? anyone can helpme to found them?

Thanks!

-- 
________________________________
Jordi Rubió Romero
Ingeniero de Software
Dpto. Seguridad Gestionada
jr...@is...

Internet Security Auditors
www.isecauditors.com

c. Santander, 101. Edif. A. 2º
E-08030 Barcelona (Spain)
Tel: +34 93 305 13 18
Fax: +34 93 278 22 48

Pº. de la Castellana, 164-166. Entlo. 1ª
E-28046 Madrid (Spain)
Tel: +34 91 788 57 78
Fax: +34 91 788 57 01
          ____________________________________
Este mensaje y los documentos que, en su caso lleve anexos, pueden
contener información CONFIDENCIAL. Por ello, se informa al destinatario
que la información contenida en el mismo es reservada y su uso no
autorizado, publicación o difusión, entera o parcialmente, tanto en
formato o medio físico como electrónico, sin el previo consentimiento de
Internet Security Auditors, está prohibida legalmente.

Si ha recibido este correo por error, le rogamos que nos lo comunique
por la misma vía o por teléfono (+34 93 305 13 18), se abstenga de
realizar copias del mensaje o remitirlo o entregarlo a otra persona y
proceda a borrarlo de inmediato.

En cumplimiento de la Ley Orgánica 15/1999 de 13 de diciembre de
protección de datos de carácter personal, Internet Security Auditors, le
informa de que sus datos personales se han incluido en ficheros
informatizados titularidad de Internet Security Auditors, que será el
único destinatario de dichos datos, y cuya finalidad exclusiva es la
gestión de clientes y acciones de comunicación comercial, y de que tiene
la posibilidad de ejercer los derechos de acceso, rectificación,
cancelación y oposición previstos en la ley mediante carta dirigida a
Internet Security Auditors, c. Santander, 101. Edif. A. 2º, E-08030
Barcelona, o vía e-mail a la siguiente dirección de correo:
le...@is...

[Mod-security-developers] Accessing modsec_rec and other internal structures of modsecurity from API

[Jordi R. R. <jr...@is...>]

Hi I interested to extend the modsecurity capabilities using their API, 
¿how can access to modsecurity structures from transformation functions?

Thanks
-- 
_________________________________
Jordi Rubió Romero
Ingeniero de Software
Dpto. Seguridad Gestionada
jr...@is...

Internet Security Auditors, S.L.
c. Santander, 101. Edif. A. 2º
E-08030 Barcelona
Tel: +34 93 305 13 18
Fax: +34 93 278 22 48
www.isecauditors.com
           ____________________________________
Este mensaje y los documentos que, en su caso lleve anexos, pueden
contener información CONFIDENCIAL. Por ello, se informa al
destinatario que la información contenida en el mismo es reservada y
su uso no autorizado, publicación o difusión, entera o parcialmente,
tanto en formato o medio físico como electrónico, sin el previo
consentimiento de Internet Security Auditors, está prohibida legalmente.

Si ha recibido este correo por error, le rogamos que nos lo comunique
por la misma vía o por teléfono (93 305 13 18), se abstenga de
realizar copias del mensaje o remitirlo o entregarlo a otra persona y
proceda a borrarlo de inmediato.

En cumplimiento de la Ley Orgánica 15/1999 de 13 de diciembre de
protección de datos de carácter personal, Internet Security Auditors
S.L., le informa de que sus datos personales se han incluido en
ficheros informatizados titularidad de Internet Security Auditors
S.L., que será el único destinatario de dichos datos, y cuya finalidad
exclusiva es la gestión de clientes y acciones de comunicación
comercial, y de que tiene la posibilidad de ejercer los derechos de
acceso, rectificación, cancelación y oposición previstos en la ley
mediante carta dirigida a Internet Security Auditors, c. Santander,
101. Edif. A. 2º, 08030 Barcelona, o vía e-mail a la siguiente dirección 
de correo: le...@is...

Re: [Mod-security-developers] Positive filter modelization proposal

[Ivan R. <iva...@br...>]

Vincent Deffontaines wrote:
> Hello Ivan and the list,
> 
> This is the answer from INL to those 16 items you listed. We are sorry
> for the long delay.

Hi Vincent,

Thank you for your answers. I just wanted to let you know I have been
overwhelmed with work recently and that I am not finding any spare time
to dedicate to responding to your email. That does not mean I don't want
to or that I've given up. I will answer eventually, it is just that it
may be a while.

I am sorry I cannot provide a better answer at this time.

-- 
Ivan Ristic

Re: [Mod-security-developers] Positive filter modelization proposal

[Vincent D. <vin...@in...>]

Hello Ivan and the list,

This is the answer from INL to those 16 items you listed. We are sorry
for the long delay.

We'll comment on the 16 topics you mentioned to tell how our model deals
or does not deal with them, and also add an extra requirement we had in
mind.
Before that, here is a short summary:

Some parts of our model[1] are ad-hoc and do not fit well with your
approach, especially with regard to topics 3 to 5 (web site as a
collection of resources, to handle with profiles).  We would like to
redesign it to fit with your approach.  Apart from that, we like the
flexibility and versatility of our system of tags and inheritence and
suggest this be included in the final model.

On Thu, 27 Jul 2006 20:44:29 +0100 Ivan Ristic <iv...@we...> wrote:

>  1. Access to some areas of a web site need not be controlled (allow all).

In our model, an allow-all policy can be expressed with a rule that
matches every resource of a given area.

>  2. Access to some areas of a web site must never be allowed (deny all).

Our model allows for a deny-all policy for a chosen area through the use
of tags and inheritence.  One can set a tag to block every rule which
inherits this tag (deny all).  However the tag/inheritence system may
not be the most simple device to deal with this since one has to add the
tag to every relevant rule (though a user interface or a script could
make this easy).

>  3. Web site is a collection of resources.
>  4. Each resource can be used in one or more different ways (resource
profiles).
>  5. Each profile needs to support a different set of parameters. (For
>     example, one set of parameters is used when the resource is invoked
>     with GET, another set of parameters when resource invoked with POST.)

Due to special requirements in our goals, we saw it the other way around:
our model is a collection of rules.  We think it needs to be redesigned
in order to switch to your approach, which seems better.

>  6. Parameters are identified by their name (choose between case sensitive
>     match and case insensitive match). Parameter attributes:

We match them with regular expressions.

>     6.5. Character distribution

What do you mean?

>     6.2. Cardinality
>     6.3. Character class (or a fixed range of acceptable values)
>          [we will probably need to support regular expressions and
>           functions here]
>     6.4. Length (minimum and maximum)
>
>     [One change I would like to make to my own model is to make the
>      attributes listed above to be standalone constraints on parameters.]
>  7. Additional support for parameter arrays (dynamically created
paramters) is needed.

We use regular expressions for that.

>  9. Support for parameters transported in the URI (e.g. URI append) is
needed.

Not fully supported in our model but this would typically be redesigned
along with the modsecurity rule format.

> 11. Metadata. A set of rules must be wrapped somehow, signed by a
>     publisher and published. Change control (e.g. serial number) needs
>     to be supported.

Our model is an ordered collection of rules.  Consequently, rules from
external publishers would have either to be placed correctly by the user
or to be independant from other possible rules. Maybe the model should
evolve to something not ordered, but rather with a "priority" tag, to
deal with rules that "conflict" with each other. But this seems like a
heavy task for the (admin) user interface : the interface would need to
detect which rules conflict with one another, which seems tough if rules
refer to regular expressions.
How do you see it?

> 12. Support for partial models, e.g. continuous learning.

If we understand correctly, our model is not very good at that because
of its ordered rules (see previous paragraph).

> 16. The format must be simple enough that can be written by hand.

Our format, based on XML, can be painfully written by hand; it was not
specifically designed for this.  It is supposed to be generated with a
user interface or a script.  The most obvious flaw is the numerical
attribute "id" (the ids are unique).  For inheritence, parents must be
specified with their id.  Thus, readability relies on comments, e.g. :
<parent id="3"/>  <!-- host some-site.com -->


We would add a requirement to this list:

17. It should be possible to register which rule blocked a request.
This is for debugging but also improving the set of rules.  This
concerns more the implementation than the model design.  Modsecurity
already allows for this.


Finally, here we copy the requirements which our model does not meet (as
is):

>     6.1. Type (query string field, body field, or file)
>     6.6. Max file size (for files only)
>     6.7. File type (not based on the supplied C-T, of course)
>  8. Support for parameters transported in PATH_INFO is needed.
> 10. Support for parameters that come from other places, for example
request
>     headers and cookies.
> 13. Compatibility with WAPL (see http://www.modsecurity.org/projects/ppr/)
>     also see the same document for ideas that related to the requirements
>     listed here.
> 14. Integration with negative security. E.g. if a part of the model is
>     weak run the request through the negative security tests. If the
anomaly
>     score is low use the request for learning. If it is high - don't.

Regards,



[1] What is mentionned "our model" herein is work we suggested in last
email to this list, and is indeed based on Ivan's initial proposal.

-- 
François Toussenel
Vincent Deffontaines
INL, http://www.inl.fr/

Re: [Mod-security-developers] Proposal on html form protection [Second attempt]

[Ivan R. <iv...@we...>]

Carles Bonamusa wrote:
>
> Hi Ivan,
> 
> I'm back from holidays with renewed energy and willing to get on coding 
> again. First of all I'd like to apologise for having been offline so 
> long, specially because our last conversation got kind of "suspended". 

No need to apologise! Not at all. Especially because I am on my holiday 
right now and will only be back in two weeks :)

Ivan

Re: [Mod-security-developers] Proposal on html form protection [Second attempt]

[Ivan R. <iv...@we...>]

Carles Bonamusa wrote:
>
> We've been also working on a first approach to comment removal. As far
> as it goes we're having some issues trying to manage tricky situations where
> non-standard markup can fool libxml and so fool our comment stripping code,
> but for the very standard case this new feature is almost coded and tested.

  I can see how there will be many "modules" that will want to access the
  HTML output, either in the form of SAX events or as a DOM tree. Do you
  have any suggestions how to implement that? At the very least a module
  could say "I want a DOM tree of the response" or "Parse this page as
  HTML and send me the SAX events". That way there will be only one parser
  with data consumed by many modules


> To send gathered data back to the serer there are two available ways,
> one of them must be selected when writing form markup by using *method*
> attribute to define submission method. When *GET* method is used, client
> browser will build the request as a url based on form's *action*
> attribute, appending successful controls name-value pairs as url
> parameters. The other option is to use *POST* method. Using this,
> /form-dataset/ containing control's name-value pairs is passed in the
> body of the request instead of being chained that data to the request url.

  Yes, in a general case, the parameters can appear in both places
  at the same time (only for POST requests, obviously).


> All previous flaws can be easily identified if some data about form
> structure is stored when form is sent to the user. So the idea is to
> filter outgoing html code issued by the back-end server searching for
> form definitions. Whenever a form markup tag is found , then for every
> submit button defined (which eventually means a different url for the
> action attribute) a form-description block is build containing the
> following data.

  Why do you want to build one block per submit button? A submit button
  as such does not guarantee that a different action attribute value
  will be used.

  I think you should build only one block per form. I understand that you
  need to bind the block to something. This is why I want to suggest to
  create a unique per-form token and to embed it into the form as a hidden
  field. This field will serve as a key to bind the form to. The key
  can be global or session specific. When a form is submitted:

  1) If you see the token you know which form it belongs to.

  2) If you don't see a token then you assume it's a CSRF attack :)


> * A list of defined parameters (name, type, and size if one is
>  defined)
> * A list of hidden fields and their respective preassigned values.
> * For each select control, a list of its predefined available
>   options.
> * submission method.
> * action url.

  You also need to store the encoding method (e.g. multipart or not).


> This data is then encrypted and sent to the user attached to the

  The performance would increase dramatically if you leave the data
  on the server. It would probably be a bit safer to. (No attacks against
  the encrypted content.)


> corresponding form (it could be added as an additional hidden field, or
> issued as a cookie).

  A hidden field is better.


> In all the previous described process, it is assumed that the full form
> definition is known when serving the page. That is, all form components
> are defined by server side application.
> 
> This scenario, although the most frequent, it is not the only one which
> is possible.

  As you say, the proposed defence method is not foolproof. So my question
  is why do it? Wouldn't it be simpler to implement this type of functionality
  through application monitoring and construction of a dynamic positive-security
  model? The difference being you would not need to look at HTML for clues
  - you would be looking at the actual data, meaning this alternative
  approach would also form for forms constructed and changed from JavaScript.

  (Personally I would implement to per-form token thing as a defence against
   CSRF attacks.)

-- 
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall
Apache Security (O'Reilly): http://www.apachesecurity.net

[Mod-security-developers] Proposal on html form protection [Second attempt]

[Carles B. <cbo...@is...>]

Hi Ivan,

Next step in our road-map of improving mod_security is to add some
protection
to html forms. Attached to this mail I've set an initial proposal of our
ideas
of some ways of achieving html forms security. It would be great if you
could
check this ideas out and share your comments/ideas as you kindly did back
when we discussed about cookie management.

We've been also working on a first approach to comment removal. As far=20
as it
goes we're having some issues trying to manage tricky situations where
non-standard markup can fool libxml and so fool our comment stripping cod=
e,
but for the very standard case this new feature is almost coded and teste=
d.

By the way, I would like to congratulate you for the work you've done on
latest 2.0 version. It is really a huge rewrite that has brought so much
clarity and readability to ms code, in addition to all new features and t=
he
ones to come due to brand new ms internal structure.

Last but not least, we expect to be able to port/merge our contributions =
to
2.x branch as soon as I'm back from my holidays and form protection is
coded and
tested.

Best Regards.

PD : Attached documents are text and html version of the proposal. I've
started to use ReStructuredText to write our internal documentation. It
allows
you to write documentation in simple txt with minor and almost invisible
markup (basically the usual space/tab/cr markup used in ordinary text), b=
ut
provides you a set of tools that let you "export"/transform this
"source" text
files to html/latex/pdf. I've found all that very useful because it all l=
et
you focus on important things not losing so much time in writing. It also
allows you to store versioned documentation and access/modify it with
the only
help of a simple text-capable editor. If you're interested, check it out =
at
http://docutils.sourceforge.net/rst.html

--=20
_________________________________
Carles Bonamusa P=E9rez
Ingeniero de Software
Dpto. Desarrollo de Soluciones
cbo...@is...

Internet Security Auditors, S.L.
c. Santander, 101. Edif. A. 2=BA 1=AA
08030 Barcelona
Tel: 93 305 13 18
Fax: 93 278 22 48
www.isecauditors.com
          ____________________________________
Este mensaje y los documentos que, en su caso lleve anexos, pueden
contener informaci=F3n confidencial. Por ello, se informa a quien lo
reciba por error que la informaci=F3n contenida en el mismo es reservada
y su uso no autorizado est=E1 prohibido legalmente, por lo que en tal
caso le rogamos que nos lo comunique por la misma v=EDa o por tel=E9fono
(93 305 13 18), se abstenga de realizar copias del mensaje o remitirlo
o entregarlo a otra persona y proceda a borrarlo de inmediato.

En cumplimiento de la Ley Org=E1nica 15/1999 de 13 de diciembre de
protecci=F3n de datos de car=E1cter personal, Internet Security Auditors
S.L., le informa de que sus datos personales se han incluido en
ficheros informatizados titularidad de Internet Security Auditors
S.L., que ser=E1 el =FAnico destinatario de dichos datos, y cuya finalida=
d
exclusiva es la gesti=F3n de clientes y acciones de comunicaci=F3n
comercial, y de que tiene la posibilidad de ejercer los derechos de
acceso, rectificaci=F3n, cancelaci=F3n y oposici=F3n previstos en la ley
mediante carta dirigida a Internet Security Auditors, c. Santander,
101. Edif. A. 2=BA 1=AA, 08030 Barcelona, o v=EDa e-mail a la siguiente
direcci=F3n de correo: le...@is...

[Mod-security-developers] Proposal on Html Form Protection

[Carles B. <cbo...@is...>]

Hi Ivan,

Next step in our road-map of improving mod_security is to add some
protection
to html forms. Attached to this mail I've set an initial proposal of our
ideas
of some ways of achieving html forms security. It would be great if you
could
check this ideas out and share your comments/ideas as you kindly did back
when we discussed about cookie management.

We've been also working on a first approach to comment removal. As far as=
 it
goes we're having some issues trying to manage tricky situations where
non-standard markup can fool libxml and so fool our comment stripping cod=
e,
but for the very standard case this new feature is almost coded and teste=
d.

By the way, I would like to congratulate you for the work you've done on
latest 2.0 version. It is really a huge rewrite that has brought so much
clarity and readability to ms code, in addition to all new features and t=
he
ones to come due to brand new ms internal structure.

Last but not least, we expect to be able to port/merge our contributions =
to
2.x branch as soon as I'm back from my holidays and form protection is
coded and
tested.

Best Regards.

PD : Attached documents are text and html version of the proposal. I've
started to use ReStructuredText to write our internal documentation. It
allows
you to write documentation in simple txt with minor and almost invisible
markup (basically the usual space/tab/cr markup used in ordinary text), b=
ut
provides you a set of tools that let you "export"/transform this
"source" text
files to html/latex/pdf. I've found all that very useful because it all l=
et
you focus on important things not losing so much time in writing. It also
allows you to store versioned documentation and access/modify it with
the only
help of a simple text-capable editor. If you're interested, check it out =
at
http://docutils.sourceforge.net/rst.html

--=20
_________________________________
Carles Bonamusa P=E9rez
Ingeniero de Software
Dpto. Desarrollo de Soluciones
cbo...@is...

Internet Security Auditors, S.L.
c. Santander, 101. Edif. A. 2=BA 1=AA
08030 Barcelona
Tel: 93 305 13 18
Fax: 93 278 22 48
www.isecauditors.com
           ____________________________________
Este mensaje y los documentos que, en su caso lleve anexos, pueden
contener informaci=F3n confidencial. Por ello, se informa a quien lo
reciba por error que la informaci=F3n contenida en el mismo es reservada
y su uso no autorizado est=E1 prohibido legalmente, por lo que en tal
caso le rogamos que nos lo comunique por la misma v=EDa o por tel=E9fono
(93 305 13 18), se abstenga de realizar copias del mensaje o remitirlo
o entregarlo a otra persona y proceda a borrarlo de inmediato.

En cumplimiento de la Ley Org=E1nica 15/1999 de 13 de diciembre de
protecci=F3n de datos de car=E1cter personal, Internet Security Auditors
S.L., le informa de que sus datos personales se han incluido en
ficheros informatizados titularidad de Internet Security Auditors
S.L., que ser=E1 el =FAnico destinatario de dichos datos, y cuya finalida=
d
exclusiva es la gesti=F3n de clientes y acciones de comunicaci=F3n
comercial, y de que tiene la posibilidad de ejercer los derechos de
acceso, rectificaci=F3n, cancelaci=F3n y oposici=F3n previstos en la ley
mediante carta dirigida a Internet Security Auditors, c. Santander,
101. Edif. A. 2=BA 1=AA, 08030 Barcelona, o v=EDa e-mail a la siguiente
direcci=F3n de correo: le...@is...

Re: [Mod-security-developers] Positive filter modelization proposal

[Ivan R. <iv...@we...>]

Vincent Deffontaines wrote:
> Ivan Ristic wrote:

Vincent,

First of all my apologies for the extended delay.


> How do you want us to work for the better?
> We can try to provide comments about all changes we made, and why we did,
> in order to open a discussion.

That would certainly be interested. Personally I would advance by
compiling a definitive requirement list. It is something we could
use later on to test if we did all we set out to do.

Here are my original requirements:

 1. Access to some areas of a web site need not be controlled (allow all).
 2. Access to some areas of a web site must never be allowed (deny all).
 3. Web site is a collection of resources.
 4. Each resource can be used in one or more different ways (resource profiles).
 5. Each profile needs to support a different set of parameters. (For
    example, one set of parameters is used when the resource is invoked
    with GET, another set of parameters when resource invoked with POST.)

 - It sounds like 4 and 5 describe the same thing.

 6. Parameters are identified by their name (choose between case sensitive
    match and case insensitive match). Parameter attributes:

    6.1. Type (query string field, body field, or file)
    6.2. Cardinality
    6.3. Character class (or a fixed range of acceptable values)
         [we will probably need to support regular expressions and
          functions here]
    6.4. Length (minimum and maximum)
    6.5. Character distribution
    6.6. Max file size (for files only)
    6.7. File type (not based on the supplied C-T, of course)

    [One change I would like to make to my own model is to make the
     attributes listed above to be standalone constraints on parameters.]

 7. Additional support for parameter arrays (dynamically created paramters) is needed.
 8. Support for parameters transported in PATH_INFO is needed.
 9. Support for parameters transported in the URI (e.g. URI append) is needed.
10. Support for parameters that come from other places, for example request
    headers and cookies.
11. Metadata. A set of rules must be wrapped somehow, signed by a
    publisher and published. Change control (e.g. serial number) needs
    to be supported.
12. Support for partial models, e.g. continuous learning.
13. Compatibility with WAPL (see http://www.modsecurity.org/projects/ppr/)
    also see the same document for ideas that related to the requirements
    listed here.
14. Integration with negative security. E.g. if a part of the model is
    weak run the request through the negative security tests. If the anomaly
    score is low use the request for learning. If it is high - don't.
16. The format must be simple enough that can be written by hand.

-- 
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall
Apache Security (O'Reilly): http://www.apachesecurity.net

Re: [Mod-security-developers] Proposal on html form protection [Second attempt]

[Ivan R. <iv...@we...>]

Carles Bonamusa wrote:
> Hi Ivan,
>
> Next step in our road-map of improving mod_security is to add some
> protection
> to html forms. Attached to this mail I've set an initial proposal of our
> ideas
> of some ways of achieving html forms security. It would be great if you
> could
> check this ideas out and share your comments/ideas as you kindly did back
> when we discussed about cookie management.

  Thanks! I'll review it over the weekend and respond by Monday.


> We've been also working on a first approach to comment removal. As far
> as it
> goes we're having some issues trying to manage tricky situations where
> non-standard markup can fool libxml and so fool our comment stripping code,
> but for the very standard case this new feature is almost coded and tested.

  Great!


> By the way, I would like to congratulate you for the work you've done on
> latest 2.0 version. It is really a huge rewrite that has brought so much
> clarity and readability to ms code, in addition to all new features and the
> ones to come due to brand new ms internal structure.

  Thanks. It's been a lot of hard work. But it's something I have planned
  for many months. Just FYI there's another bit missing, which I plan
  to implement to accommodate your contributions - a framework for ModSecurity
  modules. (There's not much work left to do.)


> Last but not least, we expect to be able to port/merge our contributions to
> 2.x branch as soon as I'm back from my holidays and form protection is
> coded and
> tested.

  Excellent. I will do my best to make that happen sooner rather than
  later.

-- 
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall
Apache Security (O'Reilly): http://www.apachesecurity.net

[Mod-security-developers] Proposal on html form protection [Second attempt]

[Carles B. <cbo...@is...>]

Hi Ivan,

Next step in our road-map of improving mod_security is to add some
protection
to html forms. Attached to this mail I've set an initial proposal of our
ideas
of some ways of achieving html forms security. It would be great if you
could
check this ideas out and share your comments/ideas as you kindly did back
when we discussed about cookie management.

We've been also working on a first approach to comment removal. As far
as it
goes we're having some issues trying to manage tricky situations where
non-standard markup can fool libxml and so fool our comment stripping cod=
e,
but for the very standard case this new feature is almost coded and teste=
d.

By the way, I would like to congratulate you for the work you've done on
latest 2.0 version. It is really a huge rewrite that has brought so much
clarity and readability to ms code, in addition to all new features and t=
he
ones to come due to brand new ms internal structure.

Last but not least, we expect to be able to port/merge our contributions =
to
2.x branch as soon as I'm back from my holidays and form protection is
coded and
tested.

Best Regards.

PD : Attached documents are text and html version of the proposal. I've
started to use ReStructuredText to write our internal documentation. It
allows
you to write documentation in simple txt with minor and almost invisible
markup (basically the usual space/tab/cr markup used in ordinary text), b=
ut
provides you a set of tools that let you "export"/transform this
"source" text
files to html/latex/pdf. I've found all that very useful because it all l=
et
you focus on important things not losing so much time in writing. It also
allows you to store versioned documentation and access/modify it with
the only
help of a simple text-capable editor. If you're interested, check it out =
at
http://docutils.sourceforge.net/rst.html

--=20
_________________________________
Carles Bonamusa P=E9rez
Ingeniero de Software
Dpto. Desarrollo de Soluciones
cbo...@is...

Internet Security Auditors, S.L.
c. Santander, 101. Edif. A. 2=BA 1=AA
08030 Barcelona
Tel: 93 305 13 18
Fax: 93 278 22 48
www.isecauditors.com
           ____________________________________
Este mensaje y los documentos que, en su caso lleve anexos, pueden
contener informaci=F3n confidencial. Por ello, se informa a quien lo
reciba por error que la informaci=F3n contenida en el mismo es reservada
y su uso no autorizado est=E1 prohibido legalmente, por lo que en tal
caso le rogamos que nos lo comunique por la misma v=EDa o por tel=E9fono
(93 305 13 18), se abstenga de realizar copias del mensaje o remitirlo
o entregarlo a otra persona y proceda a borrarlo de inmediato.

En cumplimiento de la Ley Org=E1nica 15/1999 de 13 de diciembre de
protecci=F3n de datos de car=E1cter personal, Internet Security Auditors
S.L., le informa de que sus datos personales se han incluido en
ficheros informatizados titularidad de Internet Security Auditors
S.L., que ser=E1 el =FAnico destinatario de dichos datos, y cuya finalida=
d
exclusiva es la gesti=F3n de clientes y acciones de comunicaci=F3n
comercial, y de que tiene la posibilidad de ejercer los derechos de
acceso, rectificaci=F3n, cancelaci=F3n y oposici=F3n previstos en la ley
mediante carta dirigida a Internet Security Auditors, c. Santander,
101. Edif. A. 2=BA 1=AA, 08030 Barcelona, o v=EDa e-mail a la siguiente
direcci=F3n de correo: le...@is...

Re: [Mod-security-developers] Positive filter modelization proposal

[Vincent D. <vin...@in...>]

Ivan Ristic wrote:

>> Greetings,
>>
>> Here is a suggestion of XML modelization for an HTTP filter, based on a
>> positive model (ie, not blacklisting known attacks, but rather accepting
>> only known, validated good requests). This model is heavily inspired
>> from Ivan's publication of november 2005.
>
> I am somewhat confused. You took my work, changed it heavily, and
> now you are sending it back to me as your proposal. Wouldn't it
> have been better to point to the inefficiencies of my work so that
> we can improve it through discussion?

Yes, it would have been better, and I am sorry we couldn't.
As I thought I mentionned in former private emails, we had to work in big
hurry on the project, and we had to discuss the model internally in order
to get something we could use quickly. That is why we didn't have that
normal exchange, that I agree would have been much better for everyone.
Though, I think we have advanced, and I hope you can agree on some points
that we have changed from your initial proposal.

How do you want us to work for the better?
We can try to provide comments about all changes we made, and why we did,
in order to open a discussion.

PS : sorry I am destroying the thread this was started in ; for some
reason I did not receive your answer, even though I am subscribed to the
mailing list. I had to paste it from archives to write this answer.

Re: [Mod-security-developers] Positive filter modelization proposal

[Ivan R. <iv...@we...>]

Vincent Deffontaines wrote:
> Greetings,
> 
> Here is a suggestion of XML modelization for an HTTP filter, based on a
> positive model (ie, not blacklisting known attacks, but rather accepting
> only known, validated good requests). This model is heavily inspired
> from Ivan's publication of november 2005.

  I am somewhat confused. You took my work, changed it heavily, and
  now you are sending it back to me as your proposal. Wouldn't it
  have been better to point to the inefficiencies of my work so that
  we can improve it through discussion?

-- 
Ivan Ristic, Technical Director
Thinking Stone, http://www.thinkingstone.com
ModSecurity: Open source Web Application Firewall
Apache Security (O'Reilly): http://www.apachesecurity.net

[Mod-security-developers] Positive filter modelization proposal

[Vincent D. <vin...@in...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Greetings,

Here is a suggestion of XML modelization for an HTTP filter, based on a
positive model (ie, not blacklisting known attacks, but rather accepting
only known, validated good requests). This model is heavily inspired
from Ivan's publication of november 2005.

When/if this model is validated, we [INL] will be in position to publish
some work we have started working on, especially about :
- - generating rules expressed in this model, based on [mod_security] log
analysis (log of traffic considered as good)
- - building mod_security rules from the XML model.
- - possibly [at least partially] converting rules from proprietary filters

Should this model be amended (as will probably be), we will of course
adapt our (alpha stage) tools to take advantage of it.

All comments and suggestions will, of course, be most welcome.

Vincent Deffontaines
INL
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFErWSCam6ayBS3Nb4RAjmLAJ9xvUiJfRZ3dXgdjkEKjGJtyjs79wCdHl3P
G/xuVXCxjUg2ZX1ylqT+SGo=
=E34K
-----END PGP SIGNATURE-----

[Mod-security-developers] test

[Daniel F. B. <dfe...@is...>]

test
--=20
_________________________________
Daniel Fern=E1ndez Bleda
Gerente de cuentas
Consultor en Seguridad
CISA, CISSP
OPSA/OPST Trainer, CHFI Instructor
dfe...@is...

Internet Security Auditors, S.L.
c. Santander, 101. Edif. A. 2=BA 1=AA.
08030 Barcelona
Tel: 93 305 13 18
Fax: 93 278 22 48
www.isecauditors.com
          ____________________________________
Este mensaje y los documentos que, en su caso lleve anexos, pueden
contener informaci=F3n confidencial. Por ello, se informa a quien lo
reciba por error que la informaci=F3n contenida en el mismo es reservada
y su uso no autorizado est=E1 prohibido legalmente, por lo que en tal
caso le rogamos que nos lo comunique por la misma v=EDa o por tel=E9fono
(93 305 13 18), se abstenga de realizar copias del mensaje o remitirlo
o entregarlo a otra persona y proceda a borrarlo de inmediato.

En cumplimiento de la Ley Org=E1nica 15/1999 de 13 de diciembre de
protecci=F3n de datos de car=E1cter personal, Internet Security Auditors
S.L., le informa de que sus datos personales se han incluido en
ficheros informatizados titularidad de Internet Security Auditors
S.L., que ser=E1 el =FAnico destinatario de dichos datos, y cuya finalida=
d
exclusiva es la gesti=F3n de clientes y acciones de comunicaci=F3n
comercial, y de que tiene la posibilidad de ejercer los derechos de
acceso, rectificaci=F3n, cancelaci=F3n y oposici=F3n previstos en la ley
mediante carta dirigida a Internet Security Auditors, c. Santander,
101. Edif. A. 2=BA 1=AA, 08030 Barcelona, o v=EDa e-mail a la siguiente
direcci=F3n de correo: le...@is...