RE: [Owasp-input-api-developers] Testing tool/framework anddevelopment platform survey

[Christopher T. <ch...@ch...>]

Matt,

> The problem will be coming up with enough different cases to
> cover as many methods of "attack" as possible.

Actually, we can simply ask all the gurus on the webappsec list to submit
test cases to us.  I'm sure they'll come up with a ton.

> I don't see why a web server is necessary to test these cases?

When it comes to testing, I always assume it is best to test code in an
environment that mimcs the production environment.  Different web servers
and web app servers will process the input before web apps see it, and they
might play with things like character encodings, etc.

> While OWASP is geared towards web
> applications, I don't see why someone couldn't use these filters in
> other applications.

An excellent point, and one I hope we pursue.  Then again, the filtering
needs of web applications are different from many standalone apps, and our
first goal is to help web app developers.

> We could write a test suite application in an appropriate language (I'm
> thinking perl) that could invoke a simple application written in each
> language that filters test data by a supplied filter, this simple app
> then returns the filtered data to the test suite application which
> determines if the output is correct.  This can be easily run after each
> build.

I think the tricky part here is finding standalone interpreters for all the
languages we will (eventually) support.  I am thinking of things like Cold
Fusion, Cocoon, Zope and Enhydra (which uses XMLC).  Granted, these are not
hugely popular platforms (please no flames, I'm only talking about market
share :-), and I'm not suggesting we support them out of the box.  But
eventually, it might be nice.  And if we were to do a JSP taglib version of
the Filter API (which would also be nice), that would have to be run in a
JSP page.

Perl, PHP (you can use the compiled CGI binary), Python, Java, C, and C++
all have standalone interpreters.  With respect to your Microsoft/ASP
comment below, most ASP pages are written in VBScript (someone correct me if
I'm wrong), for which there is Windows Scripting Host.  I don't know about
.NET, though.

Your suggestion would be simpler than mine, and would generally make things
easier on API porters.  My thought on having just one tool for doing all the
testing for all the languages we support is that maintaining that tool and
the test cases would require less overhead, and would provide greater
consistency across the languages.

> ./testSuite --lang=java -f tests/xssTests.xml ?
>
> The test suite application doesn't have to run everything, only what we
> tell it so people who aren't developing on other languages don't need to
> install anything extra, other than the languages they are using and of
> course the language that the test suite is written (which is why perl
> would probably be best, its available on most major unix distributions
> and I hear ActiveState Java is pretty decent).  Besides, I would think
> that we would want to package up the tools for each language for
> individual distribution.  ( I don't need the java filters for my php
> applications).

An excellent point.  It's just that I'm not a very good Perl hacker.  :-)
Most developers hate writing tests (or comments or documentation), and I
love writing all three, so I thought I could help there.

On the flip side, it means we won't have one testing framework, we'll have
one for every language we support, which could become a maintenance burden.

> AND DAMMIT.  I just realized that my test application idea breaks down
> when we throw ASP into the mix.
> (/me mumbles something dirty about Microsoft)

See my comment above about VBScript.

Chris