Re: [phpesp-dev] updated version

[Bishop B. <ph...@id...>]

Matthew raises a very good point.  In one of our products, we had a  
similar hurdle: lots of very old code (early PHP 4 based) that was  
rapidly produced in a not-so-modular fashion.  Wholesale rewriting,  
while desired, was not feasible given the clients using the code and  
the sheer number of lines to change.

To incorporate standardized testing, we first defined exactly what we  
wanted tested (ie, specific areas that were prone to changes or known  
to be problematic) and how we wanted to test (ie, unit, stress,  
penetration, and coverage testing).

Then, we settled on a robust framework -- phpunit2 -- that would grow  
with us in that, and other, products.

Finally, we made two rules:
1. All new code SHOULD conform to our testing policy
2. All bugs MUST get a test case to expose them and to prove they have  
been fixed

(Interpret MUST and SHOULD according to RFC2119)

That process allowed us to slowly migrate a consistent testing  
strategy into the old code.  It has been more cost effective than a  
whole sale restart.

Here's how we usually set up tests structure:

test/
   smoke/
   regression/
    stress/
    security/

Smoke tests are those that absolutely must pass each build.  phpESP  
doesn't really have a "build" process (IMO, it should), but the type  
of tests in this category are "config file exists, is readable, and  
parseable by PHP", "database can be accessed and has good schema",  
"web directory exists and is readable by the web server", etc.

Regression tests are every thing else: bug fix tests, new code tests, etc.

Stress tests are a particular breed of regression tests, in that  
they're designed to wallop the web UI or DB with a large number of  
requests to exercise the code under significant duress and make sure  
it meets its performance/responsiveness requirements.

Security tests, also a breed of regression, specifically try to pry  
open the application: SQL injection attacks, man in the middle  
attacks, brute-force or dictionary attacks, etc.

Then as new scenarios come up, we drop unit tests into the appropriate  
spot.  We exercise the software with our build tool phing, such as:  
"phing build", "phing regression", "phing stress", and "phing penetrate"

We definitely didn't get there overnight.  It's been a steady task of  
adding in tests.  Over time, it becomes a natural reaction of the  
developer and a significant suite is developed.  We just had to draw a  
line in the sand and exclaim "Today, we incorporate tests."


So, leaders of phpESP: what is the testing policy?  Do we incorporate  
slowly as legacy evolves, or start fresh?  I'm in the incorporate  
slowly camp, but maybe that's because I never worked for Netscape:
http://www.joelonsoftware.com/articles/fog0000000069.html


bishop


Quoting Matthew Gregg <mat...@gm...>:

> Unit testing sounds like a great thing to add.  My concern is the work
> to add and maintain would be significant given phpESP legacy code base.
> Perhaps more work than starting over using an off the shelf framework
> that supports proper unit testing.
>
> Thoughts?
>
> On Sat, 2008-04-19 at 12:52 -0400, Bishop Bettini wrote:
> ...snip...
>> Now, a bigger question: any votes on adding a unit test framework to
>> phpESP so that this bug can be installed as a test case?  (We use
>> phpunit extensively on our other products, and I recommend it.)  One
>> problem is that the phpESP code is a bit resilient to unit testing:
>> the UI, business logic, and DB code are all wrapped into one.  I
>> suppose it could be done by munging POST, including the file, and
>> checking the results both in the DB and a regex/DOM parse of the
>> output buffer.  Thoughts?


-- 
Bishop Bettini
ideacode, Inc.
(main) +1 919 341 5170 / (fax) +1 919 521 4100

Visit us on the web at:
ideacode.com          Professional software research and development
reviewmysoftware.com  Improve sales! Review your software before you release
bytejar.com           Solutions to those annoying development problems