Webware for Python

You might find BuildBot useful.
http://buildbot.sourceforge.net/

Ed

On Mon, 6 Dec 2004 10:04:18 -0500
Winston Wolff <stuff2@...> wrote:

> Hi Mark and other Webware developers-
>=20
> Here are my first thoughts on organizing automated test cases.  To
> make it easy for people to contribute automated tests, I think we
> need:
> 	=95 Test discussion - A way to get feedback when you are thinking
> 	about=20
> writing a test.  Although a test might be only a few lines of code,=20
> it's affect on Webware is significant.  Writing a test implies that
> all of Webware must now comply with this test and the "proper" result
> may not be obvious for some cases.  So people who are new might need
> help to know what to test for, especially taking into account other=20
> platforms.  Simply discussing on the development list is probably=20
> sufficient, but the issue should be mentioned in the documentation.
> 	=95 Easy to run tests - When writing a test, I want to know that I
> 	am=20
> not breaking things on other platforms that I cannot test on.  So
> there should be a way to get feedback quickly after checking in a new
> test.  Conversely, all the people who are developing with Webware are=20
> reluctant to download new versions and run tests if they break things=20
> and disrupt their own development.  So I think Webware needs to be=20
> arranged so you can have two separate versions on your machine.  One=20
> that you develop with, and a second that you can download often just
> to run the tests.  This implies that we will not use disutils to
> install Webware into the Python standard places, and that we have a
> way to control the python path to determine which version of Webware
> is being used.
> 	=95 Testing Configuration - With an automated testing system, we
> 	need a=20
> way to specify what tests run on my computer, e.g. I am on a Mac so=20
> don't run COMKit tests, or I have only PostgreSQL so don't run MySQL
> or MSSQL tests, or  I have Apache with mod_kit or I use the cgi
> adapter.
> 	=95 Testing daemon - Eventually, we might setup a "continuous
> 	testing=20
> daemon" that people can run on their machine which will check for new=20
> versions in CVS, download them to a separate place, run the tests, and
>=20
> email any failures to a mailing list.  I've heard of one for Ant,=20
> perhaps we could use that, or write a simple one in Python.  That
> would allow developers to get feedback on new tests from all platforms
> within minutes of committing a new test.
>=20
> So to accomplish this we need:
> 	=95 Documentation describing the need to discuss a test on the=20
> webware-devel list, and describing how to add a test, run the tests,=20
> etc.
> 	=95=A0alltests.py in each kit - We need a standard module that
> 	contains=20
> all the automated tests for that component.  I have always called it=20
> alltests.py.  It has a suite() function that returns the suite of=20
> unittest.TestCases for that component.
> 	=95 Test Launcher - A script that will set the python path, choose
> 	the=20
> local machine's configuration, and then run all or just some tests.
>=20
> How does that sound?
>=20
> -winston
>=20
> On Nov 29, 2004, at 6:06 PM, Mark Phillips wrote:
> > On Nov 29, 2004, at 12:13 PM, Winston Wolff wrote:
> >
> >> I have used WebUnit very successfully for high level tests. =20
> >> (http://mechanicalcat.net/tech/webunit)
> >>
> >> I haven't been able to figure out how to unit-test pages since=20
> >> anything that extends Page requires the AppServer to be running. =20
> >> Perhaps part of the dependency can be eliminated with a mock=20
> >> HTTPTransaction, but I'm not sure.
> >>
> >> I think it would be good to organize a working group of people=20
> >> willing to work on tests.  I know several people including myself=20
> >> have volunteered to work on the release, but not to lead it=20
> >> themselves citing inexperience.  However I have not been "called to
> >
> >> service" yet.  A difficulty with volunteering to do a release is to
> >
> >> know if I've made a mistake in the release process.  Hence the need
> >
> >> for tests.  Once we have a group together, we can pick a few=20
> >> priorities in the tests, and settle anything needed to let people
> >get > to work individually, such as how our tests will fit into the
> >larger > set of test cases.
> >>
> >> -winston
> >>
> >
> > Thanks for the reply, Winston. I will add your commentary and take a
> >=20
> > look at WebUnit when I get the chance.
> >
> > I am open to discussing how a "release team" might be organized and=20
> > operate. Feel free to send me a note off-list if you wish.
> >
> > Mark Phillips
> _________________________________________
> winston wolff - (646) 827-2242 - http://www.stratolab.com - learning by=20
> creating
>=20

Yes, BuildBot is perfect.

So does this sound like something we should work towards?  If others=20
agree, I'll start working on it as time permits.

-ww

On Dec 7, 2004, at 3:00 AM, Ed Rahn wrote:

> You might find BuildBot useful.
> http://buildbot.sourceforge.net/
>
> Ed
>
> On Mon, 6 Dec 2004 10:04:18 -0500
> Winston Wolff <stuff2@...> wrote:
>
>> Hi Mark and other Webware developers-
>>
>> Here are my first thoughts on organizing automated test cases.  To
>> make it easy for people to contribute automated tests, I think we
>> need:
>> 	=95 Test discussion - A way to get feedback when you are =
thinking
>> 	about
>> writing a test.  Although a test might be only a few lines of code,
>> it's affect on Webware is significant.  Writing a test implies that
>> all of Webware must now comply with this test and the "proper" result
>> may not be obvious for some cases.  So people who are new might need
>> help to know what to test for, especially taking into account other
>> platforms.  Simply discussing on the development list is probably
>> sufficient, but the issue should be mentioned in the documentation.
>> 	=95 Easy to run tests - When writing a test, I want to know that =
I
>> 	am
>> not breaking things on other platforms that I cannot test on.  So
>> there should be a way to get feedback quickly after checking in a new
>> test.  Conversely, all the people who are developing with Webware are
>> reluctant to download new versions and run tests if they break things
>> and disrupt their own development.  So I think Webware needs to be
>> arranged so you can have two separate versions on your machine.  One
>> that you develop with, and a second that you can download often just
>> to run the tests.  This implies that we will not use disutils to
>> install Webware into the Python standard places, and that we have a
>> way to control the python path to determine which version of Webware
>> is being used.
>> 	=95 Testing Configuration - With an automated testing system, we
>> 	need a
>> way to specify what tests run on my computer, e.g. I am on a Mac so
>> don't run COMKit tests, or I have only PostgreSQL so don't run MySQL
>> or MSSQL tests, or  I have Apache with mod_kit or I use the cgi
>> adapter.
>> 	=95 Testing daemon - Eventually, we might setup a "continuous
>> 	testing
>> daemon" that people can run on their machine which will check for new
>> versions in CVS, download them to a separate place, run the tests, =
and
>>
>> email any failures to a mailing list.  I've heard of one for Ant,
>> perhaps we could use that, or write a simple one in Python.  That
>> would allow developers to get feedback on new tests from all =
platforms
>> within minutes of committing a new test.
>>
>> So to accomplish this we need:
>> 	=95 Documentation describing the need to discuss a test on the
>> webware-devel list, and describing how to add a test, run the tests,
>> etc.
>> 	=95=A0alltests.py in each kit - We need a standard module that
>> 	contains
>> all the automated tests for that component.  I have always called it
>> alltests.py.  It has a suite() function that returns the suite of
>> unittest.TestCases for that component.
>> 	=95 Test Launcher - A script that will set the python path, =
choose
>> 	the
>> local machine's configuration, and then run all or just some tests.
>>
>> How does that sound?
>>
>> -winston
>>
>> On Nov 29, 2004, at 6:06 PM, Mark Phillips wrote:
>>> On Nov 29, 2004, at 12:13 PM, Winston Wolff wrote:
>>>
>>>> I have used WebUnit very successfully for high level tests.
>>>> (http://mechanicalcat.net/tech/webunit)
>>>>
>>>> I haven't been able to figure out how to unit-test pages since
>>>> anything that extends Page requires the AppServer to be running.
>>>> Perhaps part of the dependency can be eliminated with a mock
>>>> HTTPTransaction, but I'm not sure.
>>>>
>>>> I think it would be good to organize a working group of people
>>>> willing to work on tests.  I know several people including myself
>>>> have volunteered to work on the release, but not to lead it
>>>> themselves citing inexperience.  However I have not been "called to
>>>
>>>> service" yet.  A difficulty with volunteering to do a release is to
>>>
>>>> know if I've made a mistake in the release process.  Hence the need
>>>
>>>> for tests.  Once we have a group together, we can pick a few
>>>> priorities in the tests, and settle anything needed to let people
>>> get > to work individually, such as how our tests will fit into the
>>> larger > set of test cases.
>>>>
>>>> -winston
>>>>
>>>
>>> Thanks for the reply, Winston. I will add your commentary and take a
>>>
>>> look at WebUnit when I get the chance.
>>>
>>> I am open to discussing how a "release team" might be organized and
>>> operate. Feel free to send me a note off-list if you wish.
>>>
>>> Mark Phillips
>> _________________________________________
>> winston wolff - (646) 827-2242 - http://www.stratolab.com - learning by
>> creating
>>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real=20
> users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Webware-devel mailing list
> Webware-devel@...
> https://lists.sourceforge.net/lists/listinfo/webware-devel
>
_________________________________________
winston wolff - (646) 827-2242 - http://www.stratolab.com - learning by=20
creating

On Mon, 2004-12-06 at 10:04 -0500, Winston Wolff wrote:
> So to accomplish this we need:
> Documentation describing the need to discuss a test on the webware-devel
> list, and describing how to add a test, run the tests, etc.

> alltests.py in each kit - We need a standard module that contains all
> the automated tests for that component.  I have always called it
> alltests.py.  It has a suite() function that returns the suite of
> unittest.TestCases for that component. 

Hi Winston,

You may want to take a look at the stuff in WebKit/Tests which I checked
in last week.  It's a framework to run tests using the built-in HTTP
server, but would work with cgi and mod_webkit, too, if the user has
configured their webserver for that (the nice thing about the built-in
HTTP server is that it doesn't need the user to do any configuration
in order to run tests).  Right now the suite only has one test to see that the
AppServer can be started -- that was my proof-of concept.  Tests can look at
the AppServer output (implemented), and could also make HTTP requests using
webunit or something and test the responses (unimplemented).

> Test Launcher - A script that will set the python path, choose the local
> machine's configuration, and then run all or just some tests.

Re: setting the python path, check out WebKit/Tests/FixPath.py.  I grabbed this 
from Chuck's MiddleKit test suite, and it sets the python path (sys.path) to the 
Webware tree in which the test is located.  

> How does that sound?

This sounds wonderful.  Do you have CVS write access?  If not, maybe get Geoff 
to give you access.

peace,
Jason

I've put up a page on the wiki in SubProjects > UnitTestingProj 
(http://wiki.w4py.org/unittestingproj)  There was already an old 
project by this name, and since the last date was 2002 and destination 
version was v0.7, it seemed like it was done.  So I put my page over 
the old page to get rid of outdated stuff.  If Tavis Rudd, the old 
project leader doesn't like that, I can make a different page.

I'm going to start examining the codebase to assess the current state 
of tests.  I'll report my findings here.

If anybody would like to add their names to the project, I would be 
grateful.  I think the work will mostly entail converting the existing 
tests that just output results to the user, into automated tests.

-winston

_________________________________________
winston wolff - (646) 827-2242 - http://www.stratolab.com - learning by 
creating

Hi Jason-

Your base AutomatedTestCase looks great.  I will plan to use it for our 
new project.

Who is this Geoff person, and how to I contact him to get CVS access?

-winston

On Dec 9, 2004, at 9:24 AM, Jason Hildebrand wrote:

> On Mon, 2004-12-06 at 10:04 -0500, Winston Wolff wrote:
>> So to accomplish this we need:
>> Documentation describing the need to discuss a test on the 
>> webware-devel
>> list, and describing how to add a test, run the tests, etc.
>
>> alltests.py in each kit - We need a standard module that contains all
>> the automated tests for that component.  I have always called it
>> alltests.py.  It has a suite() function that returns the suite of
>> unittest.TestCases for that component.
>
> Hi Winston,
>
> You may want to take a look at the stuff in WebKit/Tests which I 
> checked
> in last week.  It's a framework to run tests using the built-in HTTP
> server, but would work with cgi and mod_webkit, too, if the user has
> configured their webserver for that (the nice thing about the built-in
> HTTP server is that it doesn't need the user to do any configuration
> in order to run tests).  Right now the suite only has one test to see 
> that the
> AppServer can be started -- that was my proof-of concept.  Tests can 
> look at
> the AppServer output (implemented), and could also make HTTP requests 
> using
> webunit or something and test the responses (unimplemented).
>
>> Test Launcher - A script that will set the python path, choose the 
>> local
>> machine's configuration, and then run all or just some tests.
>
> Re: setting the python path, check out WebKit/Tests/FixPath.py.  I 
> grabbed this
> from Chuck's MiddleKit test suite, and it sets the python path 
> (sys.path) to the
> Webware tree in which the test is located.
>
>> How does that sound?
>
> This sounds wonderful.  Do you have CVS write access?  If not, maybe 
> get Geoff
> to give you access.
>
> peace,
> Jason
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real 
> users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Webware-devel mailing list
> Webware-devel@...
> https://lists.sourceforge.net/lists/listinfo/webware-devel
>
_________________________________________
winston wolff - (646) 827-2242 - http://www.stratolab.com - learning by 
creating

Below is a list of all the tests that I've found in the current CVS=20
version of Webware.  UserKit, WebUtils, and TaskKit have some automated=20=

test extending unittest.TestCase.  But they also have some=20
non-automated test.  MiddleKit has a bunch of automated tests, but they=20=

don't extend unittest.

The current situation makes sense--the easy compartmentalized things=20
have some unittests, but the harder tests that use the appserver are=20
not automated.

I will get all the tests running on my machine, then try to automate=20
them.  The goal is to have things arranged such that one can:
	=95 download the latest version,
	=95 do as little setup as possible, hopefully none, and then
	=95 run some or all tests and they will "just work."

-winston
-------------

WebKit/Tests/=09
	Jason H's automated functional base class and test suite
			AppServerTest.py - Base class that runs Webware =
in a thread and then=20
runs unittests.
			Test.py
			Basic/	- A webware environment for the purpose =
of these tests.
				Test.py	- Jason's one sample =
TestBasicFunctionality, using=20
AppServerTest
			=09
WebKit/Testing/=09
	Contains a number of tests that are not automated, i.e. the =
result
	must be evaluated by the user to see if the test worked or not.
			EmbeddedServlet.py - Tests "extra path =
info"-such as:
				=
http://localhost/WebKit.cgi/Servlet/Extra/Path/Info
				Where the servlet is embedded in the =
URL, rather than being the=20
last component.
			Forward*.py - Tests =
trans.application().forward()
			IncludeURLTest.py - test the callMethodOfServlet =
and includeURL=20
redirection.
			Main.py
			Servlet.py - test req.serverSidePath() and =
req.extraURLPath()
			SetCookie.py - tests setting and retrieving =
cookies
			TestIMS.py - Test If-Modified-Since support in =
Webware
			stress.py - Purpose: Hit the WebKit AppServer =
with lots of a=20
requests in order to:
				Test for memory leaks, concurrency, =
Investigate performance

WebUtils/Tests/
		TestFuncs.py - tests performace of utility functions in =
WebUtils vs.=20
their standard Python alternatives.
		TestHTMLTag.py - unittest.TestCase for =
WebUtils.HTMLTag.HTMLReader=20
class
=09
UserKit/Tests/
		Test.py - unittest.TestCase which seems quite extensive.
		=09
TaskKit/Tests/
		BasicTest.py - tests task scheduling.
		Test.py - unittest.TestCase with some setup, but no =
actual outcome=20
testing.  Perhpas it tests that the scheduler at least doesn't fire=20
internal assertions.
		=09
PSP/	- no tests.
		=09
MiddleKit/Tests/	- A hand-rolled(?) testing framework.  There are =
24=20
models, and each model has a
				TestEmpty.py and/or TestSamples.py file =
which has self-evaluating=20
tests inside it.
		=09
COMKit/	- Seems to be empty.  Is that right?

_________________________________________
winston wolff - (646) 827-2242 - http://www.stratolab.com - learning by=20
creating

Can anybody think of a good reason to have an alternate version of 
unittest.py in MiscUtils, since it is included in Python already? It's 
been standard since Python 2.1.  I propose we remove 
MiscUtils/unittest.py since it is confusing things to have two versions 
of unittest.

Also, regarding timing, how do people feel about these changes and the 
new release?  I suspect I should NOT include them since I'm starting 
late.  I can check them in for the next release (9.1).

-winston

Are other people getting failures if they run this?  I get 10 failures 
out of 23 tests.

	cd Webware/UserKit/Tests/
	python Test.py

-winston

On Mon, 2004-12-13 at 15:28 -0500, Winston Wolff wrote:
> I've put up a page on the wiki in SubProjects > UnitTestingProj (http://
> wiki.w4py.org/unittestingproj)  There was already an old project by this
> name, and since the last date was 2002 and destination version was v0.7, it
> seemed like it was done.  So I put my page over the old page to get rid of
> outdated stuff.  If Tavis Rudd, the old project leader doesn't like that, I
> can make a different page.

It's good to get rid of the cruft.  I haven't seen Tavis around these
lists since 2002, so I don't think he will mind.

peace,
Jason

On Mon, 2004-12-13 at 19:39 -0500, Winston Wolff wrote:
> Can anybody think of a good reason to have an alternate version of
> unittest.py in MiscUtils, since it is included in Python already? It's
> been standard since Python 2.1.  I propose we remove
> MiscUtils/unittest.py since it is confusing things to have two versions
> of unittest.

Sounds good to me.  A survey done a while ago showed that nearly all 
Webware users are using Python 2.2 or newer, so I don't think this is an
issue.

> Also, regarding timing, how do people feel about these changes and the
> new release?  I suspect I should NOT include them since I'm starting
> late.  I can check them in for the next release (9.1).

Yes, I'd be hesitant to make too many changes just prior to release.
Waiting until after the release to check in your changes sounds like a 
good plan to me.

peace,
Jason

On Mon, 2004-12-13 at 20:23 -0500, Winston Wolff wrote:
> Are other people getting failures if they run this?  I get 10 failures
> out of 23 tests.

> 	cd Webware/UserKit/Tests/
> 	python Test.py

I get 29 failures out of 61 tests.  I don't know why it's trying to run
more tests than it does for you.  In case you don't know this, though,
UserKit is in an incomplete state and hasn't really been worked on in
two years.  It's possible that these tests have _NEVER_ passed.

In the documentation, UserKit is clearly marked as pre-alpha, so this
should be clear to users.  As it seems unlikely (after two years)
that Chuck or anyone else will ever continue work on UserKit, I think
we should actually consider removing UserKit -- this decision may not
take place any time soon, though.

For the purposes of testing, I'd suggest simply excluding UserKit from the
test suite.


peace,
Jason

On Tue, 14 Dec 2004 11:17:50 -0600, Jason Hildebrand wrote:
> On Mon, 2004-12-13 at 20:23 -0500, Winston Wolff wrote:
>
>> Are other people getting failures if they run this?  I get 10
>> failures out of 23 tests.
>>
>
>> cd Webware/UserKit/Tests/
>> python Test.py
>>
>
> I get 29 failures out of 61 tests.  I don't know why it's trying to
> run more tests than it does for you.  In case you don't know this,
> though, UserKit is in an incomplete state and hasn't really been
> worked on in two years.  It's possible that these tests have
> _NEVER_ passed.

UserKit was built and working (but only in combination with MiddleKit for=
 its backing storage). Originally, ALL tests passed.


> In the documentation, UserKit is clearly marked as pre-alpha, so
> this should be clear to users.  As it seems unlikely (after two
> years) that Chuck or anyone else will ever continue work on
> UserKit, I think we should actually consider removing UserKit --
> this decision may not take place any time soon, though.

I developed and used UserKit at a client of mine that I no longer work for.=
 I will forward this message to their current developer and see if he's=
 interested in looking at this.


-Chuck