More built-in PHPUnit and Selenium testing would be nice; I fear that it'll be hard to find someone willing to set these up, unless maybe there's an opportunity for paid development; but I'd love to be proven wrong.

I think there's a simpler, low-tech tactic, though, that will help a lot in preventing major bugs from getting into releases: have an explicit alpha/testing period between when code changes are done and when the version is released, and ask people to upgrade to the latest SVN version during that period to help with testing.

As a case in point, one of the errors in version 1.5.1 was that the class name was changed from "OWLExport" to "SMWOWLExport", but not everywhere. This change happened on May 30, and version 1.5.1 was released on May 31. I then discovered and fixed the bug on three days later, on June 3:

http://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/SemanticMediaWiki/includes/export/SMW_OWLExport.php?view=log

...even if I hadn't found it, I'm sure someone else would have found it before too long.

Markus did send out a "please test the SVN code" email, but that was also on May 30. A period of, say, two weeks would I think be a better approach.

I should note that I used to do the same thing, or even worse - I would make a flurry of SVN changes right before releasing a new version of one of my extensions. That led to a constant cycle of each version trying to fix the bugs discovered in the version before it, and introducing new bugs along the way. Now I try to check in each code change or patch as soon as I have it, to give more time for testing it before a new version was released; I think it's resulted in better code.

There's no substitute for user testing, especially given the wide variety of uses that SMW gets, and the different MediaWiki setups, browsers, database systems, etc. that it's run on - unit testing just can't capture all of it. I think a substantial "pencils down" period in which users are asked to test everything out would help a lot.

-Yaron


On Wed, Jun 9, 2010 at 1:50 PM, Lane, Ryan <Ryan.Lane@ocean.navo.navy.mil> wrote:
MediaWiki is including support for PHPUnit and Selenium in core. It may be
good to follow the efforts there. You can use me as a Selenium POC for now.
If anyone would like to be added to the Selenium planning contacts, let me
know and I'll introduce you to the other devs working on it. We are doing
all planning on wikitech-l, but it is good to know who the others actively
working on it are.

Respectfully,

Ryan Lane

> -----Original Message-----
> From: Stephan Robotta [mailto:robotta@ontoprise.de]
> Sent: Wednesday, June 09, 2010 6:23 AM
> To: semediawiki-devel@lists.sourceforge.net
> Subject: Re: [SMW-devel] Code quality, and how to improve it
>
> Hello folks,
>
> Jeroen stated a very important point here. In the Halo project (which
> includes various extensions) we suffer from the same problem that it
> costs us to keep the quality of the software.
> To have a greater output from our efforts we have created a build
> environent and run integration tests there.
>
> At one hand we use PHPUnit [1] to test single functions or portions of
> code in an extension. Depending on the specification the tests should
> contain all kinds of input that a function allows and validate the
> result.
>
> In the next level we use Canoo Webtest [2] to test a whole
> installation. Webtests try to simulate a browser and click through the
> application as defined by the testcase. After each test step the html
> of the resulting page can be validated against xpaths to
> check that certain
> elements exist on a page. This is a quite powerful software
> and it gets
> improved continuously by a vivid community. What webtest cannot yet
> serve sufficiently is to control GUI elements. Also javascripts are
> tested only within the Webtest but not like in unit test as it's
> possible for the PHP code.
>
> We have set up tests to test each extension separatly. Test
> are written
> as we add new features to the extension. Initially we also added
> testcases for existing functionality. What still costs us is to keep
> up the tests to a current state and enhance them. For every bug fix a
> new testcase should be added that tests exactly the issue
> where the bug
> arose.
>
> We also plan to have another test scenario that tests a installation
> with various extensions enabled. Often an extension itself works fine
> but in conjunction with others something may break.
>
> With these messurements we try to improve the quality of the software
> and we try to be more prepared to add new features to the
> existing code
> base. The whole testing process needs some effort and
> dicipline so that
> it is acutally useful for the entire project. Test results
> are reported
> within our build system so that after each build we have feedback of
> the quality of our work.
>
> Best regards, Stephan
>
> [1] http://www.phpunit.de/
> [2] http://webtest.canoo.com/webtest/manual/WebTestHome.html
>
> --------------------------------------------------------------
> ----------------
> ThinkGeek and WIRED's GeekDad team up for the Ultimate
> GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
> lucky parental unit. See the prize list and enter to win:
> http://p.sf.net/sfu/thinkgeek-promo
> _______________________________________________
> Semediawiki-devel mailing list
> Semediawiki-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
>

------------------------------------------------------------------------------
ThinkGeek and WIRED's GeekDad team up for the Ultimate
GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
lucky parental unit. See the prize list and enter to win:
http://p.sf.net/sfu/thinkgeek-promo
_______________________________________________
Semediawiki-devel mailing list
Semediawiki-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/semediawiki-devel




--
WikiWorks MediaWiki Consulting http://wikiworks.com