Hi.

I agree with Yaron. There is a great community built around SMW and there is always a number of people that heavily use a particular extension. I always run through Semantic Forms and Semantic Forms Inputs when a new version is imminent. Contributing your time to testing is just as important as the code writing in my opinion.

It would be great if the extension developers could give a list of what fixes their upcoming release addresses. The person who reported the bug could then test before the new version is released. Same goes for new functionality. It would be great if new stuff could be tested before rather than after general release. From experience I know developers hate writing documentation, but as I've demonstrated to them over the years it saves time and stress in the long run :)

The part you have to be rigorous with is the final code freeze. It is always tempting to put a few last minutes fixes into a release. The trouble is this can sometimes cause more problems than it's worth, especially without some form of automated regression testing. Best to leave it until the next release. People who really really need the fix can always use the SVN version.

A "Dummies Guide" on the wiki about usage and gotchas of SVN might be useful. I know a lot of people are wary of using it or simply don't know how to. Whilst they might not be technical people, as day to day users they will still make excellent testers!

Finally, the use of Bugzilla should be encouraged more. It's is not really being utilised properly at the moment and could be used much better to disseminate progress and release plan information. Perhaps people find it too fiddly to use as it is currently configured?
 
Cheers
Neill.

On 10/06/10 01:10, Yaron Koren wrote:
More built-in PHPUnit and Selenium testing would be nice; I fear that it'll be hard to find someone willing to set these up, unless maybe there's an opportunity for paid development; but I'd love to be proven wrong.

I think there's a simpler, low-tech tactic, though, that will help a lot in preventing major bugs from getting into releases: have an explicit alpha/testing period between when code changes are done and when the version is released, and ask people to upgrade to the latest SVN version during that period to help with testing.

As a case in point, one of the errors in version 1.5.1 was that the class name was changed from "OWLExport" to "SMWOWLExport", but not everywhere. This change happened on May 30, and version 1.5.1 was released on May 31. I then discovered and fixed the bug on three days later, on June 3:

http://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/SemanticMediaWiki/includes/export/SMW_OWLExport.php?view=log

...even if I hadn't found it, I'm sure someone else would have found it before too long.

Markus did send out a "please test the SVN code" email, but that was also on May 30. A period of, say, two weeks would I think be a better approach.

I should note that I used to do the same thing, or even worse - I would make a flurry of SVN changes right before releasing a new version of one of my extensions. That led to a constant cycle of each version trying to fix the bugs discovered in the version before it, and introducing new bugs along the way. Now I try to check in each code change or patch as soon as I have it, to give more time for testing it before a new version was released; I think it's resulted in better code.

There's no substitute for user testing, especially given the wide variety of uses that SMW gets, and the different MediaWiki setups, browsers, database systems, etc. that it's run on - unit testing just can't capture all of it. I think a substantial "pencils down" period in which users are asked to test everything out would help a lot.

-Yaron


On Wed, Jun 9, 2010 at 1:50 PM, Lane, Ryan <Ryan.Lane@ocean.navo.navy.mil> wrote:
MediaWiki is including support for PHPUnit and Selenium in core. It may be
good to follow the efforts there. You can use me as a Selenium POC for now.
If anyone would like to be added to the Selenium planning contacts, let me
know and I'll introduce you to the other devs working on it. We are doing
all planning on wikitech-l, but it is good to know who the others actively
working on it are.

Respectfully,

Ryan Lane

> -----Original Message-----
> From: Stephan Robotta [mailto:robotta@ontoprise.de]
> Sent: Wednesday, June 09, 2010 6:23 AM
> To: semediawiki-devel@lists.sourceforge.net
> Subject: Re: [SMW-devel] Code quality, and how to improve it
>
> Hello folks,
>
> Jeroen stated a very important point here. In the Halo project (which
> includes various extensions) we suffer from the same problem that it
> costs us to keep the quality of the software.
> To have a greater output from our efforts we have created a build
> environent and run integration tests there.
>
> At one hand we use PHPUnit [1] to test single functions or portions of
> code in an extension. Depending on the specification the tests should
> contain all kinds of input that a function allows and validate the
> result.
>
> In the next level we use Canoo Webtest [2] to test a whole
> installation. Webtests try to simulate a browser and click through the
> application as defined by the testcase. After each test step the html
> of the resulting page can be validated against xpaths to
> check that certain
> elements exist on a page. This is a quite powerful software
> and it gets
> improved continuously by a vivid community. What webtest cannot yet
> serve sufficiently is to control GUI elements. Also javascripts are
> tested only within the Webtest but not like in unit test as it's
> possible for the PHP code.
>
> We have set up tests to test each extension separatly. Test
> are written
> as we add new features to the extension. Initially we also added
> testcases for existing functionality. What still costs us is to keep
> up the tests to a current state and enhance them. For every bug fix a
> new testcase should be added that tests exactly the issue
> where the bug
> arose.
>
> We also plan to have another test scenario that tests a installation
> with various extensions enabled. Often an extension itself works fine
> but in conjunction with others something may break.
>
> With these messurements we try to improve the quality of the software
> and we try to be more prepared to add new features to the
> existing code
> base. The whole testing process needs some effort and
> dicipline so that
> it is acutally useful for the entire project. Test results
> are reported
> within our build system so that after each build we have feedback of
> the quality of our work.
>
> Best regards, Stephan
>
> [1] http://www.phpunit.de/
> [2] http://webtest.canoo.com/webtest/manual/WebTestHome.html
>
> --------------------------------------------------------------
> ----------------
> ThinkGeek and WIRED's GeekDad team up for the Ultimate
> GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
> lucky parental unit.  See the prize list and enter to win:
> http://p.sf.net/sfu/thinkgeek-promo
> _______________________________________________
> Semediawiki-devel mailing list
> Semediawiki-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
>

------------------------------------------------------------------------------
ThinkGeek and WIRED's GeekDad team up for the Ultimate
GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
lucky parental unit.  See the prize list and enter to win:
http://p.sf.net/sfu/thinkgeek-promo
_______________________________________________
Semediawiki-devel mailing list
Semediawiki-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/semediawiki-devel




--
WikiWorks · MediaWiki Consulting · http://wikiworks.com
------------------------------------------------------------------------------ ThinkGeek and WIRED's GeekDad team up for the Ultimate GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the lucky parental unit. See the prize list and enter to win: http://p.sf.net/sfu/thinkgeek-promo
_______________________________________________ Semediawiki-devel mailing list Semediawiki-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/semediawiki-devel