Hi,

Okay, I guess I extrapolated too much from the example I gave - it was the first example I could think of, and it turned out to be an unusual case.

I still think a pre-release testing period could help a lot, especially if there's a strong request made for people to switch to that version - in many cases, people could uncover bugs without even doing real testing, just by trying to run the same functionality as before using the new code.

-Yaron

On Thu, Jun 10, 2010 at 4:44 AM, Markus Krötzsch <markus@semantic-mediawiki.org> wrote:
Now an email on the SMW 1.5.1 release schedule.

In general, I completely agree with you, Yaron. The special situation for SMW
1.5.1 was that there was some pressure to release swiftly so as to coordinate
with the new Semantic Maps which has some interdependencies with SMW 1.5.1.
The release of SM was hard to defer since Jeroen needs to start on his GSoC
project soon.

Yet, we were already quite ready on Friday and did only minor updates over the
weekend. It was certainly not planned to have any major changes on May 30, and
the renaming accident was really bad luck. When discussing one such typo, it
should be kept in mind how many bugs we successfully fixed before the release.
SMW 1.5.1 includes a large number of very minor syntactic changes, e.g. since
all code has been aligned with the MW coding conventions now. Such style
changes are more error-prone than one would expect. So even though SMW 1.5.1
is a minor update in terms of functionality, there were plenty of
opportunities for typos. We were quite successful in fixing most of these
before the release.

We can try to have more of a testing phase for future releases, but my
experience in the past was that real user testing only starts after the
release anyway. The problem with LIKE search is an example of a case where a
bug went unnoticed for quite some time. Especially Selenium-style testing
could be very valuable to have a way of detecting these. Another option could
be official pre-releases for testers.

Regards,

Markus

P.S. One thing to keep in mind when implementing testing phases is to fork the
SVN directory when starting the test. We have had some problems with SVN-wide
automatic commits that simply removed MW 1.15 compatibility from all
extensions shortly before the planned release, and that required us to roll-
back many files to restore this compatibility for SMW 1.5.1.


On Donnerstag, 10. Juni 2010, Yaron Koren wrote:
<snip>
>
> I think there's a simpler, low-tech tactic, though, that will help a lot in
> preventing major bugs from getting into releases: have an explicit
> alpha/testing period between when code changes are done and when the
>  version is released, and ask people to upgrade to the latest SVN version
>  during that period to help with testing.
>
> As a case in point, one of the errors in version 1.5.1 was that the class
> name was changed from "OWLExport" to "SMWOWLExport", but not everywhere.
> This change happened on May 30, and version 1.5.1 was released on May 31. I
> then discovered and fixed the bug on three days later, on June 3:
>
> http://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/SemanticMediaWik
> i/includes/export/SMW_OWLExport.php?view=log
>
> <http://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/SemanticMediaWi
> ki/includes/export/SMW_OWLExport.php?view=log>...even if I hadn't found it,
>  I'm sure someone else would have found it before too long.
>
> Markus did send out a "please test the SVN code" email, but that was also
>  on May 30. A period of, say, two weeks would I think be a better approach.
>
> I should note that I used to do the same thing, or even worse - I would
>  make a flurry of SVN changes right before releasing a new version of one
>  of my extensions. That led to a constant cycle of each version trying to
>  fix the bugs discovered in the version before it, and introducing new bugs
>  along the way. Now I try to check in each code change or patch as soon as
>  I have it, to give more time for testing it before a new version was
>  released; I think it's resulted in better code.
>
> There's no substitute for user testing, especially given the wide variety
>  of uses that SMW gets, and the different MediaWiki setups, browsers,
>  database systems, etc. that it's run on - unit testing just can't capture
>  all of it. I think a substantial "pencils down" period in which users are
>  asked to test everything out would help a lot.
>
> -Yaron
>
> On Wed, Jun 9, 2010 at 1:50 PM, Lane, Ryan
<Ryan.Lane@ocean.navo.navy.mil>wrote:
> > MediaWiki is including support for PHPUnit and Selenium in core. It may
> > be good to follow the efforts there. You can use me as a Selenium POC for
> > now. If anyone would like to be added to the Selenium planning contacts,
> > let me know and I'll introduce you to the other devs working on it. We
> > are doing all planning on wikitech-l, but it is good to know who the
> > others actively working on it are.
> >
> > Respectfully,
> >
> > Ryan Lane
> >
> > > -----Original Message-----
> > > From: Stephan Robotta [mailto:robotta@ontoprise.de]
> > > Sent: Wednesday, June 09, 2010 6:23 AM
> > > To: semediawiki-devel@lists.sourceforge.net
> > > Subject: Re: [SMW-devel] Code quality, and how to improve it
> > >
> > > Hello folks,
> > >
> > > Jeroen stated a very important point here. In the Halo project (which
> > > includes various extensions) we suffer from the same problem that it
> > > costs us to keep the quality of the software.
> > > To have a greater output from our efforts we have created a build
> > > environent and run integration tests there.
> > >
> > > At one hand we use PHPUnit [1] to test single functions or portions of
> > > code in an extension. Depending on the specification the tests should
> > > contain all kinds of input that a function allows and validate the
> > > result.
> > >
> > > In the next level we use Canoo Webtest [2] to test a whole
> > > installation. Webtests try to simulate a browser and click through the
> > > application as defined by the testcase. After each test step the html
> > > of the resulting page can be validated against xpaths to
> > > check that certain
> > > elements exist on a page. This is a quite powerful software
> > > and it gets
> > > improved continuously by a vivid community. What webtest cannot yet
> > > serve sufficiently is to control GUI elements. Also javascripts are
> > > tested only within the Webtest but not like in unit test as it's
> > > possible for the PHP code.
> > >
> > > We have set up tests to test each extension separatly. Test
> > > are written
> > > as we add new features to the extension. Initially we also added
> > > testcases for existing functionality. What still costs us is to keep
> > > up the tests to a current state and enhance them. For every bug fix a
> > > new testcase should be added that tests exactly the issue
> > > where the bug
> > > arose.
> > >
> > > We also plan to have another test scenario that tests a installation
> > > with various extensions enabled. Often an extension itself works fine
> > > but in conjunction with others something may break.
> > >
> > > With these messurements we try to improve the quality of the software
> > > and we try to be more prepared to add new features to the
> > > existing code
> > > base. The whole testing process needs some effort and
> > > dicipline so that
> > > it is acutally useful for the entire project. Test results
> > > are reported
> > > within our build system so that after each build we have feedback of
> > > the quality of our work.
> > >
> > > Best regards, Stephan
> > >
> > > [1] http://www.phpunit.de/
> > > [2] http://webtest.canoo.com/webtest/manual/WebTestHome.html
> > >
> > > --------------------------------------------------------------
> > > ----------------
> > > ThinkGeek and WIRED's GeekDad team up for the Ultimate
> > > GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
> > > lucky parental unit.  See the prize list and enter to win:
> > > http://p.sf.net/sfu/thinkgeek-promo
> > > _______________________________________________
> > > Semediawiki-devel mailing list
> > > Semediawiki-devel@lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
> >
> > -------------------------------------------------------------------------
> >----- ThinkGeek and WIRED's GeekDad team up for the Ultimate
> > GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
> > lucky parental unit.  See the prize list and enter to win:
> > http://p.sf.net/sfu/thinkgeek-promo
> > _______________________________________________
> > Semediawiki-devel mailing list
> > Semediawiki-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
>


--
Markus Krötzsch  <markus@semantic-mediawiki.org>
* Personal page: http://korrekt.org
* Semantic MediaWiki: http://semantic-mediawiki.org
* Semantic Web textbook: http://semantic-web-book.org
--



--
WikiWorks · MediaWiki Consulting · http://wikiworks.com