statcvs-users

[Statcvs-users] Release early and often?

[Richard C. <ri...@cy...>]

Hi all,

Brian's request for a dev snapshot made me think about the StatCVS 
release process.

I tend to release late. New features and bugfixes can sit in the CVS 
for many month before I put together a new release. That's because I 
often feel that the new features need more testing and are not yet 
stable enough. I don't want to release a new version with significant 
new bugs.

But recently I think that this is the wrong strategy. Common wisdom in 
the open source community is to "release early, release often". And 
when people rely on inofficial dev builds because important new 
features and bugfixes are not available in the official release, then 
the process is obviously broken.

So what can I do to improve the release process?

Publish release candidates?

Offer a stable and a beta download?

Just stop worrying about bugs and release without much testing?

Other ideas?

What do you think? How should an open source release process for a 
project like StatCVS work? Both developer's and user's point of views 
are welcome.

Thanks,
Richard

Re: [Statcvs-users] Release early and often?

[Martin d'A. <md...@ne...>]

> What do you think? How should an open source release process for a 
> project like StatCVS work? Both developer's and user's point of views 
> are welcome.

I am really interested in people's opinions as well. Note that I have not
used StatCVS in years. So forgive me for stating the obvious here.

Software evolves by making bug fixes or adding features (by adding 
features I mean the general sense including code rewrites that does not 
necessarily add a feature visible to the user - like restructuring code
for future developments). So I am going to consider two cases: bug and 
features in the release process. Then there are the tests for the bug or 
the specific feature and the existing test suite, and then whether a 
stable release should be made or not depending on the result of the test

Case   Test for bug/feature Existing test  Stable release?
       has passed           suite passed
bug    yes                  yes            yes
bug    yes                  no             no
bug    no                   yes            only if there are new features
                                           in the release and the tests 
                                           for those have passed
feat.  yes                  yes            yes
feat.  yes                  no             no
feat.  no                   yes            only if there are bug fixes
                                           in the release and the tests
                                           have passed

In other words, a good the test suite speeds up stable releases. If you 
don't have a good test suite, then you have to rely on users' generosity.

Martin

PS: Here is another view of the same table for those with whitespace 
impaired email clients:

<table>
<th>
<td>Case</td>
<td>Test for bug/feature has passed</td>
<td>Existing test suite has passed</td>
<td>Do a stable release?</td>
</th>
<tr>
<td>bug</td>
<td>yes</td>
<td>yes</td>
<td>yes</td>
</tr>
<tr>
<td>bug</td>
<td>yes</td>
<td>no</td>
<td>no</td>
</tr>
<tr>
<td>bug</td>
<td>no</td>
<td>yes</td>
<td>only if there are new features in the release and the tests for those have passed</td>
</tr>

<tr>
<td>feature</td>
<td>yes</td>
<td>yes</td>
<td>yes</td>
</tr>
<tr>
<td>feature</td>
<td>yes</td>
<td>no</td>
<td>no</td>
</tr>
<tr>
<td>feature</td>
<td>no</td>
<td>yes</td>
<td>only if there are bug fixes in the release and the tests for those have passed</td>
</tr>

</table>

Re: [Statcvs-users] Release early and often?

[Richard C. <ri...@cy...>]

Hi Martin,

Am 23.03.2005 um 23:42 schrieb Martin d'Anjou:

[...]
> In other words, a good the test suite speeds up stable releases. If you
> don't have a good test suite, then you have to rely on users' 
> generosity.

That's a good point. With a good test suite, the code likely has less 
bugs and no regressions. This means you can release with greater 
confidence.

I wouldn't go as far as saying that you safely can do a stable release 
if all tests pass. A green bar is no guarantee that your software is 
bug-free and fulfills user's requirements.

The stuff that you can't catch with a test suite (e.g. problems with 
certain system configurations) is unlikely to be found without a bit of 
circulation. I guess that's the rationale for doing beta builds or 
release candidates.

Richard

Re: [Statcvs-users] Release early and often?

[Jukka U. <juk...@dn...>]

Richard Cyganiak wrote:
> 
> I wouldn't go as far as saying that you safely can do a stable release 
> if all tests pass. A green bar is no guarantee that your software is 
> bug-free and fulfills user's requirements.
> 

Agree. Even good junit test suites with green bar has the great effort 
for testing and bug-free programs, i like to see good test
suite is good collection junit test cases and collection just written
test cases derived from use cases or user stories.

If these all tests pass, i think stable release or least release 
candidate is quite near.

- Jukka -

Re: [Statcvs-users] Release early and often?

[Jukka U. <juk...@dn...>]

Richard Cyganiak kirjoitti:
> 
> What do you think? How should an open source release process for a 
> project like StatCVS work? Both developer's and user's point of views 
> are welcome.
> 

Hi,

This is pretty intresting question. I think it is good idea to make time 
between releases so short as possible, but not shorter than possible. I
mean if you have certain process, it will take certain time and if you 
try to make it faster, you don't follow your process. Only way is 
improve your process. And now we discussing how to improve statcvs 
release process.

How about process as follows:

1. Decide what new features are in next release

2. Provide nightly-build. Nightly-build is released from cvs head branch 
if there are any changes in cvs, code gets compiled and passes unit tests.

3. When all features in next release are implemented, provide 
release-candidate. This point there should be branch in cvs for release-
candidate bugfixes. Propably you will need couple of release-candidates.

4. When you think release-candidate is stable enough, just release 
official release or maybe uses take a vote, but propably everybody will 
use some release candidate as production at this point.

Nighly-builds requires automated build system and good enough unit test
cases. Do you have checked coverage of unit tests?

Like Martin says good test suite is important thing, but i assume that
application like statcvs, good unit test suite has most effort for 
testing. There are no user interactions etc, which are hard to test with
junit.

In open source projects, tight schedules aren't critical issue. People 
are voluntarily working and everybody has possibility to contribute to 
get releases ready earlier.

- Jukka -

Re: [Statcvs-users] Release early and often?

[Richard C. <ri...@cy...>]

Hi Jukka,

Thanks for the thoughtful response! Responses/questions inline.

Am 24.03.2005 um 21:40 schrieb Jukka Uusisalo:
[...]
> How about process as follows:
>
> 1. Decide what new features are in next release
>
> 2. Provide nightly-build. Nightly-build is released from cvs head 
> branch if there are any changes in cvs, code gets compiled and passes 
> unit tests.
>
> 3. When all features in next release are implemented, provide 
> release-candidate. This point there should be branch in cvs for 
> release-
> candidate bugfixes. Propably you will need couple of 
> release-candidates.
>
> 4. When you think release-candidate is stable enough, just release 
> official release or maybe uses take a vote, but propably everybody 
> will use some release candidate as production at this point.

I think I'd rather skip #1. StatCVS contributions (including my own) 
come from people scratching an itch, and it's kind of hard to plan 
that. So I'd rather just wait until there's enough new stuff.

Having a release candidate branch seems like a good idea. This means 
work on new stuff can continue while the candidate matures.

Any guesses how long we might have to stay in RC mode before the 
candidate is well-tested enough?

Also, is it worth the trouble for such a small project? I don't want to 
spend much time managing the process.


> Nighly-builds requires automated build system and good enough unit test
> cases. Do you have checked coverage of unit tests?

I like the idea of nightly builds. Get the new stuff out as fast as 
possible.

Our test coverage is not too good, especially in the HTML output code. 
But I wouldn't worry about that too much, the most important parts are 
well-tested. I believe that nightly builds would drive test quality 
because we would actually rely on the tests.

Any recommendations for a test coverage tool? (I guess the answer will 
have something to do with Maven :-)

How could we implement nightly builds with SourceForge? Just run Ant or 
Maven with a cronjob and dump the dist file on the SourceForge project 
web server? Anyone has experience with that?

Questions, questions, questions ...


> - Jukka -

Richard

Re: [Statcvs-users] Release early and often?

[Jukka U. <juk...@dn...>]

Richard Cyganiak wrote:
> Hi Jukka,
> 
> Thanks for the thoughtful response! Responses/questions inline.
> 

Hi,

some comments below ...


> Am 24.03.2005 um 21:40 schrieb Jukka Uusisalo:
> [...]
> 
>> How about process as follows:
>>
>> 1. Decide what new features are in next release
>>
>> 2. Provide nightly-build. Nightly-build is released from cvs head 
>> branch if there are any changes in cvs, code gets compiled and passes 
>> unit tests.
>>
>> 3. When all features in next release are implemented, provide 
>> release-candidate. This point there should be branch in cvs for release-
>> candidate bugfixes. Propably you will need couple of release-candidates.
>>
>> 4. When you think release-candidate is stable enough, just release 
>> official release or maybe uses take a vote, but propably everybody 
>> will use some release candidate as production at this point.
> 
> 
> I think I'd rather skip #1. StatCVS contributions (including my own) 
> come from people scratching an itch, and it's kind of hard to plan that. 
> So I'd rather just wait until there's enough new stuff.
>

But if we talking about short and small releases, why we have to wait?
If there are some new stuff, you can make release plan like at this
moment release plan would be just, next release supports tags.

Release plan does not have to be detailed list of all coming changes or 
bugfixes, just major issues of next release.


> Having a release candidate branch seems like a good idea. This means 
> work on new stuff can continue while the candidate matures.
> 
> Any guesses how long we might have to stay in RC mode before the 
> candidate is well-tested enough?
> 
> Also, is it worth the trouble for such a small project? I don't want to 
> spend much time managing the process.
> 

If you look statcvs cvs activity, release candidates may be wortless.
If there are nightly-build available and new commits haven't done to
cvs and no new bugs are not reported, what we need more for "official" 
release. ;)

> 
>> Nighly-builds requires automated build system and good enough unit test
>> cases. Do you have checked coverage of unit tests?
> 
> 
> I like the idea of nightly builds. Get the new stuff out as fast as 
> possible.
> 
> Our test coverage is not too good, especially in the HTML output code. 
> But I wouldn't worry about that too much, the most important parts are 
> well-tested. I believe that nightly builds would drive test quality 
> because we would actually rely on the tests.
> 
> Any recommendations for a test coverage tool? (I guess the answer will 
> have something to do with Maven :-)
> 

I just played little bit with Clover. They will give free licenses to
open source projects. But i just played, nothing any serious. First 
impressions are pretty good.


- Jukka -

Re: [Statcvs-users] Release early and often?

[Brian G. P. <br...@br...>]

I am an active developer/manager in many open source projects, and have used 
StatCVS for several years/versions.  So, my comments come from this 
perspective.

On Wednesday 23 March 2005 03:54 pm, Richard Cyganiak wrote:
> So what can I do to improve the release process?

I'd Start with nightly builds.  I find that these often allow people who can't 
or won't build the software themselves to test new features as they make it 
into CVS.

If you have JUnit tests for StatCVS, place a link to the test results on the 
same page as the nightly build tarball.  This communicates to developers and 
users what the potential state of the code is without requiring constant 
effort on your part.

> Publish release candidates?
> Offer a stable and a beta download?

My usual approach is to publish a release candidate when the development team 
thinks the code is ready for release, and usually allow a few days for 
testing.  People are generally quite responsive to calls for testing a 
release candidate.  I don't maintain a separate branch in CVS for the release 
candidate, as the entire process usually only takes a few days.

This is a little different on a large project like Squirrelmail.  The 
Squirrelmail development is *always* split into Stable and Development 
branches, and changes are merged back to stable form development.  StatCVS 
does not seem to be a large enough project to require the level of effort 
that goes with maintaining two separate CVS branches.

> Just stop worrying about bugs and release without much testing?

Nightly builds with a test process will find a lot of bugs (though not all of 
them!)

> What do you think? How should an open source release process for a
> project like StatCVS work? Both developer's and user's point of views
> are welcome.

Thanks for the great software!  I rely on it to keep an eye on what the 
development team is doing across multiple projects.  StatCVS has been a great 
help to me, keep it up!

Regards,

   - Brian