I think Dimitry makes some valid points. I think most users understand that the beta version is likely to have bugs, and if you tell them about the bugs you're aware of, they'll be reasonable, especially if the program is, by and large, stable. I also believe that any user community is sensitive to perceived responsiveness on the part of the developers. They are more willing to put up with problems if they see that those problems are addressed fairly quickly.

As Dimitry points out, a lot of the bugs found by Beta users are difficult to nail down. To me, that suggests that we may need improved logging on Beta versions. We can, of course, disable the logging later in the production release, and we'll need to take precautions to make sure we don't fill up someone's disk with log files, but if we do find that some bugs are nearly impossible to find, adding additional logging and asking the Beta community to help is a valid approach.

I have found a number of really nice tools that we can use to help our testing. I'm trying some of them out now, when I have time, trying to see how best to integrate them into our development process. One of the things I'd like to try to do, is write a new test case to match each bug that is reported. That will help whoever is working on it fix it. (Obviously, this won't be necessary for really minor bugs.)

I've also learned that Eclipse can integrate quite closely with Bugzilla, making it quite easy to track bugs, who is working on them, and the progress made towards fixing them. If we settle on Eclipse as our standard development platform, we could set up the Bugzilla integration. This would make it easier for bug reporters to find out what's going on with the bug they submitted, and much of the work would be automated.

A while back, I submitted a bug to Open Office. It was a rather obscure bug that occurred when working with large master documents. (My novel is organized as a master document with each chapter in a separate file.) I didn't expect very rapid response. All my previous bug reporting experience had been with proprietary software companies. I was extremely pleased that I got regular emails (automated) alerting me to the progress made on my bug and letting me know what release my bug was scheduled to be fixed in. Two releases later, about 3 months, the bug was fixed. It impressed me.

By using the right tools and setting up the right processes, we can do much of what Dimitry is suggesting without making someone use up all their free time on it. I'll be happy to help.


On a related topic. We are badly in need of updated documentation. Do we have any technical writers, or is it just up to us? If it's up to us, I can put some time in on that as well.


Dimitry Polivaev wrote:

the most important task of today is the coming release. We are 
interested in getting it stable work ASAP. That's why I am starting with 
the discussion about how to let the community participate on the testing.

(NB: Certainly there are two sides: the FreeMind team and the others
called community. Because the discussed topics concerns the both of
them, I write into the closed freemind-developer list for the internal
team discussion and to the open discussion forum to let the community to
participate in the discussion too.)

I start with analysis of the current project practices.

Currently the Beta versions are released rather seldom: the Beta 9 is
released on the 2007-02-21, the Beta 12 is released exactly 5 months
later on the 2007-07-21. (The Beta 10 and Beta 11 are also released in
Juli, but they have been hidden because of some bugs).

Each new Beta version encourage the beta users to submit the bug reports 
so that the remaining bugs can be removed. In about two weeks after the 
beta release all most important bugs are found, and the users want to 
get them away. But after fixing of the annoying submitted bugs (which 
takes in the most cases about a week) the freemind team does not 
immediately produce the next bug fix release. Instead it works on 
further testing and improvements not requested by the users.

The users get disappointed and send mails like

I am also wondering when beta 13 will be published. I still see beta
12 in the download list, and the images are still broken, and editing
a node(not long), still always has the text and cursor as black, even
if I set my selected node colors otherwise.

Such mails are not always answered but often just ignored. It probably 
reduce both the will to submit bug reports and the consequently the 
chances that the bugs are found and reported by the users. As a 
consequence the team has to perform more tests by itself which course an 
additional delay of the next beta release.

Another point: developers are generally less effective in performing the 
tests than the users, because the most bugs require some special 
conditions to become visible. Different users handling  the program in 
different ways are more likely to find a bug than a single developer 
always trying more ore less the same things. The bugs happen mostly 
because of some wrong dependencies between different program parts. The 
program seems to work fine, and one need a coincidence to see the bug. 
That's why many beta tester are more likely to find the bug than one 

Now I would like to describe what could be done another way.

Instead of full concentration on the testing the team (or at least one 
developer) could concentrate on producing new beta releases and on the 
dialog with the community giving them response and paying the tribute to 
the people submitting the bug reports.

The more people get involved in the testing and above all in submitting 
the bug reports, the more bugs can be recognized and fixed.

It requires that both the team and the community accept that the bugs 
are still not unlikely to be present. And I have never had an impression 
that the community is not satisfied with the quality of the current beta 
versions. Because the newer versions fixing the old bugs are better than 
the older once, we produce a positive feedback loop: if the bugs get 
fixed, submitting of the new bug reports seems to make more sense.

Further I think that releasing of the next beta does not require all of 
the known bugs to be fixed. The remaining bugs could be just listed in 
the version description, because it is better to work with a version 
where 70% of the reported bugs have been fixed than to wait for a 
version without bugs for many months.

Let me conclude.

I think that more frequently producing of the beta versions and staying 
in the dialog with the community is a better way for achieving both 
higher user satisfaction and less error rate in the release. The Beta 
versions could be produced every month. I would like to do the job, but 
first of all the proposed ideas should be discussed within the team and 
within the community.

I am looking forward to your responses.

Best regards, Dimitry

This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
Freemind-developer mailing list