On Mon, Nov 9, 2009 at 1:46 PM, John Reese <jreese@leetcode.net> wrote:
Robert Munteanu wrote:
> Another way is to forget unit tests and focus on end-to-end tests,
> using something like Selenium RC (
> http://seleniumhq.org/projects/remote-control/ ) and run the whole
> browser interface. These , like the SOAP API, bring in no dependencies
> - just clean up after yourself - and require no other changes, but on
> the other hand are more brittle, slower, and need more care when
> making changes - adding a row can mess up the fixture.

Selenium has been mentioned before, and in principle I really like the idea of
getting that set up.  In practice though, none of the core (that I know of) have
any experience with Selenium, and even less time to spend learning it and
setting it up.

I welcome anyone familiar with it to set up and contribute a test suite,
documentation on how to set it up and run it for Mantis, as well as any
necessary bits to automate the process.

I think there are benefits in both options.  However, if I'm to choose one, it will for sure be the unit test approach.  That is exactly why I developed the MantisEnum example.  I've also setup the soap test cases, since I found no other reliable way to check if the SOAP API is not broken and to verify fixes (in the past I used to use the MantisConnect .NET API unit tests to achieve the same, but I couldn't get others to run that and hence I decided to move the unit tests closer to the code).

I know that my suggestions below will probably be conceived as big changes and may indicate a re-write.  However, I personally don't believe in re-writes for open source projects, but I believe if we have a goal / direction, we can gradually move towards it.

I would love to see us refactor to achieve the following:

1. Refactor business logic with following characteristics:
a. Use classes rather than functions / use new coding conventions suggested by Gianluca from the Zend Framework..
b. Don't have view logic (i.e. formatting, UI messages, etc).
c. Throw exceptions rather than trigger errors.  This will be useful for unit tests as well as SOAP API and any future APIs.
d. Abstract access to configuration / database so that we can have a production / unit test implementation of such stores.

2. Change the core APIs to maintain the same interface / semantics on top of the classes developed in 1.  This is an interim solution until all page scripts are changed to use the classes directly and then the APIs can be retired.  We will probably also have to provide some grace period for such APIs before retiring it to allow for plugins to use the new classes directly.

3. Have a common pattern for data access layer.  I would rather if for these classes we use a common infrastructure / interface, rather than just 100% custom per entity (Paul is looking at ADODB replacement, we should have a strategy as part of this effort).

4. Utility methods like string_apis should be refactored into easily testable code like MantisEnum.

5. Refactor print_apis or any APIs that generate HTML into APIs that return a data structure / array / value and another one that renders that.  This moves us closer to templates and allows us to unit test APIs that prepare the data.  At one point, I started prepare_api to move to it the logic from print_api for preparing the data and also moved some of the logic to the entity specific APIs (e.g. users, bugs, files, etc).

6. Make sure we have unit tests and continuous integration going for all refactored code, documentation, etc.

7. For the viewing / templates logic, we can use plain PHP as the template engine or we can potentially use a framework.  However, I think the above refactoring is a stepping stone before taking dependency on a templating engine while our core code spits html.

This doesn't mean we shouldn't consider the GUI based end-to-end test and I welcome any contributions there, however, as a core team, I think we have the skills necessary to deliver on unit testing and we should do our best effort to gradually move in this direction.

-Victor