Menu
▾
▴
Re: [Rails-devel] Automated testing
|
From: Stefan F. <ste...@we...> - 2011-08-02 05:15:18
|
Erik: the myth that "small report format changes render test cases invalid" is not getting true by repetition. It only shows that either I was not able to explain the workflow of the tests better or that you are not trying to get a better understanding. Maybe both. See a better explanation below. I will add this to the wiki too. In general everyone can decide to use the tests or to simply ignore them. I work with them all the time I simply reset the tests if you have broken them. However it always worries me if I realize that tests fail after a code change of yours as it is very tedious for me to find out if this was an intentional change of the report or a bug (or maybe both). And more important it is not my task to quality check your changes. Thus if you prefer not to use them it does not effect me going on using the tests for my own coding. However I would love to increase the quality of the whole package Rails by convincing that it is better to test instead of fixing bugs after the community reports about broken games. Sorry for insisting on that but I consider this as a major time saver for all of us, Stefan Automated Game Tests: *** What the test does: * The test compares a stored game report (which is considered valid) with a game report created with current compiled version of Rails. Any difference is marked as a failed test. * If the game throws a Java error, this test is marked as an error. * As the game report contains all items you refer to below, the test includes all required functionality you define below. * There is no such thing as "the test case is invalid". See below for a workflow and further remarks. Either you did a valid change to the game report (it was your intention to change it) then you reset the base game report (claim the new report as valid), or there has been a code change that broke any of the test games. *** Based on this the workflow for using the test is: => You have a code change that did not change the game report of the test games: * Run the tests. (In package test look for TestGameBuilder and RunAs => JUnit Test). A graphical UI shows which test games pass, fail or raise an error. * If all tests pass, you are happy. * If a test reports an error, load that game and check what kind of error is reported. Fix code and rerun tests. * If a test reports a failed, click on the first row in FailureTrace and the first line where the current report and the expected report differ. Either think about your premise (no change of game report) or fix code and rerun tests. => You have a code change that did change the game report of the test games: * Run tests, all games that are effected by your change of the game report will fail. Potentially this can be all tests. * Reset the tests. (In package test look for TestGameBuilder and RunAs => Application). The UI allows you to select either single games or whole folders to reset the tests to the now current code base. * If you now run the tests again, all tests will pass. *** Further Remarks: - Running the tests is a matter of seconds. Thus one is able to test often. This makes identifying the reason of tests failing easier. - Due to this workflow try to isolate report changes from other code changes. - If test games cannot replayed at a specific point, because the action is considered invalid by Rails, this shows as failed. However if it makes sense, the games can be truncated and the report reset. - The game report of the test uses a special language setting (TE_ST), which ignores the entries in LocalisedText (have a look at one of the reports in test/data). Thus changing the entries in LocalisedText only will not have tests fail. On Monday, August 01, 2011 09:46:16 pm Erik Vos wrote: > It's a unfortunate that even small report format changes render test cases > invalid, and it makes this way of testing much less valuable for me. > I understand that this is the only simple way, but perhaps we should start > thinking about a somewhat more intelligent test evaluation method. > > When I do my own regression testing with saved games, the main thing I'm > looking for is that the game runs to the end without throwing exceptions, > and that the end result looks reasonable. > An automated test would of course need to be a bit more thorough: > - are the player and companies turn sequences identical, > - are the cash amounts the same, > And perhaps a few more. > > Only checking the status overview at the end of each round would already do > a lot. Now that overview only reports cash, and I'm wondering if it would > help to add other possessions, > to create a complete written status overview (like the zines publish about > the games they run), and check only that? > > However, that is for the future. For now, I can't guarantee anything. I'm > myself occasionally wrestling with saved games that no longer load as a > result of some improvement. > For instance, removing redundant actions (where only 'Done' is possible) > has had that effect in some cases. I still occasionally have to remove > 'Done' actions from saved games to make these loadable again. Fixing bugs > has equally well invalidated test cases in the past. So did > implementation of the dynamic company operating order. > > I can't tell if any such thing would apply to the current test set, which I > haven't studied. The more recent the file is, the larger the chance that it > is OK. > > By private mail I'm sending you some files of the kind that I regularly use > to check if things still work, and that cover some interesting aspects, in > particular in relation to train trading. > But in general I think recently completed games are more valuable (trusting > that the players haven't missed any errors!). > > Erik. > > > -----Original Message----- > > From: Stefan Frey [mailto:ste...@we...] > > Sent: Monday, August 01, 2011 7:27 AM > > To: 'Development list for Rails: an 18xx game' > > Subject: [Rails-devel] Automated testing > > > > I have added the games that I have received to automated game testing. > > Thanks to all contributing save files. > > There is also a table of those in the wiki (under developer information). > > > > In the next few days (as time permits) I will add the existing > > information > > on > > > game testing (from older mails) to the wiki and I hope it will increase > > the > > > usage of those tests. > > > > Currently all tests fail as Erik has changed many things that impacted > > the Game Report (phase names and train obsolete messages). > > > > Question to Erik: If you think all changes are merely phase name changes > > and > > > adding/removing additional messages and you are sure that everything > > works ok, give me a nod and then the game reports can be updated to the > > current states and all tests will show green again. > > > > Unfortunately my master still automatically merges on pull, I will setup > > the > > > master fresh as soon as I have pushed all my current local branches. > > > > Stefan > > --------------------------------------------------------------------------- > - -- > > > Got Input? Slashdot Needs You. > > Take our quick survey online. Come on, we don't ask for help often. > > Plus, you'll get a chance to win $100 to spend on ThinkGeek. > > http://p.sf.net/sfu/slashdot-survey > > _______________________________________________ > > Rails-devel mailing list > > Rai...@li... > > https://lists.sourceforge.net/lists/listinfo/rails-devel > > --------------------------------------------------------------------------- > --- BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA > The must-attend event for mobile developers. Connect with experts. > Get tools for creating Super Apps. See the latest technologies. > Sessions, hands-on labs, demos & much more. Register early & save! > http://p.sf.net/sfu/rim-blackberry-1 > _______________________________________________ > Rails-devel mailing list > Rai...@li... > https://lists.sourceforge.net/lists/listinfo/rails-devel |
