Menu
▾
▴
Re: [Rails-devel] Automated testing
|
From: Erik V. <eri...@xs...> - 2011-08-02 19:03:01
|
Stefan, Thanks for writing it all up again. I have printed it and will try to keep it handy and use it. Maybe one day it will settle into my brains... Erik. > -----Original Message----- > From: Stefan Frey [mailto:ste...@we...] > Sent: Tuesday, August 02, 2011 7:17 AM > To: Development list for Rails: an 18xx game > Subject: Re: [Rails-devel] Automated testing > > Erik: > the myth that "small report format changes render test cases invalid" is not > getting true by repetition. It only shows that either I was not able to explain > the workflow of the tests better or that you are not trying to get a better > understanding. Maybe both. > > See a better explanation below. I will add this to the wiki too. > > In general everyone can decide to use the tests or to simply ignore them. > I work with them all the time I simply reset the tests if you have broken > them. However it always worries me if I realize that tests fail after a code > change of yours as it is very tedious for me to find out if this was an > intentional change of the report or a bug (or maybe both). And more > important it is not my task to quality check your changes. > > Thus if you prefer not to use them it does not effect me going on using the > tests for my own coding. However I would love to increase the quality of the > whole package Rails by convincing that it is better to test instead of fixing > bugs after the community reports about broken games. > > Sorry for insisting on that but I consider this as a major time saver for all of us, > Stefan > > > Automated Game Tests: > > *** What the test does: > > * The test compares a stored game report (which is considered valid) with a > game report created with current compiled version of Rails. Any difference is > marked as a failed test. > > * If the game throws a Java error, this test is marked as an error. > > * As the game report contains all items you refer to below, the test includes > all required functionality you define below. > > * There is no such thing as "the test case is invalid". See below for a > workflow and further remarks. Either you did a valid change to the game > report > (it was your intention to change it) then you reset the base game report > (claim the new report as valid), or there has been a code change that broke > any of the test games. > > *** Based on this the workflow for using the test is: > > => You have a code change that did not change the game report of the test > games: > > * Run the tests. (In package test look for TestGameBuilder and RunAs => > JUnit > Test). A graphical UI shows which test games pass, fail or raise an error. > > * If all tests pass, you are happy. > > * If a test reports an error, load that game and check what kind of error is > reported. Fix code and rerun tests. > > * If a test reports a failed, click on the first row in FailureTrace and the > first line where the current report and the expected report differ. > Either think about your premise (no change of game report) or fix code and > rerun tests. > > => You have a code change that did change the game report of the test > games: > > * Run tests, all games that are effected by your change of the game report > will fail. Potentially this can be all tests. > > * Reset the tests. (In package test look for TestGameBuilder and RunAs => > Application). The UI allows you to select either single games or whole folders > to reset the tests to the now current code base. > > * If you now run the tests again, all tests will pass. > > *** Further Remarks: > - Running the tests is a matter of seconds. Thus one is able to test often. > This makes identifying the reason of tests failing easier. > > - Due to this workflow try to isolate report changes from other code changes. > > - If test games cannot replayed at a specific point, because the action is > considered invalid by Rails, this shows as failed. However if it makes sense, > the games can be truncated and the report reset. > > - The game report of the test uses a special language setting (TE_ST), which > ignores the entries in LocalisedText (have a look at one of the reports in > test/data). Thus changing the entries in LocalisedText only will not have > tests fail. > > > > > On Monday, August 01, 2011 09:46:16 pm Erik Vos wrote: > > It's a unfortunate that even small report format changes render test cases > > invalid, and it makes this way of testing much less valuable for me. > > I understand that this is the only simple way, but perhaps we should start > > thinking about a somewhat more intelligent test evaluation method. > > > > When I do my own regression testing with saved games, the main thing I'm > > looking for is that the game runs to the end without throwing exceptions, > > and that the end result looks reasonable. > > An automated test would of course need to be a bit more thorough: > > - are the player and companies turn sequences identical, > > - are the cash amounts the same, > > And perhaps a few more. > > > > Only checking the status overview at the end of each round would already > do > > a lot. Now that overview only reports cash, and I'm wondering if it would > > help to add other possessions, > > to create a complete written status overview (like the zines publish about > > the games they run), and check only that? > > > > However, that is for the future. For now, I can't guarantee anything. I'm > > myself occasionally wrestling with saved games that no longer load as a > > result of some improvement. > > For instance, removing redundant actions (where only 'Done' is possible) > > has had that effect in some cases. I still occasionally have to remove > > 'Done' actions from saved games to make these loadable again. Fixing bugs > > has equally well invalidated test cases in the past. So did > > implementation of the dynamic company operating order. > > > > I can't tell if any such thing would apply to the current test set, which I > > haven't studied. The more recent the file is, the larger the chance that it > > is OK. > > > > By private mail I'm sending you some files of the kind that I regularly use > > to check if things still work, and that cover some interesting aspects, in > > particular in relation to train trading. > > But in general I think recently completed games are more valuable (trusting > > that the players haven't missed any errors!). > > > > Erik. > > > > > -----Original Message----- > > > From: Stefan Frey [mailto:ste...@we...] > > > Sent: Monday, August 01, 2011 7:27 AM > > > To: 'Development list for Rails: an 18xx game' > > > Subject: [Rails-devel] Automated testing > > > > > > I have added the games that I have received to automated game testing. > > > Thanks to all contributing save files. > > > There is also a table of those in the wiki (under developer information). > > > > > > In the next few days (as time permits) I will add the existing > > > information > > > > on > > > > > game testing (from older mails) to the wiki and I hope it will increase > > > > the > > > > > usage of those tests. > > > > > > Currently all tests fail as Erik has changed many things that impacted > > > the Game Report (phase names and train obsolete messages). > > > > > > Question to Erik: If you think all changes are merely phase name changes > > > > and > > > > > adding/removing additional messages and you are sure that everything > > > works ok, give me a nod and then the game reports can be updated to > the > > > current states and all tests will show green again. > > > > > > Unfortunately my master still automatically merges on pull, I will setup > > > > the > > > > > master fresh as soon as I have pushed all my current local branches. > > > > > > Stefan > > > > --------------------------------------------------------------------------- > > - -- > > > > > Got Input? Slashdot Needs You. > > > Take our quick survey online. Come on, we don't ask for help often. > > > Plus, you'll get a chance to win $100 to spend on ThinkGeek. > > > http://p.sf.net/sfu/slashdot-survey > > > _______________________________________________ > > > Rails-devel mailing list > > > Rai...@li... > > > https://lists.sourceforge.net/lists/listinfo/rails-devel > > > > --------------------------------------------------------------------------- > > --- BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA > > The must-attend event for mobile developers. Connect with experts. > > Get tools for creating Super Apps. See the latest technologies. > > Sessions, hands-on labs, demos & much more. Register early & save! > > http://p.sf.net/sfu/rim-blackberry-1 > > _______________________________________________ > > Rails-devel mailing list > > Rai...@li... > > https://lists.sourceforge.net/lists/listinfo/rails-devel > > ---------------------------------------------------------------------------- -- > BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA > The must-attend event for mobile developers. Connect with experts. > Get tools for creating Super Apps. See the latest technologies. > Sessions, hands-on labs, demos & much more. Register early & save! > http://p.sf.net/sfu/rim-blackberry-1 > _______________________________________________ > Rails-devel mailing list > Rai...@li... > https://lists.sourceforge.net/lists/listinfo/rails-devel |
