Menu
▾
▴
rails-devel
|
From: Stefan F. <ste...@we...> - 2010-03-23 19:55:39
Attachments:
testing.txt
rails_testreport.html
|
I have added automated regression testing for Rails in CVS.
How does it work:
* For a set of Rails save files (which are known that they are correct) game
reports are created and stored alongside with the save files.
Then each time the tests are run, each game is replayed by Rails on the new
code base and the game reports are compared to the stored ones.
* Anytime a difference in the game reports occurs, this test reports the error
and displays the first line, where the game reports deviate.
More instructions and comments for developers see attached text file. In
addition I have added an example report of a current test run.
Stefan
I have not uploaded my preliminary test suite so far, as I want to ask for
opinion on the tree structure first.
My proposal is:
Convention for file names:
{18xx}_{noMap - if used}_{author}_{nb of players}_{end round}_{end
condition}_{special remarks}
-> Bug reports
The game name should include the bug number if possible.
below that with categories, that still have to be defined.
-> Finished games
real plays
below that with one folder for each 18xx
-> Pbem games
currently running pbem games
below that with one folder for each 18xx
-> Test cases
hand crafted settings for specific testing
below that with one folder for each 18xx
-> Test games
simulated games
below that with one folder for each 18xx
|
|
From: Erik V. <eri...@xs...> - 2010-03-23 20:18:33
|
Looks very good to me.
I generally agree with your proposals. Just one remark: can't the tree
hierarchy be better reversed?
For what it's worth: I would tend to select a game first, then look what
saved files/test cases exist for that game.
And a warning: most of the saved files I have sent you off-list do NOT
constitute valid games.
The revenues have often been faked, either to construct a particular test
case, or just because I was lazy.
Never trust my saved files!
Erik.
-----Original Message-----
From: Stefan Frey [mailto:ste...@we...]
Sent: Tuesday 23 March 2010 19:53
To: Development list for Rails: an 18xx game
Subject: [Rails-devel] Automated testing for Rails
I have added automated regression testing for Rails in CVS.
How does it work:
* For a set of Rails save files (which are known that they are correct) game
reports are created and stored alongside with the save files.
Then each time the tests are run, each game is replayed by Rails on the new
code base and the game reports are compared to the stored ones.
* Anytime a difference in the game reports occurs, this test reports the
error
and displays the first line, where the game reports deviate.
More instructions and comments for developers see attached text file. In
addition I have added an example report of a current test run.
Stefan
I have not uploaded my preliminary test suite so far, as I want to ask for
opinion on the tree structure first.
My proposal is:
Convention for file names:
{18xx}_{noMap - if used}_{author}_{nb of players}_{end round}_{end
condition}_{special remarks}
-> Bug reports
The game name should include the bug number if possible.
below that with categories, that still have to be defined.
-> Finished games
real plays
below that with one folder for each 18xx
-> Pbem games
currently running pbem games
below that with one folder for each 18xx
-> Test cases
hand crafted settings for specific testing
below that with one folder for each 18xx
-> Test games
simulated games
below that with one folder for each 18xx
|
|
From: Stefan F. <ste...@we...> - 2010-03-23 20:59:38
|
Erik:
- Reversing the tree makes sense.
- For faked games there is the category test_cases.
For testing reasons it does not matter if revenues are faked, as long as the
game runs through. I did the same: Testing game end is easy if you have one
company pay out $20.000 in OR 2.
In any case it makes sense that you add your test save files yourself to the
tree.
And I promise that I will never trust any of your save files...
Stefan
On Tuesday 23 March 2010 21:18:21 Erik Vos wrote:
> Looks very good to me.
>
> I generally agree with your proposals. Just one remark: can't the tree
> hierarchy be better reversed?
> For what it's worth: I would tend to select a game first, then look what
> saved files/test cases exist for that game.
>
> And a warning: most of the saved files I have sent you off-list do NOT
> constitute valid games.
> The revenues have often been faked, either to construct a particular test
> case, or just because I was lazy.
> Never trust my saved files!
>
> Erik.
>
> -----Original Message-----
> From: Stefan Frey [mailto:ste...@we...]
> Sent: Tuesday 23 March 2010 19:53
> To: Development list for Rails: an 18xx game
> Subject: [Rails-devel] Automated testing for Rails
>
> I have added automated regression testing for Rails in CVS.
>
> How does it work:
> * For a set of Rails save files (which are known that they are correct)
> game
>
> reports are created and stored alongside with the save files.
> Then each time the tests are run, each game is replayed by Rails on the new
> code base and the game reports are compared to the stored ones.
>
> * Anytime a difference in the game reports occurs, this test reports the
> error
> and displays the first line, where the game reports deviate.
>
> More instructions and comments for developers see attached text file. In
> addition I have added an example report of a current test run.
>
> Stefan
>
> I have not uploaded my preliminary test suite so far, as I want to ask for
> opinion on the tree structure first.
>
> My proposal is:
>
> Convention for file names:
> {18xx}_{noMap - if used}_{author}_{nb of players}_{end round}_{end
> condition}_{special remarks}
>
> -> Bug reports
> The game name should include the bug number if possible.
> below that with categories, that still have to be defined.
>
> -> Finished games
> real plays
> below that with one folder for each 18xx
>
> -> Pbem games
> currently running pbem games
> below that with one folder for each 18xx
>
> -> Test cases
> hand crafted settings for specific testing
> below that with one folder for each 18xx
>
> -> Test games
> simulated games
> below that with one folder for each 18xx
>
>
>
> ---------------------------------------------------------------------------
>--- Download Intel® Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Rails-devel mailing list
> Rai...@li...
> https://lists.sourceforge.net/lists/listinfo/rails-devel
|
|
From: brett l. <bre...@gm...> - 2010-03-23 20:44:52
|
On Tue, Mar 23, 2010 at 11:52 AM, Stefan Frey <ste...@we...> wrote:
> I have added automated regression testing for Rails in CVS.
Excellent. We've needed this for a while, and this is a good start.
> How does it work:
> * For a set of Rails save files (which are known that they are correct) game
> reports are created and stored alongside with the save files.
> Then each time the tests are run, each game is replayed by Rails on the new
> code base and the game reports are compared to the stored ones.
>
This is fine as a starting point, but it's not how most traditional
automated testing works.
One goal that I hope we work toward is building on this foundation by
developing unit and regression tests for each object that doesn't
require a save file.
Ideally, we should be able to build out mock objects of most things.
If we can't, that might be an indicator that refactoring is needed. If
nothing else, it should spawn some javadoc about why the object is
difficult/impossible to test.
> My proposal is:
>
> Convention for file names:
> {18xx}_{noMap - if used}_{author}_{nb of players}_{end round}_{end
> condition}_{special remarks}
This filenaming convention is going to be very unwieldy.
How about this:
Save files: {game name}-{current round}-{unique identifier}.save
Index file: savegame-tests.txt
The save game will simply be something like 1870-OR5-1.save,
1830-SR2-4.save, etc.
Then, the index file will track our notes about each saved game, and
what tests it applies to. I think there's going to need to be more
information than can be contained within the filename itself.
Alternately, we can have each saved game with it's own
{conventiion}.txt file that documents this, but that may be overkill.
Of course, all of this should probably go under /test/savedgames, or
something similarly named and organized away from the rest of the
code.
---Brett.
|
|
From: Stefan F. <ste...@we...> - 2010-03-23 22:17:49
|
Brett,
see my comments below.
Stefan
> Excellent. We've needed this for a while, and this is a good start.
Thanks. I am happy to have that too, as it now easier to refactor classes.
>
> This is fine as a starting point, but it's not how most traditional
> automated testing works.
Yes and no. It is automated testing, but obviously it is not (traditiional)
unit testing, but belongs to the class of integration or system tests. But
all are equally important.
>
> One goal that I hope we work toward is building on this foundation by
> developing unit and regression tests for each object that doesn't
> require a save file.
>
> Ideally, we should be able to build out mock objects of most things.
> If we can't, that might be an indicator that refactoring is needed. If
> nothing else, it should spawn some javadoc about why the object is
> difficult/impossible to test.
>
The major issue I see why people are reluctant to write unit tests for
existing code in Rails currently, is that it is much work to do with no
immediate reward. This is very similar to ask someone for writing
documentation for existing code.
For the integration tests above I could leverage on existing mechanisms:
The action stack (already used for the save files) to replay games and the
ReportBuffer, that collects the game report. Thus easy to do.
My goal for working on this project is to use it for playing 18xx: That is the
basis for all my contributions so far: We usually play with a real map =>
nomap Mode. I want to be able to make corrections on the fly => correction
actions. I like 1889 => 1889!
And why the tests now? My next focus will be 1870. 1870 adds some weird stuff
to the stockround. For this I would like to refactor this class to make it
modular. Therefore I need refactoring support, thus the automated testing.
Very simple.
> How about this:
>
> Save files: {game name}-{current round}-{unique identifier}.save
> Index file: savegame-tests.txt
> The save game will simply be something like 1870-OR5-1.save,
> 1830-SR2-4.save, etc.
The extension for the save files can be changed in test.properties, but I
prefer the .rails extension to the .save (which could be anything).
I would suggest then to reverse the order and add a game end indicator:
{game name}_{unique identifier}_{current round}{[_e,_b]}
where e indicates normal end (bank out of money), b indicates bankrupt.
Examples: 1830_4_sr2.rails, 1830_4_or7_e.rails, 1830_3_or5_b.rails
>
> Then, the index file will track our notes about each saved game, and
> what tests it applies to. I think there's going to need to be more
> information than can be contained within the filename itself.
>
> Alternately, we can have each saved game with it's own
> {conventiion}.txt file that documents this, but that may be overkill.
>
> Of course, all of this should probably go under /test/savedgames, or
> something similarly named and organized away from the rest of the
> code.
>
That is how it is laid out: Root save directory can be changed in
test.properties. Currently set to test/data
|
|
From: brett l. <bre...@gm...> - 2010-03-24 00:04:43
|
On Tue, Mar 23, 2010 at 3:17 PM, Stefan Frey <ste...@we...> wrote:
> Brett,
> see my comments below.
> Stefan
>
>> Excellent. We've needed this for a while, and this is a good start.
>
> Thanks. I am happy to have that too, as it now easier to refactor classes.
>>
>> This is fine as a starting point, but it's not how most traditional
>> automated testing works.
>
> Yes and no. It is automated testing, but obviously it is not (traditiional)
> unit testing, but belongs to the class of integration or system tests. But
> all are equally important.
>
Agreed. I phrased it badly, but yes, we're saying the same things.
>>
>> One goal that I hope we work toward is building on this foundation by
>> developing unit and regression tests for each object that doesn't
>> require a save file.
>>
>> Ideally, we should be able to build out mock objects of most things.
>> If we can't, that might be an indicator that refactoring is needed. If
>> nothing else, it should spawn some javadoc about why the object is
>> difficult/impossible to test.
>>
>
> The major issue I see why people are reluctant to write unit tests for
> existing code in Rails currently, is that it is much work to do with no
> immediate reward. This is very similar to ask someone for writing
> documentation for existing code.
>
Yup. Mostly I was pointing out that we have a starting point. We can
build on that incrementally.
I didn't mean to make it sound like you should do all of the work. It
was more that, now we can consider building tests as we touch old code
and/or as we write new code.
These things are always much easier if we tackle them a little bit at
a time, as we work on other things.
> For the integration tests above I could leverage on existing mechanisms:
> The action stack (already used for the save files) to replay games and the
> ReportBuffer, that collects the game report. Thus easy to do.
>
> My goal for working on this project is to use it for playing 18xx: That is the
> basis for all my contributions so far: We usually play with a real map =>
> nomap Mode. I want to be able to make corrections on the fly => correction
> actions. I like 1889 => 1889!
>
> And why the tests now? My next focus will be 1870. 1870 adds some weird stuff
> to the stockround. For this I would like to refactor this class to make it
> modular. Therefore I need refactoring support, thus the automated testing.
> Very simple.
>
That's totally okay by me. I'm not trying to suggest that you do it
differently.
Work on the things that interest you. That's how we've gotten this far. :-)
>> How about this:
>>
>> Save files: {game name}-{current round}-{unique identifier}.save
>> Index file: savegame-tests.txt
>> The save game will simply be something like 1870-OR5-1.save,
>> 1830-SR2-4.save, etc.
>
>
> The extension for the save files can be changed in test.properties, but I
> prefer the .rails extension to the .save (which could be anything).
>
> I would suggest then to reverse the order and add a game end indicator:
>
> {game name}_{unique identifier}_{current round}{[_e,_b]}
>
> where e indicates normal end (bank out of money), b indicates bankrupt.
>
> Examples: 1830_4_sr2.rails, 1830_4_or7_e.rails, 1830_3_or5_b.rails
>
That's fine. I'm less concerned about the extension as I am about
trying to fit too much information into the file name.
>>
>> Then, the index file will track our notes about each saved game, and
>> what tests it applies to. I think there's going to need to be more
>> information than can be contained within the filename itself.
>
>>
>> Alternately, we can have each saved game with it's own
>> {conventiion}.txt file that documents this, but that may be overkill.
>>
>> Of course, all of this should probably go under /test/savedgames, or
>> something similarly named and organized away from the rest of the
>> code.
>>
>
> That is how it is laid out: Root save directory can be changed in
> test.properties. Currently set to test/data
>
Perfect. Sounds like you're on the right track. :-)
---Brett.
|
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X