Thread: [Barry-devel] Circular Testing
Status: Beta
Brought to you by:
ndprojects
From: Martin O. <doc...@gm...> - 2010-05-30 02:29:38
|
Hey Barry and Chris :-D I was thinking some more about format transition testing and how we can be more sure about the formats of the data we've getting out of blackberries and indeed how we can be sure it's all safe. I propose the following method of testing and organisation for testers to see if something can be set up small scale for this limited set of devices. We use a method of circular testing which involves having a pre-defined data set with all the fields we want to test (multiple data sets if required) and the system would simply feed them into the blackberry, then feed them out and compare the data with the original. This would be more effective with opensync of course, but I think just testing say the contacts would be useful. Now I say to make sure we have device coverage perhaps we can have a sign up sheet so people can sign up to test with the test cases each release of barry with the phone model they've specified. I'd be willing to sign up my Blackberry Pearl 2120 (t-mobile) and we can make sure to test as many as we can for as many models as people are willing to test. Your thoughts? Martin, |
From: Chris F. <cd...@fo...> - 2010-05-30 03:09:03
|
Hi Martin, I love the idea, of course. I also think there is another way of going about this. What if multiple people donated small backup tarball files for the databases we support? Rigging up an automated test suit that would parse, and perhaps build, the databases would make testing easy, and almost part of the build. I like both ideas though. My idea probably requires a bit more code to start with, but would make testing easier in the long run. In the general sense, though, I welcome more Barry testing of all kinds. - Chris On Sat, May 29, 2010 at 10:29:33PM -0400, Martin Owens wrote: > Hey Barry and Chris :-D > > I was thinking some more about format transition testing and how we can > be more sure about the formats of the data we've getting out of > blackberries and indeed how we can be sure it's all safe. > > I propose the following method of testing and organisation for testers > to see if something can be set up small scale for this limited set of > devices. > > We use a method of circular testing which involves having a pre-defined > data set with all the fields we want to test (multiple data sets if > required) and the system would simply feed them into the blackberry, > then feed them out and compare the data with the original. > > This would be more effective with opensync of course, but I think just > testing say the contacts would be useful. > > Now I say to make sure we have device coverage perhaps we can have a > sign up sheet so people can sign up to test with the test cases each > release of barry with the phone model they've specified. > > I'd be willing to sign up my Blackberry Pearl 2120 (t-mobile) and we can > make sure to test as many as we can for as many models as people are > willing to test. > > Your thoughts? > > Martin, > > > ------------------------------------------------------------------------------ > > _______________________________________________ > Barry-devel mailing list > Bar...@li... > https://lists.sourceforge.net/lists/listinfo/barry-devel |
From: Martin O. <doc...@gm...> - 2010-05-30 20:33:15
|
Hey Chris, On Sat, 2010-05-29 at 23:08 -0400, Chris Frey wrote: > What if multiple people donated small backup tarball files for the > databases we support? Rigging up an automated test suit that would > parse, and perhaps build, the databases would make testing easy, and > almost part of the build. The big problem with having donated data for testing is apart from a security issue, we'd also not be sure of what we're testing, not completely sure of what features and such the blackberry supports and how to translate them. We'd effectively be trading in a knowledge of the data structure for real world data. Both perhaps are useful since problems in the real world data would allow you to improve the known data sets. I do love the idea of having real world data though, and it should be kept up to date too. Now that I think about it I'm more sure of having both in place. Real World Tests -> Clean World Tests -> Data Transition Verified. Martin, |
From: Chris F. <cd...@fo...> - 2010-05-31 19:52:09
|
On Sun, May 30, 2010 at 04:32:57PM -0400, Martin Owens wrote: > I do love the idea of having real world data though, and it should be > kept up to date too. Now that I think about it I'm more sure of having > both in place. > > Real World Tests -> Clean World Tests -> Data Transition Verified. Do you want to write up a page of step-by-step instructions that volunteers would need to follow? If you write it in HTML, I can include it in the doc/www/ documentation and put it on the website for handy reference. - Chris |
From: Martin O. <doc...@gm...> - 2010-05-31 22:25:18
|
On Mon, 2010-05-31 at 15:52 -0400, Chris Frey wrote: > Do you want to write up a page of step-by-step instructions that volunteers > would need to follow? If you write it in HTML, I can include it in > the doc/www/ documentation and put it on the website for handy reference. I could, but there aren't any tests written. |
From: Chris F. <cd...@fo...> - 2010-05-31 22:36:39
|
On Mon, May 31, 2010 at 06:24:35PM -0400, Martin Owens wrote: > I could, but there aren't any tests written. Ahh... perhaps I'm misunderstanding your plan of testing. I thought it might be as simple as having some data that btool could read, and then do a btool -s... -f... and btool -d... -f... and compare the output. - Chris |
From: Martin O. <doc...@gm...> - 2010-05-31 22:45:21
|
On Mon, 2010-05-31 at 18:36 -0400, Chris Frey wrote: > I thought it might be as simple as having some data that btool could read, > and then do a btool -s... -f... and btool -d... -f... and compare > the output. Sounds like a script, easy to write and easy to make into a standard test. :-) Martin, |
From: Chris F. <cd...@fo...> - 2010-06-04 06:02:02
|
On Mon, May 31, 2010 at 06:45:13PM -0400, Martin Owens wrote: > On Mon, 2010-05-31 at 18:36 -0400, Chris Frey wrote: > > > I thought it might be as simple as having some data that btool could read, > > and then do a btool -s... -f... and btool -d... -f... and compare > > the output. > > Sounds like a script, easy to write and easy to make into a standard > test. > > :-) I'm not sure of this will help, but I added a new -I option to btool, which sorts the database output before dumping to stdout. With this, it may be possible to script some load/save/delete/compare scripts. Also keep in mind the 'brecsum' command if you want to do a very strict data comparison. That is more for backup restore testing though. Let me know if this helps your test scripts. You should be able to create a set of test data, then save it with: btool -d Calendar -f calendar.dat Restore it with: btool -s Calendar -f calendar.dat and the results *should* be the same. :-) - Chris |