I have some questions about the 'normal and planned usage of the plugin'
I)
How should one actually use the plans? The answer may be addition(s) to the already existing wiki, I think?
In our approach they are considered as a test round for each week and I'm having doubts that this may not work in the long term.
We have used it now so, that we have only on catalog for each platform we need to test. and we do create a plan for each test round we are running. The thing with this approach is that now we can have the status of the test round in the plan, but the problem is that we'd get like 52 plans pretty easily in a year if we'd do a plan only for each cycle. well we do create a plan for each device tested so it's easily up to 4-8*52 test plans annually, unless we start to remove old and possibly obsolete plans.
I'm not sure how to handle the problem with this amount of plans in the future. Of course by naming them like '2011-wk38 -bla la bla DeviceX' would keep them in alphabetical order anyway the list will be pretty big in no time.
I see at least few approaches to possibly handle the issue.
1) a filter to the plan view is probably the easiest.
2) Or the we would just have a one plan we would test again and again but we would need to set a round for the plan in order to actually somehow see the history of the plan.
This 2nd one might easily broke the existing database structure unless we would have the possibility to move plans under other plan as rounds, or that's the case in our case at the moment, of course we could sacrifice the history and delete them or just leave the plans as they are and continue with the possible new way.
And this is only if our way to use the plans is anywhere in the same planet it's been planned to be used.
II)
Should there be different catalogs for full test and smoke test sets?
The other issue I've been wondering is how to separate or point out smoke test cases from the full test set.
I can see only one way to do it at this point and it would be to copy the smoke test set into another catalog. Even I'd like to do it so that I could actually select the test cases from a catalog into a test plan and in that plan only test that set. This would also help in the cases when the smoke test set has to be modified in order to pay attention into a new or modified feature set.
If we would need to just create a new catalog for different smoke test sets we would end up in a situation that we would need to update X catalogs including the same test case we have to update.
and the hierarchy may start to cause some problems if not think thoroughly
III)
This can actually be seen as another general question:
How should the catalogs be organized if one wants to keep the specification and tickets created from certain version of a test case in synch?
as we have catalogs
Paltform A
Platform B
We would actually need something like
Full Plat A
Full Plat B
Smoke Plat A
Smoke Plat B
and under those a subset of the specifications like
Full Plat A
Specification 0.1
Specification 0.3
Specification 1.0
and then just create plans from the Specification level
What are your ideas about this, has it been thought how to actually do something like it
At this point we do not actually have the separation for each specification version as we only have one test case that is modified when needed, so actually when we create an ticket from a test case and look it afterwards we should also check for the test case history and the day of the ticket to see how it has been tested IF the test case has been modified after the ticket has been created.
Sorry this became a bit long.
Best Regards
Jouko
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Jouko,
I don't have any answers as to how the plugin is intended to be used… I can only tell you how I am using it in my environment.
We don't have platforms and we don't have smoke and full test rounds, we have a few test plans only, one for each release of an application.
Anyway, I think your way of using it is not wrong at all, only you need some more functionality to be able to fulfill all of your requirements.
I don't think there is currently a solution to any of your questions above, so let's think about how we may improve the plugin to let it work well in your environment.
From what you say it may be ok to do the following:
- When creating a new test plan on a catalog, the User may be presented with a popup window to:
a) Select the test cases to be included into the test plan, or just click "select all".
b) Specify whether new test cases added to the catalogs would be included or excluded from the test plan.
c) Specify whether the test plan should refer to a current snapshot of the versions of the test cases, or always point to the latest version of a test case.
Point a) and b) above should help solve you issue #2, while point c) should help with #3.
As for your issue #1, the proliferation of test plans as the number of rounds grow, your idea of a filter is simple and should be sufficient, while adding the concept of test rounds would be much more complicated.
Anyway, I like the idea, and I'd like to hear form you how you would see the use case (I see you're an usability expert): what would be the actual user scenario? How would a User be able to define test rounds, view and manage them?
Ciao
Roberto
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- allowing the User to choose the options for a) and c) above, when creating a new test plan.
- You will see that a test plan can now have a subset of the test cases in a catalog (first select the test cases and only then click "New test plan")
- You will see that if you specify to take a snapshot of the test catalog, and you later change some test case descriptions, when you browse the test cases in the plan you will see the snapshot version of the wiki pages
Being able to filter the test plans that will be displayed in the test results would be VERY helpful. Do you see this being a filter on the name of the test plans or on some new variable associated with the test plan? The first option would be helpful but the second option would provide a lot more flexibility in how the test management is done.
Another difficulty I have with test plans is finding them. The way I've structured my test catalog is so that I can create test plans at the sub-catalog level and assign the test plan to another person for execution. Test plans are only shown at the subcatalog where they were created.
One suggestion might be to have another count displayed after the catalog/subcatalog that identifies how many test plans are available. Right now you show in parenthesis how many test cases, may a second parenthesis in a different color for how many test plans.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thank you for the compliment about my usability perspective. :D I do have some studies and motivation on that area.
I have to try your latest modifications asap, they do sound great and most likely will meet the needs our testing team have been somehow lacking. I may need an week or two to fully try this out. Especially, because for a great deal of our team, the forthcoming week wil be a holiday. Did you by any chance check how the reports will be handling the new
The filter for the test rounds does sound clean and clear solution. I may want to filter it by time and by name, no you think it would be easily implemented to do both at the same time.
I can imagine at least one use case shortly: someone looking for a plan for a platform XY on certain time period.
If you like to imagine the case on UI perpetive it could be something like user would give the platform name into text field (might be a bit hard to implement a combo box with selectables) and by identifying a time period (maybe the dates would be selectable from a calendar? or just a preformatted text field by minimum)
I do like the idea about the subtestpalns having more visibility from bkononen, too
Thank you Roberto, for the great work you've been doing with the very usefull Trac plugin
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The first try out for the 1.4.8 dev version has been done and it does seem to meet the request about the limited amount of test cases in the test plan. I have not tried out the history of the test case reference just yet.
There may be some issues related to the statistical graphics, though.
Did you check how the statistical reports work with this upgrade? I think they may see all the test cases in the original catalog, not just the selected subset.
For example:
I chose 3 test cases out of 15 for the plan and set one failed, one success and one as N/A, so it should be 0 untested on this plan, since the one is
1)The test activity trend is showing 15 test cases (all of the original catalog) even I only selected 3 to be the subset in this plan. I also noticed that If I have more choices on the Yellow (untested) they will only show up as untested, not under the 'user defined' value. But this has been like this already on previous releases.
2) The Current test status does show 1 failed, 1 successful and 4 as not tested, even I have set one of the test cases to N/A as not applicable. It seems like the pie chart can't separate the one's that has been defined by the user into the 'yellow' part from the not tested into something else that is also 'yellow'
(we have N/A, not tested and Not possible set to be in the yellow zone. Have to check why we have almost two identical here :/ )
But why is there like 1+1+4 as a total when I only selected 3 to exist in the plan is a mystery to me. the 1+1+1 was expected.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi guys,
finally I found some time and, also thanks to Andreas contribution, I fixed the statistics problem and added a couple of nice things.
Release 1.4.8 is out for you to download.
o Strongly enhanced the upgrade mechanism. Now it's more robust, should work with all the databases and between arbitrary Test Manager versions.
The only drawback is that upgrade is only supported from 1.4.7, not from previous versions.
o Enhancement #9077 (Track-Hacks): Ability to separate and report on test plans by product
o Enhancement #9208 (Track-Hacks): Test plan with only selected test cases from the catalog, take snapshot version of test cases.
This is an important one. Many users were asking for a way of including only selected test cases into
a Test Plan, for different reasons. Now you have it :D
o Added ability to search through the test plans of a catalog.
o Added French language catalog! Thanks to someone who doesn't want to be cited :D
o Fixed Ticket #9141 (Track-Hacks): Update installation 1.4.6 -> 1.4.7 not possible
o Fixed Ticket #9167 (Track-Hacks): installation of 1.4.7 with postgres database not possible
o Fixed Ticket #9187 (Track-Hacks): Current test status report should consider only last result of a testcase in the plan.
Thanks to Andreas for his contribution to fixing this one!
Thanks to you guys for your ideas, testing and language translations!!!
(Actually there is the need for updating the translations. If anyone wishes to contribute…)
Please, let me know how it goes. Ciao.
Roberto
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have few points that could be worth considering:
1. There is a possibility to create a ticket from a test case, but the other way - create test case from a ticket. This is the commom workflow in software development.
Such new test cases could be created in folders which structure would be the same as ticket, so:
Component -> Version -> Milestone -> Test Case
2. If I create a ticket from a test case in fields 'Test Case:' and 'Test Plan:' I get values like "TC_TT17_TC20" or "TC_TT17" that dont't say much. Those aren't even links to relevant test case or test plan.
Maybe test cases could be numbered just like tickets?
regards,
Piotr
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have some questions about the 'normal and planned usage of the plugin'
I)
How should one actually use the plans? The answer may be addition(s) to the already existing wiki, I think?
In our approach they are considered as a test round for each week and I'm having doubts that this may not work in the long term.
We have used it now so, that we have only on catalog for each platform we need to test. and we do create a plan for each test round we are running. The thing with this approach is that now we can have the status of the test round in the plan, but the problem is that we'd get like 52 plans pretty easily in a year if we'd do a plan only for each cycle. well we do create a plan for each device tested so it's easily up to 4-8*52 test plans annually, unless we start to remove old and possibly obsolete plans.
I'm not sure how to handle the problem with this amount of plans in the future. Of course by naming them like '2011-wk38 -bla la bla DeviceX' would keep them in alphabetical order anyway the list will be pretty big in no time.
I see at least few approaches to possibly handle the issue.
1) a filter to the plan view is probably the easiest.
2) Or the we would just have a one plan we would test again and again but we would need to set a round for the plan in order to actually somehow see the history of the plan.
This 2nd one might easily broke the existing database structure unless we would have the possibility to move plans under other plan as rounds, or that's the case in our case at the moment, of course we could sacrifice the history and delete them or just leave the plans as they are and continue with the possible new way.
And this is only if our way to use the plans is anywhere in the same planet it's been planned to be used.
II)
Should there be different catalogs for full test and smoke test sets?
The other issue I've been wondering is how to separate or point out smoke test cases from the full test set.
I can see only one way to do it at this point and it would be to copy the smoke test set into another catalog. Even I'd like to do it so that I could actually select the test cases from a catalog into a test plan and in that plan only test that set. This would also help in the cases when the smoke test set has to be modified in order to pay attention into a new or modified feature set.
If we would need to just create a new catalog for different smoke test sets we would end up in a situation that we would need to update X catalogs including the same test case we have to update.
and the hierarchy may start to cause some problems if not think thoroughly
III)
This can actually be seen as another general question:
How should the catalogs be organized if one wants to keep the specification and tickets created from certain version of a test case in synch?
as we have catalogs
Paltform A
Platform B
We would actually need something like
Full Plat A
Full Plat B
Smoke Plat A
Smoke Plat B
and under those a subset of the specifications like
Full Plat A
Specification 0.1
Specification 0.3
Specification 1.0
and then just create plans from the Specification level
What are your ideas about this, has it been thought how to actually do something like it
At this point we do not actually have the separation for each specification version as we only have one test case that is modified when needed, so actually when we create an ticket from a test case and look it afterwards we should also check for the test case history and the day of the ticket to see how it has been tested IF the test case has been modified after the ticket has been created.
Sorry this became a bit long.
Best Regards
Jouko
Hi Jouko,
I don't have any answers as to how the plugin is intended to be used… I can only tell you how I am using it in my environment.
We don't have platforms and we don't have smoke and full test rounds, we have a few test plans only, one for each release of an application.
Anyway, I think your way of using it is not wrong at all, only you need some more functionality to be able to fulfill all of your requirements.
I don't think there is currently a solution to any of your questions above, so let's think about how we may improve the plugin to let it work well in your environment.
From what you say it may be ok to do the following:
- When creating a new test plan on a catalog, the User may be presented with a popup window to:
a) Select the test cases to be included into the test plan, or just click "select all".
b) Specify whether new test cases added to the catalogs would be included or excluded from the test plan.
c) Specify whether the test plan should refer to a current snapshot of the versions of the test cases, or always point to the latest version of a test case.
Point a) and b) above should help solve you issue #2, while point c) should help with #3.
As for your issue #1, the proliferation of test plans as the number of rounds grow, your idea of a filter is simple and should be sufficient, while adding the concept of test rounds would be much more complicated.
Anyway, I like the idea, and I'd like to hear form you how you would see the use case (I see you're an usability expert): what would be the actual user scenario? How would a User be able to define test rounds, view and manage them?
Ciao
Roberto
Hi Jouko,
I found some time today to work on the features outlined above.
You can find in attachment to this ticket https://trac-hacks.org/ticket/9208 a development 1.4.8 version, which includes:
- more robust database upgrade mechanism ( see tickets http://trac-hacks.org/ticket/9141 and http://trac-hacks.org/ticket/9167 ). Important Note: Upgrade works only from 1.4.7, not from previous versions.
- allowing the User to choose the options for a) and c) above, when creating a new test plan.
- You will see that a test plan can now have a subset of the test cases in a catalog (first select the test cases and only then click "New test plan")
- You will see that if you specify to take a snapshot of the test catalog, and you later change some test case descriptions, when you browse the test cases in the plan you will see the snapshot version of the wiki pages
Let me know what you think.
Ciao,
Robertonull
Being able to filter the test plans that will be displayed in the test results would be VERY helpful. Do you see this being a filter on the name of the test plans or on some new variable associated with the test plan? The first option would be helpful but the second option would provide a lot more flexibility in how the test management is done.
Another difficulty I have with test plans is finding them. The way I've structured my test catalog is so that I can create test plans at the sub-catalog level and assign the test plan to another person for execution. Test plans are only shown at the subcatalog where they were created.
One suggestion might be to have another count displayed after the catalog/subcatalog that identifies how many test plans are available. Right now you show in parenthesis how many test cases, may a second parenthesis in a different color for how many test plans.
Hello All, especially Roberto,
Thank you for the compliment about my usability perspective. :D I do have some studies and motivation on that area.
I have to try your latest modifications asap, they do sound great and most likely will meet the needs our testing team have been somehow lacking. I may need an week or two to fully try this out. Especially, because for a great deal of our team, the forthcoming week wil be a holiday. Did you by any chance check how the reports will be handling the new
The filter for the test rounds does sound clean and clear solution. I may want to filter it by time and by name, no you think it would be easily implemented to do both at the same time.
I can imagine at least one use case shortly: someone looking for a plan for a platform XY on certain time period.
If you like to imagine the case on UI perpetive it could be something like user would give the platform name into text field (might be a bit hard to implement a combo box with selectables) and by identifying a time period (maybe the dates would be selectable from a calendar? or just a preformatted text field by minimum)
I do like the idea about the subtestpalns having more visibility from bkononen, too
Thank you Roberto, for the great work you've been doing with the very usefull Trac plugin
The first try out for the 1.4.8 dev version has been done and it does seem to meet the request about the limited amount of test cases in the test plan. I have not tried out the history of the test case reference just yet.
There may be some issues related to the statistical graphics, though.
Did you check how the statistical reports work with this upgrade? I think they may see all the test cases in the original catalog, not just the selected subset.
For example:
I chose 3 test cases out of 15 for the plan and set one failed, one success and one as N/A, so it should be 0 untested on this plan, since the one is
1)The test activity trend is showing 15 test cases (all of the original catalog) even I only selected 3 to be the subset in this plan. I also noticed that If I have more choices on the Yellow (untested) they will only show up as untested, not under the 'user defined' value. But this has been like this already on previous releases.
2) The Current test status does show 1 failed, 1 successful and 4 as not tested, even I have set one of the test cases to N/A as not applicable. It seems like the pie chart can't separate the one's that has been defined by the user into the 'yellow' part from the not tested into something else that is also 'yellow'
(we have N/A, not tested and Not possible set to be in the yellow zone. Have to check why we have almost two identical here :/ )
But why is there like 1+1+4 as a total when I only selected 3 to exist in the plan is a mystery to me. the 1+1+1 was expected.
Hi guys,
finally I found some time and, also thanks to Andreas contribution, I fixed the statistics problem and added a couple of nice things.
Release 1.4.8 is out for you to download.
o Strongly enhanced the upgrade mechanism. Now it's more robust, should work with all the databases and between arbitrary Test Manager versions.
The only drawback is that upgrade is only supported from 1.4.7, not from previous versions.
o Enhancement #9077 (Track-Hacks): Ability to separate and report on test plans by product
o Enhancement #9208 (Track-Hacks): Test plan with only selected test cases from the catalog, take snapshot version of test cases.
This is an important one. Many users were asking for a way of including only selected test cases into
a Test Plan, for different reasons. Now you have it :D
o Added ability to search through the test plans of a catalog.
o Added French language catalog! Thanks to someone who doesn't want to be cited :D
o Fixed Ticket #9141 (Track-Hacks): Update installation 1.4.6 -> 1.4.7 not possible
o Fixed Ticket #9167 (Track-Hacks): installation of 1.4.7 with postgres database not possible
o Fixed Ticket #9187 (Track-Hacks): Current test status report should consider only last result of a testcase in the plan.
Thanks to Andreas for his contribution to fixing this one!
Thanks to you guys for your ideas, testing and language translations!!!
(Actually there is the need for updating the translations. If anyone wishes to contribute…)
Please, let me know how it goes. Ciao.
Roberto
hi,
I have few points that could be worth considering:
1. There is a possibility to create a ticket from a test case, but the other way - create test case from a ticket. This is the commom workflow in software development.
Such new test cases could be created in folders which structure would be the same as ticket, so:
Component -> Version -> Milestone -> Test Case
2. If I create a ticket from a test case in fields 'Test Case:' and 'Test Plan:' I get values like "TC_TT17_TC20" or "TC_TT17" that dont't say much. Those aren't even links to relevant test case or test plan.
Maybe test cases could be numbered just like tickets?
regards,
Piotr