#11 Parameterized tests - move test case methods to parameter type

[Michael L. Rilee]

There seems to be excessive overhead to run the same test on a bunch of lat/lon points.

Can we move some of the test case methods to the parameter type, to avoid making the user define a new test case type and new methods on it?

Question: Would doing this for user-defined parameters impact MPI tests in general?

[Sean Patrick Santos]

I'd just like to weigh in on what I would like to see.

To start, let me specify some of the use cases we've identified for parameterized tests:

• Testing several different functions which share a set of special cases to test at. For instance, I may be testing a bunch of functions that find properties of water vapor for a given temperature/pressure. So I would do a test for a nominal case above freezing, a nominal case below freezing, at a melting point, at a boiling point, at the triple point, very high/low temperature, and very high/low pressure. Each function should have well-defined behavior for each case, so temperature/pressure make good parameters here.

• Testing multiple implementations of the same function, in cases where the implementation can be chosen at run time. For instance, if I have a wrapper function that can use multiple system-specific back ends, but it presents the same interface to hide this complexity. Or say I have multiple ways of approximating the same curve, but I want to ensure that they all give good answers within some region.

• Testing the same implementation of the same function, but with some numerical details changed. For instance, changing the decomposition of a problem over MPI tasks or threads.

So, right now, for a minimal parameterized test case, I have to define the test parameter and test case types, the test case constructor, getParameters, getParameterString, and runMethod.

What I would like to see is a scenario where I could define just the types, getParameters, and getParameterString. The test case's constructor and runMethod always seem to end up as copy/pasted boilerplate, which means that it's probably no real benefit for the user to specify them.

My thought would be that this could be addressed by making some minor changes to the existing type, and expanding the smarts of pFUnitParser.py. But I don't have a perfect conception of how that would happen.

One way to simplify the test case constructor would be if the test case actually had an AbstractTestParameter component, so that individual fields wouldn't have to be moved by the constructor as they are now. That would let you have a default test case constructor, without making the user define one. The only problem there is that the test methods still need to see the individual data fields; they can't use an AbstractTestParameter object without figuring out what subclass it actually is.

(If this isn't making any sense, maybe I can find a clearer way to explain this.)

Additionally, it makes more sense to me for getParameterString to be a method on the parameter object, rather than the test case. But this is not really that important, and I don't want to belabor the point if implementing this complicates other changes.

Finally, I'm still not sure how MPI test cases play into all this. Even now, they don't really deal with their parameters in the same way that user-defined parameterized cases do. So maybe they are an orthogonal concern? Or MpiTestCase could be re-designed to look more like a user-defined parameterized case would.

[Tom Clune]

Hi Sean,

Thank you for your interest in this, and I certainly hope you can help us make major simplifications to this facility within pFUnit. Note that I have identified a few things that I think the preprocessor could do beyond the current implementation, but they are not enough to make a big difference to new users. Please see below for some more specific comments.

On Dec 16, 2013, at 4:31 PM, Sean Patrick Santos quantheory@users.sf.net wrote:

I'd just like to weigh in on what I would like to see.

To start, let me specify some of the use cases we've identified for parameterized tests:

Testing several different functions which share a set of special cases to test at. For instance, I may be testing a bunch of functions that find properties of water vapor for a given temperature/pressure. So I would do a test for a nominal case above freezing, a nominal case below freezing, at a melting point, at a boiling point, at the triple point, very high/low temperature, and very high/low pressure. Each function should have well-defined behavior for each case, so temperature/pressure make good parameters here.

Yes - this sounds very much like the type of thing that the facility was intended to support.

Testing multiple implementations of the same function, in cases where the implementation can be chosen at run time. For instance, if I have a wrapper function that can use multiple system-specific back ends, but it presents the same interface to hide this complexity. Or say I have multiple ways of approximating the same curve, but I want to ensure that they all give good answers within some region.

I'm not sure this is much different in practice to the situation above. "Implementation can be chosen at run time" must somehow be converted into a parameter that is used to set the precondition for a given test. That last bit is actually very close to the case that actually spawned parameterized tests in pfunit 1.0. It stemmed from ESMF development where I had loaned out one of my team members to help ESMF with their testing. I did not seriously expect them to use pfunit back then, but I took it as a challenge to show that a testing framework can "readily" cover such situations. Of course ParameterizedTestCase in pfunit 1.0 was more of a kludge, which ironically probably made it easier to use.

Testing the same implementation of the same function, but with some numerical details changed. For instance, changing the decomposition of a problem over MPI tasks or threads.

Definitely. Of course, if the parameter is just the number of MPI processes, then the existing @mpiTest should make this fairly easy. Or at least this is one of the major improvements in 2.0 that I've been very happy with. I hope to work on a threads solution over the holidays.

So, right now, for a minimal parameterized test case, I have to define the test parameter and test case types, the test case constructor, getParameters, getParameterString, and runMethod.

Yes. With the latter being the most annoying. BTW, getParameterString() is not essential. It is just there if you want informative messages that vary with the parameters. If the usual assert message with the values in the assertEqual() is sufficient for your purposes, this can be skipped.

What I would like to see is a scenario where I could define just the types, getParameters, and getParameterString. The test case's constructor and runMethod always seem to end up as copy/pasted boilerplate, which means that it's probably no real benefit for the user to specify them.

My thought would be that this could be addressed by making some minor changes to the existing type, and expanding the smarts of pFUnitParser.py. But I don't have a perfect conception of how that would happen.

Agreed. I just feel very guilty about relying too much on the preprocessor. But I am planning to introduce macros for these parts. Feel free to suggest syntax.

One way to simplify the test case constructor would be if the test case actually had an AbstractTestParameter component, so that individual fields wouldn't have to be moved by the constructor as they are now. That would let you have a default test case constructor, without making the user define one. The only problem there is that the test methods still need to see the individual data fields; they can't use an AbstractTestParameter object without figuring out what subclass it actually is.

Yes - most unfortunate. I'm not sure that this is the only issue with a generic constructor. But 'tis enough.
(If this isn't making any sense, maybe I can find a clearer way to explain this.)

Additionally, it makes more sense to me for getParameterString to be a method on the parameter object, rather than the test case. But this is not really that important, and I don't want to belabor the point if implementing this complicates other changes.

Good point. We would definitely want an abstract base class then so that there would be a default getParameterString() method.

Finally, I'm still not sure how MPI test cases play into all this. Even now, they don't really deal with their parameters in the same way that user-defined parameterized cases do. So maybe they are an orthogonal concern? Or MpiTestCase could be re-designed to look more like a user-defined parameterized case would.

This may require further discussion. MpiTestCase is just a particular instance of ParameterizedTestCase. However, the framework is "aware" of this special instance and various aspects of the superstructure, e.g. the run() methods, detect this case and ensure things like creating a subcommunicator and collecting messages across processors. Other operational details are hidden within the ParallelContext subclasses.

If you subclass MpiTestCase, the framework should continue to do these things for you. Possibly I have a "TYPE" somewhere there should have been a "CLASS", so we may find things that need fixing, but hopefully nothing fundamental.

Cheers,

• Tom

[feature-requests:#11] Parameterized tests - move test case methods to parameter type

Status: open
Labels: Parameters ParameterizedTests Preprocessor MPI
Created: Fri Nov 29, 2013 05:14 PM UTC by Michael L. Rilee
Last Updated: Fri Nov 29, 2013 05:14 PM UTC
Owner: Michael L. Rilee

There seems to be excessive overhead to run the same test on a bunch of lat/lon points.

Can we move some of the test case methods to the parameter type, to avoid making the user define a new test case type and new methods on it?

Question: Would doing this for user-defined parameters impact MPI tests in general?

Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/pfunit/feature-requests/11/

To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

Thomas Clune, Ph. D. Thomas.L.Clune@nasa.gov
Chief, Software Systems Support Office Code 610.3
NASA GSFC 301-286-4635
MS 610.8 B33-C128 http://ssso.gsfc.nasa.gov
Greenbelt, MD 20771

[Tom Clune]

Ouch - I'm just now seeing that the mailing list obliterated almost all the visual queues between what text was mine and what text was Sean's. Apologies for that.

• Tom

On Dec 18, 2013, at 9:40 AM, Tom Clune tom_clune@users.sf.net wrote:

Hi Sean,

Thank you for your interest in this, and I certainly hope you can help us make major simplifications to this facility within pFUnit. Note that I have identified a few things that I think the preprocessor could do beyond the current implementation, but they are not enough to make a big difference to new users. Please see below for some more specific comments.

On Dec 16, 2013, at 4:31 PM, Sean Patrick Santos quantheory@users.sf.net wrote:

I'd just like to weigh in on what I would like to see.

To start, let me specify some of the use cases we've identified for parameterized tests:

Testing several different functions which share a set of special cases to test at. For instance, I may be testing a bunch of functions that find properties of water vapor for a given temperature/pressure. So I would do a test for a nominal case above freezing, a nominal case below freezing, at a melting point, at a boiling point, at the triple point, very high/low temperature, and very high/low pressure. Each function should have well-defined behavior for each case, so temperature/pressure make good parameters here.

Yes - this sounds very much like the type of thing that the facility was intended to support.

Testing multiple implementations of the same function, in cases where the implementation can be chosen at run time. For instance, if I have a wrapper function that can use multiple system-specific back ends, but it presents the same interface to hide this complexity. Or say I have multiple ways of approximating the same curve, but I want to ensure that they all give good answers within some region.

I'm not sure this is much different in practice to the situation above. "Implementation can be chosen at run time" must somehow be converted into a parameter that is used to set the precondition for a given test. That last bit is actually very close to the case that actually spawned parameterized tests in pfunit 1.0. It stemmed from ESMF development where I had loaned out one of my team members to help ESMF with their testing. I did not seriously expect them to use pfunit back then, but I took it as a challenge to show that a testing framework can "readily" cover such situations. Of course ParameterizedTestCase in pfunit 1.0 was more of a kludge, which ironically probably made it easier to use.

Testing the same implementation of the same function, but with some numerical details changed. For instance, changing the decomposition of a problem over MPI tasks or threads.

Definitely. Of course, if the parameter is just the number of MPI processes, then the existing @mpiTest should make this fairly easy. Or at least this is one of the major improvements in 2.0 that I've been very happy with. I hope to work on a threads solution over the holidays.

So, right now, for a minimal parameterized test case, I have to define the test parameter and test case types, the test case constructor, getParameters, getParameterString, and runMethod.

Yes. With the latter being the most annoying. BTW, getParameterString() is not essential. It is just there if you want informative messages that vary with the parameters. If the usual assert message with the values in the assertEqual() is sufficient for your purposes, this can be skipped.

What I would like to see is a scenario where I could define just the types, getParameters, and getParameterString. The test case's constructor and runMethod always seem to end up as copy/pasted boilerplate, which means that it's probably no real benefit for the user to specify them.

My thought would be that this could be addressed by making some minor changes to the existing type, and expanding the smarts of pFUnitParser.py. But I don't have a perfect conception of how that would happen.

Agreed. I just feel very guilty about relying too much on the preprocessor. But I am planning to introduce macros for these parts. Feel free to suggest syntax.

One way to simplify the test case constructor would be if the test case actually had an AbstractTestParameter component, so that individual fields wouldn't have to be moved by the constructor as they are now. That would let you have a default test case constructor, without making the user define one. The only problem there is that the test methods still need to see the individual data fields; they can't use an AbstractTestParameter object without figuring out what subclass it actually is.

Yes - most unfortunate. I'm not sure that this is the only issue with a generic constructor. But 'tis enough.
(If this isn't making any sense, maybe I can find a clearer way to explain this.)

Additionally, it makes more sense to me for getParameterString to be a method on the parameter object, rather than the test case. But this is not really that important, and I don't want to belabor the point if implementing this complicates other changes.

Good point. We would definitely want an abstract base class then so that there would be a default getParameterString() method.

Finally, I'm still not sure how MPI test cases play into all this. Even now, they don't really deal with their parameters in the same way that user-defined parameterized cases do. So maybe they are an orthogonal concern? Or MpiTestCase could be re-designed to look more like a user-defined parameterized case would.

This may require further discussion. MpiTestCase is just a particular instance of ParameterizedTestCase. However, the framework is "aware" of this special instance and various aspects of the superstructure, e.g. the run() methods, detect this case and ensure things like creating a subcommunicator and collecting messages across processors. Other operational details are hidden within the ParallelContext subclasses.

If you subclass MpiTestCase, the framework should continue to do these things for you. Possibly I have a "TYPE" somewhere there should have been a "CLASS", so we may find things that need fixing, but hopefully nothing fundamental.

Cheers,

Tom
[feature-requests:#11] Parameterized tests - move test case methods to parameter type

Status: open
Labels: Parameters ParameterizedTests Preprocessor MPI
Created: Fri Nov 29, 2013 05:14 PM UTC by Michael L. Rilee
Last Updated: Fri Nov 29, 2013 05:14 PM UTC
Owner: Michael L. Rilee

There seems to be excessive overhead to run the same test on a bunch of lat/lon points.

Can we move some of the test case methods to the parameter type, to avoid making the user define a new test case type and new methods on it?

Question: Would doing this for user-defined parameters impact MPI tests in general?

Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/pfunit/feature-requests/11/

To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

Thomas Clune, Ph. D. Thomas.L.Clune@nasa.gov
Chief, Software Systems Support Office Code 610.3
NASA GSFC 301-286-4635
MS 610.8 B33-C128 http://ssso.gsfc.nasa.gov
Greenbelt, MD 20771

[feature-requests:#11] Parameterized tests - move test case methods to parameter type

Status: open
Labels: Parameters ParameterizedTests Preprocessor MPI
Created: Fri Nov 29, 2013 05:14 PM UTC by Michael L. Rilee
Last Updated: Mon Dec 16, 2013 09:31 PM UTC
Owner: Michael L. Rilee

There seems to be excessive overhead to run the same test on a bunch of lat/lon points.

Can we move some of the test case methods to the parameter type, to avoid making the user define a new test case type and new methods on it?

Question: Would doing this for user-defined parameters impact MPI tests in general?

Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/pfunit/feature-requests/11/

To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

Thomas Clune, Ph. D. Thomas.L.Clune@nasa.gov
Chief, Software Systems Support Office Code 610.3
NASA GSFC 301-286-4635
MS 610.8 B33-C128 http://ssso.gsfc.nasa.gov
Greenbelt, MD 20771

[Michael L. Rilee]

Note: our current example illustrating parameterized tests is a bit trivial. We need a better motivated example.

[Michael L. Rilee]

The driving issue seems to have abated since the release of 3.0.