I am a member of the team responsible for the development of a commercial computational electromagnetic solution kernel, and since the majority of our development is done in Fortran, I started looking at pfUnit as a possible unit testing framework. Up until now, my investigations have been more on a personal basis, but I had a sit-down with my manager last week, and tried to sell him on the idea of unit testing to improve our code quality, and to augment our validation process.
Since the code has been under development for around 23 years now, there is obviously quite a bit of legacy baggage which will need to be addressed, but I have started working through "Working Effectively with Legacy Code", and feel confident that, given time, we could make some in-roads. One thing that concerns me is my manager's skepticism with regards to unit testing in terms of value added to the project. I have a feeling that many of the other developers, all of who have been with the team much longer than myself, will share his feelings.
Now, despite his skepticism, he has granted me time to investigate the use of unit tests and test driven development in our processes, and even though I do not have too much experience myself, I feel motivated enough to learn, and feel quite strongly that these will prove to be valuable -- at least in the long run.
My questions then:
Do you have any advice for me, both in terms of highlighting the merits (and possible caveats) of unit testing to my manager and senior developers?
How do I make an objective decision when considering whether such methods should be employed in our development process (I don't want a "when all you have is a hammer, everything looks like a nail" situation)?
Any resources geared towards unit testing in Fortran in general, and CSE software in particular would also be much appreciated.
Kind regards
Evan
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Sorry that I've been slow responding to your message on Monday. Just got back from travel and am still catching up. You'll have a response from me by late evening in the worst case.
I am a member of the team responsible for the development of a commercial computational electromagnetic solution kernel, and since the majority of our development is done in Fortran, I started looking at pfUnit as a possible unit testing framework. Up until now, my investigations have been more on a personal basis, but I had a sit-down with my manager last week, and tried to sell him on the idea of unit testing to improve our code quality, and to augment our validation process.
Since the code has been under development for around 23 years now, there is obviously quite a bit of legacy baggage which will need to be addressed, but I have started working through "Working Effectively with Legacy Code", and feel confident that, given time, we could make some in-roads. One thing that concerns me is my manager's skepticism with regards to unit testing in terms of value added to the project. I have a feeling that many of the other developers, all of who have been with the team much longer than myself, will share his feelings.
Now, despite his skepticism, he has granted me time to investigate the use of unit tests and test driven development in our processes, and even though I do not have too much experience myself, I feel motivated enough to learn, and feel quite strongly that these will prove to be valuable -- at least in the long run.
My questions then:
Do you have any advice for me, both in terms of highlighting the merits (and possible caveats) of unit testing to my manager and senior developers?
How do I make an objective decision when considering whether such methods should be employed in our development process (I don't want a "when all you have is a hammer, everything looks like a nail" situation)?
Any resources geared towards unit testing in Fortran in general, and CSE software in particular would also be much appreciated.
Thanks for the informative response, and don't worry about the delay. I know that I gave you a lot to think about!
Although there are still some uncertainties as to how I will approach this from my side, the following spring to mind.
You mention some difficulty in when tackling complex logic using TDD. The approach you follow seems to be something along the lines of the exploratory phase of the TriBITS software life cycle (see the attached paper). There Roscoe Bartlett advocates using this phase to gain understanding of the problem, but to throw the initial attempt away when starting "real" development.
Thanks for the reference. I've heard others describe a similar process before. And I'm probably confusing two different issues. There is the "I've not done something quite like this before and it's complicated." and then there is the "All of my procedures should be intuitively obvious when I'm done." So I struggle with whether I've simply not yet stumbled on the interfaces to make the implementation "obvious" or if text books somehow steer clear of the real-world cases that don't back up their world view on clean code. (I'm very much inspired by "Clean Code" by R. Martin.)
The mention of "wedges" is something that we can definitely pay more attention to here. We do have a tendency to modify existing subroutines and functions, instead of calling new ones. Some here argue that this is for performance reasons, but it is my feeling that there is only a small section of our code base which is really performance critical.
Then lastly, since we have a large number of common blocks to share data between various parts of the system (and also for MPI communication), I have also been thinking of wrapping these in some test class that allows for their states to be saved. The thought is then to ensure that only the expected global variables have changed at some interface. Since we do also support dynamic allocation, and some of the common block variables are Fortran pointers, I will have to perform some deep copy operation. This seems related to the "empirical" testing system that you mention.
Definitely. It is easy to test the ones that you expect to change. Much harder to check all the others that should not have - at least in any automated manner.
I will be interested to hear how the progress goes on these various developments (also the mocking support for pfUnit), and will be more than happy to offer any assistance that I can!
I'd be happy to have you as a collaborator on the proposal I'm just now starting. Might even be a bit of room for actual funding, but since the teaming is not yet quite in place I'm wary of overcommitting.
By the way, we are welcome to continue this discussion on the pfUnit boards if you like (if you paste your response there, I will do the same with mine). It may also be a good idea to add Mike to the CC list (although I didn't want your responses to influence his)!
Done. But I'm meeting with him in 30 minutes anyway …
I am a member of the team responsible for the development of a commercial computational electromagnetic solution kernel, and since the majority of our development is done in Fortran, I started looking at pfUnit as a possible unit testing framework. Up until now, my investigations have been more on a personal basis, but I had a sit-down with my manager last week, and tried to sell him on the idea of unit testing to improve our code quality, and to augment our validation process.
Since the code has been under development for around 23 years now, there is obviously quite a bit of legacy baggage which will need to be addressed, but I have started working through "Working Effectively with Legacy Code", and feel confident that, given time, we could make some in-roads. One thing that concerns me is my manager's skepticism with regards to unit testing in terms of value added to the project. I have a feeling that many of the other developers, all of who have been with the team much longer than myself, will share his feelings.
Now, despite his skepticism, he has granted me time to investigate the use of unit tests and test driven development in our processes, and even though I do not have too much experience myself, I feel motivated enough to learn, and feel quite strongly that these will prove to be valuable -- at least in the long run.
My questions then:
Do you have any advice for me, both in terms of highlighting the merits (and possible caveats) of unit testing to my manager and senior developers?
How do I make an objective decision when considering whether such methods should be employed in our development process (I don't want a "when all you have is a hammer, everything looks like a nail" situation)?
Any resources geared towards unit testing in Fortran in general, and CSE software in particular would also be much appreciated.
Apologies again for the slow response. I could tell from the first few lines that it would require a thoughtful response. Unfortunately, that meant it got bumped to last as I worked off the pile of emails.
I think I understand your concerns, and I feel rather fortunate that I've got a fair bit of flexibility in how I use my own time within NASA and a certain degree of support from management. But I know from attending other SE conferences that management is often the highest barrier to adopting advanced development practices and in particular certain types of testing. Proper testing can easily double the number of lines of code written, and it becomes rather difficult to prove that the decreased frequency of bugs, improved productivity, etc. are a net win.
My own motivation for using unit-testing is to avoid the stress I used to have when modifying large codes. I recall waking up in the middle of the night as a postdoc suddenly terrified that I had introduced bugs in our code that had not been noticed yet. It is also partly an attempt to find a unique role for myself, as at NASA I'm not directly involved in the development of any models, but rather provide support to several such projects. Unit testing gives me a soap box to talk about other software quality issues.
Although stress reduction is my personal motivation, there are a number of other benefits that I've observed. TDD, a restricted form of unit-testing, is pretty much the only way that I develop new software components at this point. Can't be used for most work with legacy code though -- more on that later. The times that I do find bugs in code that I have developed, I invariably discover that the bug is in a section where I fell off the TDD bandwagon.
In practice, I find that it is somewhat easier to predict progress and that it is far easier to pick software up 6 months or a year later. This statement is line with the fact that unit-tests (when regularly exercised) serve as a form of living documentation. Most other documentation becomes stale quite quickly, but unit tests provide other developers/users concrete scenarios for using any given interface.
I should also add some caveats as to where I have struggled with TDD:
Teaching others: Although other developers in my group have demonstrated enthusiasm for the concept, I generally find that it is difficult for them to understand how to write useful tests. It is a rather different skill set than coding, and I'm mostly dealing with mid-career scientist/engineers that have moved into IT. So depending on your team, the advantages may not be strong. It may also simply be that none of my staff have had the freedom to use TDD to develop things from scratch the way that I have. They mostly pick up projects when I've matured them to the point that I need someone else to start maintenance and fill in functionality.
Refactoring inertia: When you do need to refactor code, you also need to refactor tests. OTOH, it is hard to imagine refactoring a code without solid tests. Sometimes regression tests are sufficient, but those are very blunt instruments.
Complex logic: On those occasions where I am developing a rather unfamiliar algorithm/structure, I often struggle with using TDD. I often switch to prototyping and when I get an idea of the implementation, I switch back to TDD. In some cases I've never quite been happy with the results. This is a small fraction of the code which is written in practice though.
Scientific parameterizations: I'm still trying to find funding to make the attempt to derive a physical parameterization (e.g. a cloud component for a climate model) using TDD. The results would be quite different than existing (very ugly) implementations, but it is not clear how much more effort would be required to work in that manner.
But the elephant in the room is legacy code. It is very difficult to introduce unit-tests into legacy code. At the very least, one can attempt to avoid falling into the trap of "wedge" programming in which blocks of new logic are inserted into existing procedures. The new logic should be a separate routine and then the "wedge" should just introduce the call. But even that is of limited benefit for many things we want to do with code. (I've been hoping that refactoring in Photran would mature faster, as that would at least somewhat reduce the need for unit tests.)
Coming back around to your real question: how do you sell this to management and senior developers. In general each group needs a different argument. And you know your particular audience, so pick/emphasize from these for your own purposes:
Senior developers:
(1) Many/most experienced software developers become ardent converts to TDD after only a few days. Fortran developers might be different as many of them are primarily domain experts and only secondarily software experts. But if you have the flexibility and can bring in a consultant to provide 1 day of training and have everyone use the technique for 1 week, that would be a way to decide. And you should expect that some will never want to change the way they work.
(2) Have them be very honest about how they know their code really works. At least subconsciously they must have some reason to have splattered the various characters into the source file. I've found that many good developers are doing some form of unit testing already but simply have not done a few steps that would greatly enhance the value. E.g. they may write a simple driver and then discard it once the driver does what they expected. For those, we are really asking them to adopt some simple conventions:
Maintain their tests as permanent components in the repository.
Register their tests with the test framework.
Use asserts rather than print statements to enable crisp reports of success/failure.
(3) Appeal to professionalism.
(4) Emphasize that 100% coverage is impossible and even measurable coverage in a legacy application will take a long time. Instead focus on new code, code that is subject to frequent change, and code that is extremely high-risk.
Management:
(1) Ask them what consequences there are for bugs that are discovered after deployment. Bugs that are discovered earlier are much less expensive to correct. (I don't have the reference handy, but this a fairly well established factoid.)
(2) Could there even be liability for bugs that could have been found by standard techniques? In my world, we have argued that the intense scrutiny on climate change means that the software for climate models must focus more on quality and reliability than is typical for research codes. Medical and space flight software are all very heavily tested.
(3) Mention that testing in general is a good talking point when going to customers.
(4) Emphasize that the introduction can be as gradual as necessary and requires relatively little training for developers.
Start with regression testing if that is not already standard practice. Unit testing is then just a natural expansion of the overall testing.
Legacy code: I'm hoping to get funding for a new effort which will provide the means to create "empirical" unit/system tests. Essentially the system/technique would capture state from an executing legacy code on both sides of selected interface. Those interfaces could then be played back and the results compared to those that were saved. Hardly 100% coverage but it gives some confidence as the legacy code is being refactored. And of course if you don't need to modify any legacy code, then it also does not really need unit tests …
Hope this helps some. This was all very much off the top of my head, so possibly more will occur to me later. I'll be happy to continue the conversation if you want to tease out more nuances from any of this.
Cheers,
Tom
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I should also mention that many TDD practitioners have advocated that you simply don't tell management. Let them merely bask in the improved productivity and product quality. Can be very risky depending on how tightly controlled the developers are. But it a tactic that some claim works.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Sorry for the adding and deleting of comments, I only later realised that you had already added my response above!
With regards to:
I'd be happy to have you as a collaborator on the proposal I'm just now starting. Might even be a bit of room for actual funding, but since the teaming is not yet quite in place I'm wary of overcommitting.
I don't mind helping out here, but I would assume that it would have to be in my personal capacity (at least for the moment), as I am not exactly sure what the processes here are in this regard. As soon as I have something working on my side, I will be sure to let you know. (good luck with the funding).
I have also been working through "Clean Code", and have recommended it as reading material for the rest of the team -- with some unexpected results, I must add.
Evan
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Sorry for the adding and deleting of comments, I only later realised that you had already added my response above!
With regards to:
I'd be happy to have you as a collaborator on the proposal I'm just now starting. Might even be a bit of room for actual funding, but since the teaming is not yet quite in place I'm wary of overcommitting.
I don't mind helping out here, but I would assume that it would have to be in my personal capacity (at least for the moment), as I am not exactly sure what the processes here are in this regard. As soon as I have something working on my side, I will be sure to let you know. (good luck with the funding).
I have also been working through "Clean Code", and have recommended it as reading material for the rest of the team -- with some unexpected results, I must add.
'Unexpected' as in surprisingly positive? Or 'unexpected' as in informative debates on the merits of the advice?
On Jun 18, 2014, at 6:03 AM, Evan Lezar evanlezar@users.sf.net wrote:
Sorry for the adding and deleting of comments, I only later realised that you had already added my response above!
With regards to:
I'd be happy to have you as a collaborator on the proposal I'm just now starting. Might even be a bit of room for actual funding, but since the teaming is not yet quite in place I'm wary of overcommitting.
I don't mind helping out here, but I would assume that it would have to be in my personal capacity (at least for the moment), as I am not exactly sure what the processes here are in this regard. As soon as I have something working on my side, I will be sure to let you know. (good luck with the funding).
I have also been working through "Clean Code", and have recommended it as reading material for the rest of the team -- with some unexpected results, I must add.
'Unexpected' as in surprisingly positive? Or 'unexpected' as in informative debates on the merits of the advice?
I am a member of the team responsible for the development of a commercial computational electromagnetic solution kernel, and since the majority of our development is done in Fortran, I started looking at pfUnit as a possible unit testing framework. Up until now, my investigations have been more on a personal basis, but I had a sit-down with my manager last week, and tried to sell him on the idea of unit testing to improve our code quality, and to augment our validation process.
Since the code has been under development for around 23 years now, there is obviously quite a bit of legacy baggage which will need to be addressed, but I have started working through "Working Effectively with Legacy Code", and feel confident that, given time, we could make some in-roads. One thing that concerns me is my manager's skepticism with regards to unit testing in terms of value added to the project. I have a feeling that many of the other developers, all of who have been with the team much longer than myself, will share his feelings.
Now, despite his skepticism, he has granted me time to investigate the use of unit tests and test driven development in our processes, and even though I do not have too much experience myself, I feel motivated enough to learn, and feel quite strongly that these will prove to be valuable -- at least in the long run.
My questions then:
Any resources geared towards unit testing in Fortran in general, and CSE software in particular would also be much appreciated.
Kind regards
Evan
Hi Evan,
Sorry that I've been slow responding to your message on Monday. Just got back from travel and am still catching up. You'll have a response from me by late evening in the worst case.
Cheers,
On Jun 11, 2014, at 2:13 AM, Evan Lezar evanlezar@users.sf.net wrote:
Thomas Clune, Ph. D. Thomas.L.Clune@nasa.gov
ASTG Code 606.0
NASA GSFC 301-286-4635
MS 610.8 B33-C128
Greenbelt, MD 20771
Hi Evan,
On Jun 12, 2014, at 6:00 AM, Evan Lezar evanlezar@gmail.com wrote:
Thanks for the reference. I've heard others describe a similar process before. And I'm probably confusing two different issues. There is the "I've not done something quite like this before and it's complicated." and then there is the "All of my procedures should be intuitively obvious when I'm done." So I struggle with whether I've simply not yet stumbled on the interfaces to make the implementation "obvious" or if text books somehow steer clear of the real-world cases that don't back up their world view on clean code. (I'm very much inspired by "Clean Code" by R. Martin.)
Definitely. It is easy to test the ones that you expect to change. Much harder to check all the others that should not have - at least in any automated manner.
I'd be happy to have you as a collaborator on the proposal I'm just now starting. Might even be a bit of room for actual funding, but since the teaming is not yet quite in place I'm wary of overcommitting.
Done. But I'm meeting with him in 30 minutes anyway …
Cheers,
On Jun 11, 2014, at 2:13 AM, Evan Lezar evanlezar@users.sf.net wrote:
Thomas Clune, Ph. D. Thomas.L.Clune@nasa.gov
ASTG Code 606.0
NASA GSFC 301-286-4635
MS 610.8 B33-C128
Greenbelt, MD 20771
Hi Evan,
Apologies again for the slow response. I could tell from the first few lines that it would require a thoughtful response. Unfortunately, that meant it got bumped to last as I worked off the pile of emails.
I think I understand your concerns, and I feel rather fortunate that I've got a fair bit of flexibility in how I use my own time within NASA and a certain degree of support from management. But I know from attending other SE conferences that management is often the highest barrier to adopting advanced development practices and in particular certain types of testing. Proper testing can easily double the number of lines of code written, and it becomes rather difficult to prove that the decreased frequency of bugs, improved productivity, etc. are a net win.
My own motivation for using unit-testing is to avoid the stress I used to have when modifying large codes. I recall waking up in the middle of the night as a postdoc suddenly terrified that I had introduced bugs in our code that had not been noticed yet. It is also partly an attempt to find a unique role for myself, as at NASA I'm not directly involved in the development of any models, but rather provide support to several such projects. Unit testing gives me a soap box to talk about other software quality issues.
Although stress reduction is my personal motivation, there are a number of other benefits that I've observed. TDD, a restricted form of unit-testing, is pretty much the only way that I develop new software components at this point. Can't be used for most work with legacy code though -- more on that later. The times that I do find bugs in code that I have developed, I invariably discover that the bug is in a section where I fell off the TDD bandwagon.
In practice, I find that it is somewhat easier to predict progress and that it is far easier to pick software up 6 months or a year later. This statement is line with the fact that unit-tests (when regularly exercised) serve as a form of living documentation. Most other documentation becomes stale quite quickly, but unit tests provide other developers/users concrete scenarios for using any given interface.
I should also add some caveats as to where I have struggled with TDD:
Teaching others: Although other developers in my group have demonstrated enthusiasm for the concept, I generally find that it is difficult for them to understand how to write useful tests. It is a rather different skill set than coding, and I'm mostly dealing with mid-career scientist/engineers that have moved into IT. So depending on your team, the advantages may not be strong. It may also simply be that none of my staff have had the freedom to use TDD to develop things from scratch the way that I have. They mostly pick up projects when I've matured them to the point that I need someone else to start maintenance and fill in functionality.
Refactoring inertia: When you do need to refactor code, you also need to refactor tests. OTOH, it is hard to imagine refactoring a code without solid tests. Sometimes regression tests are sufficient, but those are very blunt instruments.
Complex logic: On those occasions where I am developing a rather unfamiliar algorithm/structure, I often struggle with using TDD. I often switch to prototyping and when I get an idea of the implementation, I switch back to TDD. In some cases I've never quite been happy with the results. This is a small fraction of the code which is written in practice though.
Scientific parameterizations: I'm still trying to find funding to make the attempt to derive a physical parameterization (e.g. a cloud component for a climate model) using TDD. The results would be quite different than existing (very ugly) implementations, but it is not clear how much more effort would be required to work in that manner.
But the elephant in the room is legacy code. It is very difficult to introduce unit-tests into legacy code. At the very least, one can attempt to avoid falling into the trap of "wedge" programming in which blocks of new logic are inserted into existing procedures. The new logic should be a separate routine and then the "wedge" should just introduce the call. But even that is of limited benefit for many things we want to do with code. (I've been hoping that refactoring in Photran would mature faster, as that would at least somewhat reduce the need for unit tests.)
Coming back around to your real question: how do you sell this to management and senior developers. In general each group needs a different argument. And you know your particular audience, so pick/emphasize from these for your own purposes:
Senior developers:
(1) Many/most experienced software developers become ardent converts to TDD after only a few days. Fortran developers might be different as many of them are primarily domain experts and only secondarily software experts. But if you have the flexibility and can bring in a consultant to provide 1 day of training and have everyone use the technique for 1 week, that would be a way to decide. And you should expect that some will never want to change the way they work.
(2) Have them be very honest about how they know their code really works. At least subconsciously they must have some reason to have splattered the various characters into the source file. I've found that many good developers are doing some form of unit testing already but simply have not done a few steps that would greatly enhance the value. E.g. they may write a simple driver and then discard it once the driver does what they expected. For those, we are really asking them to adopt some simple conventions:
Maintain their tests as permanent components in the repository.
Register their tests with the test framework.
Use asserts rather than print statements to enable crisp reports of success/failure.
(3) Appeal to professionalism.
(4) Emphasize that 100% coverage is impossible and even measurable coverage in a legacy application will take a long time. Instead focus on new code, code that is subject to frequent change, and code that is extremely high-risk.
Management:
(1) Ask them what consequences there are for bugs that are discovered after deployment. Bugs that are discovered earlier are much less expensive to correct. (I don't have the reference handy, but this a fairly well established factoid.)
(2) Could there even be liability for bugs that could have been found by standard techniques? In my world, we have argued that the intense scrutiny on climate change means that the software for climate models must focus more on quality and reliability than is typical for research codes. Medical and space flight software are all very heavily tested.
(3) Mention that testing in general is a good talking point when going to customers.
(4) Emphasize that the introduction can be as gradual as necessary and requires relatively little training for developers.
Start with regression testing if that is not already standard practice. Unit testing is then just a natural expansion of the overall testing.
Legacy code: I'm hoping to get funding for a new effort which will provide the means to create "empirical" unit/system tests. Essentially the system/technique would capture state from an executing legacy code on both sides of selected interface. Those interfaces could then be played back and the results compared to those that were saved. Hardly 100% coverage but it gives some confidence as the legacy code is being refactored. And of course if you don't need to modify any legacy code, then it also does not really need unit tests …
Hope this helps some. This was all very much off the top of my head, so possibly more will occur to me later. I'll be happy to continue the conversation if you want to tease out more nuances from any of this.
Cheers,
I should also mention that many TDD practitioners have advocated that you simply don't tell management. Let them merely bask in the improved productivity and product quality. Can be very risky depending on how tightly controlled the developers are. But it a tactic that some claim works.
Last edit: Evan Lezar 2014-06-18
Sorry for the adding and deleting of comments, I only later realised that you had already added my response above!
With regards to:
I don't mind helping out here, but I would assume that it would have to be in my personal capacity (at least for the moment), as I am not exactly sure what the processes here are in this regard. As soon as I have something working on my side, I will be sure to let you know. (good luck with the funding).
I have also been working through "Clean Code", and have recommended it as reading material for the rest of the team -- with some unexpected results, I must add.
Evan
On Jun 18, 2014, at 6:03 AM, Evan Lezar evanlezar@users.sf.net wrote:
Cheers,
Thomas Clune, Ph. D. Thomas.L.Clune@nasa.gov
Head ASTG,Code 606
NASA GSFC
MS 610.8 B33-C128
Greenbelt, MD 20771
301-286-4635
I have to apologize for how SourceForge is formatting mail that was nicely quoted on my Mac mail tool. The only line I added was:
On Jun 18, 2014, at 8:16 AM, Tom Clune tom_clune@users.sf.net wrote:
Thomas Clune, Ph. D. Thomas.L.Clune@nasa.gov
Head ASTG,Code 606
NASA GSFC
MS 610.8 B33-C128
Greenbelt, MD 20771
301-286-4635
'Unexpected' with regards to who of the developers were positive about the advice.
There is still some work to do, but I am giving a presentation to the team tomorrow where I hope to have a bit of a discussion.