[Sebastian]

On Miros latest blog post I posted an off-topic comment. I thought it would be best to move the discussion here:

http://embeddedgurus.com/state-space/2013/04/dual-targeting-and-agile-prototyping-of-embedded-software-on-windows/#comment-39464

I don’t think that a unit for testing is a state handler in this case. A natural unit test is an active object. So testing consists of (1) bringing the AO to a specific state (setup), (2) injecting event(s) to it and (3) verifying the behavior.

This approach seems not to be that different: you test a Statehandler with an event. After that you have to check the state of the statechart...?

With this approach I see some problems I dont know how to solve:
- I can see if the event lead to a Q_TRAN (the state changed) but i don't know if it was handled or ignored...?
- I do have to run the framework while running the tests, with my approach I could test without using the framework.
- "verifying the behavior" - If posting to another event queue or dispatching to a orthogonal component or some other framework functions is part of the behaviour, it would be easy to mock the framework functions when not testing under the framework. This seems hard to me when using the framework.

And what is still an unclear point to me: do I really have to test the whole topology of the statechart, as said before this seems tedious. And: in practice how would you proceed? Do a "real" test DRIVEN development, that is:
0. run the test for the first behaviour you want to test an let it fail
1. draw the simplest statechart possible that solves the test (eg: A-->B)
2. next test ...
3. next state ...

I really like this approach with "conventional" programming, but with QM I rather paint a "picture" that makes sense and most of the time this is not possible with the most simple statechart.

[Richard]

Sebastian

These are some of the thoughts I have had as well when thinking about TDD and QP. TDD would suggest that you test the individual functions, but that would be very difficult without the framework because the "functions" are states and would need to transition. A lot of the functionality of the state is handled by the framework. I'm not saying it would be impossible, but it is definitely non trivial. I like the idea of creating a framework mock, but once again this would require a lot of effort and I'm still not sure it would be worth it. Miro's suggestion of testing objects would require the use of the framework in the testing, which seems to me to violate the objectives of TDD.

Another problem I have with using TDD with a QP framework is that TDD wants you to develop the system as guided by the tests. One of the great benefits of a UML framework is the fact that by the time you have finished drawing you UML statechart, you design is 80% done, and crafting the code from the statechart is fairly easy. I believe this is what you meant when you said, "but with QM I rather paint a "picture" that makes sense."

I think there must be a way to get them to work well together, but it is not obvious to me right now. I guess I haven't really added much to the discussion, but I wanted to let you know that I am also interested in finding a way to effectively use them together.

[Richard]

If you did test just the functions, or states, then you could create fakes for the framework calls like Q_TRAN to make sure it was called, or let the actual macro be used and create fakes for the state and then test the state of the system just after the return and also check the return value to see if what happened was what was expected. The problem I see with this is the fact that the state action is usually a small part of what happens when a transition happens. The framework does most of the work in the transition, which as I understand TDD, should not be tested with the current function under test. Does that mean that you also have to run tests on the framework to make sure it is handling everything correctly? Where is the separation for testing? I think it should be at the function level, but I am new to TDD and so could certainly be wrong. I guess I just like the idea of testing only one function at a time rather than testing a function and all it's framework interactions at the same time. If the framework is tested separately, can we just assume that it works correctly and doesn't need to be tested along with the objects?

[Sebastian]

TDD would suggest that you test the individual functions, but that would be very difficult without the framework because the "functions" are states and would need to transition

If you only test a QStatehandler its simple:
A QStatehandler just a
- method of a statechart (so we need some dummy statechart object)
- taking an QEvt argument
- returning int
- having a possible side effect: it changes the temp-attribute of the statechart object if Q_TRAN or Q_SUPER are returned

Say you have a statehandler_to_test() which you expect to Q_TRAN() to statehandler_target() if you inject test_evt. In unity you could do:

ret = statehandler_to_test(&statechart_dummy, test_evt);
TEST_ASSERT_EQUAL_PTR(statehandler_target, statechart_dummy.temp);
TEST_ASSERT_EQUAL_UINT8(Q_RET_TRAN, ret);

A lot of the functionality of the state is handled by the framework
I like the idea of creating a framework mock, but once again this would require a lot of effort

If you want to include in the test all the stuff that happens during a transition (all ENTRY/EXIT/INIT handlers) you need the framework or at least the QEP or at least you'd end up reimplementing it if you mock it.
But as I understand TDD I've to agree with you, this would violate its objectives.
But: if you don't care about the stuff that happens during a transition but only test a statehandler, you'd only need to test its behaviour which would require mocks of at least
- QACTIVE_POST
- QXsm_defer

One of the great benefits of a UML framework is the fact that by the time you have finished drawing you UML statechart, you design is 80% done

Exactly, I also can't bring this together. And as said before, I also find it tedious (and most likely useless) to test the topology I've created with QM.

If the framework is tested separately, can we just assume that it works correctly and doesn't need to be tested along with the objects?

When using TDD you don't test the LIBs you use.

[Richard]

It does seem that testing the individual functions, whether state handlers or helpers, would be the easiest and best way to use TDD with a state based system. I think that I would also be inclined to use the testing part of TDD, but use the the design part of the framework, namely the UML statecharts, to design the system. I can see how an Active Object would be a good "unit" to test, but it carries a lot of baggage with it. Specifically, how do you test all the dependencies of a transition? Even when you get to the target state, there might still be initial transitions that would then have to be taken. Do you create spy's for all these sub-states or use the actual code? Like you said, you don't test the LIBs you use, but it does get you to you destination, and if you are testing the AO as a unit, you need to insure that the destination you arrive at is the right one.

[Richard]

Having tried for a while, I am a little frustrated by trying to get TDD and Active Objects (AO) to work together. One of the main problems is that AO are so encapsulated that testing the states of an AO means breaking that encapsulation to let the test have access to the states. Otherwise, you would have to let use the dispatcher to set the AO up for testing. I wouldn't even be so bothered by breaking encapsulation for testing if there was a way to easily un-break it for production code, but that means that the production code is now different that the tested code, and can no longer be tested. This encapsulation is part of what is so great about AO, but makes using TDD with them really hard. If you only test the AO as a module, then there is a LOT going on inside the AO that is never really tested. The beauty and difficulty of an AO is the fact that it's interface is just an event buffer. There is not much there to test unless the AO posts to another AO or something similar. There is just so much going on inside that is unreachable without breaking open the AO. If any else has had any ideas on how to do this whithout branching the code, I would love to hear about it.

[Quantum Leaps]

Hi Richard,
First, I suppose that the encapsulation is only a problem in QP/C++, because in QP/C there is no way of enforcing "privateness" of a struct member.

So, if we are talking about QP/C++, the issues here should be quite generic to doing TDD on any C++ code. I mean, there is always tension between encapsulation and the need for "white-box" tests seeing inside.

So, specifically to getting access to the QMsm.m_state member (inherited subsequently in QHsm, QActive and QMActive classes), how about adding a friendship to a specific class or function, such as friend class Test;. The Test class would then be able to inspect the m_state member. Would that be a step in the right direction for you?

--MMS

[Panopticon]

In addition to what Miro said, can I ask - would QSPY tracing be an option? QSPY will let you observe all the internal details - message posting, state transitions, even changes of application-specific data/variables (application-specific trace records).

True, QSPY uses memory and CPU, and it changes timing, and it's testing not the identical code that you'd ship, but IMO it's about as close as you can get.

Without knowing more about your constraints, it's hard to offer more specific tactics than that.