Re: [myhdl-list] Pythonic Test frameworks

[Christopher F. <chr...@gm...>]

On 11/8/11 7:04 PM, Bob Cunningham wrote:
> On 11/07/2011 08:07 PM, Christopher Felton wrote:
>> On 11/7/11 7:21 PM, Bob Cunningham wrote:
>>> FWIW, py.test seems to be the way MyHDL itself is going...
>>>
>>> -BobC
>> ...
>>
>> The following, http://bit.ly/sc3vrI, is a brief mention of py.test used
>> instead of unittest (from a couple years ago) from the myhdl-list.
>>
>> But I do believe py.test is limited.  If you are ok writing with only
>> simple asserts, I believe py.test is ok.  But unittest supports more
>> advanced features, automatically verify raised exceptions (continue if
>> raised fail if not).
>>
>> I guess an argument against py.test, is that it is another package that
>> needs to be installed.  Where as, unittest is part of the standard
>> Python library, hmmm.
>>
>> Regards,
>> Chris
>
> Well, MyHDL is, itself, "another package that needs to be installed"!
>
> See the end of this page: http://www.myhdl.org/doku.php/cookbook:stopwatch

Thanks for the reminder, this is a good commentary on py.test versus 
unittest.  This is a good read for anyone trying to determine which test 
framework to use.

>
> And py.test does handle *expected* exceptions: Unexpected exceptions will crash the test with a suitable dump.  See http://pytest.org/latest/getting-started.html#asserting-that-a-certain-exception-is-raised
>

I came across similar information while refreshing on py.test.  It does 
not seem as limited as I first thought.  Also, the setup (and teardown) 
have been enhance.  The "setup_method" is not used anymore but rather 
the funcarg.

> FWIW, py.test is faster and easier, and doesn't much care about the structure of your MyHDL code, but it has limitations.  Unittest is more powerful, but is harder to use.
>
I believe you mean faster to develop, faster to come up to speed.  Not 
necessarily faster execution.

> As a true newbie to both digital design and MyHDL, I've been coding and testing in the smallest increments possible, with tiny blocks glued together.  For this specific development path, py.test seems ideal.  At this point, I have no idea if it will work as well with more complex designs.
>
I think it will scale nicely with complex designs.  But only time and 
experience will prove if it does.

> However, I am quite biased: In my embedded/real-time software systems I often used "programming by contract" (PBC), which is extensively assert-based (to validate the contracts), so py.test seems quite natural to me.  It is also a form of 'white-box' testing.
>
> The complementary approach, test-driven design (TDD), puts more burden on the test harness, which means the test tools must be more powerful and sophisticated.  This is generally a form of 'black-box' testing.
>
> In the software world, the two test methods work well together in practice:  If you use PBC at the low level, the higher-level tests can be simpler.
>
> Has anyone tried to see if py.test and unittest can play nice together (at different levels of abstraction) within a single MyHDL project?
No I have not, I guess in theory you could have both.  I am not sure why 
you would.  I think once you go through the trouble of using unittest 
might as well stay in that framework.  But I guess, if you started with 
py.test, defined a bunch of tests.  Then found something that was not 
supported nicely in py.test, unittest could be added separately.  I 
think they would, essentially, be separate but testing the same entity.

Regards,
Chris