#334 Test coverage is incomplete

normal
open
nobody
None
4
2015-08-16
2013-07-25
Derek Lamb
No

At a BARE MINIMUM, we should test each top-level PDL function. Currently there are no tests for which_both, one2nd, and probably others. We should use Devel::Cover to flush out the test suite. That module also allows one to test for statement and branch coverage. More complete statement and branch coverage would be a worthy secondary goal, and would probably help flush out many problems. I think full top-level coverage is critical for the next PDL release, but statement and branch coverage would probably go under "feature request" and would be less critical, but certainly helpful.

Related

Bugs: #334

Discussion

  • Chris Marshall

    Chris Marshall - 2013-07-28

    Also, the tests should all be migrated to use Test::More at a minimum and eventually to use Test::PDL as well. Adding test names to the subtests makes it much easier to track down where any failures are happening. E.g., for some t/image2d.t failures in the last subtest, it would have been nice to know that the failures were in the polyfill and polyfillv subtests without having to read the t/image2d.t file to deduce.

     
  • Chris Marshall

    Chris Marshall - 2013-07-28

    Problem reported to MooX::Types::MooseLike at rt.cpan.org.

     
  • William Parker

    William Parker - 2013-08-07

    I'm new around here; my guess it that this should go here, but please
    correct me if I'm wrong. I'm interested in contributing, and working on the
    test suite seems like a good place for someone without extensive knowledge of
    the code base to do so.

    A few questions first, though. Is the test suite being run in the build
    process or independently? The ExtUtils::MakeMaker docs say it does so, but
    deliberately creating a test that would fail didn't have any effect on the build
    process. I can't find anything in them about how the tests are supposed to be
    run, just that they are. I confirmed that the test was a failure by manually
    running Test::Harness on all .t files in the t directory. More generally, does
    anyone have guidance on what precisely would be useful, logistics, etc.? Or if
    I'd just be totally over my head, please tell me. Basically, I have a good bit
    of experience with Perl, and I have a decent understanding of the math involved
    (senior undergrad math major), but I don't pretend to be a programming expert
    or professional applied mathematician.

     
  • Derek Lamb

    Derek Lamb - 2013-08-07

    Yes, this is an excellent way for somebody with some Perl experience but new to PDL to contribute, and it is a good way to learn the code base. Welcome!

    The idea here is that the test suite should exercise as much of the PDL code base as possible. The test suite is usually run (as 'make test') after PDL is built ('make') into the blib directory, and before it is installed ('make install'). So a failed test shouldn't make the build fail, just the test suite. I recommend installing Devel::Cover from cpan, then checking out the latest PDL git, building at least the core of it (don't worry about external dependencies such as GSL, FFTW, PLPlot, OpenGL (aka POGL) for the moment), then running Devel::Cover's test metrics ('cover -test'). Once that is done the cover_db/coverage.html document will show you what code was tested.

    There is a lot of stuff there, and maybe too much for a beginner. I have been working on blib/lib/PDL/Basic.pm. I suggest you look at blib/lib/PDL/Primitive.pm or blib/lib/PDL/Ufunc.pm to start. You can click on the "sub" column and it will show you what functions (in red) are not exercised by the test suite (for example, in Primitive: conv1d, fibonacci, one2nd, which_both, & intersect--note that there are two primitive test scripts, primitive.t and primitive2.t).

    I recommend subscribing to the pdl-porters list (see http://pdl.perl.org/?page=mailing-lists) and coordinating test suite development there. You can either submit patches to the list, or to the SF patches ticket category. Do one of those and we'll be happy to give you commit access to the main git repo. Thank you for your offer to help!

    Derek

    On Aug 6, 2013, at 7:32 PM, William Parker wrote:

    I'm new around here; my guess it that this should go here, but please
    correct me if I'm wrong. I'm interested in contributing, and working on the
    test suite seems like a good place for someone without extensive knowledge of
    the code base to do so.

    A few questions first, though. Is the test suite being run in the build
    process or independently? The ExtUtils::MakeMaker docs say it does so, but
    deliberately creating a test that would fail didn't have any effect on the build
    process. I can't find anything in them about how the tests are supposed to be
    run, just that they are. I confirmed that the test was a failure by manually
    running Test::Harness on all .t files in the t directory. More generally, does
    anyone have guidance on what precisely would be useful, logistics, etc.? Or if
    I'd just be totally over my head, please tell me. Basically, I have a good bit
    of experience with Perl, and I have a decent understanding of the math involved
    (senior undergrad math major), but I don't pretend to be a programming expert
    or professional applied mathematician.

    [bugs:#334] Test coverage is incomplete

    Status: open
    Created: Thu Jul 25, 2013 10:27 PM UTC by Derek Lamb
    Last Updated: Sun Jul 28, 2013 08:01 PM UTC
    Owner: nobody

    At a BARE MINIMUM, we should test each top-level PDL function. Currently there are no tests for which_both, one2nd, and probably others. We should use Devel::Cover to flush out the test suite. That module also allows one to test for statement and branch coverage. More complete statement and branch coverage would be a worthy secondary goal, and would probably help flush out many problems. I think full top-level coverage is critical for the next PDL release, but statement and branch coverage would probably go under "feature request" and would be less critical, but certainly helpful.

    Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/pdl/bugs/334/

    To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

     

    Related

    Bugs: #334

  • Chris Marshall

    Chris Marshall - 2013-08-07

    In addition to test coverage, the existing test suite is
    inconsistent in implementation and usage. If they
    were refactored to use current PDL standards, it would
    be a big help for improved coverage and would offer
    you a chance to see how things have been done in
    the past.

    • use Test::More for testing. See the TODO file in the
      PDL distribution for a list of which tests were missing
      what---although a few have been converted since that
      list was last updated.

    • use File::Temp for tempfile() generation since the
      existing $PDL::Config{TEMPDIR} with roll your own
      is clunky and not always robust.

    • PDL::Test has recently been developed to improve
      PDL test checking but no one has had time to add
      that in.

    As always, if you have any suggestions, questions,
    or whatever.... please feel free to bring up in discussion
    on the pdl-porters mailing list.

    --Chris

    On Wed, Aug 7, 2013 at 1:13 PM, Derek Lamb lambd@users.sf.net wrote:

    Yes, this is an excellent way for somebody with some Perl experience but new
    to PDL to contribute, and it is a good way to learn the code base. Welcome!

    The idea here is that the test suite should exercise as much of the PDL code
    base as possible. The test suite is usually run (as 'make test') after PDL
    is built ('make') into the blib directory, and before it is installed ('make
    install'). So a failed test shouldn't make the build fail, just the test
    suite. I recommend installing Devel::Cover from cpan, then checking out the
    latest PDL git, building at least the core of it (don't worry about external
    dependencies such as GSL, FFTW, PLPlot, OpenGL (aka POGL) for the moment),
    then running Devel::Cover's test metrics ('cover -test'). Once that is done
    the cover_db/coverage.html document will show you what code was tested.

    There is a lot of stuff there, and maybe too much for a beginner. I have
    been working on blib/lib/PDL/Basic.pm. I suggest you look at
    blib/lib/PDL/Primitive.pm or blib/lib/PDL/Ufunc.pm to start. You can click
    on the "sub" column and it will show you what functions (in red) are not
    exercised by the test suite (for example, in Primitive: conv1d, fibonacci,
    one2nd, which_both, & intersect--note that there are two primitive test
    scripts, primitive.t and primitive2.t).

    I recommend subscribing to the pdl-porters list (see
    http://pdl.perl.org/?page=mailing-lists) and coordinating test suite
    development there. You can either submit patches to the list, or to the SF
    patches ticket category. Do one of those and we'll be happy to give you
    commit access to the main git repo. Thank you for your offer to help!

    Derek

    On Aug 6, 2013, at 7:32 PM, William Parker wrote:

    I'm new around here; my guess it that this should go here, but please
    correct me if I'm wrong. I'm interested in contributing, and working on the
    test suite seems like a good place for someone without extensive knowledge
    of
    the code base to do so.

    A few questions first, though. Is the test suite being run in the build
    process or independently? The ExtUtils::MakeMaker docs say it does so, but
    deliberately creating a test that would fail didn't have any effect on the
    build
    process. I can't find anything in them about how the tests are supposed to
    be
    run, just that they are. I confirmed that the test was a failure by manually
    running Test::Harness on all .t files in the t directory. More generally,
    does
    anyone have guidance on what precisely would be useful, logistics, etc.? Or
    if
    I'd just be totally over my head, please tell me. Basically, I have a good
    bit
    of experience with Perl, and I have a decent understanding of the math
    involved
    (senior undergrad math major), but I don't pretend to be a programming
    expert
    or professional applied mathematician.

    [bugs:#334] Test coverage is incomplete

    Status: open
    Created: Thu Jul 25, 2013 10:27 PM UTC by Derek Lamb
    Last Updated: Sun Jul 28, 2013 08:01 PM UTC
    Owner: nobody

    At a BARE MINIMUM, we should test each top-level PDL function. Currently
    there are no tests for which_both, one2nd, and probably others. We should
    use Devel::Cover to flush out the test suite. That module also allows one to
    test for statement and branch coverage. More complete statement and branch
    coverage would be a worthy secondary goal, and would probably help flush out
    many problems. I think full top-level coverage is critical for the next PDL
    release, but statement and branch coverage would probably go under "feature
    request" and would be less critical, but certainly helpful.

    Sent from sourceforge.net because you indicated interest in
    https://sourceforge.net/p/pdl/bugs/334/

    To unsubscribe from further messages, please visit
    https://sourceforge.net/auth/subscriptions/


    [bugs:#334] Test coverage is incomplete

    Status: open
    Created: Thu Jul 25, 2013 10:27 PM UTC by Derek Lamb
    Last Updated: Wed Aug 07, 2013 01:32 AM UTC
    Owner: nobody

    At a BARE MINIMUM, we should test each top-level PDL function. Currently
    there are no tests for which_both, one2nd, and probably others. We should
    use Devel::Cover to flush out the test suite. That module also allows one to
    test for statement and branch coverage. More complete statement and branch
    coverage would be a worthy secondary goal, and would probably help flush out
    many problems. I think full top-level coverage is critical for the next PDL
    release, but statement and branch coverage would probably go under "feature
    request" and would be less critical, but certainly helpful.


    Sent from sourceforge.net because you indicated interest in
    https://sourceforge.net/p/pdl/bugs/334/

    To unsubscribe from further messages, please visit
    https://sourceforge.net/auth/subscriptions/

     

    Related

    Bugs: #334

  • Chris Marshall

    Chris Marshall - 2015-02-22
    • Priority: 7 --> 4
     
  • Chris Marshall

    Chris Marshall - 2015-08-16
    • Group: critical --> normal
     

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks