Menu

#47 Implement reproducible tests

0.2.0
done
nobody
None
nobody
2017-06-21
2014-07-21
No

Occasionally, some tests fail because the random parameters give results that exceed the error bounds. Thinking about this a bit suggests that we do not actually use random numbers for testing "real" randomness, but just for getting arbitrary input data without accidental symmetries or such. Hence, there is no real need for randomizing the input with every run.

As a solution, tests should always run with the same random numbers, so that results are strictly reproducible and we have no tests that occasionally fail for a bad rng.

Things to do:

  • Fix the random number for each test. The random numbers must be independent of how you run the test runner (which tests are run, which order etc.) and should not change if you add or remove tests to/from a test suite.
  • The tensor library does not allow access to its random number seed. As a consequence, add a custom function for getting a random tensor and use it in the Wavepacket tests.
  • Check the error bounds in the tests and try to make them as strict as possible

Further Todo's if they are then still relevant:

Some tests have essentially the same code as the library code, but when you run them, the output differs. Best example: operator/OperationsTest, here especially the multiplication. In principle, the tests do the same thing as the product / sum operators, but they give different results. Figure out why, and try to fix this, to allow stricter error bounds for the low-level tests.

Discussion

  • Ulf Lorenz

    Ulf Lorenz - 2017-04-23
    • Milestone: 0.1.4 --> Backlog
     
  • Ulf Lorenz

    Ulf Lorenz - 2017-06-01
    • status: open --> assigned
    • assigned_to: Ulf Lorenz
    • Milestone: Backlog --> 0.2.0
     
  • Ulf Lorenz

    Ulf Lorenz - 2017-06-21

    Two annotations:

    • The product / sum operator problem was not fixed; the error bound is 1e-12, which should be fine enough, especially since we cannot guarantee that we get the same machine code anyway.
    • The random number seed consists of two components: a fixed part defined as a static variable and a testcase-specific part that are the first six characters of each file's sha1sum. The exact value does not really matter here, though, it should just be kind of random. The fixed seed can be modified to test different configurations. That is not really optimal, but accessing the command line parameters or so seems more involved. This can still be added if systematic variations become a goal.
     
  • Ulf Lorenz

    Ulf Lorenz - 2017-06-21
    • status: assigned --> done
    • assigned_to: Ulf Lorenz --> nobody
     

Log in to post a comment.

Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.