Re: [matplotlib-devel] Automated test harness discussion...

[John H. <jd...@gm...>]

On Tue, Dec 16, 2008 at 11:12 AM, Drain, Theodore R
<the...@jp...> wrote:

> - Embed a font with the tests to eliminate font server differences (no experience with this so I'm not sure how hard this would be).  We could even create a dummy font that just has black squares for each character - it still tests that everything is drawn in the correct place and runs properly and eliminates subtle character differences.

Since we ship our own fonts, it is pretty easy to test against a known
set.  Just set up the rc to use Vera, and cm* or stix* for math.

> - Create a testing backend that records the drawing commands as a set of meta-data like (draw red line from point 1 to point 2).  The test case then checks that the proper commands were issued by the test script.  This eliminates drawing completely.  A nice  comparison suite would allow loose comparisons like "make sure a vertical line was drawn from (10,20) to (30,40) with a pixel slop of 2 pixels.

I think this idea has promise.

I'll also add that we can do *a lot* more with simple API tests that
do not look at the output but at least make sure the inputs, in all
their variety, are at least accepted.  backend_driver does this to an
extent, but I will be adding nose tests for any new features I add.
Eg, for the recent markevery, which can be None, an integer or a
length 2 tuple, I added this simple test to unit/nose_tests


def test_markevery():
    x, y = np.random.rand(2, 100)

    # check marker only plot
    fig = plt.figure()
    ax = fig.add_subplot(111)
    ax.plot(x, y, 'o', label='default')
    ax.plot(x, y, 'd', markevery=None, label='mark all')
    ax.plot(x, y, 's', markevery=10, label='mark every 10')
    ax.plot(x, y, '+', markevery=(5, 20), label='mark every 5 starting at 10')
    ax.legend()
    fig.canvas.draw()
    plt.close(fig)

    # check line/marker combos
    fig = plt.figure()
    ax = fig.add_subplot(111)
    ax.plot(x, y, '-o', label='default')
    ax.plot(x, y, '-d', markevery=None, label='mark all')
    ax.plot(x, y, '-s', markevery=10, label='mark every 10')
    ax.plot(x, y, '-+', markevery=(5, 20), label='mark every 5 starting at 10')
    ax.legend()
    fig.canvas.draw()
    plt.close(fig)

Having well defined tests that heavily exercise the frontend API would
be a significant step forward, and the harder question of output
comparison could be added in.

JDH