matplotlib-devel

[matplotlib-devel] new simplified tests; location of test baseline images

[Andrew S. <str...@as...>]

Today I committed to svn a simplified testing infrastructure, which I've
committed to matplotlib/testing/*. A few sample tests are in
matplotlib/tests/*. I also wrote some docs, which are now in the
developer coding guide. See that (
http://matplotlib.svn.sourceforge.net/viewvc/matplotlib/trunk/matplotlib/doc/devel/coding_guide.rst?revision=7654&view=markup
, starting at line 675) for information about how to write tests.

Now, I have a question. As currently written, the baseline (a.k.a.
expected) images are stored in the sample_data directory and
matplotlib.cbook.get_sample_data() downloads the images. However, I
suspect that the Mac Sage buildslave doesn't like to download stuff 
while not in an interactive login. (Remember the initial problems
running tests on that machine?) That's probably a good indication that
we probably don't want to require network access to run the tests. So,
the next question is whether we want to install baseline images with
standard MPL installs so that any user can run the full test suite? That
would be my preference, as it would be the simplest and most robust to
implement, but it comes at the cost of using additional disk space.
Otherwise, I'm open to suggestions.

(John, to confirm my suspicions about the network access issue, could
you ssh into the Sage Mac and run test/_buildbot_mac_sage.sh by hand to
see if that eliminates the bus error we're getting when run from the
buildbot?)

-Andrew

Re: [matplotlib-devel] new simplified tests; location of test baseline images

[Andrew S. <str...@as...>]

Andrew Straw wrote:
> Today I committed to svn a simplified testing infrastructure, which I've
> committed to matplotlib/testing/*. A few sample tests are in
> matplotlib/tests/*. I also wrote some docs, which are now in the
> developer coding guide. See that (
> http://matplotlib.svn.sourceforge.net/viewvc/matplotlib/trunk/matplotlib/doc/devel/coding_guide.rst?revision=7654&view=markup
> , starting at line 675) for information about how to write tests.
>   
I should also say that I plan to migrate the existing tests to this new
simplified architecture. You can get your testing feet wet by joining
the fun.

-Andrew

Re: [matplotlib-devel] new simplified tests; location of test baseline images

[John H. <jd...@gm...>]

On Sun, Sep 6, 2009 at 1:00 AM, Andrew Straw<str...@as...> wrote:
> Andrew Straw wrote:
>> Today I committed to svn a simplified testing infrastructure, which I've
>> committed to matplotlib/testing/*. A few sample tests are in
>> matplotlib/tests/*. I also wrote some docs, which are now in the
>> developer coding guide. See that (
>> http://matplotlib.svn.sourceforge.net/viewvc/matplotlib/trunk/matplotlib/doc/devel/coding_guide.rst?revision=7654&view=markup
>> , starting at line 675) for information about how to write tests.
>>
> I should also say that I plan to migrate the existing tests to this new
> simplified architecture. You can get your testing feet wet by joining
> the fun.

I should have some time today to play as well.  One thing I would like
to do is to continue the clean up on naming conventions to make them
compliant with the coding guide.  Thanks for your efforts so far on
this -- one thing left to do here that I can see is to rename the
modules to test_axes.py rather than TestAxes.py, etc..., and to finish
renaming the methods which use the wrong convention, eg
TestAxes.TestAxes.tearDown should be test_axes.TestAxes.tear_down
(module_lower_under.ClassMixedUpper.method_lower_under).

When I get some time to work later I'll ping you to make sure we
aren't working on the same thing at the same time.

JDH

Re: [matplotlib-devel] new simplified tests; location of test baseline images

[John H. <jd...@gm...>]

On Sun, Sep 6, 2009 at 12:58 AM, Andrew Straw<str...@as...> wrote:
> Today I committed to svn a simplified testing infrastructure, which I've
> committed to matplotlib/testing/*. A few sample tests are in
> matplotlib/tests/*. I also wrote some docs, which are now in the
> developer coding guide. See that (
> http://matplotlib.svn.sourceforge.net/viewvc/matplotlib/trunk/matplotlib/doc/devel/coding_guide.rst?revision=7654&view=markup
> , starting at line 675) for information about how to write tests.
>
> Now, I have a question. As currently written, the baseline (a.k.a.
> expected) images are stored in the sample_data directory and
> matplotlib.cbook.get_sample_data() downloads the images. However, I
> suspect that the Mac Sage buildslave doesn't like to download stuff
> while not in an interactive login. (Remember the initial problems
> running tests on that machine?) That's probably a good indication that
> we probably don't want to require network access to run the tests. So,
> the next question is whether we want to install baseline images with
> standard MPL installs so that any user can run the full test suite? That
> would be my preference, as it would be the simplest and most robust to
> implement, but it comes at the cost of using additional disk space.
> Otherwise, I'm open to suggestions.
>
> (John, to confirm my suspicions about the network access issue, could
> you ssh into the Sage Mac and run test/_buildbot_mac_sage.sh by hand to
> see if that eliminates the bus error we're getting when run from the
> buildbot?)

I was able to run the buildbot mac script when logged into sage with::

    Test plotting empty axes with dates along one axis. ... KNOWNFAIL:
Fails due to SF bug 2850075
    Test Some formatter and ticker issues. ... ok
    Test the 'is_string_like cookbook' function. ... ok
    Test DateFormatter ... ok
    Test RRuleLocator ... ok
    Basic Annotations ... ok
    Polar Plot Annotations ... ok
    Polar Coordinate Annotations ... ok
    Test the fill method with unitized-data. ... ok
    Test constant xy data. ... ok
    Test numpy shaped data. ... ok
    Test single-point date plots. ... ok
    Test single-point plots. ... ok
    Test polar plots with unitized data. ... ok
    Test polar plots where data crosses 0 degrees. ... ok
    Test the axhspan method with Epochs. ... ok
    Test the axvspan method with Epochs. ... ok
    very simple example test ... ok
    very simple example test that should fail ... KNOWNFAIL: Test known to fail
    matplotlib.tests.test_transforms.test_Affine2D_from_values ... ok
    matplotlib.tests.test_spines.test_spines_axes_positions ... ok

    ----------------------------------------------------------------------
    Ran 21 tests in 8.342s

    OK (KNOWNFAIL=2)

I am not sure I want to distribute the baseline images with the main
mpl distribution, but I am open to considering it.  As the number of
tests and baseline images grows, which hopefully will happen soon,
this could potentially become large -- the reason I added
get_sampledata in the first place was to get the distribution size
down.  We could add support to get_sampledata to use an environment
variable for the cache directory.  Then I could do an svn checkout of
the sample_data tree and svn up this dir in my buildbot script.  If I
point the sample data environment var to this directory, it would have
the latest data for the buildbot and would not need to make an http
request (it is odd though that svn checkouts on the sage buildbot work
fine even when not interactively logged in but http requests
apparently do not).

If you think the sample_data w/ support for local svn checkouts is the
way to go for the baseline data and images, let me know.  I would like
to utilize a subdir, eg, sample_data/baseline, if we go this route, to
keep the top-level directory a bit cleaner for user data.   We could
also release a tarball of the sample_data/baseline directory with each
release, so people who want to untar, set the environment var and test
could do so.

I am not sure this is the right approach by any means, just putting it
up for consideration.  One disadvantage of the sample_data approach is
that it would probably work well with HEAD but not with releases,
because as the baseline images changes, it becomes difficult to test
existing releases against it, which may be assuming a prior baseline.
This is why I mentioned releasing the baseline images too, but it does
raise the barrier for doing tests.

JDH

Re: [matplotlib-devel] new simplified tests; location of test baseline images

[Andrew S. <str...@as...>]

John Hunter wrote:
> I was able to run the buildbot mac script when logged into sage with::
>   
So it seems the bus error on Mac is due to networking (DNS lookups)
being broken in non-interactive logins. This is a pain for the
get_sample_data() approach. (Although I suspect we could work around it
by giving the IP address of the svn repo just like we did for the main
MPL checkout. In this case, however, the IP address would be hardcoded
into cbook.)
> I am not sure I want to distribute the baseline images with the main
> mpl distribution, but I am open to considering it.  As the number of
> tests and baseline images grows, which hopefully will happen soon,
> this could potentially become large -- the reason I added
> get_sampledata in the first place was to get the distribution size
> down.  We could add support to get_sampledata to use an environment
> variable for the cache directory.  Then I could do an svn checkout of
> the sample_data tree and svn up this dir in my buildbot script.  If I
> point the sample data environment var to this directory, it would have
> the latest data for the buildbot and would not need to make an http
> request (it is odd though that svn checkouts on the sage buildbot work
> fine even when not interactively logged in but http requests
> apparently do not).
>   

After letting the implications settle a bit, I think I'm in favor of
baseline images living in the matplotlib svn trunk so that they're
always in sync with the test scripts and available to those who have
done svn checkouts. Another important consideration is that this
approach plays will with branching the repo.

Just because they'd be in the main repository directory, though, doesn't
mean that we have to ship source or binaries with them in place --
that's a decision that could be discussed when release day gets closer.
Many of these images will be .pngs with large regions of white, so
they're relatively small files. But, I agree, hopefully there will be a
lot of tests and thus a lot of images, which will add up. As far as the
linux packaging goes -- the packagers can decide how to ship their own
binaries, but I'm sure they'd appreciate a mechanism for shipping the
test image data separately from the main binary package. This could
cause us to come up with a nice mechanism which we enable when building
Mac and Windows binary packages. As for the source packages, I think I'd
tend toward including the test images for more or less the same reasons
as including them in the svn trunk.

Also, we could set it up such that we skip image_comparison tests if the
baseline images weren't available (or simply not compare the results).

> If you think the sample_data w/ support for local svn checkouts is the
> way to go for the baseline data and images, let me know.  I would like
> to utilize a subdir, eg, sample_data/baseline, if we go this route, to
> keep the top-level directory a bit cleaner for user data.   We could
> also release a tarball of the sample_data/baseline directory with each
> release, so people who want to untar, set the environment var and test
> could do so.
>   
OK, I will move them to a new subdir if we decide to keep the
sample_data approach. I thought I read a preference to keep sample_data
flat, and I wasn't sure about Windows path names.
> I am not sure this is the right approach by any means, just putting it
> up for consideration.  One disadvantage of the sample_data approach is
> that it would probably work well with HEAD but not with releases,
> because as the baseline images changes, it becomes difficult to test
> existing releases against it, which may be assuming a prior baseline.
> This is why I mentioned releasing the baseline images too, but it does
> raise the barrier for doing tests.
>   
Likewise, I'm not sure my idea is best, either, but I think it plays
best with version control, which IMO is a substantial benefit.

> I should have some time today to play as well.  One thing I would like
> to do is to continue the clean up on naming conventions to make them
> compliant with the coding guide.  Thanks for your efforts so far on
> this -- one thing left to do here that I can see is to rename the
> modules to test_axes.py rather than TestAxes.py, etc..., and to finish
> renaming the methods which use the wrong convention, eg
> TestAxes.TestAxes.tearDown should be test_axes.TestAxes.tear_down
> (module_lower_under.ClassMixedUpper.method_lower_under).
I think we should forget about subclassing unittest.TestCase and simply
use flat functions as our tests. In particular, we should drop setUp and
tearDown altogether (which IIRC have to be named what they are because
they're a subclass of unittest.TestCase and override baseclass methods).
If we need any setup and teardown functionality, let's make a new
decorator to support it.

Then, I think the tests in test/test_matplotlib/TestAxes.py should go
into lib/matplotlib/tests/test_axes.py and each test should be it's own
function at module level, such as test_empty_datetime().

-Andrew

Re: [matplotlib-devel] new simplified tests; location of test baseline images

[John H. <jd...@gm...>]

On Sun, Sep 6, 2009 at 11:17 AM, Andrew Straw<str...@as...> wrote:

> After letting the implications settle a bit, I think I'm in favor of
> baseline images living in the matplotlib svn trunk so that they're
> always in sync with the test scripts and available to those who have
> done svn checkouts. Another important consideration is that this
> approach plays will with branching the repo.

OK, we can put them in the trunk alongside the tests, anywhere you
think they should go, and we just won't ship them by default in all
releases.  We can decide on that at release time, but I had gotten
trapped into thinking that if it lived alongside the tests we would
need to ship the baseline images, too, but of course we do not.  It
will make svn checkouts a little slower, but this is a rare event and
worth the cost.

I am working on this stuff now and am near a solution for the empty
datetime bug which is cleaner and more general.  I'll populate tests
for this stuff so just let me know where to put the baselines.

We should probably have a quick call if you are working on this stuff
now -- 773-955-7825

Re: [matplotlib-devel] new simplified tests; location of test baseline images

[John H. <jd...@gm...>]

On Sun, Sep 6, 2009 at 3:44 PM, John Hunter<jd...@gm...> wrote:
>
> I am working on this stuff now and am near a solution for the empty
> datetime bug which is cleaner and more general.  I'll populate tests
> for this stuff so just let me know where to put the baselines.

Hey Andrew -- I finally got the new date tests working.  Just run
lib/matplotlib/tests/test_dates.py to generate the new baseline images
and add them wherever you want them to go.  Or if you'd rather I add
them create the baseline images dirs in svn and I'll add/commit them.
But I'll be offline for a bit so go ahead and add them if you want.

JDH

Re: [matplotlib-devel] new simplified tests; location of test baseline images

[Andrew S. <str...@as...>]

John Hunter wrote:
> On Sun, Sep 6, 2009 at 3:44 PM, John Hunter<jd...@gm...> wrote:
>   
>> I am working on this stuff now and am near a solution for the empty
>> datetime bug which is cleaner and more general.  I'll populate tests
>> for this stuff so just let me know where to put the baselines.
>>     
>
> Hey Andrew -- I finally got the new date tests working.  Just run
> lib/matplotlib/tests/test_dates.py to generate the new baseline images
> and add them wherever you want them to go.  Or if you'd rather I add
> them create the baseline images dirs in svn and I'll add/commit them.
> But I'll be offline for a bit so go ahead and add them if you want.
>   
OK, great, we now have several examples of tests in matplotlib.tests,
and they seem to be passing on the buildbot slaves now.

I added the images to lib/matplotlib/tests/baseline_images.

I also removed ".png" from the filenames in anticipation of auto-backend
selection and comparison of other formats. (However, that's not
something I plan to implement any time soon.)

We now have 26 tests running on the buildbots and 11 with the plain
"import matplotlib; matplotlib.test()", which tests only the new
simplified tests. That means there are 15 tests left in the original
testing infrastructure that remain to be ported over, which I don't
think will be too hard now that I ported over JPL's units stuff into
matplotlib.testing.jpl_units and test_matplotlib/TestAxes.py to
matplotlib.tests.test_axes.py.

-Andrew