matplotlib-users

[Matplotlib-users] GPF on XP

[Robert L. <ro...@le...>]

When attempting to generate a larger number of graph sets (i.e. 3 graphs of 
similar style over different data ranges), I'm intermittently getting a GPF on 
XP in na_backend_agg.pyd according to the report that M$ offers to send to itself.

It is repeatable in one sense, in that if I restart the graph generation from 
the beginning it will fail at the same set, but if skip the first set of graphs 
it doesn't produce one additional set and then die, it dies at 15 (5 sets) 
earlier. I can restart from any of the sets where it failed and it will continue 
on for some random number before GPF'ing again - anything from 9 thru to 150 
graphs or so.

If I use Numeric (23.7, the latest) it is a lot worse - meaning fewer sets 
before failure. Also matplotlib 0.72 was a lot worse with either Numeric and 
numarray.

The environment is Python 2.4, XP (sp2), matplotlib 0.72.1, numarray 1.2.2, 
Numeric 23.7 with the data coming from Postgres 8.0 via SQLObject and psycopg.

I'm not sure of the best way to proceed from here - is this a known issue or 
related to one or should I attempt to produce a standalone test that causes the 
problem?

Robert

Re: [Matplotlib-users] GPF on XP

[John H. <jdh...@ac...>]

>>>>> "Robert" == Robert Leftwich <ro...@le...> writes:

    Robert> When attempting to generate a larger number of graph sets
    Robert> (i.e. 3 graphs of similar style over different data
    Robert> ranges), I'm intermittently getting a GPF on XP in
    Robert> na_backend_agg.pyd according to the report that M$ offers
    Robert> to send to itself.

Ouch.

    Robert> It is repeatable in one sense, in that if I restart the
    Robert> graph generation from the beginning it will fail at the
    Robert> same set, but if skip the first set of graphs it doesn't
    Robert> produce one additional set and then die, it dies at 15 (5
    Robert> sets) earlier. I can restart from any of the sets where it
    Robert> failed and it will continue on for some random number
    Robert> before GPF'ing again - anything from 9 thru to 150 graphs
    Robert> or so.

Repeatable is good.  Standalone much better.  So you are running the
pure Agg backend (no GUI?).  It would help to post the output of 

  c:> python myscript.py --verbose-helpful

    Robert> If I use Numeric (23.7, the latest) it is a lot worse -
    Robert> meaning fewer sets before failure. Also matplotlib 0.72
    Robert> was a lot worse with either Numeric and numarray.

It probably won't happen with 0.71 and this would be worth testing.  I
did a bunch of changes in backend agg in 0.72, including using the
numeric API rather than the sequence protocol.  If you want to verify
that the problem was introduced in 0.72 (which will help me narrow
down the possible culprits) remove site-packages/matplotlib and then
install 0.71 and see if the crash disappears.

    Robert> I'm not sure of the best way to proceed from here - is
    Robert> this a known issue or related to one or should I attempt
    Robert> to produce a standalone test that causes the problem?

That would help immensely.

One thing I can do is send you a debug build of mpl for windows that
has a bunch of extra diagnostic information turned on.  This might
help isolate which function is causing the problem.  But if you can
get a standalone script, that would be most efficient.

Thanks,

Re: [Matplotlib-users] GPF on XP

[Robert L. <ro...@le...>]

John Hunter wrote:
> 
> Repeatable is good.  Standalone much better.  So you are running the
> pure Agg backend (no GUI?).  It would help to post the output of 
> 
>   c:> python myscript.py --verbose-helpful

Correct, no gui, verbose output:

matplotlib data path C:\Python24\share\matplotlib
loaded rc file C:\Python24\share\matplotlib\.matplotlibrc
matplotlib version 0.72.1
verbose.level helpful
interactive is False
platform is win32
numerix numarray 1.2.2
font search path ['C:\\Python24\\share\\matplotlib']
loaded ttfcache file c:\home\robert\.ttffont.cache
backend TkAgg version 8.4

> It probably won't happen with 0.71 and this would be worth testing.

Everything works without error on 0.71, using either numarray or Numeric.

> But if you can
> get a standalone script, that would be most efficient.

Working on it now.

Robert

Re: [Matplotlib-users] GPF on XP

[John H. <jdh...@ac...>]

>>>>> "Robert" == Robert Leftwich <ro...@le...> writes:

    Robert> John Hunter wrote:
    >> Repeatable is good.  Standalone much better.  So you are
    >> running the pure Agg backend (no GUI?).  It would help to post
    >> the output of c:> python myscript.py --verbose-helpful

    Robert> Correct, no gui, verbose output:

You say no gui, but the verbose report says tkagg:

    backend TkAgg version 8.4

Do you see the problem when using agg alone?

    >> But if you can get a standalone script, that would be most
    >> efficient.

    Robert> Working on it now.

Looking forward to it :-)

JDH

Re: [Matplotlib-users] GPF on XP

[Robert L. <ro...@le...>]

John Hunter wrote:
 >>>>>>"Robert" == Robert Leftwich <ro...@le...> writes:
 >
 >     Robert> Correct, no gui, verbose output:
 >
 > You say no gui, but the verbose report says tkagg:
 >
 >     backend TkAgg version 8.4

My bad, forgot to change the rc when re-installing 0.72.

 >
 > Do you see the problem when using agg alone?

Yes, in fact it seems to be worse.

 >
 >     >> But if you can get a standalone script, that would be most
 >     >> efficient.
 >
 >     Robert> Working on it now.
 >
 > Looking forward to it :-)

I sent it direct to you, rather than everyone on the list.

Robert

Re: [Matplotlib-users] GPF on XP

[John H. <jdh...@ac...>]

>>>>> "Robert" == Robert Leftwich <ro...@le...> writes:

    John> But if you can get a standalone script, that would be most
    John> efficient.

    Robert> I sent it direct to you, rather than everyone on the list.

OK, the good news is that my first hunch was correct.  The fact that
the minimal script (off-list) needed more iterations than the full,
and that the number of iterations on my system before the crash was
different than yours indicated to me that it was a memory problem.
matplotlib uses a fair number of cyclic references (figure refers to
canvas and canvas refers to figure -- this one was actually added in
0.72 which is where I suspect the culprit is).

In the pylab interface, I call gc.collect in the destroy method of
each figure manager to force the garbage collector to clean up.  In
your script, which doesn't use the pylab interface, you need to
manually add this call yourself.  I'll make it a FAQ because it is
fairly important, and add a link or the faq in the agg_oo example.

import gc


    def graphAll(self):
        for i in range(100):
            print i
            self.graphRSIDailyBenchmark()
            gc.collect()

Should cure what ails you!

python gurus -- would it be advisable for me to insert a gc.collect()
somewhere in the matplotlib OO hierarchy since I know cyclic
references are a problem?  Eg in the call to the FigureCanvas init
function?

JDH

Re: [Matplotlib-users] GPF on XP

[Robert L. <ro...@le...>]

John Hunter wrote:
> 
>             gc.collect()
> 
> Should cure what ails you!
> 

The good news is that it is a huge improvement, but the bad news is that I'm 
still getting a GPF, just a lot less often :-( Try bumping the minimal test loop 
up to 5k, it failed at 3057 for me.

Robert

Re: [Matplotlib-users] GPF on XP

[John H. <jdh...@ac...>]

>>>>> "Robert" == Robert Leftwich <ro...@le...> writes:

    Robert> John Hunter wrote:
    >> gc.collect() Should cure what ails you!
    >> 

    Robert> The good news is that it is a huge improvement, but the
    Robert> bad news is that I'm still getting a GPF, just a lot less
    Robert> often :-( Try bumping the minimal test loop up to 5k, it
    Robert> failed at 3057 for me.

Bet you had to wait a while for that one.  Maybe you should use
the full test script.  At lease you'll fail faster  :-)

matplotlib does some caching in various places for efficiency which
will slowly eat memory.  We need to add some automated means to clear
this cache but we don't have it yet.

What happens if you comment out this line in text.py

    self.cached[key] = ret

and this line in backend_agg.py

  _fontd[key] = font

We have a linux/unix specific script for testing for memory leaks in
the mpl src distro unit/memleak_hawaii3.py.  You may want to adapt
something like this for windows xp so we can get firm numbers on how
much is leaking per figure.  See also
http://matplotlib.sourceforge.net/faq.html#LEAKS.  Todd Miller knows a
clever way of getting python to report how may object references it
has a hold of, but I can't remember the magic command right now.

With matplotlib CVS on linux, that script is reporting no detectable
leak.  But your script may be exposing a leak not covered by that
one.

JDH

Re: [Matplotlib-users] GPF on XP

[Robert L. <ro...@le...>]

John Hunter wrote:
>>>>>>"Robert" == Robert Leftwich <ro...@le...> writes:
> 
> 
>     Robert> The good news is that it is a huge improvement, but the
>     Robert> bad news is that I'm still getting a GPF, just a lot less
>     Robert> often :-( Try bumping the minimal test loop up to 5k, it
>     Robert> failed at 3057 for me.
> 
> Bet you had to wait a while for that one. 

Not on the new laptop!

 > Maybe you should use
> the full test script.  At lease you'll fail faster  :-)

Yep, using the real data, it fails pretty quickly.

> 
> 
> What happens if you comment out this line in text.py
> 
>     self.cached[key] = ret
> 
> and this line in backend_agg.py
> 
>   _fontd[key] = font

It's worse, the minimal test fails at 4!, but the real data takes 180 or so.

> 
> We have a linux/unix specific script for testing for memory leaks in
> the mpl src distro unit/memleak_hawaii3.py.  You may want to adapt
> something like this for windows xp so we can get firm numbers on how
> much is leaking per figure.  

I'll attempt this when time permits, possibly late today, but sometime later 
this week is more likely.

Robert