[Matplotlib-users] How to do million data-point plots with Matplotlib?

[David S. <dav...@gm...>]

I have been working on a program that uses Matplotlib to plot data
consisting of around one million points.  Sometimes the plots succeed but
often I get an exception: OverFlowError: Agg rendering complexity exceeded.

I can make this message go away by plotting the data in "chunks" as
illustrated in the demo code below.  However, the extra code is a chore
which I don't think should be necessary - I hope the developers will
be able to fix this issue sometime soon.  I know that the development
version has some modifications to addressing this issue.  I wonder if it is
expected to make the problem go away?

By the way, this plot takes about 30 seconds to render on my I7 2600k.
The main program reaches the show() statement quickly and prints
"Done plotting?".   Then I see that the program reaches 100% usage
on one CPU core (4 real, 8 virtual on the 2600k) until the plot is
displayed.  I wonder if there is any way to persuade Matplotlib to run
some of the chunks in parallel so as to use more CPU cores?

Plotting something other than random data, the plots run faster and
the maximum chunk size is smaller.  The maximum chunk size
also depends on the plot size - it is smaller for larger plots.  I am
wondering if I could use this to plot course and fine versions of the
plots.  The course plot is zoomed in version of the small-sized raster.
That would be better than decimation as all the points would at least
be there.

Thanks in advance,

David

--------------------------- start code ---------------------------------
## Demo program shows how to "chunk" plots to avoid the exception:
##
##    OverflowError: Agg rendering complexity exceeded.
##    Consider downsampling or decimating your data.
##
## David Smith December 2011.

from pylab import *
import numpy as np

nPts=600100
x = np.random.rand(nPts)
y = np.random.rand(nPts)

## This seems to always succeed if Npts <= 20000, but fails
## for Npts > 30000.  For points between, it sometimes succeeds
## and sometimes fails.
figure(1)
plot (x, y)

## Chunking the plot alway succeeds.
figure(2)
chunk_size=20000
iStarts=range(x.size/chunk_size)
for iStart in iStarts:
    print "Plotting chunk starting at %d\n" % iStart
    plot(x[iStart:iStart+chunk_size], y[iStart:iStart+chunk_size], '-b')

left_overs = nPts % chunk_size
if left_overs > 0:
    print "Leftovers %d points\n" % left_overs
    plot(x[-left_overs-1:], y[-left_overs-1:], '-r')

print "done plotting?"
show()
---------------------------------- end code ------------------------
Please don't reply to this post "It is rediculous to plot 1 million points on
screen".  I am routinely capturing million-point traces from oscilloscopes and
other test equipment and to I need to be able to spot features in the
data (glitches if you will) that may not show up plotting decimated data.
I can then zoom the plot to inspect these features.