Re: [Matplotlib-users] large data sets and performance

[Peter G. <pgr...@ge...>]

Perry:
Currently using connected line plots, but do not want to limit myself in 
any way when it comes to presenting data. I am certain that at one 
point, I will use every plot available in the matplotlib arsenal. On a 
3.2Ghz P4 with 2GB RAM get ~90 seconds for a 100,000 data set, ~50 
seconds for 50,000 and ~9 seconds for a 10,000 (sort of linear). This is 
way to long for my purposes. I was hoping more for ~5 seconds for 
100,000 points.

John:

I routinely plot data sets this large.  500,000 data points is a

>I routinely plot data sets this large.  500,000 data points is a
>typical 10 seconds of EEG, which is the application that led me to
>  
>
That sounds good!

>If your xdata are sorted, ie like time, the following
>
>  l = plot(blah, blah)
>  set(l, 'lod', True)
>
>could be a big win.
>
>Whether this is appropriate or not depends on the data set of course,
>whether it is continuous, and so on.  Can you describe your dataset in
>more detail, because I would like to add whatever optimizations are
>appropriate -- if others can pipe in here too that would help.
>  
>
Will mostly be plotting time Vs value(time) but in certain cases will 
need plots of other data, and therefore have to look at the worst case 
scenario.
Not exactly sure what you mean by "continuous" since all are descrete 
data points. The data may not be smooth (could have misbehaving sensors 
giving garbage) and jump all over the place.

>econdly, the standard gdmodule will iterate over the x, y values in a
>python loop in gd.py.  This is slow for lines with lots of points.  I
>have a patched gdmodule that I can send you (provide platform info)
>that moves this step to the extension module.  Potentially a very big
>win.
>  
>
Yes, that would be great!
System info:

OS: RedHat9 ( kernel 2.4.20)

gcc version from running 'gcc -v':
Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/3.2.2/specs
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man 
--infodir=/usr/share/info --enable-shared --enable-threads=posix 
--disable-checking --with-system-zlib --enable-__cxa_atexit 
--host=i386-redhat-linux
Thread model: posix
gcc version 3.2.2 20030222 (Red Hat Linux 3.2.2-5)

Python: Python 2.2.2 (#1, Feb 24 2003, 19:13:11)

matplotlig: matplotlib-0.50e

gdpython: 0.51 (with modified _gdmodule.c)

gd: gd-2.0.21

>Another possibility: change backends.  The GTK backend is
>significantly faster than GD.  If you want to work off line (ie, draw
>to image only and not display to screen ) and are on a linux box, you
>can do this with GTK and Xvfb.  I'll give you instructions if
>interested.  In the next release of matplotlib, there will be a libart
>paint backend (cross platform) that may be faster than GD.  I'm
>working on an Agg backend that should be considerably faster than all
>the other backends since it does everything in extension code -- we'll
>see 
>
Yes I am only planning to work offline. Want to be able to pipe the 
output images to stdout. I am looking for the fastest solution possible.

Thanks again.
Peter