Re: [Matplotlib-users] large data sets and performance

[Peter G. <pgr...@ge...>]

Thanks for the prompt answers.

>Bad terminology: for x I meant sorted (monotonic) and for y the ideal
>cases is smooth and not varying too rapidly.  Try the lod feature and
>see if it works for you.
>
Although the data I'm playing with right now is monotonic (in x), I 
cannot assume that this will always be the case, and need an efficient 
solutions for all situations.

the 'lod' option in:
l = plot(arange(10000), arange(20000,30000))    #dummy data.. 10,000 pairs
set(l, 'lod', True)
option does not work for me. It's still roughly 1000 points/second

>    >> econdly, the standard gdmodule will iterate over the x, y
>    >> values in a python loop in gd.py.  This is slow for lines with
>    >> lots of points.  I have a patched gdmodule that I can send you
>    >> (provide platform info) that moves this step to the extension
>    >> module.  Potentially a very big win.
>
>    Peter> Yes, that would be great!  System info:
>
>Here is the link
>
>http://nitace.bsd.uchicago.edu:8080/files/share/gdmodule-0.52b.tar.gz
>
>You must also upgrade gd to 2.0.22 (alas 2.0.21 is obsolete!) since I
>needed the latest version to get this sucker ported to win32.
>  
>
Installed gd 2.0.22, and gdmodule-0.52b (from the link you provided) but 
there is no change in the times. Not sure why.. I should probably notice 
at least a little difference.

>I don't know how to write a GTK pixbuf to stdout.  I inquired on the
>pygtk mailing list, so perhaps we'll learn something soon.  To use GTK
>in Xvfb, make sure you have Xvfb (X virtual frame buffer) installed
>(/usr/X11R6/bin/Xvfb).  There is probably an RPM, but I don't
>remember.
>
[...]
Installed Xvfb, and ran the little script you included. It complained about:
File 
"/usr/lib/python2.2/site-packages/matplotlib/backends/backend_gtk.py", 
line 528, in _quit_after_print_xvfb
    if len(manager.drawingArea._printQued): break
AttributeError: FigureManagerGTK instance has no attribute 'drawingArea'

Didn't inquire further because in my case it is crucial to have stdout 
output.. I have to be able to pipe these plots to cgi scrips.

If you have any other ideas, please let me know. Can anyone else tell me 
what kind of performance they're getting doing these 10k, 50k, 100k plots?

Best,
Peter