RE: [Matplotlib-users] large data sets and performance

[Perry G. <pe...@st...>]

John Hunter writes:
> >>>>> "Perry" == Perry Greenfield <pe...@st...> writes:
> 
>     Perry> What I was alluding to was that if a backend primitive was
>     Perry> added that allowed plotting a symbol (patch?) or point for
>     Perry> an array of points.  The base implementation would just do
>     Perry> a python loop over the single point case so there is no
>     Perry> requirement for a backend to overload this call. But it
>     Perry> could do so if it wanted to loop over all points in C.  How
>     Perry> flexible to make this is open to discussion (e.g., allowing
>     Perry> x, and y scaling factors, as arrays, for the symbol to be
>     Perry> plotted, and other attributes that may vary with point such
>     Perry> as color)
> 
> To make this work in the current design, you'll need more than a new
> backend method.
> 
[much good explanation of why...]

OK, I understand.

> My first response to this problem was to use a naive container class,
> eg Circles, and an appropriate backend method, eg, draw_circles.  In
> this case, scatter would instantiate a Circles instance with a list of
> circles.  When Circles was called to render, it would need to create a
> sequence of location data and a sequence of gcs
[...]
I'd agree that this doesn't seem worth the trouble
> 
> Much better is to implement a GraphicsContextCollection, where the
> relevant properties can be either individual elements or
> len(collection) sequences.  If a property is an element, it's
> homogeneous across the collection.  If it's len(collection), iterate
> over it.  The CircleCollection, instead of storing individual Circle
> instances as I wrote about above, stores just the location and size
> data in arrays and a single GraphicsContextCollection.
> 
> def scatter(x, y, s, c):
> 
>   collection = CircleCollection(x, y, s)
>   gc = GraphicsContextCollection()
>   gc.set_linewidth(1.0)  # a single line width
>   gc.set_foreground(c)   # a len(x) array of facecolors
>   gc.set_edgecolor('k')  # a single edgecolor
> 
>   collection.set_gc(gc)
> 
>   axes.add_collection(collection)
>   return collection
> 
> And this will be blazingly fast compared to the solution above, since,
> for example, you transform the x, y, and s coordinates as numeric
> arrays rather than individually.  And there is almost no function call
> overhead.  And as you say, if the backend doesn't implement a
> draw_circles method, the CircleCollection can just fall back on
> calling the existing methods in a loop.
> 
> Thoughts?
>         
I like the sounds of this approach even more. But I wonder if
it can be made somewhat more generic. This approach (if I read
it correctly seems to need a backend function for each shape:
perhaps only for circle?). What I was thinking was if there was a way
to pass it the vectors or path for a symbol (for very often, 
many points will share the same shape, if not all the same x,y
scale). Here the circle is a bit of a special case compared to
crosses, error bars triangles and other symbols that are usually
made up of a few straight lines. In these cases you could pass
the backend the context collection along with the shape
(and perhaps some scaling info if that isn't part of the context).
That way only one backend routine is needed. 

I suppose circle and other curved items could be handled with 
A bezier type call. 

But perhaps I still misunderstand.

Thanks for your very detailed response.

Perry