Re: [matplotlib-devel] image module questions

[John H. <jdh...@ac...>]

>>>>> "Perry" == Perry Greenfield <pe...@st...> writes:

    >> Create an image extension built around agg (not backend_agg).
    >> This would be a free standing extension not tied to any of the
    >> backends with the responsibility of loading image data from a
    >> variety of sources into a pixel buffer, and resizing it to a
    >> desired pixel size (dependent on the axes window) with
    >> customizable interpolation.

    Perry> I guess I'm confused by terminology. What do you intend
    Perry> "backend" to mean for images. A common interface for
    Perry> reading different image formats? Speaking of which...

Sorry for the confusion.  I meant that I was considering using
antigrain to load/store/scale/convert/process the pixel buffers in an
image module in the same way that PIL could be used, and that this has
nothing per se to do with backend_agg.  By "backends", I meant the
same old backends we already have, backend_gd, backend_wx, backend_ps,
etc..., each of which would implement a new method draw_image to
render the image object to its respective canvases/displays.  Whether
this image object is PIL based or Agg based under the hood is an open
question.

I hope I this clarifies rather than muddies the waters....

    Perry> But initially, reading images into arrays seems like the
    Perry> most flexible and straightforward thing to do.

Agreed - I like the idea of making the user deal with endianess, etc,
in loading the image data into arrays, and passing those to image
module.  Todd, is it reasonably straightforward to write extension
code that is Numeric/numarray neutral for rank 2 and rank 3 arrays?

    Perry> These arrays are passed to matplotlib rendering methods or
    Perry> functions and the dimensionality will tell the rendering
    Perry> engine how to interpret it. The question is how much the
    Perry> engine needs to know about the depth and representation of
    Perry> the display buffer and how much of these details are
    Perry> handled by agg (or other backends)

One thing that the rest of matplotlib tries to do is insulate as much
complexity from the backends as possible.  For example, the backends
only know one coordinate system (display) and the various figure
objects transform themselves before requesting, for example,
draw_lines or draw_text.  Likewise, the backends don't know about
Line2D objects or Rectangle objects; the only know how to draw a line
from x1, y1 to x2, y2, etc...

This suggests doing as much work as possible in the image module
itself.  For example, if the image module converted all image data to
an RGBA array of floats, this would be totally general and the
backends would only have to worry about one thing vis-a-vis images:
dumping rgba floats to the canvas.  Nothing about byte-order, RGB
versus BGR, grayscale, colormaps and so on.  Most or all of these
things could be supported in the image module: the image module scales
the image, handles the interpolation, and converts all pixel formats
to an array of rgba floats.  Then the backend takes over and renders.
An array of RGBA UInt16s would probably suffice for just about
everything, however.

The obvious potential downside here is performance and memory.  You
might be in a situation where the user passes in UInt8, the image
module converts to floats, and the backend converts back to UInt8 to
pass to display.  Those of you who deal with image data a lot: to what
extent is this a big concern?  My first reaction is that on a
reasonably modern PC, even largish images could be handled with
reasonable speed.

On the subject of PIL versus agg for the workhorse under the image
module hood: I agree with the positive points things about PIL that
you and Andrew brought up (stability, portability, wide user base,
relevant functionality already implemented).  In favor of agg I would
add

  * we're already distributing agg so negligible code bloat; PIL is
    largish and not the easiest install.  Distributing fonttools has
    been a mess that I don't want to repeat.

  * impressive suite of pixel conversion funcs, interpolators, filters
    and transforms built in

  * easy and efficient integration with backend_agg, which the GUIs
    are converging around.  

  * numeric/numarray support from the ground up

Downsides

  * less portable - requires modern c++ compiler

  * more up front work (but most is already done in C++ and just needs
    exposing)

JDH