Re: [Gmod-ajax] storing primitives in memory

[Mitchell S. <mi...@ar...>]

Well, I've been spending some quality time with D.
melanogaster chromosome 4 and perl's DProf, and
I've learned a few things:

* the TiledImage::AUTOLOAD routine was taking up a
significant proportion of the time, especially on
primitive-intensive tracks like the yeast_chr1
translation tracks at full zoom.  I'm working on
reimplementing the accessors using closures or
Class::Struct, and reimplementing the GD primitive
interception with closures like this (in
TiledImage):

    foreach my $sub (keys %intercept) {
        no strict "refs";
        *$sub = sub  {
            my ($self, @args) = @_;
            my @originalArgs = @args;

            # get bounding box
            my @bb;
            @bb =  $self->getBoundingBox ($sub, @args);
            ...

* on some tracks, GD::filledRectangle was taking the
lion's share of the time (96% on one run that I
measured for a track that was using the "transcript"
glyph).  I looked at the gd c code, and
gdImageFilledRectangle calls the retail 
gdImageSetPixel function (which is non-trivial) in
a nested loop.  Since AFAIK we're not doing any
antialiasing or other compositing, it seems like
this could be improved a lot.  For the immediate
future the advice is to avoid rectangle based glyphs.

* for the less primitive-intensive tracks (like gene
or mRNA) at high zoom levels, the gridlines are big
memory users (for the in-memory primitive storage
approach).  Rendering Dmel chrom 4 mRNA at zoom 1
without gridlines reduced peak memory usage (RSS as
measured by ps) from 653 megabytes to 169 megabytes.
If we can find a way to avoid rendering the gridlines
(like using PNG transparency as mentioned on the
wiki) or if we can find a way to avoid storing the
gridline primitives then I don't think memory usage
will be a problem for these kinds of tracks.

* for the primitive-intensive tracks like the
translation and dna tracks, I'm hoping that rendering
smaller tile ranges will make the jobs fit in RAM.
I'm not sure if we can avoid generating all
primitives when we're not rendering the whole track,
but in any case we should be able to avoid storing
all those primitives.

If we can make all those improvements (not a huge if
IMO; call it a medium-small if) then I think
pre-rendering is the way to go.

Unless there are other reasons than speed and memory
usage to do render-on-demand?  I've been assuming
that rendering on demand would be a nontrivial amount
of work, not to mention that it's yet another CGI
sitting on top of a database (not sure how well it'll
scale), but admittedly I haven't thought all the way
through it.

If we don't need to render on demand, then AFAICT
we don't need the primitive database (?), and then
I'll feel comfortable producing a patch that takes
out the database bits.  It's not like I have some
kind of vendetta against the database bits, it's
just that making database/memory a run-time option
is that much extra work.

Mitch