Re: [Gmod-ajax] rendered larger chromsome

[Ian H. <ih...@be...>]

Mitch,

> I've put up Dmel 3R here:
> http://genome.biowiki.org/gbrowse/dmel51/prototype_gbrowse.html

cool -- looks good (a bit slow to load but maybe others are looking at 
it right now...? could also be the ginormous html)

> It rendered in less than 8 hours on one CPU, and the tiles and HTML take 
> up around 13 gigabytes on disk.  Memory usage maxed out at 342 MB.  The 
> 3R chromosome arm is the largest Dmel one at 28 Mb, or about 20 times 
> the size of chromosome 4.  So to do human we just have to scale another 
> order of magnitude; it's CPU bound and we've got a fair amount of CPU to 
> throw at it, but if memory usage grows linearly with the amount of 
> sequence then we may start running into problems. 

as a web service we can easily afford a few 4Gb-RAM machines or even 
64Gb. disk space, similarly, is cheap - for us. as an app that can be 
downloaded and installed by anyone (i.e. like the existing GBrowse), 
you're right: this narrows the user base a bit. still i'm not too 
worried about it yet.

> We could do some more 
> work on rendering on demand, and that would help, but you've still got 
> to have all the features in memory at the same time to do layout, so we 
> may not be able to reduce max memory usage very much that way.  Maybe we 
> can partition the layout job if there are empty regions of the chromosome.

again.. not worried yet. maybe we can start thinking idly about 
bounded-space layout algorithms.

> One problem with the current approach is that the HTML files get quite 
> large for the lower (more zoomed-out) zoom levels.  For the Genes track 
> the entire_landmark HTML file is 1.6 megabytes; for CDS it's 7.7 
> megabytes.  The HTML for entire_landmark for all tracks totals 24 
> megabytes, which is how much you would have to download to actually view 
> the entire_landmark zoom level.  Maybe we should have some kind of limit 
> above which we turn off the HTML.  Also, if we had a way of lazily 
> loading feature information then the HTML could be a lot smaller.

mmm, this is a bit more scary. it does make lazy feature-info-loading 
look appealing.

> With a large number of features the browser starts to slow down; for me 
> this starts to be noticeable around the 1Mbp zoom level on Windows.  On 
> other platforms firefox starts to bog down at higher (less dense) zoom 
> levels.  Again, if we turn off the html above some feature density 
> threshold this should be less of a problem.

i suspect that other people (probably some on this list, or the other 
GMOD lists) have thought quite hard about this kind of issue (collapsing 
features, density thresholds, etc) and we'd benefit from consulting with 
them.

ian


> 
> Mitch
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys-and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Gmod-ajax mailing list
> Gmo...@li...
> https://lists.sourceforge.net/lists/listinfo/gmod-ajax