gmod-ajax — JBrowse list

Re: [Gmod-ajax] AJAX GBrowse poster feedback from RECOMB

[Seth C. <sjc...@lb...>]

On Mon, 2007-04-23 at 13:28 -0700, Ian Holmes wrote:
> > learned if there's a good reason for it.  But why not Java applets?
> > The only two reasons we came up with are (1) long load time, and (2)
> > need to install Java, but that's not hard...
> 
> Interesting. I'd like to know what others think of this.

I can think of a few!

I've done stuff before with Java applets as web 2.0 type thingys, and my
recollection is that it was very nasty. One still vivid nasty part is
that there was no good way to have the Java applet and JavaScript
communicate with each other, thereby trapping what you're doing on one
side or the other. There were classes and objects to get around this,
but the implementations were buggy and platform dependent (LiveConnect,
etc.).

Another big ding against applets was presentation/layout issues. For
example, font resizing and layout won't work with the rest of the
browser, making styles very hard to use and control. Also, printing just
gave me some truncation and looked pretty bad when I tried it with an
applet from Sun's Java site (the spreadsheet applet) and cut-and-paste
didn't work at all. Scraping data would also be impossible.

It's hard to just stick with the tangibles: things were very...fiddly.
It may have all changed since I last did this years ago, but doing a
cursory google search right now it looks to be in more or less the same
state as I remember. 

It's hard to have the browser as a platform when it's actually Java.

-Seth


> My answer to this question would be "of course we will welcome your Java 
> code contributions", delivered with a polite grin.
> 
> In all seriousness, I think we already have an emerging dream of a 
> language-agnostic message bus, certainly on the the server side.
> 
> I've been asked about standalone clients before. Google Earth often 
> comes up in these discussions. Sticking with the analogy, I usually 
> point out that (A) Google Maps came first and (B) the two applications 
> are designed to be compatible, but they occupy distinct niches in the 
> ecosystem.
> 
> One can argue language merits endlessly. Java has clear advantages.
> 
> Still, there ARE good reasons for scripting the web browser, rather than 
> budding off an applet, mainly to do with the interface. "Long load time" 
> and "no plugin" are part of it, but there are a whole lot of subtle 
> issues that go along with this: look & feel, web-browser integration, 
> compatibility with other components.
> 
> When language discussions go negative it's all over, but having said 
> that, if you DO get backed up against a wall and need a good reason 
> AGAINST Java, go and take a look at some of the Java genome browsers out 
> there. They just don't feel as light on their feet. I'm sure their 
> codebase is more stable, but they're not as fun.
> 
> I definitely DON'T want to block Java-heads from hitching a ride on our 
> message bus, so we probably need to avoid using JSON or other 
> Javascript-only tricks, and keep casual eye on compatibility. But I 
> don't want to switch to Java ;)


> > Related to this, I got many questions/suggestions that we
> > should shift layout/rendering to the client side completely.  There
> > were quite a few people in favor of that idea and seem to think even
> > JavaScript can do what we need.
> 
> I am gradually being persuaded that this is practical. Mitch has been 
> trying to get me on board his boatload of divs for some time now. I 
> realize it makes mashups easier too.

I think this is part of the 

Re: [Gmod-ajax] AJAX GBrowse poster feedback from RECOMB

[Ian H. <ih...@be...>]

Hi Andrew, thanks for passing on all this feedback.

> less-than-reusable fashion.  A certain Bay Area company (should I
> mention who?  I don't know what the etiquette rules are...)

Unless they asked you not to mention them (which would be odd at an open 
poster session), I don't see why not.

> "Pragmatic Ajax" book)... then they invited us to demo our version for
> them sometime.

Can you pass on contact details and I'll try to arrange a date?

> learned if there's a good reason for it.  But why not Java applets?
> The only two reasons we came up with are (1) long load time, and (2)
> need to install Java, but that's not hard...

Interesting. I'd like to know what others think of this.

My answer to this question would be "of course we will welcome your Java 
code contributions", delivered with a polite grin.

In all seriousness, I think we already have an emerging dream of a 
language-agnostic message bus, certainly on the the server side.

I've been asked about standalone clients before. Google Earth often 
comes up in these discussions. Sticking with the analogy, I usually 
point out that (A) Google Maps came first and (B) the two applications 
are designed to be compatible, but they occupy distinct niches in the 
ecosystem.

One can argue language merits endlessly. Java has clear advantages.

Still, there ARE good reasons for scripting the web browser, rather than 
budding off an applet, mainly to do with the interface. "Long load time" 
and "no plugin" are part of it, but there are a whole lot of subtle 
issues that go along with this: look & feel, web-browser integration, 
compatibility with other components.

When language discussions go negative it's all over, but having said 
that, if you DO get backed up against a wall and need a good reason 
AGAINST Java, go and take a look at some of the Java genome browsers out 
there. They just don't feel as light on their feet. I'm sure their 
codebase is more stable, but they're not as fun.

I definitely DON'T want to block Java-heads from hitching a ride on our 
message bus, so we probably need to avoid using JSON or other 
Javascript-only tricks, and keep casual eye on compatibility. But I 
don't want to switch to Java ;)


> - I got a couple of questions about whether our approach will scale.
> One gentleman did point out that storage requirements for e.g. a human
> genome would be ridiculous, even if we get rid of storing feature info
> in HTML.

So we definitely need HARD FACTS AND FIGURES at our disposal. 
Anticipating such questions is due diligence we should be doing.

Do we have an actual estimate for human track size (fully-rendered)? A 
gigabyte?

Even before having done the back-of-envelope calculation, I am going to 
(perhaps unwisely) stick my neck out: I don't think the human genome 
requirements are ridiculous when you stop to think about what you 
actually have to achieve.

Specifically, I think the problem mostly melts away with a few simple 
tricks, e.g.:

(1) Most tiles are whitespace, and you only really need to store one 
image of an empty tile; all the other empty tiles can be symlinks back 
to this one (this was/is the idea behind my MD5-filename hack for tile 
images)

(2) Most close-up tiles won't ever get viewed, so rendering on demand 
saves a lot of space;

(3) If the annotation upload facility takes off, you might not need more 
than a few human genome installations at first, and the large 
storage/cluster costs become centralized and benefit from economy of 
scale (c.f. UCSC);

(4) Client-side rendering has to show up in this list of tricks; but, to 
me, reimplementing parts of GBrowse in javascript is just another rabbit 
we can pull out of the hat to address problems of scale, not a goal in 
itself.

(5) Maybe we can even dip in and out of classic (CGI) GBrowse when we 
need to, though Mitch has pointed out that this might complicate the 
user experience somewhat.


> Related to this, I got many questions/suggestions that we
> should shift layout/rendering to the client side completely.  There
> were quite a few people in favor of that idea and seem to think even
> JavaScript can do what we need.

I am gradually being persuaded that this is practical. Mitch has been 
trying to get me on board his boatload of divs for some time now. I 
realize it makes mashups easier too.

I still see it as just one thing in our bag of tricks for providing a 
smooth experience, rather than an end in itself ("wow! porting code is 
so much fun! why don't you rewrite the entire thing in F# and run it on 
the Mono VM! Go on, I'll watch" </snark>)

Mitch has done quite well nudging forward Javascript rendering on the 
sly. I'll probably get very excited about it as soon as it's 
demonstrably the quickest route to a significant milestone, such as a 
functioning prototype of a human genome wiki.

Our methodology so far has been to develop fluid, functional prototypes. 
I hope we can keep doing that, and that this will lead us to the human 
genome by small steps, each yielding an intermediate product of 
practical interest.

Short answer: "Yes, clientside rendering offers many benefits and is 
likely to help with scaling issues, like storage & rendering time. It is 
definitely on our list."


 > A lot of people asked "can I do this or that track" or "this or that
 > species"... it appears people have an impression that genome browsers
 > are very restricted to the underlying data

I think this really makes it worth emphasizing the "wiki farm" idea -- 
that anyone will eventually be able to create their own new species via 
the web interface, a la Blogger or Wordpress.

To a Unix geek who can checkout & build GBrowse, that's an almost 
trivial piece of functionality to expose via the web interface, but 
clearly there are a lot of people who would benefit from it.

I.

[Gmod-ajax] AJAX GBrowse poster feedback from RECOMB

[Andrew U. <and...@gm...>]

Hi everyone...

Mitch and I presented a poster on AJAX GBrowse yesterday at RECOMB,
and it was well received.  Y'all can see the poster itself here:

http://biowiki.org/twiki/pub/GBrowse/SkinnerUzilovMungallHolmesRecomb2007Poster/RECOMB2007posterSkinnerUzilovMungallSteinHolmes.pdf

I just wanted to get some poster session impressions out of my head
before I forget them... this is by no means a complete summary of what
was mentioned... Mitch, you should add anything I'm missing:

- We had a laptop set up running the demo... original GBrowse and AJAX
GBrowse side-to-side to demonstrate.  This was a good idea.

- This is not the first time someone is making an AJAX or JavaScript
genome browser, but ours is further along and more reusable than the
other attempts.  As a matter of fact, there seems to be a tendency for
people to homebrew their own browsers for specific purposes in
less-than-reusable fashion.  A certain Bay Area company (should I
mention who?  I don't know what the etiquette rules are...) actually
went as far as to reverse engineer the obfuscated JavaScript of Google
Maps to make a spiffy custom browser (they never heard of the
"Pragmatic Ajax" book)... then they invited us to demo our version for
them sometime.  I think I convinced some people to at least use
GBrowse instead of homebrewing their own throwaway implementation.

- A lot of people asked "can I do this or that track" or "this or that
species"... it appears people have an impression that genome browsers
are very restricted to the underlying data; they're used to them being
purpose-specific.  I tried to dispell this notion.

- I also got some questions about file formats and database schemas.
In general, it seems that it wasn't obvious that this is a generic
browser useful for ANY feature data.

- I got quite a few questions about why we're not using Java applets
or SVG or Flash... one guy was particularly adamant in favor of Java
applets (a few people seemed to think they were a bad idea), but I
realized I can't really come up with a convincing reason AGAINST Java.
 SVG implementation is currently shaky, something which people at the
session mentioned, and we simply don't know Flash, but that can be
learned if there's a good reason for it.  But why not Java applets?
The only two reasons we came up with are (1) long load time, and (2)
need to install Java, but that's not hard...

- I got a couple of questions about whether our approach will scale.
One gentleman did point out that storage requirements for e.g. a human
genome would be ridiculous, even if we get rid of storing feature info
in HTML.  Related to this, I got many questions/suggestions that we
should shift layout/rendering to the client side completely.  There
were quite a few people in favor of that idea and seem to think even
JavaScript can do what we need.

Anyway... there's other stuff I'm forgetting, but this is it for now.


Andrew

Re: [Gmod-ajax] Google Summer of Code

[Lars B. <lba...@gm...>]

I just wanted to say hi, and that I really appreciate the opportunity  
to work on this project over the summer. I'll be spending the next  
couple weeks digging through the material on the biowiki and the  
ideas page for my project in preparation for coding in earnest. I'm  
sure I'll have lots of questions. Also, if anyone has any suggestions  
for things I should look into, important features, random thoughts,  
etc., please let me know!

And again, I'm very excited about this opportunity, and I look  
forward to working with you all over the summer!

-Lars


On Apr 10, 2007, at 6:29 PM, Ian Holmes wrote:

> Hi Lars,
>
> You've been selected for your Google Summer of Code application  
> "Ajax interface for the XRate command-line tool". Congratulations!  
> The competition on this particular project was very tough.
>
> I'll be your primary mentor; though, since there are a couple of  
> ajax projects in the NESCent group, we'll probably also pool  
> mentorship resources, e.g. via a mailing list.

[Gmod-ajax] Google Summer of Code

[Ian H. <ih...@be...>]

Hi Lars,

You've been selected for your Google Summer of Code application "Ajax 
interface for the XRate command-line tool". Congratulations! The 
competition on this particular project was very tough.

I'll be your primary mentor; though, since there are a couple of ajax 
projects in the NESCent group, we'll probably also pool mentorship 
resources, e.g. via a mailing list.

In the meantime, can you please subscribe to the gmod-ajax mailing list, 
register yourself at biowiki.org, and skim through some of the other 
pages linked below:

gmod-ajax page:
https://lists.sourceforge.net/lists/listinfo/gmod-ajax

Main "ideas page" for your project (please also check out the links from 
this page):
http://biowiki.org/AjaxPhyloinformatics

Early Ajax alignment viewer prototype:
http://biowiki.org/RnaAlignmentViewer

Another Ajax alignment viewer prototype you might want to check out:
http://www.ridenow.org/viewer/


Look forward to working with you this summer! Perhaps we can also fly 
you out to California at some point (or meet up in or around NJ; I'm 
over on the East Coast next month for the CSHL Biology of Genomes meeting).

Best wishes,
Ian Holmes

[Gmod-ajax] Getting started with gmod-ajax

[Mitch S. <mit...@be...>]

Since some new names have shown up on the list, I put up a new guide to 
getting started here:

http://biowiki.org/view/GBrowse/GettingStarted

I tried to strip down my machine to test the dependency list; I cleared 
out my ~/lib/perl5 directory but I've got a bunch of perl module RPMs 
installed that other stuff depends on.  So I may have missed some 
dependencies; if I did, please add them to that wiki page.

Mitch

Re: [Gmod-ajax] kudos!

[Mitch S. <mit...@be...>]

On Sun, 2007-04-08 at 09:42 -0500, Jay Hannah wrote:
> Since I've got you, I'm guessing you have commit perms to the whole  
> gbrowse project, so you might want to add "my " on line 29 to get rid  
> of this error. I pulled trunk from CVS so I'm pretty sure this is  
> just a simple vanilla bug:

I mostly just stay in my little ajax sandbox; if I wanted to get a patch
in anywhere else I'd bring it up on the mailing lists first, so I'd
prefer if you raised this on gmod-gbrowse along with your other stuff.

> Either the lib/ or libnew/ version(s)...?

FWIW, it seems like the stable branch is the better one to work with at
the moment.

Mitch

Re: [Gmod-ajax] A quick hello.

[Ian H. <ih...@be...>]

Great to hear from you Michele! Thanks for the encouragement.

Our two main goals at present are (1) to get the ajax interface up to
par with the cgi interface and (2) to enable community annotation (i.e.
persistent upload of features to the database, by anyone).

Mitch Skinner here in Berkeley is working on this full-time. Actually 
he'll be at the honeybee annotation meeting in CSHL, which is right 
before Genomes, so if you're at Genomes then maybe some introductions 
can be effected.

In the meantime any questions that you may have will be v welcome.

cheers..
Ian


Michele Clamp wrote:
> Hello chaps,
> 
> Just poking my head round the door to see what you're all up to.   
> Great to see someone tackling this - surely ajax is the way forward  
> for genome browsers.
> 
> Anyway I'm going to get my feet wet over the next couple of weeks so  
> I will probably be asking many stupid questions.
> 
> All the best,
> 
> Michele.
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys-and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Gmod-ajax mailing list
> Gmo...@li...
> https://lists.sourceforge.net/lists/listinfo/gmod-ajax

[Gmod-ajax] A quick hello.

[Michele C. <mc...@br...>]

Hello chaps,

Just poking my head round the door to see what you're all up to.   
Great to see someone tackling this - surely ajax is the way forward  
for genome browsers.

Anyway I'm going to get my feet wet over the next couple of weeks so  
I will probably be asking many stupid questions.

All the best,

Michele.

Re: [Gmod-ajax] kudos!

[Jay H. <ja...@ja...>]

On Apr 7, 2007, at 6:18 PM, Mitch Skinner wrote:
> Reading your email again, I see you've got plenty of experience with
> BioPerl.  Sorry if my last email was patronizing; I was thinking about
> how I'd update our installation documentation.

-laugh-  Not at all!  I've spent 6 hours this weekend fighting  
dependencies in the GD, lib*, Apache stacks. Always humbling. :)

Since I've got you, I'm guessing you have commit perms to the whole  
gbrowse project, so you might want to add "my " on line 29 to get rid  
of this error. I pulled trunk from CVS so I'm pretty sure this is  
just a simple vanilla bug:

[Sun Apr 08 08:58:10 2007] [error] [client 72.196.27.155] Global  
symbol "$obj" requires explicit package name at /usr/lib/perl5/ 
site_perl/5.8.6/x86_64-linux-thread-multi/Bio/Graphics/Browser/ 
PluginSet.pm line 29, <DATA> line 192.

Either the lib/ or libnew/ version(s)...?

Got gbrowse going, now on to AJAX!

Yee-ha!

j
seqlab.net
http://www.bioperl.org/wiki/User:Jhannah

Re: [Gmod-ajax] kudos!

[Mitch S. <mit...@be...>]

On Sat, 2007-04-07 at 13:15 -0500, Jay Hannah wrote:
> Can I just take a second to say "wow"!?
> 
> GBrowse is awesome. AJAX GBrowse is awesome + 1. Very impressive. I'll 
> be trying to install it today.

Cool!

The current documentation on the wiki about installing is out of date.
I'll be spending some time today updating the docs, but the gist of it
is that you need a recent bioperl, which you can install using CPAN like
so:
cpan[1]> install S/SE/SENDU/bioperl-1.5.2_102.tar.gz

And you also need a recent GBrowse; the version here should work:
http://sourceforge.net/project/showfiles.php?group_id=27707&package_id=34513&release_id=491673
(if the URL wraps, you can also just go to gmod.org and follow the
GBrowse download link).

Actually installing GBrowse can be a little involved, but if you're just
interested in rendering tiles I believe you can just put the full path
to Generic-Genome-Browser-1.67/lib into your PERL5LIB environment
variable.  GBrowse also has some requirements that you should be able to
satisfy from CPAN.  The GBrowse requirements are listed at the gmod
site:
http://blog.gmod.org/gbrowse_installation
and from that page this is the list of required modules:
            CGI                  (2.56 or higher)
            GD                   (2.07 or higher)
            CGI::Session         (4.03 or higher)
            DBI                  (any version)
            DBD::mysql           (any version)
            Digest::MD5          (any version)
            Text::Shellwords     (any version)
            Class::Base          (any version)

And from their list of optional modules I believe we need the XML ones
and GD::SVG.

Sorry about the documentation being out of date; please do follow up to
the list with any problems you have.  When you're knee deep in it it's
easy to forget about the difficulties of getting started.

Regards,
Mitch

[Gmod-ajax] kudos!

[Jay H. <ja...@ja...>]

Can I just take a second to say "wow"!?

GBrowse is awesome. AJAX GBrowse is awesome + 1. Very impressive. I'll 
be trying to install it today.

Thanks for all your hard work!

j
seqlab.net
http://www.bioperl.org/wiki/User:Jhannah

[Gmod-ajax] Slides from aforeposted Raph Koster talk

[Ian H. <ih...@be...>]

http://www.raphkoster.com/2007/03/29/etech07-the-core-of-fun/

It's a bit abstract really (and I find his idea of a "theory of fun" 
somewhat oxymoronic) but it is good to think about.

In response to Andrew & Mitch's earlier posts:

Re metrics: I think Raph was talking about metrics of user enjoyment, 
not user contributions; subtly different but let's assume they're the 
same for now (contributions are clearly more measurable).

Having a metric of contribution is important. However, as the original 
wiki crowd (the XP guys) put it: you have to "do the simplest thing that 
could possibly work". To my mind that means avoiding unnecessary coding; 
and where coding is unavoidable, you avoid unnecessary research into 
user interface design.

For example:- TWiki and (perhaps?) MediaWiki already have built-in 
metrics for who has posted the most stuff to the wiki. What they don't 
have is a ratings system, but there are lots of ratings systems out 
there. I really, really don't want to get sidetracked into experimenting 
with developing the next version of Slashdot's "karma" or Kos's "mojo" 
or Advogato's "Trust Metric", or whatever...[*]

Where I think this kind of "game design" mentality can help is in 
thinking about the fundamental steps of using the browser: locating a 
gene, pulling up the external links (GO, Flybase, PDB...), looking at 
neighboring genes, looking at other nearby features on the chromosome, 
etc. (compare this with the slide from Raph's talk listing all the 
things that someone does when they go to Amazon)

I don't quite know what we do with this list of steps yet, and I don't 
mean to imply that we should do frivolous things. The smooth scrolling 
interface (that we started with) may superficially seem frivolous but 
it's not: it directly benefits the "look at neighboring genes" and "look 
at nearby features" steps. I agree that Mitch's panning animation might 
possibly help with a sense of place too (BTW Mitch, I think that a 
quadratic (x^2) function for the pan would probably physics-y in the 
sense of Newton's 2nd law....)

The steps we have not yet really delved into are the external links, 
especially things like ontology/browser mashups (which could be really 
fun), and also the wiki upload interface. We haven't yet got a working 
upload, but when we do, the user should DEFINITELY feel rewarded for 
having uploaded something.

NB (at risk of repeating myself) I don't mean we should add frivolous 
bells & whistles, just that this concept of reward is probably a good 
one to hold in mind when developing & evaluating an interface.

Of course, this may all be a little premature, since we don't have the 
basics yet, but it's also good to think about.


[*] OK, as I re-read this, I realize a lot of those other trust metrics 
are secret... Actually, I think that if we allow "hotlists" of some kind 
(i.e. ability to aggregate other peoples' annotations and offer them up 
as a preset list), then the need for a trust metric might go away, or 
the metric might emerge naturally from the hotlist somehow.

Re: [Gmod-ajax] "Game designers obsess about making all the micro bits entertaining"

[Mitch S. <mit...@be...>]

I also liked this bit:
> Ratings. Metrics of contribution. Other people need to see it to 
> measure against it.

I think we would want to have some metrics of contribution for a wiki 
annotation project.  How to balance different kinds of users 
(computational users add lots of features, and biological users add lots 
of functional annotations; you don't want to privilege one kind of 
contribution over the other) and make it difficult to game the system 
are hard problems, though.  In my blue-sky fantasy tenure boards in 
certain fields would look at your ranking on the genome wiki as part of 
their decision making process.

Mitch

Re: [Gmod-ajax] "Game designers obsess about making all the micro bits entertaining"

[Mitch S. <mit...@be...>]

As far as the browser goes, the main entertaining things I've been 
thinking about involve animation.  Mainly animating pans along the 
chromosome and animating zoom level transitions.  I was thinking I'd 
have to justify that kind of functionality with the argument that it 
helps you keep your gut sense of where you are if the application can 
engage your visual system's object tracking functionality.  But I think 
it's also fun, or at least when I put together the experiment below I 
spent some time clicking on the arrow buttons-

http://genome.biowiki.org/test/boatdiv/boat.html

I think the animation above is cool because it uses a non-linear 
function (cos) to control the pan (and some of the js frameworks have 
even fancier animations that begin to approach physics-model 
slickness).  At some point I intend to put that pan animation into an 
actual browser.

The original idea behind that experiment was to see how many divs the 
browser could handle before it started to bog down.  When I was looking 
at the Simile::Timeline demos, the panning was pretty slow on Linux 
(although it was much smoother on windows and mac).  I wondered whether 
the slowness came from the number of divs or from how wide the view 
was.  On my hardware on linux the demo above can do a few thousand divs 
before panning starts to slow down, so I think if we do a good job of 
attaching and detaching things from the DOM (rather than having a very 
wide view like Simile::Timeline does) then a div-based genome browser 
could be pretty snappy.  Especially if we switch to a feature-density 
plot when the view starts to get busy.

It works in Firefox, Safari, and IE 6.  IE doesn't like the manual 
dragging for some reason, but the animation works fine.

Mitch

[Gmod-ajax] "Game designers obsess about making all the micro bits entertaining"

[Ian H. <ih...@be...>]

Raph Koster on how game designers would do web apps.

Some interesting stuff but it's a fairly terse set of notes (made during 
his O'Reilly Emerging Tech talk). Perhaps Raph will re-post it on his 
site...

http://www.wonderlandblog.com/wonderland/2007/03/etech_07_raph_k.html

Re: [Gmod-ajax] rendered larger chromsome

[Ian H. <ih...@be...>]

Mitch Skinner wrote:
> As an experiment I implemented this:
> http://genome.biowiki.org/gbrowse/dmel51/prototype_gbrowse.html

I like it..

> Unfortunately, the scale is currently getting rendered off-tile, and I 
> think the only way to change this is to add some padding on the side, 
> which will throw off a lot of other position calculations (on both the 
> client and server side).  The Right Thing to do in this case is have 
> special client-side support for always showing the scale at the edge of 
> the view,

At some point we should allow the black/grey stripes on the left-hand 
side to be replaced with arbitrary images, I guess.

Ian

Re: [Gmod-ajax] rendered larger chromsome

[Mitch S. <mit...@be...>]

I wrote:
> One of the things I like about the feature density plot is that it's 
> obvious that there's nothing to click on.

As an experiment I implemented this:
http://genome.biowiki.org/gbrowse/dmel51/prototype_gbrowse.html

On the conference call yesterday Lincoln said that there was some 
feature density functionality already in GBrowse, and I see some code 
for it in Bio::DB::GFF but AFAICS it's not implemented for 
Bio::DB::SeqFeature, so I did my own version.  The stuff in Bio::DB::GFF 
is per-datasource (there's some SQL to count the number of features in 
each histogram bin) and that's more efficient than retrieving all the 
features and sorting them into bins, but since we have to get all of the 
features anyway a generic implementation doesn't cost us much.  Point 
being, I don't think I'm duplicating effort by doing things this way.

Also (at Lincoln's suggestion) I made it so it doesn't generate imagemap 
regions for 1-pixel wide features.  I'm a little concerned that this 
might be confusing, since you can mouseover some features but not 
others, but then again I don't think it's too likely that people will 
try to mouseover 1-pixel targets but will probably try to zoom in some 
more before trying it.  I'd be interested to hear opinions on this--try 
looking at the exon track at 500kbp.  I should try to measure how much 
this saves on HTML size.

Unfortunately, the scale is currently getting rendered off-tile, and I 
think the only way to change this is to add some padding on the side, 
which will throw off a lot of other position calculations (on both the 
client and server side).  The Right Thing to do in this case is have 
special client-side support for always showing the scale at the edge of 
the view, so trying to add padding is not only problematic but it 
doesn't really solve the problem either.  Point being, when judging the 
usefulness of the feature density histogram, imagine that there's a 
scale on the side.

Mitch

Re: [Gmod-ajax] rendered larger chromsome

[Mitch S. <mi...@ar...>]

Andrew Uzilov wrote:
> Yeah, it would definately still be useful.  When you say "confusing,"
> do you mean confusing to implement or confusing to the user?  The user
> can be just notified with a little warning at the top of the page or
> on the track... kind of like GBrowse warns you currently.
>   

I meant confusing for the user; we could have a warning but I think it 
could be easy to miss (or annoying, if we made it really in-your-face).  
One of the things I like about the feature density plot is that it's 
obvious that there's nothing to click on.

Mitch

Re: [Gmod-ajax] rendered larger chromsome

[Andrew U. <and...@gm...>]

On 3/23/07, Mitch Skinner <mit...@be...> wrote:

[snip]

> I'm not too concerned with the amount of graphical space that the
> features take up (especially since it's easy to turn tracks on and off),
> although I guess that as we try to do larger chromosomes the
> entire_landmark zoom level will start to get quite tall.

Hmm... I think that, above some point, a single track will get so tall
(>1000px or so) as to render it worthless on any monitor.  I just
don't see anyone finding something so feature-dense useful, but I
might be wrong.  I think the Dmel demo suffers from this severely at
low zoom levels.  Maybe we should allow users to specify max track
height in pixels when they do track upload?  When in doubt, just let
them set their own parameter and pass it to
generate_tiles/Bio::Graphics

> The main
> problem in my view is the number of imagemap regions that get created
> for dense tracks.  Even if we load feature info lazily there's a minimum
> bound on the size of the html required to define those imagemap
> regions.  And the browser starts to have problems when we create large
> numbers of them.
>
> If we turn off the HTML above some threshold then those problems goes
> away.  IMO the question is whether or not the graphical view is useful
> even when you can't click on things, and I think the answer is "yes",
> but it would be confusing for the user to be able to click on features
> at some zoom levels but not others.

Yeah, it would definately still be useful.  When you say "confusing,"
do you mean confusing to implement or confusing to the user?  The user
can be just notified with a little warning at the top of the page or
on the track... kind of like GBrowse warns you currently.

Re: [Gmod-ajax] rendered larger chromsome

[Mitch S. <mit...@be...>]

Ian Holmes wrote:
> i suspect that other people (probably some on this list, or the other 
> GMOD lists) have thought quite hard about this kind of issue 
> (collapsing features, density thresholds, etc) and we'd benefit from 
> consulting with them.

Well, the GBrowse code that we're using certainly has facilities for 
dealing with high feature densities; right now the generate-tiles.pl 
code for turning off labels looks like this:

            # if the average number of features per tile is
            # less than $feature_thresh, we print labels
            if (($#features  / $num_tiles) < $feature_thresh) {
                $track->configure(-bump => 1, -label => 1, -description 
=> 1);
            } else {
                $track->configure(-bump => 1, -label => 0, -description 
=> 0);
            }

where $feature_thresh is set using the new -f command line option 
(default: 50, which is what I used to render 3R).

We could certainly add a second threshold above which we turn off 
bumping; I'm not a big fan of the non-bumped view since I don't think 
it's as useful as a density plot but I'd certainly be interested in 
hearing arguments for it.  Glyph also has a bump_limit option that I 
should try out at some point.

Right now this is how flybase deals with dense views:
====
/Detailed view is limited to 1 Mbp. Click in the overview to select a 
region 20 kbp wide.
/====
so we're not covering terribly heavily trodden ground by exploring 
different alternatives.

This ties in with what Andrew said:
> Yeah, we definately need some sort of heuristic where we compact
> features above some threshold.  Suzi's suggestion to turn them into a
> feature density plot or some such was a good idea, but would require a
> bit of work... maybe for now we can just drop the smaller features
> out, or turn them off completely (e.g. how GBrowse stops displaying
> features above the 1MB zoom level) or do something smarter than that,
> where we have a "max track height" above which feature display would
> shut off. 

I'm not too concerned with the amount of graphical space that the 
features take up (especially since it's easy to turn tracks on and off), 
although I guess that as we try to do larger chromosomes the 
entire_landmark zoom level will start to get quite tall.  The main 
problem in my view is the number of imagemap regions that get created 
for dense tracks.  Even if we load feature info lazily there's a minimum 
bound on the size of the html required to define those imagemap 
regions.  And the browser starts to have problems when we create large 
numbers of them.

If we turn off the HTML above some threshold then those problems goes 
away.  IMO the question is whether or not the graphical view is useful 
even when you can't click on things, and I think the answer is "yes", 
but it would be confusing for the user to be able to click on features 
at some zoom levels but not others.

Mitch

Re: [Gmod-ajax] rendered larger chromsome

[Chris M. <cj...@fr...>]

great to see a real dmel Chromosome up!

so how hard would the lazy-feature-info be?

(still getting the oldPrimaryDivNum error but I expect you're aware  
of that one)...

On Mar 23, 2007, at 3:54 AM, Andrew Uzilov wrote:

> Good work improving the rendering time!  It's great to see we can do a
> large chromosome in a good amount of time.
>
> Yeah, we definately need some sort of heuristic where we compact
> features above some threshold.  Suzi's suggestion to turn them into a
> feature density plot or some such was a good idea, but would require a
> bit of work... maybe for now we can just drop the smaller features
> out, or turn them off completely (e.g. how GBrowse stops displaying
> features above the 1MB zoom level) or do something smarter than that,
> where we have a "max track height" above which feature display would
> shut off.
>
> Andrew
>
> On 3/22/07, Mitch Skinner <mit...@be...> wrote:
>> I've put up Dmel 3R here:
>> http://genome.biowiki.org/gbrowse/dmel51/prototype_gbrowse.html
>>
>> It rendered in less than 8 hours on one CPU, and the tiles and  
>> HTML take
>> up around 13 gigabytes on disk.  Memory usage maxed out at 342  
>> MB.  The
>> 3R chromosome arm is the largest Dmel one at 28 Mb, or about 20 times
>> the size of chromosome 4.  So to do human we just have to scale  
>> another
>> order of magnitude; it's CPU bound and we've got a fair amount of  
>> CPU to
>> throw at it, but if memory usage grows linearly with the amount of
>> sequence then we may start running into problems.  We could do  
>> some more
>> work on rendering on demand, and that would help, but you've still  
>> got
>> to have all the features in memory at the same time to do layout,  
>> so we
>> may not be able to reduce max memory usage very much that way.   
>> Maybe we
>> can partition the layout job if there are empty regions of the  
>> chromosome.
>>
>> One problem with the current approach is that the HTML files get  
>> quite
>> large for the lower (more zoomed-out) zoom levels.  For the Genes  
>> track
>> the entire_landmark HTML file is 1.6 megabytes; for CDS it's 7.7
>> megabytes.  The HTML for entire_landmark for all tracks totals 24
>> megabytes, which is how much you would have to download to  
>> actually view
>> the entire_landmark zoom level.  Maybe we should have some kind of  
>> limit
>> above which we turn off the HTML.  Also, if we had a way of lazily
>> loading feature information then the HTML could be a lot smaller.
>>
>> With a large number of features the browser starts to slow down;  
>> for me
>> this starts to be noticeable around the 1Mbp zoom level on  
>> Windows.  On
>> other platforms firefox starts to bog down at higher (less dense)  
>> zoom
>> levels.  Again, if we turn off the html above some feature density
>> threshold this should be less of a problem.
>>
>> Mitch
>>
>> --------------------------------------------------------------------- 
>> ----
>> Take Surveys. Earn Cash. Influence the Future of IT
>> Join SourceForge.net's Techsay panel and you'll get the chance to  
>> share your
>> opinions on IT & business topics through brief surveys-and earn cash
>> http://www.techsay.com/default.php? 
>> page=join.php&p=sourceforge&CID=DEVDEV
>> _______________________________________________
>> Gmod-ajax mailing list
>> Gmo...@li...
>> https://lists.sourceforge.net/lists/listinfo/gmod-ajax
>>
>
> ---------------------------------------------------------------------- 
> ---
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to  
> share your
> opinions on IT & business topics through brief surveys-and earn cash
> http://www.techsay.com/default.php? 
> page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Gmod-ajax mailing list
> Gmo...@li...
> https://lists.sourceforge.net/lists/listinfo/gmod-ajax
>

Re: [Gmod-ajax] [GMOD-devel] [Gmod-schema] [Gmod-gbrowse] Creating a 'complete' GMOD virtual machine for distribution

[Lincoln S. <ls...@cs...>]

Hi,

David Hinds gave me a stripped down VMPlayer image that includes a running
gbrowse with human gene annotations. Compressed the whole thing was just 200
M, and performance is good enough to make a compelling demo. It is
configured with two virtual disks -- one for the base system, and one to
hold the mysql databases; the idea is that the user will add his own data to
the virtual disk. I'd be happy to provide this to anyone as a starting
place.

Lincoln

On 3/22/07, andy law (RI) <and...@bb...> wrote:
>
>
> >
> > Can I ask why use VMware instead of e.g. a liveCD or Knoppix
> > type arrangement?
>
>
> In our experience there are several advantages and one disadvantage to
> using VMWare.
>
> As Scott has mentioned, the ability to create and distribute VMWare
> images is a significant plus, although the images are (or can be) large
> and take time to move around even on a LAN. Another significant plus is
> the ability to take snapshots and rollback to them in the event of a
> disaster (or other reason). For example, you could set up a group of
> machines, run a course that involved all the users installing stuff
> within the machine and then roll each machine back in a matter of
> seconds to restore each virtual machine to its pristine state ready for
> the next group. I often use a VM to try out a new configuration safe in
> the knowledge that I can go back to where I started with *no* pain at
> all.
>
> It is possible to run the virtual machines as separate servers (i.e.
> separate from the host OS IP address) too, although our sysadmins are
> not keen on us doing that using DHCP-allocated addresses for some reason
> which makes our management a bit harder (we have to configue each
> machine with a unique hard-coded IP address which makes the distribution
> more complicated than it should be).
>
> With the VMPlayer, there is also no need to buy licenses for every
> client machine either, so the costs are now pretty reasonable.
>
>
> The main disadvantage in our hands is performance. I get reasonable
> performance (as good as I would expect given the processor) on my cra**y
> laptop, but moving the same VM onto a higher specced desktop machine
> does not - in our experience - give a proportionate increase in
> performance. I have no idea why this should be the case.
>
>
> Later,
>
> Andy
>
> -------------
> Yada, yada, yada...
>
> Roslin Institute is a company limited by guarantee, registered in
> Scotland (registered number SC157100) and a Scottish Charity (registered
> number SC023592). Our registered office is at Roslin, Midlothian, EH25
> 9PS. VAT registration number 847380013.
>
> The information contained in this e-mail (including any attachments) is
> confidential and is intended for the use of the addressee only.   The
> opinions expressed within this e-mail (including any attachments) are
> the opinions of the sender and do not necessarily constitute those of
> Roslin Institute (Edinburgh) ("the Institute") unless specifically
> stated by a sender who is duly authorised to do so on behalf of the
> Institute.
>
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share
> your
> opinions on IT & business topics through brief surveys-and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Gmod-devel mailing list
> Gmo...@li...
> https://lists.sourceforge.net/lists/listinfo/gmod-devel
>



-- 
Lincoln D. Stein
Cold Spring Harbor Laboratory
1 Bungtown Road
Cold Spring Harbor, NY 11724
(516) 367-8380 (voice)
(516) 367-8389 (fax)
FOR URGENT MESSAGES & SCHEDULING,
PLEASE CONTACT MY ASSISTANT,
SANDRA MICHELSEN, AT mic...@cs...

Re: [Gmod-ajax] rendered larger chromsome

[Andrew U. <and...@gm...>]

Good work improving the rendering time!  It's great to see we can do a
large chromosome in a good amount of time.

Yeah, we definately need some sort of heuristic where we compact
features above some threshold.  Suzi's suggestion to turn them into a
feature density plot or some such was a good idea, but would require a
bit of work... maybe for now we can just drop the smaller features
out, or turn them off completely (e.g. how GBrowse stops displaying
features above the 1MB zoom level) or do something smarter than that,
where we have a "max track height" above which feature display would
shut off.

Andrew

On 3/22/07, Mitch Skinner <mit...@be...> wrote:
> I've put up Dmel 3R here:
> http://genome.biowiki.org/gbrowse/dmel51/prototype_gbrowse.html
>
> It rendered in less than 8 hours on one CPU, and the tiles and HTML take
> up around 13 gigabytes on disk.  Memory usage maxed out at 342 MB.  The
> 3R chromosome arm is the largest Dmel one at 28 Mb, or about 20 times
> the size of chromosome 4.  So to do human we just have to scale another
> order of magnitude; it's CPU bound and we've got a fair amount of CPU to
> throw at it, but if memory usage grows linearly with the amount of
> sequence then we may start running into problems.  We could do some more
> work on rendering on demand, and that would help, but you've still got
> to have all the features in memory at the same time to do layout, so we
> may not be able to reduce max memory usage very much that way.  Maybe we
> can partition the layout job if there are empty regions of the chromosome.
>
> One problem with the current approach is that the HTML files get quite
> large for the lower (more zoomed-out) zoom levels.  For the Genes track
> the entire_landmark HTML file is 1.6 megabytes; for CDS it's 7.7
> megabytes.  The HTML for entire_landmark for all tracks totals 24
> megabytes, which is how much you would have to download to actually view
> the entire_landmark zoom level.  Maybe we should have some kind of limit
> above which we turn off the HTML.  Also, if we had a way of lazily
> loading feature information then the HTML could be a lot smaller.
>
> With a large number of features the browser starts to slow down; for me
> this starts to be noticeable around the 1Mbp zoom level on Windows.  On
> other platforms firefox starts to bog down at higher (less dense) zoom
> levels.  Again, if we turn off the html above some feature density
> threshold this should be less of a problem.
>
> Mitch
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys-and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Gmod-ajax mailing list
> Gmo...@li...
> https://lists.sourceforge.net/lists/listinfo/gmod-ajax
>

Re: [Gmod-ajax] rendered larger chromsome

[Ian H. <ih...@be...>]

Mitch,

> I've put up Dmel 3R here:
> http://genome.biowiki.org/gbrowse/dmel51/prototype_gbrowse.html

cool -- looks good (a bit slow to load but maybe others are looking at 
it right now...? could also be the ginormous html)

> It rendered in less than 8 hours on one CPU, and the tiles and HTML take 
> up around 13 gigabytes on disk.  Memory usage maxed out at 342 MB.  The 
> 3R chromosome arm is the largest Dmel one at 28 Mb, or about 20 times 
> the size of chromosome 4.  So to do human we just have to scale another 
> order of magnitude; it's CPU bound and we've got a fair amount of CPU to 
> throw at it, but if memory usage grows linearly with the amount of 
> sequence then we may start running into problems. 

as a web service we can easily afford a few 4Gb-RAM machines or even 
64Gb. disk space, similarly, is cheap - for us. as an app that can be 
downloaded and installed by anyone (i.e. like the existing GBrowse), 
you're right: this narrows the user base a bit. still i'm not too 
worried about it yet.

> We could do some more 
> work on rendering on demand, and that would help, but you've still got 
> to have all the features in memory at the same time to do layout, so we 
> may not be able to reduce max memory usage very much that way.  Maybe we 
> can partition the layout job if there are empty regions of the chromosome.

again.. not worried yet. maybe we can start thinking idly about 
bounded-space layout algorithms.

> One problem with the current approach is that the HTML files get quite 
> large for the lower (more zoomed-out) zoom levels.  For the Genes track 
> the entire_landmark HTML file is 1.6 megabytes; for CDS it's 7.7 
> megabytes.  The HTML for entire_landmark for all tracks totals 24 
> megabytes, which is how much you would have to download to actually view 
> the entire_landmark zoom level.  Maybe we should have some kind of limit 
> above which we turn off the HTML.  Also, if we had a way of lazily 
> loading feature information then the HTML could be a lot smaller.

mmm, this is a bit more scary. it does make lazy feature-info-loading 
look appealing.

> With a large number of features the browser starts to slow down; for me 
> this starts to be noticeable around the 1Mbp zoom level on Windows.  On 
> other platforms firefox starts to bog down at higher (less dense) zoom 
> levels.  Again, if we turn off the html above some feature density 
> threshold this should be less of a problem.

i suspect that other people (probably some on this list, or the other 
GMOD lists) have thought quite hard about this kind of issue (collapsing 
features, density thresholds, etc) and we'd benefit from consulting with 
them.

ian


> 
> Mitch
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys-and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Gmod-ajax mailing list
> Gmo...@li...
> https://lists.sourceforge.net/lists/listinfo/gmod-ajax