gmod-cmap — Discussion of the CMAP comparative map viewer

Re: [gmod-cmap] Maximum number of comparative maps?

[graham e. \(JIC\) <gra...@bb...>]

Hi Ben,
That's exactly what I was asking and it works perfectly now!
Thanks so much for your help.
Best wishes,
Graham


-----Original Message-----
From: Ben Faga [mailto:fag...@gm...]
Sent: Wed 6/4/2008 19:24
To: graham etherington (JIC)
Cc: gmo...@li...
Subject: Re: [gmod-cmap]  Maximum number of comparative maps?
 
Hi Graham,

I think I know what you are asking but I want to make sure.  Are you
trying to display 4 corresponding maps with correspondences to the
reference map?  If so, the following applies.

If they are all from the same map set, you can place them in the same
"slot" (the -1, -2, 1 and 2 numbers).  Then your url becomes the
following:

http://myserver/cgi-bin/cmap/viewer?ref_map_set_acc=3;ref_map_accs=3.7;highlight=1056,773,874,1510,1403;comparative_maps=1%3dmap_acc%3d2.4%3A1%3dmap_acc%3d2.8%3A1%3dmap_acc%3d4.6%3A1%3dmap_acc%3d4.4%3A

CMap can only display correspondences between adjacent "slots".  The
reference map(s) is(are) in one slot and you can show comparisons with
map sets on either sides.  If you find it useful, you can then show
comparisons between the end comparative maps and some otherset.

Let me know if this helped.

Ben

On Wed, 2008-06-04 at 09:34 -0700, Graham Etherington wrote:
> Hi,
> Is there a maximum number of comparative maps (with correspondences) that I
> can show at any one time, or is there anything in the configuration that
> stops me from displaying more than 3 comparative maps (the reference map
> plus two other maps with correspondences).
> I'm producing URL's on the fly and here is an example of a url which should
> display 5 maps (the reference map and 4 corresponding maps at placement -1,
> -2, +1 and +2), but only displays 3 (the reference map and 2 corresponding
> maps at placement -1 and +1).
> http://myserver/cgi-bin/cmap/viewer?ref_map_set_acc=3;ref_map_accs=3.7;highlight=1056,773,874,1510,1403;comparative_maps=-1%3dmap_acc%3d2.4%3A1%3dmap_acc%3d2.8%3A-2%3dmap_acc%3d4.6%3A2%3dmap_acc%3d4.4%3A
> (Please note that my map_acc numbers are decimals)
> 
> Can anyone see any reason why all the corresponding maps will not display?
> Many thanks,
> Graham

Re: [gmod-cmap] Maximum number of comparative maps?

[Ben F. <fag...@gm...>]

Hi Graham,

I think I know what you are asking but I want to make sure.  Are you
trying to display 4 corresponding maps with correspondences to the
reference map?  If so, the following applies.

If they are all from the same map set, you can place them in the same
"slot" (the -1, -2, 1 and 2 numbers).  Then your url becomes the
following:

http://myserver/cgi-bin/cmap/viewer?ref_map_set_acc=3;ref_map_accs=3.7;highlight=1056,773,874,1510,1403;comparative_maps=1%3dmap_acc%3d2.4%3A1%3dmap_acc%3d2.8%3A1%3dmap_acc%3d4.6%3A1%3dmap_acc%3d4.4%3A

CMap can only display correspondences between adjacent "slots".  The
reference map(s) is(are) in one slot and you can show comparisons with
map sets on either sides.  If you find it useful, you can then show
comparisons between the end comparative maps and some otherset.

Let me know if this helped.

Ben

On Wed, 2008-06-04 at 09:34 -0700, Graham Etherington wrote:
> Hi,
> Is there a maximum number of comparative maps (with correspondences) that I
> can show at any one time, or is there anything in the configuration that
> stops me from displaying more than 3 comparative maps (the reference map
> plus two other maps with correspondences).
> I'm producing URL's on the fly and here is an example of a url which should
> display 5 maps (the reference map and 4 corresponding maps at placement -1,
> -2, +1 and +2), but only displays 3 (the reference map and 2 corresponding
> maps at placement -1 and +1).
> http://myserver/cgi-bin/cmap/viewer?ref_map_set_acc=3;ref_map_accs=3.7;highlight=1056,773,874,1510,1403;comparative_maps=-1%3dmap_acc%3d2.4%3A1%3dmap_acc%3d2.8%3A-2%3dmap_acc%3d4.6%3A2%3dmap_acc%3d4.4%3A
> (Please note that my map_acc numbers are decimals)
> 
> Can anyone see any reason why all the corresponding maps will not display?
> Many thanks,
> Graham

[gmod-cmap] Maximum number of comparative maps?

[Graham E. <gra...@bb...>]

Hi,
Is there a maximum number of comparative maps (with correspondences) that I
can show at any one time, or is there anything in the configuration that
stops me from displaying more than 3 comparative maps (the reference map
plus two other maps with correspondences).
I'm producing URL's on the fly and here is an example of a url which should
display 5 maps (the reference map and 4 corresponding maps at placement -1,
-2, +1 and +2), but only displays 3 (the reference map and 2 corresponding
maps at placement -1 and +1).
http://myserver/cgi-bin/cmap/viewer?ref_map_set_acc=3;ref_map_accs=3.7;highlight=1056,773,874,1510,1403;comparative_maps=-1%3dmap_acc%3d2.4%3A1%3dmap_acc%3d2.8%3A-2%3dmap_acc%3d4.6%3A2%3dmap_acc%3d4.4%3A
(Please note that my map_acc numbers are decimals)

Can anyone see any reason why all the corresponding maps will not display?
Many thanks,
Graham
-- 
View this message in context: http://www.nabble.com/Maximum-number-of-comparative-maps--tp17650954p17650954.html
Sent from the gmod-cmap mailing list archive at Nabble.com.

Re: [gmod-cmap] CMap-GFF Specification - Request for Comments

[Ben F. <fag...@gm...>]

Hi Ken,

You make good points.  My responses are in line.

On Mon, 2008-05-19 at 16:06 -0500, Ken Youens-Clark wrote:
> 
> While I applaud your efforts to create One File to Rule all the  
> Imports, I wonder if the effort to generate the correct GFF syntax  
> with all the extensions you described in the call aren't as big a  
> hurdle as generating an XML file to describe the various CMap  
> objects.  

First, the import file doesn't have to be all encompassing.  It doesn't
even need to have all the new CMap extentions.  It can just be a regular
GFF3 file and be imported into a map set just like the original cmap tab
delimited file format.  The same file that you are using for GBrowse can
be used for CMap.

Second, (and this might only be true for me because I am so very
familiar with the CMap data structure) the extensions are pretty
straight forward and logical.  Define a species, define a map set to go
in it, add map to map set, add features to map.

> The hybrid XML/tab system I describe is hacky, but it's also  
> pretty speedy.  The XML is pretty loose -- I generate it with  
> XML::Simple dumping a big hash structure, so that's easy enough.  

While the XML objects are pretty neat, there are a couple drawbacks.

One is that they use the ids to reference other objects.  I know why it
does this but it makes it hard to import the xml files into a
pre-existing database (id clashes).  Of course I could be wrong about
how the import works.

The other is that regular people aren't used to XML.  A tab delimited
file is a lot like Excel and therefor more familiar.

> And  
> the tab import for features is about as lean as I can imagine (though  
> I wish it would fail differently when importing a feature type not  
> defined in a config file).

How would you like it to handle undefined feature types?  I'm open to
suggestions.  Right now it checks the file before loading anything and
dies right away. 

> 
> The problem is that, as simple as the CMap db strives to be, it's  
> still somewhat complicated (and certainly hierarchical) data that is  
> difficult to type out in whatever form, whether from the command line  
> (cmap_admin.pl) or into some web form or into some XML-ish document.   
> It's like that saying "There's a reason they call it 'work.'"  :-)  I  
> don't know that GFF is going to be that much simpler, esp. for  
> correspondences.

That's the trick though.  The GFF can be as simple or as complex as you
need it to be.

BTW, thanks for the input.  I really appreciate it (even if I argue
every point).

Ben

Re: [gmod-cmap] CMap-GFF Specification - Request for Comments

[Ken Youens-C. <kc...@cs...>]

On May 19, 2008, at 3:24 PM, Ben Faga wrote:
> I have written a specification for the CMap extension of GFF3 and I am
> interested in hearing comments or questions about it.
>
> The idea is to allow a single file to hold all the data needed to  
> import
> into CMap (rather than messing with files for features and files for
> correspondences and manual creation of species, etc).
>
> I am also interested in hearing if you think think this is:
>
> A) a great idea that you will use,
> B) a terrible idea and why am I wasting my time on it
> C) a waste of perfectly good space in my inbox

Ben,

While I applaud your efforts to create One File to Rule all the  
Imports, I wonder if the effort to generate the correct GFF syntax  
with all the extensions you described in the call aren't as big a  
hurdle as generating an XML file to describe the various CMap  
objects.  The hybrid XML/tab system I describe is hacky, but it's also  
pretty speedy.  The XML is pretty loose -- I generate it with  
XML::Simple dumping a big hash structure, so that's easy enough.  And  
the tab import for features is about as lean as I can imagine (though  
I wish it would fail differently when importing a feature type not  
defined in a config file).

The problem is that, as simple as the CMap db strives to be, it's  
still somewhat complicated (and certainly hierarchical) data that is  
difficult to type out in whatever form, whether from the command line  
(cmap_admin.pl) or into some web form or into some XML-ish document.   
It's like that saying "There's a reason they call it 'work.'"  :-)  I  
don't know that GFF is going to be that much simpler, esp. for  
correspondences.

ky

[gmod-cmap] CMap-GFF Specification - Request for Comments

[Ben F. <fag...@gm...>]

Hello everyone,

I have written a specification for the CMap extension of GFF3 and I am
interested in hearing comments or questions about it.

The idea is to allow a single file to hold all the data needed to import
into CMap (rather than messing with files for features and files for
correspondences and manual creation of species, etc).

I am also interested in hearing if you think think this is:

 A) a great idea that you will use, 
 B) a terrible idea and why am I wasting my time on it
 C) a waste of perfectly good space in my inbox

Thanks,

Ben

Re: [gmod-cmap] Attribute in CMap

[Ben F. <fag...@gm...>]

The <attribute> option is only for describing the feature type.  An
example would be 

<attribute>
   name Description
   value This is the description of this type of feature.
   is_public 1
</attribute>

It is only visible on the "Feature Types" page.

Ben

On Fri, 2008-05-16 at 08:43 +0200, Jacques-gustave Audrey wrote:
> Thank you!!!!!
> 
> The column in the import file it's exactly what I want. So, for a  
> feature, if I have ... 3 attributes for example, in the column  
> "feature_attributes" I write :
> attribute_label1:"value_attribute1";attribute_label2:"value_attribute2";attribute_label3:"value_attribute3"?
> 
> It's good for what I have to do. But can you tell me more about the  
> "<attribute>" option? Because I want to put it  in the config file in  
> each <feature_type> tag.
> 
> Audrey
> 
> Quoting Ben Faga <fag...@gm...>:
> 
> > The "<attribute>" option that you are setting should show up on the
> > "Feature Types" page (not on each feature which is confusing).
> >
> > To add an attribute to the feature when importing the data simply add a
> > "feature_attributes" column to the import file.  Then for each feature,
> > add attributes in that column like this:
> >
> >  Genbank ID: "BH245189"; Description:"This is a description";
> >
> > If the data is already in the database, then you can write a script to
> > insert them.
> >
> > Let me know if you want to go this route and I'll describe it more.
> >
> > Ben
> >
> > On Thu, 2008-05-15 at 10:07 +0200, Jacques-gustave Audrey wrote:
> >> Hello
> >>
> >> I have a question : I need to create some attributes for a
> >> feature_type: QTL for example. I tried in the config file to put the
> >> <attribute></attribute> in the feature_type definition but nothing
> >> happens when I create a new feature with the type QTL(no attribute in
> >> the Web Admin Tool's "Attributes" part of the feature). It is possible
> >> to create attribute(s) when importing data in another way?
> >>
> >> Thanks for your help
> >>
> >> Audrey
> >>
> >>
> >> -------------------------------------------------------------------------
> >> This SF.net email is sponsored by: Microsoft
> >> Defy all challenges. Microsoft(R) Visual Studio 2008.
> >> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> >> _______________________________________________
> >> gmod-cmap mailing list
> >> gmo...@li...
> >> https://lists.sourceforge.net/lists/listinfo/gmod-cmap
> >
> >
> 
> 
> 

Re: [gmod-cmap] Attribute in CMap

[Jacques-gustave A. <aud...@et...>]

Thank you!!!!!

The column in the import file it's exactly what I want. So, for a  
feature, if I have ... 3 attributes for example, in the column  
"feature_attributes" I write :
attribute_label1:"value_attribute1";attribute_label2:"value_attribute2";attribute_label3:"value_attribute3"?

It's good for what I have to do. But can you tell me more about the  
"<attribute>" option? Because I want to put it  in the config file in  
each <feature_type> tag.

Audrey

Quoting Ben Faga <fag...@gm...>:

> The "<attribute>" option that you are setting should show up on the
> "Feature Types" page (not on each feature which is confusing).
>
> To add an attribute to the feature when importing the data simply add a
> "feature_attributes" column to the import file.  Then for each feature,
> add attributes in that column like this:
>
>  Genbank ID: "BH245189"; Description:"This is a description";
>
> If the data is already in the database, then you can write a script to
> insert them.
>
> Let me know if you want to go this route and I'll describe it more.
>
> Ben
>
> On Thu, 2008-05-15 at 10:07 +0200, Jacques-gustave Audrey wrote:
>> Hello
>>
>> I have a question : I need to create some attributes for a
>> feature_type: QTL for example. I tried in the config file to put the
>> <attribute></attribute> in the feature_type definition but nothing
>> happens when I create a new feature with the type QTL(no attribute in
>> the Web Admin Tool's "Attributes" part of the feature). It is possible
>> to create attribute(s) when importing data in another way?
>>
>> Thanks for your help
>>
>> Audrey
>>
>>
>> -------------------------------------------------------------------------
>> This SF.net email is sponsored by: Microsoft
>> Defy all challenges. Microsoft(R) Visual Studio 2008.
>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>> _______________________________________________
>> gmod-cmap mailing list
>> gmo...@li...
>> https://lists.sourceforge.net/lists/listinfo/gmod-cmap
>
>

Re: [gmod-cmap] Attribute in CMap

[Ben F. <fag...@gm...>]

The "<attribute>" option that you are setting should show up on the
"Feature Types" page (not on each feature which is confusing).

To add an attribute to the feature when importing the data simply add a
"feature_attributes" column to the import file.  Then for each feature,
add attributes in that column like this:

 Genbank ID: "BH245189"; Description:"This is a description";

If the data is already in the database, then you can write a script to
insert them.

Let me know if you want to go this route and I'll describe it more.

Ben

On Thu, 2008-05-15 at 10:07 +0200, Jacques-gustave Audrey wrote:
> Hello
> 
> I have a question : I need to create some attributes for a  
> feature_type: QTL for example. I tried in the config file to put the  
> <attribute></attribute> in the feature_type definition but nothing  
> happens when I create a new feature with the type QTL(no attribute in  
> the Web Admin Tool's "Attributes" part of the feature). It is possible  
> to create attribute(s) when importing data in another way?
> 
> Thanks for your help
> 
> Audrey
> 
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft 
> Defy all challenges. Microsoft(R) Visual Studio 2008. 
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> gmod-cmap mailing list
> gmo...@li...
> https://lists.sourceforge.net/lists/listinfo/gmod-cmap

[gmod-cmap] Attribute in CMap

[Jacques-gustave A. <aud...@et...>]

Hello

I have a question : I need to create some attributes for a  
feature_type: QTL for example. I tried in the config file to put the  
<attribute></attribute> in the feature_type definition but nothing  
happens when I create a new feature with the type QTL(no attribute in  
the Web Admin Tool's "Attributes" part of the feature). It is possible  
to create attribute(s) when importing data in another way?

Thanks for your help

Audrey

Re: [gmod-cmap] How are You Importing Data

[Ben F. <fag...@gm...>]

Thanks Ken,

These are some interesting ideas.  More comments in-line.

On Fri, 2008-05-09 at 14:02 -0500, Ken Youens-Clark wrote:
> On May 9, 2008, at 9:01 AM, Ben Faga wrote:
> > On a side note, I am thinking about getting rid of the evidence table
> > for the next release.  Are there really any cases where you want to  
> > make
> > multiple evidences for the same correspondence?  And if so, will it  
> > mess
> > up the correspondence counts to have 2 corrs instead of just one?
> 
> Ben,
> 
> If you're really considering changing the schema, I'd recommend two  
> very large changes that could create some big performance gains:
> 
> - Flatten the correspondences completely to one table.  Remove the  
> evidence or, like you said, create correspondences for each evidence  
> type.  I don't know if you really want duplicates, though, as you'd  
> have to do a lot of UNIQUEs in your queries, e.g., for building the  
> correspondence matrix.  Also, I would kill the lookup table and have  
> correspondences be one-way.  So you'd have to create both "A->B" and  
> "B->A," but that's what the lookup table was doing.  If you did all  
> this, you'd go from three correspondence tables to one.

I'm not sure how removing the lookup table helps performance.  It
actually helps quite a bit since it doesn't require looking for a corr
from both directions.  You don't need any uniques because you set one
end of the corr to a specific map set, map or feature.

Plus the lookup table reduces some joins by including feature and map
information.

> 
> - Move all feature names into one table.  This gets rid of that ugly  
> UNION in the feature search (unless you've nixed it already).  It's a  
> bit tricky as you create a circular relationship betwee the features  
> and the names (the "name" table has a "feature_id," and the "feature"  
> table has a "display_name_id" field).  We've done this in Gramene's  
> markers db, and it really makes for extremely zippy name lookups, esp.  
> considering we're up to 30M markers and 78M synonyms.

I remember you telling me how this has sped up Gramene but I'm wondering
if it would be useful w/ CMap.  I don't know how often people do a
feature search versus simply loading maps.  

While it would simplify things like name based correspondence creation,
it would add an extra join to getting features to draw the maps. 

Do you think the speed increases of various name searches outweigh the
potential slowing of the drawing?

Ben 

> 
> Those are some really major db changes, so I only offer them for  
> consideration.  However, I did learn one valuable thing from that  
> miserable season playing basketball in 6th grade -- if you're going to  
> foul, foul big.  So, if you're going to break the db, do it once and  
> make all the changes you want rather than incrementally.
> 
> Do I smell 1.01 in the offing?

I agree with you about the foul big philosophy (although you don't want
to foul so big that you get a technical foul and ejected from the game).

Thanks for the ideas.

Ben

Re: [gmod-cmap] How are You Importing Data

[Ken Youens-C. <ky...@gm...>]

On May 9, 2008, at 9:01 AM, Ben Faga wrote:
> On a side note, I am thinking about getting rid of the evidence table
> for the next release.  Are there really any cases where you want to  
> make
> multiple evidences for the same correspondence?  And if so, will it  
> mess
> up the correspondence counts to have 2 corrs instead of just one?

Ben,

If you're really considering changing the schema, I'd recommend two  
very large changes that could create some big performance gains:

- Flatten the correspondences completely to one table.  Remove the  
evidence or, like you said, create correspondences for each evidence  
type.  I don't know if you really want duplicates, though, as you'd  
have to do a lot of UNIQUEs in your queries, e.g., for building the  
correspondence matrix.  Also, I would kill the lookup table and have  
correspondences be one-way.  So you'd have to create both "A->B" and  
"B->A," but that's what the lookup table was doing.  If you did all  
this, you'd go from three correspondence tables to one.

- Move all feature names into one table.  This gets rid of that ugly  
UNION in the feature search (unless you've nixed it already).  It's a  
bit tricky as you create a circular relationship betwee the features  
and the names (the "name" table has a "feature_id," and the "feature"  
table has a "display_name_id" field).  We've done this in Gramene's  
markers db, and it really makes for extremely zippy name lookups, esp.  
considering we're up to 30M markers and 78M synonyms.

Those are some really major db changes, so I only offer them for  
consideration.  However, I did learn one valuable thing from that  
miserable season playing basketball in 6th grade -- if you're going to  
foul, foul big.  So, if you're going to break the db, do it once and  
make all the changes you want rather than incrementally.

Do I smell 1.01 in the offing?

ky

Re: [gmod-cmap] How are You Importing Data

[Ben F. <fag...@gm...>]

I've been thinking a lot about Gramene when doing the GFF stuff.  So
much so that I downloaded the CMap sql files.

Gramene is ginormous.  The evidence table data file was 5G.  I had to
kill the gunzip because it was going to fill my drive.

I am going to use Gramene as my goal.  You shouldn't have to use such a
mishmash of tech to get (relatively) speedy imports.

On a side note, I am thinking about getting rid of the evidence table
for the next release.  Are there really any cases where you want to make
multiple evidences for the same correspondence?  And if so, will it mess
up the correspondence counts to have 2 corrs instead of just one?

Ben

On Thu, 2008-05-08 at 16:21 -0500, Ken Youens-Clark wrote:
> On May 5, 2008, at 5:36 PM, Ben Faga wrote:
> > Hi everyone,
> >
> > I have been thinking a lot about importing data into CMap and I  
> > think it
> > needs improvement.
> >
> > My question to everyone is, how are you getting data into your CMap
> > databases?   Are you using cmap_admin.pl, the programing API,  
> > inserting
> > directly into the database or a mix?  Or are you importing in a way  
> > that
> > I haven't thought of?
> >
> > I am interested in hearing any issues or suggestions anyone has
> > regarding importing data.  I want CMap to be as easy to use as  
> > possible.
> 
> Ever since Gramene started relying on our markers db as the primary  
> source for CMap, I've been using a hybrid XML/tab import.  We have  
> basically the same hierarchy in our markers db -- map sets, maps, but  
> markers are separate from their mappings.  I export the map set and  
> map data (species, names, types, start/stop coords, etc.) in XML and  
> all the features in the tab files.  I then create shell scripts that  
> specify the data source, map set accessions, import file names,  
> actions, and such to push it all into CMap.
> 
> For the correspondences, I actually create a gigantor tab-file for  
> direct import by "mysqlimport" into the "cmap_feature_correspondence"  
> and "cmap_correspondence_evidence" because the API is too slow (too  
> much checking, and since I wrote the validation and have validated my  
> data externally, I skip it).  Then I recreate the  
> "cmap_correspondence_lookup" table with some SQL from you.  All in  
> all, it takes me 2-3 days to build a CMap instance for a Gramene  
> release.  Usually I build it once, find several mistakes in the  
> Gramene markers db or some CMap configuration file, then wipe it out  
> and do it all again.  CMap has become a validation tool as well as a  
> visualization tool for Gramene.
> 
> ky
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
> Don't miss this year's exciting event. There's still time to save $100. 
> Use priority code J8TL2D2. 
> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
> _______________________________________________
> gmod-cmap mailing list
> gmo...@li...
> https://lists.sourceforge.net/lists/listinfo/gmod-cmap

Re: [gmod-cmap] How are You Importing Data

[Ken Youens-C. <ky...@gm...>]

On May 5, 2008, at 5:36 PM, Ben Faga wrote:
> Hi everyone,
>
> I have been thinking a lot about importing data into CMap and I  
> think it
> needs improvement.
>
> My question to everyone is, how are you getting data into your CMap
> databases?   Are you using cmap_admin.pl, the programing API,  
> inserting
> directly into the database or a mix?  Or are you importing in a way  
> that
> I haven't thought of?
>
> I am interested in hearing any issues or suggestions anyone has
> regarding importing data.  I want CMap to be as easy to use as  
> possible.

Ever since Gramene started relying on our markers db as the primary  
source for CMap, I've been using a hybrid XML/tab import.  We have  
basically the same hierarchy in our markers db -- map sets, maps, but  
markers are separate from their mappings.  I export the map set and  
map data (species, names, types, start/stop coords, etc.) in XML and  
all the features in the tab files.  I then create shell scripts that  
specify the data source, map set accessions, import file names,  
actions, and such to push it all into CMap.

For the correspondences, I actually create a gigantor tab-file for  
direct import by "mysqlimport" into the "cmap_feature_correspondence"  
and "cmap_correspondence_evidence" because the API is too slow (too  
much checking, and since I wrote the validation and have validated my  
data externally, I skip it).  Then I recreate the  
"cmap_correspondence_lookup" table with some SQL from you.  All in  
all, it takes me 2-3 days to build a CMap instance for a Gramene  
release.  Usually I build it once, find several mistakes in the  
Gramene markers db or some CMap configuration file, then wipe it out  
and do it all again.  CMap has become a validation tool as well as a  
visualization tool for Gramene.

ky

Re: [gmod-cmap] How are You Importing Data

[Ben F. <fag...@gm...>]

Thanks Andrew and Audrey,

You comments are in fact quite useful.  

It has been a while since I've thought about how cumbersome the
next_number table is.  I don't do straight sql data entry anymore (I use
the CMap objects which handle all the extraneous stuff) but I don't do
much large scale importing.

The idea of letting RDBMSs w/ automatic ids handling the id is an
interesting one but on first thought it seems like it would break some
import methods that reserve ids from next_number (as a faster way to
work w/ next_number).  I'll have to do more thinking on this because I'm
sure it would remove a lot of head aches.

My idea for GFF3 is to have a simple(ish) format that a fair number of
people understand that can encode the whole CMap database.  I've worked
out how to represent species, map sets and maps (in custom gff
directives) as well as encoding correspondences.  But it will also
accept regular GFF3 files from other programs that would behave much
like the cmap formatted files in use now.

A secondary advantage is that it will allow you to output GFF3 to be
viewed in GBrowse.

I'll probably send a longer message on GFF3 later this week or early
next week.

I'm interested on anyone's thoughts on the matter.

Ben

On Wed, 2008-05-07 at 20:51 -0600, Andrew Farmer wrote:
> Hi Ben-
> At LIS, we've been mostly using cmap_admin.pl on tab-delimited files, 
> with the occasional direct insert or use of the web-based admin pages to 
> deal with adding generic xrefs. As someone else mentioned, doing direct 
> SQL inserts against CMap can be a bit of a pain when one forgets to 
> update the cmap_next_number table before using the perl code to do 
> further importing. But sometimes you really want to be able to do a 
> batch insert of data via SQL from some related database, rather than go 
> the flatfile route. It might be nice to have an option to configure the 
> perl so that it assumes the DBMS was managing this (e.g. if the admin 
> wanted to change the schema files to use their system's implementation 
> of sequences)?
> 
> At any rate, we do get most of our map data from collaborators via 
> spreadsheets, so it's generally very handy to be able to transform to a 
> tab-delimited file and import from that.  The current tab-delimited 
> structure strikes a good balance between the simplicity of the core data 
> and the flexibility of the representation of xrefs and attributes.
> 
> I'm curious about your vision for use of GFF3; are you imagining that 
> its use could bring in more of the "controlled vocab/ontology" flavor to 
> attributes and types? That could useful. Currently, we use some 
> simplistic naming conventions to encode a very shallow hierarchy into 
> our feature types, but that approach can only go so far before becoming 
> cumbersome.
> 
> Hope these comments from the peanut gallery are of some small use.
> 
> Regards
> 

Re: [gmod-cmap] How are You Importing Data

[Andrew F. <ad...@nc...>]

Hi Ben-
At LIS, we've been mostly using cmap_admin.pl on tab-delimited files, 
with the occasional direct insert or use of the web-based admin pages to 
deal with adding generic xrefs. As someone else mentioned, doing direct 
SQL inserts against CMap can be a bit of a pain when one forgets to 
update the cmap_next_number table before using the perl code to do 
further importing. But sometimes you really want to be able to do a 
batch insert of data via SQL from some related database, rather than go 
the flatfile route. It might be nice to have an option to configure the 
perl so that it assumes the DBMS was managing this (e.g. if the admin 
wanted to change the schema files to use their system's implementation 
of sequences)?

At any rate, we do get most of our map data from collaborators via 
spreadsheets, so it's generally very handy to be able to transform to a 
tab-delimited file and import from that.  The current tab-delimited 
structure strikes a good balance between the simplicity of the core data 
and the flexibility of the representation of xrefs and attributes.

I'm curious about your vision for use of GFF3; are you imagining that 
its use could bring in more of the "controlled vocab/ontology" flavor to 
attributes and types? That could useful. Currently, we use some 
simplistic naming conventions to encode a very shallow hierarchy into 
our feature types, but that approach can only go so far before becoming 
cumbersome.

Hope these comments from the peanut gallery are of some small use.

Regards

-- 

Andrew Farmer
ad...@nc...
(505) 995-4464
Database Administrator/Software Developer
National Center for Genome Resources

---
"To live in the presence of great truths and eternal laws,
to be led by permanent ideals-
that is what keeps a man patient when the world ignores him,
and calm and unspoiled when the world praises him."
-Balzac
---

Re: [gmod-cmap] cmap.conf is not a directory

[RobertoRun <Rob...@gm...>]

This problem is resolved,just because the SELinux, change the SELinux from
Forcing to Permission, it's OK!

-- 
View this message in context: http://www.nabble.com/cmap.conf-is-not-a-directory-tp16762552p17117620.html
Sent from the gmod-cmap mailing list archive at Nabble.com.

Re: [gmod-cmap] How are You Importing Data

[Jacques-gustave A. <aud...@et...>]

Hello

For importing data into my CMap Database I'm using cmap_admin.pl. I  
have tried directly to insert into the database (PostgreSQL with  
phpPgAdmin) but is not sympathetic with cmap_admin.pl because this  
script generate automatically IDs for features. So I think that is not  
possible to mixe this 2 possibilities :it's one or the other.

Audrey

[gmod-cmap] How are You Importing Data

[Ben F. <fag...@gm...>]

Hi everyone,

I have been thinking a lot about importing data into CMap and I think it
needs improvement.

My question to everyone is, how are you getting data into your CMap
databases?   Are you using cmap_admin.pl, the programing API, inserting
directly into the database or a mix?  Or are you importing in a way that
I haven't thought of?

I am interested in hearing any issues or suggestions anyone has
regarding importing data.  I want CMap to be as easy to use as possible.

As a side note, I am working on an adaptor to import GFF3 into CMap
(including species, map set and correspondence data).  Is there anyone
who would put GFF3 to use?

Thanks,

Ben

[gmod-cmap] Updated Postgres sql file for v1.0

[Ben F. <fag...@gm...>]

Hello,

It has come to my attention that the Postgres sql schema creation file
that was included with the CMap 1.0 distribution has a couple flaws.

If you are installing on CMap on Postgres, please use the updated
Postgres file from SourceForge.

http://gmod.cvs.sourceforge.net/*checkout*/gmod/cmap/sql/cmap.create.postgresql

Thanks to Stephen Ficklin and the Cotton Marker Database
(http://www.cottonmarker.org/ ) for the fix.

Ben

Re: [gmod-cmap] Installation issue with cmap 1.0

[Ben F. <fag...@gm...>]

Hi,

There are two things that you should probably try first.

1. The cgi-bin directory in ubuntu isn't created when apache is
installed (I don't know why).  To create it run the following command:

 $ sudo mkdir /usr/lib/cgi-bin

2. You shouldn't have to include the PREFIX or WEB_DOCUMENT_ROOT
anymore.  CMap 1.0 will now try to guess directories to install things
and will ask you if they are okay.  Try installing without the flags.

Let me know how it goes. 

Ben

On Tue, 2008-04-22 at 00:47 -0300, Sebastian Bassi wrote:
> Hello,
> 
> I'm trying to install cmap without success :(
> The base system is a standard Ubuntu server. I meet all the
> dependencies and then I did:
> 
> sbassi@JeOS1:~$ sudo perl Build.PL PREFIX=/var/www/cmap/
> WEB_DOCUMENT_ROOT=/var/www/
> sbassi@JeOS1:~$ sudo ./Build
> sbassi@JeOS1:~$ sudo ./Build install
> 
> When I go to the page (http://192.168.117.130/cmap/), I see this:
> http://i28.tinypic.com/2uqbv4w.jpg
> 
> But when I click on Maps (or on almost any other link), I get a 404:
> 
> Not Found
> The requested URL /cgi-bin/cmap/viewer was not found on this server.
> Apache/2.2.4 (Ubuntu) PHP/5.2.3-1ubuntu6 mod_perl/2.0.2 Perl/v5.8.8
> Server at 192.168.117.130 Port 80
> 
> The apacheconf is here: http://www.pastecode.com.ar/f3c19becc
> 
> I don't know what is going on. Does anybody can help me with this problem?
> 
> 

[gmod-cmap] Installation issue with cmap 1.0

[Sebastian B. <sb...@cl...>]

Hello,

I'm trying to install cmap without success :(
The base system is a standard Ubuntu server. I meet all the
dependencies and then I did:

sbassi@JeOS1:~$ sudo perl Build.PL PREFIX=/var/www/cmap/
WEB_DOCUMENT_ROOT=/var/www/
sbassi@JeOS1:~$ sudo ./Build
sbassi@JeOS1:~$ sudo ./Build install

When I go to the page (http://192.168.117.130/cmap/), I see this:
http://i28.tinypic.com/2uqbv4w.jpg

But when I click on Maps (or on almost any other link), I get a 404:

Not Found
The requested URL /cgi-bin/cmap/viewer was not found on this server.
Apache/2.2.4 (Ubuntu) PHP/5.2.3-1ubuntu6 mod_perl/2.0.2 Perl/v5.8.8
Server at 192.168.117.130 Port 80

The apacheconf is here: http://www.pastecode.com.ar/f3c19becc

I don't know what is going on. Does anybody can help me with this problem?


-- 
Sebastián Bassi (セバスティアン). Diplomado en Ciencia y Tecnología.
Curso Biologia molecular para programadores: http://tinyurl.com/2vv8w6
Mostrá tu código: http://www.pastecode.com.ar
GPG Fingerprint: 9470 0980 620D ABFC BE63 A4A4 A3DE C97D 8422 D43D

Re: [gmod-cmap] some info about "image_only parameter that will cause CMap to only return the htmp tag pointing to the image"

[Ben F. <fag...@gm...>]

Normally, CMap dynamically creates the images when the cgi-script is
called.  This image is stored in the web directory /cmap/tmp/.

The take home I guess is that if you are doing this in php, you won't be
able to integrate CMap (unless you use the image_only tag that I
mentioned before).

If you can access perl, you can do it this way:
 
The MapViewer.pm module handles the request calls so that can be your
template but the following should work (untested because I am leaving
for a wedding soon).

First craft a CMap url.

my $admin = Bio::GMOD::CMap::Admin->new( data_source => $data_source, );
my $apr = CGI->new( $url );
my %parsed_url_options = Bio::GMOD::CMap::Utils->parse_url( $apr,
$admin )
        or die $admin->error();

my $drawer = Bio::GMOD::CMap::Drawer->new(
                    apr => $apr,
                    %parsed_url_options,
                    )
                    or die
$self->error( Bio::GMOD::CMap::Drawer->error );

my $image_loc = $drawer->cache_dir."/".$drawer->image_name;

That should get you mostly there.  If anything goes wrong, take a look
at MapViewer.pm or email us.

Ben



On Fri, 2008-04-18 at 17:01 +0200, Jacques-gustave Audrey wrote:
> Re
> 
> Ok, but I want to create this image by querying the CMAP database. The  
> %parse_url_option is not very interesting for me.
> When the picture is created, how can I found it?
> 
> 
> 

Re: [gmod-cmap] some info about "image_only parameter that will cause CMap to only return the htmp tag pointing to the image"

[Jacques-gustave A. <aud...@et...>]

Re

Ok, but I want to create this image by querying the CMAP database. The  
%parse_url_option is not very interesting for me.
When the picture is created, how can I found it?

Re: [gmod-cmap] some info about "image_only parameter that will cause CMap to only return the htmp tag pointing to the image"

[Ben F. <fag...@gm...>]

Hi,

The image_only parameter is passed to the cmap cgi script.  So, to embed
an image into another page, you load the CMap viewer in something like
an iframe with the image_only param set to one. 

The following is an example.  Note the image_only bit at the end.

http://dev.gramene.org/db/cmap/viewer?mapMenu=&featureMenu=&corrMenu=&displayMenu=&advancedMenu=&ref_map_accs=wig2005b-12&sub=Draw+Maps&ref_map_start=&ref_map_stop=&ft_ssr=2&prev_ref_species_acc=rice&prev_ref_map_accs=&ref_map_set_acc=wig2005b&ref_species_acc=rice&data_source=Build26&image_only=1

It returns an html tag for the image.  In this case

<img src="/cmap/tmp/GZaGzykqZ" height="706" width="165" usemap="#zoom" border="0">

Of course this assumes that the whole thing is on the same server as
that is a relative link.  If you want to do cross server stuff, I will
have to code for that and release a patch.

BTW, if you are set on creating the image in a script, check out
Bio/GMOD/CMap/Apache/MapViewer.pm.  This calls the drawer which
generates the image.  The relavant section is this:  

    $drawer = Bio::GMOD::CMap::Drawer->new(
        apr => $apr,
        %parsed_url_options,
        )
     or return $self->error( Bio::GMOD::CMap::Drawer->error );

Ben


On Fri, 2008-04-18 at 16:18 +0200, Jacques-gustave Audrey wrote:
> Hello
> 
> I'm very interesting about the possibility to draw a map in another  
> page than CMap-Viewer. I try to do this with the  
> Bio::GMOD::CMap::Drawer module, but it's very difficult to understand  
> exactly what should I use to get a picture to include it in a php page.
> 
> This is my perl script :
> 
> ///////
> #!/usr/bin/perl
> 
> use Bio::GMOD::CMap::Admin;
> use Bio::GMOD::CMap::Drawer;
> use Bio::GMOD::CMap::Data::Generic;
> use Bio::GMOD::CMap::Config;
> 
> my $data_source='CMAP_DEMO';
> my $admin=Bio::GMOD::CMap::Admin->new(
>      data_source=>$data_source,
>      );
> my $sql_object=$admin->sql();
> 
> #avec comme parametre le map_id
> sub draw_a_map{
>      $i=0;
>      while ($_[i]!=0){
> 	my $map=$sql_object->get_maps(
> 	    map_id=>$_[0],
> 	    );
> 	my $slot={$i}{'maps'}{$map->map_acc}=(
> 	    'start'=>$map->map_start,
> 	    'stopt'=>$map->map_stop,
> 	    );
> 	$i++;
> 
>      }
>      my $drawer =Bio::GMOD::CMap::Drawer->new(
> 	slots=>$slot,
> 	data_source=>$data_source,
> 	);
> 
> 
>      return $drawer;
> }
> 
> $draw=draw_a_map(@ARGV);
> 
> //////
> Where is this "fabulous" image_only parameter? Because I don't think  
> that that's script is good.
> 
> Some help please?
> 
> 
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
> Don't miss this year's exciting event. There's still time to save $100. 
> Use priority code J8TL2D2. 
> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
> _______________________________________________
> gmod-cmap mailing list
> gmo...@li...
> https://lists.sourceforge.net/lists/listinfo/gmod-cmap