[Treebase-devel] Data migration for Dec 2009 is (mostly) compete

[Vladimir G. <vga...@ne...>]

A copy of the migrated data is now in the treebasestage DB instance.   
The http://treebase-stage.nescent.org front end should be up shortly.   
We have "named" backups for the pre- and post- migration copies. Also,  
the treebase-dev instance is reasonably close to the pre-migration  
copy, as it was restored earlier this week from a production backup.

I suggest that the staging DB instance be used for any necessary  
inspecting and cleaning -- all interested parties should have suitable  
access rights. We'll then migrate the Dec-March delta into this  
modified instance (or to its copy on production).

We should probably discuss (recall?) what are the next steps.

To keep track of any possible data cleaning activity, Hilmar was  
suggesting to set up a directory in SVN -- I will soon follow up with  
a proposal. Besides this, I will now turn to documenting and  
depositing migration scripts, etc., until I hear what else needs to be  
done to finalize the migration.


I am aware of a few issues with the post-migration data that may  
warrant further attention:

(1) Matrix loading broke a couple times, running out of heap space  
(there is probably a memory leak).  This left the matrices on which it  
broke in inconsistent state.  I scraped out all (I hope) data  
associated with these matrices and loaded them again.  Still, this was  
a deviation from the straight path.  The matrices were
     Filename       matrix_id(scraped)   matrix_id(reloaded)
     M4374.nex      4678                           4718
     M4622.nex      4210                           4717

(2) The citations.txt contained 4 "Book Section" entries, which broke  
the citation import tool.  Since these were only a handful, instead of  
tracking down the problem I decided to skip these entries (I commented  
them out).  Hopefully, it will be possible to enter them by hand  
later.  FYI, the 4 entries are appended below.

(3) As we previously noted with Bill, it appeared that the migration  
process picked up a few more matrix and tree files than Bill expected  
were new ones in the delta. Here are the orphaned trees and matrices  
that I could detect after the migration.
Compared to the list of matrices with null study_id in treebase-dev  
(which is almost identical to the pre-migration copy), the post- 
migration copy has 4 more such matrices:
  matrix_id | nexusfilename
       4227 | M4864.nex
       4280 | M4470.nex
       4456 | M4886.nex
       4528 | M4863.nex
Trees that belong to the fake study (#2264) used by the migration  
process for initial uploading (and do not belong to this study in  
treebase-dev):
  phylotree_id |  nexusfilename
--------------+-----------------
          6074 | S1934A11000.tre
          6075 | S1934A11000.tre
          6176 | S1815A10024.tre
          6358 | S1934A11001.tre
          6433 | S1319A11058.tre
          6521 | S1319A11057.tre
I should mention that there are about 26 more matrices and 52 more  
trees with the null study_id, coming from the pre-migration instance.


--Vladimir



=== These citations were skipped during the migration ====

*Book Section   Columbus, J. T.; Peterson, P. M.; Refulio Rodriguez,  
N. F.; Cerros Tlatilpa, R.; Kinney, M. S.  2009    Phylogenetics of  
Muhlenbergiinae (Poaceae, Chloridoideae, Cynodonteae) based \
on ITS and trnL-F DNA sequences    Seberg, O.; Petersen, G.; Barfod,  
A. S.; Davis, J. I.   Proceedings of the Fourth International  
Conference on the Comparative Biology of the Monocotyledons and Th\
e Fifth International Symposium on Grass Systematics and  
Evolution 
                                                                             in 
  press                Muhlenbergiinae are a subtribe \
in the grass (Poaceae) subfamily Chloridoideae, tribe Cynodonteae. The  
morphologically diverse group includes ten genera and 168 species and  
is restricted almost entirely to the New World, with a c\
enter of diversity in Mexico (125 species). With 147 species,  
Muhlenbergia is by far the largest genus, and is divided into two  
subgenera, Muhlenbergia and Trichochloa, the latter with two sections\
. The other, much smaller genera are Aegopogon (4 species), Bealia  
(1), Blepharoneuron (2), Chaboissaea (4), Lycurus (3), Pereilema (4),  
Redfieldia (1), Schaffnerella (1), and Schedonnardus (1). We\
  conducted a phylogenetic study of Muhlenbergiinae based on parsimony  
analysis of DNA sequences of the nuclear ribosomal internal  
transcribed spacer region (ITS1 + 5.8S + ITS2) and chloroplast trnL\
  intron, trnL 3′ exon, and trnL trnF intergenic spacer. All  
genera were sampled, including 52 species of Muhlenbergia representing  
both subgenera and sections. Muhlenbergia and Pereilema are \
not monophyletic in the resulting trees. The species of Pereilema and  
the other small genera are nested within Muhlenbergia in three main  
lineages. One of the lineages includes a monophyletic Muhle\
nbergia subgen. Trichochloa. Another lineage comprises species having  
leaf anatomy predictive of the PCK subtype of C4 photosynthesis. Based  
on the results of this study, we favor expanding the cir\
cumscription of Muhlenbergia to include the other nine genera of the  
subtribe.        S2438

*Book Section   Duvall, M. R.; Leseberg, C. H.; Grennan, C. P.;  
Morris, L. M.   2009    Molecular evolution and phylogenetics of  
complete chloroplast genomes in Poaceae        Davis, J.       Fifth\
  International Symposium on Grass Systematics and  
Evolution 
                                                                                 in 
  press                Phylogenetic issues in Poaceae not\
  resolved by previous multi-gene analyses can be usefully  
investigated by small genome-scale analyses. In this pilot study,  
complete or nearly complete chloroplast genomes (plastomes) were  
sequence\
d from six selected graminoids. Representatives of Anomochlooideae,  
Puelioideae, Bambusoideae, both major tribes of Panicoideae and  
Joinvilleaceae were newly sampled to supplement previously publis\
hed plastome data from Ehrhartoideae, Pooideae, and Andropogoneae. For  
amplification and sequencing, over 200 pairs of primers were designed  
in conserved regions of published grass plastomes that w\
ere positioned to flank overlapping 1200-base pair fragments around  
the entire plastome. As expected, gene order and number were highly  
conserved. Concurrent with the high conservation of the plast\
ome was considerable cumulative variation useful for studies within  
the family and even within a single tribe. Readily interpreted  
mutational patterns were observed, such as small inversions of the\
  loop in hairpin-loop regions and indels resulting from slipped- 
strand mispairings. Phylogenetic analyses were conducted on these and  
eight previously published plastomes. Maximum or near-maximum s\
upport was observed in all likelihood and parsimony bootstrap analyses  
including shallow nodes, such as those within a clade corresponding to  
a complex of four Andropogoneae, and deep nodes, such a\
s the one uniting the bambusoid/ehrhartoid/pooid (BEP) clade.      S2429

*Book Section   Prince, L.      2009    Phylogenetic relationships and  
species delimitation in Canna (Cannaceae)        Seberg, O.; Petersen,  
G.; Barfod, A. S.; Davis, J.      Proceedings of the Fo\
urth International Conference on the Comparative Biology of the  
Monocotyledons and The Fifth International Symposium on Grass  
Systematics and Evolution                                              \
                                               Canna lilies are a  
conspicuous component of tropical and subtropical humid Neotropics  
where they are native, and the Asian Paleotropics where they have\
  been introduced. Cannas have been cultivated as a food item  
(rhizome), for wrapping (leaves), and as beads (seeds) for millennia  
by indigenous people. In both tropical and temperate regions they h\
ave a long history as ornamental plants as well. With only a few dozen  
taxa in a single genus, Cannaceae has much lower generic and species  
diversity than its sister family, Marantaceae (550 specie\
s in 31 genera). Parsimony and Bayesian analyses of nuclear ribosomal  
internal transcribed spacer (ITS) and chloroplast non-coding sequence  
data (trnE-T intergenic spacer and rpL16 intron) were use\
d to infer evolutionary relationships among species. Potential causes  
of non-monophyly of nuclear ITS haplotypes and conflict between  
nuclear and plastid phylogenies for some samples are discussed.\
  Chloroplast (rbcL, ndhF) DNA data indicate a North American taxon,  
Canna flaccida, is sister to all other species in the genus.  
Phylogenetic analyses are consistent with the hypothesis of a South \
American origin for the genus, followed by dispersal and migration to  
North and Central America, and the Caribbean.     S2373

*Book Section   Roncal, J.; Borchsenius, F.; Asmussen-Lange, C. B.;  
Balslev, H. 2009    Divergence times in tribe Geonomateae (Arecaceae)  
coincide with tertiary geological events. The Comparative B\
iology of the Monocotyledons               Proceedings of the Fourth  
International Conference              Aarhus University  
Press                                                         in  
press  \
               The Geonomateae is a species-rich palm tribe restricted  
to the Neotropics with a concentration of species in western Colombia  
and adjacent Central America with extensions along the An\
des. We estimated divergence times for the Geonomateae based on a  
phylogeny resulting from analysis of two low-copy nuclear DNA genes  
and using a Bayesian relaxed molecular clock method. We obtaine\
d calibration points from the fossil record and previous dated  
phylogenies in the Arecaceae. The results indicated a diversification  
of the tribe during the Oligocene at around 31 million years ago\
. Divergence time of a high elevation Geonoma clade from 3.8 to 9.2  
million years ago coincided with the Andean uplift. A clade of Geonoma  
species from the Brazilian Shield was contemporary with Mi\
ocene marine incursions in South America. The most likely scenario to  
explain the arrival of the Calyptronoma-Calyptrogyne ancestor in the  
Greater Antilles is a migration through a dry-land connect\
ion between Central or South America prior to the formation of the  
Panamanian isthmus and after 27.8 million years ago. The molecular  
dating results were consistent with the growing evidence of a T\
ertiary diversification for most Neotropical biota and contradicted  
the Pleistocene refugia theory.     S2295