Menu
▾
▴
treebase-devel — TreeBASE developer mailing list
You can subscribe to this list here.
| 2009 |
Jan
|
Feb
|
Mar
(1) |
Apr
(41) |
May
(41) |
Jun
(50) |
Jul
(14) |
Aug
(21) |
Sep
(37) |
Oct
(8) |
Nov
(4) |
Dec
(135) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2010 |
Jan
(145) |
Feb
(110) |
Mar
(216) |
Apr
(101) |
May
(42) |
Jun
(42) |
Jul
(23) |
Aug
(17) |
Sep
(33) |
Oct
(15) |
Nov
(18) |
Dec
(6) |
| 2011 |
Jan
(8) |
Feb
(10) |
Mar
(8) |
Apr
(41) |
May
(48) |
Jun
(62) |
Jul
(7) |
Aug
(9) |
Sep
(7) |
Oct
(11) |
Nov
(49) |
Dec
(1) |
| 2012 |
Jan
(17) |
Feb
(63) |
Mar
(4) |
Apr
(13) |
May
(17) |
Jun
(21) |
Jul
(10) |
Aug
(10) |
Sep
|
Oct
|
Nov
|
Dec
(16) |
| 2013 |
Jan
(10) |
Feb
|
Mar
(1) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2014 |
Jan
(5) |
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(5) |
Nov
|
Dec
|
| 2015 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(1) |
Nov
|
Dec
|
|
From: William P. <wil...@ya...> - 2010-08-06 07:05:22
|
On Aug 6, 2010, at 8:22 AM, Hilmar Lapp wrote: > > On Aug 5, 2010, at 7:18 PM, William Piel wrote: > >> This seems okay to me, given our constraints (Hilmar - WDYT?). > > > Sounds OK to me too. It's hard to imagine that there'd be very little to notice at 123 GB size, but an annoying degree of slowdown at 180 GB. We just need to keep in mind that whatever slowdown we see, it's not going to be any less for the real database, but quite possibly more. > > -hilmar As a test file, I have a large NEXUS (attached) which I uploaded to the treebase.org site, and it took me 7 minutes 51 seconds. Course, I'm doing it from China (which has fairly slow internet) but I suspect that the real slowdown is with the insertions. In fact, this file is too big to delete -- once uploaded, trying to delete it causes a error trace (probably because postgres times out) -- evidently, our upload methods are more efficient than our delete methods. With 378 taxa and 15993 characters, we're looking at 6,045,354 insertions for the matrix element table alone. The submitter has uploaded a bunch of these (located in submission 10698 / study 10708), but we can't delete them. Since we don't have "cascade delete" constraints, and with all the FKs in circular relationships, it's also not easy to issue a SQL command to delete these. But that's beside the point. We have a benchmark file that takes a significant time to import, so we can use it for testing. bp |
|
From: Hilmar L. <hl...@ne...> - 2010-08-06 00:22:09
|
On Aug 5, 2010, at 7:18 PM, William Piel wrote: > This seems okay to me, given our constraints (Hilmar - WDYT?). Sounds OK to me too. It's hard to imagine that there'd be very little to notice at 123 GB size, but an annoying degree of slowdown at 180 GB. We just need to keep in mind that whatever slowdown we see, it's not going to be any less for the real database, but quite possibly more. -hilmar -- =========================================================== : Hilmar Lapp -:- Durham, NC -:- informatics.nescent.org : =========================================================== |
|
From: William P. <wil...@ya...> - 2010-08-05 23:18:59
|
On Aug 5, 2010, at 11:54 PM, Jon Auman wrote: > Treebaseprod (on dev server) is 123 GB as compared to 180 GB for the production database on the main treebase.org website. I'm OK with testing the performance on treebaseprod as it would save us a day of restores and the size is still significant. I will need to point treebase-dev.nescent.org to that database should that be the way we want to go. > > Let me know. > > Thanks, > Jon This seems okay to me, given our constraints (Hilmar - WDYT?). Even if the 123 GB version underestimates performance slow-down, and so when the final deployment happens we're not happy with the result -- I assume we can still go for a plan-b option? bp |
|
From: Jon A. <jon...@ne...> - 2010-08-05 15:54:39
|
Treebaseprod (on dev server) is 123 GB as compared to 180 GB for the production database on the main treebase.org website. I'm OK with testing the performance on treebaseprod as it would save us a day of restores and the size is still significant. I will need to point treebase-dev.nescent.org to that database should that be the way we want to go. Let me know. Thanks, Jon On Aug 5, 2010, at 10:02 AM, William Piel wrote: > > On Aug 5, 2010, at 8:41 PM, Hilmar Lapp wrote: > >> I would expect the primary source of >> performance degradation, if there is one, to come from updating large >> indexes, which, if done record by record that is inserted > > Okay. On treebasedb-dev there is a database called treebaseprod, which might be closest in size to the current production database. If we have a build that points to that, perhaps that would be our best performance comparison? Or alternatively, perhaps delete treebaseprod and replace it with a duplicate of the current database -- that would provide the closest size for performance comparison. > > bp > > > > ------------------------------------------------------------------------------ > The Palm PDK Hot Apps Program offers developers who use the > Plug-In Development Kit to bring their C/C++ apps to Palm for a share > of $1 Million in cash or HP Products. Visit us here for more details: > http://p.sf.net/sfu/dev2dev-palm > _______________________________________________ > Treebase-devel mailing list > Tre...@li... > https://lists.sourceforge.net/lists/listinfo/treebase-devel ------------------------------------------------------- Jon Auman Systems Administrator National Evolutionary Synthesis Center Duke University http:www.nescent.org jon...@ne... ------------------------------------------------------ |
|
From: William P. <wil...@ya...> - 2010-08-05 14:03:08
|
On Aug 5, 2010, at 8:41 PM, Hilmar Lapp wrote: > I would expect the primary source of > performance degradation, if there is one, to come from updating large > indexes, which, if done record by record that is inserted Okay. On treebasedb-dev there is a database called treebaseprod, which might be closest in size to the current production database. If we have a build that points to that, perhaps that would be our best performance comparison? Or alternatively, perhaps delete treebaseprod and replace it with a duplicate of the current database -- that would provide the closest size for performance comparison. bp |
|
From: Hilmar L. <hl...@ne...> - 2010-08-05 12:41:47
|
On Aug 5, 2010, at 2:44 AM, William Piel wrote: > Is the anticipated slowness in the time it takes to upload files to > a directory for parsing, and not the time it takes to insert records > once the file is parsed? Or is the anticipated slowness for both of > these steps? It might be both, though the first step would be independent of the size of the database, and thus I'd expect the main problem to be the second step. > Do we need to wait for something to be built on dev before we can do > the tests, or is it enough to test: > > http://treebase-dev.nescent.org/treebase-web/ I understood from Jon that this is a small database with much fewer records than the production one. I would expect the primary source of performance degradation, if there is one, to come from updating large indexes, which, if done record by record that is inserted (and there is hardly an alternative way if the database is online for concurrent querying), is highly I/O intensive, and the more so the larger the index is. -hilmar -- =========================================================== : Hilmar Lapp -:- Durham, NC -:- informatics.nescent.org : =========================================================== |
|
From: William P. <wil...@ya...> - 2010-08-05 06:47:47
|
On Aug 5, 2010, at 3:22 AM, Jon Auman wrote: > This task will take several days since it now takes about a full day to restore the treebase database. So moving to SAN does not require a rebuild from dump... What would you estimate the down time for this? bp |
|
From: William P. <wil...@ya...> - 2010-08-05 06:44:54
|
On Aug 5, 2010, at 3:27 AM, Hilmar Lapp wrote: > Just to clarify, that testing for satisfactory performance would need > to specifically test uploading large submissions. According to Jon, > the SAN seems not too bad for read access compared to local disk, but > write access can be significantly slower. So an OK test for query > performance would not suffice to make the decision. Is the anticipated slowness in the time it takes to upload files to a directory for parsing, and not the time it takes to insert records once the file is parsed? Or is the anticipated slowness for both of these steps? Do we need to wait for something to be built on dev before we can do the tests, or is it enough to test: http://treebase-dev.nescent.org/treebase-web/ vs http://www.treebase.org/treebase-web/ as they stand currently... bp |
|
From: Hilmar L. <hl...@ne...> - 2010-08-04 18:28:02
|
On Aug 4, 2010, at 2:22 PM, Jon Auman wrote: > I propose that we restore the current production database to dev, > and someone (Bill?) test the performance. It it is acceptable, I can > then move the production database to the SAN as well. Just to clarify, that testing for satisfactory performance would need to specifically test uploading large submissions. According to Jon, the SAN seems not too bad for read access compared to local disk, but write access can be significantly slower. So an OK test for query performance would not suffice to make the decision. -hilmar -- =========================================================== : Hilmar Lapp -:- Durham, NC -:- informatics.nescent.org : =========================================================== |
|
From: Jon A. <jon...@ne...> - 2010-08-04 18:23:08
|
We are running out of space on the treebase database server. I estimate we've got about a month or less before we completely run out of space. I've got two options: 1) move treebase to the SAN, where treebase-dev resides 2) install more local disks on the virtualization server, where treebase production currently lives The problem with #2 is that I will need to bring down the VMs on that server, reinstall/reconfigure the virtualization software, and restore treebase from backup. This task will take several days since it now takes about a full day to restore the treebase database. We have grown from 50 GB in February to nearly 200 GB 6 months later. I propose that we restore the current production database to dev, and someone (Bill?) test the performance. It it is acceptable, I can then move the production database to the SAN as well. So, unless anyone has any objections, I would like to restore the production database to dev this weekend. thanks, -Jon ------------------------------------------------------ Jon Auman Systems Administrator National Evolutionary Synthesis Center Duke University http:www.nescent.org jon...@ne... ------------------------------------------------------ |
|
From: youjun g. <you...@ya...> - 2010-07-22 14:29:18
|
Dear All, The Dryad-TreeBASE handshaking feature is available at " http://treebasedb-dev.nescent.org/treebase-web" for integrity test. The workflow is: Dryad send bagit package to TreeBASE via rest service at "../treebase-web//handshaking/ dryadImport" The rest service store the bagit package and return a unique url Dryad present the url to the user User use the url to log in to the TreeBASE and declare the ownership of the data TreeBASE process the bagit and add the data into user's account Due to the apache-tomcat setup, treebase application always get the user ip as 127.0.0.1(apache server's IP ), so the ip based authentication may better be implemented by apache configuration. The resource need to be protected is "../treebase-web//handshaking/dryadImport". Youjun |
|
From: youjun g. <you...@ya...> - 2010-07-20 18:20:45
|
Hi, Jon, I am testing the handshaking feature on treebase-dev, but it looks like the server is down. Youjun |
|
From: Hilmar L. <hl...@ne...> - 2010-07-19 03:18:33
|
I don't recall a feature on the website that allows one to BLAST sequences against TreeBASE. Wouldn't that be pretty useful? For example, the blast results would identify each tree in which the matching sequence is (or is part of) a OTU, which would allow me to find trees by sequence match. -hilmar -- =========================================================== : Hilmar Lapp -:- Durham, NC -:- informatics.nescent.org : =========================================================== |
|
From: Hilmar L. <hl...@ne...> - 2010-07-14 18:51:15
|
On Jul 14, 2010, at 10:25 AM, William Piel wrote: > No subdirectory with Hilmar's stuff. Not sure what's happening there. I've now shared the folder directly with you. Do you see it now? -hilmar -- =========================================================== : Hilmar Lapp -:- Durham, NC -:- informatics.nescent.org : =========================================================== |
|
From: William P. <wil...@ya...> - 2010-07-14 14:25:16
|
On Jul 14, 2010, at 7:56 AM, Rutger Vos wrote: > It was still pending as an invite to pi...@tr..., I've sent one to wil...@ya... Thanks -- I'm registered under "wil...@ya...". When I do "Open Dropbox Folder" I see: Dropbox/ievobio/doc/tb2/ with 25 items in it. Looks like it's all "rise of the machines," though. No subdirectory with Hilmar's stuff. bp |
|
From: Rutger V. <rut...@gm...> - 2010-07-14 11:57:04
|
> > > >> I've now put a copy of the poster (.pages and .pdf) in the shared > >> folder. > > > > Is this with Dropbox? Can I have access > > You should have already, because I created it as a subfolder to > Rutger's TreeBASE talk folder, which I believe Rutger shared with you. > So if you accepted that request, there should be an intuitively named > folder for the poster in there now. > > Let me know if for some reason you don't see the folder. > > It was still pending as an invite to pi...@tr..., I've sent one to wil...@ya... -- Dr. Rutger A. Vos School of Biological Sciences Philip Lyle Building, Level 4 University of Reading Reading RG6 6BX United Kingdom Tel: +44 (0) 118 378 7535 http://www.nexml.org http://rutgervos.blogspot.com |
|
From: Hilmar L. <hl...@ne...> - 2010-07-13 19:16:16
|
On Jul 13, 2010, at 12:29 PM, William Piel wrote: >> there really is no "Download all checked trees" button??). > > I'm afraid so... Looks like this ought to be a high-priority feature request then. > >> I've now put a copy of the poster (.pages and .pdf) in the shared >> folder. > > Is this with Dropbox? Can I have access You should have already, because I created it as a subfolder to Rutger's TreeBASE talk folder, which I believe Rutger shared with you. So if you accepted that request, there should be an intuitively named folder for the poster in there now. Let me know if for some reason you don't see the folder. -hilmar -- =========================================================== : Hilmar Lapp -:- Durham, NC -:- informatics.nescent.org : =========================================================== |
|
From: William P. <wil...@ya...> - 2010-07-13 16:29:20
|
On Jul 12, 2010, at 7:50 PM, Hilmar Lapp wrote: > I've finally managed to put the poster together yesterday and got it printed. fantastic. > and wanted to download the trees found (I could not > find how to do this - there really is no "Download all checked trees" > button??). I'm afraid so... The only option, really, is to do this: http://purl.org/phylo/treebase/phylows/taxon/find?query=dcterms.title==%22Homo%20sapiens%22&format=rss1&recordSchema=tree And then programmatically fetch each tree one-by-one. In addition to having a "Download all checked trees" , it would be helpful if these /phylows/ queries were functional (so far they revert to format=html): http://purl.org/phylo/treebase/phylows/taxon/find?query=dcterms.title==%22Homo%20sapiens%22&format=nexml&recordSchema=tree http://purl.org/phylo/treebase/phylows/taxon/find?query=dcterms.title==%22Homo%20sapiens%22&format=nexus&recordSchema=tree > I've now put a copy of the poster (.pages and .pdf) in the shared folder. Is this with Dropbox? Can I have access (I'm "wil...@ya...") thanks, bp |
|
From: Hilmar L. <hl...@ne...> - 2010-07-12 23:50:48
|
On Jul 8, 2010, at 6:49 AM, Rutger Vos wrote: > I prefer we move the folder you made inside my dropbox to elsewhere: Sure, this was meant only temporary. I've finally managed to put the poster together yesterday and got it printed. It's far from perfect but it's been up since yesterday afternoon. Too late for yesterday's poster session, but in time for today's, which was for even-numbered posters - and mine is E38. There were two people, one of whom was Dannie Durand (http://www.cmu.edu/bio/faculty/durand.html - cool stuff!) who wanted to do a variety of (taxon-based) searches but found them failing (I showed her how to - horrible interface though indeed), and wanted to download the trees found (I could not find how to do this - there really is no "Download all checked trees" button??). I encouraged her to bombard us with requests, hopefully she follows up on that. There was also Vincent Ranwez, the author of PhyloExplorer (http://www.ncbi.orthomam.univ-montp2.fr/phyloexplorer/ ), which apparently does queries to TreeBASE. We should be in touch with these folks to make sure they make the most effective use of the TreeBASE API. I've now put a copy of the poster (.pages and .pdf) in the shared folder. Copy it out of there if you want to keep it as I'll delete them and the enclosing folder in a couple of days. -hilmar -- =========================================================== : Hilmar Lapp -:- Durham, NC -:- informatics.nescent.org : =========================================================== |
|
From: Rutger V. <rut...@gm...> - 2010-07-08 10:49:15
|
Hi Hilmar, all the originals I used in my talk are in that same folder, yes. I prefer we move the folder you made inside my dropbox to elsewhere: as it is it is now inside my ievobio project structure, which I'd like to zip up and move to archive (as it is a "finished" project. That's how I try to do this :)) It's fine for now seeing that you're in a scramble, but it some point let's rearrange! Rutger On Wed, Jul 7, 2010 at 3:28 AM, Hilmar Lapp <hl...@ne...> wrote: > I'm leaving for Boston tomorrow in the wee hours of the morning. > Somehow I'll try to cobble together the poster, although I also have > to prepare several other things. The poster needs to go up Sat night, > and so the latest time I can have it printed (presumably at a local > Fedex/Kinkos) is on Sat morning. > > Any and all help with this would be highly appreciated. If you have, > or can make, screen shots or diagrams that weren't already in Rutger's > talk (Rutger - I assume the originals for that are all on the shared > Dropbox folder?), please drop those into the shared Dropbox folder I > just created underneath the one Rutger created. It's called 'TreeBASE2 > Poster ISMB2010'. If you don't have access but have material you'd > like to share, please let me know and I'll share the folder with you. > > Sorry for my failure to stay on top of this - I've got a huge amount > of review work dropped in my lap which is due at the same time as ISMB. > > -hilmar > -- > =========================================================== > : Hilmar Lapp -:- Durham, NC -:- informatics.nescent.org : > =========================================================== > > > > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by Sprint > What will you do first with EVO, the first 4G phone? > Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first > _______________________________________________ > Treebase-devel mailing list > Tre...@li... > https://lists.sourceforge.net/lists/listinfo/treebase-devel > -- Dr. Rutger A. Vos School of Biological Sciences Philip Lyle Building, Level 4 University of Reading Reading RG6 6BX United Kingdom Tel: +44 (0) 118 378 7535 http://www.nexml.org http://rutgervos.blogspot.com |
|
From: Hilmar L. <hl...@ne...> - 2010-07-07 16:57:48
|
On Jul 6, 2010, at 11:42 PM, William Piel wrote: > Attached is the eBiosphere poster Thanks. > The original is in Canvas X, which you may or may not have access > to. I guess I could convert it to Illustrator if that's helpful. I don't have Canvas X nor Illustrator. Any chance you can convert to PPT? -hilmar -- =========================================================== : Hilmar Lapp -:- Durham, NC -:- informatics.nescent.org : =========================================================== |
|
From: William P. <wil...@ya...> - 2010-07-07 03:43:11
|
On Jul 6, 2010, at 10:28 PM, Hilmar Lapp wrote: > I'm leaving for Boston tomorrow in the wee hours of the morning. > Somehow I'll try to cobble together the poster, although I also have > to prepare several other things. The poster needs to go up Sat night, > and so the latest time I can have it printed (presumably at a local > Fedex/Kinkos) is on Sat morning. Oh wow. Indeed, time flies. I leave tomorrow morning to a conference in Colorado. Attached is the eBiosphere poster -- in case there are icons/text that you'd like to copy/paste from. The original is in Canvas X, which you may or may not have access to. I guess I could convert it to Illustrator if that's helpful. bp |
|
From: Hilmar L. <hl...@ne...> - 2010-07-07 02:28:43
|
I'm leaving for Boston tomorrow in the wee hours of the morning. Somehow I'll try to cobble together the poster, although I also have to prepare several other things. The poster needs to go up Sat night, and so the latest time I can have it printed (presumably at a local Fedex/Kinkos) is on Sat morning. Any and all help with this would be highly appreciated. If you have, or can make, screen shots or diagrams that weren't already in Rutger's talk (Rutger - I assume the originals for that are all on the shared Dropbox folder?), please drop those into the shared Dropbox folder I just created underneath the one Rutger created. It's called 'TreeBASE2 Poster ISMB2010'. If you don't have access but have material you'd like to share, please let me know and I'll share the folder with you. Sorry for my failure to stay on top of this - I've got a huge amount of review work dropped in my lap which is due at the same time as ISMB. -hilmar -- =========================================================== : Hilmar Lapp -:- Durham, NC -:- informatics.nescent.org : =========================================================== |
|
From: Hilmar L. <hl...@ne...> - 2010-07-06 03:50:32
|
FYI. I hope this won't put anyone in tears. -hilmar Begin forwarded message: From: "ISCB" <st...@is...> Date: July 5, 2010 10:31:34 PM EDT To: hl...@ne... Subject: ISMB2010 - Killer App Award Notification Reply-To: st...@is... Dear Killer App Award submitter: we received 65 submissions for the ISMB 2010 Killer App Award, including many excellent submissions. Unfortunately we could only select four finalists, and after a series of reviews by a panel of judges, we are sorry to tell you that your submission did not make it to the final round this year. Our criteria were very demanding and multifaceted : a large number of submissions clearly addressed important questions, but currently lacked a sufficient level of ease-of-use to be immediately adopted by experimental biologists. In such cases, we hope that you will continue to develop your tool or service, and will submit an improved version for the Killer App award next year. In any case, we would like to thank you for participating in this challenge - the strong response to the first Killer App submission was very encouraging. We invite you to attend the Killer App Award session at ISMB on Tuesday, July 13: 11:45 a.m. - 12:40 p.m, in Room: 306 (TT34). Here, each of the four finalists will give a short 10 minute presentation on their work, and answer questions from the judges and the general audience. The final winner of the Kill App Award will be announced with the other awards at the ISMB closing ceremony. The four finalists will shortly be announced at http://www.iscb.org/cms_addon/conferences/ismb2010/technologytrack.php Sincerely, Anita de Waard, Elsevier, Netherlands Seán O'Donoghue, EMBL, Germany |
|
From: youjun g. <you...@ya...> - 2010-07-05 13:02:45
|
Dear TreeBASEers and Dryaders, As you may already know, I am transferring to the Yale Office of Digital Assets and Infrastructure on 26th this month. I want to let you know that I really enjoy working on the TreeBASE and Dryad project with you in the past year, and feel sorry to leave. I will keep following the development of the projects and offering any help if you need me. Thanks Youjun |
9 messages has been excluded from this view by a project administrator.
