Menu
▾
▴
treebase-devel — TreeBASE developer mailing list
You can subscribe to this list here.
| 2009 |
Jan
|
Feb
|
Mar
(1) |
Apr
(41) |
May
(41) |
Jun
(50) |
Jul
(14) |
Aug
(21) |
Sep
(37) |
Oct
(8) |
Nov
(4) |
Dec
(135) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2010 |
Jan
(145) |
Feb
(110) |
Mar
(216) |
Apr
(101) |
May
(42) |
Jun
(42) |
Jul
(23) |
Aug
(17) |
Sep
(33) |
Oct
(15) |
Nov
(18) |
Dec
(6) |
| 2011 |
Jan
(8) |
Feb
(10) |
Mar
(8) |
Apr
(41) |
May
(48) |
Jun
(62) |
Jul
(7) |
Aug
(9) |
Sep
(7) |
Oct
(11) |
Nov
(49) |
Dec
(1) |
| 2012 |
Jan
(17) |
Feb
(63) |
Mar
(4) |
Apr
(13) |
May
(17) |
Jun
(21) |
Jul
(10) |
Aug
(10) |
Sep
|
Oct
|
Nov
|
Dec
(16) |
| 2013 |
Jan
(10) |
Feb
|
Mar
(1) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2014 |
Jan
(5) |
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(5) |
Nov
|
Dec
|
| 2015 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(1) |
Nov
|
Dec
|
|
From: Mark D. <mj...@ge...> - 2009-05-28 16:07:52
|
On Thu, 2009-05-28 at 11:45 -0400, Mark Dominus wrote: > It should suffice to move the old sandbox aside and check out a new one > from SF into the same place. I'll do this. Done. (Notice "revision 3" at the bottom of the page instead of "revision 6504".) |
|
From: Rutger V. <rut...@gm...> - 2009-05-28 16:07:11
|
Mmmm... I did a clean checkout from SDSC, blew away the .svn folders and committed the result to sf.net. The big idea was that this way I wouldn't accidentally add files that are in our local projects *but not under svn*, such as the .properties file that contains the passwords. Is it possible that the pom.xml belongs in that category? On Thu, May 28, 2009 at 12:00 PM, Mark Dominus <mj...@ge...> wrote: > On Wed, 2009-05-27 at 17:53 -0400, Rutger Vos wrote: >> Hi, >> >> I committed the source code to our sourceforge svn repository. > > It seems that some files may be missing. For example, trunk/pom.xml is > not in the SF SVN repository. I will correct these as I notice them, > but perhaps you could do a more systematic check for missing files and > check in anything else that might be missing. > > > > ------------------------------------------------------------------------------ > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT > is a gathering of tech-side developers & brand creativity professionals. Meet > the minds behind Google Creative Lab, Visual Complexity, Processing, & > iPhoneDevCamp as they present alongside digital heavyweights like Barbarian > Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com > _______________________________________________ > Treebase-devel mailing list > Tre...@li... > https://lists.sourceforge.net/lists/listinfo/treebase-devel > -- Dr. Rutger A. Vos Department of zoology University of British Columbia http://www.nexml.org http://rutgervos.blogspot.com |
|
From: Mark D. <mj...@ge...> - 2009-05-28 16:00:43
|
On Wed, 2009-05-27 at 17:53 -0400, Rutger Vos wrote: > Hi, > > I committed the source code to our sourceforge svn repository. It seems that some files may be missing. For example, trunk/pom.xml is not in the SF SVN repository. I will correct these as I notice them, but perhaps you could do a more systematic check for missing files and check in anything else that might be missing. |
|
From: Mark D. <mj...@ge...> - 2009-05-28 15:45:22
|
On Wed, 2009-05-27 at 17:53 -0400, Rutger Vos wrote: > Hi, > > I committed the source code to our sourceforge svn repository. I guess > the next step is to (re-)configure our IDEs and the publish routine on > 8ball and we're good to go. The publish command doesn't need to change; it just does SVN checkout. It should suffice to move the old sandbox aside and check out a new one from SF into the same place. I'll do this. |
|
From: Hilmar L. <hl...@du...> - 2009-05-27 22:12:38
|
To do it cleanly, I suppose yes (i.e., if they control your SVN repository - a simple pre-commit hook rejecting all commits would preserve the current state of the code yet disallows all changes). But you can always resort to just moving the entire code base to some new path with obvious semantics (e.g., /MOVED-TO-SF.NET-DO-NOT- CHANGE), which should make any subsequent commit against the current path fail. Just be creative :-) -hilmar On May 27, 2009, at 6:00 PM, Rutger Vos wrote: > Ah, erm, I suppose this involves asking for favours from SDSC? > > On Wed, May 27, 2009 at 5:58 PM, Hilmar Lapp <hl...@du...> wrote: >> Cool! You might want to lock the now former repository just to >> prevent any >> inadvertent commits to the wrong place. >> >> -hilmar >> >> On May 27, 2009, at 5:53 PM, Rutger Vos wrote: >> >>> Hi, >>> >>> I committed the source code to our sourceforge svn repository. I >>> guess >>> the next step is to (re-)configure our IDEs and the publish >>> routine on >>> 8ball and we're good to go. >>> >>> Rutger >>> >>> -- >>> Dr. Rutger A. Vos >>> Department of zoology >>> University of British Columbia >>> http://www.nexml.org >>> http://rutgervos.blogspot.com >>> >>> >>> ------------------------------------------------------------------------------ >>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT >>> is a gathering of tech-side developers & brand creativity >>> professionals. >>> Meet >>> the minds behind Google Creative Lab, Visual Complexity, >>> Processing, & >>> iPhoneDevCamp as they present alongside digital heavyweights like >>> Barbarian >>> Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com >>> _______________________________________________ >>> Treebase-devel mailing list >>> Tre...@li... >>> https://lists.sourceforge.net/lists/listinfo/treebase-devel >> >> -- >> =========================================================== >> : Hilmar Lapp -:- Durham, NC -:- hlapp at duke dot edu : >> =========================================================== >> >> >> >> >> > > > > -- > Dr. Rutger A. Vos > Department of zoology > University of British Columbia > http://www.nexml.org > http://rutgervos.blogspot.com -- =========================================================== : Hilmar Lapp -:- Durham, NC -:- hlapp at duke dot edu : =========================================================== |
|
From: Rutger V. <rut...@gm...> - 2009-05-27 22:00:29
|
Ah, erm, I suppose this involves asking for favours from SDSC? On Wed, May 27, 2009 at 5:58 PM, Hilmar Lapp <hl...@du...> wrote: > Cool! You might want to lock the now former repository just to prevent any > inadvertent commits to the wrong place. > > -hilmar > > On May 27, 2009, at 5:53 PM, Rutger Vos wrote: > >> Hi, >> >> I committed the source code to our sourceforge svn repository. I guess >> the next step is to (re-)configure our IDEs and the publish routine on >> 8ball and we're good to go. >> >> Rutger >> >> -- >> Dr. Rutger A. Vos >> Department of zoology >> University of British Columbia >> http://www.nexml.org >> http://rutgervos.blogspot.com >> >> >> ------------------------------------------------------------------------------ >> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT >> is a gathering of tech-side developers & brand creativity professionals. >> Meet >> the minds behind Google Creative Lab, Visual Complexity, Processing, & >> iPhoneDevCamp as they present alongside digital heavyweights like >> Barbarian >> Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com >> _______________________________________________ >> Treebase-devel mailing list >> Tre...@li... >> https://lists.sourceforge.net/lists/listinfo/treebase-devel > > -- > =========================================================== > : Hilmar Lapp -:- Durham, NC -:- hlapp at duke dot edu : > =========================================================== > > > > > -- Dr. Rutger A. Vos Department of zoology University of British Columbia http://www.nexml.org http://rutgervos.blogspot.com |
|
From: Hilmar L. <hl...@du...> - 2009-05-27 21:58:45
|
Cool! You might want to lock the now former repository just to prevent any inadvertent commits to the wrong place. -hilmar On May 27, 2009, at 5:53 PM, Rutger Vos wrote: > Hi, > > I committed the source code to our sourceforge svn repository. I guess > the next step is to (re-)configure our IDEs and the publish routine on > 8ball and we're good to go. > > Rutger > > -- > Dr. Rutger A. Vos > Department of zoology > University of British Columbia > http://www.nexml.org > http://rutgervos.blogspot.com > > ------------------------------------------------------------------------------ > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT > is a gathering of tech-side developers & brand creativity > professionals. Meet > the minds behind Google Creative Lab, Visual Complexity, Processing, & > iPhoneDevCamp as they present alongside digital heavyweights like > Barbarian > Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com > _______________________________________________ > Treebase-devel mailing list > Tre...@li... > https://lists.sourceforge.net/lists/listinfo/treebase-devel -- =========================================================== : Hilmar Lapp -:- Durham, NC -:- hlapp at duke dot edu : =========================================================== |
|
From: Rutger V. <rut...@gm...> - 2009-05-27 21:53:43
|
Hi, I committed the source code to our sourceforge svn repository. I guess the next step is to (re-)configure our IDEs and the publish routine on 8ball and we're good to go. Rutger -- Dr. Rutger A. Vos Department of zoology University of British Columbia http://www.nexml.org http://rutgervos.blogspot.com |
|
From: Mark D. <mj...@ge...> - 2009-05-27 19:10:57
|
On Wed, 2009-05-27 at 14:25 -0400, Hilmar Lapp wrote: > You can easily refer to a > previous email or thread by its URL. Yes, but they cannot easily be replied to. Since people were repeating the same points that I had already made twice, I hoped to re-inject them into the discussion so that we would not rehash absolutely everything. |
|
From: Mark D. <mj...@ge...> - 2009-05-27 19:06:14
|
On Wed, 2009-05-27 at 14:28 -0400, Hilmar Lapp wrote: > How does the IXF format look like? Is it text-based? It's a binary format. I have attached an IXF dump of the USERROLE table, whose contents are as follows: AUTHORITY | USERROLE_ID | VERSION ---------------------------------------- User | 2 | 1 Associate Editor | 3 | 1 TEST | 4 | 0 Admin | 1 | 1 TEST | 24 | 0 TEST | 25 | 0 > PostgreSQL > supports delimited files for import in various formats) DB2 will dump a delimited file or a CSV file, but it truncates the data to 250 characters. |
|
From: Hilmar L. <hl...@du...> - 2009-05-27 18:29:07
|
I would be surprised if Hibernate supports database dumps really well. (It's an ORM framework.) How does the IXF format look like? Is it text-based? Is it column- oriented and delimited? Can you post a small sample? (PostgreSQL supports delimited files for import in various formats) -hilmar On May 27, 2009, at 1:07 PM, Rutger Vos wrote: > Hi, > > Mark-Jason and I just met to discuss how to transfer data from DB2 > (sdsc) to pg (nescent). We considered the following options: > > 1. DB2 uses an export file format called IXF, which pg doesn't > support. There are different versions of IXF, googling for tools that > might convert/import IXF only turns up things that look dodgy. An > unattractive possibility is that we write our own IXF parser. > > 2. Perhaps Hibernate has a facility to dump a database's contents to a > format that can be imported (say, select statements). You'd think that > Hibernate would have this, but we haven't found it. > > 3. Write our own exporter. Mark-Jason did some preliminary work on a > java-based "dump table to delimited file" exporter. Since we can run > DBD::DB2 on gigahertz.sdsc.edu we might do something along those lines > in perl. We note Hilmar's remarks about the dangers with this: need to > keep track of character encodings and escape sequences. > > I am now investigating option 2. We may have to open door 3, but we'd > like to hear your suggestions/remarks. > > Rutger > > -- > Dr. Rutger A. Vos > Department of zoology > University of British Columbia > http://www.nexml.org > http://rutgervos.blogspot.com > > ------------------------------------------------------------------------------ > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT > is a gathering of tech-side developers & brand creativity > professionals. Meet > the minds behind Google Creative Lab, Visual Complexity, Processing, & > iPhoneDevCamp as they present alongside digital heavyweights like > Barbarian > Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com > _______________________________________________ > Treebase-devel mailing list > Tre...@li... > https://lists.sourceforge.net/lists/listinfo/treebase-devel -- =========================================================== : Hilmar Lapp -:- Durham, NC -:- hlapp at duke dot edu : =========================================================== |
|
From: Hilmar L. <hl...@du...> - 2009-05-27 18:26:47
|
Hi Mark - the mailing list is archived :-) You can easily refer to a previous email or thread by its URL. (though I agree that can get a bit messy through Sf.net's interface). -hilmar On May 27, 2009, at 1:27 PM, Mark Dominus wrote: > > I'll re-send the earlier messages that I sent to the list on this > subject. > > > > ------------------------------------------------------------------------------ > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT > is a gathering of tech-side developers & brand creativity > professionals. Meet > the minds behind Google Creative Lab, Visual Complexity, Processing, & > iPhoneDevCamp as they present alongside digital heavyweights like > Barbarian > Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com > _______________________________________________ > Treebase-devel mailing list > Tre...@li... > https://lists.sourceforge.net/lists/listinfo/treebase-devel -- =========================================================== : Hilmar Lapp -:- Durham, NC -:- hlapp at duke dot edu : =========================================================== |
|
From: Hilmar L. <hl...@du...> - 2009-05-27 18:17:28
|
Feel free to override me on this, but I wouldn't spend time or efforts on an approach that involves sucking the data over the wire at the same time it is being loaded. Given the size of the database, this will have poor or no repeatability, and a high likelihood of crashing or data corruption. Keep in mind that optimistically we are trying to transfer about 50GB; at a throughput of 1Mbit this will take 4 days and 18 hours, and I'm pretty sure that neither JDBC nor DBI::Pg have 1Mbit throughput, let alone the network between here and SDSC. If the networks drops for only one second during that time, the load is toast and you can start over. Conversely, there are a lot of possibilities and tools that deal with speeding up, nicely chunking, and resuming interrupted downloads of plain old files. And for repeatability we need file(s) anyway here. All else failing, the dump can be chunked using unix command line tools and can be shipped on 5 DVDs (costs a few dollars), or on a single hard disk. It's hard to imagine why those options would not work, they're low tech, and we already know the tools, and how to do it. I'm not sure why I would not start with that. Of course, that assumes we have a dump file to begin with. -hilmar On May 27, 2009, at 1:31 PM, William Piel wrote: > > What do you think of this tool: http://www.sobolsoft.com/ > postgresqldb2/ > > Too dodgy? But for only $30, perhaps it's worth giving it a shot. > > Bill > > > > On May 27, 2009, at 1:07 PM, Rutger Vos wrote: > >> Hi, >> >> Mark-Jason and I just met to discuss how to transfer data from DB2 >> (sdsc) to pg (nescent). We considered the following options: >> >> 1. DB2 uses an export file format called IXF, which pg doesn't >> support. There are different versions of IXF, googling for tools that >> might convert/import IXF only turns up things that look dodgy. An >> unattractive possibility is that we write our own IXF parser. >> >> 2. Perhaps Hibernate has a facility to dump a database's contents >> to a >> format that can be imported (say, select statements). You'd think >> that >> Hibernate would have this, but we haven't found it. >> >> 3. Write our own exporter. Mark-Jason did some preliminary work on a >> java-based "dump table to delimited file" exporter. Since we can run >> DBD::DB2 on gigahertz.sdsc.edu we might do something along those >> lines >> in perl. We note Hilmar's remarks about the dangers with this: need >> to >> keep track of character encodings and escape sequences. >> >> I am now investigating option 2. We may have to open door 3, but we'd >> like to hear your suggestions/remarks. >> >> Rutger >> >> -- >> Dr. Rutger A. Vos >> Department of zoology >> University of British Columbia >> http://www.nexml.org >> http://rutgervos.blogspot.com >> >> ------------------------------------------------------------------------------ >> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT >> is a gathering of tech-side developers & brand creativity >> professionals. Meet >> the minds behind Google Creative Lab, Visual Complexity, >> Processing, & >> iPhoneDevCamp as they present alongside digital heavyweights like >> Barbarian >> Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com >> _______________________________________________ >> Treebase-devel mailing list >> Tre...@li... >> https://lists.sourceforge.net/lists/listinfo/treebase-devel > > > > > > ------------------------------------------------------------------------------ > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT > is a gathering of tech-side developers & brand creativity > professionals. Meet > the minds behind Google Creative Lab, Visual Complexity, Processing, & > iPhoneDevCamp as they present alongside digital heavyweights like > Barbarian > Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com > _______________________________________________ > Treebase-devel mailing list > Tre...@li... > https://lists.sourceforge.net/lists/listinfo/treebase-devel -- =========================================================== : Hilmar Lapp -:- Durham, NC -:- hlapp at duke dot edu : =========================================================== |
|
From: Hilmar L. <hl...@du...> - 2009-05-27 18:06:06
|
That's a useful document but doesn't address data migration based on
what I can see. It's going to be a nice reference though for the
schema and code migration.
-hilmar
On May 27, 2009, at 1:37 PM, William Piel wrote:
>
> Don't know if you saw this document ("DB2 UDB To PostgreSQL Conversion
> Guide"):
>
> http://wiki.postgresql.org/images/d/d1/DB2UDB-to-PG.pdf
>
> bp
>
>
>
> ------------------------------------------------------------------------------
> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
> is a gathering of tech-side developers & brand creativity
> professionals. Meet
> the minds behind Google Creative Lab, Visual Complexity, Processing, &
> iPhoneDevCamp as they present alongside digital heavyweights like
> Barbarian
> Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com
> _______________________________________________
> Treebase-devel mailing list
> Tre...@li...
> https://lists.sourceforge.net/lists/listinfo/treebase-devel
--
===========================================================
: Hilmar Lapp -:- Durham, NC -:- hlapp at duke dot edu :
===========================================================
|
|
From: Mark D. <mj...@ge...> - 2009-05-27 17:55:11
|
On Wed, 2009-05-27 at 13:37 -0400, William Piel wrote:
> Don't know if you saw this document ("DB2 UDB To PostgreSQL Conversion
> Guide"):
I hadn't. Thanks.
|
|
From: Jon A. <jon...@du...> - 2009-05-27 17:50:11
|
The sobolsoft software looks worth a try to me. It follows the spirit of Mark's suggestion, that is, transfer the data across the wire without creating intermediate files. However, it is Windows only and my gut feeling is that it will blow up on the very large tables. Then again, maybe not. I found another solution that could work called RazorSQL (http://www.razorsql.com/features/db2_export.html ). Though it has a gui, it also has a command line interface which we could easily pipe the output into a psql statement to import directly into a PostgreSQL database (http://www.razorsql.com/docs/command_line.html ). Something like: java -Xms512M -Xmx2048M -jar razorsql.jar -export "[Treebase_DB2_db]" "select * from table1" sql | psql -Utreebase -hdarwin.nescent.org -d treebase-dev -f - I can either give someone access to a database on our Postgresql server, or I can take a stab at from this end if it is possible to get read only access to the DB2 database. -Jon ------------------------------------------------------- Jon Auman Systems Administrator National Evolutionary Synthesis Center Duke University http:www.nescent.org jon...@ne... ------------------------------------------------------ On May 27, 2009, at 1:31 PM, William Piel wrote: > > What do you think of this tool: http://www.sobolsoft.com/ > postgresqldb2/ > > Too dodgy? But for only $30, perhaps it's worth giving it a shot. > > Bill > > > > On May 27, 2009, at 1:07 PM, Rutger Vos wrote: > >> Hi, >> >> Mark-Jason and I just met to discuss how to transfer data from DB2 >> (sdsc) to pg (nescent). We considered the following options: >> >> 1. DB2 uses an export file format called IXF, which pg doesn't >> support. There are different versions of IXF, googling for tools that >> might convert/import IXF only turns up things that look dodgy. An >> unattractive possibility is that we write our own IXF parser. >> >> 2. Perhaps Hibernate has a facility to dump a database's contents >> to a >> format that can be imported (say, select statements). You'd think >> that >> Hibernate would have this, but we haven't found it. >> >> 3. Write our own exporter. Mark-Jason did some preliminary work on a >> java-based "dump table to delimited file" exporter. Since we can run >> DBD::DB2 on gigahertz.sdsc.edu we might do something along those >> lines >> in perl. We note Hilmar's remarks about the dangers with this: need >> to >> keep track of character encodings and escape sequences. >> >> I am now investigating option 2. We may have to open door 3, but we'd >> like to hear your suggestions/remarks. >> >> Rutger >> >> -- >> Dr. Rutger A. Vos >> Department of zoology >> University of British Columbia >> http://www.nexml.org >> http://rutgervos.blogspot.com >> >> ------------------------------------------------------------------------------ >> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT >> is a gathering of tech-side developers & brand creativity >> professionals. Meet >> the minds behind Google Creative Lab, Visual Complexity, >> Processing, & >> iPhoneDevCamp as they present alongside digital heavyweights like >> Barbarian >> Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com >> _______________________________________________ >> Treebase-devel mailing list >> Tre...@li... >> https://lists.sourceforge.net/lists/listinfo/treebase-devel > > > > > > ------------------------------------------------------------------------------ > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT > is a gathering of tech-side developers & brand creativity > professionals. Meet > the minds behind Google Creative Lab, Visual Complexity, Processing, & > iPhoneDevCamp as they present alongside digital heavyweights like > Barbarian > Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com > _______________________________________________ > Treebase-devel mailing list > Tre...@li... > https://lists.sourceforge.net/lists/listinfo/treebase-devel |
|
From: William P. <wil...@ya...> - 2009-05-27 17:37:45
|
Don't know if you saw this document ("DB2 UDB To PostgreSQL Conversion
Guide"):
http://wiki.postgresql.org/images/d/d1/DB2UDB-to-PG.pdf
bp
|
|
From: Mark D. <mj...@ge...> - 2009-05-27 17:37:16
|
On Fri, 2009-05-01 at 09:40 -0400, Hilmar Lapp wrote: > There'll be more on this next week. I'm tied up during the day (as I > have been the whole week), more on this tonight and tomorrow. What is the status of this? Do we have an ETA? If not, do we have an ETA for the ETA? > On Apr 30, 2009, at 12:59 PM, Mark Dominus wrote: > > > Hilmar Lapp wrote: > >> On Apr 21, 2009, at 4:56 PM, Rutger Vos wrote: > >>> > >>> I'm curious about this step: what does it involve in practical terms > >>> to get to the point where I can ssh into > >>> rv...@tr... (or some such)? > >> > >> Purchase & delivery of the hardware, virtualization environment to be > >> set up, virtual slices to be created, OS installed and imaged, > >> accounts to be created. > >> > >> Jon would have more details. We'll also be doing testing of the host > >> slices using our own development sites, and we'll be looking this and > >> next week whether we can fast-track some of the hardware purchases so > >> we can start testing earlier. > >> > > > > What's the status of this? Do we have an ETA? If not, do we have an > > ETA for the ETA? > -- Mark Jason Dominus mj...@ge... Penn Genome Frontiers Institute +1 215 573 5387 |
|
From: William P. <wil...@ya...> - 2009-05-27 17:31:36
|
What do you think of this tool: http://www.sobolsoft.com/postgresqldb2/ Too dodgy? But for only $30, perhaps it's worth giving it a shot. Bill On May 27, 2009, at 1:07 PM, Rutger Vos wrote: > Hi, > > Mark-Jason and I just met to discuss how to transfer data from DB2 > (sdsc) to pg (nescent). We considered the following options: > > 1. DB2 uses an export file format called IXF, which pg doesn't > support. There are different versions of IXF, googling for tools that > might convert/import IXF only turns up things that look dodgy. An > unattractive possibility is that we write our own IXF parser. > > 2. Perhaps Hibernate has a facility to dump a database's contents to a > format that can be imported (say, select statements). You'd think that > Hibernate would have this, but we haven't found it. > > 3. Write our own exporter. Mark-Jason did some preliminary work on a > java-based "dump table to delimited file" exporter. Since we can run > DBD::DB2 on gigahertz.sdsc.edu we might do something along those lines > in perl. We note Hilmar's remarks about the dangers with this: need to > keep track of character encodings and escape sequences. > > I am now investigating option 2. We may have to open door 3, but we'd > like to hear your suggestions/remarks. > > Rutger > > -- > Dr. Rutger A. Vos > Department of zoology > University of British Columbia > http://www.nexml.org > http://rutgervos.blogspot.com > > ------------------------------------------------------------------------------ > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT > is a gathering of tech-side developers & brand creativity > professionals. Meet > the minds behind Google Creative Lab, Visual Complexity, Processing, & > iPhoneDevCamp as they present alongside digital heavyweights like > Barbarian > Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com > _______________________________________________ > Treebase-devel mailing list > Tre...@li... > https://lists.sourceforge.net/lists/listinfo/treebase-devel |
|
From: Mark D. <mj...@ge...> - 2009-05-27 17:31:17
|
-- Mark Jason Dominus mj...@ge... Penn Genome Frontiers Institute +1 215 573 5387 |
|
From: Mark D. <mj...@ge...> - 2009-05-27 17:28:25
|
I'll re-send the earlier messages that I sent to the list on this subject. |
|
From: Rutger V. <rut...@gm...> - 2009-05-27 17:07:21
|
Hi, Mark-Jason and I just met to discuss how to transfer data from DB2 (sdsc) to pg (nescent). We considered the following options: 1. DB2 uses an export file format called IXF, which pg doesn't support. There are different versions of IXF, googling for tools that might convert/import IXF only turns up things that look dodgy. An unattractive possibility is that we write our own IXF parser. 2. Perhaps Hibernate has a facility to dump a database's contents to a format that can be imported (say, select statements). You'd think that Hibernate would have this, but we haven't found it. 3. Write our own exporter. Mark-Jason did some preliminary work on a java-based "dump table to delimited file" exporter. Since we can run DBD::DB2 on gigahertz.sdsc.edu we might do something along those lines in perl. We note Hilmar's remarks about the dangers with this: need to keep track of character encodings and escape sequences. I am now investigating option 2. We may have to open door 3, but we'd like to hear your suggestions/remarks. Rutger -- Dr. Rutger A. Vos Department of zoology University of British Columbia http://www.nexml.org http://rutgervos.blogspot.com |
|
From: Hilmar L. <hl...@du...> - 2009-05-01 14:03:03
|
There'll be more on this next week. I'm tied up during the day (as I have been the whole week), more on this tonight and tomorrow. -hilmar On Apr 30, 2009, at 12:59 PM, Mark Dominus wrote: > Hilmar Lapp wrote: >> On Apr 21, 2009, at 4:56 PM, Rutger Vos wrote: >>> >>> I'm curious about this step: what does it involve in practical terms >>> to get to the point where I can ssh into >>> rv...@tr... (or some such)? >> >> Purchase & delivery of the hardware, virtualization environment to be >> set up, virtual slices to be created, OS installed and imaged, >> accounts to be created. >> >> Jon would have more details. We'll also be doing testing of the host >> slices using our own development sites, and we'll be looking this and >> next week whether we can fast-track some of the hardware purchases so >> we can start testing earlier. >> > > What's the status of this? Do we have an ETA? If not, do we have an > ETA for the ETA? > > ------------------------------------------------------------------------------ > Register Now & Save for Velocity, the Web Performance & Operations > Conference from O'Reilly Media. Velocity features a full day of > expert-led, hands-on workshops and two days of sessions from industry > leaders in dedicated Performance & Operations tracks. Use code > vel09scf > and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf > _______________________________________________ > Treebase-devel mailing list > Tre...@li... > https://lists.sourceforge.net/lists/listinfo/treebase-devel -- =========================================================== : Hilmar Lapp -:- Durham, NC -:- hlapp at duke dot edu : =========================================================== |
|
From: Mark D. <mj...@ge...> - 2009-04-30 16:59:07
|
Hilmar Lapp wrote: > On Apr 21, 2009, at 4:56 PM, Rutger Vos wrote: >> >> I'm curious about this step: what does it involve in practical terms >> to get to the point where I can ssh into >> rv...@tr... (or some such)? > > Purchase & delivery of the hardware, virtualization environment to be > set up, virtual slices to be created, OS installed and imaged, > accounts to be created. > > Jon would have more details. We'll also be doing testing of the host > slices using our own development sites, and we'll be looking this and > next week whether we can fast-track some of the hardware purchases so > we can start testing earlier. > What's the status of this? Do we have an ETA? If not, do we have an ETA for the ETA? |
|
From: Mark D. <mj...@ge...> - 2009-04-30 16:38:50
|
Mark Dominus wrote:
> I will have it produce figures for the other tables.
Quick summary: 27 GB.
==============================================================================
Regenerating the statistics was taking too long, perhaps because of some
sort of deadlock. But using the statistics from last week, here is a
list of tables whose sizes exceed 1 MB:
CHARSET_COLRANGE 216902 5639452
COLUMNRANGE 252922 9611036
DISCRETECHARSTATE 168094 10758016
MATRIXCOLUMN 5630858 310000000
MATRIXELEMENT 300000000 25800000000
MATRIXROW 182396 39032744
PHYLOCHAR 32405 1944300
PHYLOTREE 5603 1933035
PHYLOTREENODE 431761 53970124
SUB_TAXONLABEL 102749 2671474
TAXON 424785 33558016
TAXONLABEL 232262 16722864
TAXONLABELSET_TAXONLABEL 201327 6039810
TAXONVARIANT 548169 62491264
The second column is the number of records; the third is DB2's estimate
of the total data size, excluding LOBs. To calculate this, it takes the
average row size, adds ten bytes of overhead, and multiplies by the
number of records. The row size includes the descriptor for any LOB
fields but not the LOB data itself.
To estimate LOB data, I got DB2 to give me a list of LOB fields; there
were four. Then I did 'select sum(length(FIELD)) from TABLE'. The
results are as follows:
PHYLOTREE.NEWICKSTRING 6001170
STUDY_NEXUSFILE.NEXUS 333845436
HELP.HELPTEXT 24071
MATRIXROW.SYMBOLSTRING 307037386
Total size estimate exclusive of LOB data: 26 357.582 372 MB
Total LOB estimate: 646.908 063 MB
TOTAL ESTIMATED SIZE: 27 GB.
|
9 messages has been excluded from this view by a project administrator.
