Re: [Treebase-devel] [eX-purgate bulk]: db2->pg data transfer

[Hilmar L. <hl...@du...>]

Feel free to override me on this, but I wouldn't spend time or efforts  
on an approach that involves sucking the data over the wire at the  
same time it is being loaded.

Given the size of the database, this will have poor or no  
repeatability, and a high likelihood of crashing or data corruption.  
Keep in mind that optimistically we are trying to transfer about 50GB;  
at a throughput of 1Mbit this will take 4 days and 18 hours, and I'm  
pretty sure that neither JDBC nor DBI::Pg have 1Mbit throughput, let  
alone the network between here and SDSC. If the networks drops for  
only one second during that time, the load is toast and you can start  
over.

Conversely, there are a lot of possibilities and tools that deal with  
speeding up, nicely chunking, and resuming interrupted downloads of  
plain old files. And for repeatability we need file(s) anyway here.  
All else failing, the dump can be chunked using unix command line  
tools and can be shipped on 5 DVDs (costs a few dollars), or on a  
single hard disk. It's hard to imagine why those options would not  
work, they're low tech, and we already know the tools, and how to do  
it. I'm not sure why I would not start with that.

Of course, that assumes we have a dump file to begin with.

	-hilmar

On May 27, 2009, at 1:31 PM, William Piel wrote:

>
> What do you think of this tool: http://www.sobolsoft.com/ 
> postgresqldb2/
>
> Too dodgy? But for only $30, perhaps it's worth giving it a shot.
>
> Bill
>
>
>
> On May 27, 2009, at 1:07 PM, Rutger Vos wrote:
>
>> Hi,
>>
>> Mark-Jason and I just met to discuss how to transfer data from DB2
>> (sdsc) to pg (nescent). We considered the following options:
>>
>> 1. DB2 uses an export file format called IXF, which pg doesn't
>> support. There are different versions of IXF, googling for tools that
>> might convert/import IXF only turns up things that look dodgy. An
>> unattractive possibility is that we write our own IXF parser.
>>
>> 2. Perhaps Hibernate has a facility to dump a database's contents  
>> to a
>> format that can be imported (say, select statements). You'd think  
>> that
>> Hibernate would have this, but we haven't found it.
>>
>> 3. Write our own exporter. Mark-Jason did some preliminary work on a
>> java-based "dump table to delimited file" exporter. Since we can run
>> DBD::DB2 on gigahertz.sdsc.edu we might do something along those  
>> lines
>> in perl. We note Hilmar's remarks about the dangers with this: need  
>> to
>> keep track of character encodings and escape sequences.
>>
>> I am now investigating option 2. We may have to open door 3, but we'd
>> like to hear your suggestions/remarks.
>>
>> Rutger
>>
>> --  
>> Dr. Rutger A. Vos
>> Department of zoology
>> University of British Columbia
>> http://www.nexml.org
>> http://rutgervos.blogspot.com
>>
>> ------------------------------------------------------------------------------
>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
>> is a gathering of tech-side developers & brand creativity
>> professionals. Meet
>> the minds behind Google Creative Lab, Visual Complexity,  
>> Processing, &
>> iPhoneDevCamp as they present alongside digital heavyweights like
>> Barbarian
>> Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com
>> _______________________________________________
>> Treebase-devel mailing list
>> Tre...@li...
>> https://lists.sourceforge.net/lists/listinfo/treebase-devel
>
>
>
>
>
> ------------------------------------------------------------------------------
> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
> is a gathering of tech-side developers & brand creativity  
> professionals. Meet
> the minds behind Google Creative Lab, Visual Complexity, Processing, &
> iPhoneDevCamp as they present alongside digital heavyweights like  
> Barbarian
> Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com
> _______________________________________________
> Treebase-devel mailing list
> Tre...@li...
> https://lists.sourceforge.net/lists/listinfo/treebase-devel

-- 
===========================================================
: Hilmar Lapp  -:-  Durham, NC  -:- hlapp at duke dot edu :
===========================================================