Re: [Treebase-devel] TreeBASE migration project

[Hilmar L. <hl...@du...>]

On Apr 22, 2009, at 4:45 PM, Mark Dominus wrote:

> [...]
> Hibernate will generate the database schema definition and create the
> tables if we ask it do

I know, and in fact that's one possible way to migrate the schema from  
DB2 to PostgreSQL.

It's not a good way to manage schema versions, though. I.e., there  
would be none, and accordingly, you then can't have software that uses  
the database directly rather than through Hibernate and still relies  
on anything, and you can't make any direct modifications to the schema.

For software projects such as TreeBASE it is best practice to manage  
versioning of the schema as you manage versioning of the software, and  
to change a schema by writing and applying a migration script.

> [...]
> I understand there is probably some way to dump the TB2-format data as
> it currently exists at SDSC, transfer the dump files to NESCent, and
> bulk-load them into the database on the NESCent side.

In theory yes, but in practice each RDBMS has its own dump format.  
Ideally we can get DB2 to dump the data as SQL standard-compliant  
INSERT statements, but I don't know DB2 enough yet to know whether it  
does that, and aside from that there's more than the data itself, such  
as the sequence(s), grants, etc that may not dump in a format that's  
readily ingestible by Pg.

Also, the deliverable that we want from this is a fully scripted  
process that takes the dumps from DB2, does its thing, and at the end  
the data is fully imported into the NESCent PostgreSQL instance. The  
reason is that we want this to be repeatable. I.e., we will test  
first, fix, rerun, etc, until we can switch over in a well-coordinated  
fashion that involves only minimal downtime.

> [...] I would not like to perform the TB1->TB2 migration a second  
> time, if at
> all possible.

That would be a bad idea indeed once data have been added to TB2.

>> 1d. unit testing of TB2 code
>
> The current codebase is severely lacking in unit tests.  What tests
> there are are often extremely slow and are more properly system tests
> than unit tests.

I think it's going to be *very* important to have unit tests, and  
semantic data integrity tests (i.e., tests for integrity that go  
beyond the constraints that the database enforces) so that we can have  
confidence in the migration result. Otherwise we'll be sort of betting.

	-hilmar
-- 
===========================================================
: Hilmar Lapp  -:-  Durham, NC  -:- hlapp at duke dot edu :
===========================================================