gramps

On Wed, Oct 29, 2003 at 05:22:08PM -0700, Don Allingham wrote:
> I've been giving some thought to changes after the 1.0 release. The
> biggest issue is going to be moving to a real database.

Have you taken a look at SQLite? I haven't used it, but it is quite
small, fast, easy to set up, and has python bindings. Gnucash is
considering using it for its db backend. I am amazed at the things
that have been accomplished in such a small package. From the sqlite
documentation:

    * Implements most of SQL92. (Features not supported)
    * A complete database (with multiple tables and indices) is stored
      in a single disk file.
    * Atomic commit and rollback protect data integrity.
    * Database files can be freely shared between machines with
      different byte orders.
    * Supports databases up to 2 terabytes (2^ 41bytes) in size.
    * Small memory footprint: less than 25K lines of C code.
    * Two times faster than PostgreSQL and MySQL for many common
      operations.
    * Very simple C/C++ interface requires the use of only three
      functions and one opaque structure.
    * TCL bindings included. Bindings for many other languages available
      separately.
    * Simple, well-commented source code.
    * Automated test suite provides near 100% code coverage.
    * Self-contained: no external dependencies.
    * Built and tested under Linux and Windows.
    * Sources are in the public domain. Use for any purpose.

Perhaps SQL is overkill for a genealogy program. On the other hand,
you could have multiple access a single database simultaneously!

For more information, see http://www.sqlite.org

-- 
James (Jay) Treacy
treacy@...

I haven't looked at SQLite. From the sounds of it, it has some pretty
nice features. The commit/rollback would be very nice, as would be the
single disk file.

I'll look into this.

I'd appreciate any help and advice by those on the list. I'm a
definitely not a database expert by any stretch of the imagination.

Don

On Wed, 2003-10-29 at 21:54, James A. Treacy wrote:
> On Wed, Oct 29, 2003 at 05:22:08PM -0700, Don Allingham wrote:
> > I've been giving some thought to changes after the 1.0 release. The
> > biggest issue is going to be moving to a real database.
> 
> Have you taken a look at SQLite? I haven't used it, but it is quite
> small, fast, easy to set up, and has python bindings. Gnucash is
> considering using it for its db backend. I am amazed at the things
> that have been accomplished in such a small package. From the sqlite
> documentation:
> 
>     * Implements most of SQL92. (Features not supported)
>     * A complete database (with multiple tables and indices) is stored
>       in a single disk file.
>     * Atomic commit and rollback protect data integrity.
>     * Database files can be freely shared between machines with
>       different byte orders.
>     * Supports databases up to 2 terabytes (2^ 41bytes) in size.
>     * Small memory footprint: less than 25K lines of C code.
>     * Two times faster than PostgreSQL and MySQL for many common
>       operations.
>     * Very simple C/C++ interface requires the use of only three
>       functions and one opaque structure.
>     * TCL bindings included. Bindings for many other languages available
>       separately.
>     * Simple, well-commented source code.
>     * Automated test suite provides near 100% code coverage.
>     * Self-contained: no external dependencies.
>     * Built and tested under Linux and Windows.
>     * Sources are in the public domain. Use for any purpose.
> 
> Perhaps SQL is overkill for a genealogy program. On the other hand,
> you could have multiple access a single database simultaneously!
> 
> For more information, see http://www.sqlite.org
-- 
Don Allingham <dallingham@...>
GRAMPS - Open Source Genealogy

Don> I've been giving some thought to changes after the 1.0
Don> release. The biggest issue is going to be moving to a real
Don> database.

I think it will be a big win overall, but mostly in terms of memory
used, which is pretty ugly right now.  

Don> Currently, GRAMPS has an experimental ZODB interface. This
Don> interface has never been the best, and had never been stable. I
Don> think it would probably be wise to remove the interface for the
Don> 1.0 release.

Hear hear!

Don> 1) Choice of database. I would prefer a simple to setup database.
Don> I'd rather not have Aunt Martha get frustrated because she has to
Don> configure a database server. Ideally, no additional software
Don> would have to be loaded. Currently, I'm favoring the bsddb
Don> interface. It is standard in python 2.3, and does not require any
Don> complex setup.

Hear hear!  And if you can package it in with Gramps by default, so
that user's don't even need to know it's there, that's even better.  

Don> 2) The database is going to have to be modified to eliminate all
Don> references to other objects, and using GRAMPS IDs as keys.  This
Don> will allow us to only load objects that we need as we need
Don> them. The person/family objects are going to have to be
Don> fixed. Currently, families keep a list of people, and people keep
Don> lists of families. This will have to be changed so that lists of
Don> IDs are kept. (References won't work, since the objects won't be
Don> in memory)

I'm not sure you'll have to do this.  Can't you keep different objects
in different tables, with cross references between them?  In any case,
good DB programming requires that you use unique keys for each table,
even if the data in them is supposedly unique.  If you guess wrong
about your data, you're ok.  

This means that the re-writting of the gramps back end is a good time
to maybe abstract out how data is handled internally, so we can put in
a proper data model here.

I guess I really need to learn python and spend the time understand
gramps internally.  Good thing it's winter and I'll be inside more
now.  :]  

Back to the references issue.  Can you describe the current schema
used for gramps and how it stores records/objects currently?  Just
from looking at the XML file from an 0.8.0-pl2 version of the code
(it's what I had handy), you've got the following objects:

      People
      Families
      Events
      Places
      Sources

And they all look to use the proper keys to reference each other.
This looks good.  I'm probably missing the internal issues though...

Don> 3) When does data get updated? Do we move to a mode where data is
Don> updated as changed? This is how most databases (and genealogy
Don> programs) work. 

I would suspect that you'd want to write changes to disk once they are
made.  Don't keep them in memory except for caching reasons.  

Don> However, if we are still working with an XML database, we will
Don> have to have the load/save model. Would this be confusing?

Why would we want to keep the XML DB around after 1.0 realease?  It's
going to be a big enough change that I think it should be just dropped
completely.  

What advantages do you see for the XML interface?  The only one I see
is that the DB is in a human readable format, but it would be easy
enough to have an import/export filter for XML formated files.  But
the internal setup should be purely based on a DB format.

John

On Thu, 2003-10-30 at 07:55, John Stoffel wrote:
> Don> 2) The database is going to have to be modified to eliminate all
> Don> references to other objects, and using GRAMPS IDs as keys.  This
> Don> will allow us to only load objects that we need as we need
> Don> them. The person/family objects are going to have to be
> Don> fixed. Currently, families keep a list of people, and people keep
> Don> lists of families. This will have to be changed so that lists of
> Don> IDs are kept. (References won't work, since the objects won't be
> Don> in memory)
> 
> I'm not sure you'll have to do this.  Can't you keep different objects
> in different tables, with cross references between them?  In any case,
> good DB programming requires that you use unique keys for each table,
> even if the data in them is supposedly unique.  If you guess wrong
> about your data, you're ok.  

Actually, this is what I meant. Currently, we store python references,
not keys. Something like (but vastly simplified)

 a = Person()
 f = Family()

 f.add(a)

In this case, we are making a reference to the in-memory representation
of "a". This type of structure does not work well for a real database,
since "a" may not be (and probably isn't) in memory.

We need to change to:

 a = Person()
 f = Family()

 f.add(a.getId())

This way, we are storing a key to the person in the family structure.
The person does not have to remain in memory. However, a lookup is
needed to convert the key back to a person. So instead of:

  my_person = f.get()

We would need to do:

  my_person_id = f.get()
  my_person = db.person_table.lookup(my_person_id)



-- 
Don Allingham <dallingham@...>
GRAMPS - Open Source Genealogy

John Stoffel <stoffel@...>, Thu 30 Oct 2003 09:55 -0500:

> Why would we want to keep the XML DB around after 1.0 realease?  It's
> going to be a big enough change that I think it should be just dropped
> completely.  

In my opinion, an XML database is useful if and only if gramps 1.0 is 
not shipped with an internal, real database.  For small researches (I'm 
working with 1500 individuals and I'm fine with gramps), a separate 
package may discourage new, unskilled users.

On the other hand, two db backends are more difficult to maintain.


-- 
email: lorenzo.cappelletti@...
Jabber: lolo@...
Fingerprint: 8CDD 3408 53B2 6122 99DA EE37 1523 68FC D906 4C08

Don Allingham <dallingham@...>, Wed 29 Oct 2003 17:22 -0700:

>    need them. The person/family objects are going to have to be
>    fixed. Currently, families keep a list of people, and people
>    keep lists of families. This will have to be changed so that 

I'm not an expert in databases either, but I think you've got two 
solutions here:

1) A list of individuals: a family table has got the columns such as ID, 
father, mother, child1, child2, child3,...  This solution implies a 
fixed number of child columns which may waste space if too high or be 
problematic if too low.

2) Table of individual-family pairs: there are three tables, one for the 
families, one for the individuals, and one that binds individuals to 
families.  The latter has got only two columns: IndividualID and 
FamilyID.  This way, a family may have any number of individuals and an 
individual may belong to an arbitral number of families.  Because a real 
database engine is optimized for searches like these, performance should 
be very high.

-- 
email: lorenzo.cappelletti@...
Jabber: lolo@...
Fingerprint: 8CDD 3408 53B2 6122 99DA EE37 1523 68FC D906 4C08