On Wed, Dec 30, 2009 at 12:10 PM, Gerald Britton <gerald.britton@gmail.com> wrote:

> 3/The, 'use 7 blob tables in sql' suggestion of Gerald. That goes
> against all sql usage (storage optimization, columns, ...). If you
> want that, you should use bsddb :-).

It's not a matter of what is wanted, but what is the best solution.
It is clear to me from the discussion in this thread that the choice
of an SQL implementation and the Django framework has directly led to
unacceptable performance levels.  I have not measured the differences
in storage use, but are we talking about an order of magnitude or just
a few percentage points?

What lead to unacceptable performance was attempting to use gen.lib to get all of the parts of an object. At the topmost layer in the Django interface is a query to select the relevant people, perhaps something like:

people = Person.objects.filter(...some crtieria based on search string private settings etc...)

Then using that to populate the view with an extended gen.lib-based DbDjango, further wrapped in proxies:

for person in people:

Using the relational tables, each call to get_person_from_handle executes at least 30 queries per person, and depending on the proxies involved, can be much more (300 per person).

How bad is this? Professional websites would expect somewhere between a few queries and up to maybe 15 or 30 total queries per view. However, trying to use gen.lib turns every disk access into a query. There are better ways where you can get all of the related data in one query, but that is a query-driven, top-down data access method. gen.lib is a bottom-up access method.
SQL should not be an end in itself.  It is a tool, that's all.  Let's
choose the best tool for the job...and yes, that means a tool that
performs well while causing the least collateral damage.

>Most importantly, see 1/, one
> would not be able to use a framework for the html side, which is the
> reason the gramps-connect is already somewhat usable.

I cannot see how the choice of the storage access method would
influence the choice of HTML framework.  There are plenty of other
choices out there for Python developers.

In Django, one gets an integrated HTML framework with the database models. For example, one can make a form from a model, and models know how to create HTML widgets on forms.

We feel that we've picked the best tool for the job, and we are working towards "collateral damage" that will actually improve DbBsdb as well.

Gerald Britton

This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon's best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
Gramps-devel mailing list