Re: [Modeling-users] (LONG) Other OR-bridges?

[Mario R. <ma...@ru...>]

Thanks for the new core release. Pygresql seems to work just fine with=20=

it.
However there are some problems running the tests, which i will mail=20
about
shortly. But before, please see my comments to some points below.

...

>> Having to write XML instead of an excel file is not a major issue --
>> but it is less comfortable (but maybe the ZModel tool is even more
>> comfortable?)
>
>   Sure, using the ZModeler is much, much more comfortable than editing
>   an xml-file. But I understand installing and launching Zope just for
>   that can be a burden --I'm currently looking for an alternate
>   solution.

Yes, but this should not be considered high priority.
I feel it is more important to expose (to document) the interaction=20
between the
different layers (tool - model description (xml) - python & sql).=20
Specifically,
documentation for the XML format, documentation of the constraints on=20
the
generated python classes and on the constraints on the generated sql=20
for the
tables, and the utility or code to generate the python classes and the=20=

sql to
init and/or update the database with the corresponding tables. As I=20
understand,
generation of python classes and sql statements is only available via=20
the Z tool
at the moment? (Apologies beforehand if this info is in the doc already=20=

and
I have missed it...)

Given this, it is quite likely that others may come up with such=20
tools...
For example it could be quite simple to, borrowing the CSV idea,=20
generate
your XML from a spreadsheet... opening up the possibility to use any old
spreadsheet program as a basic model editor. Or, if the XML will not map
nicely to CSV, some other cooked editor for this XML doctype.

>> For large queries, can either of the frameworks do paging?  In
>> Modeling you can get the count, but, if the count is big, can you
>> specify to fetch N objects at a time (respecting the original
>> qualifiers, and sort, if any) ?
>>
>> Can you sort? (This is done in the store or in the db (much faster))
>
>   (I've no idea of what are the MiddleKit's capabilities on these=20
> points)

To my knowledge there is absolutely no such chunking management in
MiddleKit, but this question was no longer in the context of a=20
comparison between
the two. Such chunking management becomes very useful when=20
user-interfacing
to large sets of data.

As for sort, MK does it on the DB side, by specifying sql order by=20
clauses.
However do not know if that implies refetching all the objects each=20
time (imagine
a typical interface showing a list of items, and clicking on a column=20
heading to
sort those items...) , or simply merging the sort order with the=20
already loaded
objects.

>   For both questions, wrt. the Modeling, answer is: not yet.
>
>   1. Work on paging has not began yet,

Again, this would only be needed for large datasets... so later. But it=20=

is good
to foresee as many limitations as possible, and their possible=20
solutions.

>   2. Modeling.SortOrdering is specifically designed for sorting=20
> purpose,
>      to be used along with Qualifiers in FetchSpecifications, but for
>      the moment it can only be used for in-memory sorts.

This could be an optimization later -- instead of sorting on the store=20=

it might
be faster to do a db query to get only the sort info (get only the id?)=20=

and then iterate
on that sort order using the data in the store (or getting it when=20
missing).

>> I must say that this is a problem! I have not yet managed to install
>> Modeling because i could get neither pgdb nor psycopg installed on my
>> OSX machine, as for some reason they will not compile (am following =
it
>> up on the psycopg list).  PyPgSQL is installed however... how
>> difficult is it to add support to another adaptor such as pypgsgl?
>
>   Not *that* difficult ;) I released 0.8.1 today, adding support for
>   pypgsql. BTW, I am *very* interesting in hearing from your and=20
> others'
>   experiences with the modeling and MacOS X (and windows as well) =
since
>   the only OS I have at hand is linux.

Great, thanks a lot for adding support for PyPgSQL. More soon... \-;

...

>   Being able to fetch raw rows that can be later converted to objects=20=

> is
>   indeed on the todo list (no ETA, still: it highly depends on users'
>   requests :).

OK...

> [...]
>> So you mean you would have an objectstore cached in each user =
session?
>> (Very memory expensive?)
>> Or a shared application-level object store that all users that fetch
>> would go through? (To modify they will have to refetch into a new=20
>> store?
>> Or this is where the new feature you link to about nested contexts
>> comes in?)
>
> Sure, both designs are possible, that's an architectural choice. Back=20=

> to
> your questions:
>
>   1. [an ObjectStore in each session] yes, since each object store=20
> holds
>      its own copies of objects, this can be memory expensive. Future
>      features related to that topic includes:
>
>        - trying to maintain the number of objects per ObjectStore =
under
>          a given, user-defined, limit (I say ``trying'', because when
>          all objects in an object store are modified it is definitely
>          not possible to reduce the memory footprint)

This can be very important for web applications with many open user
sessions, all possibly accessing different datasets.

>        - having a shared ObjectStore where mostly-read-only objects
>          lives and are shared by all other object store. This is =
called
>          a Shared Editing Context in the EOF.

Could be powerful in combination with previous feature -- all requests
for view-only data are serviced by a common read-only object store,=20
while
all modify-data requests are serviced by a (short-lived) user store that
could be independent from read-only store, or could be a nested store
to it.  The cost of refetching objects that are to be modified is much=20=

less
critical that refetching objects for reading -- ***most*** of the time
data is browsed for reading.

>   2. Now what if you decide to have a shared app.-level object store?
>      The first consequence is that any modifications made by one =
users,
>      including non-committed ones, will be seen by others ; by the =
way,
>      modifications are made as usual on objects and do not require any
>      re-fetch.

Yes, this would not be the correct way to go.

>> Or this is where the new feature you link to about nested contexts
>> comes in?
>
>   It will, to a certain extend. I suppose you're thinking of having a
>   shared (root) object store, with one child (nested) object store
>   e.g. created for each user when modifications are about to be done.=20=

> In
>   this configuration, changes will not be seen by other users until
>   changes made in the child are committed back to the parent (the
>   parent/child configuration will really bring transactional abilities
>   to the sets of objects). Last, fetching objects in a child object
>   store that were already fetched in a parent obj.store is just as
>   expensive as refetching already fetched objects in a single=20
> obj.store:
>   definitely not as expensive as the first fetch, as shown in my
>   previous message.

Again, cost of fetching is (much) less important here. In addition,=20
normally data
is modified in small units, typically 1 object at a time. But when=20
browsing, it is
common to request fetches of 1000s of objects...

>     However, even in the still-uncommited changes, the framework lacks
>   an important feature which is likely to be the next thing I'll work
>   on: the ability to broadcast the changes committed on one object=20
> store
>   to the others (in fact, the notifications are already made, but they
>   are not properly handled yet).

Yes, this would be necessary. Not sure how the broadcast model will work
(may not always know who the end clients are -- data on a client web=20
page
will never know about these changes even if you broadcast). But,=20
possibly
a feasible simple strategy could be that each store will keep a copy of=20=

the
original object as it received it. When committing a modify it compares=20=

the
original object(s) being modified to those in the db, if still=20
identical, locks the
objects and commits the changes, otherwise returns an error "object has
changed since". This behaviour could be application specific, i.e. in=20
some
cases you want to go ahead and modify anyway, and in others you want to
do something else. Thus, this could be a store option, with default=20
being the
error.

In combination with this, other stores may still be notified of=20
changes, and it
is up to them (or their configuration) whether to reload the modified=20
objects,
or to ignore the broadcasts. Otherwise, on each fetch the framework=20
could
check if any changes have taken place (may be expensive) for the=20
concerned
objects, and reload them automatically. This has the advantage of=20
covering
those cases when changes to data in the db are done by some way beyond=20=

the
framework, and thus the framework can never be sure to know about it.

> Last and although this is not directly connected to that topic, I'd=20
> like
> to make an other point more precise: the framework does not implement
> yet any locking, either optimistic or pessimistic, on DB's rows. This
> means that if other processes modify a row in the DB, an ObjectStore
> will not notice the changes and will possibly override them if the
> corresponding object has been modified in the meantime.

Yes, but locking objects should be very short lived -- only the time to=20=

commit
any changes, as mentioned above. This also makes it much less important,
and only potentially a problem when two processes try to modify the same
object at the same time.

>> How about POM?
>> Which could be interpreted as PyObjectManager, with the additional=20
>> merit
>> of retaining an implicit credit to Apple, as well as to its
>> non-american author...
>> never know, it might one day become Le Big Pomme :-)
>
> Very nice proposition, I'll think of it. Really!

Ah, very good.

mario

>         Best regards,
>
> -- S=E9bastien.