modeling-users

[Modeling-users] Some more questions

[Ernesto R. <er...@si...>]

Hi again,

Queries with large resultsets:
I remember I read about this but I don't see where. What is returned for =
a fetch with a result set of, say, 10000 rows? A list? Or an iterator? =
Is there a concept like database cursors, which iterate over objects =
(wherever they are stored), instead of database rows?

Multi-user environment:
I read the deployment instructions of the user doc. Under SQLite, when =
two separe editing context (either different applications or one =
application but different editing context) save changes to the same =
object, the latter one overwrites silently the earlier ones, which is =
not that nice. Does this behave the same way with Postgres? And with =
mySQL?

Grand Totals, Counters, etc.
Say I have a heavily accesed table and I want to have grand totals per =
month. What is the best way to accomplish this? (The problem is that if =
I do this within the domain class, it has to read and update the total =
with added/updated/deleted data, but during these operations other =
people may also hace to need to update the same field, causing =
collisions or overwriting. If there was a 'add' operation which would be =
translated to SQL (field=3Dfiel+increment), this problem would =
disappear, as it would be executed inside a transaction. Also, the grand =
total should perhaps be a volatile field and no cached, that is, =
whenever we access entity.getGrandTotal() it would be read from the =
database.) In databases with triggers, this is easily resolved inside =
the DB, but we still need volatile attributes.

Simple question:
can an editingContext access (fetch and update) classes of different =
models at the same time?
(I think this is true, but still haven't tried it. It can also treat =
heterogenous data sources, although there may be no relations between =
different models, right?)

Would it be possible to create a Wiki? Wouldn't it be the right place to =
put questions like these?

best regards,
Erny

Re: [Modeling-users] Some more questions

[Jerome K. <Jer...@fi...>]

Ernesto Revilla wrote:

>Hi again,
>
>Queries with large resultsets:
>I remember I read about this but I don't see where. What is returned for a fetch with a result set of, say, 10000 rows? A list? Or an iterator? Is there a concept like database cursors, which iterate over objects (wherever they are stored), instead of database rows?
>  
>
You can use both . either fetch using Ec ( via relationships..) so you 
will get a list of object
or using fetchRow and you will get a dict per row ( which isn't exactly 
why the Modeling is done for )

>Multi-user environment:
>I read the deployment instructions of the user doc. Under SQLite, when two separe editing context (either different applications or one application but different editing context) save changes to the same object, the latter one overwrites silently the earlier ones, which is not that nice. Does this behave the same way with Postgres? And with mySQL?
>  
>
2 things :
- SQLite isn't a multiuser DB. you can't open the db twice. so in 
Modeling open / fetch / close is the circle i
   think . that's why you get this .
- MySQL / Postgres and others can handle this smoothly. but again take 
care that Modeling have some caching
  functions (on fetch) that can drive you in the wall if you doesn't 
take care about that.
  here a sample
  authors = ec.fetchWith(....)
  will retrieve the authors from the DB .. and build Authors object w/
   now do authors[0].setName('Jean Pascal')
   ec.saveChanges()
   This will change the right row .. etc . but if on another host (or 
even another program ) you do the
    same thing as the same time, you won't get the right result cause 
 authors is in the cache . so the second program won't see the changes 
until the first one call ec.saveChanges() . and worst case happen . since
authors is in cache so won't be retrieve at second call ..

This really can be a severe issue, you can avoid this by using Nested EC 
.. or doing some ec.dispose ( with all the drawback this generate )


>Grand Totals, Counters, etc.
>Say I have a heavily accesed table and I want to have grand totals per month. What is the best way to accomplish this? (The problem is that if I do this within the domain class, it has to read and update the total with added/updated/deleted data, but during these operations other people may also hace to need to update the same field, causing collisions or overwriting. If there was a 'add' operation which would be translated to SQL (field=fiel+increment), this problem would disappear, as it would be executed inside a transaction. Also, the grand total should perhaps be a volatile field and no cached, that is, whenever we access entity.getGrandTotal() it would be read from the database.) In databases with triggers, this is easily resolved inside the DB, but we still need volatile attributes.
>  
>
Last time i work w/ the Modeling i got this kind of trouble. We use a 
manual SQL lock on a db table to ensure that other EC ( running on other 
hosts) won't try to access the DB at the same time. Pay caution that you 
CAN'T lock a Modeling  table as you don't know when modeling will access 
this (when fetch occur) so we used a empty table for locking only. for 
example we work on a table STATS we got a STATS_LOCK table which is SQL 
manually lock/unlock between statistics update.

>Simple question:
>can an editingContext access (fetch and update) classes of different models at the same time?
>(I think this is true, but still haven't tried it. It can also treat heterogenous data sources, although there may be no relations between different models, right?)
>  
>
It should yes.. but haven't tested . Sebastien should know that.

>Would it be possible to create a Wiki? Wouldn't it be the right place to put questions like these?
>  
>
Sebastien try to keep the documentation as simple / explicit as he can, 
i think. and he usually do some great job about this. now it's clear 
that he seems to have a lot of work right now, and miss some mails .


Bye Bye ..

Re: [Modeling-users] Some more questions

[Sebastien B. <sbi...@us...>]

Jerome Kerdreux <Jer...@fi...> wrote:
> Ernesto Revilla wrote:
> > Would it be possible to create a Wiki? Wouldn't it be the right
> > place to put questions like these?
> >
> Sebastien try to keep the documentation as simple / explicit as he can, i
> think. and he usually do some great job about this. now it's clear that he
> seems to have a lot of work right now, and miss some mails .

That's right, I can miss some emails and I sometimes forget about
specific issues posted here, just because it runs out of my mind --just
like the issue w/ ZODB.Persistent we discussed earlier.

   As a general rule, it is a good idea, if you have some time for that,
to submit a bug report along with the post on the mailing-list (that's
what I did for the documentation issue you reported today), since bug
reports are persistent on sf.net even if they are not in my brain ;)
It's definitely a great help for me to remember what should be fixed.

  Now back on the wiki: as far I know sf.net does not offer such a
  possibility and I currently do not have the time (and the money) to
  add such things to the project's page. Frankly, even if I had a place,
  I'd need someone to manage the wiki to keep it organized and tidy, I
  can't see how I could find the time for this. If any of you have
  suggestions, or want to take the lead on this, please speak, I'm
  really open to that.


-- S=E9bastien.

Re: [Modeling-users] Some more questions

[Sebastien B. <sbi...@us...>]

"Ernesto Revilla" <er...@si...> wrote:
> Hi again,
>=20
> Queries with large resultsets:
> I remember I read about this but I don't see where. What is returned
> for a fetch with a result set of, say, 10000 rows? A list? Or an
> iterator? Is there a concept like database cursors, which iterate over
> objects (wherever they are stored), instead of database rows?

A list, yes, either a list of objects or a list of dictionaries if you
fetch with 'rawRows=3D1' (which is probably what you need for such a large
number of objects).

  There are some pending works for supporting OFFSET and LIMIT, but this
  is still unfinished.

> Multi-user environment:
> I read the deployment instructions of the user doc. Under SQLite, when
> two separe editing context (either different applications or one
> application but different editing context) save changes to the same
> object, the latter one overwrites silently the earlier ones, which is
> not that nice. Does this behave the same way with Postgres? And with
> mySQL?

The current default database policy is: no locking, which basically
means that no checks are done, ever. I do not know what you would need
in such a case, but maybe optimistic locking could do the
trick. Optimistic locking means: if the data has changed between the
moment you fetched an object, and the moment you save it, than raise
(and then you get the opportunity to ask the user/refresh the
data/whatever).
The last option would be pessimistic locking, which means that as long
as an object lives in an EC, its db row remains locked.

  Pessimistic locking is a really strong requirement and I'm not going
  into this unless there is a real need. However optimistic locking is a
  natural expectation and this could be added sooner than (I!) expected
  if there is a real need for this (and even better, if you plan to
  experiment and test it in various ways).

Last, as Soaf said, there are currently solutions involving using nested
ECs, and/or locking special tables by hand for that purpose, but all
these tricks are just what they are: tricks. But beware with sqlite,
it's probably not mt-safe (gotta check however).

> Grand Totals, Counters, etc.
> Say I have a heavily accesed table and I want to have grand totals per
> month. What is the best way to accomplish this? (The problem is that
> if I do this within the domain class, it has to read and update the
> total with added/updated/deleted data, but during these operations
> other people may also hace to need to update the same field, causing
> collisions or overwriting. If there was a 'add' operation which would
> be translated to SQL (field=3Dfiel+increment), this problem would
> disappear, as it would be executed inside a transaction. Also, the
> grand total should perhaps be a volatile field and no cached, that is,
> whenever we access entity.getGrandTotal() it would be read from the
> database.) In databases with triggers, this is easily resolved inside
> the DB, but we still need volatile attributes.

That's a too long questions to answer now; I'll answer later, probably
this evening.

> Simple question:
> can an editingContext access (fetch and update) classes of different
> models at the same time?
> (I think this is true, but still haven't tried it. It can also treat
> heterogenous data sources, although there may be no relations between
> different models, right?)

Absolutely, as many models can be loaded at the same time, the only
conditions is that two entities in the models cannot have the same name.

  Ok, I've got to go back to work now, I'll be back this evening.

        Regards,

-- S=E9bastien.



> Would it be possible to create a Wiki? Wouldn't it be the right place
> to put questions like these?

=20=20

Re: [Modeling-users] Some more questions

[Sebastien B. <sbi...@us...>]

  Back on an unanswered questions of yours:

"Ernesto Revilla" <er...@si...> wrote:
> Hi again,
>=20
> Grand Totals, Counters, etc.
> Say I have a heavily accesed table and I want to have grand totals per
> month. What is the best way to accomplish this? (The problem is that
> if I do this within the domain class, it has to read and update the
> total with added/updated/deleted data, but during these operations
> other people may also hace to need to update the same field, causing
> collisions or overwriting. If there was a 'add' operation which would
> be translated to SQL (field=3Dfiel+increment), this problem would
> disappear, as it would be executed inside a transaction. Also, the
> grand total should perhaps be a volatile field and no cached, that is,
> whenever we access entity.getGrandTotal() it would be read from the
> database.) In databases with triggers, this is easily resolved inside
> the DB, but we still need volatile attributes.

Due to time constraint tonight I won't exhaust all possibilities, but
basically you get it right: if you know how to do this with triggers
(which I surely the best way to achieve this in an highly-stressed db),
then all that you need is a way to get the actual data in the db, not a
cached version of your data.

  Right, this is yet unsupported, however this is planned, something
  like a 'refresh=3Dtrue' parameter in method ec.fetch().

This makes me think that this feature and the other one we discussed
today, Optimistic Locking, requires that EditingContexts get notified of
updates saved by an other EC, and updates their data accordingly. This
is the very basis for both of these features. Now I need priority. As I
said earlier, I'm currently working on dynamically instanciating
modules/classes from a (py)model, either classic-style or new-style
(with metaclasses). I can change priorities of tasks at demand, given
that this demand is loudly expressed ;) As always, users' requests
usually take precedence on my own willingness to code this or that, I
just need to know your timeframe and when you think you'll need this.

        Regards,

-- S=E9bastien.