modeling-users

[Modeling-users] How the transactions are done in the saveChanges?

[Marco B. <m.b...@ic...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all.

I'm experimenting with ModelingFramework, and I've some questions.

I've tried the sample model in from the website, and I've analyzed the
database logs. I've noticed that the ec.saveChanges() in the model is
composed of 5 transactions: the first 4 transactions get the key numbers
from the sequence, the last one is the transaction actually doing the work.

Is this a correct behaviour? What are the reasons behind it?

Also, I've noticed that the ec.fetch() is a transaction which terminates
with an abort. Is this correct?

I've not yet experimented with Zope. I see I can connect the
ec.saveChanges() to the transactions machinery in Zope. Some questions:

1) how many db transaction are done for each saveChanges (i.e. same as
above?)

2) how are the db transaction executed/completed inside a Zope
transaction? In a non-zope application, I can see each fetch() produces
a transaction, as well as the ec.saveChanges(). Is this true also in
Zope applications?

Thanks for the support.

Regards
Marco
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBGRKfXhfyAQQVoaIRApwhAJ4gN0VvFmGwjVMGbcWPNJX8//7UuwCfWz23
ms4DuwY4buaqGurMlFB2yYg=
=fnEH
-----END PGP SIGNATURE-----

Re: [Modeling-users] How the transactions are done in the saveChanges?

[Sebastien B. <sbi...@us...>]

Marco Bizzarri wrote:
> Hi all.
> 
> I'm experimenting with ModelingFramework, and I've some questions.
> 
> I've tried the sample model in from the website, and I've analyzed the 
> database logs. I've noticed that the ec.saveChanges() in the model is 
> composed of 5 transactions: the first 4 transactions get the key 
> numbers from the sequence, the last one is the transaction actually 
> doing the work.
> 
> Is this a correct behaviour? What are the reasons behind it?

The first 4 transactions get the PK value from the sequence; in fact, you'll
notice that there are as many transactions for obtaining PK values as there are
newly inserted objects that will be saved.

  Is this correct? It is, even if it's inefficient --this is a TODO item, PKs
values could be obtained in a single roundtrip to the database!

By the way, the PK values are obtained before any change is saved, just because
saving changes requires that those PKs values are known (because of
relationships).  Now if ec.saveChanges() fails, those PKs values won't be used,
ever.

  So, in theory, there are two transactions, one for obtaining the
  new primary keys values (in practice you get N transactions there),
  the other one for actually saving the changes.

BTW: if you set the env. variable 'MDL_ENABLE_DATABASE_LOGGING' then you'll get
a copy of the framework db-related actions on stderr (or anywhere else if you
modify Modeling.logging) --see the appropriate annex in the User's Guide.

> Also, I've noticed that the ec.fetch() is a transaction which 
> terminates with an abort. Is this correct?

This is intentional, anyhow ;)  Such a transaction should be terminated, and I
cannot see any reason why it should be COMMITed, hence it is ABORTed!  But if
you think differently, please let us know!

> I've not yet experimented with Zope. I see I can connect the
> ec.saveChanges() to the transactions machinery in Zope. Some
> questions:
> 
> 1) how many db transaction are done for each saveChanges (i.e. same as
> above?)

Yes.

> 2) how are the db transaction executed/completed inside a Zope 
> transaction? In a non-zope application, I can see each fetch() 
> produces a transaction, as well as the ec.saveChanges(). Is this true 
> also in Zope applications?

Yes it is.  The zope support basically consists in providing a default
EditingContext per session, possibly also connecting this EC's
saveChanges() to the transaction machinery (i.e. to each request/response loop).
 That's all, and everything else will be handled in your ZProducts' code, just
the same way you would make it in pure python.

	Hope this helps,

-- Sébastien.

Re: [Modeling-users] How the transactions are done in the saveChanges?

[Marco B. <m.b...@ic...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sebastien Bigaret wrote:
...

|
| BTW: if you set the env. variable 'MDL_ENABLE_DATABASE_LOGGING' then
you'll get
| a copy of the framework db-related actions on stderr (or anywhere else
if you
| modify Modeling.logging) --see the appropriate annex in the User's Guide.
|

Ok, I will take a look

|>Also, I've noticed that the ec.fetch() is a transaction which
|>terminates with an abort. Is this correct?
|
|
| This is intentional, anyhow ;)  Such a transaction should be
terminated, and I
| cannot see any reason why it should be COMMITed, hence it is ABORTed!
  But if
| you think differently, please let us know!
|

One of the basic reason for this is in the debugging/maintenance of the
system.

When you need to look for the DB log, it is useful to spot the problems
by mean of ABORTed transactions. If the aborted transactions are the
standard, this can be a problem.

|
|>2) how are the db transaction executed/completed inside a Zope
|>transaction? In a non-zope application, I can see each fetch()
|>produces a transaction, as well as the ec.saveChanges(). Is this true
|>also in Zope applications?
|
|
| Yes it is.  The zope support basically consists in providing a default
| EditingContext per session, possibly also connecting this EC's
| saveChanges() to the transaction machinery (i.e. to each
request/response loop).
|  That's all, and everything else will be handled in your ZProducts'
code, just
| the same way you would make it in pure python.
|
| 	Hope this helps,
|
| -- Sébastien.

Yes and no... I would prefer to be able to tell ModelingFramework when
it should start/stop a transaction on the db... howevr, thanks of the
answer...
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBHHJxXhfyAQQVoaIRAogVAKC4xp1v+Ttxp6wmjxWotOpYPtvq6gCdHIyC
ric78zFY9j+Ifbc5eoVMilE=
=hFeZ
-----END PGP SIGNATURE-----

Re: [Modeling-users] How the transactions are done in the saveChanges?

[Sebastien B. <sbi...@us...>]

        Hi,

Marco Bizzarri wrote:
[...]
> Sebastien Bigaret wrote:
> |>Also, I've noticed that the ec.fetch() is a transaction which 
> |>terminates with an abort. Is this correct?
> |
> |
> | This is intentional, anyhow ;)  Such a transaction should be
> | terminated, and I cannot see any reason why it should be COMMITed, 
> | hence it is ABORTed! But if you think differently, please let us 
> | know!
> 
> One of the basic reason for this is in the debugging/maintenance of 
> the system.
> 
> When you need to look for the DB log, it is useful to spot the 
> problems by mean of ABORTed transactions. If the aborted transactions 
> are the standard, this can be a problem.

Understood, this seems reasonable. I think this could be controlled by
an option / env. variable / whatever

> |>2) how are the db transaction executed/completed inside a Zope 
> |>transaction? In a non-zope application, I can see each fetch() 
> |>produces a transaction, as well as the ec.saveChanges(). Is this 
> |>true also in Zope applications?
> |
> | Yes it is.  The zope support basically consists in providing a
> | default EditingContext per session, possibly also connecting this 
> | EC's saveChanges() to the transaction machinery (i.e. to 
> | request/response loop).  That's all, and everything else will be 
> | handled in your ZProducts' code, just the same way you would make it 
> | in pure python.
> |
> | 	Hope this helps
> 
> Yes and no... I would prefer to be able to tell ModelingFramework when 
> it should start/stop a transaction on the db... howevr, thanks of the 
> answer...

Could you be a little more explicit on your use-case(s)?
I'm curious here, and I'd like to better understand what your needs are
/ could be in terms of finer control on transactions.

-- Sébastien.

Re: [Modeling-users] How the transactions are done in the saveChanges?

[Marco B. <m.b...@ic...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sebastien Bigaret wrote:
|         Hi,
|
| Marco Bizzarri wrote:
[...]
|>When you need to look for the DB log, it is useful to spot the
|>problems by mean of ABORTed transactions. If the aborted transactions
|>are the standard, this can be a problem.
|
|
| Understood, this seems reasonable. I think this could be controlled by
| an option / env. variable / whatever
|

That would be great :)


|>|>2) how are the db transaction executed/completed inside a Zope
|>|>transaction? In a non-zope application, I can see each fetch()
|>|>produces a transaction, as well as the ec.saveChanges(). Is this
|>|>true also in Zope applications?
|>|
|>| Yes it is.  The zope support basically consists in providing a
|>| default EditingContext per session, possibly also connecting this
|>| EC's saveChanges() to the transaction machinery (i.e. to
|>| request/response loop).  That's all, and everything else will be
|>| handled in your ZProducts' code, just the same way you would make it
|>| in pure python.
|>|
|>| 	Hope this helps
|>
|>Yes and no... I would prefer to be able to tell ModelingFramework when
|>it should start/stop a transaction on the db... howevr, thanks of the
|>answer...
|
|
| Could you be a little more explicit on your use-case(s)?
| I'm curious here, and I'd like to better understand what your needs are
| / could be in terms of finer control on transactions.

Up to now we have used SQL transactions inside the Zope TM. This means
that, for each request/response, you've just one transaction, which is
either commited or aborted.

The Modeling Framework, on the other hand, would perform several
transactions on RDBMS,

This could be a problem since I could see, inside a transaction,
possibily different values for the same row. Of course I could decide to
manage directly these issues, but I would prefer to have them handled by
the Modeling Framework.

Let's say I've two classes, Customer and Order. The association is one
to many from Customer to Order.

Now let's say I'm retrieve a Customer C, using an ec.fetch(),  which
causes an RDBMS transaction. After this, another thread commits
something, modifying both the Customer and its orders. I then retrieve
the Orders, which is another RDBMS transaction. Data on the order are up
to date, but not on the Customer.

Actually, I would not having to deal with RDBMS transactions at all.
Instead, I would prefer to see a single BEGIN/COMMIT, or BEGIN/ABORT for
each of my Zope Transaction.

| -- Sébastien.

Regards
Marco
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBIz2DXhfyAQQVoaIRAj4FAJ9Lf7OMeTO9GhdfVmGUkzSL5tVbTwCdGuKF
50IZki3s5k7I8Vsf+NjmmqI=
=ZzDr
-----END PGP SIGNATURE-----

Re: [Modeling-users] How the transactions are done in the saveChanges?

[Sebastien B. <sbi...@us...>]

Marco Bizzarri wrote:
> |>When you need to look for the DB log, it is useful to spot the 
> |>problems by mean of ABORTed transactions. If the aborted 
> |>transactions are the standard, this can be a problem.
> |
> | Understood, this seems reasonable. I think this could be 
> | controlled by an option / env. variable / whatever
> |
> 
> That would be great :)

Ok, we're going that way then.  Could you submit a RFE @sf.net?
That would make it sure I won't forget...

[...]
> | Could you be a little more explicit on your use-case(s)?
> | I'm curious here, and I'd like to better understand what your needs 
> | are / could be in terms of finer control on transactions.
> 
> Up to now we have used SQL transactions inside the Zope TM. 
> This means that, for each request/response, you've just one 
> transaction, which is either commited or aborted.
> 
> The Modeling Framework, on the other hand, would perform 
> several transactions on RDBMS,

Right, and the main reason is that they do not work at the same level.
The Zope TM usually directly "impacts" rdbms databases.
The MDL framework on the other side works at the object level: you 
make whatever changes you need to make on a graph of objects, then
you save them as a bunch (and here, obviously,inside a  single 
transaction) --that's the saveChanges() stuff.  And if you have
a situation where there can be several saveChanges() *before* the 
decision is made on whether they should be permanently saved into
the database, then the parent/child EC configuration is for you
(you can consider such configurations as sub-transactions at the
object level).

  Okay, enough with that --it was probably off-topic here, 
  but I took the opportunity to make it clear, just in case ;)

> 
> This could be a problem since I could see, inside a 
> transaction, possibily different values for the same row. Of 
> course I could decide to manage directly these issues, but I 
> would prefer to have them handled by the Modeling Framework.
> 
> Let's say I've two classes, Customer and Order. The 
> association is one to many from Customer to Order.
> 
> Now let's say I'm retrieve a Customer C, using an ec.fetch(), 
>  which causes an RDBMS transaction. After this, another 
> thread commits something, modifying both the Customer and its 
> orders. I then retrieve the Orders, which is another RDBMS 
> transaction. Data on the order are up to date, but not on the 
> Customer.

Okay, I think I see the point now, I'll try to rephrase it, you'll
correct me if I misunderstood.

First, let's consider you have a single transaction per thread: you 
probably agree that a problem remains --different problem, but still
a problem:

  a. thread-1 fetches customer (and leave the transaction opened)
  b. thread-2 modifies customers and orders
  c. thread-1 fetches orders, not seeing changes made in b.

Then, what should thread-1 do? (ok, the "not seeing changes made in b."
actually depends on the db, the cursor's isolation level, ...)

You may not have the problem of inconsistency between data fetched
during step a. and c., but still there remains the problem of answering
the question "what should thread-1 do?"  Refresh the (possibly already 
modified) data fetched during step a.?  Discard any modifications and 
retry the whole stuff?  Or else...?

> Actually, I would not having to deal with RDBMS transactions 
> at all. Instead, I would prefer to see a single BEGIN/COMMIT, 
> or BEGIN/ABORT for each of my Zope Transaction.

  As far as I can understand, your problem is not with transactions, but
with inconsistencies that can occur in data retrieved from the db when it
is concurrently accessed and modified, is that right?

  Currently (as of release 0.9pre17), each EC currently ignores the
  changes done by others.  This means, in particular, that if ec1 and ec2
  fetch then save the same object, the one that saves its changes last
  overrides the other one changes.

Now if you want every EC to be notified (hence, at the object level) of
not only _when_ an EC saves its changes, but also _what_ the changes are,
then the 'optimistic locking' patch available at @sf.net is for you :)

--> Search the archives, we discussed this in detail not a long time ago,
I believe it was w/ Duncan and Ernesto, and as far as I remember Duncan
uses it successfully since it's been posted.
(the patch will probably be included in the next release)

  BTW, the way an EC is notified of changes is not documented neither in
the messages nor in the patch, but it's there; and the default behaviour
this patch proposes is, roughly: ec1 and ec2 fetch() an object, ec2
modifies then saves it, ec1 gets the modifications, applies them then
re-applies the changes that has been made in its own scope, if 
appropriate.

  Could such a mechanism be of some help? I guess so, if I did not 
misunderstood your point, since in particular it would offer an 
automatic way of being notified of changes concurrently made so that
a decision can be made on what should be done (possibly depending on
the situation, on which objects are involved, etc.)

> Actually, I would not having to deal with RDBMS transactions 
> at all. Instead, I would prefer to see a single BEGIN/COMMIT, 
> or BEGIN/ABORT for each of my Zope Transaction.

Back on this, just a few words: having a single BEGIN/COMMIT or ABORT
per zope transaction means that you want to restrict what the 
framework is able to do. Updates in an EC can take more than one zope
transaction to be achieved, for example if they are saved after several
html pages have been served one after the other (think of a wizard, at
the end of which you confirm or cancel the changes).

  I can understand that you feel uncomfortable with several 
db-transactions occuring during a single zope transaction, but now if
the inconsistencies can be solved at the object level with full control
of what should be done (or a reasonable default behaviour), would the
discomfort eventually remain?


-- Sébastien.

Re: [Modeling-users] How the transactions are done in the saveChanges?

[Marco B. <m.b...@ic...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Pfeeww... I didn't expect such a looong answer... :)

Let's see..

Sebastien Bigaret wrote:
| Marco Bizzarri wrote:
|
|>|>When you need to look for the DB log, it is useful to spot the
|>|>problems by mean of ABORTed transactions. If the aborted
|>|>transactions are the standard, this can be a problem.
|>|
|>| Understood, this seems reasonable. I think this could be
|>| controlled by an option / env. variable / whatever
|>|
|>
|>That would be great :)
|
|
| Ok, we're going that way then.  Could you submit a RFE @sf.net?
| That would make it sure I won't forget...

https://sourceforge.net/tracker/index.php?func=detail&aid=1011515&group_id=58935&atid=489338

| [...]
|
|>| Could you be a little more explicit on your use-case(s)?
|>| I'm curious here, and I'd like to better understand what your needs
|>| are / could be in terms of finer control on transactions.
|>
|>Up to now we have used SQL transactions inside the Zope TM.
|>This means that, for each request/response, you've just one
|>transaction, which is either commited or aborted.
|>
|>The Modeling Framework, on the other hand, would perform
|>several transactions on RDBMS,
|
|
| Right, and the main reason is that they do not work at the same level.
| The Zope TM usually directly "impacts" rdbms databases.
| The MDL framework on the other side works at the object level: you
| make whatever changes you need to make on a graph of objects, then
| you save them as a bunch (and here, obviously,inside a  single
| transaction) --that's the saveChanges() stuff.  And if you have
| a situation where there can be several saveChanges() *before* the
| decision is made on whether they should be permanently saved into
| the database, then the parent/child EC configuration is for you
| (you can consider such configurations as sub-transactions at the
| object level).
|
|   Okay, enough with that --it was probably off-topic here,
|   but I took the opportunity to make it clear, just in case ;)
|
|

Yes, but I think there is another point (please forgive me if I'm
repeating the same things you're saying).

In a web environment, like Zope or whatever, you've a clear distinction
of a transaction: the arrive of a request, the sending of response. All
inside these two events can be considered a transaction; at least, this
is what the user can perceive as a transaction.

In non zope (I would say non web based) programming, the boundaries of
the transactions are more difficult to be pointed. Indeed, the
boundaries are of two kinds:

1) read transactions, each having its own RDB transaction;
2) write transactions, which are directly activated by the developer
(with saveChanges())


[...]

|
| Okay, I think I see the point now, I'll try to rephrase it, you'll
| correct me if I misunderstood.
|
| First, let's consider you have a single transaction per thread: you
| probably agree that a problem remains --different problem, but still
| a problem:
|
|   a. thread-1 fetches customer (and leave the transaction opened)
|   b. thread-2 modifies customers and orders
|   c. thread-1 fetches orders, not seeing changes made in b.
|
| Then, what should thread-1 do? (ok, the "not seeing changes made in b."
| actually depends on the db, the cursor's isolation level, ...)

Ok, I'm assuming serialization isolation level. Inside a single
transaction, I think you will end having some serialization error (it
depends on your changes, but you should).

| You may not have the problem of inconsistency between data fetched
| during step a. and c., but still there remains the problem of answering
| the question "what should thread-1 do?"  Refresh the (possibly already
| modified) data fetched during step a.?  Discard any modifications and
| retry the whole stuff?  Or else...?

Yes, this is a problem, and, as you correctly stated, this is a
different. problem.

Actually, this is a classic "business transaction problem", and it is
not limited to the case we are discussing. A tipical would be:

1) user 1 reads Customer C, and has a page to modify it.
2) user 2 actually modifies it
3) user 1 modifies the page and submits the page.

What should be done? First of all, we would have to detect the problem,
and this can be done with different methods (version numbers,
timestamps, etc.)

Then, the application will inform the user, who will have to decide what
to do.

|
|>Actually, I would not having to deal with RDBMS transactions
|>at all. Instead, I would prefer to see a single BEGIN/COMMIT,
|>or BEGIN/ABORT for each of my Zope Transaction.
|
|
|   As far as I can understand, your problem is not with transactions, but
| with inconsistencies that can occur in data retrieved from the db when it
| is concurrently accessed and modified, is that right?

Yes, it is.

|   Currently (as of release 0.9pre17), each EC currently ignores the
|   changes done by others.  This means, in particular, that if ec1 and ec2
|   fetch then save the same object, the one that saves its changes last
|   overrides the other one changes.
|
| Now if you want every EC to be notified (hence, at the object level) of
| not only _when_ an EC saves its changes, but also _what_ the changes are,
| then the 'optimistic locking' patch available at @sf.net is for you :)
|
| --> Search the archives, we discussed this in detail not a long time ago,
| I believe it was w/ Duncan and Ernesto, and as far as I remember Duncan
| uses it successfully since it's been posted.
| (the patch will probably be included in the next release)

I will take a look at the patch, thanks :)

|   BTW, the way an EC is notified of changes is not documented neither in
| the messages nor in the patch, but it's there; and the default behaviour
| this patch proposes is, roughly: ec1 and ec2 fetch() an object, ec2
| modifies then saves it, ec1 gets the modifications, applies them then
| re-applies the changes that has been made in its own scope, if
| appropriate.
|
|   Could such a mechanism be of some help? I guess so, if I did not
| misunderstood your point, since in particular it would offer an
| automatic way of being notified of changes concurrently made so that
| a decision can be made on what should be done (possibly depending on
| the situation, on which objects are involved, etc.)

I think I should review a little... consider an application where you
have a large number of users, 1000, perhaps, each having its own ec...
should we notify each user of the changes?



|
|>Actually, I would not having to deal with RDBMS transactions
|>at all. Instead, I would prefer to see a single BEGIN/COMMIT,
|>or BEGIN/ABORT for each of my Zope Transaction.
|
|
| Back on this, just a few words: having a single BEGIN/COMMIT or ABORT
| per zope transaction means that you want to restrict what the
| framework is able to do. Updates in an EC can take more than one zope
| transaction to be achieved, for example if they are saved after several
| html pages have been served one after the other (think of a wizard, at
| the end of which you confirm or cancel the changes).

I don't think this is possible in the current integration of MDL and Zope.

IIRC, MDL register itself with the _finish method...

~  def _finish(self):
~    self.ec.lock()
~    self.ec.saveChanges()
~    self.ec.unlock()

Now, this means that, at the end of *each* Zope transaction you will do
a saveChanges on your ec. Therefore, if you update your objects inside
the page of the wizard, they will be stored on the RDB.

Also, even though you could in principle do some saveChanges() by
yourself, this would be looking for havoc. Indeed, Zope relies on the
fact that a transaction can be redone from the start, if something goes
wrong.

|   I can understand that you feel uncomfortable with several
| db-transactions occuring during a single zope transaction, but now if
| the inconsistencies can be solved at the object level with full control
| of what should be done (or a reasonable default behaviour), would the
| discomfort eventually remain?

I've to see the Optimistic lock patch before answering.

I confess I'm probably too much Zope-oriented. However, I would explore
this possibility:

1) when a first fetch is done on an EC, the EC, if it has not done yet,
opens a RDB transaction;

2) whenever you have a saveChanges(), the EC either opens and then
commits the transaction, or simply commit the transaction.

The above behaviour could be configurable. You could have an
"auto-commit" EC and a "non-auto-commit" EC (i.e. different subclasses),
and let the developer choose according to its need.

|
|
| -- Sébastien.

Thanks for your support.

Regards
Marco
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBI31yXhfyAQQVoaIRAgx6AJ9WzgmDLTEYWQw8H6mL64MdhvGn/gCfdb4y
Gel67mlAS0o1pQvgEr+3hvQ=
=iAGJ
-----END PGP SIGNATURE-----

Re: [Modeling-users] How the transactions are done in the saveChanges?

[Sebastien B. <sbi...@us...>]

        Hi,

To all: I'll probably be off for a day or two from now, so 
do not expect me to answer quickly in this period ;)

Marco Bizzarri wrote:
> Pfeeww... I didn't expect such a looong answer... :)

This will be very short one, before a more complete in the coming days.
I just want to correct something quickly:

[...]
> In a web environment, like Zope or whatever, you've a clear 
> distinction of a transaction: the arrive of a request, the 
> sending of response. All inside these two events can be 
> considered a transaction; at least, this is what the user can 
> perceive as a transaction.
[...]
> | Back on this, just a few words: having a single 
> BEGIN/COMMIT or ABORT 
> | per zope transaction means that you want to restrict what the 
> | framework is able to do. Updates in an EC can take more 
> than one zope 
> | transaction to be achieved, for example if they are saved after 
> | several html pages have been served one after the other (think of a 
> | wizard, at the end of which you confirm or cancel the changes).
> 
> I don't think this is possible in the current integration of 
> MDL and Zope.
> 
> IIRC, MDL register itself with the _finish method...
> 
> ~  def _finish(self):
> ~    self.ec.lock()
> ~    self.ec.saveChanges()
> ~    self.ec.unlock()
> 
> Now, this means that, at the end of *each* Zope transaction 
> you will do a saveChanges on your ec. Therefore, if you 
> update your objects inside the page of the wizard, they will 
> be stored on the RDB.


No, a session's EC is NOT bound by default to the zope TM, 
(even if the _finish() method is used when it is bound to it)
cf. section 6.4. & 6.4.1 in the User's Guide.  If you observed
the oppposite this is definitely a bug that should be fixed.

So, basically, you should get exactly what you describe:
> The above behaviour could be configurable. You could have an 
> "auto-commit" EC and a "non-auto-commit" EC (i.e. different 
> subclasses), and let the developer choose according to its need.

based on the ZEditingContextSessioning's property 
'bind_saveChanges_to_zope_transactions'. It IS possible to 
decorrelate EC.saveChanges() and the zope TM, it should even
be the default (or, I repeat, this is definitely a bug).

[...]
> Also, even though you could in principle do some 
> saveChanges() by yourself, this would be looking for havoc. 
> Indeed, Zope relies on the fact that a transaction can be 
> redone from the start, if something goes wrong.

Zope does not insist to rely on this if you insist of ignoring
its TM ;) 
Seriously, it's possible to build a whole application on this principle.
E.g. you can design a whole application after the MVC model, with the
V/C part being handled by zpt/python Products and the M part being handled
by plain pure-python modules --which saveChanges() at some point, 
completely decorrelated from the zope transaction machinery.  And the whole
zope machinery is then just used for its "regular" application server
(serving pages, load-balancing between threads, handling sessions, etc.)

> I think I should review a little... consider an application 
> where you have a large number of users, 1000, perhaps, each 
> having its own ec... should we notify each user of the changes?

Wow, I only meant notifying the ECs, not the users ;) unless you decide
so, obviously. 

To be clear, in my language, notifications are only a way to inform
the  different ECs so that a (programmatic) decision can be made on what
should be done (merge changes, discard changes, etc.) --see the 
Notification Framework for details!  Just because something has to be
done, even if this "something" ultimately means "do nothing/ignore the
changes" in some situations :)

  Hopefully this succeeds some of my points in the previsou email
  a little clearer. Again, yes, this would be a lot clearer with a tutorial,
  a zope dedicated section and a sample toy application...

-- Sébastien.

Re: [Modeling-users] How the transactions are done in the saveChanges?

[Marco B. <m.b...@ic...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sebastien Bigaret wrote:
|         Hi,
|
| To all: I'll probably be off for a day or two from now, so
| do not expect me to answer quickly in this period ;)

No problem :)

| Marco Bizzarri wrote:
|
|>Pfeeww... I didn't expect such a looong answer... :)
|
|
| This will be very short one, before a more complete in the coming days.
| I just want to correct something quickly:
|
| [...]
|
|>In a web environment, like Zope or whatever, you've a clear
|>distinction of a transaction: the arrive of a request, the
|>sending of response. All inside these two events can be
|>considered a transaction; at least, this is what the user can
|>perceive as a transaction.
|
| [...]
|
|>| Back on this, just a few words: having a single
|>BEGIN/COMMIT or ABORT
|>| per zope transaction means that you want to restrict what the
|>| framework is able to do. Updates in an EC can take more
|>than one zope
|>| transaction to be achieved, for example if they are saved after
|>| several html pages have been served one after the other (think of a
|>| wizard, at the end of which you confirm or cancel the changes).
|>
|>I don't think this is possible in the current integration of
|>MDL and Zope.
|>
|>IIRC, MDL register itself with the _finish method...
|>
|>~  def _finish(self):
|>~    self.ec.lock()
|>~    self.ec.saveChanges()
|>~    self.ec.unlock()
|>
|>Now, this means that, at the end of *each* Zope transaction
|>you will do a saveChanges on your ec. Therefore, if you
|>update your objects inside the page of the wizard, they will
|>be stored on the RDB.
|
|
|
| No, a session's EC is NOT bound by default to the zope TM,
| (even if the _finish() method is used when it is bound to it)
| cf. section 6.4. & 6.4.1 in the User's Guide.  If you observed
| the oppposite this is definitely a bug that should be fixed.

No, I was assuming the bind_saveChanges_to_zope_transactions behaviour.

| So, basically, you should get exactly what you describe:
|
|>The above behaviour could be configurable. You could have an
|>"auto-commit" EC and a "non-auto-commit" EC (i.e. different
|>subclasses), and let the developer choose according to its need.
|
|
| based on the ZEditingContextSessioning's property
| 'bind_saveChanges_to_zope_transactions'. It IS possible to
| decorrelate EC.saveChanges() and the zope TM, it should even
| be the default (or, I repeat, this is definitely a bug).
|
| [...]
|
|>Also, even though you could in principle do some
|>saveChanges() by yourself, this would be looking for havoc.
|>Indeed, Zope relies on the fact that a transaction can be
|>redone from the start, if something goes wrong.
|
|
| Zope does not insist to rely on this if you insist of ignoring
| its TM ;)

I disagree.

Zope itself uses a conflict resolution approach, where, aside from
interactions with the RDB, you can have ConflictErrors, which are
raised, catched by the ZPublisher which redo the transaction from the
beginning.

| Seriously, it's possible to build a whole application on this principle.

[...]

This is a point which is very interesting to me. However, I think you
could be very limited on what you can use inside Zope... ZCatalogs, for
example, should be exluded, because they can easily generate
ReadConflictError.

However, I suspect you have much more experience on this topic than me,
and I would appreciate your point of view.

|
|>I think I should review a little... consider an application
|>where you have a large number of users, 1000, perhaps, each
|>having its own ec... should we notify each user of the changes?
|
|
| Wow, I only meant notifying the ECs, not the users ;) unless you decide
| so, obviously.
|
[...]

Of course, my wording was not that good. If I have 1000 users active, I
have 1000 sessions, with 1000 EC. I could have to notify to 999 EC when
one makes a change... or not?

|
|   Hopefully this succeeds some of my points in the previsou email
|   a little clearer. Again, yes, this would be a lot clearer with a
tutorial,
|   a zope dedicated section and a sample toy application...

I would be happy to contribute something on this point...

| -- Sébastien.

Regards
Marco
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBJFkLXhfyAQQVoaIRAt3DAJ9hGUCP0LE821psO9UZSQip8dsVBQCfcwYX
0crnvRdxieb384XpvloS+fg=
=j+Jq
-----END PGP SIGNATURE-----

Re: [Modeling-users] How the transactions are done in the saveChanges?

[Sebastien B. <sbi...@us...>]

        Hi,

  Since this discussion was pending for a while, I leave here almost the
  whole the context

Marco Bizzarri <m.b...@ic...> wrote:
> Sebastien Bigaret wrote:
> |         Hi,
> |
> | To all: I'll probably be off for a day or two from now, so
> | do not expect me to answer quickly in this period ;)
>=20
> No problem :)
>=20
> | Marco Bizzarri wrote:
> |
> |>Pfeeww... I didn't expect such a looong answer... :)
> |
> |
> | This will be very short one, before a more complete in the coming days.
> | I just want to correct something quickly:
> |
> | [...]
> |
> |>In a web environment, like Zope or whatever, you've a clear
> |>distinction of a transaction: the arrive of a request, the
> |>sending of response. All inside these two events can be
> |>considered a transaction; at least, this is what the user can
> |>perceive as a transaction.
> |
> | [...]
> |
> |>| Back on this, just a few words: having a single
> |>BEGIN/COMMIT or ABORT
> |>| per zope transaction means that you want to restrict what the
> |>| framework is able to do. Updates in an EC can take more
> |>than one zope
> |>| transaction to be achieved, for example if they are saved after
> |>| several html pages have been served one after the other (think of a
> |>| wizard, at the end of which you confirm or cancel the changes).
> |>
> |>I don't think this is possible in the current integration of
> |>MDL and Zope.
> |>
> |>IIRC, MDL register itself with the _finish method...
> |>
> |>~  def _finish(self):
> |>~    self.ec.lock()
> |>~    self.ec.saveChanges()
> |>~    self.ec.unlock()
> |>
> |>Now, this means that, at the end of *each* Zope transaction
> |>you will do a saveChanges on your ec. Therefore, if you
> |>update your objects inside the page of the wizard, they will
> |>be stored on the RDB.
> |
> |
> |
> | No, a session's EC is NOT bound by default to the zope TM,
> | (even if the _finish() method is used when it is bound to it)
> | cf. section 6.4. & 6.4.1 in the User's Guide.  If you observed
> | the oppposite this is definitely a bug that should be fixed.
>=20
> No, I was assuming the bind_saveChanges_to_zope_transactions behaviour.

Okay, fine.

> | So, basically, you should get exactly what you describe:
> |
> |>The above behaviour could be configurable. You could have an
> |>"auto-commit" EC and a "non-auto-commit" EC (i.e. different
> |>subclasses), and let the developer choose according to its need.
> |
> |
> | based on the ZEditingContextSessioning's property
> | 'bind_saveChanges_to_zope_transactions'. It IS possible to
> | decorrelate EC.saveChanges() and the zope TM, it should even
> | be the default (or, I repeat, this is definitely a bug).
> |
> | [...]
> |
> |>Also, even though you could in principle do some
> |>saveChanges() by yourself, this would be looking for havoc.
> |>Indeed, Zope relies on the fact that a transaction can be
> |>redone from the start, if something goes wrong.
> |
> |
> | Zope does not insist to rely on this if you insist of ignoring
> | its TM ;)
>=20
> I disagree.
>=20
> Zope itself uses a conflict resolution approach, where, aside from
> interactions with the RDB, you can have ConflictErrors, which are
> raised, catched by the ZPublisher which redo the transaction from the
> beginning.
>=20
> | Seriously, it's possible to build a whole application on this principle.
>=20
> [...]
>=20
> This is a point which is very interesting to me. However, I think you
> could be very limited on what you can use inside Zope... ZCatalogs, for
> example, should be exluded, because they can easily generate
> ReadConflictError.
>=20
> However, I suspect you have much more experience on this topic than me,
> and I would appreciate your point of view.

  Well, AFAIK (Read)ConflictErrors are handled within each storage and
  do not interfere with others --unless that, after the transaction has
  been retried n times, it is abandoned and makes the whole transaction
  fail.

However I can be wrong on this point and have no time to double-check
what I say, so if you have informations suggesting or proving the
contrary I'd be happy to hear from you.


  What I was saying with:

> | Zope does not insist to rely on this if you insist of ignoring
> | its TM ;)

  is that you can build a whole application without binding the DB txns
  (ec.saveChanges()) to zope txns  --and in fact that's exactly what I
  do.  This mean, in particular, that I never worked w/ applications
  where ec.saveChanges() are bound to the zope txn machinery, so I
  cannot be absolutely positive about what I'm saying above.

But some documents, such as TM.py or:
http://www.zope.org/Members/petrilli/WritingADA


  Now, I must double-check the design in ZECSessioning, I think I've
  just see a flaw in it: it should probably implement method tpc_vote()
  calling self.ec.validateChanges(), before any txns is actually
  committed.  Any comment, anyone?


> |>I think I should review a little... consider an application
> |>where you have a large number of users, 1000, perhaps, each
> |>having its own ec... should we notify each user of the changes?
> |
> |
> | Wow, I only meant notifying the ECs, not the users ;) unless you decide
> | so, obviously.
> |
> [...]
>=20
> Of course, my wording was not that good. If I have 1000 users active, I
> have 1000 sessions, with 1000 EC. I could have to notify to 999 EC when
> one makes a change... or not?

  Yes, you do have to --if you don't, then you can get inconsistencies.
  Of course, the notification process is quite fast, and the appropriate
  methods (solving inconsistencies) are only triggered when needed.


> |   Hopefully this succeeds some of my points in the previsou email a
> |   little clearer. Again, yes, this would be a lot clearer with a >
> |   tutorial, a zope dedicated section and a sample toy application...
>=20
> I would be happy to contribute something on this point...

Great :)

-- S=E9bastien.