modeling-users

[Modeling-users] Warning: possible problem with Zope, maybe MT-related

[Sebastien B. <sbi...@us...>]

        Hi all,

  Just a quick note to warn you about the fact that I currently have a very
  annoying bug revealing itself on a zope-based application using the
  modeling.

  It is still not clear where the bug is, but it has been reported that it was
  also observed with zope and ZPsycopgDA alone.

It might be a MT-issue, or just a Zope-issue (this is just my guess at this
point).

  Description of the problem: at some point, the psycopg adaptor stops adding
  BEGIN and END or ABORT at the begin/end of the txn.
  From that point, none of the sql statements sent to the pg server is
  committed anymore, hence they are lost.

It do not have any other solution than monitoring the application and
restarting it when it happens --ok, this is not a solution, rather an ugly
workaround.

My initial post is at:
http://lists.initd.org/pipermail/psycopg/2003-March/001885.html

  I'll post here if and when a solution will be found.

-- Sebastien.

Re: [Modeling-users] Warning: possible problem with Zope, maybe MT-related

[Sebastien B. <sbi...@us...>]

        Hi,

>   Just a quick note to warn you about the fact that I currently have a
>   very annoying bug revealing itself on a zope-based application using
>   the modeling.
>=20
>   It is still not clear where the bug is, but it has been reported
>   that it was also observed with zope and ZPsycopgDA alone.
>=20
> It might be a MT-issue, or just a Zope-issue (this is just my guess at
> this point).
>=20
>   Description of the problem: at some point, the psycopg adaptor stops
>   adding BEGIN and END or ABORT at the begin/end of the txn.  From
>   that point, none of the sql statements sent to the pg server is
>   committed anymore, hence they are lost.

It seems that the problem is triggered when the same psycopg cursor is
used in two different threads.

  I submitted and fixed bug #699272 for the modeling framework: when
  calling EditingContext.objectsCountWithFetchSpecification() the
  underlying adaptorChannel was not closed at the end. Hence, even if
  MDL_PERMANENT_DB_CONNECTION was unset, the corresponding
  (psycopg-)connection to the database was not closed. This makes it
  possible for the same cursor to be reused by an other request,
  especially in a different thread (when used within zope e.g.) and as a
  consequence it made the bug more likely to occur.

--> If you are using the framework in a MT environment, please update
--> your copy of DatabaseChannel w/ v.1.10 or apply the patch attached
--> below. Additionally, please remember that you are strongly advised
--> against setting MDL_PERMANENT_DB_CONNECTION in a multi-threaded
--> environment.

  This does not guarantee that the problem won't show up, but it will
  dramatically reduce its frequency. Hopefully a complete resolution for
  the bug will be found, I'll keep you informed.

        Regards,

-- S=E9bastien.



------------------------------------------------------------------------
retrieving revision 1.9
retrieving revision 1.10
diff -r1.9 -r1.10
32c32
<     $Id: DatabaseChannel.py,v 1.9 2003/02/20 11:59:52 sbigaret Exp $
---
>     $Id: DatabaseChannel.py,v 1.10 2003/03/07 09:27:18 sbigaret Exp $
36c36
< __version__=3D'$Revision: 1.9 $'[11:-2]
---
> __version__=3D'$Revision: 1.10 $'[11:-2]
292c292
<       self.__isFetchInProgress=3D0
---
>       self.cancelFetch()
------------------------------------------------------------------------

Re: [Modeling-users] Warning: possible problem with Zope, maybe MT-related

[Jerome K. <Jer...@fi...>]

On Fri, Mar 07, 2003 at 10:53:08AM +0100, Sebastien Bigaret wrote:
> 
> It seems that the problem is triggered when the same psycopg cursor is
> used in two different threads.


This is really sound strange as, i know zope use medusa asyncore, and 
this avoid the use of thread (for the http..) so how could you achieve
to have several thread using the same db connection. ? 


Is there someone else using modeling over zope ? cause i read severals
post over last weeks about the lack on this kind of things in zope 
(mainly #plone ) and the zope community seems to try different ways
as CMFTypes and Adaptable Storage. 


/Soaf which is going to test modeling over mysql :) 

Re: [Modeling-users] Warning: possible problem with Zope, maybe MT-related

[Sebastien B. <sbi...@us...>]

Jerome Kerdreux <Jer...@fi...> writes:
> On Fri, Mar 07, 2003 at 10:53:08AM +0100, Sebastien Bigaret wrote:
> >=20
> > It seems that the problem is triggered when the same psycopg cursor=
 is
> > used in two different threads.
>=20
>=20
> This is really sound strange as, i know zope use medusa asyncore, and=
=20
> this avoid the use of thread (for the http..) so how could you achieve
> to have several thread using the same db connection. ?=20
>=20

  You made me doubt about this, but I checked and Zope does use threads
  indeed. I included at the end the short list of modules I visited to
  make sure: Zope uses the asyncore to dispatch its request (and, right,
  the dispatch itself is done without forking & w/ no thread, but with
  select()), and these requests are then handled by a dedicated number
  of threads. There's also a nice explanation of this mechanism at
  http://lists.initd.org/pipermail/psycopg/2003-March/001906.html but
  beware! There is a guru wearing a warm cloak of invisibility (+2)
  roaming around! ;)

> Is there someone else using modeling over zope ? cause i read severals
> post over last weeks about the lack on this kind of things in zope=20
> (mainly #plone ) and the zope community seems to try different ways
> as CMFTypes and Adaptable Storage.=20

  I'm using it, but that you already know. And yes, the way the
  framework can be used along w/ zope is still to be documented.

> /Soaf which is going to test modeling over mysql :)=20

  Great! I'll wait for your reports then.

        Cheers,

-- S=E9bastien.



------------------------------------------------------------------------
Zope uses threads (possible errors of interpretation of code are all
                   mine):

z2.py
        from ZServer import setNumberOfThreads
        setNumberOfThreads(NUMBER_OF_THREADS)
        [...]
        if HTTP_PORT:
          [...]
          hs =3D zhttp_server( [...]

ZServer/HTTPServer.py:=20
    zhttp_server derives from http_server, and uses zhttp_channel for
    its channel_class

    zhttp_channel itself derives from http_channel, and uses
    ZServer.PubCore.handle in its work() method

Last, ZServer.PubCore.__init__ declares handle as
ZRendezvous.ZRendevous(_n).handle, where ZRendeVous is declared in
ZServer.PubCore.ZRendezvous and creates a pool of threads:

(and where _n is assigned by setNumberOfThreads() above in z2.py for
option '-t n': "The number of threads to use, if ZODB3 is used.")

        class ZRendevous:
            def __init__(self, n=3D1):
                [...]
                while n > 0:
                    l=3Dthread.allocate_lock()
                    l.acquire()
                    pool.append(l)
                    thread.start_new_thread(ZServerPublisher,=20
                                            (self.accept,))

------------------------------------------------------------------------