Re: [Quickfix-developers] QuickFIX Python and Twisted

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

This sounds reasonable enough to me, but I'm not familiar with python threading.

What happens if the python callbacks are long running (i.e. SQL
query). Would it block the main quickfix loop in this case?

On 11/15/06, Oren Miller <or...@qu...> wrote:
> To get around the 1 bytecode assignment for long running C methods,
> Python provides a couple of macros.  Py_BEGIN_ALLOW_THREADS and
> Py_END_ALLOW_THREADS which are to be called before and after your
> long running method respectively.  This would normally work well
> enough, but like Reggie said, QuickFIX is calling into Python, so
> doing this could be dangerous since the QuickFIX thread could end up
> accessing python resources when it essentially promised it wouldn't.
>
> I think perhaps the problem can be resolved this way.
>
> When calling start, in C++ we would do what we normally do when
> calling a long running method.  Something like...
>
> Py_BEGIN_ALLOW_THREADS
>    Call C++ start method
> Py_END_ALLOW_THREADS
>
> At this point the only danger is when QuickFIX delegates to the
> python callbacks.  So I would think we could do the opposite when
> entering a callback.
>
> Py_END_ALLOW_THREADS
>    Invoke Python callback
> Py_BEGIN_ALLOW_THREADS
>
> Anybody see a reason why this would not work?
>
> --oren
>
> On Nov 15, 2006, at 2:36 PM, Robert Parrott wrote:
>
> > ]That's very helpful information, Reggie.
> >
> > Are there any suggestions from the quickfix developers on how one
> > might address this threading issue?
> >
> > On 11/15/06, Reggie Dugard <re...@me...> wrote:
> >> Rob,
> >>
> >> I attempted to do almost exactly the same thing with almost
> >> exactly the
> >> same results.  I believe the problem starts with Python threading
> >> behavior:
> >>
> >>       * The Python Global Interpreter Lock (GIL) allows only one
> >> python
> >>         thread to be running at any point in time.
> >>       * Python will execute 10 bytecode instructions from a
> >> particular
> >>         thread and then release the GIL allowing other threads the
> >>         opportunity to run and acquire the GIL.
> >>       * A call to a C/C++ extension function is 1 bytecode
> >>
> >> Due to this behavior I believe that Python is calling into the QF C++
> >> code and then staying there without ever giving Python the
> >> opportunity
> >> to release the GIL and allow your twisted thread to run.  I
> >> thought of
> >> trying to edit the quickfix.i file to get swig to generate code to
> >> release the GIL before calling QF, but I realize I don't have enough
> >> SWIG/C++ knowledge to do it.  One of the problems I had is that
> >> the SWIG
> >> generated code sometimes calls back into python within a method so
> >> you
> >> can't just release the GIL arbitrarily before calling any QF
> >> function.
> >>
> >> Unfortunately I have yet to get Twisted and QuickFix integrated in
> >> the
> >> same application.  Sorry I couldn't be of more help, but if you
> >> find a
> >> solution to this I'd appreciate it if you could pass it along.
> >>
> >> -Reggie
> >>
> >>
> >>
> >> On Wed, 2006-11-15 at 13:41 -0500, Robert Parrott wrote:
> >>> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/
> >>> doc/html/index.html
> >>> QuickFIX Support: http://www.quickfixengine.org/services.html
> >>>
> >>> HI Folks,
> >>>
> >>> I'm trying to  build a simply gateway based on QuickFIX with the
> >>> python bindings, and the Twisted framework (http://
> >>> twistedmatrix.com).
> >>> I'm finding some kind of threading issue.
> >>>
> >>> My aim is to wrap the quickfix engine with a simple RPC
> >>> container, so
> >>> that one can connect to the daemon and invoke calls like "start",
> >>> "stop", "subscribe to data" etc. The twisted framework provides an
> >>> asynchronous networking layer to do so.
> >>>
> >>> However things seem to lock up when I invoke the
> >>> SocketInitiator.start() method. I can start up the service, and
> >>> create
> >>> the QuickFIX application, and  setup and configure the various
> >>> QuickFIX components, ala:
> >>>
> >>>  class MyQFService:
> >>>   def __init__
> >>>         settings     = quickfix.SessionSettings(configFile)
> >>>         application  = MyQFClient(connStr)
> >>>         storeFactory = quickfix.OdbcStoreFactory(settings)
> >>>         logFactory   = quickfix.OdbcLogFactory(settings)
> >>>
> >>>         self.quickfixSettings = settings
> >>>         self.quickfixApplication = application
> >>>         self.quickfixStoreFactory = storeFactory
> >>>         self.quickfixLogFactory = logFactory
> >>>
> >>>         self.quickfixInitiator = quickfix.SocketInitiator
> >>> (application,
> >>> storeFactory, settings, logFactory)
> >>>
> >>>  When a remote "START" command is issued, the main thread of the
> >>> twisted code calls the SocketInitiator to start the QuickFIX thread:
> >>>
> >>>   def startQF(self):
> >>>         log.msg("QuickFIX Engine starting.")
> >>>         self.quickfixInitiator.start()
> >>>         log.msg("QuickFIX Engine started.")
> >>>         return defer.succeed("STARTED")
> >>>
> >>> At this point, the quickfix engine thread starts, and I see logon,
> >>> heartbeats, and other admin messages. The log message immediately
> >>> after the start() call executes, and returns correctly. However, the
> >>> main thread of the Twisted application seems to block entirely,
> >>> and I
> >>> don't get any response from the application on subsequent RPC
> >>> attempts. At this point the appliction also stops responding to
> >>> Ctrl-C
> >>> on the command line, when before it would gracefully terminate.
> >>>
> >>> It seems as if the quickfix thread has taken over, or more likely is
> >>> completely blocking the main twisted thread. Any thoughts on what
> >>> might be happening here? My first guess is that it has to do with
> >>> logging, but I can't be sure.
> >>>
> >>> thanks for any insight,
> >>> rob
> >>>
> >>> --------------------------------------------------------------------
> >>> -----
> >>> Take Surveys. Earn Cash. Influence the Future of IT
> >>> Join SourceForge.net's Techsay panel and you'll get the chance to
> >>> share your
> >>> opinions on IT & business topics through brief surveys - and earn
> >>> cash
> >>> http://www.techsay.com/default.php?
> >>> page=join.php&p=sourceforge&CID=DEVDEV
> >>> _______________________________________________
> >>> Quickfix-developers mailing list
> >>> Qui...@li...
> >>> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> >> --
> >> Reggie Dugard <re...@me...>
> >> Merfin, LLC
> >>
> >>
> >>
> >
> > ----------------------------------------------------------------------
> > ---
> > Take Surveys. Earn Cash. Influence the Future of IT
> > Join SourceForge.net's Techsay panel and you'll get the chance to
> > share your
> > opinions on IT & business topics through brief surveys - and earn cash
> > http://www.techsay.com/default.php?
> > page=join.php&p=sourceforge&CID=DEVDEV
> > _______________________________________________
> > Quickfix-developers mailing list
> > Qui...@li...
> > https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> >
>
>