quickfix-developers — QuickFIX developers forum

Re: [Quickfix-developers] AHA! : QuickFIX Python and Twisted

[Robert P. <rob...@gm...>]

OK, so I guess I need to back out on the claim that the --with-pth
option made a difference.

I rebuilt python 2.4 with all default options, and it runs as expected.

So, all I can claim is that the default, system installed python has issues.

cheers,
rob


On 11/16/06, Rich Holm <rh...@ql...> wrote:
> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
>
> Is pth what you want?  pth isn't really preemptive threading.
>
> Cheers,
> Rich
>
>
>
> Robert Parrott wrote:
> > So I updated the verions of python I was using __from__ 2.3.4
> >
> >    > python -v
> >    Python 2.3.4 (#1, Oct 11 2006, 06:18:43)
> >    [GCC 3.4.6 20060404 (Red Hat 3.4.6-3)] on linux2
> >
> > which, from the rpm spec file, was compiled with options:
> >
> >    configure --enable-ipv6 --enable-unicode=%{unicode} --enable-shared
> >
> > To python 2.4.4 complied by hand with the --with-pth option.
> >
> > After this change, it seems that the code runs fine, with the
> > initiator and main threads working in parallel. It seems to invoke
> > callbacks fine, and I was able to telnet in and subscribe to a market
> > data feed just fine.
> >
> > As such, my guess is that it's a pthreads issue (on linux) and not a
> > python version issue, as such. I know that for Zope, in order to run
> > correctly, you need to compile python with threading enabled, so this
> > may be the same way (I know that much Zope code is written in C/C++).
> >
> > cheers,
> > rob
>
>
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>

Re: [Quickfix-developers] QuickFIX Python and Twisted

[Oren M. <or...@qu...>]

Instructions are here: http://www.quickfixengine.org/developers.html

It looks like Robert found his problem, but it would still me nice  
for additional confirmation.  Looks like you need to make sure your  
python is build with pthreads for it to work.

--oren

On Nov 16, 2006, at 1:06 PM, Rich Holm wrote:

> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/ 
> html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
>
> I do I checkout the latest SVN?
>
> Cheers,
> Rich

Re: [Quickfix-developers] AHA! : QuickFIX Python and Twisted

[Rich H. <rh...@ql...>]

Is pth what you want?  pth isn't really preemptive threading.

Cheers,
Rich



Robert Parrott wrote:
> So I updated the verions of python I was using __from__ 2.3.4
>
>    > python -v
>    Python 2.3.4 (#1, Oct 11 2006, 06:18:43)
>    [GCC 3.4.6 20060404 (Red Hat 3.4.6-3)] on linux2
>
> which, from the rpm spec file, was compiled with options:
>
>    configure --enable-ipv6 --enable-unicode=%{unicode} --enable-shared
>
> To python 2.4.4 complied by hand with the --with-pth option.
>
> After this change, it seems that the code runs fine, with the
> initiator and main threads working in parallel. It seems to invoke
> callbacks fine, and I was able to telnet in and subscribe to a market
> data feed just fine.
>
> As such, my guess is that it's a pthreads issue (on linux) and not a
> python version issue, as such. I know that for Zope, in order to run
> correctly, you need to compile python with threading enabled, so this
> may be the same way (I know that much Zope code is written in C/C++).
>
> cheers,
> rob

[Quickfix-developers] AHA! : QuickFIX Python and Twisted

[Robert P. <rob...@gm...>]

So I updated the verions of python I was using __from__ 2.3.4

    > python -v
    Python 2.3.4 (#1, Oct 11 2006, 06:18:43)
    [GCC 3.4.6 20060404 (Red Hat 3.4.6-3)] on linux2

which, from the rpm spec file, was compiled with options:

    configure --enable-ipv6 --enable-unicode=%{unicode} --enable-shared

To python 2.4.4 complied by hand with the --with-pth option.

After this change, it seems that the code runs fine, with the
initiator and main threads working in parallel. It seems to invoke
callbacks fine, and I was able to telnet in and subscribe to a market
data feed just fine.

As such, my guess is that it's a pthreads issue (on linux) and not a
python version issue, as such. I know that for Zope, in order to run
correctly, you need to compile python with threading enabled, so this
may be the same way (I know that much Zope code is written in C/C++).

cheers,
rob

Re: [Quickfix-developers] QuickFIX Python and Twisted

[Rich H. <rh...@ql...>]

I do I checkout the latest SVN?

Cheers,
Rich


Oren Miller wrote:
> The generated SWIG file is making a call to PyEval_InitThreads.  I'm 
> pretty baffled why it's not working for Robert.  I've tested it on Mac 
> OS/X and Fedora Code 4 and both seem to be working.  Under OS/X I'm 
> using Python 2.3.5 and on Fedora Python 2.4.  I've even taken Robert's 
> script and am reliably getting "snores" printed out to the screen 
> every second.  Would you be able to try out the latest from svn and 
> report your results?
>
> --oren
>
> On Nov 16, 2006, at 9:07 AM, Rich Holm wrote:
>
>> QuickFIX Documentation: 
>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>
>>
>> Are you calling PyEval_InitThreads()?
>>
>> Cheers,
>> Rich
>>
>>
>>
>> Oren Miller wrote:
>>> QuickFIX Documentation: 
>>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>>
>>> Eureka.
>>>
>>> There is a -threads parameter that can be used with -python.  It
>>> looks like it automatically does exactly what we were talking about
>>> implementing and then some.  So all I had to do was update the bash
>>> script.  Easy as can be.  I checked in the results.  Give it a try
>>> and see how it goes.
>>>
>>> --oren
>

Re: [Quickfix-developers] QuickFIX Python and Twisted

[Oren M. <or...@qu...>]

The generated SWIG file is making a call to PyEval_InitThreads.  I'm  
pretty baffled why it's not working for Robert.  I've tested it on  
Mac OS/X and Fedora Code 4 and both seem to be working.  Under OS/X  
I'm using Python 2.3.5 and on Fedora Python 2.4.  I've even taken  
Robert's script and am reliably getting "snores" printed out to the  
screen every second.  Would you be able to try out the latest from  
svn and report your results?

--oren

On Nov 16, 2006, at 9:07 AM, Rich Holm wrote:

> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/ 
> html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
>
> Are you calling PyEval_InitThreads()?
>
> Cheers,
> Rich
>
>
>
> Oren Miller wrote:
>> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/ 
>> html/index.html
>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>
>> Eureka.
>>
>> There is a -threads parameter that can be used with -python.  It
>> looks like it automatically does exactly what we were talking about
>> implementing and then some.  So all I had to do was update the bash
>> script.  Easy as can be.  I checked in the results.  Give it a try
>> and see how it goes.
>>
>> --oren

Re: [Quickfix-developers] QuickFIX Python and Twisted

[Robert P. <rob...@gm...>]

I would be interested ... I'm not having much luck with the straight,
swig-generated version.

rob

On 11/16/06, Rich Holm <rh...@ql...> wrote:
> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
>
> You also need to check if you are running on the main thread (the one that
> imported your module) before changing the thread state.  I just check
> the current
> thread id (can be done on linux and windows).  I've wrapped this in a
> helper
> class... like the Locker class.  If anyone is interested, I can send it
> along.
>
> Cheers,
> Rich
>
>
> Oren Miller wrote:
> > QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html
> > QuickFIX Support: http://www.quickfixengine.org/services.html
> >
> > We should be able to use Py_BLOCK_THREADS and Py_UNBLOCK_THREADS.
> > The _save variable will need to be global and protected.
> >
> > --oren
> >
> > On Nov 15, 2006, at 5:15 PM, Robert Parrott wrote:
> >
> >
> >> BTW, looking at these macro definitions, it isn't as easy as inserting
> >> the macros directly, since there are defined to be used in pairs,
> >> before and after. Instead, for the callbacks there will need to be
> >> either explicit code or new macros:
> >>
> >> (from ceval.h of python 2.3 )
> >>
> >> PyAPI_FUNC(void) PyEval_InitThreads(void);
> >> PyAPI_FUNC(void) PyEval_AcquireLock(void);
> >> PyAPI_FUNC(void) PyEval_ReleaseLock(void);
> >> PyAPI_FUNC(void) PyEval_AcquireThread(PyThreadState *tstate);
> >> PyAPI_FUNC(void) PyEval_ReleaseThread(PyThreadState *tstate);
> >> PyAPI_FUNC(void) PyEval_ReInitThreads(void);
> >>
> >> #define Py_BEGIN_ALLOW_THREADS { \
> >>                      PyThreadState *_save; \
> >>                      _save = PyEval_SaveThread();
> >> #define Py_BLOCK_THREADS     PyEval_RestoreThread(_save);
> >> #define Py_UNBLOCK_THREADS   _save = PyEval_SaveThread();
> >> #define Py_END_ALLOW_THREADS PyEval_RestoreThread(_save); \
> >>               }
> >>
> >
> >
> > -------------------------------------------------------------------------
> > Take Surveys. Earn Cash. Influence the Future of IT
> > Join SourceForge.net's Techsay panel and you'll get the chance to share your
> > opinions on IT & business topics through brief surveys - and earn cash
> > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> > _______________________________________________
> > Quickfix-developers mailing list
> > Qui...@li...
> > https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> >
>
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>

Re: [Quickfix-developers] QuickFIX Python and Twisted

[Rich H. <rh...@ql...>]

Are you calling PyEval_InitThreads()?

Cheers,
Rich



Oren Miller wrote:
> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Eureka.
>
> There is a -threads parameter that can be used with -python.  It  
> looks like it automatically does exactly what we were talking about  
> implementing and then some.  So all I had to do was update the bash  
> script.  Easy as can be.  I checked in the results.  Give it a try  
> and see how it goes.
>
> --oren
>
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>   

Re: [Quickfix-developers] QuickFIX Python and Twisted

[Rich H. <rh...@ql...>]

You also need to check if you are running on the main thread (the one that
imported your module) before changing the thread state.  I just check 
the current
thread id (can be done on linux and windows).  I've wrapped this in a 
helper
class... like the Locker class.  If anyone is interested, I can send it 
along.

Cheers,
Rich


Oren Miller wrote:
> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> We should be able to use Py_BLOCK_THREADS and Py_UNBLOCK_THREADS.   
> The _save variable will need to be global and protected.
>
> --oren
>
> On Nov 15, 2006, at 5:15 PM, Robert Parrott wrote:
>
>   
>> BTW, looking at these macro definitions, it isn't as easy as inserting
>> the macros directly, since there are defined to be used in pairs,
>> before and after. Instead, for the callbacks there will need to be
>> either explicit code or new macros:
>>
>> (from ceval.h of python 2.3 )
>>
>> PyAPI_FUNC(void) PyEval_InitThreads(void);
>> PyAPI_FUNC(void) PyEval_AcquireLock(void);
>> PyAPI_FUNC(void) PyEval_ReleaseLock(void);
>> PyAPI_FUNC(void) PyEval_AcquireThread(PyThreadState *tstate);
>> PyAPI_FUNC(void) PyEval_ReleaseThread(PyThreadState *tstate);
>> PyAPI_FUNC(void) PyEval_ReInitThreads(void);
>>
>> #define Py_BEGIN_ALLOW_THREADS { \
>> 			PyThreadState *_save; \
>> 			_save = PyEval_SaveThread();
>> #define Py_BLOCK_THREADS	PyEval_RestoreThread(_save);
>> #define Py_UNBLOCK_THREADS	_save = PyEval_SaveThread();
>> #define Py_END_ALLOW_THREADS	PyEval_RestoreThread(_save); \
>> 		 }
>>     
>
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>   

Re: [Quickfix-developers] Fwd: QuickFIX Python and Twisted

[Robert P. <rob...@gm...>]

OK, so I built quickfix from completely vanilla sources at SVN r1865,
and I see the same thing happening.

This must be a python version issue, if the same code is showing
different behavior like this.

Anyone else want to give it try and report back?




On 11/15/06, Robert Parrott <rob...@gm...> wrote:
> Unfortuntely, I'm unable to get it running on my end.
>
> I've grabbed the QuickfixPython.cpp file straight from SVN and
> compiled.  ( The python version is 2.3.4 RHEL, and the OS is CentOS
> 4.4 on i386, if that's at all relevant. )
>
> I run this simple client,
>
> ------------------------------------------------------------------
>
> fileName = "my.conf"
> settings   = quickfix.SessionSettings(fileName)
> application  = MyClient(connStr)
> storeFactory = quickfix.FileStoreFactory(settings)
> logFactory   = quickfix.FileLogFactory(settings)
> initiator = quickfix.SocketInitiator(application, storeFactory,
> settings, logFactory)
>
> time.sleep(3)
>
> log.msg("starting initiator...")
> initiator.start()
> log.msg("...started initiator.")
>
> log.msg("starting sleep cycle")
> while True :
>         time.sleep(1)
>         log.msg("snore")
> initiator.stop()
> -----------------------------------------------------
>
> For output, I only see this:
> 2006/11/15 21:11 EST [-] Log opened.
> 2006/11/15 21:11 EST [-] MyClient: created.
> 2006/11/15 21:11 EST [-] starting initiator...
> 2006/11/15 21:11 EST [-] ...started initiator.
> 2006/11/15 21:11 EST [-] starting sleep cycle
>
> And it hangs there indefinitely. I DO notice that if I remove the
> time.sleep(1), I see a bunch of "snore"'s, but that they stop after a
> second (same happens with just print statement).
>
> I've tried adding cout's to the QuickfixPython.cpp code, and can see
> when each swig threading macro is invoked, but the cout's in the
> wrap_Initiator_start function I don't see being invoked at all, so I'm
> at a bit of a loss.
>
> Tomorrow, I'll try to get a fresh checkout and see if I can get that
> running. Has anyone
>
> BTW, I have some modifications to quickfix.i to allow for ODBC,
> PostgreSQL, and MySQL data stores and logging, so I need to regenerate
> QuickfixPython.cpp instead of just using the one from SVN. However,
> for now I'll use the File-based store and logging.
>
> cheers,
> rob
>
> On 11/15/06, Robert Parrott <rob...@gm...> wrote:
> > I've been regenerating QuickfixPython.cpp from the quickfix.i with
> > swig 1.3.29 at each stage.
> >
> > Is this the version you're using?
> >
> >
> >
> > On 11/15/06, Oren Miller <or...@qu...> wrote:
> > > That's my fault.  I checked in a file that had some of the manual
> > > threading changes I had started on.  I removed all that stuff and it
> > > seems to work here.  Can you do another update and try again?
> > >
> > > --oren
> > >
> > > On Nov 15, 2006, at 6:34 PM, Robert Parrott wrote:
> > >
> > > > QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/
> > > > html/index.html
> > > > QuickFIX Support: http://www.quickfixengine.org/services.html
> > > >
> > > > Unfortunately, I'm finding that after this change, the initiator
> > > > thread doesn't even start when Initiator.start() is called in python.
> > > > Reverting back to the interface generated without the -threads option
> > > > generates a library that runs the initiator thread.
> > > >
> > > > Oh well, no magic bullet here, it seems.
> > >
> > >
> >
>

Re: [Quickfix-developers] Function problem in vb.net

[cstrader <cst...@cs...>]

Thanks to all.... I understand the issue now, although I still haven't =
figured out how to deal with it!  My problem, a bit off topic I suppose, =
is that although textbox.invokerequired returns false, the textbox is =
still not written to.

chuck




  ----- Original Message -----=20
  From: Brian Erst=20
  To: cstrader ; qui...@li...=20
  Sent: Wednesday, November 15, 2006 10:23 AM
  Subject: Re: [Quickfix-developers] Function problem in vb.net


  You're probably having a threading issue. The .NET framework controls =
do not handle calls to them from other threads especially well =
(sometimes they work, other times they don't).

  I'm just guessing here, but I'd guess the onCreate call is probably =
running on your main thread (you instantiate a QF Application-derived =
object, which creates the sessions, which call the "onCreate" callback), =
while the onLogout is coming from the internal QF socket thread (this is =
an asynchronous callback). The first one works because you are in the =
"main" or "GUI" thread, while the second one fails because you are on a =
different thread.

  I'm not sure how VB.NET handles this (I'm a C# guy myself), but in the =
C# world you have to check the control's "InvokeRequired" property, and =
if true, use its "Invoke" method to create a message that gets handled =
by the GUI thread.


  A quick and dirty example:

  private delegate void SessionIDDelegate(SessionID sessionID);

  public void onLogout(SessionID sessionID)
  {
      // Are we on the main GUI thread?
      if (Form1.TextBox.InvokeRequired)
      {
         // No - call this function on the main GUI thread
         Form1.TextBox.Invoke(new SessionIDDelegate(onCreate), new =
Object[]{sessionID});
      }
      else
      {
          // We're on the main GUI thread - safe to update controls
          Console.Write("ONLOGOUT");
          Form1.TextBox.Text =3D "LOGOUT";
      }
  }

  Hope this helps.

  - Brian Erst
  Thynk Software, Inc.


  ----- Original Message ----
  From: cstrader <cst...@cs...>
  To: qui...@li...
  Sent: Wednesday, November 15, 2006 8:21:13 AM
  Subject: [Quickfix-developers] Function problem in vb.net


  QuickFIX Documentation: =
http://www.quickfixengine.org/quickfix/doc/html/index.html
  QuickFIX Support: http://www.quickfixengine.org/services.html


  I have a weird problem in vb.net 2005 that is causing me major =
headaches.   The problem is that some commands given within quickfix =
functions do not execute.

  For instance,

  This sub writes to both the console and the textbox:
  Public Sub onCreate(ByVal sessionID As QuickFix.SessionID) Implements =
QuickFix.Application.onCreate

      Console.write("ONCREATE")

      Form1.TextBox1.Text =3D "ONCREATE"

  End Sub

  But although this sub writes to the console, it does NOT write to =
form1.textbox1!!!



  Public Sub onLogout(ByVal sessionID As QuickFix.SessionID) Implements =
QuickFix.Application.onLogout=20

      Console.write("ONLOGOUT")

      Form1.TextBox1.Text =3D "LOGOUT"

  End Sub

  Also, the ToApp sub is OK in this regard, but the FromApp sub does not =
print to the textbox either???

  thanks!



  chuck





    ----- Original Message -----=20
    From: John Haldi=20
    To: cstrader=20
    Sent: Wednesday, October 25, 2006 9:26 AM
    Subject: RE: Any luck?


    Its very common for NetAdmins to disable pinging from external IPs.  =
But you need to get the broker's IT guy on the phone and explain that =
you're trying to connect to their FIX engine and see if he can determine =
whether there are any permission issues preventing you from connecting.



-------------------------------------------------------------------------=
---
    From: cstrader [mailto:cst...@cs...]=20
    Sent: Wednesday, October 25, 2006 10:12 AM
    To: John Haldi
    Subject: Re: Any luck?


    No...but thanks for asking.

    If I can't ping the ip address my broker has given me, and there is =
no reason that I can think of on my end, what do you think that means?


      ----- Original Message -----=20
      From: John Haldi=20
      To: cstrader=20
      Sent: Wednesday, October 25, 2006 10:01 AM
      Subject: Any luck?




      --------------------------------------
      John Haldi
      Allagash Trading, LLC
      120 Broadway, 20th Floor
      New York, NY 10271
      212.433.3958
      jo...@al...

  =
-------------------------------------------------------------------------=

  Take Surveys. Earn Cash. Influence the Future of IT
  Join SourceForge.net's Techsay panel and you'll get the chance to =
share your
  opinions on IT & business topics through brief surveys - and earn cash
  =
http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D=
DEVDEV
  _______________________________________________
  Quickfix-developers mailing list
  Qui...@li...
  https://lists.sourceforge.net/lists/listinfo/quickfix-developers

Re: [Quickfix-developers] Fwd: QuickFIX Python and Twisted

[Robert P. <rob...@gm...>]

Unfortuntely, I'm unable to get it running on my end.

I've grabbed the QuickfixPython.cpp file straight from SVN and
compiled.  ( The python version is 2.3.4 RHEL, and the OS is CentOS
4.4 on i386, if that's at all relevant. )

I run this simple client,

------------------------------------------------------------------

fileName = "my.conf"
settings   = quickfix.SessionSettings(fileName)
application  = MyClient(connStr)
storeFactory = quickfix.FileStoreFactory(settings)
logFactory   = quickfix.FileLogFactory(settings)
initiator = quickfix.SocketInitiator(application, storeFactory,
settings, logFactory)

time.sleep(3)

log.msg("starting initiator...")
initiator.start()
log.msg("...started initiator.")

log.msg("starting sleep cycle")
while True :
	time.sleep(1)
	log.msg("snore")	
initiator.stop()
-----------------------------------------------------

For output, I only see this:
2006/11/15 21:11 EST [-] Log opened.
2006/11/15 21:11 EST [-] MyClient: created.
2006/11/15 21:11 EST [-] starting initiator...
2006/11/15 21:11 EST [-] ...started initiator.
2006/11/15 21:11 EST [-] starting sleep cycle

And it hangs there indefinitely. I DO notice that if I remove the
time.sleep(1), I see a bunch of "snore"'s, but that they stop after a
second (same happens with just print statement).

I've tried adding cout's to the QuickfixPython.cpp code, and can see
when each swig threading macro is invoked, but the cout's in the
wrap_Initiator_start function I don't see being invoked at all, so I'm
at a bit of a loss.

Tomorrow, I'll try to get a fresh checkout and see if I can get that
running. Has anyone

BTW, I have some modifications to quickfix.i to allow for ODBC,
PostgreSQL, and MySQL data stores and logging, so I need to regenerate
QuickfixPython.cpp instead of just using the one from SVN. However,
for now I'll use the File-based store and logging.

cheers,
rob

On 11/15/06, Robert Parrott <rob...@gm...> wrote:
> I've been regenerating QuickfixPython.cpp from the quickfix.i with
> swig 1.3.29 at each stage.
>
> Is this the version you're using?
>
>
>
> On 11/15/06, Oren Miller <or...@qu...> wrote:
> > That's my fault.  I checked in a file that had some of the manual
> > threading changes I had started on.  I removed all that stuff and it
> > seems to work here.  Can you do another update and try again?
> >
> > --oren
> >
> > On Nov 15, 2006, at 6:34 PM, Robert Parrott wrote:
> >
> > > QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/
> > > html/index.html
> > > QuickFIX Support: http://www.quickfixengine.org/services.html
> > >
> > > Unfortunately, I'm finding that after this change, the initiator
> > > thread doesn't even start when Initiator.start() is called in python.
> > > Reverting back to the interface generated without the -threads option
> > > generates a library that runs the initiator thread.
> > >
> > > Oh well, no magic bullet here, it seems.
> >
> >
>

Re: [Quickfix-developers] Fwd: QuickFIX Python and Twisted

[Robert P. <rob...@gm...>]

I've been regenerating QuickfixPython.cpp from the quickfix.i with
swig 1.3.29 at each stage.

Is this the version you're using?



On 11/15/06, Oren Miller <or...@qu...> wrote:
> That's my fault.  I checked in a file that had some of the manual
> threading changes I had started on.  I removed all that stuff and it
> seems to work here.  Can you do another update and try again?
>
> --oren
>
> On Nov 15, 2006, at 6:34 PM, Robert Parrott wrote:
>
> > QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/
> > html/index.html
> > QuickFIX Support: http://www.quickfixengine.org/services.html
> >
> > Unfortunately, I'm finding that after this change, the initiator
> > thread doesn't even start when Initiator.start() is called in python.
> > Reverting back to the interface generated without the -threads option
> > generates a library that runs the initiator thread.
> >
> > Oh well, no magic bullet here, it seems.
>
>

Re: [Quickfix-developers] Fwd: QuickFIX Python and Twisted

[Oren M. <or...@qu...>]

That's my fault.  I checked in a file that had some of the manual  
threading changes I had started on.  I removed all that stuff and it  
seems to work here.  Can you do another update and try again?

--oren

On Nov 15, 2006, at 6:34 PM, Robert Parrott wrote:

> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/ 
> html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Unfortunately, I'm finding that after this change, the initiator
> thread doesn't even start when Initiator.start() is called in python.
> Reverting back to the interface generated without the -threads option
> generates a library that runs the initiator thread.
>
> Oh well, no magic bullet here, it seems.

[Quickfix-developers] Fwd: QuickFIX Python and Twisted

[Robert P. <rob...@gm...>]

Unfortunately, I'm finding that after this change, the initiator
thread doesn't even start when Initiator.start() is called in python.
Reverting back to the interface generated without the -threads option
generates a library that runs the initiator thread.

Oh well, no magic bullet here, it seems.

On 11/15/06, Oren Miller <or...@qu...> wrote:
> Eureka.
>
> There is a -threads parameter that can be used with -python.  It
> looks like it automatically does exactly what we were talking about
> implementing and then some.  So all I had to do was update the bash
> script.  Easy as can be.  I checked in the results.  Give it a try
> and see how it goes.
>
> --oren
>
>

Re: [Quickfix-developers] QuickFIX Python and Twisted

[Oren M. <or...@qu...>]

Eureka.

There is a -threads parameter that can be used with -python.  It  
looks like it automatically does exactly what we were talking about  
implementing and then some.  So all I had to do was update the bash  
script.  Easy as can be.  I checked in the results.  Give it a try  
and see how it goes.

--oren

Re: [Quickfix-developers] QuickFIX Python and Twisted

[Oren M. <or...@qu...>]

I discovered this code block in the generated SWIG file.  Should be  
useful:

#if defined(SWIG_PYTHON_NO_THREADS)
#  if defined(SWIG_PYTHON_THREADS)
#    undef SWIG_PYTHON_THREADS
#  endif
#endif
#if defined(SWIG_PYTHON_THREADS) /* Threading support is enabled */
#  if !defined(SWIG_PYTHON_USE_GIL) && !defined(SWIG_PYTHON_NO_USE_GIL)
#    if (PY_VERSION_HEX >= 0x02030000) /* For 2.3 or later, use the  
PyGILState calls */
#      define SWIG_PYTHON_USE_GIL
#    endif
#  endif
#  if defined(SWIG_PYTHON_USE_GIL) /* Use PyGILState threads calls */
#    ifndef SWIG_PYTHON_INITIALIZE_THREADS
#     define SWIG_PYTHON_INITIALIZE_THREADS  PyEval_InitThreads()
#    endif
#    ifdef __cplusplus /* C++ code */
        class SWIG_Python_Thread_Block {
          bool status;
          PyGILState_STATE state;
        public:
          void end() { if (status) { PyGILState_Release(state);  
status = false;} }
          SWIG_Python_Thread_Block() : status(true), state 
(PyGILState_Ensure()) {}
          ~SWIG_Python_Thread_Block() { end(); }
        };
        class SWIG_Python_Thread_Allow {
          bool status;
          PyThreadState *save;
        public:
          void end() { if (status) { PyEval_RestoreThread(save);  
status = false; }}
          SWIG_Python_Thread_Allow() : status(true), save 
(PyEval_SaveThread()) {}
          ~SWIG_Python_Thread_Allow() { end(); }
        };
#      define SWIG_PYTHON_THREAD_BEGIN_BLOCK    
SWIG_Python_Thread_Block _swig_thread_block
#      define SWIG_PYTHON_THREAD_END_BLOCK     _swig_thread_block.end()
#      define SWIG_PYTHON_THREAD_BEGIN_ALLOW    
SWIG_Python_Thread_Allow _swig_thread_allow
#      define SWIG_PYTHON_THREAD_END_ALLOW     _swig_thread_allow.end()
#    else /* C code */
#      define SWIG_PYTHON_THREAD_BEGIN_BLOCK   PyGILState_STATE  
_swig_thread_block = PyGILState_\
Ensure()
#      define SWIG_PYTHON_THREAD_END_BLOCK     PyGILState_Release 
(_swig_thread_block)
#      define SWIG_PYTHON_THREAD_BEGIN_ALLOW   PyThreadState  
*_swig_thread_allow = PyEval_SaveTh\
read()
#      define SWIG_PYTHON_THREAD_END_ALLOW     PyEval_RestoreThread 
(_swig_thread_allow)
#    endif
#  else /* Old thread way, not implemented, user must provide it */
#    if !defined(SWIG_PYTHON_INITIALIZE_THREADS)
#      define SWIG_PYTHON_INITIALIZE_THREADS
#    endif
#  else /* Old thread way, not implemented, user must provide it */
#    if !defined(SWIG_PYTHON_INITIALIZE_THREADS)
#      define SWIG_PYTHON_INITIALIZE_THREADS
#    endif
#    if !defined(SWIG_PYTHON_THREAD_BEGIN_BLOCK)
#      define SWIG_PYTHON_THREAD_BEGIN_BLOCK
#    endif
#    if !defined(SWIG_PYTHON_THREAD_END_BLOCK)
#      define SWIG_PYTHON_THREAD_END_BLOCK
#    endif
#    if !defined(SWIG_PYTHON_THREAD_BEGIN_ALLOW)
#      define SWIG_PYTHON_THREAD_BEGIN_ALLOW
#    endif
#    if !defined(SWIG_PYTHON_THREAD_END_ALLOW)
#      define SWIG_PYTHON_THREAD_END_ALLOW
#    endif
#  endif
#else /* No thread support */
#  define SWIG_PYTHON_INITIALIZE_THREADS
#  define SWIG_PYTHON_THREAD_BEGIN_BLOCK
#  define SWIG_PYTHON_THREAD_END_BLOCK
#  define SWIG_PYTHON_THREAD_BEGIN_ALLOW
#endif

On Nov 15, 2006, at 5:15 PM, Robert Parrott wrote:

> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/ 
> html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> BTW, looking at these macro definitions, it isn't as easy as inserting
> the macros directly, since there are defined to be used in pairs,
> before and after. Instead, for the callbacks there will need to be
> either explicit code or new macros:
>
> (from ceval.h of python 2.3 )
>
> PyAPI_FUNC(void) PyEval_InitThreads(void);
> PyAPI_FUNC(void) PyEval_AcquireLock(void);
> PyAPI_FUNC(void) PyEval_ReleaseLock(void);
> PyAPI_FUNC(void) PyEval_AcquireThread(PyThreadState *tstate);
> PyAPI_FUNC(void) PyEval_ReleaseThread(PyThreadState *tstate);
> PyAPI_FUNC(void) PyEval_ReInitThreads(void);
>
> #define Py_BEGIN_ALLOW_THREADS { \
> 			PyThreadState *_save; \
> 			_save = PyEval_SaveThread();
> #define Py_BLOCK_THREADS	PyEval_RestoreThread(_save);
> #define Py_UNBLOCK_THREADS	_save = PyEval_SaveThread();
> #define Py_END_ALLOW_THREADS	PyEval_RestoreThread(_save); \

Re: [Quickfix-developers] QuickFIX Python and Twisted

[Oren M. <or...@qu...>]

We should be able to use Py_BLOCK_THREADS and Py_UNBLOCK_THREADS.   
The _save variable will need to be global and protected.

--oren

On Nov 15, 2006, at 5:15 PM, Robert Parrott wrote:

> BTW, looking at these macro definitions, it isn't as easy as inserting
> the macros directly, since there are defined to be used in pairs,
> before and after. Instead, for the callbacks there will need to be
> either explicit code or new macros:
>
> (from ceval.h of python 2.3 )
>
> PyAPI_FUNC(void) PyEval_InitThreads(void);
> PyAPI_FUNC(void) PyEval_AcquireLock(void);
> PyAPI_FUNC(void) PyEval_ReleaseLock(void);
> PyAPI_FUNC(void) PyEval_AcquireThread(PyThreadState *tstate);
> PyAPI_FUNC(void) PyEval_ReleaseThread(PyThreadState *tstate);
> PyAPI_FUNC(void) PyEval_ReInitThreads(void);
>
> #define Py_BEGIN_ALLOW_THREADS { \
> 			PyThreadState *_save; \
> 			_save = PyEval_SaveThread();
> #define Py_BLOCK_THREADS	PyEval_RestoreThread(_save);
> #define Py_UNBLOCK_THREADS	_save = PyEval_SaveThread();
> #define Py_END_ALLOW_THREADS	PyEval_RestoreThread(_save); \
> 		 }

Re: [Quickfix-developers] QuickFIX Python and Twisted

[Robert P. <rob...@gm...>]

BTW, looking at these macro definitions, it isn't as easy as inserting
the macros directly, since there are defined to be used in pairs,
before and after. Instead, for the callbacks there will need to be
either explicit code or new macros:

(from ceval.h of python 2.3 )

PyAPI_FUNC(void) PyEval_InitThreads(void);
PyAPI_FUNC(void) PyEval_AcquireLock(void);
PyAPI_FUNC(void) PyEval_ReleaseLock(void);
PyAPI_FUNC(void) PyEval_AcquireThread(PyThreadState *tstate);
PyAPI_FUNC(void) PyEval_ReleaseThread(PyThreadState *tstate);
PyAPI_FUNC(void) PyEval_ReInitThreads(void);

#define Py_BEGIN_ALLOW_THREADS { \
			PyThreadState *_save; \
			_save = PyEval_SaveThread();
#define Py_BLOCK_THREADS	PyEval_RestoreThread(_save);
#define Py_UNBLOCK_THREADS	_save = PyEval_SaveThread();
#define Py_END_ALLOW_THREADS	PyEval_RestoreThread(_save); \
		 }

On 11/15/06, Robert Parrott <rob...@gm...> wrote:
> Makes sense. And since the point was to use an asynchronous threaded
> package like twisted, these callbacks can be made to be non-blocking.
>
> So these changes would need to be made in the quickfix.i code?
>
> rob
>
>
> On 11/15/06, Oren Miller <or...@qu...> wrote:
> > The allow threads macros I believe were explicitly designed for
> > blocking IO calls, you can read mode about it here: http://
> > www.gmu.edu/mlnavbar/webdev/doc-python/api/threads.html
> >
> > I would think that an IO library like one that does database queries
> > is most likely aware of this and will implement these macros.  As
> > long as they do, it should not be a problem since we called the end
> > macro, they would be able to call the begin macro, then the end
> > macro, allowing us to call the begin macro again.
> >
> > --oren
> >
> > On Nov 15, 2006, at 3:01 PM, Robert Parrott wrote:
> >
> > > This sounds reasonable enough to me, but I'm not familiar with
> > > python threading.
> > >
> > > What happens if the python callbacks are long running (i.e. SQL
> > > query). Would it block the main quickfix loop in this case?
> >
> >
>

Re: [Quickfix-developers] QuickFIX Python and Twisted

[Robert P. <rob...@gm...>]

Makes sense. And since the point was to use an asynchronous threaded
package like twisted, these callbacks can be made to be non-blocking.

So these changes would need to be made in the quickfix.i code?

rob


On 11/15/06, Oren Miller <or...@qu...> wrote:
> The allow threads macros I believe were explicitly designed for
> blocking IO calls, you can read mode about it here: http://
> www.gmu.edu/mlnavbar/webdev/doc-python/api/threads.html
>
> I would think that an IO library like one that does database queries
> is most likely aware of this and will implement these macros.  As
> long as they do, it should not be a problem since we called the end
> macro, they would be able to call the begin macro, then the end
> macro, allowing us to call the begin macro again.
>
> --oren
>
> On Nov 15, 2006, at 3:01 PM, Robert Parrott wrote:
>
> > This sounds reasonable enough to me, but I'm not familiar with
> > python threading.
> >
> > What happens if the python callbacks are long running (i.e. SQL
> > query). Would it block the main quickfix loop in this case?
>
>

Re: [Quickfix-developers] QuickFIX Python and Twisted

[Oren M. <or...@qu...>]

The allow threads macros I believe were explicitly designed for  
blocking IO calls, you can read mode about it here: http:// 
www.gmu.edu/mlnavbar/webdev/doc-python/api/threads.html

I would think that an IO library like one that does database queries  
is most likely aware of this and will implement these macros.  As  
long as they do, it should not be a problem since we called the end  
macro, they would be able to call the begin macro, then the end  
macro, allowing us to call the begin macro again.

--oren

On Nov 15, 2006, at 3:01 PM, Robert Parrott wrote:

> This sounds reasonable enough to me, but I'm not familiar with  
> python threading.
>
> What happens if the python callbacks are long running (i.e. SQL
> query). Would it block the main quickfix loop in this case?

Re: [Quickfix-developers] QuickFIX Python and Twisted

[Robert P. <rob...@gm...>]

This sounds reasonable enough to me, but I'm not familiar with python threading.

What happens if the python callbacks are long running (i.e. SQL
query). Would it block the main quickfix loop in this case?

On 11/15/06, Oren Miller <or...@qu...> wrote:
> To get around the 1 bytecode assignment for long running C methods,
> Python provides a couple of macros.  Py_BEGIN_ALLOW_THREADS and
> Py_END_ALLOW_THREADS which are to be called before and after your
> long running method respectively.  This would normally work well
> enough, but like Reggie said, QuickFIX is calling into Python, so
> doing this could be dangerous since the QuickFIX thread could end up
> accessing python resources when it essentially promised it wouldn't.
>
> I think perhaps the problem can be resolved this way.
>
> When calling start, in C++ we would do what we normally do when
> calling a long running method.  Something like...
>
> Py_BEGIN_ALLOW_THREADS
>    Call C++ start method
> Py_END_ALLOW_THREADS
>
> At this point the only danger is when QuickFIX delegates to the
> python callbacks.  So I would think we could do the opposite when
> entering a callback.
>
> Py_END_ALLOW_THREADS
>    Invoke Python callback
> Py_BEGIN_ALLOW_THREADS
>
> Anybody see a reason why this would not work?
>
> --oren
>
> On Nov 15, 2006, at 2:36 PM, Robert Parrott wrote:
>
> > ]That's very helpful information, Reggie.
> >
> > Are there any suggestions from the quickfix developers on how one
> > might address this threading issue?
> >
> > On 11/15/06, Reggie Dugard <re...@me...> wrote:
> >> Rob,
> >>
> >> I attempted to do almost exactly the same thing with almost
> >> exactly the
> >> same results.  I believe the problem starts with Python threading
> >> behavior:
> >>
> >>       * The Python Global Interpreter Lock (GIL) allows only one
> >> python
> >>         thread to be running at any point in time.
> >>       * Python will execute 10 bytecode instructions from a
> >> particular
> >>         thread and then release the GIL allowing other threads the
> >>         opportunity to run and acquire the GIL.
> >>       * A call to a C/C++ extension function is 1 bytecode
> >>
> >> Due to this behavior I believe that Python is calling into the QF C++
> >> code and then staying there without ever giving Python the
> >> opportunity
> >> to release the GIL and allow your twisted thread to run.  I
> >> thought of
> >> trying to edit the quickfix.i file to get swig to generate code to
> >> release the GIL before calling QF, but I realize I don't have enough
> >> SWIG/C++ knowledge to do it.  One of the problems I had is that
> >> the SWIG
> >> generated code sometimes calls back into python within a method so
> >> you
> >> can't just release the GIL arbitrarily before calling any QF
> >> function.
> >>
> >> Unfortunately I have yet to get Twisted and QuickFix integrated in
> >> the
> >> same application.  Sorry I couldn't be of more help, but if you
> >> find a
> >> solution to this I'd appreciate it if you could pass it along.
> >>
> >> -Reggie
> >>
> >>
> >>
> >> On Wed, 2006-11-15 at 13:41 -0500, Robert Parrott wrote:
> >>> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/
> >>> doc/html/index.html
> >>> QuickFIX Support: http://www.quickfixengine.org/services.html
> >>>
> >>> HI Folks,
> >>>
> >>> I'm trying to  build a simply gateway based on QuickFIX with the
> >>> python bindings, and the Twisted framework (http://
> >>> twistedmatrix.com).
> >>> I'm finding some kind of threading issue.
> >>>
> >>> My aim is to wrap the quickfix engine with a simple RPC
> >>> container, so
> >>> that one can connect to the daemon and invoke calls like "start",
> >>> "stop", "subscribe to data" etc. The twisted framework provides an
> >>> asynchronous networking layer to do so.
> >>>
> >>> However things seem to lock up when I invoke the
> >>> SocketInitiator.start() method. I can start up the service, and
> >>> create
> >>> the QuickFIX application, and  setup and configure the various
> >>> QuickFIX components, ala:
> >>>
> >>>  class MyQFService:
> >>>   def __init__
> >>>         settings     = quickfix.SessionSettings(configFile)
> >>>         application  = MyQFClient(connStr)
> >>>         storeFactory = quickfix.OdbcStoreFactory(settings)
> >>>         logFactory   = quickfix.OdbcLogFactory(settings)
> >>>
> >>>         self.quickfixSettings = settings
> >>>         self.quickfixApplication = application
> >>>         self.quickfixStoreFactory = storeFactory
> >>>         self.quickfixLogFactory = logFactory
> >>>
> >>>         self.quickfixInitiator = quickfix.SocketInitiator
> >>> (application,
> >>> storeFactory, settings, logFactory)
> >>>
> >>>  When a remote "START" command is issued, the main thread of the
> >>> twisted code calls the SocketInitiator to start the QuickFIX thread:
> >>>
> >>>   def startQF(self):
> >>>         log.msg("QuickFIX Engine starting.")
> >>>         self.quickfixInitiator.start()
> >>>         log.msg("QuickFIX Engine started.")
> >>>         return defer.succeed("STARTED")
> >>>
> >>> At this point, the quickfix engine thread starts, and I see logon,
> >>> heartbeats, and other admin messages. The log message immediately
> >>> after the start() call executes, and returns correctly. However, the
> >>> main thread of the Twisted application seems to block entirely,
> >>> and I
> >>> don't get any response from the application on subsequent RPC
> >>> attempts. At this point the appliction also stops responding to
> >>> Ctrl-C
> >>> on the command line, when before it would gracefully terminate.
> >>>
> >>> It seems as if the quickfix thread has taken over, or more likely is
> >>> completely blocking the main twisted thread. Any thoughts on what
> >>> might be happening here? My first guess is that it has to do with
> >>> logging, but I can't be sure.
> >>>
> >>> thanks for any insight,
> >>> rob
> >>>
> >>> --------------------------------------------------------------------
> >>> -----
> >>> Take Surveys. Earn Cash. Influence the Future of IT
> >>> Join SourceForge.net's Techsay panel and you'll get the chance to
> >>> share your
> >>> opinions on IT & business topics through brief surveys - and earn
> >>> cash
> >>> http://www.techsay.com/default.php?
> >>> page=join.php&p=sourceforge&CID=DEVDEV
> >>> _______________________________________________
> >>> Quickfix-developers mailing list
> >>> Qui...@li...
> >>> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> >> --
> >> Reggie Dugard <re...@me...>
> >> Merfin, LLC
> >>
> >>
> >>
> >
> > ----------------------------------------------------------------------
> > ---
> > Take Surveys. Earn Cash. Influence the Future of IT
> > Join SourceForge.net's Techsay panel and you'll get the chance to
> > share your
> > opinions on IT & business topics through brief surveys - and earn cash
> > http://www.techsay.com/default.php?
> > page=join.php&p=sourceforge&CID=DEVDEV
> > _______________________________________________
> > Quickfix-developers mailing list
> > Qui...@li...
> > https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> >
>
>

Re: [Quickfix-developers] QuickFIX Python and Twisted

[Oren M. <or...@qu...>]

To get around the 1 bytecode assignment for long running C methods,  
Python provides a couple of macros.  Py_BEGIN_ALLOW_THREADS and  
Py_END_ALLOW_THREADS which are to be called before and after your  
long running method respectively.  This would normally work well  
enough, but like Reggie said, QuickFIX is calling into Python, so  
doing this could be dangerous since the QuickFIX thread could end up  
accessing python resources when it essentially promised it wouldn't.

I think perhaps the problem can be resolved this way.

When calling start, in C++ we would do what we normally do when  
calling a long running method.  Something like...

Py_BEGIN_ALLOW_THREADS
   Call C++ start method
Py_END_ALLOW_THREADS

At this point the only danger is when QuickFIX delegates to the  
python callbacks.  So I would think we could do the opposite when  
entering a callback.

Py_END_ALLOW_THREADS
   Invoke Python callback
Py_BEGIN_ALLOW_THREADS

Anybody see a reason why this would not work?

--oren

On Nov 15, 2006, at 2:36 PM, Robert Parrott wrote:

> ]That's very helpful information, Reggie.
>
> Are there any suggestions from the quickfix developers on how one
> might address this threading issue?
>
> On 11/15/06, Reggie Dugard <re...@me...> wrote:
>> Rob,
>>
>> I attempted to do almost exactly the same thing with almost  
>> exactly the
>> same results.  I believe the problem starts with Python threading
>> behavior:
>>
>>       * The Python Global Interpreter Lock (GIL) allows only one  
>> python
>>         thread to be running at any point in time.
>>       * Python will execute 10 bytecode instructions from a  
>> particular
>>         thread and then release the GIL allowing other threads the
>>         opportunity to run and acquire the GIL.
>>       * A call to a C/C++ extension function is 1 bytecode
>>
>> Due to this behavior I believe that Python is calling into the QF C++
>> code and then staying there without ever giving Python the  
>> opportunity
>> to release the GIL and allow your twisted thread to run.  I  
>> thought of
>> trying to edit the quickfix.i file to get swig to generate code to
>> release the GIL before calling QF, but I realize I don't have enough
>> SWIG/C++ knowledge to do it.  One of the problems I had is that  
>> the SWIG
>> generated code sometimes calls back into python within a method so  
>> you
>> can't just release the GIL arbitrarily before calling any QF  
>> function.
>>
>> Unfortunately I have yet to get Twisted and QuickFix integrated in  
>> the
>> same application.  Sorry I couldn't be of more help, but if you  
>> find a
>> solution to this I'd appreciate it if you could pass it along.
>>
>> -Reggie
>>
>>
>>
>> On Wed, 2006-11-15 at 13:41 -0500, Robert Parrott wrote:
>>> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/ 
>>> doc/html/index.html
>>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>>
>>> HI Folks,
>>>
>>> I'm trying to  build a simply gateway based on QuickFIX with the
>>> python bindings, and the Twisted framework (http:// 
>>> twistedmatrix.com).
>>> I'm finding some kind of threading issue.
>>>
>>> My aim is to wrap the quickfix engine with a simple RPC  
>>> container, so
>>> that one can connect to the daemon and invoke calls like "start",
>>> "stop", "subscribe to data" etc. The twisted framework provides an
>>> asynchronous networking layer to do so.
>>>
>>> However things seem to lock up when I invoke the
>>> SocketInitiator.start() method. I can start up the service, and  
>>> create
>>> the QuickFIX application, and  setup and configure the various
>>> QuickFIX components, ala:
>>>
>>>  class MyQFService:
>>>   def __init__
>>>         settings     = quickfix.SessionSettings(configFile)
>>>         application  = MyQFClient(connStr)
>>>         storeFactory = quickfix.OdbcStoreFactory(settings)
>>>         logFactory   = quickfix.OdbcLogFactory(settings)
>>>
>>>         self.quickfixSettings = settings
>>>         self.quickfixApplication = application
>>>         self.quickfixStoreFactory = storeFactory
>>>         self.quickfixLogFactory = logFactory
>>>
>>>         self.quickfixInitiator = quickfix.SocketInitiator 
>>> (application,
>>> storeFactory, settings, logFactory)
>>>
>>>  When a remote "START" command is issued, the main thread of the
>>> twisted code calls the SocketInitiator to start the QuickFIX thread:
>>>
>>>   def startQF(self):
>>>         log.msg("QuickFIX Engine starting.")
>>>         self.quickfixInitiator.start()
>>>         log.msg("QuickFIX Engine started.")
>>>         return defer.succeed("STARTED")
>>>
>>> At this point, the quickfix engine thread starts, and I see logon,
>>> heartbeats, and other admin messages. The log message immediately
>>> after the start() call executes, and returns correctly. However, the
>>> main thread of the Twisted application seems to block entirely,  
>>> and I
>>> don't get any response from the application on subsequent RPC
>>> attempts. At this point the appliction also stops responding to  
>>> Ctrl-C
>>> on the command line, when before it would gracefully terminate.
>>>
>>> It seems as if the quickfix thread has taken over, or more likely is
>>> completely blocking the main twisted thread. Any thoughts on what
>>> might be happening here? My first guess is that it has to do with
>>> logging, but I can't be sure.
>>>
>>> thanks for any insight,
>>> rob
>>>
>>> -------------------------------------------------------------------- 
>>> -----
>>> Take Surveys. Earn Cash. Influence the Future of IT
>>> Join SourceForge.net's Techsay panel and you'll get the chance to  
>>> share your
>>> opinions on IT & business topics through brief surveys - and earn  
>>> cash
>>> http://www.techsay.com/default.php? 
>>> page=join.php&p=sourceforge&CID=DEVDEV
>>> _______________________________________________
>>> Quickfix-developers mailing list
>>> Qui...@li...
>>> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>> --
>> Reggie Dugard <re...@me...>
>> Merfin, LLC
>>
>>
>>
>
> ---------------------------------------------------------------------- 
> ---
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to  
> share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php? 
> page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>

Re: [Quickfix-developers] QuickFIX Python and Twisted

[Robert P. <rob...@gm...>]

That's very helpful information, Reggie.

Are there any suggestions from the quickfix developers on how one
might address this threading issue?

On 11/15/06, Reggie Dugard <re...@me...> wrote:
> Rob,
>
> I attempted to do almost exactly the same thing with almost exactly the
> same results.  I believe the problem starts with Python threading
> behavior:
>
>       * The Python Global Interpreter Lock (GIL) allows only one python
>         thread to be running at any point in time.
>       * Python will execute 10 bytecode instructions from a particular
>         thread and then release the GIL allowing other threads the
>         opportunity to run and acquire the GIL.
>       * A call to a C/C++ extension function is 1 bytecode
>
> Due to this behavior I believe that Python is calling into the QF C++
> code and then staying there without ever giving Python the opportunity
> to release the GIL and allow your twisted thread to run.  I thought of
> trying to edit the quickfix.i file to get swig to generate code to
> release the GIL before calling QF, but I realize I don't have enough
> SWIG/C++ knowledge to do it.  One of the problems I had is that the SWIG
> generated code sometimes calls back into python within a method so you
> can't just release the GIL arbitrarily before calling any QF function.
>
> Unfortunately I have yet to get Twisted and QuickFix integrated in the
> same application.  Sorry I couldn't be of more help, but if you find a
> solution to this I'd appreciate it if you could pass it along.
>
> -Reggie
>
>
>
> On Wed, 2006-11-15 at 13:41 -0500, Robert Parrott wrote:
> > QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html
> > QuickFIX Support: http://www.quickfixengine.org/services.html
> >
> > HI Folks,
> >
> > I'm trying to  build a simply gateway based on QuickFIX with the
> > python bindings, and the Twisted framework (http://twistedmatrix.com).
> > I'm finding some kind of threading issue.
> >
> > My aim is to wrap the quickfix engine with a simple RPC container, so
> > that one can connect to the daemon and invoke calls like "start",
> > "stop", "subscribe to data" etc. The twisted framework provides an
> > asynchronous networking layer to do so.
> >
> > However things seem to lock up when I invoke the
> > SocketInitiator.start() method. I can start up the service, and create
> > the QuickFIX application, and  setup and configure the various
> > QuickFIX components, ala:
> >
> >  class MyQFService:
> >   def __init__
> >         settings     = quickfix.SessionSettings(configFile)
> >         application  = MyQFClient(connStr)
> >         storeFactory = quickfix.OdbcStoreFactory(settings)
> >         logFactory   = quickfix.OdbcLogFactory(settings)
> >
> >         self.quickfixSettings = settings
> >         self.quickfixApplication = application
> >         self.quickfixStoreFactory = storeFactory
> >         self.quickfixLogFactory = logFactory
> >
> >         self.quickfixInitiator = quickfix.SocketInitiator(application,
> > storeFactory, settings, logFactory)
> >
> >  When a remote "START" command is issued, the main thread of the
> > twisted code calls the SocketInitiator to start the QuickFIX thread:
> >
> >   def startQF(self):
> >         log.msg("QuickFIX Engine starting.")
> >         self.quickfixInitiator.start()
> >         log.msg("QuickFIX Engine started.")
> >         return defer.succeed("STARTED")
> >
> > At this point, the quickfix engine thread starts, and I see logon,
> > heartbeats, and other admin messages. The log message immediately
> > after the start() call executes, and returns correctly. However, the
> > main thread of the Twisted application seems to block entirely, and I
> > don't get any response from the application on subsequent RPC
> > attempts. At this point the appliction also stops responding to Ctrl-C
> > on the command line, when before it would gracefully terminate.
> >
> > It seems as if the quickfix thread has taken over, or more likely is
> > completely blocking the main twisted thread. Any thoughts on what
> > might be happening here? My first guess is that it has to do with
> > logging, but I can't be sure.
> >
> > thanks for any insight,
> > rob
> >
> > -------------------------------------------------------------------------
> > Take Surveys. Earn Cash. Influence the Future of IT
> > Join SourceForge.net's Techsay panel and you'll get the chance to share your
> > opinions on IT & business topics through brief surveys - and earn cash
> > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> > _______________________________________________
> > Quickfix-developers mailing list
> > Qui...@li...
> > https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> --
> Reggie Dugard <re...@me...>
> Merfin, LLC
>
>
>

Re: [Quickfix-developers] QuickFIX Python and Twisted

[Reggie D. <re...@me...>]

Rob,

I attempted to do almost exactly the same thing with almost exactly the
same results.  I believe the problem starts with Python threading
behavior:

      * The Python Global Interpreter Lock (GIL) allows only one python
        thread to be running at any point in time. 
      * Python will execute 10 bytecode instructions from a particular
        thread and then release the GIL allowing other threads the
        opportunity to run and acquire the GIL.
      * A call to a C/C++ extension function is 1 bytecode

Due to this behavior I believe that Python is calling into the QF C++
code and then staying there without ever giving Python the opportunity
to release the GIL and allow your twisted thread to run.  I thought of
trying to edit the quickfix.i file to get swig to generate code to
release the GIL before calling QF, but I realize I don't have enough
SWIG/C++ knowledge to do it.  One of the problems I had is that the SWIG
generated code sometimes calls back into python within a method so you
can't just release the GIL arbitrarily before calling any QF function.

Unfortunately I have yet to get Twisted and QuickFix integrated in the
same application.  Sorry I couldn't be of more help, but if you find a
solution to this I'd appreciate it if you could pass it along.

-Reggie



On Wed, 2006-11-15 at 13:41 -0500, Robert Parrott wrote:
> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
> 
> HI Folks,
> 
> I'm trying to  build a simply gateway based on QuickFIX with the
> python bindings, and the Twisted framework (http://twistedmatrix.com).
> I'm finding some kind of threading issue.
> 
> My aim is to wrap the quickfix engine with a simple RPC container, so
> that one can connect to the daemon and invoke calls like "start",
> "stop", "subscribe to data" etc. The twisted framework provides an
> asynchronous networking layer to do so.
> 
> However things seem to lock up when I invoke the
> SocketInitiator.start() method. I can start up the service, and create
> the QuickFIX application, and  setup and configure the various
> QuickFIX components, ala:
> 
>  class MyQFService:
>   def __init__
>         settings     = quickfix.SessionSettings(configFile)
>         application  = MyQFClient(connStr)
>         storeFactory = quickfix.OdbcStoreFactory(settings)
>         logFactory   = quickfix.OdbcLogFactory(settings)
> 
>         self.quickfixSettings = settings
>         self.quickfixApplication = application
>         self.quickfixStoreFactory = storeFactory
>         self.quickfixLogFactory = logFactory
> 
>         self.quickfixInitiator = quickfix.SocketInitiator(application,
> storeFactory, settings, logFactory)
> 
>  When a remote "START" command is issued, the main thread of the
> twisted code calls the SocketInitiator to start the QuickFIX thread:
> 
>   def startQF(self):
>         log.msg("QuickFIX Engine starting.")
>         self.quickfixInitiator.start()
>         log.msg("QuickFIX Engine started.")
>         return defer.succeed("STARTED")
> 
> At this point, the quickfix engine thread starts, and I see logon,
> heartbeats, and other admin messages. The log message immediately
> after the start() call executes, and returns correctly. However, the
> main thread of the Twisted application seems to block entirely, and I
> don't get any response from the application on subsequent RPC
> attempts. At this point the appliction also stops responding to Ctrl-C
> on the command line, when before it would gracefully terminate.
> 
> It seems as if the quickfix thread has taken over, or more likely is
> completely blocking the main twisted thread. Any thoughts on what
> might be happening here? My first guess is that it has to do with
> logging, but I can't be sure.
> 
> thanks for any insight,
> rob
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
-- 
Reggie Dugard <re...@me...>
Merfin, LLC