pygccxml-development

[pygccxml-development] Feature request: Creating thread safe methods

[Matthias B. <ba...@ir...>]

Hi,

recently I added support for multi-threading in my Python/Maya package. 
This means whenever a thread is executing Python code it has to lock the 
interpreter before it can execute the code, so that other threads that 
also execute Python code won't interfere. This can be done by 
surrounding any Python invocation with PyGILState_Ensure() and 
PyGILState_Release().
For my own commands this is no problem as the code that invokes a Python 
function is written by myself and so I can add the above functions.
But when a user registers a Python plugin class, some methods of this 
class will be called by Maya (i.e. from C++ code) and then the only 
place where the above functions can be called is the wrapper of the 
plugin class. See here:

http://mail.python.org/pipermail/c++-sig/2005-December/009953.html

(also see the other mails of the thread. As I understand the issue, the 
PyGILState_Ensure() function actually has to be called before 
get_override() is called)

However, this code is generated by pyplusplus and currently I don't see 
a way to inject the extra code. So my suggestion would be to add a new 
decoration attribute "thread_safe" to the calldef decls which is a 
boolean that controls whether the code creators add the appropriate 
locking code or not. Maybe this flag could even have three states so 
that in addition to True and False there's the option to use a global 
default value. This means the user could simply set the default value to 
specify if he wants to create thread-safe bindings or not.
Another question is if pyplusplus should ever add the locking functions 
to non-virtual methods. In principle, non-virtual methods are always 
thread-safe unless the method invokes Python code internally (which is 
probably a really rare case).

Then there's another issue related to thread-safety. Locking the Python 
interpreter is one thing, but you also have to catch any exception that 
is raised in the Python code and take some appropriate actions (like 
reporting the exception and, in my case, returning an appropriate error 
status code to Maya). This is something that cannot be done 
automatically by pyplusplus because it depends on the application how 
errors are reported back to the application. So this is where that "arg 
policies" mechanism to inject user code would be handy.
I'm not sure if even the above locking code could be added via this 
mechanism.

While adding the "thread_safe" flag and updating the code creators seems 
to be pretty straightforward to me, I'm afraid adding the "arg policy" 
thing is a more delicate operation. I thought before I try to do this 
myself I rather post a message to discuss these issues before I get my 
hands dirty.

So, any comments?

- Matthias -

Re: [pygccxml-development] Feature request: Creating thread safe methods

[Roman Y. <rom...@gm...>]

On 7/31/06, Matthias Baas <ba...@ir...> wrote:
> Hi,
>
> recently I added support for multi-threading in my Python/Maya package.
> This means whenever a thread is executing Python code it has to lock the
> interpreter before it can execute the code, so that other threads that
> also execute Python code won't interfere. This can be done by
> surrounding any Python invocation with PyGILState_Ensure() and
> PyGILState_Release().

May be we should add to boost::python library simple guard:
struct gilguard{
    gilguard(){ PyGILState_Ensure() ; }
    ~gilguard(){ PyGILState_Release(); }
};

Thus the generated code will be able to use the guard and it will be exception
neutral.

> However, this code is generated by pyplusplus and currently I don't see
> a way to inject the extra code.

Now can you see the problem with virtual functions and arguments policies?
They don't work well together :-(. The main problem is that function
signature is changed.

> So my suggestion would be to add a new
> decoration attribute "thread_safe" to the calldef decls which is a
> boolean that controls whether the code creators add the appropriate
> locking code or not.

This is really simple and easy to implement solution.

> Maybe this flag could even have three states so
> that in addition to True and False there's the option to use a global
> default value. This means the user could simply set the default value to
> specify if he wants to create thread-safe bindings or not.

mb.calldefs().thread_safe = True

The point is that we don't need global flag.

> Another question is if pyplusplus should ever add the locking functions
> to non-virtual methods. In principle, non-virtual methods are always
> thread-safe unless the method invokes Python code internally (which is
> probably a really rare case).

I don't know. Also I prefer to develop py++ having real use cases.
So the answer is next: if you don't have such use case we will not
write the code that
will treat it.

> Then there's another issue related to thread-safety. Locking the Python
> interpreter is one thing, but you also have to catch any exception that
> is raised in the Python code and take some appropriate actions (like
> reporting the exception and, in my case, returning an appropriate error
> status code to Maya).

Will exception neutral code will solve this problem? If not, I think py++
should have functionality, that will allow user to add/modify/rewrite
code generated
by py++. If this functionality will be added, we could implement
"arguments policy"
functionality out side of py++.

> This is something that cannot be done
> automatically by pyplusplus because it depends on the application how
> errors are reported back to the application. So this is where that "arg
> policies" mechanism to inject user code would be handy.
> I'm not sure if even the above locking code could be added via this
> mechanism.

Every overridable virtual member function looks like:

0 virtual int pure_virtual( int x ){
1
2    bp::override func_pure_virtual = this->get_override( "pure_virtual" );
3
4 return func_pure_virtual( x );
5
6 }

It seems, that user needs to add code to 1, 3, 5 and to be able to modify 2,4.
Am I right?

> While adding the "thread_safe" flag and updating the code creators seems
> to be pretty straightforward to me, I'm afraid adding the "arg policy"
> thing is a more delicate operation. I thought before I try to do this
> myself I rather post a message to discuss these issues before I get my
> hands dirty.
>
> So, any comments?

One more comment. How I see this feature is developed. py++ will
provide all services
you need to implement this feature out side of it. After "arguments
policy" functionality
will become stable, we will merge it with py++. Thus you will be able
to experiment
and move faster. From the other side, I will be able to concentrate on
next release.


-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Feature request: Creating thread safe methods

[Matthias B. <ba...@ir...>]

Roman Yakovenko wrote:
> May be we should add to boost::python library simple guard:
> struct gilguard{
>    gilguard(){ PyGILState_Ensure() ; }
>    ~gilguard(){ PyGILState_Release(); }
> };
> 
> Thus the generated code will be able to use the guard and it will be 
> exception neutral.

This is a good idea for the case when the entire function/method has to 
be surrounded by the above calls.

But in the case of our wrapper methods it would be more efficient to do 
the original function call outside the locking to prevent other threads 
from being blocked.
Currently the wrapper code for virtual member functions looks somewhat 
like this:

virtual void foo(){
   if( bp::override func_foo = this->get_override( "foo" ) )
      func_foo( );
   else
      Foo::foo( );
}

Surrounding the entire function body with the Ensure() and Release() 
calls would work but I think it's too conservative. In the case when the 
original Foo:foo() function is invoked the interpreter lock is not 
necessary anymore and it would just prevent other threads from being 
able to execute Python code.
So I think the wrapper code should rather look like this:

virtual void foo(){
   PyGILState_STATE gstate;
   gstate = PyGILState_Ensure();

   bp::override func_foo = this->get_override( "foo" );
   if( func_foo )
   {
      func_foo( );
      // at this point the user might want to add code that
      // checks for exceptions (or does the above function
      // call already raise a C++ exception? then there
      // should be a try...catch block)
   }

   PyGILState_Release(gstate);

   if (!func_foo)
      Foo::foo( );
}

>> However, this code is generated by pyplusplus and currently I don't see
>> a way to inject the extra code.
> 
> Now can you see the problem with virtual functions and arguments policies?
> They don't work well together :-(. The main problem is that function
> signature is changed.

Well, yes, but in this case we are actually dealing with two wrapper 
methods, aren't we? One for the case when the call is made from C++ and 
one for the case when the call is made from Python.
So far, my existing arg policies are dealing with the case when the call 
is made from Python and goes down to C++. Hm, maybe the other direction 
just needs a separate set of "arg policy" classes....?

So far, the virtual methods in the Maya SDK are always called from C++, 
so I would only need one direction. But I have only considered a few 
classes and methods yet, there might still be methods where the derived 
class wants to invoke the inherited method...

>> Maybe this flag could even have three states so
>> that in addition to True and False there's the option to use a global
>> default value. This means the user could simply set the default value to
>> specify if he wants to create thread-safe bindings or not.
> 
> mb.calldefs().thread_safe = True

Well, ok. I just thought it would be more efficient and spare one query, 
but the above is fine with me.

>> Another question is if pyplusplus should ever add the locking functions
>> to non-virtual methods. In principle, non-virtual methods are always
>> thread-safe unless the method invokes Python code internally (which is
>> probably a really rare case).
> 
> I don't know. Also I prefer to develop py++ having real use cases.
> So the answer is next: if you don't have such use case we will not
> write the code that will treat it.

ok. As I already said, I think it's really a rare case where someone 
would need this. Usually, the methods will be virtual so that you can 
override the methods and implement them in Python.

>> Then there's another issue related to thread-safety. Locking the Python
>> interpreter is one thing, but you also have to catch any exception that
>> is raised in the Python code and take some appropriate actions (like
>> reporting the exception and, in my case, returning an appropriate error
>> status code to Maya).
> 
> Will exception neutral code will solve this problem? 

What do you mean by this?

> If not, I think py++
> should have functionality, that will allow user to add/modify/rewrite
> code generated
> by py++. If this functionality will be added, we could implement
> "arguments policy"
> functionality out side of py++.

If you mean that the actual policy class is outside py++ (but of course, 
the framework for this to work is inside py++), then I agree.... :)

> Every overridable virtual member function looks like:
> 
> 0 virtual int pure_virtual( int x ){
> 1
> 2    bp::override func_pure_virtual = this->get_override( "pure_virtual" );
> 3
> 4 return func_pure_virtual( x );
> 5
> 6 }
> 
> It seems, that user needs to add code to 1, 3, 5 and to be able to 
> modify 2,4. Am I right?

In principle, yes. Of course, as I wasn't dealing with virtual methods 
yet I have no distinction between 1 and 3 (as line 2 doesn't appear in 
the code generated by the existing arg policy mechanism), so both of 
them would be "pre-call" whereas 5 is "post-call". Additionally, there 
should be a way to allocate local variables (this has to be done via an 
API call so that the variable name can be adjusted if necessary to 
prevent name clashes).

- Matthias -

Re: [pygccxml-development] Feature request: Creating thread safe methods

[Roman Y. <rom...@gm...>]

On 7/31/06, Matthias Baas <ba...@ir...> wrote:
> This is a good idea for the case when the entire function/method has to
> be surrounded by the above calls.
>
> virtual void foo(){
>    PyGILState_STATE gstate;
>    gstate = PyGILState_Ensure();
>
>    bp::override func_foo = this->get_override( "foo" );
>    if( func_foo )
>    {
>       func_foo( );
>       // at this point the user might want to add code that
>       // checks for exceptions (or does the above function
>       // call already raise a C++ exception? then there
>       // should be a try...catch block)
>    }
>
>    PyGILState_Release(gstate);
>
>    if (!func_foo)
>       Foo::foo( );
> }

virtual void foo(){
   {
        gilguard guard;
        if( bp::override func_foo = this->get_override( "foo" ) )
            func_foo( );
        return;
   }
    Foo::foo( );
}

The point is that you still can use the guard. Another point, is that
we are talking
here about optimization. Lets just skip it.

> >> However, this code is generated by pyplusplus and currently I don't see
> >> a way to inject the extra code.
> >
> > Now can you see the problem with virtual functions and arguments policies?
> > They don't work well together :-(. The main problem is that function
> > signature is changed.
>
> Well, yes, but in this case we are actually dealing with two wrapper
> methods, aren't we? One for the case when the call is made from C++ and
> one for the case when the call is made from Python.
> So far, my existing arg policies are dealing with the case when the call
> is made from Python and goes down to C++. Hm, maybe the other direction
> just needs a separate set of "arg policy" classes....?

There are reasons I did not start to implement this functionality. I
don't know the
answer. It is clear to me that user will have to choose between X and Y.
But it is not clear to me what X and Y are.

It will be perfectly good to make an assumption, that most of the time users
don't embed Python within C++, but rather extend Python. We can start to discuss
a solution that will be based on this assumption.

> So far, the virtual methods in the Maya SDK are always called from C++,
> so I would only need one direction. But I have only considered a few
> classes and methods yet, there might still be methods where the derived
> class wants to invoke the inherited method...

I think, we will have to build some use cases table and for every use case to
write an expected solution.

> >> Then there's another issue related to thread-safety. Locking the Python
> >> interpreter is one thing, but you also have to catch any exception that
> >> is raised in the Python code and take some appropriate actions (like
> >> reporting the exception and, in my case, returning an appropriate error
> >> status code to Maya).
> >
> > Will exception neutral code will solve this problem?
>
> What do you mean by this?

If we will generate an exception neutral code, then we move the responsibility
to treat exception to boost.python and to the user. The only exception to this
case is when you have to translate exception to some kind of return code.

> If you mean that the actual policy class is outside py++ (but of course,
> the framework for this to work is inside py++), then I agree.... :)

I think we understand each other.  I meant, that you will define what
services you will need from py++ in order to implement this feature
efficiently. Also do you
mean "actual policy class" is something similar to classes that derives from
ArgTransformerBase?

By the way, do you want to rename it? Thread safety has nothing to do with
arguments.

> > Every overridable virtual member function looks like:
> >
> > 0 virtual int pure_virtual( int x ){
> > 1
> > 2    bp::override func_pure_virtual = this->get_override( "pure_virtual" );
> > 3
> > 4 return func_pure_virtual( x );
> > 5
> > 6 }
> >
> > It seems, that user needs to add code to 1, 3, 5 and to be able to
> > modify 2,4. Am I right?
>
> In principle, yes. Of course, as I wasn't dealing with virtual methods
> yet I have no distinction between 1 and 3 (as line 2 doesn't appear in
> the code generated by the existing arg policy mechanism), so both of
> them would be "pre-call" whereas 5 is "post-call".

>Additionally, there
> should be a way to allocate local variables (this has to be done via an
> API call so that the variable name can be adjusted if necessary to
> prevent name clashes).

I am  not sure about this, really. We should see.

Matthias, can you write some document that will contain description of
what you are going to do:
1. use case
2. proposed solution
3. user interface/API
4. pyplusplus API

Something like this. I think it will cause us to work together and
better understand
the problem and solution.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Feature request: Creating thread safe methods

[Matthias B. <ba...@ir...>]

Roman Yakovenko wrote:
> It will be perfectly good to make an assumption, that most of the time 
> users
> don't embed Python within C++, but rather extend Python. We can start to 
> discuss
> a solution that will be based on this assumption.

Well, the thing is just that in my case, I actually do embed Python in 
C++ and the cases I have to deal with now are those where the 
application does calls to Python (without knowing it).

>> >> Then there's another issue related to thread-safety. Locking the 
>> Python
>> >> interpreter is one thing, but you also have to catch any exception 
>> that
>> >> is raised in the Python code and take some appropriate actions (like
>> >> reporting the exception and, in my case, returning an appropriate 
>> error
>> >> status code to Maya).
>> >
>> > Will exception neutral code will solve this problem?
>>
>> What do you mean by this?
> 
> If we will generate an exception neutral code, then we move the 
> responsibility
> to treat exception to boost.python and to the user. The only exception 
> to this
> case is when you have to translate exception to some kind of return code.

Right, that's what I want to do...

>> If you mean that the actual policy class is outside py++ (but of course,
>> the framework for this to work is inside py++), then I agree.... :)
> 
> I think we understand each other.  I meant, that you will define what
> services you will need from py++ in order to implement this feature
> efficiently. Also do you
> mean "actual policy class" is something similar to classes that derives 
> from ArgTransformerBase?

Yes, exactly.

> By the way, do you want to rename it? Thread safety has nothing to do with
> arguments.

Yes, I know. I suppose, eventually we have to find a better name (but I 
think "call wrapper policy" also doesn't really capture it...)

> Matthias, can you write some document that will contain description of
> what you are going to do:

I tried to come up with some initial document here:
https://realityforge.vrsource.org/view/PyPP/CodeInserter

- Matthias -

Re: [pygccxml-development] Feature request: Creating thread safe methods

[Roman Y. <rom...@gm...>]

On 8/1/06, Matthias Baas <ba...@ir...> wrote:
> >> If you mean that the actual policy class is outside py++ (but of course,
> >> the framework for this to work is inside py++), then I agree.... :)
> >
> > I think we understand each other.  I meant, that you will define what
> > services you will need from py++ in order to implement this feature
> > efficiently. Also do you
> > mean "actual policy class" is something similar to classes that derives
> > from ArgTransformerBase?
>
> Yes, exactly.

Please take a look on decl_wrappers.constructor_t class. It has "body" property.
I think, all you need to implement the feature is similar property on
member_function_t
class. Actually you will need 2 properties:
1. body
2. default_virtual_body ( better name is desired ).

Or something like this. If those properties contain a text, pyplusplus
will use it
as function body, otherwise it will generate the body.

I mean something similar to what "inserting code" document describes.

> > By the way, do you want to rename it? Thread safety has nothing to do with
> > arguments.
>
> Yes, I know. I suppose, eventually we have to find a better name (but I
> think "call wrapper policy" also doesn't really capture it...)

Function transform policies?

> > Matthias, can you write some document that will contain description of
> > what you are going to do:
>
> I tried to come up with some initial document here:
> https://realityforge.vrsource.org/view/PyPP/CodeInserter

It is a pleasure to work with you :-)

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Feature request: Creating thread safe methods

[Matthias B. <ba...@ir...>]

Roman Yakovenko wrote:
> Please take a look on decl_wrappers.constructor_t class. It has "body" 
> property.
> I think, all you need to implement the feature is similar property on
> member_function_t
> class. Actually you will need 2 properties:
> 1. body
> 2. default_virtual_body ( better name is desired ).
> 
> Or something like this. If those properties contain a text, pyplusplus
> will use it
> as function body, otherwise it will generate the body.

Will the "body" attribute then replace the entire function body? (what 
about the signature?) But then I wouldn't gain anything. I do this more 
or less already since I ignore the original declaration and replace it 
with my own one (circumventing pyplusplus).
The point is that a user then has to know the code that pyplusplus would 
create so that he can add that code to his own version, too. But I would 
prefer if a user could only concentrate on his new code that he wants to 
insert and leave everything else alone (i.e. the code that pyplusplus 
already generates).

>> Yes, I know. I suppose, eventually we have to find a better name (but I
>> think "call wrapper policy" also doesn't really capture it...)
> 
> Function transform policies?

Hm, I don't know, I don't regard the addition of code as a 
transformation and the added code doesn't have to implement a policy.
How can we concisely describe "an object that is a building block for 
wrapper functions" or "a specialized partial code creator for wrapper 
functions" or....?

- Matthias -

Re: [pygccxml-development] Feature request: Creating thread safe methods

[Roman Y. <rom...@gm...>]

On 8/2/06, Matthias Baas <ba...@ir...> wrote:
> Roman Yakovenko wrote:
> > Please take a look on decl_wrappers.constructor_t class. It has "body"
> > property.
> > I think, all you need to implement the feature is similar property on
> > member_function_t
> > class. Actually you will need 2 properties:
> > 1. body
> > 2. default_virtual_body ( better name is desired ).
> >
> > Or something like this. If those properties contain a text, pyplusplus
> > will use it
> > as function body, otherwise it will generate the body.
>
> Will the "body" attribute then replace the entire function body? (what
> about the signature?)

Will replace the body, but will leave the signature.

> But then I wouldn't gain anything. I do this more
> or less already since I ignore the original declaration and replace it
> with my own one (circumventing pyplusplus).
> The point is that a user then has to know the code that pyplusplus would
> create so that he can add that code to his own version, too. But I would
> prefer if a user could only concentrate on his new code that he wants to
> insert and leave everything else alone (i.e. the code that pyplusplus
> already generates).

I understand your preferences. I will not implement this. If you will
need one, it is
a matter of few hours to add it.

> >> Yes, I know. I suppose, eventually we have to find a better name (but I
> >> think "call wrapper policy" also doesn't really capture it...)
> >
> > Function transform policies?
>
> Hm, I don't know, I don't regard the addition of code as a
> transformation and the added code doesn't have to implement a policy.
> How can we concisely describe "an object that is a building block for
> wrapper functions" or "a specialized partial code creator for wrapper
> functions" or....?

Consider to take one step back. There is some C++ function, that could
not be called
from Python. You need to create another function, that could be called
from Python,
that will call the original function. If you don't call the original
function, than you just
create new function and register it under some C++ function name. So,
in some sense
you do transform one function to another.


-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/