pygccxml-development

[pygccxml-development] Feature proposal: "Argument policies"

[Matthias B. <ba...@ir...>]

Hi,

here's a suggestion for a new feature that should deal with the problem 
that C++ functions that have more than just one output (by taking 
references) cannot be translated literally to Python. Even worse, they 
compile without errors but they are plain useless in Python because 
there's no way to invoke such functions.

One example of such a function is the already mentioned getSize(int& 
width, int& height) method that returns the resolution of an image.
It is my understanding that such a function requires a wrapper function 
that changes the signature so that the function takes no arguments and 
returns a Python tuple with the result.
Having to write those wrappers manually is quite a repetitive and 
tedious task. So my suggestion is to introduce something similar to call 
policies, that's why I called it "argument policies". The idea is just 
that for each "problematic" argument you specify that it needs special 
treatment. In the above case, pyplusplus needs to know that the 
arguments are actually used for output values instead of input values. 
This could be specified like this:

Image.Method("getSize").setArgPolicy(Output(1), Output(2))

setArgPolicy() is a new decorator that takes an arbitrary number of 
"argument policies" as input. The "Output" policy takes one integer as 
argument which is the index of the argument it applies to. So by writing 
"Output(1)", the user specifies that the first argument is actually an 
output value. This is already enough information so that a wrapper 
function can be created automatically. The wrapper function would take 
no arguments anymore and return two integers with the result.

Another example is a function that takes a pointer to a double as input. 
For example:

void getTranslation(double t[3])
void setTranslation(double t[3])

In the first case, 't' is an ouput argument used to store the result 
whereas in the second case 't' is an input argument. The above Output 
policy cannot be used because Python has no double[3] type. Instead, the 
wrapper needs some additional code to copy a Python sequence into a 
double array or vice versa:

cls.Method("getTranslation").setArgPolicy(OutputArray(1,3))
cls.Method("setTranslation").setArgPolicy(InputArray(1,3))

The first argument to the OutputArray/InputArray class is again the 
index of the argument this policy applies to. The second argument is the 
size of the array. These policies will then also take care of converting 
between Python sequences and C arrays.

More policies can be defined as appropriate. For example, variants of 
the InputArray/OutputArray policies could take the array size from an 
additional argument.

The nice thing about this framework is that another thing I wanted to 
have fits nicely into it. For the Maya SDK, I wanted to convert Status 
objects into exceptions. This can also be expressed with those policies 
by using a custom policy such as "MStatusToException" that adds code 
that throws an exception when an error has occurred. There can also be 
policies that do something with the return value, etc.

I have already implemented a prototype version of this in the 
experimental module. So far, it has worked quite fine for my Maya 
bindings (by the way, I have just released a first "preview" version of 
those bindings, so if anybody wants to have a look how I create them, 
have a look at the 'maya' package over here: http://cgkit.sourceforge.net/)

Currently, my implementation would not work when a class already 
receives wrappers generated by pyplusplus. But I think this could be 
changed so that the wrappers created by pyplusplus use the same 
framework for specifying wrapper source code (but this needs 
modification inside pyplusplus).
The above argument policies use the same "code creator" concept that 
pyplusplus already uses, the interface of those code creators is just 
different because there are actually two places where code can be 
placed, either before or after the original function invocation. And in 
addition to creating code, those special code creators can also modify 
the signature of a function. So pyplusplus would have to use those 
special code creators instead of just creating one static block of C++ 
source code for the entire wrapper function.

Any comments? Do you already see cases where this could not work or how 
it could be improved?

- Matthias -

Re: [pygccxml-development] Feature proposal: "Argument policies"

[Roman Y. <rom...@gm...>]

On 4/18/06, Matthias Baas <ba...@ir...> wrote:
> Hi,
> So my suggestion is to introduce something similar to call
> policies, that's why I called it "argument policies". The idea is just
> that for each "problematic" argument you specify that it needs special
> treatment.

I like the idea and the name you gave it. It is easy to understand and expl=
ain.
I think, this will be the focus of next release.

Before we proceed to technical details, I suggest everyone, who is
interested in this
should review two code creators: function_t and function_wrapper_t
within code_creators/calldefs.py file.

> I have already implemented a prototype version of this in the
> experimental module. So far, it has worked quite fine for my Maya
> bindings (by the way, I have just released a first "preview" version of
> those bindings, so if anybody wants to have a look how I create them,
> have a look at the 'maya' package over here: http://cgkit.sourceforge.net=
/)

:-). I will definitely take a look. It seems that we can start "who is
using pyplusplus?" list.
Who does want to contribute :-) ?

> Currently, my implementation would not work when a class already
> receives wrappers generated by pyplusplus. But I think this could be
> changed so that the wrappers created by pyplusplus use the same
> framework for specifying wrapper source code (but this needs
> modification inside pyplusplus).
>
> The above argument policies use the same "code creator" concept that
> pyplusplus already uses, the interface of those code creators is just
> different because there are actually two places where code can be
> placed, either before or after the original function invocation. And in
> addition to creating code, those special code creators can also modify
> the signature of a function. So pyplusplus would have to use those
> special code creators instead of just creating one static block of C++
> source code for the entire wrapper function.

Right

> Any comments? Do you already see cases where this could not work or how
> it could be improved?

I hope next week, I will write some PEP. Now I just want to comment the fea=
ture.

Please take a look on the classes ( function_t and function_wrapper_t ).
They are tightly coupled and contains code that treat a lot of use cases:

When generating code pyplusplus should consider

1. [ virtual | pure virtual | not virtual ]
    Different code should be generated for different access types/

2. [ public | protected | private ]
Yes, private pure virtual and private virtual functions should/could
be exposed. See
"template method" design pattern.

3. [ static | free | member ] functions

4. operators( call (), index [] and may be something else  )

5. class hierarchy with deep more then 2 abstract classes:

struct base_abstract{
void do_smth() =3D 0;
};

struct middle_abstract : public abstract{
void do_smth_else() =3D 0;
};

Class wrapper that is built for middle_abstract should define also
"function wrappers" for
do_smth function. I can explain, why but lets leave it.

6. function return value: for example virtual function, that returns
reference to some object,
could not be overridden from Python - access violation is guaranteed.

7. template functions on broken compilers ( msvc 7.1 )
    When your expose some instantiation of template class that has
template function
    pyplusplus should define few typedefs in order to generate code,
that takes address
    of the function. It does do it right now, but this should be fixed.

8. difference between fundamental types and user defined
   ( boost::ref and boost::python::ptr ).

I am sure this is not the full list. Now we should add here argument
policies. This is not going
to be easy. I see few challenges here:

0. simple user interface

1. orthogonality, it is obvious to me, that our solution should be
constructed from
    few blocks

2. use cases enumeration

3. implementation

4. testing

It is not going to be easy, but we will have a lot of fun, I am sure.

Contribution: we can start now to create comprehensive unit tests for this.
It would be grate, If some one can write C++ code that will cover all
those use case.

> - Matthias -

--
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Feature proposal: "Argument policies"

[Roman Y. <rom...@gm...>]

On 4/19/06, Roman Yakovenko <rom...@gm...> wrote:

I'd like to add next example, from what you can see that argument
policies should be able to deal with more complex case:

//read from some file
int read( byte* buffer, unsigned int buffer_size );

This is very interesting use case for "argument policies", because we
should deal
with two arguments at the same time. The generated code will not be
very different
from code generated by using OutputArray/InputArray policies.

I can create an example, that binds between return value and some argument.

I want to say that "argument policies" feature should be rename. New
feature should deal
with an examples I showed. So it does not deal any more with single
argument, but with
whole function.

Bottom line - new name needed.

Suggestions:

[ function ] wrapper call policies

Comments?

--
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Feature proposal: "Argument policies"

[Matthias B. <ba...@ir...>]

Matthias Baas wrote:
> More policies can be defined as appropriate. For example, variants of 
> the InputArray/OutputArray policies could take the array size from an 
> additional argument.

Roman Yakovenko wrote:
> I'd like to add next example, from what you can see that argument
> policies should be able to deal with more complex case:
> 
> //read from some file
> int read( byte* buffer, unsigned int buffer_size );
> 
> This is very interesting use case for "argument policies", because we
> should deal
> with two arguments at the same time. The generated code will not be
> very different
> from code generated by using OutputArray/InputArray policies.

This is an example of what I meant with the above statement from my 
initial mail.
My current prototype allows that already. An argument policy isn't 
restricted to a single argument but it can deal with as many arguments 
as it likes.

The interface of the argument policy class has four methods that each 
policy has to implement:

- prepareWrapper(wm): This method can do changes to the signature of the 
wrapper function (like adding or removing arguments, for example).

- preCall(wm): This method has to return C++ source code that is put 
before the original function invocation

- postCall(wm): This method has to return C++ source code that is put 
after the original function invocation

- cleanup(wm): This method has to return C++ source code that contains 
cleanup code in the case an error has occurred somewhen after the 
preCall() code but before the postCall() code (which won't get executed).

The argumen 'wm' is an instance of a "WrapperManager" class that can be 
seen as the code creator for the entire wrapper function. It has some 
attributes and methods that can be used by the policies, such as:

- removeArg(index): Removes an argument from the wrapper function

- insertArg(index, arg): Adds a new argument to the wrapper function

- appendResult(varname): Adds a particular variable to the result tuple

- removeResult(varname): Removes a variable from the result tuple

- replaceCallArg(index, callstr): Replaces a variable that is used for 
the original function invocation.

- declareLocal(name, type, size=None, default=None): Declares a local 
variable and returns its actual name. To prevent name clashes, every 
variable that a policy uses must be declared.

For example, the Output class simply has to implement the 
prepareWrapper() method and only do three things:

- Remove the given argument: wm.removeArg(index)
- Declare a local variable that takes over the part of the input 
variable (the variable has the same name and base type than the original 
input variable): wm.declareLocal(name, type)
- Add the new local variable to the result: wm.appendResult(name)

> I want to say that "argument policies" feature should be rename. New
> feature should deal
> with an examples I showed. So it does not deal any more with single
> argument, but with
> whole function.
> 
> Bottom line - new name needed.
> 
> Suggestions:
> 
> [ function ] wrapper call policies
> 
> Comments?

At the beginning, I was calling them "argument transformers" and only 
later noticed that they were quite similar to "call policies" and then 
called them "argument policies" to express this similarity.
The main purpose of those classes is still to do something with the 
arguments (which includes the return value), so I don't see the 
necessity to rename them.

I'm against using "call policies" in their name because they actually 
have nothing to do with Boost.Python's call policies, so I think this 
would just lead to confusion.

- Matthias -

Re: [pygccxml-development] Feature proposal: "Argument policies"

[Roman Y. <rom...@gm...>]

Okay lets see if I understand you right.

class wrapper{

return_type function( arguments ) <=3D=3D here user can modify function sig=
nature

    variables <=3D=3D here user declares few variables and can bind them
with arguments
                          and/or return value

    make "native" call ( mix of variables and arguments ) <=3D=3D here I
think he should
                         be able to bind return argument to some variable

    construct return value  <=3D=3D here user can check result of "native" =
call and
                         throw exception or construct return value.

}

Now, the communication between steps done using function_wrapper_t
code creator ( =3D=3D wrapper_manager  ).
To be more precision: using variables and arguments collections.

I think, I've got you right.

If so, what interface you give to the user? Direct access to the code creat=
or?
I am not talking about wm.declare_local( ... ), as for me this is too
low-level.

Comments?

P.S.
Small comment about clean_up( :-) ) state.
In C++ developers can use RIIA technique. So I think, that clean up
state is nessecary. I don't want to force developers to use it, but I
also don't want to
provide "too mach" help with hand written clean up actions.


--
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Feature proposal: "Argument policies"

[Roman Y. <rom...@gm...>]

On 4/20/06, Niki Spahiev <ni...@vi...> wrote:
> Roman Yakovenko wrote:
> > I'd like to add next example, from what you can see that argument
> > policies should be able to deal with more complex case:
> >
> > //read from some file
> > int read( byte* buffer, unsigned int buffer_size );
> [...]
> > Bottom line - new name needed.
> >
> > Suggestions:
> >
> > [ function ] wrapper call policies
> >
> > Comments?
> >
>
> In SWIG they are called typemaps. There are input, output, exception and
> some other kinds of typemaps.

Thanks for input. Can you send me link to the relevant piece of documentati=
on?

> As SWIG is C++ parser too, IMHO most if
> the issues to handle are the same.

I am sure they do it.

> IIRC SWIG can (or is a planing to)
> save XML.

Do you mean that they are going to save C++ declarations as XML in file?

> Niki Spahiev
>

--
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Feature proposal: "Argument policies"

[Niki S. <ni...@vi...>]

Roman Yakovenko wrote:
>> In SWIG they are called typemaps. There are input, output, exception and
>> some other kinds of typemaps.
> 
> Thanks for input. Can you send me link to the relevant piece of documentation?

Sure: http://www.swig.org/Doc1.3/Typemaps.html

>> IIRC SWIG can (or is a planing to)
>> save XML.
> 
> Do you mean that they are going to save C++ declarations as XML in file?

http://www.rexx.com/~dkuhlman/swigxml.html

Bug plus of SWIG is that it can handle incomplete sources.

Niki Spahiev

Re: [pygccxml-development] Feature proposal: "Argument policies"

[Roman Y. <rom...@gm...>]

On 4/21/06, Niki Spahiev <ni...@vi...> wrote:
> Roman Yakovenko wrote:
> >> In SWIG they are called typemaps. There are input, output, exception a=
nd
> >> some other kinds of typemaps.
> >
> > Thanks for input. Can you send me link to the relevant piece of documen=
tation?
>
> Sure: http://www.swig.org/Doc1.3/Typemaps.html

Thanks

> >> IIRC SWIG can (or is a planing to)
> >> save XML.
> >
> > Do you mean that they are going to save C++ declarations as XML in file=
?
>
> http://www.rexx.com/~dkuhlman/swigxml.html
>
> Bug plus of SWIG is that it can handle incomplete sources.

pyplusplus does not depend on C++ parser at all :-). I think it will takes =
week
or two, to integrate new C++ parser with pygccxml.

Any way, it seems to me that you used both projects, right? So stay tuned :=
-)
we definitely will need your input here and there.

> Niki Spahiev

--
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Feature proposal: "Argument policies"

[Matthias B. <ba...@ir...>]

Roman Yakovenko wrote:
> Okay lets see if I understand you right.
> 
> class wrapper{
> 
> return_type function( arguments ) <== here user can modify function signature
> 
>     variables <== here user declares few variables and can bind them
> with arguments
>                           and/or return value
> 
>     make "native" call ( mix of variables and arguments ) <== here I
> think he should
>                          be able to bind return argument to some variable
> 
>     construct return value  <== here user can check result of "native" call and
>                          throw exception or construct return value.
> 
> }

I think you mostly got it. Just to clarify, here's the basic layout of 
the wrapper functions as they are currently generated (assuming there 
are n argument policies):

return_type wrapper_function_name( [self,] arguments )
{
   <local variable declarations>

   <precall code 1>
   <precall code 2>
   ...
   <precall code n>

   result = original_function( args );

   <postcall code 1>
   <postcall code 2>
   ...
   <postcall code n>

   return single_result_or_result_tuple;
}

- By default, the wrapper function has the same signature than the 
original function but the argument policies can modify the signature. 
When the function is supposed to be a class method the "self" argument 
is added as first argument.

- When more than 1 result variables get registered the return type 
automatically turns to "boost::python::tuple" and the return statement 
becomes a "return boost::python::make_tuple(args)" statement.

- The local variables are those that the arg policies have declared + a 
variable for the original return value (if there is one).

- The argument list for the original function call is a list of variable 
names. The arg policies can modify this list to "redirect" input or 
output (this includes the name of the result variable).

> If so, what interface you give to the user? Direct access to the code creator?
> I am not talking about wm.declare_local( ... ), as for me this is too
> low-level.

Policies that are independent of a particular project (such as Output, 
OutputArray, InputArray, etc) can be made part of pyplusplus and a user 
can use them for his own project. In this case, the user doesn't have to 
know how argument policies work internally.
If the standard policies don't fit the bill of the user, he has the 
option to write his own policies. Now the user must be familiar with the 
internals of the policy mechanism.

> Small comment about clean_up( :-) ) state.
> In C++ developers can use RIIA technique. So I think, that clean up
> state is nessecary. 

Should that read "is not necessary"...?

> I don't want to force developers to use it, but I
> also don't want to provide "too mach" help with hand written clean up actions.

What do you mean? Are you for or against the cleanup() method? (by the 
way, currently I don't have this implemented yet anyway as my current 
policies don't need any cleanup. But as soon as you have to allocate 
memory it might get necessary)

- Matthias -

Re: [pygccxml-development] Feature proposal: "Argument policies"

[Roman Y. <rom...@gm...>]

On 4/24/06, Matthias Baas <ba...@ir...> wrote:
> > If so, what interface you give to the user? Direct access to the code c=
reator?
> > I am not talking about wm.declare_local( ... ), as for me this is too
> > low-level.
>
> Policies that are independent of a particular project (such as Output,
> OutputArray, InputArray, etc) can be made part of pyplusplus and a user
> can use them for his own project. In this case, the user doesn't have to
> know how argument policies work internally.
> If the standard policies don't fit the bill of the user, he has the
> option to write his own policies. Now the user must be familiar with the
> internals of the policy mechanism.
>
> > Small comment about clean_up( :-) ) state.
> > In C++ developers can use RIIA technique. So I think, that clean up
> > state is nessecary.
>
> Should that read "is not necessary"...?

Yes of course.

>
> > I don't want to force developers to use it, but I
> > also don't want to provide "too mach" help with hand written clean up a=
ctions.
>
> What do you mean? Are you for or against the cleanup() method? (by the
> way, currently I don't have this implemented yet anyway as my current
> policies don't need any cleanup. But as soon as you have to allocate
> memory it might get necessary)

std::auto_ptr, boost::scoped_ptr, boost::shared_ptr ( with custom deleter )
should work


Okay. I glad we understand each other and has an agreement.
Can you start working on this. I mean prepare use-cases and what code shoul=
d
be generated. I think we will start with one common use-case:

int read( int size, char* buffer )

and

simple use case:  error status translated to exception.

I am working on function_t function_wrapper_t code creators. If you
don't mind can you create:
- C++ code for unit test
- C++ generated code ( this will guide us what code creators should do )
- configuration pseudo code. I mean from user point of view:

  mb =3D module_builder_t( ... )
  mf =3D mb.member_function( ... )
  mf.args_policies =3D ????????


I hope, I will finish with my task in the middle of next week. After
this we can start to work
on code generation. What do you think? Do you agree with the plan? May
be you prefer to
do something else?

> - Matthias -


--
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/