Thread: [pygccxml-development] Status of code inserters
Brought to you by:
mbaas,
roman_yakovenko
From: Allen B. <al...@vr...> - 2006-08-29 13:12:41
|
Matthias: What is the status of the code inserter's? Do you have any py++ addons that I can try out yet? -Allen |
From: Matthias B. <ba...@ir...> - 2006-08-29 15:48:21
|
Allen Bierbaum wrote: > Matthias: > > What is the status of the code inserter's? Do you have any py++ addons > that I can try out yet? I was just about to post a message on this subject.... Well, I don't have a readily available addon yet, but I think the general way to proceed is more or less clear. Right now, it's like I have all the individual pieces spread out before me but I'm not sure how to assemble them... The result from my earlier wiki entry was that there are up to two functions involved: 1) the wrapper function that gets called from Python and that makes a call "down to" C++ where the actual work is done 2) the virtual function that gets called from C++ and that may make a call to Python So for the general case, an arg policy must be prepared to deal with both functions and there must be an API that allows it to do so. This led to some updates which I have already entered into the wiki here: https://realityforge.vrsource.org/view/PyppApi/CodeInserter#Arg_policy_interface Basically, there are now two sets of methods to create pre-call and post-call code, one for the wrapper function and one for the virtual function. In the previous version I have proposed a class WrapperManager (or wrapper_manager_t) that represents the code creator for the wrapper function, i.e. that created the complete C++ source code. I have changed this and made this class only a helper class for creating C++ functions. Now it basically provides text substitution services which could be used by the existing Py++ code creators. Those code creators still create the rough layout of the code and insert special variables at appropriate places that will get substituted by the wrapper manager. These variables represent individual parts of the C++ function. I have documented the class in the wiki over here: https://realityforge.vrsource.org/view/PyppApi/Wrapper_manager This wrapper manager object is also the object that the individual methods of the arg policies (sometimes I also call them "code blocks" in the wiki) receive as input ("wm") and which they have to use to modify either the wrapper function or the virtual function. I have already implemented most parts of the mentioned classes (wrapper manager, code manager, policy base class). Now I could add it to pypp_api but it could only make use of the "wrapper function" part and not the virtual functions. So if everyone (in particular Roman) agrees with this approach, I would rather try to update the Py++ code creators so that they use the text substitution service from above. But that's where I'm a bit lost because I'm not familiar with the implementation and all the dependencies involved. So before I continue with this I'd rather like to know if Roman agrees to the general approach. The things that I'd like to do next are: - Add the "wrapper manager" class to Py++. It's more or less a stand-alone module that I would stick into its own directory inside "pyplusplus". - Add a new decoration attribute to the decl_wrappers that stores the "arg policy" objects. - Update the Py++ calldef code creators so that they use the arg policies via the above text substitution mechanism. Additionally, I'd like to introduce the "thread_safe" decoration attribute. This involves the same code as above so I think it safes time if it is done at the same time. This option should be implemented right in the Py++ code creators and not via the "arg policy" mechanism because it affects code on a lower level than what the arg policies are doing. - Matthias - |
From: Allen B. <al...@vr...> - 2006-08-29 18:33:49
|
Sounds like a well thought out plan to me. I would be more then happy to help test it out once you have it implemented and integrated into py++. -Allen Matthias Baas wrote: >Allen Bierbaum wrote: > > >>Matthias: >> >>What is the status of the code inserter's? Do you have any py++ addons >>that I can try out yet? >> >> > >I was just about to post a message on this subject.... > >Well, I don't have a readily available addon yet, but I think the >general way to proceed is more or less clear. Right now, it's like I >have all the individual pieces spread out before me but I'm not sure how >to assemble them... > >The result from my earlier wiki entry was that there are up to two >functions involved: > >1) the wrapper function that gets called from Python and that makes a >call "down to" C++ where the actual work is done > >2) the virtual function that gets called from C++ and that may make a >call to Python > >So for the general case, an arg policy must be prepared to deal with >both functions and there must be an API that allows it to do so. This >led to some updates which I have already entered into the wiki here: >https://realityforge.vrsource.org/view/PyppApi/CodeInserter#Arg_policy_interface >Basically, there are now two sets of methods to create pre-call and >post-call code, one for the wrapper function and one for the virtual >function. > >In the previous version I have proposed a class WrapperManager (or >wrapper_manager_t) that represents the code creator for the wrapper >function, i.e. that created the complete C++ source code. I have changed >this and made this class only a helper class for creating C++ functions. >Now it basically provides text substitution services which could be used >by the existing Py++ code creators. Those code creators still create the >rough layout of the code and insert special variables at appropriate >places that will get substituted by the wrapper manager. These >variables represent individual parts of the C++ function. I have >documented the class in the wiki over here: >https://realityforge.vrsource.org/view/PyppApi/Wrapper_manager > >This wrapper manager object is also the object that the individual >methods of the arg policies (sometimes I also call them "code blocks" in >the wiki) receive as input ("wm") and which they have to use to modify >either the wrapper function or the virtual function. > >I have already implemented most parts of the mentioned classes (wrapper >manager, code manager, policy base class). Now I could add it to >pypp_api but it could only make use of the "wrapper function" part and >not the virtual functions. So if everyone (in particular Roman) agrees >with this approach, I would rather try to update the Py++ code creators >so that they use the text substitution service from above. But that's >where I'm a bit lost because I'm not familiar with the implementation >and all the dependencies involved. > >So before I continue with this I'd rather like to know if Roman agrees >to the general approach. The things that I'd like to do next are: > >- Add the "wrapper manager" class to Py++. It's more or less a >stand-alone module that I would stick into its own directory inside >"pyplusplus". > >- Add a new decoration attribute to the decl_wrappers that stores the >"arg policy" objects. > >- Update the Py++ calldef code creators so that they use the arg >policies via the above text substitution mechanism. > > >Additionally, I'd like to introduce the "thread_safe" decoration >attribute. This involves the same code as above so I think it safes time >if it is done at the same time. This option should be implemented right >in the Py++ code creators and not via the "arg policy" mechanism because >it affects code on a lower level than what the arg policies are doing. > > >- Matthias - > > >------------------------------------------------------------------------- >Using Tomcat but need to do more? Need to support web services, security? >Get stuff done quickly with pre-integrated technology to make your job easier >Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo >http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >_______________________________________________ >pygccxml-development mailing list >pyg...@li... >https://lists.sourceforge.net/lists/listinfo/pygccxml-development > > > |
From: Matthias B. <ba...@ir...> - 2006-09-07 15:30:31
|
Allen Bierbaum wrote: > Sounds like a well thought out plan to me. I would be more then happy > to help test it out once you have it implemented and integrated into py++. If you still want to do some initial tests you could do so with the current svn version. However, only public (non-pure) virtual member functions currently support the function transformers, all other member types ignore them. There's no convenience method to add transformers to the decl_wrapper so you just have to manipulate the new list attribute "function_transformers" directly. Example: image.member_function("get_size").function_transformers.extend([output_t(1), output_t(2)]) The attribute "function_transformers" is just a list that contains all transformers/policies that should be applied. For the time being you also have to import the transformer objects explicitly: from pyplusplus.function_transformers.arg_policies import * After that, the following four policies are available: - output_t(index): Argument index (1-based) is an output variable Example: C++: void getValue(int& v) or void getValue(int* v) Policy: output_t(1) -> Python: v = getValue() - input_t(index): Argument index is an input variable Example: C++: void setValue(int& v) or void setValue(int* v) Policy: input_t(1) -> Python: setValue(v) - input_array_t(index, size): Handles an input array of a fixed size. Example: C++: void setVec3(double* v) Policy: input_array_t(1,3) -> Python: setVec3([x,y,z]) - output_array_t(index, size): Handles an output array of a fixed size. Example: C++: void getVec3(double* v) Policy: output_array_t(1,3) -> Python: x,y,z = getVec3() - Matthias - |
From: Roman Y. <rom...@gm...> - 2006-08-29 19:05:01
|
Matthias I will be busy for next few days. In general I agree, but I will take time to think about it. -- Roman Yakovenko C++ Python language binding http://www.language-binding.net/ |
From: Roman Y. <rom...@gm...> - 2006-08-30 18:56:08
|
Good evening. I just finished to read code inserters document. * I think we should give a name to the feature. I propose "Function Transformation". This is basically what we do. * Untill this feature, the only classes that generate code were code creators. Py++ design\architecture is described here ( http://language-binding.net/pyplusplus/documentation/architecture.html ). I prefer to stay with it, hence I ask you how you see this feature is integrated within Py++. * WrapperManager. It is presented in the document as a "Template" class. From my experience it is not possible to build such class without using tons of if-else statements. * Coding convention. I really don't want to introduce I propose to begin to work on the feature. * To create C++ code and Boost.Python expected code. Thus we will see examples and the goal. Also I think it will give us better understanding of the subject. I can write do this. * To create design. I'd like to see classes and methods, with responcibility description. This will give us an ability to split the work. Also we will be able to take a use case and to see what code user will have to write in order to apply a transformation. -- Roman Yakovenko C++ Python language binding http://www.language-binding.net/ |
From: Matthias B. <ba...@ir...> - 2006-08-31 08:49:11
|
Roman Yakovenko wrote: > * I think we should give a name to the feature. I propose > "Function Transformation". > This is basically what we do. In my current implementation the "base" class for the "arg policy" objects is called code_block_base_t. But something like func_transformer_t or similar is fine with me as well. After all, this class is more or less an internal class that doesn't have to be used by the end user, its main purpose is to document the interface that the user has to implement. As to the name that is exposed to the user, in the wiki and doc strings I'm still using the name "argument policy" as I found it still describes the main usages best. All the cases where I want to use this feature for my Maya bindings deal with arguments (including the return value = output argument), see the use-cases in the wiki. Previously, the only exception would have been the thread-safety stuff, but meanwhile I think this should not be implemented using the "arg policy" mechanism anyway because this is not straightforward to do. Instead, I'd recommend to implement that separately. > * Untill this feature, the only classes that generate code were code > creators. > Py++ design\architecture is described here > ( > http://language-binding.net/pyplusplus/documentation/architecture.html ). > I prefer to stay with it, hence I ask you how you see this feature > is integrated within Py++. The existing Py++ calldef code creators will still remain the code creators for the calldefs. What would slightly change is the way *how* they create their code, i.e. they would have to add some variables at appropriate places to the code and have that variables substituted by the proposed text substitution services. > * WrapperManager. It is presented in the document as a "Template" class. > From my experience it is not possible to build such class without using > tons of if-else statements. What do you mean with "Template" class? Are you referring to the template string argument that is provided by the user of the class? (this is just an argument name) > * Coding convention. I really don't want to introduce Uhm, to introduce what? I already tried to stick to the Py++ conventions. By the way, yesterday I renamed the "wrapper manager" class and now I'm calling it "substitution manager" class. I think this describes the functionality better. The actual class name in the implementation is "subst_manager_t". The entire sub-package is currently called "wrapper_subst". > I propose to begin to work on the feature. > > * To create C++ code and Boost.Python expected code. Thus we will see > examples and the goal. Also I think it will give us better understanding > of the subject. > > I can write do this. ok. See the wiki for use cases. At least for the "wrapper" part, I am already using code in my Maya bindings for quite a while now. Its just that I couldn't test the "virtual" part yet. For testing the new implementation I have already a "new-style" Output policy. > * To create design. I'd like to see classes and methods, with > responcibility > description. This will give us an ability to split the work. Also we > will be able > to take a use case and to see what code user will have to write in order > to apply a transformation. At the lowest level, there is the substitution manager class (formerly known as wrapper manager) which has to be used by the Py++ code creators to "enhance" their generated code (see the wiki for examples). Some things to note here: - The modified code might require additional include files. The caller (i.e. the Py++ code creators) must ensure that these include files actually get included. - There might be some additional support code necessary that is generated by the substitution manager. But I think this can probably be handled much in the same way as the previous point, so it's not really a new point. - The modified code (at least the wrapper) might have a different signature than the original code. The Py++ code creators have to deal with this. In particular, it might happen that two overloaded functions end up with the same signature. - The user might add "arg policies" to methods that wouldn't have spawned a wrapper function otherwise. So the presence of policies is already a sufficient condition for the creation of a wrapper function. I have already implemented the substitution sub-package (see the wiki), but the integration into the Py++ code is not yet done. - Matthias - |
From: Roman Y. <rom...@gm...> - 2006-09-01 08:02:52
|
On 8/31/06, Matthias Baas <ba...@ir...> wrote: > In my current implementation the "base" class for the "arg policy" > objects is called code_block_base_t. But something like > func_transformer_t or similar is fine with me as well. After all, this > class is more or less an internal class that doesn't have to be used by > the end user, its main purpose is to document the interface that the > user has to implement. > > As to the name that is exposed to the user, in the wiki and doc strings > I'm still using the name "argument policy" as I found it still describes > the main usages best. All the cases where I want to use this feature for > my Maya bindings deal with arguments (including the return value = > output argument), see the use-cases in the wiki. Does it mean that you agree with proposed name? >Previously, the only > exception would have been the thread-safety stuff, but meanwhile I think > this should not be implemented using the "arg policy" mechanism anyway > because this is not straightforward to do. Instead, I'd recommend to > implement that separately. Can we put "thread safe" aside for a time being? We will come back to it. > > * Untill this feature, the only classes that generate code were code > > creators. > > Py++ design\architecture is described here > > ( > > http://language-binding.net/pyplusplus/documentation/architecture.html ). > > I prefer to stay with it, hence I ask you how you see this feature > > is integrated within Py++. > > The existing Py++ calldef code creators will still remain the code > creators for the calldefs. What would slightly change is the way *how* > they create their code, i.e. they would have to add some variables at > appropriate places to the code and have that variables substituted by > the proposed text substitution services. So, the whole feature is to create plug-ins into ( calldefs ) code creators? Nice :-) > > * WrapperManager. It is presented in the document as a "Template" class. > > From my experience it is not possible to build such class without using > > tons of if-else statements. > > What do you mean with "Template" class? Are you referring to the > template string argument that is provided by the user of the class? > (this is just an argument name) String substitution. > > * Coding convention. I really don't want to introduce > Uhm, to introduce what? > I already tried to stick to the Py++ conventions. Thanks, I pushed "send" button to early :-) > By the way, yesterday I renamed the "wrapper manager" class and now I'm > calling it "substitution manager" class. I think this describes the > functionality better. The actual class name in the implementation is > "subst_manager_t". The entire sub-package is currently called > "wrapper_subst". This is the name I looked for! "substitution manager". One comment: do you mind to write full package and class names? We always can introduce aliases in Python. I think that full names are easy for maintenance. > > * To create design. I'd like to see classes and methods, with > > responcibility > > description. This will give us an ability to split the work. Also we > > will be able > > to take a use case and to see what code user will have to write in order > > to apply a transformation. > > At the lowest level, there is the substitution manager class (formerly > known as wrapper manager) which has to be used by the Py++ code creators > to "enhance" their generated code (see the wiki for examples). Some > things to note here: > > - The modified code might require additional include files. The caller > (i.e. the Py++ code creators) must ensure that these include files > actually get included. I plan something different here: I want to associate include files with classes and other global declarations. Thus Py++ will generate better code( right now all cpp files contains includes of the whole project ). This feature will work for our case too. > - There might be some additional support code necessary that is > generated by the substitution manager. But I think this can probably be > handled much in the same way as the previous point, so it's not really a > new point. > > - The modified code (at least the wrapper) might have a different > signature than the original code. The Py++ code creators have to deal > with this. In particular, it might happen that two overloaded functions > end up with the same signature. Not a problem. > - The user might add "arg policies" to methods that wouldn't have > spawned a wrapper function otherwise. So the presence of policies is > already a sufficient condition for the creation of a wrapper function. > > > I have already implemented the substitution sub-package (see the wiki), > but the integration into the Py++ code is not yet done. > I will take a look on it pretty soon. -- Roman Yakovenko C++ Python language binding http://www.language-binding.net/ |
From: Matthias B. <ba...@ir...> - 2006-09-01 17:03:12
|
Roman Yakovenko wrote: > Does it mean that you agree with proposed name? I've already started renaming the code blocks to function transformers. So yes, when referring to the general mechanism I have no objections to use the name "function transformers". In pypp_api however, I will still keep the function setArgPolicy() for now as I believe this describes and documents better what the user is doing. >> Previously, the only >> exception would have been the thread-safety stuff, but meanwhile I think >> this should not be implemented using the "arg policy" mechanism anyway >> because this is not straightforward to do. Instead, I'd recommend to >> implement that separately. > > Can we put "thread safe" aside for a time being? We will come back to it. Right, it's not a use-case for the function transformers anyway. > So, the whole feature is to create plug-ins into ( calldefs ) code > creators? > Nice :-) I haven't seen it that way yet, but yes, it's a valid way to look at it. The function transformers correspond to plugins then. >> By the way, yesterday I renamed the "wrapper manager" class and now I'm >> calling it "substitution manager" class. I think this describes the >> functionality better. The actual class name in the implementation is >> "subst_manager_t". The entire sub-package is currently called >> "wrapper_subst". > > This is the name I looked for! "substitution manager". One comment: do you > mind to write full package and class names? We always can introduce aliases > in Python. I think that full names are easy for maintenance. ok, I'll rename as follows: subst_manager_t -> substitution_manager_t wrapper_subst -> wrapper_substitution (It's just that there's usually a typo when I try to type "subsi^Htiut^H^Htui^Htion"... ;) >> - The modified code might require additional include files. The caller >> (i.e. the Py++ code creators) must ensure that these include files >> actually get included. > > I plan something different here: I want to associate include files with > classes Sounds like a good idea! > and other global declarations. Thus Py++ will generate better code( > right now > all cpp files contains includes of the whole project ). This feature > will work for our case too. Yes, my above point is still the same though. The function transformer will just tell the code creator that it is going to create code that requires include file "strawberry.h" and now it's the code creator's responsibility to make sure that "strawberry.h" gets included at the right place. How this is done is up to the code creator, it may create another include code creator or it may use the new scheme from above. Yesterday and today I have already done the following things: - Added a property "function_transformers" to calldef_t (and while I was at it I already added the property "thread_safe" as well) - I modified mem_fun_v_wrapper_t and mem_fun_v_t so that the function transformers are used. Now the code from the wiki (option 3) is created for public virtual member functions. So far everything looks fine, but I have a slight problem with Boost.Python's override. For example, we might have the following code: virtual void spam( int & n ) { if( bp::override func_spam = this->get_override( "spam" ) ) { // The Python method will return an integer n = func_spam(); } else { Viking::spam( n ); } } This function will work and in this example everything is fine. But there might be other Python functions that return several values so the line n = func_spam(); doesn't work. So how can I invoke the function and just receive a boost::python::object object as result without that Boost.Python tries to convert it to a C++ value? I tried the following which doesn't work: boost::python::object result = func_spam(); When I run this code I get the following error: TypeError: No registered converter was able to extract a C++ reference to type boost::python::api::object from this Python object of type tuple Any ideas? - Matthias - |
From: Roman Y. <rom...@gm...> - 2006-09-01 17:58:38
|
On 9/1/06, Matthias Baas <ba...@ir...> wrote: > Roman Yakovenko wrote: > > Does it mean that you agree with proposed name? > > I've already started renaming the code blocks to function transformers. > So yes, when referring to the general mechanism I have no objections to > use the name "function transformers". Matthias where is the code you are talking about? Do you want to create function_transformers package under pyplusplus and move all relevant code to it? > > So, the whole feature is to create plug-ins into ( calldefs ) code > > creators? > > Nice :-) > > I haven't seen it that way yet, but yes, it's a valid way to look at it. > The function transformers correspond to plugins then. Finally I understand the feature. Thanks. > >> By the way, yesterday I renamed the "wrapper manager" class and now I'm > >> calling it "substitution manager" class. I think this describes the > >> functionality better. The actual class name in the implementation is > >> "subst_manager_t". The entire sub-package is currently called > >> "wrapper_subst". > > > > This is the name I looked for! "substitution manager". One comment: do you > > mind to write full package and class names? We always can introduce aliases > > in Python. I think that full names are easy for maintenance. > > ok, I'll rename as follows: > > subst_manager_t -> substitution_manager_t > wrapper_subst -> wrapper_substitution > > (It's just that there's usually a typo when I try to type > "subsi^Htiut^H^Htui^Htion"... ;) You are not alone. This is one of the reasons I use properties every where. If I see that my script does not works as expected I can put break point in "get" and see whether it is called or not. > >> - The modified code might require additional include files. The caller > >> (i.e. the Py++ code creators) must ensure that these include files > >> actually get included. > > > > I plan something different here: I want to associate include files with > > classes > > Sounds like a good idea! Thanks. I will implement it before next release. > > and other global declarations. Thus Py++ will generate better code( > > right now > > all cpp files contains includes of the whole project ). This feature > > will work for our case too. > > Yes, my above point is still the same though. The function transformer > will just tell the code creator that it is going to create code that > requires include file "strawberry.h" and now it's the code creator's > responsibility to make sure that "strawberry.h" gets included at the > right place. My points are: 1. creating function transformer is a task that requieres user attention to be concentrated on types and function imlementation. Adding something else to it will disturb to the user 2. Py++ will have only one way to control included files from declarations. We can discuss this point later. The first implementation could be done as you propose. But I propmise you we will back to it later :-) >How this is done is up to the code creator, it may create > another include code creator or it may use the new scheme from above. Only code creators factory creates them. In most cases code creators does not know about each other. > Yesterday and today I have already done the following things: > > - Added a property "function_transformers" to calldef_t Do you mean decl_wrappers.calldef_t, right > (and while I was > at it I already added the property "thread_safe" as well) Please, remove thread_safe. We will add this later. > - I modified mem_fun_v_wrapper_t and mem_fun_v_t so that the function > transformers are used. Now the code from the wiki (option 3) is created > for public virtual member functions. So far everything looks fine, but I > have a slight problem with Boost.Python's override. For example, we > might have the following code: > > virtual void spam( int & n ) { > if( bp::override func_spam = this->get_override( "spam" ) ) > { > // The Python method will return an integer > n = func_spam(); > } > else > { > Viking::spam( n ); > } > } > > This function will work and in this example everything is fine. But > there might be other Python functions that return several values so the line > > n = func_spam(); > > doesn't work. So how can I invoke the function and just receive a > boost::python::object object as result without that Boost.Python tries > to convert it to a C++ value? > > I tried the following which doesn't work: > > boost::python::object result = func_spam(); > > When I run this code I get the following error: > > TypeError: No registered converter was able to extract a C++ reference > to type boost::python::api::object from this Python object of type tuple > > Any ideas? Please commit all your code :-). -- Roman Yakovenko C++ Python language binding http://www.language-binding.net/ |
From: Matthias B. <ba...@ir...> - 2006-09-03 09:08:53
|
Roman Yakovenko wrote: > Matthias where is the code you are talking about? Do you want to create > function_transformers package under pyplusplus and move all relevant code > to it? I haven't committed the code yet because I also did changes to Py++ itself (and not only pypp_api or the independent sub-package) and committing everything would put Py++ into an inconsistent state where it would probably break everyone else's projects. Well, I could already commit the new sub-package which is currently called "wrapper_substitution" (would you like me to rename it to "function_transformers"?) and which contains the base functionality (i.e. the base function transformer mechanism + substitution services). This definitely won't break any existing code. Committing calldef_t is also safe as it only adds features but for code_creators/calldef.py I would rather suggest to post that file here so that you can inspect them without that we get a broken version of Py++ inside the repository. >> Yesterday and today I have already done the following things: >> >> - Added a property "function_transformers" to calldef_t > > Do you mean decl_wrappers.calldef_t, right Right. >> (and while I was >> at it I already added the property "thread_safe" as well) > > Please, remove thread_safe. We will add this later. Well, my thinking was that even though the features are mostly independent from each other they affect the same source code and doing them separately means that the very same code has to be touched twice. Another reason is that this is rather high priority for me because it prevents me from doing another release of the Maya bindings (Python plugins currently don't work at all when multi-threading is enabled). But I can leave this attribute only in my local copy for now, that's no problem. - Matthias - |