pygccxml-development

Re: [pygccxml-development] Classes missing from the documentation

[Roman Y. <rom...@gm...>]

On 8/31/06, Matthias Baas <ba...@ir...> wrote:
> Hi,
>
> does anybody have an idea why some classes (such as decl_wrapper_t or
> code_creator_t) don't show up in the epydoc generated documentation?
>
> When removing some parts (e.g. the alias property in decl_wrapper_t)
> then suddenly the class does show up. But epydoc never generates a
> warning or error...? (I've even updated to latest alpha3 of epydoc)

I don't know, but it is possible to open bug to epydoc. The author is
pretty responsive.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

[pygccxml-development] Feature request: Return list of files created by module builder

[Allen B. <al...@vr...>]

I have run into a build problem that led me to a feature idea.

I would like to see a way to have the module_builder_t.split_module() 
method return a list of all the files it created.

I have been running into a problem lately where a later run of my 
generation script may generate a different set of source files then a 
previous run.  (for example removing a class)  Py++ correctly generates 
the new code, but the old .cpp files are still in the source tree and 
get picked up by my build.  If module builder returned a list of files, 
then I could check this against the files on disk and automatically 
remove old files that I don't need anymore.

Thoughts?  Anyone else think this would be useful or do you use another 
way to achieve the same results?

-Allen

[pygccxml-development] Classes missing from the documentation

[Matthias B. <ba...@ir...>]

Hi,

does anybody have an idea why some classes (such as decl_wrapper_t or 
code_creator_t) don't show up in the epydoc generated documentation?

When removing some parts (e.g. the alias property in decl_wrapper_t) 
then suddenly the class does show up. But epydoc never generates a 
warning or error...? (I've even updated to latest alpha3 of epydoc)

- Matthias -

Re: [pygccxml-development] templates (again)

[Allen B. <al...@vr...>]

Neal Becker wrote:

>On Thursday 31 August 2006 4:58 am, Matthias Baas wrote:
>  
>
>>Neal Becker wrote:
>>    
>>
>>>// file test.h
>>>#include <boost/numeric/ublas/vector.hpp>
>>>using namespace boost::numeric::ublas;
>>>
>>>typedef vector<int> vector_int;
>>>
>>># File: test_setup.py
>>>from pypp_api import *
>>>
>>>mod = ModuleBuilder(
>>>    headerFiles = "test.h",
>>>    moduleName = "test"
>>>)
>>>
>>>root = mod.parse()
>>>root.Class("vector_int").expose()
>>>
>>>mod.writeModule()
>>>
>>>result:
>>>[...]
>>>    raise RuntimeError, "Query produced no results (filter: %s)"%filter
>>>RuntimeError: Query produced no results (filter: (name=='vector_int') &
>>>(type==CLASS) & (parent=='::'))
>>>      
>>>
>>Well, I have never wrapped templates so far. This part of pypp_api was
>>written by Allen. You are not using the Template() method, so I don't
>>know if this could already work or not. Or maybe the result is empty
>>because you are actually querying for a typedef instead of a class...?
>>Try Decl() instead of Class():
>>
>>root.Decl("vector_int").expose()
>>
>>    
>>
I think the problem you are running into is that the decl for the 
typedef is really just a reference to the real type.

You have to do a little more processing to get to the full type.

The code that I am using right now is in the TemplateBuilder and 
TemplateWrapper classes in the goodies contrib:

http://svn.sourceforge.net/viewvc/pygccxml/pyplusplus_dev/contrib/goodies/goodie_utils.py?revision=500&view=markup

To use this code you would do something like this:

    tb = TemplateBuilder()
    vec3f_t = tb.Template("osg::vector<float,3>")

    # Add autogen code to a header that is included in myheaders list
    # using tb.buildAutogenContents() which returns the text that needs
    # processed
    mb = moduble_builder_t([myheaders], ...)
    tb.process(mb)
 
    vec3f = vec3f_t.decl
    vec3f.method("getSize").exclude()

    mb.builde_code_creator(...)


I am not sure if you will want to use this helper class directly, but it 
should at least show you all the steps involved in getting templates 
wrapped.

-Allen


>>    
>>
>
>OK.  This time no error, but no code generated either!
>// This file has been generated by Py++.
>
>#include "boost/python.hpp"
>
>#include "test.h"
>
>namespace bp = boost::python;
>
>BOOST_PYTHON_MODULE(test){
>
>}
>---test.h
>#include <boost/numeric/ublas/vector.hpp>
>typedef boost::numeric::ublas::vector<int> vector_int;
>
>inline void instantiate() {
>  sizeof (vector_int);
>}
>
># File: pypp_setup.py
>
>from pypp_api import *
>
>mod = ModuleBuilder(
>    headerFiles = "test.h",
>    moduleName = "test"
>)
>
>root = mod.parse()
>root.Decl("vector_int").expose()
>
>mod.writeModule()
>
>
>-------------------------------------------------------------------------
>Using Tomcat but need to do more? Need to support web services, security?
>Get stuff done quickly with pre-integrated technology to make your job easier
>Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>_______________________________________________
>pygccxml-development mailing list
>pyg...@li...
>https://lists.sourceforge.net/lists/listinfo/pygccxml-development
>
>  
>

Re: [pygccxml-development] templates (again)

[Neal B. <ndb...@gm...>]

On Thursday 31 August 2006 4:58 am, Matthias Baas wrote:
> Neal Becker wrote:
> > // file test.h
> > #include <boost/numeric/ublas/vector.hpp>
> > using namespace boost::numeric::ublas;
> >
> > typedef vector<int> vector_int;
> >
> > # File: test_setup.py
> > from pypp_api import *
> >
> > mod = ModuleBuilder(
> >     headerFiles = "test.h",
> >     moduleName = "test"
> > )
> >
> > root = mod.parse()
> > root.Class("vector_int").expose()
> >
> > mod.writeModule()
> >
> > result:
> > [...]
> >     raise RuntimeError, "Query produced no results (filter: %s)"%filter
> > RuntimeError: Query produced no results (filter: (name=='vector_int') &
> > (type==CLASS) & (parent=='::'))
>
> Well, I have never wrapped templates so far. This part of pypp_api was
> written by Allen. You are not using the Template() method, so I don't
> know if this could already work or not. Or maybe the result is empty
> because you are actually querying for a typedef instead of a class...?
> Try Decl() instead of Class():
>
> root.Decl("vector_int").expose()
>
>

OK.  This time no error, but no code generated either!
// This file has been generated by Py++.

#include "boost/python.hpp"

#include "test.h"

namespace bp = boost::python;

BOOST_PYTHON_MODULE(test){

}
---test.h
#include <boost/numeric/ublas/vector.hpp>
typedef boost::numeric::ublas::vector<int> vector_int;

inline void instantiate() {
  sizeof (vector_int);
}

# File: pypp_setup.py

from pypp_api import *

mod = ModuleBuilder(
    headerFiles = "test.h",
    moduleName = "test"
)

root = mod.parse()
root.Decl("vector_int").expose()

mod.writeModule()

Re: [pygccxml-development] templates (again)

[Matthias B. <ba...@ir...>]

Neal Becker wrote:
> // file test.h
> #include <boost/numeric/ublas/vector.hpp>
> using namespace boost::numeric::ublas;
> 
> typedef vector<int> vector_int;
> 
> # File: test_setup.py
> from pypp_api import *
> 
> mod = ModuleBuilder(
>     headerFiles = "test.h",
>     moduleName = "test"
> )
> 
> root = mod.parse()
> root.Class("vector_int").expose()
> 
> mod.writeModule()
> 
> result:
> [...]
>     raise RuntimeError, "Query produced no results (filter: %s)"%filter
> RuntimeError: Query produced no results (filter: (name=='vector_int') & 
> (type==CLASS) & (parent=='::'))

Well, I have never wrapped templates so far. This part of pypp_api was 
written by Allen. You are not using the Template() method, so I don't 
know if this could already work or not. Or maybe the result is empty 
because you are actually querying for a typedef instead of a class...?
Try Decl() instead of Class():

root.Decl("vector_int").expose()


- Matthias -

Re: [pygccxml-development] Status of code inserters

[Matthias B. <ba...@ir...>]

Roman Yakovenko wrote:
> * I think we should give a name to the feature. I propose
>   "Function Transformation".
>  This is basically what we do.

In my current implementation the "base" class for the "arg policy" 
objects is called code_block_base_t. But something like 
func_transformer_t or similar is fine with me as well. After all, this 
class is more or less an internal class that doesn't have to be used by 
the end user, its main purpose is to document the interface that the 
user has to implement.

As to the name that is exposed to the user, in the wiki and doc strings 
I'm still using the name "argument policy" as I found it still describes 
the main usages best. All the cases where I want to use this feature for 
my Maya bindings deal with arguments (including the return value = 
output argument), see the use-cases in the wiki. Previously, the only 
exception would have been the thread-safety stuff, but meanwhile I think 
this should not be implemented using the "arg policy" mechanism anyway 
because this is not straightforward to do. Instead, I'd recommend to 
implement that separately.

> * Untill this feature, the only classes that generate code were code 
> creators.
>   Py++ design\architecture is described here
>   ( 
> http://language-binding.net/pyplusplus/documentation/architecture.html ).
>   I prefer to stay with it, hence I ask you how you see this feature
> is integrated  within Py++.

The existing Py++ calldef code creators will still remain the code 
creators for the calldefs. What would slightly change is the way *how* 
they create their code, i.e. they would have to add some variables at 
appropriate places to the code and have that variables substituted by 
the proposed text substitution services.

> * WrapperManager. It is presented in the document as a "Template" class. 
> From my experience it is not possible to build such class without using
> tons of if-else statements.

What do you mean with "Template" class? Are you referring to the 
template string argument that is provided by the user of the class? 
(this is just an argument name)

> * Coding convention. I really don't want to introduce

Uhm, to introduce what?
I already tried to stick to the Py++ conventions.

By the way, yesterday I renamed the "wrapper manager" class and now I'm 
calling it "substitution manager" class. I think this describes the 
functionality better. The actual class name in the implementation is 
"subst_manager_t". The entire sub-package is currently called 
"wrapper_subst".

> I propose to begin to work on the feature.
> 
> * To create C++ code and Boost.Python expected code. Thus we will see
>   examples and the goal. Also I think it will give us better understanding
>   of the subject.
> 
>   I can write do this.

ok. See the wiki for use cases. At least for the "wrapper" part, I am 
already using code in my Maya bindings for quite a while now. Its just 
that I couldn't test the "virtual" part yet.
For testing the new implementation I have already a "new-style" Output 
policy.

> * To create design. I'd like to see classes and methods, with 
> responcibility
>  description. This will give us an ability to split the work. Also we
> will be able
>  to take a use case and to see what code user will have to write in order
>  to apply a transformation.

At the lowest level, there is the substitution manager class (formerly 
known as wrapper manager) which has to be used by the Py++ code creators 
to "enhance" their generated code (see the wiki for examples). Some 
things to note here:

- The modified code might require additional include files. The caller 
(i.e. the Py++ code creators) must ensure that these include files 
actually get included.

- There might be some additional support code necessary that is 
generated by the substitution manager. But I think this can probably be 
handled much in the same way as the previous point, so it's not really a 
new point.

- The modified code (at least the wrapper) might have a different 
signature than the original code. The Py++ code creators have to deal 
with this. In particular, it might happen that two overloaded functions 
end up with the same signature.

- The user might add "arg policies" to methods that wouldn't have 
spawned a wrapper function otherwise. So the presence of policies is 
already a sufficient condition for the creation of a wrapper function.


I have already implemented the substitution sub-package (see the wiki), 
but the integration into the Py++ code is not yet done.


- Matthias -

Re: [pygccxml-development] Status of code inserters

[Roman Y. <rom...@gm...>]

Good evening.  I just finished to read code inserters document.

* I think we should give a name to the feature. I propose
   "Function Transformation".
  This is basically what we do.

* Untill this feature, the only classes that generate code were code creators.
   Py++ design\architecture is described here
   ( http://language-binding.net/pyplusplus/documentation/architecture.html ).
   I prefer to stay with it, hence I ask you how you see this feature
is integrated
   within Py++.

* WrapperManager. It is presented in the document as a "Template" class. From
  my experience it is not possible to build such class without using
tons of if-else
  statements.

* Coding convention. I really don't want to introduce

I propose to begin to work on the feature.

* To create C++ code and Boost.Python expected code. Thus we will see
   examples and the goal. Also I think it will give us better understanding
   of the subject.

   I can write do this.

* To create design. I'd like to see classes and methods, with responcibility
  description. This will give us an ability to split the work. Also we
will be able
  to take a use case and to see what code user will have to write in order
  to apply a transformation.


-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

[pygccxml-development] templates (again)

[Neal B. <ndb...@gm...>]

// file test.h
#include <boost/numeric/ublas/vector.hpp>
using namespace boost::numeric::ublas;

typedef vector<int> vector_int;

# File: test_setup.py
from pypp_api import *

mod = ModuleBuilder(
    headerFiles = "test.h",
    moduleName = "test"
)

root = mod.parse()
root.Class("vector_int").expose()

mod.writeModule()

result:
INFO gccxml 
cmd: /usr/bin/gccxml  -I"/home/nbecker/tmp"   "test.h" -fxml="/tmp/tmpjsHti0.xml"
Completed parsing in 6s.
Traceback (most recent call last):
  File "test.setup.py", line 11, in ?
    root.Class("vector_int").expose()
  
File "/usr/local/src/pygccxml/pyplusplus_dev/contrib/pypp_api/pypp_api/declwrapper.py", 
line 570, in Class
    return self.Classes(name, assert_count=1, **args)
  
File "/usr/local/src/pygccxml/pyplusplus_dev/contrib/pypp_api/pypp_api/declwrapper.py", 
line 538, in Classes
    return self.Decls(name=name, type=type|CLASS, **args)
  
File "/usr/local/src/pygccxml/pyplusplus_dev/contrib/pypp_api/pypp_api/declwrapper.py", 
line 510, in Decls
    raise RuntimeError, "Query produced no results (filter: %s)"%filter
RuntimeError: Query produced no results (filter: (name=='vector_int') & 
(type==CLASS) & (parent=='::'))

Re: [pygccxml-development] Comments on module_builder cache functionality

[Allen B. <al...@vr...>]

Roman Yakovenko wrote:

> On 8/29/06, Allen Bierbaum <al...@vr...> wrote:
>
>> So now I have to figure out how to get this to work with module builder
>> cache and there is a problem.  Namely, in module builder the call to
>> parse_declarations does not have access to the list of files that gccxml
>> had to include.  The only place where these files seem to be available
>> is in source_reader.py on line 237.  It does not appear that this
>> information remains in the pygccxml tree but I could be wrong.  Also,
>> the files list is only used to update the decl cache so it is not
>> available later during execution.
>>
>> Does anybody know a way to either a) get the computed file list from
>> source_reader back into the module builder
>
>
> You don't need this
>
>>  or b) get a list of files
>> that were included from a pygccxml decl tree?
>
>
> pygccxml.declarations.declarations_files
> But you also don't need this functionality

How would I call update then in the interface you propose below?  I need 
the list of include files if that part of the key check is going to work.

>
> My idea was next:
>
> class mbcache_t( pygccxml.parser.base_cache_t ):
>    def __init__( self, *args ):
>         pygccxml.parser.base_cache_t.__init__( self )
>         test whether there is a need to load from cache on disk
>         self.decls_cache = pygccxml.parser.file_cache_t( file_name )
>
>   def update( self, .... ):
>         self.decls_cache.update( self,  .... )
>
>   def flush( self, ... )
>        self.decls_cache.flush()
>        write to disk files passed to module_builder_t
>
> Thus, mbcache_t only needs to warry about files list that was passed to
> module_builder_t
>
> I think this is pretty simple idea, that will take few hours to 
> implement.

Maybe I am missing something, but what I don't see is how the interface 
provides the capabilities I need.  It would not be difficult for me to 
write a little helper class that wrapped the cache file and loading if 
that is the issue, but I don't see a good way to reuse the file_cache_t 
code or really a good reason to do so.  What I need is different and 
smaller.

I need something like:

class mbcache_t(object):
   def __init__(self, cache_file)
      self.cache_file = file(cache_file)

   def getCachedTree(self, sourceFiles, configuration):
      # Read and test prefix signature against sourceFiles and configuration
      # If no match, early exit
      # Read decl tree
      # Read signature and list of files used to create cache
      # If one of those files has changed, then return None
      # Else return tree

   def updateCachedTree(self, sourceFiles, configuration, includedFiles, 
declTree):
      # Open cache file for writing
      # Compute prefix sig and dump it
      # Dump the decl tree
      # Compute and dump signature for and includeFile list
      # Save and close the file

I can easily create a class like this and use it if you think that will 
make the code more modular.  But I hope you can see this is quite a bit 
different from the cache map that we use in pygccxml.  There is no map, 
there is no need for dirty/used flag, it works for multiple files, it is 
smart enough to only load if needed.  I would reuse the signature code, 
but the rest would be unique.

But, back to the question at hand.  How would I get the "includedFiles" 
for the updateCacheTree call?  Note that this call would need to happen 
builder.__init__ where all I have access to is the self.__global_ns decl 
tree that I am caching.  The gccxml scanner that is used to pass the 
parameters to the file_cache_t.update() method is not available at this 
level in the code.

Any ideas?

-Allen

>
> What do you think?
>

Re: [pygccxml-development] Status of code inserters

[Roman Y. <rom...@gm...>]

Matthias I will be busy for next few days. In general I agree, but
I will take time to think about it.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Status of code inserters

[Allen B. <al...@vr...>]

Sounds like a well thought out plan to me.  I would be more then happy 
to help test it out once you have it implemented and integrated into py++.

-Allen

Matthias Baas wrote:

>Allen Bierbaum wrote:
>  
>
>>Matthias:
>>
>>What is the status of the code inserter's?  Do you have any py++ addons 
>>that I can try out yet?
>>    
>>
>
>I was just about to post a message on this subject....
>
>Well, I don't have a readily available addon yet, but I think the 
>general way to proceed is more or less clear. Right now, it's like I 
>have all the individual pieces spread out before me but I'm not sure how 
>to assemble them...
>
>The result from my earlier wiki entry was that there are up to two 
>functions involved:
>
>1) the wrapper function that gets called from Python and that makes a 
>call "down to" C++ where the actual work is done
>
>2) the virtual function that gets called from C++ and that may make a 
>call to Python
>
>So for the general case, an arg policy must be prepared to deal with 
>both functions and there must be an API that allows it to do so. This 
>led to some updates which I have already entered into the wiki here: 
>https://realityforge.vrsource.org/view/PyppApi/CodeInserter#Arg_policy_interface
>Basically, there are now two sets of methods to create pre-call and 
>post-call code, one for the wrapper function and one for the virtual 
>function.
>
>In the previous version I have proposed a class WrapperManager (or 
>wrapper_manager_t) that represents the code creator for the wrapper 
>function, i.e. that created the complete C++ source code. I have changed 
>this and made this class only a helper class for creating C++ functions. 
>Now it basically provides text substitution services which could be used 
>by the existing Py++ code creators. Those code creators still create the 
>rough layout of the code and insert special variables at appropriate 
>places that will get substituted by the wrapper manager.  These 
>variables represent individual parts of the C++ function. I have 
>documented the class in the wiki over here: 
>https://realityforge.vrsource.org/view/PyppApi/Wrapper_manager
>
>This wrapper manager object is also the object that the individual 
>methods of the arg policies (sometimes I also call them "code blocks" in 
>the wiki) receive as input ("wm") and which they have to use to modify 
>either the wrapper function or the virtual function.
>
>I have already implemented most parts of the mentioned classes (wrapper 
>manager, code manager, policy base class). Now I could add it to 
>pypp_api but it could only make use of the "wrapper function" part and 
>not the virtual functions. So if everyone (in particular Roman) agrees 
>with this approach, I would rather try to update the Py++ code creators 
>so that they use the text substitution service from above. But that's 
>where I'm a bit lost because I'm not familiar with the implementation 
>and all the dependencies involved.
>
>So before I continue with this I'd rather like to know if Roman agrees 
>to the general approach. The things that I'd like to do next are:
>
>- Add the "wrapper manager" class to Py++. It's more or less a 
>stand-alone module that I would stick into its own directory inside 
>"pyplusplus".
>
>- Add a new decoration attribute to the decl_wrappers that stores the 
>"arg policy" objects.
>
>- Update the Py++ calldef code creators so that they use the arg 
>policies via the above text substitution mechanism.
>
>
>Additionally, I'd like to introduce the "thread_safe" decoration 
>attribute. This involves the same code as above so I think it safes time 
>if it is done at the same time. This option should be implemented right 
>in the Py++ code creators and not via the "arg policy" mechanism because 
>it affects code on a lower level than what the arg policies are doing.
>
>
>- Matthias -
>
>
>-------------------------------------------------------------------------
>Using Tomcat but need to do more? Need to support web services, security?
>Get stuff done quickly with pre-integrated technology to make your job easier
>Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>_______________________________________________
>pygccxml-development mailing list
>pyg...@li...
>https://lists.sourceforge.net/lists/listinfo/pygccxml-development
>
>  
>

Re: [pygccxml-development] special functions/operators

[Roman Y. <rom...@gm...>]

On 8/29/06, Matthias Baas <ba...@ir...> wrote:
> Neal Becker wrote:
> >>>> cls = mb.class_("X")
> >>>> f = mb.free_function( "addAssign" )
> >>>> cls.add_method( f )
> >
> > I don't understand all of what you guys said, but it sounds like this is what
> > I had in mind.
> >
> > f = mb.free_function ("addAssign")
> >
> > what is "free_function"?  Is this new or already existing?

free_function returns a reference to free function declaration. It exists.

> It's the syntax of the Py++ high-level API, the pypp_api equivalent is
> the Function() method.
>
> So back to your initial example. Using the add_method approach you could
> write the following:
>
> f = root.Function("addAssign")
> f.rename("__iadd__")
> f.setPolicy(return_self())
> root.Class("X").addMethod(f)
>
> Currently, you would have to write the following instead:
>
> root.Class("X").cdef("__iadd__", "addAssign", return_self())
>
>
> Well, if *I* had to choose in this example, I would still use the latter
> option because of its brevity. Your goal has been to be less "low level"
> and get to a "better" API. But frankly, I don't see how the first option
> should have achieved these goals.

In this specific case you are right.

> But don't get me wrong, I don't say that add_method() is always a bad
> thing and cdef() is always better. There's merit in having something
> like add_method(), but probably not in this particular case (which was
> used as an example that a new API should improve).

Matthias, I think I still need this function, because it will improve few my
scripts and I like the idea. Do you want to implement it? It could be a nice
introduction to code creators factory.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Comments on module_builder cache functionality

[Roman Y. <rom...@gm...>]

On 8/29/06, Allen Bierbaum <al...@vr...> wrote:
> So now I have to figure out how to get this to work with module builder
> cache and there is a problem.  Namely, in module builder the call to
> parse_declarations does not have access to the list of files that gccxml
> had to include.  The only place where these files seem to be available
> is in source_reader.py on line 237.  It does not appear that this
> information remains in the pygccxml tree but I could be wrong.  Also,
> the files list is only used to update the decl cache so it is not
> available later during execution.
>
> Does anybody know a way to either a) get the computed file list from
> source_reader back into the module builder

You don't need this

>  or b) get a list of files
> that were included from a pygccxml decl tree?

pygccxml.declarations.declarations_files
But you also don't need this functionality

My idea was next:

class mbcache_t( pygccxml.parser.base_cache_t ):
    def __init__( self, *args ):
         pygccxml.parser.base_cache_t.__init__( self )
         test whether there is a need to load from cache on disk
         self.decls_cache = pygccxml.parser.file_cache_t( file_name )

   def update( self, .... ):
         self.decls_cache.update( self,  .... )

   def flush( self, ... )
        self.decls_cache.flush()
        write to disk files passed to module_builder_t

Thus, mbcache_t only needs to warry about files list that was passed to
module_builder_t

I think this is pretty simple idea, that will take few hours to implement.

What do you think?

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] special functions/operators

[Matthias B. <ba...@ir...>]

Neal Becker wrote:
>>>> cls = mb.class_("X")
>>>> f = mb.free_function( "addAssign" )
>>>> cls.add_method( f )
> 
> I don't understand all of what you guys said, but it sounds like this is what 
> I had in mind.
> 
> f = mb.free_function ("addAssign")
> 
> what is "free_function"?  Is this new or already existing?

It's the syntax of the Py++ high-level API, the pypp_api equivalent is 
the Function() method.

So back to your initial example. Using the add_method approach you could 
write the following:

f = root.Function("addAssign")
f.rename("__iadd__")
f.setPolicy(return_self())
root.Class("X").addMethod(f)

Currently, you would have to write the following instead:

root.Class("X").cdef("__iadd__", "addAssign", return_self())


Well, if *I* had to choose in this example, I would still use the latter 
option because of its brevity. Your goal has been to be less "low level" 
and get to a "better" API. But frankly, I don't see how the first option 
should have achieved these goals.

But don't get me wrong, I don't say that add_method() is always a bad 
thing and cdef() is always better. There's merit in having something 
like add_method(), but probably not in this particular case (which was 
used as an example that a new API should improve).

- Matthias -

Re: [pygccxml-development] Comments on module_builder cache functionality

[Allen B. <al...@vr...>]

Roman Yakovenko wrote:

> On 8/29/06, Allen Bierbaum <al...@vr...> wrote:
>
>> >Actually, if you change any text in implementation_details.h it will
>> >rebuild the cache because the md5 hash will catch it.
>
>
> Your code does not contains the check.

It doesn't contain a check for implicit includes, no.  I have 
temporarily added the ability for the user to pass in a list of files 
they know should be used for the signature.  This is not a great 
solution, but it will help out until a better solution is found (see below).

>> >
>> >As I said on IM though there are changes that both of them miss.
>> >
>> >The thing that both cache implementations miss is that if a file
>> >included by to_be_exported.h changes, then the caches will be invalid
>> >but they will not know it.
>
>
> This is an open source project. I claim that pygccxml.parser.file_cache_t
> class will not miss.
>
>> >I have seen this problem before in the pygccxml cache and there is
>> >really no good way around it.  The only way that pygccxml has to get a
>> >full list of dependent header files (ie. build a recursive list of
>> >includes)
>
>
> GCCXML does it.
>
>> >What both of these bugs mean is that it is possible to make a change in
>> >your code and have it not get picked up.
>
>
> pygccxml does not contain this bug.

Okay.  Roman is right.  Sorry for doubting you.  I just wish I could 
have found the code quicker.  I tracked down where this test is coming 
from.  For those of you that are interested, this check is on line 218 
of declarations_cache.py.  The test doesn't test the signature by 
computing a signature against the files that should be included during 
the current run, instead it does something that works equally as well.  
It checks to see if any of the files included when the cache was built 
have changed.  If they have then it says the signature is invalid.

>> >One hybrid solution that may work is that we could add an optional
>> >parameter to the module builder that is a list of "dependent" headers
>> >that the user cares about checking.
>
>
> In pygccxml this workds out of the box.
>
So now I have to figure out how to get this to work with module builder 
cache and there is a problem.  Namely, in module builder the call to 
parse_declarations does not have access to the list of files that gccxml 
had to include.  The only place where these files seem to be available 
is in source_reader.py on line 237.  It does not appear that this 
information remains in the pygccxml tree but I could be wrong.  Also, 
the files list is only used to update the decl cache so it is not 
available later during execution.

Does anybody know a way to either a) get the computed file list from 
source_reader back into the module builder or b) get a list of files 
that were included from a pygccxml decl tree?

Note: Modifying declarations_cache.file_cache_t would not eliminate this 
need.  No matter how the cache is held I need a way to get this list of 
files in module builder so I can use it to update the cache.

-Allen

Re: [pygccxml-development] special functions/operators

[Neal B. <ndb...@gm...>]

On Monday 28 August 2006 9:22 am, Roman Yakovenko wrote:
> On 8/28/06, Matthias Baas <ba...@ir...> wrote:
> > Roman Yakovenko wrote:
> > >> We can indeed use cdef for this, but it's rather low level.  I wonder
> > >> if we
> > >> can make a better api for this, maybe something like:
> > >>
> > >> add_method (...)
> > >
> > > You and Matthias are solving different problems.
> > > Matthias generates new function from the script and than adds it to the
> > > class.
> >
> > Well, not only. In my Maya bindings I'm also doing the same thing as
> > Neal, i.e. I have manually written C++ functions that I add as methods
> > to some classes using cdef().
>
> It does not different from what I say. You did not feed the
> declarations to Py++.
>
> > > You already have function declaration, and the only thing you want to
> > > do is to associate it
> > > with the class. I think it will take only few hours to implement what
> > > you are asking for.
> > >
> > > cls = mb.class_("X")
> > > f = mb.free_function( "addAssign" )
> > > cls.add_method( f )
> >

I don't understand all of what you guys said, but it sounds like this is what 
I had in mind.

f = mb.free_function ("addAssign")

what is "free_function"?  Is this new or already existing?

cls.add_method (f) sounds right.  I guess need to be able to set some 
properties on it as well.  Usual stuff.

Re: [pygccxml-development] Status of code inserters

[Matthias B. <ba...@ir...>]

Allen Bierbaum wrote:
> Matthias:
> 
> What is the status of the code inserter's?  Do you have any py++ addons 
> that I can try out yet?

I was just about to post a message on this subject....

Well, I don't have a readily available addon yet, but I think the 
general way to proceed is more or less clear. Right now, it's like I 
have all the individual pieces spread out before me but I'm not sure how 
to assemble them...

The result from my earlier wiki entry was that there are up to two 
functions involved:

1) the wrapper function that gets called from Python and that makes a 
call "down to" C++ where the actual work is done

2) the virtual function that gets called from C++ and that may make a 
call to Python

So for the general case, an arg policy must be prepared to deal with 
both functions and there must be an API that allows it to do so. This 
led to some updates which I have already entered into the wiki here: 
https://realityforge.vrsource.org/view/PyppApi/CodeInserter#Arg_policy_interface
Basically, there are now two sets of methods to create pre-call and 
post-call code, one for the wrapper function and one for the virtual 
function.

In the previous version I have proposed a class WrapperManager (or 
wrapper_manager_t) that represents the code creator for the wrapper 
function, i.e. that created the complete C++ source code. I have changed 
this and made this class only a helper class for creating C++ functions. 
Now it basically provides text substitution services which could be used 
by the existing Py++ code creators. Those code creators still create the 
rough layout of the code and insert special variables at appropriate 
places that will get substituted by the wrapper manager.  These 
variables represent individual parts of the C++ function. I have 
documented the class in the wiki over here: 
https://realityforge.vrsource.org/view/PyppApi/Wrapper_manager

This wrapper manager object is also the object that the individual 
methods of the arg policies (sometimes I also call them "code blocks" in 
the wiki) receive as input ("wm") and which they have to use to modify 
either the wrapper function or the virtual function.

I have already implemented most parts of the mentioned classes (wrapper 
manager, code manager, policy base class). Now I could add it to 
pypp_api but it could only make use of the "wrapper function" part and 
not the virtual functions. So if everyone (in particular Roman) agrees 
with this approach, I would rather try to update the Py++ code creators 
so that they use the text substitution service from above. But that's 
where I'm a bit lost because I'm not familiar with the implementation 
and all the dependencies involved.

So before I continue with this I'd rather like to know if Roman agrees 
to the general approach. The things that I'd like to do next are:

- Add the "wrapper manager" class to Py++. It's more or less a 
stand-alone module that I would stick into its own directory inside 
"pyplusplus".

- Add a new decoration attribute to the decl_wrappers that stores the 
"arg policy" objects.

- Update the Py++ calldef code creators so that they use the arg 
policies via the above text substitution mechanism.


Additionally, I'd like to introduce the "thread_safe" decoration 
attribute. This involves the same code as above so I think it safes time 
if it is done at the same time. This option should be implemented right 
in the Py++ code creators and not via the "arg policy" mechanism because 
it affects code on a lower level than what the arg policies are doing.


- Matthias -

Re: [pygccxml-development] Comments on module_builder cache functionality

[Roman Y. <rom...@gm...>]

On 8/29/06, Allen Bierbaum <al...@vr...> wrote:
> >Actually, if you change any text in implementation_details.h it will
> >rebuild the cache because the md5 hash will catch it.

Your code does not contains the check.

> >
> >As I said on IM though there are changes that both of them miss.
> >
> >The thing that both cache implementations miss is that if a file
> >included by to_be_exported.h changes, then the caches will be invalid
> >but they will not know it.

This is an open source project. I claim that pygccxml.parser.file_cache_t
class will not miss.

> >I have seen this problem before in the pygccxml cache and there is
> >really no good way around it.  The only way that pygccxml has to get a
> >full list of dependent header files (ie. build a recursive list of
> >includes)

GCCXML does it.

> >What both of these bugs mean is that it is possible to make a change in
> >your code and have it not get picked up.

pygccxml does not contain this bug.

> >One hybrid solution that may work is that we could add an optional
> >parameter to the module builder that is a list of "dependent" headers
> >that the user cares about checking.

In pygccxml this workds out of the box.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Comments on module_builder cache functionality

[Allen B. <al...@vr...>]

Allen Bierbaum wrote:

>Roman Yakovenko wrote:
>
>  
>
>>On 8/29/06, *Allen Bierbaum* <al...@vr... 
>><mailto:al...@vr...>> wrote:
>>
>>    As requested, I have backed out these changes.  The implementation is
>>    now available for people to use in goodies.goodies_perf_overrides.py.
>>    Just import that file and you will get the full module caching as well
>>    as the create_identifier override.
>>
>>
>>I don't like the way you do it. There was good reason to remove it - 
>>critical bug.
>>I did not see the code you added to goodies, but I assume it contains 
>>same bug.
>>
>>Bug description:
>>
>>Lets say you have 2 header files:
>>implementation_details.h
>>    ...
>>
>>and
>>
>>to_be_exported.h:
>>
>>   #include "implementation_details.h"
>>   ....
>>
>>Py++ code:
>>
>>mb = module_builder_t( "to_be_exported.h" )
>>
>>The problem with Allen implementation is that you can change 
>>"implementation_details.h"
>>file, but "cache" will remain valid. In this case module_builder_t 
>>should rebuild the cache, otherwise
>>Py++ will generate wrong code. This is a very critical bug. pygccxml 
>>cache classes know
>>to deal with this situation.
>>    
>>
>
>Actually, if you change any text in implementation_details.h it will 
>rebuild the cache because the md5 hash will catch it.  This is the same 
>with the entire list of header files.  I have been using it and I can 
>say without a doubt this is how it works and it works exactly the same 
>as the pygccxml decl cache.
>
>As I said on IM though there are changes that both of them miss.
>
>The thing that both cache implementations miss is that if a file 
>included by to_be_exported.h changes, then the caches will be invalid 
>but they will not know it.  The module cache misses this and the 
>existing pygccxml cache misses it.  (if you doubt this, take a look at 
>the code.  Although pygccxml passing a set of included files into the 
>cache update method those files are not used for the key signature.  
>Thus they are not used to validate the cache entry on load).
>
>I have seen this problem before in the pygccxml cache and there is 
>really no good way around it.  The only way that pygccxml has to get a 
>full list of dependent header files (ie. build a recursive list of 
>includes) is to either a) run pygccxml on the file or b) add a scanner 
>that does this internally.  Option a is out because it would make the 
>cache worthless since we would be doing the exact work we want to 
>cache.  Option b doesn't exist in the code base and in my case I would 
>want the use to be optional because it would take significant time to 
>run. I haven't tested for sure but I would guess that in my case it 
>would have to find and scan well over 10,000 files if not more.
>
>What both of these bugs mean is that it is possible to make a change in 
>your code and have it not get picked up.  That is definitely true and I 
>don't see a great way around it.
>
>One hybrid solution that may work is that we could add an optional 
>parameter to the module builder that is a list of "dependent" headers 
>that the user cares about checking.  This list could be scanned and made 
>part of the signature.  Still not perfect, but it would allow the user 
>to tailor their caching needs.
>
>  
>
I just committed a change to the module builder override in goodies that 
implements this feature.  I have started using it for my projects and it 
seems to work well so far.

-Allen

Re: [pygccxml-development] Comments on module_builder cache functionality

[Allen B. <al...@vr...>]

Roman Yakovenko wrote:

> On 8/29/06, *Allen Bierbaum* <al...@vr... 
> <mailto:al...@vr...>> wrote:
>
>     As requested, I have backed out these changes.  The implementation is
>     now available for people to use in goodies.goodies_perf_overrides.py.
>     Just import that file and you will get the full module caching as well
>     as the create_identifier override.
>
>
> I don't like the way you do it. There was good reason to remove it - 
> critical bug.
> I did not see the code you added to goodies, but I assume it contains 
> same bug.
>
> Bug description:
>
> Lets say you have 2 header files:
> implementation_details.h
>     ...
>
> and
>
> to_be_exported.h:
>
>    #include "implementation_details.h"
>    ....
>
> Py++ code:
>
> mb = module_builder_t( "to_be_exported.h" )
>
> The problem with Allen implementation is that you can change 
> "implementation_details.h"
> file, but "cache" will remain valid. In this case module_builder_t 
> should rebuild the cache, otherwise
> Py++ will generate wrong code. This is a very critical bug. pygccxml 
> cache classes know
> to deal with this situation.

Actually, if you change any text in implementation_details.h it will 
rebuild the cache because the md5 hash will catch it.  This is the same 
with the entire list of header files.  I have been using it and I can 
say without a doubt this is how it works and it works exactly the same 
as the pygccxml decl cache.

As I said on IM though there are changes that both of them miss.

The thing that both cache implementations miss is that if a file 
included by to_be_exported.h changes, then the caches will be invalid 
but they will not know it.  The module cache misses this and the 
existing pygccxml cache misses it.  (if you doubt this, take a look at 
the code.  Although pygccxml passing a set of included files into the 
cache update method those files are not used for the key signature.  
Thus they are not used to validate the cache entry on load).

I have seen this problem before in the pygccxml cache and there is 
really no good way around it.  The only way that pygccxml has to get a 
full list of dependent header files (ie. build a recursive list of 
includes) is to either a) run pygccxml on the file or b) add a scanner 
that does this internally.  Option a is out because it would make the 
cache worthless since we would be doing the exact work we want to 
cache.  Option b doesn't exist in the code base and in my case I would 
want the use to be optional because it would take significant time to 
run. I haven't tested for sure but I would guess that in my case it 
would have to find and scan well over 10,000 files if not more.

What both of these bugs mean is that it is possible to make a change in 
your code and have it not get picked up.  That is definitely true and I 
don't see a great way around it.

One hybrid solution that may work is that we could add an optional 
parameter to the module builder that is a list of "dependent" headers 
that the user cares about checking.  This list could be scanned and made 
part of the signature.  Still not perfect, but it would allow the user 
to tailor their caching needs.

>
> There was another bug: module_builder_t.__init__ can take not only 
> file paths but also
> text, that contains valid C++ code. The Allen's code does not deal 
> with them at all.

I will have to look at this part.

Does this code come in through the files parameter?  As far as I can 
tell the handling of this parameter is still the same as it was before.  
If there is a type of "file" that I am missing then I may have to update 
the signature calcuation but that should not be too hard.

>
> I proposed to Allen to create and implement module_builder_t cache class
> in terms of pygccxml classes.

I don't have time to rewrite the existing cache code to make it so it 
could be reusable for the case I need.  I already am using all the parts 
that are reusable (the file and config signature methods).  But the 
cache can not be used or extended as is for several reasons:  1) The 
current signature keys are based on 1 file.  I have multiple files that 
make up the key.  2) The current pygccxml cache is based on the idea of 
loading a map from key to data.  These cache entries are marked as dirty 
or not before being flushed.    I don't need anything like this.  I just 
need a cache that has a signature to look at as the first entry in the 
pickled data.  And then the next entry needs to be the tree.  The 
pygccxml cache can't really work this way because it has multiple 
entries.  So the two needs are enough different that I don't see the 
commonality for reuse beyond what I have done.

-Allen

>
> I am pretty busy this days, so I don't have time to fix the bug. Thus 
> the code has been removed.
>
>
> -- 
> Roman Yakovenko
> C++ Python language binding
> http://www.language-binding.net/ 

[pygccxml-development] Status of code inserters

[Allen B. <al...@vr...>]

Matthias:

What is the status of the code inserter's?  Do you have any py++ addons 
that I can try out yet?

-Allen

Re: [pygccxml-development] special functions/operators

[Roman Y. <rom...@gm...>]

On 8/29/06, Matthias Baas <ba...@ir...> wrote:
> Roman Yakovenko wrote:
> >> But in the above example f is already a "real" declaration that will get
> >> turned into a "real" code creator, so all the above features could be
> >> used, couldn't they?
> >
> > I just wanted to save time :-). I can implement the feature, but it
> > will take some time.
> > Do you want to try to implement it?
>
> Well, currently I have no need for this feature for the Maya bindings
> (as I said, Py++ doesn't see my manually created functions so I would
> have to stick to cdef anyway (which doesn't impose any disadvantage for
> me yet)).
> Apart from that, I'd first love to hear Neal's opinion on what the API
> should look like anyway.

Neal did we understand your use case right?

> (And I'm still busy with the arg policies...)

I hope in a few days we will join the efforts.


-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] [pygccxml-commit] SF.net SVN: pygccxml: [451] pygccxml_dev/pygccxml/declarations/mdecl_wrapper.py

[Roman Y. <rom...@gm...>]

On 8/29/06, Matthias Baas <ba...@ir...> wrote:
> Roman Yakovenko wrote:
> >> ERROR: test (operators_tester.tester_t)
> >
> > Can you send the error?
>
> gccxml (0.6.0) produces errors on a Boost header which look like this:
> ...
> So it doesn't look like this is an error in Py++.

It was very convenient to reuse rational library to test operators
support. May be I should
change it. Anyway, if you want you can disable this test for your environment.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] special functions/operators

[Matthias B. <ba...@ir...>]

Roman Yakovenko wrote:
>> But in the above example f is already a "real" declaration that will get
>> turned into a "real" code creator, so all the above features could be
>> used, couldn't they?
> 
> I just wanted to save time :-). I can implement the feature, but it
> will take some time.
> Do you want to try to implement it?

Well, currently I have no need for this feature for the Maya bindings 
(as I said, Py++ doesn't see my manually created functions so I would 
have to stick to cdef anyway (which doesn't impose any disadvantage for 
me yet)).
Apart from that, I'd first love to hear Neal's opinion on what the API 
should look like anyway.
(And I'm still busy with the arg policies...)

- Matthias -