pygccxml-development

Re: [pygccxml-development] pyplusplus status

[Roman Y. <rom...@gm...>]

On 7/19/06, Allen Bierbaum <al...@vr...> wrote:
> What do you mean by "cache management" will be done in user code.  Do
> you just mean the user code will tell pyplusplus what things to cache

No

> and where the cache is

Yes

> and then it will be up to pyplusplus to read and
> update the cache?

Yes and no. Only user knows whether code chaned in a such way, that he should
drop configuration from cache and to create new configuration.

That what I mean under "cache management". Of course I could be wrong.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] pyplusplus status

[Allen B. <al...@vr...>]

Roman Yakovenko wrote:

> On 7/18/06, Allen Bierbaum <al...@vr...> wrote:
>
>> I think the keywords here are "most tasks".   I think we are already
>> close to being able to do most tasks with the higher-level API. I just
>> don't think the high-level API allows people to do everything as easily
>> as they could with Pyste or the experimental API that Matthias is using.
>
>
> I will study Matthias API and write here my thoughts.

Don't forget to review the Pyste API.  I hate to say it, but it is a 
reality of software that anything that is easy to do in the original 
tool (Pyste) but hard to do in the new tool (pyplusplus) will hurt 
adoption of the the new software package.  Pyplusplus not only has to be 
able to do more it has to be able to still do everything Pyste does and 
do it just as easily.

>
>> I would also say the simple API would require limited to no use of
>> pygccxml directly.  This may be one of the areas that gets me the most
>> because I don't always know where to look in pygccxml to do what I want.
>
>
> I thought that pygccxml is pretty simple project. In 99% of use case you
> have to understand only declarations package. I will try to improve
> documentation
> for the package. Please say what information you are missing?

You are probably right that the big thing I am missing is complete 
documentation.  For example a table with all the type traits, how to 
call them, and what they return. Similarly with the api for 
lookup/search, documentation of all the methods, all the params, and all 
the matchers.  (once again, just a table that lists everything in one 
place with a short description would be helpful).  I know from 
experience what source files to look in, but new users don't know this.  
This could be helped a lot by just having a complete tutorial with pages 
that have these tables.

>
>> If the query api was more completely documented in a tutorial form (so
>> newbies don't have to delve deeply into the reference docs), then I
>> think we would be on much better footing here.
>
>
> What is wrong with this
> http://language-binding.net/pygccxml/query_interface.html
> tutorial. If you understand how to use "member_function(s)" , you
> understand all others.
>
Yes, but you have to know what all the "others" are and that they take 
the same parameters.  Once again, I am not saying we are too far from 
this, but there is still a disconnect for a new user.

I keep using the term "tutorial", but maybe the more appropriate term 
would be "user's guide".  What I think is needed is something similar to 
the Pyste documenation and the Boost.Python Tutorial.  Something that 
shows the user-level API that can be used to accomplish most tasks.  It 
should have reference to where to go for lower-level detail, but it 
should stand on it's own as a useful resource.

>> >
>> > power - modifying directly code creators tree. This is really the
>> > "dark" area of the project,
>> > but I have an idea how to simplify it and of course I will write a
>> > documentation that will
>> > explain it.
>>
>> I would also add using anything in pygccxml that requires understanding
>> of how pygccxml is implemented and using any of pygccxml that is not
>> documented in a tutorial.  Basically, if the user has to read the code
>> to understand or find what they need to do, then that is not part of the
>> simple interface.
>
>
> I don't agree with you. I think, the project does not have enough 
> documentation.
> If I will describe every public function in terms of: precondition,
> post condition,
> arguments and return value, then you will almost never will have to
> look to source code.
>
But even then you are talking about reference documentation.  There 
needs to be some starting point that lets the user know the capability 
exists.  Then this can be used as a launching point to looking for the 
detailed documentation. 

I could be totally wrong here though.  I would love for you to prove me 
wrong.  I don't care how we get there or who's ideas work, just as long 
as new users can pick up, understand, and use pyplusplus easily.

-Allen

Re: [pygccxml-development] pyplusplus status

[Allen B. <al...@vr...>]

[snip]

>>>Would it be useful to make the tutorial and howto a "live" document so
>>>everyone can contribute, add questions, make changes?  If so, what about
>>>using a wiki so everyone can contribute and fix up the descrptions.
>>>      
>>>
>>Sounds like a good idea to me!
>>    
>>
>
>I am not against this idea. I think that before we going to it we can improve
>existing documentation, faqs and how to. I prefer to have only 1
>place, where user
>have to look for documentation.
>  
>
Agreed, but what if the one place for the tutorial and faq was a wiki or 
something similar where users and developers could contribute and make 
comments?

[snip]

>  
>
>>>>>- Cache parsing and preparing query optimizer.  If the input has not
>>>>>changed (the headers are the same) why do I have to wait for all the
>>>>>processing of the declarations and query optimization each time.  Can't
>>>>>we cache all of that?
>>>>>          
>>>>>
>>>>Yes, but you can guess what the priority of this. Also I think that
>>>>solution for this
>>>>problem will be completely in the user code.
>>>>        
>>>>
>>Wouldn't that rather be located in the high level API where the queries
>>are made? (if it's in the user code it would have to be duplicated with
>>every new project)
>>    
>>
>
>I think you missed the point. py++ can do( and may be some day will do ) the
>dirty job of updating decl wrapper classes within a cache, but the cache
>managment will be done completly from user code.
>
>  
>
What do you mean by "cache management" will be done in user code.  Do 
you just mean the user code will tell pyplusplus what things to cache 
and where the cache is and then it will be up to pyplusplus to read and 
update the cache?

-Allen

Re: [pygccxml-development] pyplusplus status

[Roman Y. <rom...@gm...>]

On 7/18/06, Matthias Baas <ba...@ir...> wrote:
> No, I also think that most of the time it should be enough to stick to
> the high-level API.
> I would say this is more or less the case with my Maya bindings, but
> then, I'm still using the "experimental" API instead of Roman's version.
> I've just committed the current version of the API into the "contrib"
> directory (subdirectory "pypp_api").

Would you mind to clean experimental directory?

> My main script to generate the bindings can be found here (pypp_setup.py):
> http://svn.sourceforge.net/viewcvs.cgi/cgkit/maya/trunk/maya_wrapper/

It will take some time to me to learn pypp_api and how you use it in
your script.


I will try to take a look on cache and ALL_AT_ONCE issue in next few days.

> > Would it be useful to make the tutorial and howto a "live" document so
> > everyone can contribute, add questions, make changes?  If so, what about
> > using a wiki so everyone can contribute and fix up the descrptions.
>
> Sounds like a good idea to me!

I am not against this idea. I think that before we going to it we can improve
existing documentation, faqs and how to. I prefer to have only 1
place, where user
have to look for documentation.

> I think you two are talking about somewhat different things here.
> If I understand Allen correctly he wants to replace a method with his
> own code instead of wrapping the original method. The original method
> might even never get called in the wrapper code but the wrapper code
> will still provide the same functionality. I also did that for methods
> that deal with stuff like function pointers that need some manual
> intervention anyway (even though in most cases I did actually call the
> original method).

"User defined transformation" does not have to call to original function.
If you talk about function "replace" you are talking transformation, otherwise
it is much easier to exclude function and to add reference to new function under
same name.

> The "call wrapper policies" (in my version they are still called arg
> policies) are rather meant to attach additional code to a wrapper method
> that surrounds the original method invocation. This surrounding code is
> mainly used to transform input arguments or return values between Python
> and C++ (which cannot be done by Boost.Python automatically in those
> cases). One example is the often mentioned "void getSize(int& width,
> int& height)" method which cannot be wrapped directly because of the
> references.

Think about method transformation as a generalization of your idea.

> >>> - Add ability to add user code to namespace or global *without* having
> >>> to use code creators directly
> >>
> >> I want to solve this problem, to this release. Can you write what you
> >> want/need/expect
> >> here? In other words I need use cases and how you see they should be
> >> solved.
> >
> > I will have to think about this and get back to you.
>
> I take it your basically talking about the addStartText(),
> addBodyTailText() methods that you implemented in the experimental
> version, right? They are still available in pypp_api and I'm also using
> them myself.

Why you work so hard? You can start the editor you use every day. To write
"start text" in it. ( The better name is - user defined  declarations. )
Then using the technique described
http://svn.sourceforge.net/viewcvs.cgi/pygccxml/pyplusplus_dev/docs/faqs/faqs.rest?view=markup
section "How to add custom exception translation?" you can add it to module
body.

> >>> - Cache parsing and preparing query optimizer.  If the input has not
> >>> changed (the headers are the same) why do I have to wait for all the
> >>> processing of the declarations and query optimization each time.  Can't
> >>> we cache all of that?
> >>
> >> Yes, but you can guess what the priority of this. Also I think that
> >> solution for this
> >> problem will be completely in the user code.
>
> Wouldn't that rather be located in the high level API where the queries
> are made? (if it's in the user code it would have to be duplicated with
> every new project)

I think you missed the point. py++ can do( and may be some day will do ) the
dirty job of updating decl wrapper classes within a cache, but the cache
managment will be done completly from user code.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] pyplusplus status

[Allen B. <al...@vr...>]

Matthias Baas wrote:

>Allen Bierbaum wrote:
>  
>
>>>>- 90% of users should never have to delve beyond the "easy API" (never
>>>>need to know about creators). The current API is close to this, but
>>>>there are still some times when IMHO users have to suddenly delve very
>>>>deep into the internal code.
>>>>        
>>>>
>>>[...]
>>>      
>>>
>>The big ones I find missing are:
>>
>>    * Do not require the user to know anything outside of the high-level
>>      API to accomplish most tasks.
>>    * Provide access to all the power of pypluplus and pygccxml *if* the
>>      user wants it
>>
>>[...]
>>
>>Am I the only one that feels this way when using the API?
>>    
>>
>
>No, I also think that most of the time it should be enough to stick to 
>the high-level API.
>I would say this is more or less the case with my Maya bindings, but 
>then, I'm still using the "experimental" API instead of Roman's version.
>I've just committed the current version of the API into the "contrib" 
>directory (subdirectory "pypp_api").
>My main script to generate the bindings can be found here (pypp_setup.py):
>http://svn.sourceforge.net/viewcvs.cgi/cgkit/maya/trunk/maya_wrapper/
>  
>
I did not know you were still keeping these up-to-date.  I have to admit 
it has been a long time since I looked at this interface but it does 
seem a little more high-level or at least more self contained.

Q: If I remember correctly, when I first wrote these I had to modify the 
code creators to change they code they output.  Is this still necessary 
or is everything captured in the api itself?  Maybe the better way to 
say this is how do you handle decorating and creating for features that 
are not supported by Roman's API?

>  
>
>>>>- template support: I would like to see an API that allows users to more
>>>>easily wrap templates.  I have had to manually create some code that
>>>>does this but I am sure something better could be done.  You can see
>>>>what I have done at:
>>>>https://realityforge.vrsource.org/viewvc/pyopensg/trunk/src/gen_bindings.py?view=markup 
>>>>
>>>>Look for the TemplateBuilder helper class.
>>>>        
>>>>
>>>My main objection to this is that users will not use it. Think about
>>>it. You can create/generate/generate on fly header file/string and to
>>>include it to the list of header files
>>>passed to module_builder_t.__init__. You don't have to learn  new API.
>>>      
>>>
>>And my main reason for saying this is needed is that doing it manually 
>>is too hard for most users.  It is *much* easier to do this with Pyste.  
>>You just say Template("class") and then instantiate templates.  Very 
>>straight forward in Pyste's domain specific lanaguage.  Unfortunately I 
>>think the domain specific language of pyplusplus is still python and 
>>this is not good IMHO.
>>    
>>
>
>I agree with Allen. If you can use a Template() function then you don't 
>have to know *how* the template class is actually wrapped. Whereas if 
>you have to create C++ source code yourself you have to know what code 
>to create, where to put that code, how the code is added to the sources, 
>etc.
>
>By the way, the pypp_api stuff still contains the Template() 
>functionality that Allen has originally written (though I haven't tested 
>it myself yet as the Maya SDK doesn't contain any templates).
>  
>
I took a quick look at it.  It looks like I implemented it quite a bit 
like the Template helper I am using now, but it was more integrated.  
One of the big reasons is that in the experimental API, the construction 
of the builder is separated from the parsing.  So you could create a 
builder, call the Template() method on it and then when parsing times 
comes around the builder could take care of all the autocreation needed 
behind the scenes for the templates.  It wasn't perfect, but it did make 
it pretty transparent to the users.

>  
>
>>>mb = module_builder_t( ...
>>>    , compilation_mode=pygccxml.parser.COMPILATION_MODE.ALL_AT_ONCE
>>>    , ... )
>>>      
>>>
>>Does this behave *exactly* the same way.  Does it only create one entry 
>>in the cache?
>>    
>>
>
>I don't think so. We were talking about that already in a previous 
>thread quite a while ago. If I remember correctly the ALL_AT_ONCE mode 
>doesn't use the cache at all, that's why I also create my own header 
>file for the Maya bindings and do not use ALL_AT_ONCE.
>
>  
>
>>Would it be useful to make the tutorial and howto a "live" document so 
>>everyone can contribute, add questions, make changes?  If so, what about 
>>using a wiki so everyone can contribute and fix up the descrptions.
>>    
>>
>
>Sounds like a good idea to me!
>  
>
Anyone else agree or disagree?

>  
>
>>>>- Wrap method: Add API for just wrapping an existing method with a new
>>>>defined method.  (right now this can be cone but you have to exclude the
>>>>current method and then call add_code with a 'def'. It is just too
>>>>complex for a new user and makes the wrapper gen script look 
>>>>complicated.)
>>>>        
>>>>
>>>http://www.language-binding.net/pyplusplus/peps/call_wrapper_policies.html 
>>>
>>>
>>>When I implement this PEP, it will be pretty easy to do what you want.
>>>Until then I don't want to introduce something, that I will not 
>>>support later.
>>>      
>>>
>>I don't see how this pep fixes the problem, but then again I don't fully 
>>understand what the pep will do.  I will just trust you that this will 
>>help out.
>>    
>>
>
>I think you two are talking about somewhat different things here.
>If I understand Allen correctly he wants to replace a method with his 
>own code instead of wrapping the original method. The original method 
>might even never get called in the wrapper code but the wrapper code 
>will still provide the same functionality. I also did that for methods 
>that deal with stuff like function pointers that need some manual 
>intervention anyway (even though in most cases I did actually call the 
>original method).
>  
>
Exactly.

>The "call wrapper policies" (in my version they are still called arg 
>policies) are rather meant to attach additional code to a wrapper method 
>that surrounds the original method invocation. This surrounding code is 
>mainly used to transform input arguments or return values between Python 
>and C++ (which cannot be done by Boost.Python automatically in those 
>cases). One example is the often mentioned "void getSize(int& width, 
>int& height)" method which cannot be wrapped directly because of the 
>references.
>
>  
>
Yes, I need these too.

>>>>- Add ability to add user code to namespace or global *without* having
>>>>to use code creators directly
>>>>        
>>>>
>>>I want to solve this problem, to this release. Can you write what you
>>>want/need/expect
>>>here? In other words I need use cases and how you see they should be 
>>>solved.
>>>      
>>>
>>I will have to think about this and get back to you.
>>    
>>
>
>I take it your basically talking about the addStartText(), 
>addBodyTailText() methods that you implemented in the experimental 
>version, right? They are still available in pypp_api and I'm also using 
>them myself.
>  
>
Yes, that is what I am thinking about.  I am actually just trying to 
port some code that used the experimental pypp_api and I could not find 
a way to do this.

>  
>
>>>>- Cache parsing and preparing query optimizer.  If the input has not
>>>>changed (the headers are the same) why do I have to wait for all the
>>>>processing of the declarations and query optimization each time.  Can't
>>>>we cache all of that?
>>>>        
>>>>
>>>Yes, but you can guess what the priority of this. Also I think that
>>>solution for this
>>>problem will be completely in the user code.
>>>      
>>>
>
>Wouldn't that rather be located in the high level API where the queries 
>are made? (if it's in the user code it would have to be duplicated with 
>every new project)
>  
>
I agree.  This should be handled by the module builder so users don't 
need to know the implementation details.  As I think about it right now, 
we are really just caching some of the module builder data structures so 
this seems like exactly something that should be encapsulated in the 
module builder.

Experimental API, pypp_api...

I did not realize that there were still two active high-level APIs being 
developed.  As a user this concerns me a little because it means that 
there are probably good ideas in both that are not being cross-mixed to 
get the best API possible.  (some of the topics above point out those 
cases).

Also, after thinking about it more I believe that many of the limits and 
issues I am having were addressed at least in some form by the 
experimental API.  The way pypp_api was designed was to provide a very 
simple and easy to use API that could accomplish what most users would 
want with a minimal understanding of pyplusplus.  One way to make 
Roman's API more complete and simple would be to re-examine the pypp_api 
and make sure that Roman's API can do all the things that pypp_api can 
do with the same level of simplicitly.

Comments?

-Allen

Re: [pygccxml-development] pyplusplus status

[Allen B. <al...@vr...>]

Roman Yakovenko wrote:

> On 7/18/06, Allen Bierbaum <al...@vr...> wrote:
>
>> What this comment really comes down to is that I don't think the current
>> API fullfills the goals set out at:
>> https://realityforge.vrsource.org/view/PyOpenSG/PyppApiDiscussion
>>
>> The big ones I find missing are:
>>
>>     * Do not require the user to know anything outside of the high-level
>>       API to accomplish most tasks.
>
>
> This will never happen :-(. py++ knows to export only limited subset 
> of C++.

I think the keywords here are "most tasks".   I think we are already 
close to being able to do most tasks with the higher-level API. I just 
don't think the high-level API allows people to do everything as easily 
as they could with Pyste or the experimental API that Matthias is using.

>
>> Please don't take offense to this Roman, but you understand the APIs of
>> pygccxml and pyplusplus so well and in so much detail that I think your
>> take on what is an "easy API" may be much more in depth then what I
>> would consider an "easy API".
>> ...
>> But having the simple API is still important
>> for allowing new users to pick it up and use the tool.
>
>
> With the power comes complexity. Also I would like to see your
> definition to "simple"
> and "power" API. My definition is next:
>
> simple - API defined by decl_wrappers package and module_builder
> package. Basically,
> this is get\set properties + "query" api.

I would also say the simple API would require limited to no use of 
pygccxml directly.  This may be one of the areas that gets me the most 
because I don't always know where to look in pygccxml to do what I want.

If the query api was more completely documented in a tutorial form (so 
newbies don't have to delve deeply into the reference docs), then I 
think we would be on much better footing here.

>
> power - modifying directly code creators tree. This is really the
> "dark" area of the project,
> but I have an idea how to simplify it and of course I will write a
> documentation that will
> explain it.

I would also add using anything in pygccxml that requires understanding 
of how pygccxml is implemented and using any of pygccxml that is not 
documented in a tutorial.  Basically, if the user has to read the code 
to understand or find what they need to do, then that is not part of the 
simple interface.

>
>> Am I the only one that feels this way when using the API?
>
>
> I definitely would like to get the answer to this question with
> suggestions from other users.
>
>> You just say Template("class") and then instantiate templates.  Very
>> straight forward in Pyste's domain specific lanaguage.
>
>
> Please contribute your code with small example and documentation( and 
> license )
> to py++ :-). I already added new directory for this in "contrib" one.

I think my code as it stands is too complex.  I will try to revisit it 
once I get back to using it (probably late this week or early next week).

>
>> > mb = module_builder_t( ...
>> >     , compilation_mode=pygccxml.parser.COMPILATION_MODE.ALL_AT_ONCE
>> >     , ... )
>>
>> Does this behave *exactly* the same way.  Does it only create one entry
>> in the cache?
>
>
> Almost. pygccxml will create temporal file. I never checked how it
> will behave with a
> cache. Can you check this, and if the behaviour is not as you expect,
> I will try to fix it.
>
>> I would recommend just treating it like you would a .decls() call.  ie.
>> if the users looks up declarations of differing types and then calls a
>> method on those decls that only makes sense for one type, then python
>> will display an error about the type problems.
>
>
> I prefer to continue to discuss this problem within the original thread.

Ok.

>
>> Would it be useful to make the tutorial and howto a "live" document so
>> everyone can contribute, add questions, make changes?  If so, what about
>> using a wiki so everyone can contribute and fix up the descrptions.
>
>
> I think, this will be useful, but not now. There is only few peoples
> who contribute
> to the project.
>
> I started to work on FAQs:
> http://svn.sourceforge.net/viewcvs.cgi/pygccxml/pyplusplus_dev/docs/faqs/faqs.rest?view=markup 
>
>
> If you have something specifiec in mind, or want to add documentation
> here and there,
> please do it.
>
>> I don't see how this pep fixes the problem, but then again I don't fully
>> understand what the pep will do.  I will just trust you that this will
>> help out.
>
>
> This is pretty simple: this paper deals with method transformations.
> For example:
>     void get_size( int& h, int&w )
> You will be able to define the transformation of get_size to
>    boost::python::tuple get_size()
>
> The function wrapper, you are talking about, it is just an "user
> defined" transformation.

What I want is not only transformations, but defining completely new 
functions to replace existing functions.

>
>> def processClass(cls):
>>    if cls.count("method"):
>>       # do something
>>
>> I have run into a few cases where something like this could be pretty
>> useful and keep the script more understandable.
>
>
> I will see what I can do.
>
Thanks,
Allen

Re: [pygccxml-development] pyplusplus status

[Matthias B. <ba...@ir...>]

Allen Bierbaum wrote:
>>> - 90% of users should never have to delve beyond the "easy API" (never
>>> need to know about creators). The current API is close to this, but
>>> there are still some times when IMHO users have to suddenly delve very
>>> deep into the internal code.
>>
>> [...]
> 
> The big ones I find missing are:
> 
>     * Do not require the user to know anything outside of the high-level
>       API to accomplish most tasks.
>     * Provide access to all the power of pypluplus and pygccxml *if* the
>       user wants it
> 
> [...]
> 
> Am I the only one that feels this way when using the API?

No, I also think that most of the time it should be enough to stick to 
the high-level API.
I would say this is more or less the case with my Maya bindings, but 
then, I'm still using the "experimental" API instead of Roman's version.
I've just committed the current version of the API into the "contrib" 
directory (subdirectory "pypp_api").
My main script to generate the bindings can be found here (pypp_setup.py):
http://svn.sourceforge.net/viewcvs.cgi/cgkit/maya/trunk/maya_wrapper/

>>> - template support: I would like to see an API that allows users to more
>>> easily wrap templates.  I have had to manually create some code that
>>> does this but I am sure something better could be done.  You can see
>>> what I have done at:
>>> https://realityforge.vrsource.org/viewvc/pyopensg/trunk/src/gen_bindings.py?view=markup 
>>>
>>> Look for the TemplateBuilder helper class.
>>
>> My main objection to this is that users will not use it. Think about
>> it. You can create/generate/generate on fly header file/string and to
>> include it to the list of header files
>> passed to module_builder_t.__init__. You don't have to learn  new API.
> 
> And my main reason for saying this is needed is that doing it manually 
> is too hard for most users.  It is *much* easier to do this with Pyste.  
> You just say Template("class") and then instantiate templates.  Very 
> straight forward in Pyste's domain specific lanaguage.  Unfortunately I 
> think the domain specific language of pyplusplus is still python and 
> this is not good IMHO.

I agree with Allen. If you can use a Template() function then you don't 
have to know *how* the template class is actually wrapped. Whereas if 
you have to create C++ source code yourself you have to know what code 
to create, where to put that code, how the code is added to the sources, 
etc.

By the way, the pypp_api stuff still contains the Template() 
functionality that Allen has originally written (though I haven't tested 
it myself yet as the Maya SDK doesn't contain any templates).

>> mb = module_builder_t( ...
>>     , compilation_mode=pygccxml.parser.COMPILATION_MODE.ALL_AT_ONCE
>>     , ... )
> 
> Does this behave *exactly* the same way.  Does it only create one entry 
> in the cache?

I don't think so. We were talking about that already in a previous 
thread quite a while ago. If I remember correctly the ALL_AT_ONCE mode 
doesn't use the cache at all, that's why I also create my own header 
file for the Maya bindings and do not use ALL_AT_ONCE.

> Would it be useful to make the tutorial and howto a "live" document so 
> everyone can contribute, add questions, make changes?  If so, what about 
> using a wiki so everyone can contribute and fix up the descrptions.

Sounds like a good idea to me!

>>> - Wrap method: Add API for just wrapping an existing method with a new
>>> defined method.  (right now this can be cone but you have to exclude the
>>> current method and then call add_code with a 'def'. It is just too
>>> complex for a new user and makes the wrapper gen script look 
>>> complicated.)
>>
>> http://www.language-binding.net/pyplusplus/peps/call_wrapper_policies.html 
>>
>>
>> When I implement this PEP, it will be pretty easy to do what you want.
>> Until then I don't want to introduce something, that I will not 
>> support later.
> 
> I don't see how this pep fixes the problem, but then again I don't fully 
> understand what the pep will do.  I will just trust you that this will 
> help out.

I think you two are talking about somewhat different things here.
If I understand Allen correctly he wants to replace a method with his 
own code instead of wrapping the original method. The original method 
might even never get called in the wrapper code but the wrapper code 
will still provide the same functionality. I also did that for methods 
that deal with stuff like function pointers that need some manual 
intervention anyway (even though in most cases I did actually call the 
original method).

The "call wrapper policies" (in my version they are still called arg 
policies) are rather meant to attach additional code to a wrapper method 
that surrounds the original method invocation. This surrounding code is 
mainly used to transform input arguments or return values between Python 
and C++ (which cannot be done by Boost.Python automatically in those 
cases). One example is the often mentioned "void getSize(int& width, 
int& height)" method which cannot be wrapped directly because of the 
references.

>>> - Add ability to add user code to namespace or global *without* having
>>> to use code creators directly
>>
>> I want to solve this problem, to this release. Can you write what you
>> want/need/expect
>> here? In other words I need use cases and how you see they should be 
>> solved.
> 
> I will have to think about this and get back to you.

I take it your basically talking about the addStartText(), 
addBodyTailText() methods that you implemented in the experimental 
version, right? They are still available in pypp_api and I'm also using 
them myself.

>>> - Cache parsing and preparing query optimizer.  If the input has not
>>> changed (the headers are the same) why do I have to wait for all the
>>> processing of the declarations and query optimization each time.  Can't
>>> we cache all of that?
>>
>> Yes, but you can guess what the priority of this. Also I think that
>> solution for this
>> problem will be completely in the user code.

Wouldn't that rather be located in the high level API where the queries 
are made? (if it's in the user code it would have to be duplicated with 
every new project)

>>> - This is a small nitpick, but unless you are making a lot of money from
>>> Google AdSense, could you remove the Google ads in the table of contents
>>> on the website.  They are just annoying and make the documentation
>>> harder to read.
>>
>> I am not, the AdSence is there to return the money I pay for running 
>> web site.
>> I hate the idea to ask for donations. User experience is importer.
>> I will do something about it.
> 
> That would be very helpful. I do not care about the ads on the left bar 
> or even if you added ones at the other sides of the pages.  But putting 
> ads in the middle of the table of contents or text body is a little 
> annoying.

I also think the web page is a bit difficult to read because of the ads. 
  Roman, I don't know how much control you have about the layout of the 
site and the ads, but would it also be possible to separate them from 
the actual content somehow? Like for example, using a different 
background color, adding separating lines or something like this. 
Currently, the distinction between content and ad is not obvious at 
first glance.


- Matthias -

Re: [pygccxml-development] pyplusplus status

[Roman Y. <rom...@gm...>]

On 7/18/06, Allen Bierbaum <al...@vr...> wrote:
> What this comment really comes down to is that I don't think the current
> API fullfills the goals set out at:
> https://realityforge.vrsource.org/view/PyOpenSG/PyppApiDiscussion
>
> The big ones I find missing are:
>
>     * Do not require the user to know anything outside of the high-level
>       API to accomplish most tasks.

This will never happen :-(. py++ knows to export only limited subset of C++.

> Please don't take offense to this Roman, but you understand the APIs of
> pygccxml and pyplusplus so well and in so much detail that I think your
> take on what is an "easy API" may be much more in depth then what I
> would consider an "easy API".
> ...
> But having the simple API is still important
> for allowing new users to pick it up and use the tool.

With the power comes complexity. Also I would like to see your
definition to "simple"
and "power" API. My definition is next:

simple - API defined by decl_wrappers package and module_builder
package. Basically,
this is get\set properties + "query" api.

power - modifying directly code creators tree. This is really the
"dark" area of the project,
but I have an idea how to simplify it and of course I will write a
documentation that will
explain it.

> Am I the only one that feels this way when using the API?

I definitely would like to get the answer to this question with
suggestions from other users.

> You just say Template("class") and then instantiate templates.  Very
> straight forward in Pyste's domain specific lanaguage.

Please contribute your code with small example and documentation( and license )
to py++ :-). I already added new directory for this in "contrib" one.

> > mb = module_builder_t( ...
> >     , compilation_mode=pygccxml.parser.COMPILATION_MODE.ALL_AT_ONCE
> >     , ... )
>
> Does this behave *exactly* the same way.  Does it only create one entry
> in the cache?

Almost. pygccxml will create temporal file. I never checked how it
will behave with a
cache. Can you check this, and if the behaviour is not as you expect,
I will try to fix it.

> I would recommend just treating it like you would a .decls() call.  ie.
> if the users looks up declarations of differing types and then calls a
> method on those decls that only makes sense for one type, then python
> will display an error about the type problems.

I prefer to continue to discuss this problem within the original thread.

> Would it be useful to make the tutorial and howto a "live" document so
> everyone can contribute, add questions, make changes?  If so, what about
> using a wiki so everyone can contribute and fix up the descrptions.

I think, this will be useful, but not now. There is only few peoples
who contribute
to the project.

I started to work on FAQs:
http://svn.sourceforge.net/viewcvs.cgi/pygccxml/pyplusplus_dev/docs/faqs/faqs.rest?view=markup

If you have something specifiec in mind, or want to add documentation
here and there,
please do it.

> I don't see how this pep fixes the problem, but then again I don't fully
> understand what the pep will do.  I will just trust you that this will
> help out.

This is pretty simple: this paper deals with method transformations.
For example:
     void get_size( int& h, int&w )
You will be able to define the transformation of get_size to
    boost::python::tuple get_size()

The function wrapper, you are talking about, it is just an "user
defined" transformation.

> def processClass(cls):
>    if cls.count("method"):
>       # do something
>
> I have run into a few cases where something like this could be pretty
> useful and keep the script more understandable.

I will see what I can do.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] pyplusplus status

[Allen B. <al...@vr...>]

Roman Yakovenko wrote:

> On 7/12/06, Allen Bierbaum <al...@vr...> wrote:
>
>> - 90% of users should never have to delve beyond the "easy API" (never
>> need to know about creators). The current API is close to this, but
>> there are still some times when IMHO users have to suddenly delve very
>> deep into the internal code.
>
>
> I would like to see use case, where you have to do this.

I will try to post the use cases as I run across them.  Unfortunately 
most of my binding generation scripts would be classified more as "power 
user" then newbie so I don't use the "easy API".

What this comment really comes down to is that I don't think the current 
API fullfills the goals set out at: 
https://realityforge.vrsource.org/view/PyOpenSG/PyppApiDiscussion

The big ones I find missing are:

    * Do not require the user to know anything outside of the high-level
      API to accomplish most tasks.
    * Provide access to all the power of pypluplus and pygccxml *if* the
      user wants it

Please don't take offense to this Roman, but you understand the APIs of 
pygccxml and pyplusplus so well and in so much detail that I think your 
take on what is an "easy API" may be much more in depth then what I 
would consider an "easy API".  For example, take a look at the Pyste 
tutorial again.  Pyste's API is very domain specific and very simple.  
This makes it easy to approach by new users.  The main problem (and what 
pyplusplus solves extremely well) is that Pyste is very limited once you 
go beyond that simple API.  But having the simple API is still important 
for allowing new users to pick it up and use the tool.

Am I the only one that feels this way when using the API?

>
>> - template support: I would like to see an API that allows users to more
>> easily wrap templates.  I have had to manually create some code that
>> does this but I am sure something better could be done.  You can see
>> what I have done at:
>> https://realityforge.vrsource.org/viewvc/pyopensg/trunk/src/gen_bindings.py?view=markup 
>>
>> Look for the TemplateBuilder helper class.
>
>
> My main objection to this is that users will not use it. Think about
> it. You can create/generate/generate on fly header file/string and to
> include it to the list of header files
> passed to module_builder_t.__init__. You don't have to learn  new API.

And my main reason for saying this is needed is that doing it manually 
is too hard for most users.  It is *much* easier to do this with Pyste.  
You just say Template("class") and then instantiate templates.  Very 
straight forward in Pyste's domain specific lanaguage.  Unfortunately I 
think the domain specific language of pyplusplus is still python and 
this is not good IMHO.

>
> There is "contrib" package, I think your class could be added to it.
> If you agree,
> you can create new directory - "templates" and to add the 
> functionality to it.
> ( You have commit permissions. )
>
>> - Merging header files.  I have found it very helpful to specify a
>> single temporary header file to the module builder that just includes
>> all of the headers for the API I want to wrap.  This can result in
>> significant speed increases when calling gccxml because the header files
>> for a library often include much code in common.  It would be nice if
>> the module builder had a flag in it's constructor (like:
>> combine_headers) that would make pypp do this automatically.
>
>
> This is already implemented:
> http://www.language-binding.net/pygccxml/apidocs/pygccxml.parser.project_reader.project_reader_t-class.html#read_files 
>
>
> mb = module_builder_t( ...
>     , compilation_mode=pygccxml.parser.COMPILATION_MODE.ALL_AT_ONCE
>     , ... )

Does this behave *exactly* the same way.  Does it only create one entry 
in the cache?

>
>> - finalize():  I would really, really, really appreciate a true
>> finalize() method being defined for all decls.  There are a lot of times
>> where I know I want the class to be final so I don't want a wrapper or
>> anything else extra.
>
>
> I will try to find time to work on this feature for next release, not 
> this one.
>
>> - mdecl returns: See the thread "mdecl and methods that return values"
>> from 5/30/2006.  This is a capability that I still very much need.
>
>
> Please, read my last mail in that thread. If you have good definition
> I will implement it.

I would recommend just treating it like you would a .decls() call.  ie. 
if the users looks up declarations of differing types and then calls a 
method on those decls that only makes sense for one type, then python 
will display an error about the type problems.

>
>> - Documentation
>>    - Summary: Make a simple, complete, tutorial document like the one
>> for Pyste.  Just show how to do the things in the "easy API" that 90% of
>> people want to do (see above)
>
>
>>    - simple, complete documentation of user API and what can be done
>>    - documentation for all paramemters of builder methods.
>>       - what does compilation_mode, start_with_declarations, etc do in
>> __init__
>>       - Same with build_code_creator
>>       - what do the params do and how could/should they be used
>>    - How do I do this...  documentation of standard things a user wants
>> to do
>
>
> HowTo is a good idea.
>
>>       - Pyste documentation is very good about this.
>
>
> :-). I am not good in writing documentation. It takes me a lot of time 
> to do it.
> I remember, in the beginning, we made a deal: I write code and you
> write documentation, right :-) ? So, if you want to contribute, we may
> spend some time on IRC channel,
> you will ask questions, I will answer them. And after this you will
> convert our conversation
> to some form of documentation. Otherwise, I will have to completely
> freeze development for
> few month and the only thing I will do is writing documentation.

Would it be useful to make the tutorial and howto a "live" document so 
everyone can contribute, add questions, make changes?  If so, what about 
using a wiki so everyone can contribute and fix up the descrptions.

>
>> - Wrap method: Add API for just wrapping an existing method with a new
>> defined method.  (right now this can be cone but you have to exclude the
>> current method and then call add_code with a 'def'. It is just too
>> complex for a new user and makes the wrapper gen script look 
>> complicated.)
>
>
> http://www.language-binding.net/pyplusplus/peps/call_wrapper_policies.html 
>
>
> When I implement this PEP, it will be pretty easy to do what you want.
> Until then I don't want to introduce something, that I will not 
> support later.

I don't see how this pep fixes the problem, but then again I don't fully 
understand what the pep will do.  I will just trust you that this will 
help out.

>
>> - Add an API to ask if has decl and/or number of found decls
>
>
> decls = mv.decls( ..., allow_empty=True )
> len( decls )

How about something shorter that is a little easier to use in 
conditionals.  Maybe something like count(). so you could do something like:

def processClass(cls):
   if cls.count("method"):
      # do something

I have run into a few cases where something like this could be pretty 
useful and keep the script more understandable.

>
>> - Add ability to add user code to namespace or global *without* having
>> to use code creators directly
>
>
> I want to solve this problem, to this release. Can you write what you
> want/need/expect
> here? In other words I need use cases and how you see they should be 
> solved.


I will have to think about this and get back to you.

>
>> - Cache parsing and preparing query optimizer.  If the input has not
>> changed (the headers are the same) why do I have to wait for all the
>> processing of the declarations and query optimization each time.  Can't
>> we cache all of that?
>
>
> Yes, but you can guess what the priority of this. Also I think that
> solution for this
> problem will be completely in the user code.
>
>> - This is a small nitpick, but unless you are making a lot of money from
>> Google AdSense, could you remove the Google ads in the table of contents
>> on the website.  They are just annoying and make the documentation
>> harder to read.
>
>
> I am not, the AdSence is there to return the money I pay for running 
> web site.
> I hate the idea to ask for donations. User experience is importer.
> I will do something about it.

That would be very helpful. I do not care about the ads on the left bar 
or even if you added ones at the other sides of the pages.  But putting 
ads in the middle of the table of contents or text body is a little 
annoying.

>
>> - Better job of not including platform specific files.  For example I
>> have a binding right now that is adding
>> /usr/include/c++/4.0.2/ext/mt_allocator.h as an include in all the
>> files.  Needless to say this doesn't compile on windows. :(
>
>
> This is another issue I have to solve. Your thoughts are welcome.

I don't really have any ideas because I don't know what is causing it.  
For now, I have just worked around it by over-riding the included 
headers using replace_included_headers() to set the included header(s) 
to be the same list that was originally passed to the module builder.  
It seems to me that this list should always work and will always include 
any headers that are actually needed by the wrapped code.

>
>> - Performance improvements:  improving performance is always good.  The
>> time waiting for all the code_creators to be created is especially long.
>
>
> Do you mind to run your code under profiler? Thus we will have clear
> picture of where
> is a bottleneck. I don't have big use case. All my scripts run in a
> minute or two and most of
> the time is spent in xml parsing.
>
Yes, I can provide you with a profile run once I get back to working on 
my big project.

-Allen

Re: [pygccxml-development] improving user experience

[Roman Y. <rom...@gm...>]

On 7/17/06, Matthias Baas <ba...@ir...> wrote:
> Roman Yakovenko wrote:
> > Also I completely forgot about "readme" functionality. May be my
> > approach is wrong.
> > May be I need to change the code. I think, that instead of reporting
> > warning from
> > "ignore" property, I need to report it from module_creator/creator.py file.
> >
> > What do you think?
>
> Well, I have to admit that I actually was somewhat surprised to see the
> __report_warning() message being invoked when accessing the "ignore"
> property but then, as long as it works.....

It was ugly hack, that does not work :-(.

> But it does sound reasonable to me if problems are reported when the
> code creator tree is created, so maybe this would make the code more clear.

I agree.

> By the way, when exactly is _exportable_impl() called

1. When pyplusplus asks for ignore property value. This is happen from code
    creators factory. ( That is why I introduced the hack )

2. When user calls for
    readme method
    exportable property
    why_not_exportable property

> (or when is the
> exportable property accessed)? Would it also be an option to log a
> warning message right at the place where it is created? (this is
> probably what I would have done in the first place)

You mean during construction, right? It is not possible. The main
reason for this
is that construction is done in 2 steps ( scanner and linker )

> Or is the generated message string used for something else as well?

No. It's main purpose it to be printed to user.

> Or is "exportable" already accessed somewhen before the user actually
> ignored the declaration? (so that the message should not be printed)

No. I don't think so, but we should be careful to not print same message ( from
same declaration ) twice.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] improving user experience

[Matthias B. <ba...@ir...>]

Roman Yakovenko wrote:
> Also I completely forgot about "readme" functionality. May be my
> approach is wrong.
> May be I need to change the code. I think, that instead of reporting
> warning from
> "ignore" property, I need to report it from module_creator/creator.py file.
> 
> What do you think?

Well, I have to admit that I actually was somewhat surprised to see the 
__report_warning() message being invoked when accessing the "ignore" 
property but then, as long as it works.....

But it does sound reasonable to me if problems are reported when the 
code creator tree is created, so maybe this would make the code more clear.

By the way, when exactly is _exportable_impl() called (or when is the 
exportable property accessed)? Would it also be an option to log a 
warning message right at the place where it is created? (this is 
probably what I would have done in the first place)
Or is the generated message string used for something else as well?
Or is "exportable" already accessed somewhen before the user actually 
ignored the declaration? (so that the message should not be printed)


- Matthias -

Re: [pygccxml-development] improving user experience

[Roman Y. <rom...@gm...>]

On 7/17/06, Matthias Baas <ba...@ir...> wrote:
> Roman Yakovenko wrote:
> >> - Even though there are several logging streams now I think warning
> >> message should still be issued using warning() instead of info() (so
> >> that I can set up a file handler at the root logger that only writes
> >> true warnings). For example, in calldef_t._exportable_impl() the message
> >> about the max_arity thing is still written using info(). Roman, do you
> >> mind if I change that to warning()?
> >
> > Please do it.
>
> Done.

Thank you!

> Yes, I have updated earlier today. I did see those messages because of
> my own modification that prints every _exportable_reason that is not empty.
> Now I've seen your __report_warning() method and realized that the
> messages actually should already have appeared in the logger. But
> because I only saw 5 messages whereas my code from above printed many
> more I thought the user has to call something like readme() himself.
> The problem is in __report_warning() which I'd say has a bug. The method
> has the following lines right at the beginning:
>
>      if reason in decl_wrapper_t.ALREADY_REPORTED_MSGS:
>          return
>      decl_wrapper_t.ALREADY_REPORTED_MSGS.add( reason )
>
> But the string "reason" only contains the "raw" message and doesn't
> contain anything that is unique to the current declaration. This means
> if there is another declaration that cannot be exported for the same
> reason, then the warning message for the second declaration will be
> suppressed (even though this is a new message as it refers to a
> different declaration).

Good catch, I did not think about this at all.

> So instead of storing "reason" in ALREADY_REPORTED_MSGS this should
> rather be "msg" (which contains the declaration string). Or maybe it
> would be more efficient if only a reference to the declaration is stored
> instead of an entire string

Yes. It will be enough.

>(or can a declaration have more than one "reason" messages?).

Yes and no.

Yes - one function could have many problems.
No - actually any message indicates that this declaration has some
"fatal" error.

Also I completely forgot about "readme" functionality. May be my
approach is wrong.
May be I need to change the code. I think, that instead of reporting
warning from
"ignore" property, I need to report it from module_creator/creator.py file.

What do you think?

If you think, that this approach is better and you have time to
implement it, please
implement it.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] improving user experience

[Matthias B. <ba...@ir...>]

Roman Yakovenko wrote:
>> - Even though there are several logging streams now I think warning
>> message should still be issued using warning() instead of info() (so
>> that I can set up a file handler at the root logger that only writes
>> true warnings). For example, in calldef_t._exportable_impl() the message
>> about the max_arity thing is still written using info(). Roman, do you
>> mind if I change that to warning()?
> 
> Please do it.

Done.

>> - Most warnings are still "hidden" from the user until he explicitly
>> obtains them using the readme() method. I did a test and modified the
>> code in decl_wrapper_t.get_exportable() so that any time
>> self._exportable_reason is set to a non-empty string this string is also
>> written to the logger.
>> [...]
>> A lot of the above new warnings are about compiler generated
>> constructors (it always seems to be the copy constructor in my case). Is
>> this really worth a message at all? I certainly don't expect pyplusplus
>> to generate wrappers for things that are not there, so I think
>> pyplusplus could just silently ignore those artifical declarations that
>> are only created by gccxml (I don't really know why they are generated
>> at all?).
> 
> I thought, I disabled them. I will check one more time. Can you check, 
> whether you are running with latest version?

Yes, I have updated earlier today. I did see those messages because of 
my own modification that prints every _exportable_reason that is not empty.
Now I've seen your __report_warning() method and realized that the 
messages actually should already have appeared in the logger. But 
because I only saw 5 messages whereas my code from above printed many 
more I thought the user has to call something like readme() himself.
The problem is in __report_warning() which I'd say has a bug. The method 
has the following lines right at the beginning:

     if reason in decl_wrapper_t.ALREADY_REPORTED_MSGS:
         return
     decl_wrapper_t.ALREADY_REPORTED_MSGS.add( reason )

But the string "reason" only contains the "raw" message and doesn't 
contain anything that is unique to the current declaration. This means 
if there is another declaration that cannot be exported for the same 
reason, then the warning message for the second declaration will be 
suppressed (even though this is a new message as it refers to a 
different declaration).
So instead of storing "reason" in ALREADY_REPORTED_MSGS this should 
rather be "msg" (which contains the declaration string). Or maybe it 
would be more efficient if only a reference to the declaration is stored 
instead of an entire string (or can a declaration have more than one 
"reason" messages?).

- Matthias -

Re: [pygccxml-development] improving user experience

[Roman Y. <rom...@gm...>]

On 7/17/06, Matthias Baas <ba...@ir...> wrote:
> Roman Yakovenko wrote:
> > Hi. I just committed a set of changes, that should improve usability
> > of pyplusplus.
> > To be more specific: by default only important messages are send to the
> > user:
> > 1. gccxml invocation command line ( info )
> > 2. warning about declaration, that could not be exported
>
> I've noticed two things:
>
> - Even though there are several logging streams now I think warning
> message should still be issued using warning() instead of info() (so
> that I can set up a file handler at the root logger that only writes
> true warnings). For example, in calldef_t._exportable_impl() the message
> about the max_arity thing is still written using info(). Roman, do you
> mind if I change that to warning()?

Please do it.

> - Most warnings are still "hidden" from the user until he explicitly
> obtains them using the readme() method. I did a test and modified the
> code in decl_wrapper_t.get_exportable() so that any time
> self._exportable_reason is set to a non-empty string this string is also
> written to the logger. What I don't know is if the "exportable"
> attribute will always be obtained by pyplusplus internally, i.e. if
> get_exportable() will always be called or not.

Yes. Please take a look on _get_ignore method. It will be always called from it.

> By running my Maya stuff
> it seems to be the case as I got much more warnings. So should I check
> that in as well?

I don't understand, sorry. What do you want to check/fix?

>And are there other methods that create warning/error
> messages that are only returned as strings instead of being written to a
> logger?

decl_wrappers.decl_wrapper_t derived classes should not directly write to log.
Instead of this, they should implement _exportable_impl method, that will return
string( explanation ) if declaration could not be exported and None if it could.

> A lot of the above new warnings are about compiler generated
> constructors (it always seems to be the copy constructor in my case). Is
> this really worth a message at all? I certainly don't expect pyplusplus
> to generate wrappers for things that are not there, so I think
> pyplusplus could just silently ignore those artifical declarations that
> are only created by gccxml (I don't really know why they are generated
> at all?).

I thought, I disabled them. I will check one more time. Can you check, whether
you are running with latest version?

Thanks
-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] improving user experience

[Matthias B. <ba...@ir...>]

Roman Yakovenko wrote:
> Hi. I just committed a set of changes, that should improve usability
> of pyplusplus.
> To be more specific: by default only important messages are send to the 
> user:
> 1. gccxml invocation command line ( info )
> 2. warning about declaration, that could not be exported

I've noticed two things:

- Even though there are several logging streams now I think warning 
message should still be issued using warning() instead of info() (so 
that I can set up a file handler at the root logger that only writes 
true warnings). For example, in calldef_t._exportable_impl() the message 
about the max_arity thing is still written using info(). Roman, do you 
mind if I change that to warning()?

- Most warnings are still "hidden" from the user until he explicitly 
obtains them using the readme() method. I did a test and modified the 
code in decl_wrapper_t.get_exportable() so that any time 
self._exportable_reason is set to a non-empty string this string is also 
written to the logger. What I don't know is if the "exportable" 
attribute will always be obtained by pyplusplus internally, i.e. if 
get_exportable() will always be called or not. By running my Maya stuff 
it seems to be the case as I got much more warnings. So should I check 
that in as well? And are there other methods that create warning/error 
messages that are only returned as strings instead of being written to a 
logger?

A lot of the above new warnings are about compiler generated 
constructors (it always seems to be the copy constructor in my case). Is 
this really worth a message at all? I certainly don't expect pyplusplus 
to generate wrappers for things that are not there, so I think 
pyplusplus could just silently ignore those artifical declarations that 
are only created by gccxml (I don't really know why they are generated 
at all?).

- Matthias -

Re: [pygccxml-development] pyplusplus status

[Roman Y. <rom...@gm...>]

On 7/12/06, Allen Bierbaum <al...@vr...> wrote:
> - 90% of users should never have to delve beyond the "easy API" (never
> need to know about creators). The current API is close to this, but
> there are still some times when IMHO users have to suddenly delve very
> deep into the internal code.

I would like to see use case, where you have to do this.

> - template support: I would like to see an API that allows users to more
> easily wrap templates.  I have had to manually create some code that
> does this but I am sure something better could be done.  You can see
> what I have done at:
> https://realityforge.vrsource.org/viewvc/pyopensg/trunk/src/gen_bindings.py?view=markup
> Look for the TemplateBuilder helper class.

My main objection to this is that users will not use it. Think about
it. You can create/generate/generate on fly header file/string and to
include it to the list of header files
passed to module_builder_t.__init__. You don't have to learn  new API.

There is "contrib" package, I think your class could be added to it.
If you agree,
you can create new directory - "templates" and to add the functionality to it.
( You have commit permissions. )

> - Merging header files.  I have found it very helpful to specify a
> single temporary header file to the module builder that just includes
> all of the headers for the API I want to wrap.  This can result in
> significant speed increases when calling gccxml because the header files
> for a library often include much code in common.  It would be nice if
> the module builder had a flag in it's constructor (like:
> combine_headers) that would make pypp do this automatically.

This is already implemented:
http://www.language-binding.net/pygccxml/apidocs/pygccxml.parser.project_reader.project_reader_t-class.html#read_files

mb = module_builder_t( ...
     , compilation_mode=pygccxml.parser.COMPILATION_MODE.ALL_AT_ONCE
     , ... )

> - finalize():  I would really, really, really appreciate a true
> finalize() method being defined for all decls.  There are a lot of times
> where I know I want the class to be final so I don't want a wrapper or
> anything else extra.

I will try to find time to work on this feature for next release, not this one.

> - mdecl returns: See the thread "mdecl and methods that return values"
> from 5/30/2006.  This is a capability that I still very much need.

Please, read my last mail in that thread. If you have good definition
I will implement it.

> - Documentation
>    - Summary: Make a simple, complete, tutorial document like the one
> for Pyste.  Just show how to do the things in the "easy API" that 90% of
> people want to do (see above)

>    - simple, complete documentation of user API and what can be done
>    - documentation for all paramemters of builder methods.
>       - what does compilation_mode, start_with_declarations, etc do in
> __init__
>       - Same with build_code_creator
>       - what do the params do and how could/should they be used
>    - How do I do this...  documentation of standard things a user wants
> to do

HowTo is a good idea.

>       - Pyste documentation is very good about this.

:-). I am not good in writing documentation. It takes me a lot of time to do it.
I remember, in the beginning, we made a deal: I write code and you
write documentation, right :-) ? So, if you want to contribute, we may
spend some time on IRC channel,
you will ask questions, I will answer them. And after this you will
convert our conversation
to some form of documentation. Otherwise, I will have to completely
freeze development for
few month and the only thing I will do is writing documentation.

> - Wrap method: Add API for just wrapping an existing method with a new
> defined method.  (right now this can be cone but you have to exclude the
> current method and then call add_code with a 'def'. It is just too
> complex for a new user and makes the wrapper gen script look complicated.)

http://www.language-binding.net/pyplusplus/peps/call_wrapper_policies.html

When I implement this PEP, it will be pretty easy to do what you want.
Until then I don't want to introduce something, that I will not support later.

> - Add an API to ask if has decl and/or number of found decls

decls = mv.decls( ..., allow_empty=True )
len( decls )

> - Add ability to add user code to namespace or global *without* having
> to use code creators directly

I want to solve this problem, to this release. Can you write what you
want/need/expect
here? In other words I need use cases and how you see they should be solved.

> - Cache parsing and preparing query optimizer.  If the input has not
> changed (the headers are the same) why do I have to wait for all the
> processing of the declarations and query optimization each time.  Can't
> we cache all of that?

Yes, but you can guess what the priority of this. Also I think that
solution for this
problem will be completely in the user code.

> - This is a small nitpick, but unless you are making a lot of money from
> Google AdSense, could you remove the Google ads in the table of contents
> on the website.  They are just annoying and make the documentation
> harder to read.

I am not, the AdSence is there to return the money I pay for running web site.
I hate the idea to ask for donations. User experience is importer.
I will do something about it.

> - Better job of not including platform specific files.  For example I
> have a binding right now that is adding
> /usr/include/c++/4.0.2/ext/mt_allocator.h as an include in all the
> files.  Needless to say this doesn't compile on windows. :(

This is another issue I have to solve. Your thoughts are welcome.

> - Performance improvements:  improving performance is always good.  The
> time waiting for all the code_creators to be created is especially long.

Do you mind to run your code under profiler? Thus we will have clear
picture of where
is a bottleneck. I don't have big use case. All my scripts run in a
minute or two and most of
the time is spent in xml parsing.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] pyplusplus status

[Allen B. <al...@vr...>]

Roman:

As promised, here is a short list of the things I have found while using 
pyplusplus recently.

- 90% of users should never have to delve beyond the "easy API" (never 
need to know about creators). The current API is close to this, but 
there are still some times when IMHO users have to suddenly delve very 
deep into the internal code.

- template support: I would like to see an API that allows users to more 
easily wrap templates.  I have had to manually create some code that 
does this but I am sure something better could be done.  You can see 
what I have done at: 
https://realityforge.vrsource.org/viewvc/pyopensg/trunk/src/gen_bindings.py?view=markup  
Look for the TemplateBuilder helper class.

- Merging header files.  I have found it very helpful to specify a 
single temporary header file to the module builder that just includes 
all of the headers for the API I want to wrap.  This can result in 
significant speed increases when calling gccxml because the header files 
for a library often include much code in common.  It would be nice if 
the module builder had a flag in it's constructor (like: 
combine_headers) that would make pypp do this automatically.

- finalize():  I would really, really, really appreciate a true 
finalize() method being defined for all decls.  There are a lot of times 
where I know I want the class to be final so I don't want a wrapper or 
anything else extra.

- mdecl returns: See the thread "mdecl and methods that return values" 
from 5/30/2006.  This is a capability that I still very much need.
 
- Documentation
   - Summary: Make a simple, complete, tutorial document like the one 
for Pyste.  Just show how to do the things in the "easy API" that 90% of 
people want to do (see above)
   - simple, complete documentation of user API and what can be done
   - documentation for all paramemters of builder methods.
      - what does compilation_mode, start_with_declarations, etc do in 
__init__
      - Same with build_code_creator
      - what do the params do and how could/should they be used
   - How do I do this...  documentation of standard things a user wants 
to do
      - Pyste documentation is very good about this.
- Wrap method: Add API for just wrapping an existing method with a new 
defined method.  (right now this can be cone but you have to exclude the 
current method and then call add_code with a 'def'. It is just too 
complex for a new user and makes the wrapper gen script look complicated.)

- Add an API to ask if has decl and/or number of found decls

- Add ability to add user code to namespace or global *without* having 
to use code creators directly

- Cache parsing and preparing query optimizer.  If the input has not 
changed (the headers are the same) why do I have to wait for all the 
processing of the declarations and query optimization each time.  Can't 
we cache all of that?

- This is a small nitpick, but unless you are making a lot of money from 
Google AdSense, could you remove the Google ads in the table of contents 
on the website.  They are just annoying and make the documentation 
harder to read.

- Better job of not including platform specific files.  For example I 
have a binding right now that is adding 
/usr/include/c++/4.0.2/ext/mt_allocator.h as an include in all the 
files.  Needless to say this doesn't compile on windows. :(

- Performance improvements:  improving performance is always good.  The 
time waiting for all the code_creators to be created is especially long.


Comments?

-Allen

Roman Yakovenko wrote:

>Hi. I just want to describe what have been done and what I think should be done
>before next version will be ready to be released.
>
>Done:
>1. pyplusplus now preserves the order of generated code. I mean if you
>run pyplusplus
>    twice, than second run will not change any file. If you see a
>different behaviour, please
>    report a bug.
>
>2. Usability of pyplusplus has been improved. Now by default it will
>print only very important
>    messages:
>    1. gccxml command line ( printed text will start with "INFO" word )
>    2. declarations, that user asked to export, but for some reason,
>pyplusplus can
>        not export them ( ( printed text will start with "WARNING" word )
>    3. a list of updated(created) files ( printed text will start with
>"INFO" word )
>
>    It is possible to configure pyplusplus to print much more information:
>
>    import logger
>    from pyplusplus import module_builder
>    module_builder.set_logger_level( logging.DEBUG )
>
>3. pyplusplus is able to generate code for indexing suite v2.
>    You need explicitly to enable it:
>    mb = module_builder.module_builder_t( ..., indexing_suite_version=2 )
>
>    The documentation for this feature is in progress
>
>What still should be done?
>
>1. Georgiy Dernovoy contributed code, that extracts doxygen
>documentation from source
>    files. I will add code to pyplusplus that will generate documentation.
>
>2. I should find solution for long standing problem: the number of
>include directives
>    in generated source files.
>
>3. I am going to work a little on file_writers package. The main
>missing future here is
>   to add user code to generated source file.
>
>4. Documentation.
>
>For items (2) and (3), I'd like to know your opinion:
>    1. What code pyplusplus should generate?
>    2. How do you prefer to configure the feature? ( user interface )
>
>I don't want to set release day,  but from now I will try to keep
>source code in
>"release ready" mode. If you can try source code from "Subversion" and
>report bugs
>it will be grate.
>
>  
>

Re: [pygccxml-development] demand driven instantiation

[Roman Y. <rom...@gm...>]

On 7/12/06, Neal Becker <ndb...@gm...> wrote:
> One of the challenges of integrating algorithms written in c++ with python
> wrappers is that we want to write generic c++ code.  It is not obvious how to
> support compile-time polymorphism in this framework.
>
> I wonder is pyplusplus can help here?  Maybe a python program interface to
> call pyplusplus to generate a specific instantiation wrapper, call the
> compiler, and load the module?

This will work. pyplusplus has all functionality to do it. You can
take a look on
pyplusplus unittests. This is almost exactly what you want.

There is one big minus - such solution will complicate product delivery.
0. different os and environments
1. good compiler
2. boost.python
3. gccxml
4. pygccxml + pyplusplus
5. scons or other build system, or you will have to generate makefiles
6. your product

It is going to be nightmare :-).

May be you should study Python bindings for boost.graph library.
Or to create all possible or very useful template instantiations and to export
them?

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

[pygccxml-development] demand driven instantiation

[Neal B. <ndb...@gm...>]

One of the challenges of integrating algorithms written in c++ with python 
wrappers is that we want to write generic c++ code.  It is not obvious how to 
support compile-time polymorphism in this framework.

I wonder is pyplusplus can help here?  Maybe a python program interface to 
call pyplusplus to generate a specific instantiation wrapper, call the 
compiler, and load the module? 

[pygccxml-development] pyplusplus status

[Roman Y. <rom...@gm...>]

Hi. I just want to describe what have been done and what I think should be done
before next version will be ready to be released.

Done:
1. pyplusplus now preserves the order of generated code. I mean if you
run pyplusplus
    twice, than second run will not change any file. If you see a
different behaviour, please
    report a bug.

2. Usability of pyplusplus has been improved. Now by default it will
print only very important
    messages:
    1. gccxml command line ( printed text will start with "INFO" word )
    2. declarations, that user asked to export, but for some reason,
pyplusplus can
        not export them ( ( printed text will start with "WARNING" word )
    3. a list of updated(created) files ( printed text will start with
"INFO" word )

    It is possible to configure pyplusplus to print much more information:

    import logger
    from pyplusplus import module_builder
    module_builder.set_logger_level( logging.DEBUG )

3. pyplusplus is able to generate code for indexing suite v2.
    You need explicitly to enable it:
    mb = module_builder.module_builder_t( ..., indexing_suite_version=2 )

    The documentation for this feature is in progress

What still should be done?

1. Georgiy Dernovoy contributed code, that extracts doxygen
documentation from source
    files. I will add code to pyplusplus that will generate documentation.

2. I should find solution for long standing problem: the number of
include directives
    in generated source files.

3. I am going to work a little on file_writers package. The main
missing future here is
   to add user code to generated source file.

4. Documentation.

For items (2) and (3), I'd like to know your opinion:
    1. What code pyplusplus should generate?
    2. How do you prefer to configure the feature? ( user interface )

I don't want to set release day,  but from now I will try to keep
source code in
"release ready" mode. If you can try source code from "Subversion" and
report bugs
it will be grate.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] improving user experience

[Roman Y. <rom...@gm...>]

Hi. I just committed a set of changes, that should improve usability
of pyplusplus.
To be more specific: by default only important messages are send to the user:
1. gccxml invocation command line ( info )
2. warning about declaration, that could not be exported

If you want to see all information, that py++ have to say you can
change logger level to
DEBUG:

import logging
import module_builder

module_builder.set_logger_level( logging.DEBUG )

I did not implemented the original idea: to write as comment
declarations that could not
be exported - too much work. I'd like to get feedback about current situation.
If the situation is still bad, I will do that.


Thanks

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

[pygccxml-development] verbose is going to be deprecated

[Roman Y. <rom...@gm...>]

Hi. I am going to deprecate "verbose" flag in pygccxml. Instead of
this I am going to add
another logger, to preserve the functionality.

Here is a hierarchy of loggers for pygccxml project:
pygccxml
    queries_engine
        all messages to this logger will be written with log level "DEBUG".
    gccxml
        gccxml invocation command line and result will be written with
log level "INFO". All
        other messages will be written with log level DEBUG.
    cache
        only open cache message will be written with "INFO" level. All
other messages will be
        written with "DEBUG" level.

All loggers will be configured to "INFO" level. I think this change
will reduce amount
of information written by pygccxml.

Thoughts?

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

[pygccxml-development] code generation changes

[Roman Y. <rom...@gm...>]

Hi. I just committed changes. Those changes have 2 goals:
1. To fix msvc 7.1 compilation bug.

    x.def( "__call__", ( void ( X::* )( int ) )( &X::operator() ) );

   This code compiles fine with gcc, but could not be compiled with MSVC 7.1

2. To improve generated code readability:
    The previous statement will be generated like this:

    typedef void (X::*function_ptr_t)( int );
    x.def(
          "__call__"
        , function_ptr_t( &X::operator() )
        , ... )

Sorry for inconvenience, but I think those changes are absolutely necessary.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Order of enum values

[Roman Y. <rom...@gm...>]

On 7/8/06, Matthias Baas <ba...@ir...> wrote:
> I've replaced the dict in enumeration_t with a list and (hopefully)
> changed all other places that are affected by this. Here's what I changed:
>
> In pygccxml:
> ============
>
> enumeration_t:
> --------------
>
> - Store the values as an ordered list (same order as in the source
> file). List items are tuples (name(str), value(int))

I added new function called get_name2value_dict. There were too many places
in my unittests, where I needed it.

> - The 'values' property returns a *copy* of the internal list so that
> external code cannot mess with internal data.
> (Is that ok? Or is it intentional that a user receives the internal object?)

I don't have an opinion. In C++ I would return const reference to container,
but in Python ...? I prefere to think about declarations classes as
read only ones.
If user wants to modify them, I suppose he knows what he is doing.

> - Added a method append_value() that can be used to gradually construct
> the enum (instead of that the scanner has to know about internal details
> of the class)

Scanner and linker know almost all internal details of every class. In
this case,
it is possible to elimintate implementation details.

>
> It's not clear to me why the code creators also define properties
> value_aliases and export_values. I didn't use them in _create_impl() but
> I left them in anyway because I don't know if other code depends on them.

It is possible to rename an exported enum value

enum X{ x };

enum_<X>( "X" )
    .value( "XYZ", x );


> Finally, I have some "criticism" towards the code (you should read this
> as suggestions for improvements in future code).

I am always glad, when somebody does code review to what I wrote.

>  I think the
> enumeration_t class (and its wrapper class) wasn't properly
> encapsulating its data which I would regard as a design flaw. If they
> were encapsulating it, a modification like the above (changing internal
> data structures) should have been possible by only modifying
> enumeration_t itself. But other parts of the code were getting access to
> the internal dict and could do whatever they wanted with it without that
> the class has the chance to validate the data. That's why I have
> introduced new methods like append_value() which the scanner uses to add
> a new value to the enum without having to worry how exactly this has to
> be done (the enum class knows that better than the scanner). And for the
> same reason I modified the get/set methods for the 'values' property.
> Previously, they were passing the value right through to the internal
> attribute. This means a caller could obtain a reference to the internal
> dict and mess with it in any way. Or you could just set a value that
> contained corrupt data or that wasn't a dict at all. Now, retrieving the
> list gives you a copy so you can do with that copy what you want without
> accidentally corrupting the internal class data. When setting a new
> value some checks are done so that all in all, it's now much more
> difficult to get the enumeration class into an invalid state.
>
> This is not yet true for the enumeration wrapper class. This one still
> provides access to its internal _value_aliases and _export_values
> attributes. But I noticed in the unit tests that they're obviously
> supposed to be used like that so that you can add new values to those
> variables. I think a different way, such as using methods for adding
> values, where the class has the chance to validate the data would have
> been a more safe interface.
>
> As I said above, these were just some observations from me that I
> thought I should mention. I'm not expecting any immediate changes of the
> code (as long as it works it's ok), it's just supposed to be "food for
> thought" for future designs.

I understand, what you are talking about. Interface, that exposes functions,
is much better, stable and clear than interface built arround data structures
and that gives access to it's variables. I thought, that providing interface
built on "data structures" will simplify it. I will reconsider my approach.


Thank you very much for patch and very constructive "criticism".

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Order of enum values

[Matthias B. <ba...@ir...>]

Roman Yakovenko wrote:
>> Could the value actually also be something else than an int?
> 
> I am not sure. May be it possible to create an enum, with some 
> expression as
> value. And may be in this case gccxml will dump it as a string. I don't 
> know.
> So, small investigation is needed. If it could be expression, then
> 'int' in comment should be replaced to 'string'

I did some more reading and a couple of tests and it appears that it's 
always an int.

>> - Store the values inside a list instead of a dict. The items can be
>> either just the enum names or tuples (name, value) if the value is still
>> required somewhen.
> 
> You loose a convenience of dictionary: value = x[name].

Yes, but...

>> so maybe there are cases where a value has to be looked up quickly which
>> justifies using a dict.
> 
> There is no such place.

...as such a lookup is never used, we don't lose anything.

> If gccxml reports enums, using the order from source file, you can
> replace dict with
> a list, otherwise any order you define will surprise a user.

In my tests, gccxml always reported the enum values in the same order as 
they appear in the source file.

An enum like this

   enum Color { red=1, green=3, blue=2};

is reported in the order red, green, blue whereas it would be red, blue, 
green if sorted by numeric value.

> I prefer to make a change in pygccxml, but I am not sure, that it will
> be enough.
> Please take a look on decl_wrappers.enumeration_t. It also defines few
> containers,
> may be you will have to change those containers too. Please take a look 
> also
> on code_creators.enum_t and code_creators.unnamed_enum_t. They generate
> the code, I think you will have to fix them too :-(.
> 
> Please, implement what ever you think is right and I try to help you as 
> much as I can.

I've replaced the dict in enumeration_t with a list and (hopefully) 
changed all other places that are affected by this. Here's what I changed:

In pygccxml:
============

enumeration_t:
--------------

- Store the values as an ordered list (same order as in the source 
file). List items are tuples (name(str), value(int))

- The 'values' property returns a *copy* of the internal list so that 
external code cannot mess with internal data.
(Is that ok? Or is it intentional that a user receives the internal object?)

- Added a method append_value() that can be used to gradually construct 
the enum (instead of that the scanner has to know about internal details 
of the class)

- Added a method has_value_name(). (This method would be more efficient 
with a dict but I didn't notice any slowdown so it's probably not worth 
spending any time on this. But it would be no problem adding an 
additional dict to the class)

=> The internal attribute _values is completely hidden by the class


scanner_t:
----------

- Converted the numeric value into an int

- Uses the append_value() method instead of directly manipulating class 
internals


parser.patcher.py:
------------------

- Updated __find_enum()
- Updated __is_unqualified_enum()


In pyplusplus:
==============

enumeration_wrapper.enumeration_t:
----------------------------------

- Added a property no_export_values that is the complementary of 
export_values.


code_creators.enum_t / code_creators.unnamed_enum_t:
----------------------------------------------------

- Updated code creation to reflect the changes of the enum declaration 
objects.

It's not clear to me why the code creators also define properties 
value_aliases and export_values. I didn't use them in _create_impl() but 
I left them in anyway because I don't know if other code depends on them.


I've tested the stuff with my Maya bindings and it seems to work fine 
(this will safe me quite some compilation time! :)
I've already committed the modifications so please have a look at them 
to make sure I didn't accidentally break anything else. Don't forget to 
remove any cache you have lying around (I stumbled across that one 
again! *sigh*)

Finally, I have some "criticism" towards the code (you should read this 
as suggestions for improvements in future code). I think the 
enumeration_t class (and its wrapper class) wasn't properly 
encapsulating its data which I would regard as a design flaw. If they 
were encapsulating it, a modification like the above (changing internal 
data structures) should have been possible by only modifying 
enumeration_t itself. But other parts of the code were getting access to 
the internal dict and could do whatever they wanted with it without that 
the class has the chance to validate the data. That's why I have 
introduced new methods like append_value() which the scanner uses to add 
a new value to the enum without having to worry how exactly this has to 
be done (the enum class knows that better than the scanner). And for the 
same reason I modified the get/set methods for the 'values' property. 
Previously, they were passing the value right through to the internal 
attribute. This means a caller could obtain a reference to the internal 
dict and mess with it in any way. Or you could just set a value that 
contained corrupt data or that wasn't a dict at all. Now, retrieving the 
list gives you a copy so you can do with that copy what you want without 
accidentally corrupting the internal class data. When setting a new 
value some checks are done so that all in all, it's now much more 
difficult to get the enumeration class into an invalid state.
This is not yet true for the enumeration wrapper class. This one still 
provides access to its internal _value_aliases and _export_values 
attributes. But I noticed in the unit tests that they're obviously 
supposed to be used like that so that you can add new values to those 
variables. I think a different way, such as using methods for adding 
values, where the class has the chance to validate the data would have 
been a more safe interface.

As I said above, these were just some observations from me that I 
thought I should mention. I'm not expecting any immediate changes of the 
code (as long as it works it's ok), it's just supposed to be "food for 
thought" for future designs.


- Matthias -