pygccxml-development

[pygccxml-development] AttributeError: can't set attribute error when importing a compiled boost.python module generated by Py++

[Pecevski D. <de...@ig...>]

Hi,

I'm using py++ to generate boost.python wrappers in my project. The 
generated source code compiles well, but when I try
to import the generated .so library in python, I get this strange poor 
error message:
AttributeError: can't set attribute error

Apparently there is a problem with the generated pypp code.

After googling, I found this post related to some bug in pyste that 
produces the same error:
http://mail.python.org/pipermail/c++-sig/2003-August/005108.html
It has to do something with generating multiple def()'s  in boost.python 
code.

Does somebody have any idea what can be the problem?

I'm using linux with the latest gccxml cvs dump, and latest version of 
pygccxml and py++ (0.8.2).

Thanks in advance,
Dejan

Re: [pygccxml-development] Another way to speed up the generation process...

[Roman Y. <rom...@gm...>]

On 10/30/06, Matthias Baas <ba...@ir...> wrote:
> Roman Yakovenko wrote:
> > May be you can investigate this and create small explanation to "best
> > practices"
> > section:
> > http://language-binding.net/pyplusplus/documentation/best_practices.html ?
>
> I have just added (and committed) a section about the -fxml-start option
> to best_practices.rest (I hope I got the markup right, I didn't try to
> run the docutils).

Thank you. I will review it later and correct (docutils ) if needed.
I am still pretty busy.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Another way to speed up the generation process...

[Matthias B. <ba...@ir...>]

Roman Yakovenko wrote:
> May be you can investigate this and create small explanation to "best 
> practices"
> section:
> http://language-binding.net/pyplusplus/documentation/best_practices.html ?

I have just added (and committed) a section about the -fxml-start option 
to best_practices.rest (I hope I got the markup right, I didn't try to 
run the docutils).

- Matthias -

Re: [pygccxml-development] FT - recent changes

[Roman Y. <rom...@gm...>]

On 10/25/06, Matthias Baas <ba...@ir...> wrote:
> Roman Yakovenko wrote:
> >> > do_smth.add_transformation() #<- use Py++ default behaviour
> >> > do_smth.add_transformation( raise_on_error( 0, ... ) )
> >>
> >> How do you rename the first version to "do_smth_no_raise" (without
> >> renaming the other one)?
> >
> > do_smth.add_transformation( "do_smth_no_raise", raise_on_error( 0, ... ) )
>
> What about other decoration operations (such as assigning call policies,
> assigning a doc string, etc.)?

This is exactly function_transformation_t class for.

add_transformation is just a convenience method.

> Why isn't the existing interface used? The user could obtain a handle to
> a "clone" of the original declaration (either returned by
> add_transformation() or maybe even by a dedicated clone() method) that
> provides the same interface as the regular declarations. Then the user
> could do to the clone whatever he could do to every other declaration
> using an interface that he knows already.

In general asking a user to modify his data is a bad idea. Py++ should
not change
the declarations tree, but work with it as is.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] FT - recent changes

[Matthias B. <ba...@ir...>]

Roman Yakovenko wrote:
>> > do_smth.add_transformation() #<- use Py++ default behaviour
>> > do_smth.add_transformation( raise_on_error( 0, ... ) )
>>
>> How do you rename the first version to "do_smth_no_raise" (without
>> renaming the other one)?
> 
> do_smth.add_transformation( "do_smth_no_raise", raise_on_error( 0, ... ) )

What about other decoration operations (such as assigning call policies, 
assigning a doc string, etc.)?
Why isn't the existing interface used? The user could obtain a handle to 
a "clone" of the original declaration (either returned by 
add_transformation() or maybe even by a dedicated clone() method) that 
provides the same interface as the regular declarations. Then the user 
could do to the clone whatever he could do to every other declaration 
using an interface that he knows already.

- Matthias -

Re: [pygccxml-development] FT - recent changes

[Roman Y. <rom...@gm...>]

On 10/25/06, Matthias Baas <ba...@ir...> wrote:
> Roman Yakovenko wrote:
> > For example:
> >
> > struct error_t{};
> >
> > void do_smth( ..., error_t& );
> >
> > I want to expose do_smth in 2 ways:
> >
> > 1. as is: def( "do_smth_no_raise", &do_smth );
> > 2. error status is converted by Py++ to exception:
> >
> > void do_smth_raise(){
> >     error_t x;
> >     do_smth( x );
> >     if x contains error:
> >        throw actual error;
> > }
> >
> > def( "do_smth", &do_smth_raise );
> >
> >
> > do_smth.add_transformation() #<- use Py++ default behaviour
> > do_smth.add_transformation( raise_on_error( 0, ... ) )
>
> How do you rename the first version to "do_smth_no_raise" (without
> renaming the other one)?

do_smth.add_transformation( "do_smth_no_raise", raise_on_error( 0, ... ) )

I want to make it clear: this functionality is disabled now.
But in future function_transformation_t class will get new property - alias.
add_transformation method will change:
def add_transformation( *args ):
    alias = self.alias
    if args and isinstance( args[0], str ):
        alias = args[0]
        args = args[1:]
    ....

So it should be pretty simple to add it. I just want to concentrate my attention
on small refactoring steps.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Another way to speed up the generation process...

[Matthias B. <ba...@ir...>]

Allen Bierbaum wrote:
>> I'm not exactly sure which customizations you mean, but I don't think 
>> I'm interfering with your customizations. The only modification to 
>> Py++ that I needed is already checked in (the modification to pygccxml 
>> so that XML files get cached). The actual pruning happens outside Py++ 
>> and doesn't need any further customization.
>>
>> And for the query cache the only customization I needed was to 
>> override the constructor of declaration_t which I can do within my own 
>> code so it's not necessary to modify Py++ itself.
> 
> Ok. The modification I did in goodies was to override the module builder.
> 
> I as not worried at all about interference, I was actually hoping that 
> what you were doing would find it's way into Py++ so I could get rid of 
> my extensions.  I would like to see a way that everyone can see the 
> additional performance of better caching.  I hope to try out some of 
> your performance improvements soon.  I can always find a need for better 
> performance. :)

I still have to commit the changes to pypp_api (where the query cache is 
located). I'm not sure if it would be just as straightforward to 
implement the cache in the official query functions. From the examples 
I've seen so far the regular query functions seem to rely on lambda 
functions more heavily than is done in pypp_api, whose filters are 
rather "value-based" instead of being "function-based". So it is not as 
straightforward anymore to create a unique key for the cache entries 
(i.e. to find out if a particular function that was passed to a query 
operation is still the same as in the previous run). You could probably 
extract the source code of the function and compare that one, but this 
could also break if the function uses another function and only that 
other function has been changed.

- Matthias -

Re: [pygccxml-development] FT - recent changes

[Matthias B. <ba...@ir...>]

Roman Yakovenko wrote:
> For example:
> 
> struct error_t{};
> 
> void do_smth( ..., error_t& );
> 
> I want to expose do_smth in 2 ways:
> 
> 1. as is: def( "do_smth_no_raise", &do_smth );
> 2. error status is converted by Py++ to exception:
> 
> void do_smth_raise(){
>     error_t x;
>     do_smth( x );
>     if x contains error:
>        throw actual error;
> }
> 
> def( "do_smth", &do_smth_raise );
> 
> 
> do_smth.add_transformation() #<- use Py++ default behaviour
> do_smth.add_transformation( raise_on_error( 0, ... ) )

How do you rename the first version to "do_smth_no_raise" (without 
renaming the other one)?

- Matthias -

Re: [pygccxml-development] Another way to speed up the generation process...

[Allen B. <al...@vr...>]

Matthias Baas wrote:
> Allen Bierbaum wrote:
>>> What I did were two things: pruning the declaration tree and caching 
>>> query operations. I don't use the pruning function from my earlier 
>>> mail anymore because this pruning is only done after the header files 
>>> were already parsed and stored in the cache. So while the function 
>>> could speed up the decoration stage it could not speed up the parsing 
>>> stage (which took more than a minute on OSX using the regular cache). 
>>> So now I prune the declarations at an earlier stage, namely right 
>>> after the XML file was created by gccxml and before pygccxml reads it. 
>>> I wrote a little utility that directly operates on the XML file and 
>>> outputs a pruned XML file which I use as input for Py++ (I think I'll 
>>> put it into contrib eventually). This should now even be safer than 
>>> before because I also keep all dependencies even when they are not 
>>> inside any of the allowed headers. With the pruned tree alone I am 
>>> already at the above parsing time (when the cache is used which 
>>> required a small modification to pygccxml) and decoration was also 
>>> noticeably faster (~1:40min) because there were less declarations to 
>>> consider (as already mentioned in my earlier mail).
>> Is this similar to the code I have in goodies that customizes the module 
>> builder calls to cache the module builder's state?  I guess what I am 
>> asking, is will your changes replace the customizations I am doing in 
>> goodies?
> 
> I'm not exactly sure which customizations you mean, but I don't think 
> I'm interfering with your customizations. The only modification to Py++ 
> that I needed is already checked in (the modification to pygccxml so 
> that XML files get cached). The actual pruning happens outside Py++ and 
> doesn't need any further customization.
> 
> And for the query cache the only customization I needed was to override 
> the constructor of declaration_t which I can do within my own code so 
> it's not necessary to modify Py++ itself.

Ok. The modification I did in goodies was to override the module builder.

I as not worried at all about interference, I was actually hoping that 
what you were doing would find it's way into Py++ so I could get rid of 
my extensions.  I would like to see a way that everyone can see the 
additional performance of better caching.  I hope to try out some of 
your performance improvements soon.  I can always find a need for better 
performance. :)

Thanks for taking the time to look into this and improve the project.

-Allen

[pygccxml-development] FT - recent changes

[Roman Y. <rom...@gm...>]

Hi. I committed few small changes to FT feature.

The main one is the interface change:

Now in order to add new transformation user have to write:

f.add_transformation( output(1), output(2) )

f.transformations property provides access to all function transformations that
should be applied.

The change was necessary because I want the way to apply few function
transformations on a single function.

For example:

struct error_t{};

void do_smth( ..., error_t& );

I want to expose do_smth in 2 ways:

1. as is: def( "do_smth_no_raise", &do_smth );
2. error status is converted by Py++ to exception:

void do_smth_raise(){
    error_t x;
    do_smth( x );
    if x contains error:
       throw actual error;
}

def( "do_smth", &do_smth_raise );


do_smth.add_transformation() #<- use Py++ default behaviour
do_smth.add_transformation( raise_on_error( 0, ... ) )


-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Another way to speed up the generation process...

[Roman Y. <rom...@gm...>]

On 10/24/06, Matthias Baas <ba...@ir...> wrote:
> Roman Yakovenko wrote:
> > I don't have enough time to analize what you did, but take a look on next
> > document:
> > http://gccxml.org/HTML/Running.html
> > Search for -fxml-start=
>
> Ah, I wasn't aware of that option. Thanks for the hint! It's almost what
> I did with the difference that I was filtering based on the header file
> that a declaration appears in.
> I did a quick test and passed all Maya classes to the xml-start option
> (the SDK doesn't use namespaces). It almost worked except that friend
> functions were not picked up. But I suppose it would have been possible
> to stick everything in a namespace of my own and use that one as a
> starting declaration.

Also I am aware of the option it never worked for me and I never had
enough time
to find out why.

May be you can investigate this and create small explanation to "best practices"
section:
http://language-binding.net/pyplusplus/documentation/best_practices.html ?

Thanks.
-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Another way to speed up the generation process...

[Matthias B. <ba...@ir...>]

Allen Bierbaum wrote:
>> What I did were two things: pruning the declaration tree and caching 
>> query operations. I don't use the pruning function from my earlier 
>> mail anymore because this pruning is only done after the header files 
>> were already parsed and stored in the cache. So while the function 
>> could speed up the decoration stage it could not speed up the parsing 
>> stage (which took more than a minute on OSX using the regular cache). 
>> So now I prune the declarations at an earlier stage, namely right 
>> after the XML file was created by gccxml and before pygccxml reads it. 
>> I wrote a little utility that directly operates on the XML file and 
>> outputs a pruned XML file which I use as input for Py++ (I think I'll 
>> put it into contrib eventually). This should now even be safer than 
>> before because I also keep all dependencies even when they are not 
>> inside any of the allowed headers. With the pruned tree alone I am 
>> already at the above parsing time (when the cache is used which 
>> required a small modification to pygccxml) and decoration was also 
>> noticeably faster (~1:40min) because there were less declarations to 
>> consider (as already mentioned in my earlier mail).
> 
> Is this similar to the code I have in goodies that customizes the module 
> builder calls to cache the module builder's state?  I guess what I am 
> asking, is will your changes replace the customizations I am doing in 
> goodies?

I'm not exactly sure which customizations you mean, but I don't think 
I'm interfering with your customizations. The only modification to Py++ 
that I needed is already checked in (the modification to pygccxml so 
that XML files get cached). The actual pruning happens outside Py++ and 
doesn't need any further customization.

And for the query cache the only customization I needed was to override 
the constructor of declaration_t which I can do within my own code so 
it's not necessary to modify Py++ itself.

- Matthias -

Re: [pygccxml-development] Another way to speed up the generation process...

[Allen B. <al...@vr...>]

Matthias Baas wrote:
> Hi,
> 
> I spent a little more time on reducing the execution time of the code 
> generation process and now I'm below 1 minute on both, Linux and OSX. :)
> 
> Linux: 59s (previously 4:40 minutes)
> OSX: 44s (previously 7:20 minutes)
> 
> Here are the timings of the individual steps on OSX:
> 
> Parsing: 3s
> Decoration: 13s
> Building code creators: 14s
> Writing files: 14s
> 
> (this is a run where all caches could be fully utilized and where no 
> file had to be written)
> 
> What I did were two things: pruning the declaration tree and caching 
> query operations. I don't use the pruning function from my earlier mail 
> anymore because this pruning is only done after the header files were 
> already parsed and stored in the cache. So while the function could 
> speed up the decoration stage it could not speed up the parsing stage 
> (which took more than a minute on OSX using the regular cache). So now I 
> prune the declarations at an earlier stage, namely right after the XML 
> file was created by gccxml and before pygccxml reads it. I wrote a 
> little utility that directly operates on the XML file and outputs a 
> pruned XML file which I use as input for Py++ (I think I'll put it into 
> contrib eventually). This should now even be safer than before because I 
> also keep all dependencies even when they are not inside any of the 
> allowed headers. With the pruned tree alone I am already at the above 
> parsing time (when the cache is used which required a small modification 
> to pygccxml) and decoration was also noticeably faster (~1:40min) 
> because there were less declarations to consider (as already mentioned 
> in my earlier mail).

Is this similar to the code I have in goodies that customizes the module 
builder calls to cache the module builder's state?  I guess what I am 
asking, is will your changes replace the customizations I am doing in 
goodies?

-Allen

> The next step in speeding things up was to cache the query operations. I 
> did an experimental implementation in pypp_api that works in conjunction 
> with the regular cache. Using this query cache, I get the above results 
> and the decoration stage is no longer the bottleneck in my case.
> 
> I'm afraid any further optimizations require quite a bit of 
> restructuring inside Py++. Running the profiler on the last two stages 
> (building code creators and writing files) reports the following "hot 
> spots" (the list is sorted by total time):
> 
>           26608350 function calls (26440374 primitive calls) in 341.650 
> CPU seconds
> 
>     Ordered by: internal time
>     List reduced from 856 to 10 due to restriction <10>
> 
>     ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>    9370513   60.550    0.000   60.550    0.000 :0(isinstance)
>    4574860   53.200    0.000   82.480    0.000 
> decl_wrappers/algorithm.py:87(<lambda>)
> 61199/49370   36.710    0.001  194.330    0.004 :0(filter)
>     948954   20.230    0.000   35.240    0.000 
> type_traits.py:42(remove_alias)
>     320939    7.480    0.000   23.030    0.000 matchers.py:205(__call__)
>     211696    6.120    0.000   12.180    0.000 matchers.py:224(check_name)
> 398580/387050    5.300    0.000    8.350    0.000 
> code_creators/algorithm.py:42(proceed_single)
>     369799    5.190    0.000   13.540    0.000 
> code_creators/algorithm.py:40(make_flatten_generator)
>     105595    4.990    0.000    8.060    0.000 
> class_declaration.py:105(_get_name_impl)
>     102200    4.790    0.000   36.450    0.000 
> container_traits.py:40(get_container_or_none)
> 
> 
> - Matthias -
> 
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> pygccxml-development mailing list
> pyg...@li...
> https://lists.sourceforge.net/lists/listinfo/pygccxml-development
> 

Re: [pygccxml-development] Another way to speed up the generation process...

[Matthias B. <ba...@ir...>]

Roman Yakovenko wrote:
> I don't have enough time to analize what you did, but take a look on next
> document:
> http://gccxml.org/HTML/Running.html
> Search for -fxml-start=

Ah, I wasn't aware of that option. Thanks for the hint! It's almost what 
I did with the difference that I was filtering based on the header file 
that a declaration appears in.
I did a quick test and passed all Maya classes to the xml-start option 
(the SDK doesn't use namespaces). It almost worked except that friend 
functions were not picked up. But I suppose it would have been possible 
to stick everything in a namespace of my own and use that one as a 
starting declaration.

- Matthias -

Re: [pygccxml-development] Modifying source_reader_t.read_xml_file()

[Roman Y. <rom...@gm...>]

On 10/24/06, Matthias Baas <ba...@ir...> wrote:
> Hi,
>
> the optimizations mentioned in my previous mail depend on a small
> modification to pygccxml, namely that the result of parsing XML files is
> also stored in the cache (otherwise parsing the file takes longer and my
> query cache wouldn't work at all). So I'm suggesting an updated version
> of source_reader_t.read_xml_file() that makes use the cache (if the user
> wouldn't want to cache the file he would not specify a cache in the
> first place, would he?):
>
>
>      def read_xml_file(self, gccxml_created_file):
>          """
>          Reads GCC-XML generated XML file.
>
>          @param gccxml_created_file: path to GCC-XML generated file
>          @type gccxml_created_file: str
>
>          @return: declarations tree
>          """
>          self.logger.debug( "Reading xml file: [%s]" % gccxml_created_file )
>
>          assert(self.__config!=None)
>          declarations = self.__dcache.cached_value( gccxml_created_file,
> self.__config )
>          if not declarations:
>              self.logger.debug( "File has not been found in cache,
> parsing..." )
>              declarations, files = self.__parse_gccxml_created_file(
> gccxml_created_file )
>              self.__dcache.update( gccxml_created_file, self.__config,
> declarations, [] )
>          else:
>              self.logger.debug( "File has not been changed, reading
> declarations from cache." )
>
>          return declarations
>
>
> So, Roman, is it ok when I commit this modification?

Yes if you add small changes to it: please use full file name to
update and read from
cache.

Thanks.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Another way to speed up the generation process...

[Roman Y. <rom...@gm...>]

On 10/24/06, Matthias Baas <ba...@ir...> wrote:
> Hi,
>
> I spent a little more time on reducing the execution time of the code
> generation process and now I'm below 1 minute on both, Linux and OSX. :)
>
> Linux: 59s (previously 4:40 minutes)
> OSX: 44s (previously 7:20 minutes)
>
> Here are the timings of the individual steps on OSX:
>
> Parsing: 3s
> Decoration: 13s
> Building code creators: 14s
> Writing files: 14s
>
> (this is a run where all caches could be fully utilized and where no
> file had to be written)
>
> What I did were two things: pruning the declaration tree and caching
> query operations. I don't use the pruning function from my earlier mail
> anymore because this pruning is only done after the header files were
> already parsed and stored in the cache. So while the function could
> speed up the decoration stage it could not speed up the parsing stage
> (which took more than a minute on OSX using the regular cache). So now I
> prune the declarations at an earlier stage, namely right after the XML
> file was created by gccxml and before pygccxml reads it. I wrote a
> little utility that directly operates on the XML file and outputs a
> pruned XML file which I use as input for Py++ (I think I'll put it into
> contrib eventually).

I don't have enough time to analize what you did, but take a look on next
document:
http://gccxml.org/HTML/Running.html
Search for -fxml-start=

http://language-binding.net/pyplusplus/documentation/apidocs/pygccxml.parser.config.config_t-class.html#start_with_declarations
start_with_declarations is a relevant argument.

So may be you can drop your utility.

>This should now even be safer than before because I
> also keep all dependencies even when they are not inside any of the
> allowed headers. With the pruned tree alone I am already at the above
> parsing time (when the cache is used which required a small modification
> to pygccxml) and decoration was also noticeably faster (~1:40min)
> because there were less declarations to consider (as already mentioned
> in my earlier mail).
> The next step in speeding things up was to cache the query operations. I
> did an experimental implementation in pypp_api that works in conjunction
> with the regular cache. Using this query cache, I get the above results
> and the decoration stage is no longer the bottleneck in my case.
>
> I'm afraid any further optimizations require quite a bit of
> restructuring inside Py++. Running the profiler on the last two stages
> (building code creators and writing files) reports the following "hot
> spots" (the list is sorted by total time):
>
>           26608350 function calls (26440374 primitive calls) in 341.650
> CPU seconds
>
>     Ordered by: internal time
>     List reduced from 856 to 10 due to restriction <10>
>
>     ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>    9370513   60.550    0.000   60.550    0.000 :0(isinstance)
>    4574860   53.200    0.000   82.480    0.000
> decl_wrappers/algorithm.py:87(<lambda>)
> 61199/49370   36.710    0.001  194.330    0.004 :0(filter)
>     948954   20.230    0.000   35.240    0.000
> type_traits.py:42(remove_alias)
>     320939    7.480    0.000   23.030    0.000 matchers.py:205(__call__)
>     211696    6.120    0.000   12.180    0.000 matchers.py:224(check_name)
> 398580/387050    5.300    0.000    8.350    0.000
> code_creators/algorithm.py:42(proceed_single)
>     369799    5.190    0.000   13.540    0.000
> code_creators/algorithm.py:40(make_flatten_generator)
>     105595    4.990    0.000    8.060    0.000
> class_declaration.py:105(_get_name_impl)
>     102200    4.790    0.000   36.450    0.000
> container_traits.py:40(get_container_or_none)
>
>
> - Matthias -
>
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> pygccxml-development mailing list
> pyg...@li...
> https://lists.sourceforge.net/lists/listinfo/pygccxml-development
>


-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

[pygccxml-development] Modifying source_reader_t.read_xml_file()

[Matthias B. <ba...@ir...>]

Hi,

the optimizations mentioned in my previous mail depend on a small 
modification to pygccxml, namely that the result of parsing XML files is 
also stored in the cache (otherwise parsing the file takes longer and my 
query cache wouldn't work at all). So I'm suggesting an updated version 
of source_reader_t.read_xml_file() that makes use the cache (if the user 
wouldn't want to cache the file he would not specify a cache in the 
first place, would he?):


     def read_xml_file(self, gccxml_created_file):
         """
         Reads GCC-XML generated XML file.

         @param gccxml_created_file: path to GCC-XML generated file
         @type gccxml_created_file: str

         @return: declarations tree
         """
         self.logger.debug( "Reading xml file: [%s]" % gccxml_created_file )

         assert(self.__config!=None)
         declarations = self.__dcache.cached_value( gccxml_created_file, 
self.__config )
         if not declarations:
             self.logger.debug( "File has not been found in cache, 
parsing..." )
             declarations, files = self.__parse_gccxml_created_file( 
gccxml_created_file )
             self.__dcache.update( gccxml_created_file, self.__config, 
declarations, [] )
         else:
             self.logger.debug( "File has not been changed, reading 
declarations from cache." )

         return declarations


So, Roman, is it ok when I commit this modification?

- Matthias -

Re: [pygccxml-development] Another way to speed up the generation process...

[Matthias B. <ba...@ir...>]

Hi,

I spent a little more time on reducing the execution time of the code 
generation process and now I'm below 1 minute on both, Linux and OSX. :)

Linux: 59s (previously 4:40 minutes)
OSX: 44s (previously 7:20 minutes)

Here are the timings of the individual steps on OSX:

Parsing: 3s
Decoration: 13s
Building code creators: 14s
Writing files: 14s

(this is a run where all caches could be fully utilized and where no 
file had to be written)

What I did were two things: pruning the declaration tree and caching 
query operations. I don't use the pruning function from my earlier mail 
anymore because this pruning is only done after the header files were 
already parsed and stored in the cache. So while the function could 
speed up the decoration stage it could not speed up the parsing stage 
(which took more than a minute on OSX using the regular cache). So now I 
prune the declarations at an earlier stage, namely right after the XML 
file was created by gccxml and before pygccxml reads it. I wrote a 
little utility that directly operates on the XML file and outputs a 
pruned XML file which I use as input for Py++ (I think I'll put it into 
contrib eventually). This should now even be safer than before because I 
also keep all dependencies even when they are not inside any of the 
allowed headers. With the pruned tree alone I am already at the above 
parsing time (when the cache is used which required a small modification 
to pygccxml) and decoration was also noticeably faster (~1:40min) 
because there were less declarations to consider (as already mentioned 
in my earlier mail).
The next step in speeding things up was to cache the query operations. I 
did an experimental implementation in pypp_api that works in conjunction 
with the regular cache. Using this query cache, I get the above results 
and the decoration stage is no longer the bottleneck in my case.

I'm afraid any further optimizations require quite a bit of 
restructuring inside Py++. Running the profiler on the last two stages 
(building code creators and writing files) reports the following "hot 
spots" (the list is sorted by total time):

          26608350 function calls (26440374 primitive calls) in 341.650 
CPU seconds

    Ordered by: internal time
    List reduced from 856 to 10 due to restriction <10>

    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   9370513   60.550    0.000   60.550    0.000 :0(isinstance)
   4574860   53.200    0.000   82.480    0.000 
decl_wrappers/algorithm.py:87(<lambda>)
61199/49370   36.710    0.001  194.330    0.004 :0(filter)
    948954   20.230    0.000   35.240    0.000 
type_traits.py:42(remove_alias)
    320939    7.480    0.000   23.030    0.000 matchers.py:205(__call__)
    211696    6.120    0.000   12.180    0.000 matchers.py:224(check_name)
398580/387050    5.300    0.000    8.350    0.000 
code_creators/algorithm.py:42(proceed_single)
    369799    5.190    0.000   13.540    0.000 
code_creators/algorithm.py:40(make_flatten_generator)
    105595    4.990    0.000    8.060    0.000 
class_declaration.py:105(_get_name_impl)
    102200    4.790    0.000   36.450    0.000 
container_traits.py:40(get_container_or_none)


- Matthias -

Re: [pygccxml-development] Problems with reading XML files directly

[Roman Y. <rom...@gm...>]

On 10/22/06, Brian OBrien <bob...@ya...> wrote:
> I have a c++ class for reading and parsing xml data...
> Interested?

I am not sure. First I don't know what additional value your class will provide.
And second even if it x20 faster than built-in Python parser( I am
sure it is :- )
it will introduce additional dependencies for the project. This is a
huge headache for
the users. Most of them drop an idea to use Py++ because of this.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Problems with reading XML files directly

[Brian O. <bob...@ya...>]

I have a c++ class for reading and parsing xml data...
Interested?

--- Roman Yakovenko <rom...@gm...> wrote:

> On 10/22/06, Matthias Baas <ba...@ir...> wrote:
> > Roman Yakovenko wrote:
> > >> I just noticed that when I read a XML file
> directly (by using
> > >> create_gccxml_fc()), Py++ creates an include
> directive where it tries to
> > >> include the XML file (which, of course, won't
> compile).
> > >
> > > What reader do you use? project_reader_t or
> source_reader_t?
> >
> > I'm using the project_reader_t class. But I found
> the mistake, it was my
> > fault, sorry. I forgot that I was passing a list
> to creator_t.create().
> > By the way, in create() there's the following 'if'
> statement right in
> > the beginning:
> >
> >     if not decl_headers:
> >         self._create_includes()
> >
> > Could the expression be changed from "not
> decl_headers" to "decl_headers
> > is None"? Because when I pass an empty list to
> disable the creation of
> > include files the expression "not decl_headers" is
> True and the includes
> > get generated anyway.
> 
> Go ahead :-)
> 
> -- 
> Roman Yakovenko
> C++ Python language binding
> http://www.language-binding.net/
> 
>
-------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support
> web services, security?
> Get stuff done quickly with pre-integrated
> technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1
> based on Apache Geronimo
>
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> pygccxml-development mailing list
> pyg...@li...
>
https://lists.sourceforge.net/lists/listinfo/pygccxml-development
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: [pygccxml-development] Problems with reading XML files directly

[Roman Y. <rom...@gm...>]

On 10/22/06, Matthias Baas <ba...@ir...> wrote:
> Roman Yakovenko wrote:
> >> I just noticed that when I read a XML file directly (by using
> >> create_gccxml_fc()), Py++ creates an include directive where it tries to
> >> include the XML file (which, of course, won't compile).
> >
> > What reader do you use? project_reader_t or source_reader_t?
>
> I'm using the project_reader_t class. But I found the mistake, it was my
> fault, sorry. I forgot that I was passing a list to creator_t.create().
> By the way, in create() there's the following 'if' statement right in
> the beginning:
>
>     if not decl_headers:
>         self._create_includes()
>
> Could the expression be changed from "not decl_headers" to "decl_headers
> is None"? Because when I pass an empty list to disable the creation of
> include files the expression "not decl_headers" is True and the includes
> get generated anyway.

Go ahead :-)

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] Problems with reading XML files directly

[Matthias B. <ba...@ir...>]

Roman Yakovenko wrote:
>> I just noticed that when I read a XML file directly (by using
>> create_gccxml_fc()), Py++ creates an include directive where it tries to
>> include the XML file (which, of course, won't compile).
> 
> What reader do you use? project_reader_t or source_reader_t?

I'm using the project_reader_t class. But I found the mistake, it was my 
fault, sorry. I forgot that I was passing a list to creator_t.create().
By the way, in create() there's the following 'if' statement right in 
the beginning:

    if not decl_headers:
        self._create_includes()

Could the expression be changed from "not decl_headers" to "decl_headers 
is None"? Because when I pass an empty list to disable the creation of 
include files the expression "not decl_headers" is True and the includes 
get generated anyway.

>> Another nuisance/bug is that the cache isn't used for XML files, so they
>> get parsed every time.
> 
> By design. XML file is already a cache. It uses slightly ( :-) )
> different format than cPickle.
> For my projects the difference between XML and cPickle is not that big.

In my case, it's 20s for parsing the XML file vs 7s for reading the 
cache. So I would prefer the cache. :)

- Matthias -

Re: [pygccxml-development] Problems with reading XML files directly

[Roman Y. <rom...@gm...>]

On 10/22/06, Matthias Baas <ba...@ir...> wrote:
> Hi,
>
> I just noticed that when I read a XML file directly (by using
> create_gccxml_fc()), Py++ creates an include directive where it tries to
> include the XML file (which, of course, won't compile).

What reader do you use? project_reader_t or source_reader_t?

If first one, than please submit small test case that reproduces the problem.
If second one, than it has special function to read xml generated
files: read_xml_file

> Another nuisance/bug is that the cache isn't used for XML files, so they
> get parsed every time.

By design. XML file is already a cache. It uses slightly ( :-) )
different format than cPickle.
For my projects the difference between XML and cPickle is not that big.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

[pygccxml-development] Problems with reading XML files directly

[Matthias B. <ba...@ir...>]

Hi,

I just noticed that when I read a XML file directly (by using 
create_gccxml_fc()), Py++ creates an include directive where it tries to 
include the XML file (which, of course, won't compile).

Another nuisance/bug is that the cache isn't used for XML files, so they 
get parsed every time.

- Matthias -

Re: [pygccxml-development] Another way to speed up the generation process...

[Matthias B. <ba...@ir...>]

Roman Yakovenko wrote:
> I appreciate your efforts to speed up cod generation process. This
> specific patch
> will not find its way to Py++ :-(. It is too dangerous:

It's not meant to be a patch anyway. I just posted the function so that 
anyone who thinks it is useful can incorporate it into their generation 
scripts (in my case, I made it available via pypp_api (but it still has 
to be called by the user explicitly, it's not called automatically by 
pypp_api)).

> While remove_declarations works as you expect, the functionality is broken.
> For example: base and derived classes. if you remove base class from the
> declaration tree, base class still will be accessible from the derived 
> "bases"
> property. Or if some function takes as argument instance of some class
> and you remove the class from the declaration tree, the type of argument 
> will continue to keep reference to the class.

Looks like those kind of problems do not appear in my case. I'm not 
aware that the Maya SDK uses any class from outside the SDK. The 
generated code after pruning is the same as before, so I guess it's ok 
for me to take advantage of the pruning. :)

So the feature I'd like to see eventually is just that I can tell Py++ 
in advance that I will only be querying (and exposing) declarations from 
a particular set of header files and that all other declarations are not 
interesting to me and can be safely ignored for any query operation and 
they also don't have to be stored in the cache (unfortunately, I don't 
have the latter yet). Then it's still up to Py++ if it has to keep some 
of the declarations but at least I could expect that they won't slow 
down a query operation anymore and that the cache is kept at a 
reasonable size. As mentioned in my previous post, under OSX I'm only 
interested in about 1.5% of the declarations in the top-level namespace. 
Everything else is junk that just slows down the generation process.
Reading the cache still takes about 1:10 minutes (about 1/3 of the 
entire execution time) while parsing the code without cache takes about 
1:45 minutes (on Linux it is ~7s vs ~1:30min).

> Can you post your profiler results? May be we can find something that
> could\should be fixed.

Sorry, I didn't use a profiler. I was just using the regular output of 
the driver script.

- Matthias -