From: <rom...@us...> - 2008-01-05 20:47:40
|
Revision: 1218 http://pygccxml.svn.sourceforge.net/pygccxml/?rev=1218&view=rev Author: roman_yakovenko Date: 2008-01-05 12:47:42 -0800 (Sat, 05 Jan 2008) Log Message: ----------- improving docs Added Paths: ----------- pyplusplus_dev/docs/documentation/how_to/ pyplusplus_dev/docs/documentation/how_to/best_practices.rest pyplusplus_dev/docs/documentation/how_to/exception_translation.rest pyplusplus_dev/docs/documentation/how_to/hints.rest pyplusplus_dev/docs/documentation/how_to/how_to.rest pyplusplus_dev/docs/documentation/how_to/templates.rest pyplusplus_dev/docs/documentation/how_to/www_configuration.py Added: pyplusplus_dev/docs/documentation/how_to/best_practices.rest =================================================================== --- pyplusplus_dev/docs/documentation/how_to/best_practices.rest (rev 0) +++ pyplusplus_dev/docs/documentation/how_to/best_practices.rest 2008-01-05 20:47:42 UTC (rev 1218) @@ -0,0 +1,176 @@ +============== +Best practices +============== + +.. contents:: Table of contents + +------------ +Introduction +------------ + +`Py++`_ has reach interface and a lot of functionality. Sometimes reach +interface helps, but sometimes it can confuse. This document will describe how +effectively to use `Py++`_. + +------------ +Big projects +------------ + +Definition +---------- + +First of all, let me to define "big project". "Big project" is a project with +few hundred of header files. `Py++`_ was born to create `Python`_ bindings +for such projects. If you take a look `here`__ you will find few such projects. + +.. __ : ./../../pyplusplus/quotes.html + +Tips +---- + +* Create one header file, which will include all project header files. + + Doing it this way makes it so `GCC-XML`_ is only called once and it reduces the + overhead that would occur if you pass `GCC-XML`_ all the files individually. + Namely `GCC-XML`_ would have to run hundreds of times and each call would + actually end up including quite a bit of common code anyway. This way takes a + `GCC-XML`_ processing time from multiple hours with gigabytes of caches to a + couple minutes with a reasonable cache size. + + You can read more about different caches supported by `pygccxml`_ `here`__. + ``module_builder_t.__init__`` method takes reference to an instance of cache + class or ``None``: + + .. code-block:: Python + + from module_builder import * + mb = module_builder_t( ..., cache=file_cache_t( path to project cache file ), ... ) + +* Single header file, will also improve performance compiling the generated bindings. + + When `Py++`_ generated the bindings, you have a lot of .cpp files to + compile. The project you are working on is big. I am sure it takes a lot of + time to compile projects that depend on it. Generated code also depend on it, + more over this code contains a lot of template instantiations. So it could + take a great deal of time to compile it. Allen Bierbaum investigated this + problem. He found out that most of the time is really spent processing all the + headers, templates, macros from the project and from the boost library. So he + come to conclusion, that in order to improve compilation speed, user should + be able to control( to be able to generate ) precompiled header file. He + implemented an initial version of the functionality. After small discussion, + we agreed on next interface: + + .. code-block:: Python + + class module_builder_t( ... ): + ... + def split_module( self, directory_path, huge_classes=None, precompiled_header=None ): + ... + + ``precompiled_header`` argument could be ``None`` or string, that contains + name of precompiled header file, which will be created in the directory. + `Py++`_ will add to it header files from `Boost.Python`_ library and + your header files. + + What is ``huge_classes`` argument for? ``huge_classes`` could be ``None`` or + list of references to class declarations. It is there to provide a solution to + `this error`_. `Py++`_ will automatically split generated code for the + huge classes to few files: + + .. code-block:: Python + + mb = module_builder_t( ... ) + ... + my_big_class = mb.class_( my_big_class ) + mb.split_module( ..., huge_classes=[my_big_class], ... ) + + * **Caveats** + + Consider next file layout: + :: + + boost/ + date_time/ + ptime.hpp + time_duration.hpp + date_time.hpp //main header, which include all other header files + + Py++ currently does not handle relative paths as input very well, so it is + recommended that you use "os.path.abspath()" to transform the header file to + be processed into an absolute path: + + .. code-block:: Python + + #Next code will expose nothing + mb = module_builder( [ 'date_time/date_time.hpp' ], ... ) + + #while this one will work as expected + import os + mb = module_builder( [ os.path.abspath('date_time/date_time.hpp') ], ... ) + +.. _`this error` : http://boost.org/libs/python/doc/v2/faq.html#c1204 +.. __ : ./../../pygccxml/design.html + +* Keep the declaration tree small. + + When parsing the header files to build the declaration tree, there will also + be the occasional "junk" declaration inside the tree that is not relevant to + the bindings you want to generate. These extra declarations come from header + files that were included somewhere in the header files that you were actually + parsing (e.g. if that library uses the STL or OpenGL or other system headers + then the final declaration tree will contain those declarations, too). + It can happen that the majority of declarations in your declaration tree are + such "junk" declarations that are not required for generating your bindings + and that just slow down the generation process (reading the declaration cache + and doing queries will take longer). + + To speed up your generation process you might want to consider making the + declaration tree as small as possible and only store those declarations that + somehow have an influence on the bindings. Ideally, this is done as early + as possible and luckily gccxml provides an option that allows you to reduce + the number of declarations that it will store in the output XML file. You can + specify one or more declarations using the ``-fxml-start`` option and only + those sub-tree starting at the specified declarations will be written. For + example, if you specify the name of a particular class, only this class + and all its members will get written. Or if your project already uses + a dedicated namespace you can simply use this namespace as a starting point + and all declarations stemming from system headers will be ignored (except + for those declarations that are actually used within your library). + + In the ``pygccxml`` package you can set the value for the ``-fxml-start`` + option using the ``start_with_declarations`` attribute of the + ``pygccxml.parser.config_t`` object that you are passing to the parser. + +* Use `Py++`_ repository of generated files md5 sum. + + `Py++`_ is able to store md5 sum of generated files in a file. Next time you + will generate code, `Py++`_ will compare generated file content against the sum, + instead of loading the content of the previously generated file from the disk + and comparing against it. + + .. code-block:: Python + + mb = module_builder_t( ... ) + ... + my_big_class = mb.class_( my_big_class ) + mb.split_module( ..., use_files_sum_repository=True ) + + `Py++`_ will generate file named "<your module name>.md5.sum" in the directory + it will generate all the files. + + Enabling this functionality should give you 10-15% of performance boost. + + * **Caveats** + + If you changed manually some of the files - don't forget to delete the relevant + line from "md5.sum" file. You can also delete the whole file. If the file is + missing, `Py++`_ will use old plain method of comparing content of the files. + It will not re-write "unchanged" files and you will not be forced to recompile + the whole project. + + +.. _`Py++` : ./../pyplusplus.html +.. _`pygccxml` : ./../../pygccxml/pygccxml.html +.. _`Boost.Python`: http://www.boost.org/libs/python/doc/index.html +.. _`Python`: http://www.python.org +.. _`GCC-XML`: http://www.gccxml.org Added: pyplusplus_dev/docs/documentation/how_to/exception_translation.rest =================================================================== --- pyplusplus_dev/docs/documentation/how_to/exception_translation.rest (rev 0) +++ pyplusplus_dev/docs/documentation/how_to/exception_translation.rest 2008-01-05 20:47:42 UTC (rev 1218) @@ -0,0 +1,65 @@ +========================================= +How to register an exception translation? +========================================= + +.. contents:: Table of contents + +------------ +Introduction +------------ + +Boost.Python provides functionality to translate any C++ exception to a Python one. +`Py++`_ provides a convenient API to do this. + +By the way, be sure to take a look on "`troubleshooting guide - exceptions`_". +The guide will introduces a complete solution for handling exceptions within +Python scripts. + +.. _`troubleshooting guide - exceptions` : ./../../troubleshooting_guide/exceptions/exceptions.html + +-------- +Solution +-------- + +`Boost.Python exception translator documentation`_ contains a complete explanation +what should be done. I will use that example, to show how it could be done with +`Py++`_: + +.. code-block:: Python + + from pyplusplus import module_builder_t + + mb = module_builder_t( ... ) + my_exception = mb.class_( 'my_exception' ) + + translate_code = 'PyErr_SetString(PyExc_RuntimeError, exc.what();' + my_exception.exception_translation_code = translate_code + +That's all, really. `Py++`_ will generate for you the ``translate`` function +definition and than will register it. + +I think this is a most popular use case - translate a C++ exception to a string +and than to create an instance of Python built-in exception. That is exactly why +`Py++`_ provides additional API: + +.. code-block:: Python + + mb = module_builder_t( ... ) + my_exception = mb.class_( 'my_exception' ) + + my_exception.translate_exception_to_string( 'PyExc_RuntimeError', 'exc.what()') + +The first argument of ``translate_exception_to_string`` method is exception type, +The second one is a string - code that converts your exception to ``const char*``. + +As you see, it is really simple to add exception translation to your project. + +One more point, in both pieces of code I used "``exc``" as the name of ``my_exception`` +class instance. This is a predefined name. I am not going to change it without +any good reason, any time soon :-). + +.. _`Boost.Python exception translator documentation` : http://boost.org/libs/python/doc/v2/exception_translator.html +.. _`Py++` : ./../../pyplusplus.html +.. _`Boost.Python`: http://www.boost.org/libs/python/doc/index.html +.. _`Python`: http://www.python.org +.. _`GCC-XML`: http://www.gccxml.org Added: pyplusplus_dev/docs/documentation/how_to/hints.rest =================================================================== --- pyplusplus_dev/docs/documentation/how_to/hints.rest (rev 0) +++ pyplusplus_dev/docs/documentation/how_to/hints.rest 2008-01-05 20:47:42 UTC (rev 1218) @@ -0,0 +1,64 @@ +===== +Hints +===== + +.. contents:: Table of contents + +---------------------------------- +Class template instantiation alias +---------------------------------- + +`Py++`_ has nice feature. If you define ``typedef`` for instantiated class +template, than `Py++`_ will use it as a `Python`_ class name. + +For example: + +.. code-block:: C++ + + #include <vector> + typedef std::vector< int > numbers; + numbers generate_n(){ + ... + } + +`Py++`_ will use "numbers" as Python class name: + +.. code-block:: C++ + + using boost::python; + class_< std::vector< int > >( "numbers" ) + ... + ; + +`Py++`_ will pick up the alias, only in case the class has single "typedef". + +``pyplusplus::aliases`` namespace +--------------------------------- + +The previous approach is "implicit" - `Py++`_ does something behind the scene. +Recently (version 0.8.6 ), another approach was introduced: + +.. code-block:: C++ + + #include <vector> + + namespace pyplusplus{ namespace aliases{ + //^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + typedef std::vector< int > numbers; + + } } //pyplusplus::aliases + +The idea is that you create namespace with a special name - ``pyplusplus::aliases`` +and `Py++`_ automatically picks the class aliases from it. In case you accidentally +introduced two or more different aliases to the same class, it will pick the +longest one and print a warning. Other advantages of the approach: + +* you are not forced to learn new API + +* you continue to use your favorite editor and familiar language + +.. _`Py++` : ./../pyplusplus.html +.. _`Boost.Python`: http://www.boost.org/libs/python/doc/index.html +.. _`Python`: http://www.python.org +.. _`GCC-XML`: http://www.gccxml.org + Added: pyplusplus_dev/docs/documentation/how_to/how_to.rest =================================================================== --- pyplusplus_dev/docs/documentation/how_to/how_to.rest (rev 0) +++ pyplusplus_dev/docs/documentation/how_to/how_to.rest 2008-01-05 20:47:42 UTC (rev 1218) @@ -0,0 +1,175 @@ +============ +How to ... ? +============ + +.. contents:: Table of contents + +`How to deal with templates?`_ + +.. _`How to deal with templates?` : ./templates.html + +`How to add custom exception translation?`_ + +.. _ `How to add custom exception translation?` : custom_exception_translation.html + +------------------------------------------------------- +How to expose function, which has hand-written wrapper? +------------------------------------------------------- +.. code-block:: C++ + + struct window_t{ + ... + void get_size( int& height, int& widht ) const; + }; + +You can not expose ``get_size`` function as is - ``int`` is immutable type in +Python. So, we need to create a wrapper to the function: + +.. code-block:: C++ + + boost::python::tuple get_size_wrapper( const window_t& win ){ + int height(0), width( 0 ); + win.get_size( height, widht ); + return boost::python::make_tuple( height, width ); + } + +.. code-block:: C++ + + class_<window_t>( ... ) + .def( "get_size", &get_size_wrapper ) + ... + ; + +Now, after you know how this problem is solved. I will show how this solution +could be integrated with `Py++`_. + +.. code-block:: Python + + wrapper_code = \ + """ + static boost::python::tuple get_size( const window_t& win ){ + int height(0), width( 0 ); + win.get_size( height, width ); + return boost::python::make_tuple( height, width ); + } + """ + +.. code-block:: Python + + registration_code = 'def( "get_size", &%s::get_size )' % window.wrapper_alias + +.. code-block:: Python + + mb = module_builder_t( ... ) + window = mb.class_( "window_t" ) + window.member_function( "get_size" ).exclude() + window.add_wrapper_code( wrapper_code ) + window.registration_code( registration_code ) + +That's all. + +------------------------------------------------------------- +Fatal error C1204:Compiler limit: internal structure overflow +------------------------------------------------------------- + +If you get this error, than the generated file is too big. You will have to split +it to few files. Well, not you but `Py++`_, you will only have to tell it to do +that. + +If you are using ``module_builder_t.write_module`` method, consider to switch +to ``module_builder_t.split_module``. + +If you are using ``split_module``, but still the generated code for some class +could not be compiled, because of the error, you can ask `Py++`_ to split the +code generated for class to be split to few cpp files. + +For more information, please read the documentation. + + +------------------------------------- +Py++ generated file name is too long! +------------------------------------- +There are use cases, when `Py++`_ generated file name is too long. In some cases +the code generation process even fails because of this. What can you do in order +to eliminate the problem? + +First of all the problem arises when you expose template instantiated classes +and you did not set the class alias. + +Let me explain: + +.. code-block:: C++ + + template < class T> + struct holder{ ... }; + + +Lets say that you want to export ``holder< int >`` class. Class name in `Python`_ +has few `constraints`_. `Py++`_ is aware of the `constraints`_ and if you didn't +set an alias to the class, `Py++`_ will do it for you. In this case, +"holder_less_int_grate\_" is the generated alias. Obviously it is much longer +and not readable. + +.. _`constraints` : http://www.python.org/doc/current/ref/identifiers.html + +There are few pretty good reasons for this behavior: + +* when you just start to work on `Python`_ bindings concentrate your attention + on really important things + +* if you forgot to set the class alias your users still can use the class + functionality, however the class name will be a little bit ugly. + + +In this case the generate alias for ``holder`` instantiation is relatively short. +Imagine how long it could be for ``std::map`` instantiation. + +`Py++`_ uses class alias for the file name. So if you want to force `Py++`_ to +generate files with short name, you have to set class alias: + +.. code-block:: Python + + from pyplusplus import module_builder + + mb = module_builder_t( ... ) + holder = mb.class_( 'holder< int >' ) + holder.alias = 'IntHolder' + #next line has same effect as the previous one: + holder.rename( 'IntHolder' ) + +The nice thing about this approach is that now `Python`_ users have "normal" +class name and you have short file name. + +------------------- +Full\relative paths +------------------- + +Consider next file layout: +:: + + boost/ + date_time/ + ptime.hpp + time_duration.hpp + date_time.hpp //main header, which include all other header files + +Py++ currently does not handle relative paths as input very well, so it is +recommended that you use ``os.path.abspath()`` to transform the header file to +be processed into an absolute path: + +.. code-block:: Python + + #Next code will expose nothing + mb = module_builder( [ 'date_time/date_time.hpp' ], ... ) + mb.split_module( ... ) + + #while this one will work as expected + import os + mb = module_builder( [ os.path.abspath('date_time/date_time.hpp') ], ... ) + mb.split_module( ... ) + + +.. _`Py++` : ./../pyplusplus.html +.. _`Boost.Python`: http://www.boost.org/libs/python/doc/index.html +.. _`Python`: http://www.python.org +.. _`GCC-XML`: http://www.gccxml.org Added: pyplusplus_dev/docs/documentation/how_to/templates.rest =================================================================== --- pyplusplus_dev/docs/documentation/how_to/templates.rest (rev 0) +++ pyplusplus_dev/docs/documentation/how_to/templates.rest 2008-01-05 20:47:42 UTC (rev 1218) @@ -0,0 +1,196 @@ +=========================== +How to deal with templates? +=========================== + +.. contents:: Table of contents + +------------ +Introduction +------------ + +I would like to introduce next piece of code I will use for most exlanations. + +.. code-block:: C++ + + // file point.h + template< class T> + struct point_t{ + T x, y; + }; + + template <class T> + double distance( const point_t<T>& point ){ + return sqrt( point.x * point.x + point.y*point.y ); + } + + struct environment_t{ + ... + template< class T> + T get_value(const std::string& name); + ... + }; + +---------------------- +Template instantiation +---------------------- + +First of all you should understand, that you can not export template itself, but +only its instantiations. + +You can instantiate template class using operator ``sizeof``: + +.. code-block:: C++ + + sizeof( point_t<int> ); + + +In order to instantiate a function you have to call it: + +.. code-block:: C++ + + void instantiate(){ + double x = distance( point_t<t>() ); + + environment_t env; + std::string path = env.get_value< std::string >( "PATH" ); + int version = env.get_value< int >( "VERSION" ); + } + +You should put that code in some header file, parsed by GCC-XML. + +"Dynamic" instantiation +----------------------- +If you have a template class, which should be instantiated with many types, you +can create a small code generator, which will "instantiate the class". It is +pretty easy to blend together the generated code and the existing one: + +.. code-block:: Python + + from module_builder import module_builder_t, create_text_fc + + def generate_instantiations_string( ... ): + ... + + code = generate_instantiations_string( ... ) + + mb = module_builder_t( [ create_text_fc( code ), other header files ], ... ) + ... + +Function ``create_text_fc`` allows you to extract declarations from the string, +which contains valid C++ code. It creates temporal header file and compiles it. + +.. __ : ./../../../pygccxml/design.html#parser-configuration-classes + +---------------------------------- +Functions templated on return type +---------------------------------- + +.. code-block:: C++ + + environment_t env; + std::string path = env.get_value< std::string >( "PATH" ); + int version = env.get_value< int >( "VERSION" ); + + +`GCC-XML`_ provides information for both instantiations: + + * ``get_value<int>`` + + * ``get_value< std::string >`` + +But, in this case there is a catch: the name of both functions is "**get_value**". +The only difference is "return type". + +In this situation, `Py++`_ will generate code that contains errors. If your are +lucky, it depends on the compiler you use, the generated code will not compile. +Otherwise, you will discover the errors while testing the bindings. + +Generated code: + +.. code-block:: C++ + + bp::class_< environment_t >( "environment_t" ) + ... + .def( "get_value" + , (int ( ::environment_t::* )( ::std::string const & ) )( &::environment_t::get_value ) ) + .def( "get_value" + , (::std::string ( ::environment_t::* )( ::std::string const & ) )( &::environment_t::get_value ) ); + +The correct code: + +.. code-block:: C++ + + bp::class_< environment_t >( "environment_t" ) + .def( "get_value" + , (int ( ::environment_t::* )( ::std::string const & ) )( &::environment_t::get_value< int > ) ) + //--------------------------------------------------------------------------^^^^^^^^^^^^^^^^ + .def( "get_value" + , (::std::string ( ::environment_t::* )( ::std::string const & ) )( &::environment_t::get_value< std::string > ) ); + //------------------------------------------------------------------------------------^^^^^^^^^^^^^^^^^^^^^^^^ + +The perfect one: + +.. code-block:: C++ + + bp::class_< environment_t >( "environment_t" ) + ... + .def( "get_value", &::environment_t::get_value< int > ) + .def( "get_value", &::environment_t::get_value< std::string > ); + + +Work-around +----------- + +`Py++`_ contains a work-around to the problem: + +.. code-block:: Python + + mb = module_builder_t( ..., optimize_queries=False, ... ) + environment = mb.class_( "environment_t" ) + for f in environment.member_functions( "get_value" ): + #set the function alias + f.alias = f.name + "_" + f.return_type.decl_string + #correct function name + f.name = f.demangled_name + #you still want the queries to run fast + mb.run_query_optimizer() + +Before you read the rest of the solution, you should understand what is +"name mangling" means. If you don't, consider reading about it on `Wikipedia`__ . + +.. __ : http://en.wikipedia.org/wiki/Name_mangling + +The solution is pretty simple. `GCC-XML`_ reports mangled and demangled function +names. The demangled function name contains "real" function name: +``get_value< used type >``. You only have to instruct `Py++`_ to use it. + +`Py++`_ does not use by default demangled function name for mainly one reason. +Demangled function name is a string that contains a lot of information. `Py++`_ +implements a parser, which extracts the only relevant one. The parser +implementation is a little bit complex and was not heavily tested. By "heavily" I +mean that I tested it on a lot of crazy use cases and on a real project, but +there is always some new use case out there. I am almost sure it will work for +you. The problem, we deal with, is rare, so by default "demangled_name" +feature is turned off. + +By the way, almost the same problem exists for template classes. But, in the +classes use case `Py++`_ uses demangled name by default. + +----------- +Help wanted +----------- +I understand that the provided solutions are not perfect and that something +better and simpler should be done. Unfortunatelly the priority of this task is +low. + +Allen Bierbaum has few suggestion that could improve `Py++`_. He created a +`wiki page`_, that discuss possible solutions. Your contribution is welcome too! + +.. _`wiki page` : https://realityforge.vrsource.org/view/PyppApi/TemplateSupport + + + +.. _`Py++` : ./../pyplusplus.html +.. _`Boost.Python`: http://www.boost.org/libs/python/doc/index.html +.. _`Python`: http://www.python.org +.. _`GCC-XML`: http://www.gccxml.org Added: pyplusplus_dev/docs/documentation/how_to/www_configuration.py =================================================================== --- pyplusplus_dev/docs/documentation/how_to/www_configuration.py (rev 0) +++ pyplusplus_dev/docs/documentation/how_to/www_configuration.py 2008-01-05 20:47:42 UTC (rev 1218) @@ -0,0 +1,10 @@ +name = 'how to ... ?' +#main_html_file = 'index.html' + +names = { 'hints' : 'hints' + , 'how_to' : 'how to' + , 'templates' : 'deal with templates' + , 'best_practices' : 'best practices' + , 'exception_translation' : 'exception translation' +} + This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |