pygccxml-development

[pygccxml-development] [ANN] Py++ based ctypes code generator

[Roman Y. <rom...@gm...>]

Hello.

I would like to announce a new code generator for ctypes package.

The new code generator is integrated with Py++ package and reuse most
of its infrastructure:
* GCC-XML - generates an XML description of a C++ program from GCC's
internal representation
* pygccxml - powerful and simple Python package to work GCCXML generated file.
* Py++ design, code and utilities
* It was build with future ( ctypes-cpp ) functionality in mind.
* It has cross-platform ( Windows XP and Linux Ubuntu 8.10 ) unit tests

How it works:

1. Using pygccxml, the source files are parsed ( using pygccxml ) and
in-memory description of the code is created
2. .dll, .map or .so file is parsed ( pygccxml ) and exported symbols
list is created
3. Using decorated and undecorated names, mapping between declarations
and exported symbols is created.
4. The exported symbols and the mapping are used to:
   * define for what declarations the code should be generated
   * find out function calling convention
5. The last step - the code is generated

Supported functionality:

* shared libraries, that were compiled using C or C++ compiler
* free functions, including variable number of arguments
* [global|member] [named|anonymous] variables
* [named|anonymous|opaque|nested]  structures
* enumerations
* typedefs

Examples:

    Is it useful? - Yes. I created example, which generates code for
GMP( http://gmplib.org/ ) library. The following link (
http://pygccxml.svn.sourceforge.net/viewvc/pygccxml/pyplusplus_dev/examples/gmplib_dev/
) will take you to the example.
    * dev - directory, which contains code generator source code
    * pygmplib - directory, which contains generated code - Python
package, which provides access to GMP lib functionality
    * test.py - Python script - tester.

Want to try?
    Windows users:
        http://sourceforge.net/project/showfiles.php?group_id=118209
contains few packages
        * ctypes code generator - stand alone executable, which
generates code for a .dll file
        * gccxml-setup - gccxml setup file
    Linux users:
        Sorry, no binaries :-(
        * you need to check-out( http://gccxml.org/HTML/Download.html
) and build gccxml ( http://gccxml.org/HTML/Install.html )
        * you need to check-out Py++ and pygccxml (
http://sourceforge.net/svn/?group_id=118209 )
        * wrap_library.py (
http://pygccxml.svn.sourceforge.net/viewvc/pygccxml/pyplusplus_dev/scripts/
) is the script, which generates code for .so file.

Your feedback is welcome.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] [ANN] Py++ based ctypes code generator

[Gustavo C. <gjc...@gm...>]

2009/1/20 Roman Yakovenko <rom...@gm...>

> Hello.
>
> I would like to announce a new code generator for ctypes package.
>
> The new code generator is integrated with Py++ package and reuse most
> of its infrastructure:
> * GCC-XML - generates an XML description of a C++ program from GCC's
> internal representation
> * pygccxml - powerful and simple Python package to work GCCXML generated
> file.
> * Py++ design, code and utilities
> * It was build with future ( ctypes-cpp ) functionality in mind.
> * It has cross-platform ( Windows XP and Linux Ubuntu 8.10 ) unit tests
>
> How it works:
>
> 1. Using pygccxml, the source files are parsed ( using pygccxml ) and
> in-memory description of the code is created
> 2. .dll, .map or .so file is parsed ( pygccxml ) and exported symbols
> list is created
> 3. Using decorated and undecorated names, mapping between declarations
> and exported symbols is created.
> 4. The exported symbols and the mapping are used to:
>   * define for what declarations the code should be generated
>   * find out function calling convention
> 5. The last step - the code is generated
>
> Supported functionality:
>
> * shared libraries, that were compiled using C or C++ compiler
> * free functions, including variable number of arguments
> * [global|member] [named|anonymous] variables
> * [named|anonymous|opaque|nested]  structures
> * enumerations
> * typedefs
>
> Examples:
>
>    Is it useful? - Yes. I created example, which generates code for
> GMP( http://gmplib.org/ ) library. The following link (
>
> http://pygccxml.svn.sourceforge.net/viewvc/pygccxml/pyplusplus_dev/examples/gmplib_dev/
> ) will take you to the example.
>    * dev - directory, which contains code generator source code
>    * pygmplib - directory, which contains generated code - Python
> package, which provides access to GMP lib functionality
>    * test.py - Python script - tester.
>
> Want to try?
>    Windows users:
>        http://sourceforge.net/project/showfiles.php?group_id=118209
> contains few packages
>        * ctypes code generator - stand alone executable, which
> generates code for a .dll file
>        * gccxml-setup - gccxml setup file
>    Linux users:
>        Sorry, no binaries :-(
>        * you need to check-out( http://gccxml.org/HTML/Download.html
> ) and build gccxml ( http://gccxml.org/HTML/Install.html )
>        * you need to check-out Py++ and pygccxml (
> http://sourceforge.net/svn/?group_id=118209 )
>        * wrap_library.py (
> http://pygccxml.svn.sourceforge.net/viewvc/pygccxml/pyplusplus_dev/scripts/
> ) is the script, which generates code for .so file.
>
> Your feedback is welcome.


This is potentially very useful!  Generated Python code can be shipped with
any project and is (almost) platform independent without need for
compilation; you just run Python code.

However I have to wonder how (if) do you support C++ calling conventions.
Things like:
   - How to pass the 'this' parameter to call methods;
   - Function/method overloading.

I have followed the path of C++ code generation precisely because of the
difficulty of calling C++ code via libffi, or else I would probably have
contributed to the gobject-introspection and pybank projects.  I thought
ctypes did not support C++?

Cheers,

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert

Re: [pygccxml-development] [ANN] Py++ based ctypes code generator

[Roman Y. <rom...@gm...>]

On Tue, Jan 20, 2009 at 9:22 PM, Gustavo Carneiro <gjc...@gm...> wrote:
> This is potentially very useful!  Generated Python code can be shipped with
> any project and is (almost) platform independent without need for
> compilation; you just run Python code.

This is my goal :-)

> However I have to wonder how (if) do you support C++ calling conventions.

I think, I was not clear. You can compile C code using C and/or C++
compiler. Py++ supports both modes. In the second mode the calling
convention is same as in C, but name mangling is different

> Things like:
>    - How to pass the 'this' parameter to call methods;
>    - Function/method overloading.
>
> I have followed the path of C++ code generation precisely because of the
> difficulty of calling C++ code via libffi, or else I would probably have
> contributed to the gobject-introspection and pybank projects.  I thought
> ctypes did not support C++?

Just read few mails, with ctypes-cpp in the subject:
http://sourceforge.net/mailarchive/forum.php?forum_name=ctypes-users

and take a look on easybmp example(
http://pygccxml.svn.sourceforge.net/viewvc/pygccxml/pyplusplus_dev/examples/pyeasybmp_dev/ctypes/
)

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] [ANN] Py++ based ctypes code generator

[Gustavo C. <gjc...@gm...>]

2009/1/20 Roman Yakovenko <rom...@gm...>

> On Tue, Jan 20, 2009 at 9:22 PM, Gustavo Carneiro <gjc...@gm...>
> wrote:
> > This is potentially very useful!  Generated Python code can be shipped
> with
> > any project and is (almost) platform independent without need for
> > compilation; you just run Python code.
>
> This is my goal :-)
>
> > However I have to wonder how (if) do you support C++ calling conventions.
>
> I think, I was not clear. You can compile C code using C and/or C++
> compiler. Py++ supports both modes. In the second mode the calling
> convention is same as in C, but name mangling is different
>
> > Things like:
> >    - How to pass the 'this' parameter to call methods;
> >    - Function/method overloading.
> >
> > I have followed the path of C++ code generation precisely because of the
> > difficulty of calling C++ code via libffi, or else I would probably have
> > contributed to the gobject-introspection and pybank projects.  I thought
> > ctypes did not support C++?
>
> Just read few mails, with ctypes-cpp in the subject:
> http://sourceforge.net/mailarchive/forum.php?forum_name=ctypes-users
>
> and take a look on easybmp example(
>
> http://pygccxml.svn.sourceforge.net/viewvc/pygccxml/pyplusplus_dev/examples/pyeasybmp_dev/ctypes/
> )
>

I still don't understand one thing.  As far as I know, there is no standard
C++ ABI, and no standard calling convention for C++ methods, no standard
layout for class vtable; all those things are compiler dependent.  So, if
your code works with MSVC, it may not work with GCC, or vice-versa.  Am I
wrong?

Not trying to put down your excellent work.  I am just trying to understand
the limitations (which may or not matter).

Thanks in advance,

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert

Re: [pygccxml-development] [ANN] Py++ based ctypes code generator

[Roman Y. <rom...@gm...>]

On Tue, Jan 20, 2009 at 11:56 PM, Gustavo Carneiro <gjc...@gm...> wrote:
> I still don't understand one thing.  As far as I know, there is no standard
> C++ ABI, and no standard calling convention for C++ methods, no standard
> layout for class vtable; all those things are compiler dependent.  So, if
> your code works with MSVC, it may not work with GCC, or vice-versa.  Am I
> wrong?

No, you are right. I can continue the list: different version of the
same compiler, memory management, exceptions and etc.

* calling conventions - there is a lot of documentation, that explains
it, so it should be possible
* ABI - name mangling is different, but this problem is solved. Today
pygccxml is able to read exported symbols from the .dll, .map or .so
file and map it to the declaration.
* vtable - reverse engineering should work. I believe that in simple
case ( class A{}; class B : A{}, class C : B{} ) it should not be too
difficult to reconstruct vtable.
* ctypes will have to deal with different compilers and operating system
* exceptions - it is possible to solve the problem on Windows, not
sure about Linux
* memory management is the complex and hard-to-solve issue

> Not trying to put down your excellent work.

The whole credit should go to Thomas Heller.

> I am just trying to understand the limitations (which may or not matter).

I understand.


-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

Re: [pygccxml-development] [ANN] Py++ based ctypes code generator

[Roman Y. <rom...@gm...>]

On Tue, Jan 20, 2009 at 11:56 PM, Gustavo Carneiro <gjc...@gm...> wrote:
> I still don't understand one thing.  As far as I know, there is no standard
> C++ ABI, and no standard calling convention for C++ methods, no standard
> layout for class vtable; all those things are compiler dependent.  So, if
> your code works with MSVC, it may not work with GCC, or vice-versa.  Am I
> wrong?

No, you are right. I can continue the list: different version of the
same compiler, memory management, exceptions and etc.

* calling conventions - there is a lot of documentation, that explains
it, so it should be possible
* ABI - name mangling is different, but this problem is solved. Today
pygccxml is able to read exported symbols from the .dll, .map or .so
file and map it to the declaration.
* vtable - reverse engineering should work. I believe that in simple
case ( class A{}; class B : A{}, class C : B{} ) it should not be too
difficult to reconstruct vtable.
* ctypes will have to deal with different compilers and operating system
* exceptions - it is possible to solve the problem on Windows, not
sure about Linux
* memory management is the complex and hard-to-solve issue

> Not trying to put down your excellent work.

The whole credit should go to Thomas Heller.

> I am just trying to understand the limitations (which may or not matter).

I understand.


-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/