PyCXX Python-C++ Connection

Barry,

Thanks for the reply.

BTW, I am using Python2 parts of version 2.6.1 of PyCXX.

I used simple.cxx as my starting point. But all of the methods in 
"new_style_class" return Py::None. How do I create and return an instance of 
new_style_class to the python code?  (In my application, I have more than one 
class, and a method of one class needs to return an instance of another class.)

With some help from someone who knows c++ better than me, I managed to do what I 
needed and it seems to work. But it felt like a herculean effort.

I added a second constructor to PythonClass and modified its self() to take an 
optional "owned" flag. I give all the code further down. 


Let us call my subclasses of PythonClass A and B, they each wrap a pre-existing 
C++ class and hold a shared pointer to instances of those wrapped classes. Let 
us call the pointers a and b.

I am now able to return  a  PythonClassInstance of B from a method A as follows:
    const WrappedClass* x =  < something that returns an instance of 
WrappedClass>;
    return (new B(x))->self(true);

I must be being monumentally stupid because there is a vast gap between what I 
had to do and your comment "simple.cxx is all you need". I don't see how that 
can be the case when simple.cxx doesn't have a single method that returns 
anything, let alone instances of user defined classes.

Regardless of whether I did more than what was necessary, my extension works 
beautifully. I think PyCXX is a very nice tool. The fact that I could make 
changes to it is an indication of how well written it is. I am grateful for your 
work on PyCXX.

thanks,
Murali

My changes to PythonClass, in header file CXX/Python2/ExtensionType.hxx:

    template<TEMPLATE_TYPENAME T> class PythonClass
    : public PythonExtensionBase
    {
    protected:
[...]
        static PythonClassInstance* alloc_empty() {
            PyObject *py_optlib_expr = extension_object_new(type_object(), 
Tuple().ptr(), Dict().ptr());
            return reinterpret_cast<PythonClassInstance*>(py_optlib_expr);
        }

        PythonClass()
            : PythonExtensionBase()
            , m_class_instance( alloc_empty() )
        {
            reinterpret_cast< Py::PythonClassInstance * 
>(selfPtr())->m_pycxx_object = 

                static_cast<PythonExtensionBase*>(this);
        }
public:
        static const T* obj_from_arg(Object arg) {
            PythonClassInstance * b = reinterpret_cast<PythonClassInstance *> 
(arg.ptr());
            const T *val = static_cast<T *>(b->m_pycxx_object);
            return val;
        }
        
        virtual Object self(bool owned = false) // customization: Add optional 
argument owned.
        {
            return Object( reinterpret_cast<PyObject *>( m_class_instance ), 
owned );
        }
[...]


Murali




________________________________
From: Barry Scott <barry@...>
To: Murali Donthireddy <donthireddy@...>
Cc: PyCXX and improvement <cxx-users@...>
Sent: Thu, October 28, 2010 4:59:03 PM
Subject: Re: PyCXX and new style classes



On 20 Oct 2010, at 19:14, Murali Donthireddy wrote:

Hi Barry,
>
>I apologize for directly emailing you with a question. Completely understandable 
>if you don't reply.
>


We have a user list that I'm CC'ing.


>I am unable to figure out how to create, in C++ code, an instance of a new style 
>class. The constructor requires a mysterious Py::PythonClassInstance* that I 
>don't know how to generate.
>

You do not call the c'tor directly to create a instance.
You have to create instances using python methods.


>The example simple.cxx seems to be the only source of info on about new style 
>classes anywhere on the Internet, short of reading the implementation of PyCXX, 
>but my C++ isn't good enough for that. 
>

Have you tried compiling and running simple? It shows the new style class stuff 
all working.


>Any pointer or example code would be great.
>

simple.cxx is all you need. It shows you how to do most of the interesting 
stuff.

What exactly are you trying to do that simple.cxx does not do?

Barry

I also added a constructor in my class B:
B( const WrappedClass* x )  :  m_wrapped_obj(*x) {}


Murali



________________________________
From: Murali Donthireddy <donthireddy@...>
To: Barry Scott <barry@...>
Cc: PyCXX and improvement <cxx-users@...>
Sent: Thu, October 28, 2010 7:24:18 PM
Subject: Re: PyCXX and new style classes


Barry,

Thanks for the reply.

BTW, I am using Python2 parts of version 2.6.1 of PyCXX.

I used simple.cxx as my starting point. But all of the methods in 
"new_style_class" return Py::None. How do I create and return an instance of 
new_style_class to the python code?  (In my application, I have more than one 
class, and a method of one class needs to return an instance of another class.)

With some help from someone who knows c++ better than me, I managed to do what I 
needed and it seems to work. But it felt like a herculean effort.

I added a second constructor to PythonClass and modified its self() to take an 
optional "owned" flag. I give all the code further down. 


Let us call my subclasses of PythonClass A and B, they each wrap a pre-existing 
C++ class and hold a shared pointer to instances of those wrapped classes. Let 
us call the pointers a and b.

I am now able to return  a  PythonClassInstance of B from a method A as follows:
    const WrappedClass* x =  < something  that returns an instance of 
WrappedClass>;
    return (new B(x))->self(true);

I must be being monumentally stupid because there is a vast gap between what I 
had to do and your comment "simple.cxx is all you need". I don't see how that 
can be the case when simple.cxx doesn't have a single method that returns 
anything, let alone instances of user defined classes.

Regardless of whether I did more than what was necessary, my extension works 
beautifully. I think PyCXX is a very nice tool. The fact that I could make 
changes to it is an indication of how well written it is. I am grateful for your 
work on PyCXX.

thanks,
Murali

My changes to PythonClass, in header file CXX/Python2/ExtensionType.hxx:

    template<TEMPLATE_TYPENAME T> class PythonClass
    : public PythonExtensionBase
    {
    protected:
[...]
        static PythonClassInstance* alloc_empty() {
            PyObject *py_optlib_expr = extension_object_new(type_object(), 
Tuple().ptr(), Dict().ptr());
            return reinterpret_cast<PythonClassInstance*>(py_optlib_expr);
        }

        PythonClass()
            :  PythonExtensionBase()
            , m_class_instance( alloc_empty() )
        {
            reinterpret_cast< Py::PythonClassInstance * 
>(selfPtr())->m_pycxx_object = 

                static_cast<PythonExtensionBase*>(this);
        }
public:
        static const T* obj_from_arg(Object arg) {
            PythonClassInstance * b = reinterpret_cast<PythonClassInstance *> 
(arg.ptr());
            const T *val = static_cast<T  *>(b->m_pycxx_object);
            return val;
        }
        
        virtual Object self(bool owned = false) // customization: Add optional 
argument owned.
        {
            return Object( reinterpret_cast<PyObject *>( m_class_instance ), 
owned );
        }
[...]


Murali




________________________________
From: Barry Scott <barry@...>
To: Murali Donthireddy <donthireddy@...>
Cc: PyCXX and improvement <cxx-users@...>
Sent: Thu, October 28, 2010 4:59:03 PM
Subject: Re: PyCXX and new style classes



On 20 Oct 2010, at 19:14, Murali Donthireddy wrote:

Hi Barry,
>
>I apologize for directly emailing you with a question. Completely understandable 
>if you don't reply.
>


We have a user list that I'm CC'ing.


>I am unable to figure out how to create, in C++ code, an instance of a new style 
>class. The constructor requires a mysterious Py::PythonClassInstance* that I 
>don't know how to generate.
>

You do not call the c'tor directly to create a instance.
You have to create instances using python methods.


>The example simple.cxx seems to be the only source of info on about new style 
>classes anywhere on the Internet, short of reading the implementation of PyCXX, 
>but my C++ isn't good enough for that. 
>

Have you tried compiling and running simple? It shows the new style class stuff 
all working.


>Any pointer or example code would be great.
>

simple.cxx is all you need. It shows you how to do most of the interesting 
stuff.

What exactly are you trying to do that simple.cxx does not do?

Barry

On 29 Oct 2010, at 00:24, Murali Donthireddy wrote:

> Barry,
> 
> Thanks for the reply.
> 
> BTW, I am using Python2 parts of version 2.6.1 of PyCXX.
> 
> I used simple.cxx as my starting point. But all of the methods in "new_style_class" return Py::None. How do I create and return an instance of new_style_class to the python code?  (In my application, I have more than one class, and a method of one class needs to return an instance of another class.)
> 
> With some help from someone who knows c++ better than me, I managed to do what I needed and it seems to work. But it felt like a herculean effort.

It would do. You cannot do this from C++ on its own.
You have to create instances of your new style class using python calls.
Python does a lot of house keeping for new style classes that using raw C++
will bypass and lead to who knows what problems.

In pure python you would do this:

class new_style_class:
	def __init__( self ):
		...

instance = new_style_class()

"new_style_class" is in fact a type. You can call the type to create instances of that type.

In PyCXX the type is available as new_style_class::type().
In simple.cxx you can see that being exposed as the "new_style_class" in the module dictionary.

To create an instance in C++ you need code like this:

        Py::Callable class_type( new_style_class::type() );

        Py::PythonClassObject<new_style_class> new_style_obj( class_type.apply( args, kwds ) );

I have updated the simple.cxx to include a new function (make_instance) that shows this.

https://cxx.svn.sourceforge.net/svnroot/cxx/trunk/CXX/Demo/Python2/simple.cxx


> 
> I added a second constructor to PythonClass and modified its self() to take an optional "owned" flag. I give all the code further down. 

I think this will lead to problems.

> 
> Let us call my subclasses of PythonClass A and B, they each wrap a pre-existing C++ class and hold a shared pointer to instances of those wrapped classes. Let us call the pointers a and b.
> 
> I am now able to return  a  PythonClassInstance of B from a method A as follows:
>     const WrappedClass* x =  < something that returns an instance of WrappedClass>;
>     return (new B(x))->self(true);

Python may try to delete this instance and that may fail.

> 
> I must be being monumentally stupid because there is a vast gap between what I had to do and your comment "simple.cxx is all you need". I don't see how that can be the case when simple.cxx doesn't have a single method that returns anything, let alone instances of user defined classes.
> 
> Regardless of whether I did more than what was necessary, my extension works beautifully. I think PyCXX is a very nice tool. The fact that I could make changes to it is an indication of how well written it is. I am grateful for your work on PyCXX.

> 
> thanks,
> Murali
> 
> My changes to PythonClass, in header file CXX/Python2/ExtensionType.hxx:
> 
>     template<TEMPLATE_TYPENAME T> class PythonClass
>     : public PythonExtensionBase
>     {
>     protected:
> [...]
>         static PythonClassInstance* alloc_empty() {
>             PyObject *py_optlib_expr = extension_object_new(type_object(), Tuple().ptr(), Dict().ptr());
>             return reinterpret_cast<PythonClassInstance*>(py_optlib_expr);
>         }
> 
>         PythonClass()
>             : PythonExtensionBase()
>             , m_class_instance( alloc_empty() )
>         {
>             reinterpret_cast< Py::PythonClassInstance * >(selfPtr())->m_pycxx_object = 
>                 static_cast<PythonExtensionBase*>(this);
>         }
> public:
>         static const T* obj_from_arg(Object arg) {
>             PythonClassInstance * b = reinterpret_cast<PythonClassInstance *> (arg.ptr());
>             const T *val = static_cast<T *>(b->m_pycxx_object);

Any time you see reinterpret_cast in C++ is often a good sign that the code has a problem.
(Deep library code will need to do this).

Here is a type safe way to do what you want (take from func() in simple.cxx):

                Py::PythonClassObject<new_style_class> x2( x );
                std::cout << "C++ pointer " << x2.getCxxObject() << std::endl;


>             return val;
>         }
>         
>         virtual Object self(bool owned = false) // customization: Add optional argument owned.
>         {
>             return Object( reinterpret_cast<PyObject *>( m_class_instance ), owned );
>         }

Because python did not create the class for you you do not have a "self" that is sane.

Given the changes you'll make to get python to create your objects this is not needed.

Barry

Thank  you, Barry! I will modify my code as you suggest.

>> Python may try to delete this instance and that may fail.
fwiw,
I did some testing by having my python code assign the result of this method to 
a variable and then set the variable to None.
At first it wasn't calling the destructor of my new style class. I printed out 
reference counts of various objects in python and I noticed that these objects 
created in C++ code had reference counts one higher than other objects. That is 
when I added the "owned" argument to self() so that I could get an object with 
the right reference count. After that, my destructors were being called and I 
had no problems. I take that to mean Python is deleting the instance 
successfully. Of course there could be other things wrong that I haven't tested, 
and it is possible I have been lucky not to run into problems.

Murali



________________________________
From: Barry Scott <barry@...>
To: Murali Donthireddy <donthireddy@...>
Cc: PyCXX and improvement <cxx-users@...>
Sent: Sun, October 31, 2010 6:23:32 AM
Subject: Re: PyCXX and new style classes



On 29 Oct 2010, at 00:24, Murali Donthireddy wrote:

Barry,
>
>Thanks for the reply.
>
>BTW, I am using Python2 parts of version 2.6.1 of PyCXX.
>
>I used simple.cxx as my starting point. But all of the methods in 
>"new_style_class" return Py::None. How do I create and return an instance of 
>new_style_class to the python code?  (In my application, I have more than one 
>class, and a method of one class needs to return an instance of another class.)
>
>With some help from someone who knows c++ better than me, I managed to do what I 
>needed and it seems to work. But it felt like a herculean effort.
>

It would do. You cannot do this from C++ on its own.
You have to create instances of your new style class using python calls.
Python does a lot of house keeping for new style classes that using raw C++
will bypass and lead to who knows what problems.

In pure python you would do this:

class new_style_class:
def __init__( self ):
...

instance = new_style_class()

"new_style_class" is in fact a type. You can call the type to create instances 
of that type.

In PyCXX the type is available as new_style_class::type().
In simple.cxx you can see that being exposed as the "new_style_class" in the 
module dictionary.

To create an instance in C++ you need code like this:

        Py::Callable class_type( new_style_class::type() );

        Py::PythonClassObject<new_style_class> new_style_obj( class_type.apply( 
args, kwds ) );

I have updated the simple.cxx to include a new function (make_instance) that 
shows this.

https://cxx.svn.sourceforge.net/svnroot/cxx/trunk/CXX/Demo/Python2/simple.cxx



>I added a second constructor to PythonClass and modified its self() to take an 
>optional "owned" flag. I give all the code further down. 
>

I think this will lead to problems.


>Let us call my subclasses of PythonClass A and B, they each wrap a pre-existing 
>C++ class and hold a shared pointer to instances of those wrapped classes. Let 
>us call the pointers a and b.
>
>I am now able to return  a  PythonClassInstance of B from a method A as 
follows:
>    const WrappedClass* x =  < something that returns an instance of 
>WrappedClass>;
>    return (new B(x))->self(true);
>

Python may try to delete this instance and that may fail.


>I must be being monumentally stupid because there is a vast gap between what I 
>had to do and your comment "simple.cxx is all you need". I don't see how that 
>can be the case when simple.cxx doesn't have a single method that returns 
>anything, let alone instances of user defined classes.
>
>Regardless of whether I did more than what was necessary, my extension works 
>beautifully. I think PyCXX is a very nice tool. The fact that I could make 
>changes to it is an indication of how well written it is. I am grateful for your 
>work on PyCXX.
>


>thanks,
>Murali
>
>My changes to PythonClass, in header file CXX/Python2/ExtensionType.hxx:
>
>    template<TEMPLATE_TYPENAME T> class PythonClass
>    : public PythonExtensionBase
>    {
>    protected:
>[...]
>        static PythonClassInstance* alloc_empty() {
>            PyObject *py_optlib_expr = extension_object_new(type_object(), 
>Tuple().ptr(), Dict().ptr());
>            return reinterpret_cast<PythonClassInstance*>(py_optlib_expr);
>        }
>
>        PythonClass()
>            : PythonExtensionBase()
>            , m_class_instance( alloc_empty() )
>        {
>            reinterpret_cast< Py::PythonClassInstance * 
>>(selfPtr())->m_pycxx_object = 
>                static_cast<PythonExtensionBase*>(this);
>        }
>public:
>        static const T* obj_from_arg(Object arg) {
>            PythonClassInstance * b = reinterpret_cast<PythonClassInstance *> 
>(arg.ptr());
>            const T *val = static_cast<T *>(b->m_pycxx_object);
>

Any time you see reinterpret_cast in C++ is often a good sign that the code has 
a problem.
(Deep library code will need to do this).

Here is a type safe way to do what you want (take from func() in simple.cxx):

                Py::PythonClassObject<new_style_class> x2( x );
                std::cout << "C++ pointer " << x2.getCxxObject() << std::endl;


            return val;
>        }
>        
>        virtual Object self(bool owned = false) // customization: Add optional 
>argument owned.
>        {
>            return Object( reinterpret_cast<PyObject *>( m_class_instance 
>), owned );
>        }
>

Because python did not create the class for you you do not have a "self" that is 
sane.

Given the changes you'll make to get python to create your objects this is not 
needed.

Barry

On 31 Oct 2010, at 23:59, Murali Donthireddy wrote:

> Thank  you, Barry! I will modify my code as you suggest.
> 
> >> Python may try to delete this instance and that may fail.
> fwiw,
> I did some testing by having my python code assign the result of this method to a variable and then set the variable to None.
> At first it wasn't calling the destructor of my new style class. I printed out reference counts of various objects in python and I noticed that these objects created in C++ code had reference counts one higher than other objects. That is when I added the "owned" argument to self() so that I could get an object with the right reference count. After that, my destructors were being called and I had no problems. I take that to mean Python is deleting the instance successfully. Of course there could be other things wrong that I haven't tested, and it is possible I have been lucky not to run into problems.

I was seeing some strange crashes getting the new style class support working.

The way new style classes work in python is that there is a python self object that
is extended by PyCXX. In the extension I put a pointer to the C++ class instance.
Because you need this special self from python just wrapping a C++ pointer will
not work.

Barry