cxx-users — Discuss the use and improvement of PyCXX

PyCXX question on __module__ attribute

[kishor s. <kis...@gm...>]

Hi CXX Users,

I am using PyCXX to write extension classes. I am trying to pickle the
extension object using cPickle. cPickle break as __module__ attribute is
not on the objects
e.g.

In [1]: import example
range object created 0x196b290
range object destroyed 0x196b290

In [2]: example.range.__name__
Out[2]: 'range'

In [3]: example.range.__doc__
Out[3]: 'range(start,stop,stride)'

*In [4]: example.range.__module__*

*In [5]: example.range.__module__ is None*
*Out[5]: True*

I tried debugging the code it seems to be passing correct module name to
Py_InitModule4 function but it is not reflected in the python.

Any help/pointers on how to fix?

Thanks,
Kishor Sharma

PyCXX question on __module__ attribute

[kishor s. <kis...@gm...>]

Hi CXX Users,

I am using PyCXX to write extension classes. I am trying to pickle the
extension object using cPickle. cPickle break as __module__ attribute is
not on the objects
e.g.

In [1]: import example
range object created 0x196b290
range object destroyed 0x196b290

In [2]: example.range.__name__
Out[2]: 'range'

In [3]: example.range.__doc__
Out[3]: 'range(start,stop,stride)'

*In [4]: example.range.__module__*

*In [5]: example.range.__module__ is None*
*Out[5]: True*

I tried debugging the code it seems to be passing correct module name to
Py_InitModule4 function but it is not reflected in the python.

Any help/pointers on how to fix?

Thanks,
Kishor Sharma

Re: Batch of unsolved external symbols

[Barry S. <ba...@ba...>]

On Thursday 05 November 2015 21:52:44 Ilçeren Inçelami Çadormoç wrote:
> I see, I didn't get the feeling it was necessary since the documentation
> stated what source files you needed for each header.

You are right it is missing from both the py2 and py3 docs. I have fixed
that are r311. It will be in the new release.

> I'll try that as soon
> as I get to work tomorrow. I normally prefer Visual Studio to do the
> compiling and linking, but that may be useful when I port it to Linux.

<personal-option>

The problem with the Visual Studio stuff is controlling the compiler options
on the sources files. There used to be bugs where setting options in the GUI
would not result in them being used in the compile correctly. I have lost too
much time to VS bugs that it made sense to write the setup_makefile.py.

With setup_makefile.py the time spent maintain the build systems has
gone down to almost nothing. (I use the same method on pysvn and
Barry's Emacs).

Trying to go from win32 to win64 was a night mare in VS last time I tried.

I also like to have a set of build files that can be compared textually to
see what changes have been made. VS reorders sections in its project
files making that hard.

I also want to do builds that can be 100% automated and error checked
by computer with full logs to prove it. That is hard to do with VS.

With nmake/make I can build on all lots of OS with 100% reproducible
results, automatically.

I do use the VS debugger when that is needed.

</personal-option>

> Thank you very much. Ilceren
> 
> > From: ba...@ba...
> > To: cxx...@li...
> > CC: il...@ho...
> > Subject: Re: Batch of unsolved external symbols
> > Date: Thu, 5 Nov 2015 20:12:43 +0000
> > 
> > The missing symbols are in IndirectPythonInterface.cxx
> > It is not optional.
> > 
> > You can generate a windows make file using the following:
> > 
> > c:\python27\python.exe setup_makefile.py win32 example.mak
> > 
> > change the path to python to suit your needs.
> > 
> > If you have nmake on the path you can then build the examples
> > 
> > nmake -f example.make
> > 
> > Barry

RE: Batch of unsolved external symbols

[Ilçeren I. Ç. <il...@ho...>]

I see, I didn't get the feeling it was necessary since the documentation stated what source files you needed for each header. I'll try that as soon as I get to work tomorrow. I normally prefer Visual Studio to do the compiling and linking, but that may be useful when I port it to Linux. Thank you very much.
Ilceren

> From: ba...@ba...
> To: cxx...@li...
> CC: il...@ho...
> Subject: Re: Batch of unsolved external symbols
> Date: Thu, 5 Nov 2015 20:12:43 +0000
> 
> The missing symbols are in IndirectPythonInterface.cxx
> It is not optional.
> 
> You can generate a windows make file using the following:
> 
> c:\python27\python.exe setup_makefile.py win32 example.mak
> 
> change the path to python to suit your needs.
> 
> If you have nmake on the path you can then build the examples
> 
> nmake -f example.make
> 
> Barry
> 
 		 	   		  

Re: Batch of unsolved external symbols

[Barry S. <ba...@ba...>]

On Thursday 05 November 2015 18:19:13 Ilçeren Inçelami Çadormoç wrote:
> Good afternoon,I've came across this PyCXX project just recently as I was
> looking for information on how to make extension modules for Python so I
> could hook up some C++ code to a Python back-end. I've used the
> "example.cxx" file heavily for reference in order to code the interface
> with Python using an ExtensionModule, added the path to the installation
> folder for the header inclusion, included the necessary headers at the
> start of my code and added the three PyCXX code files required for
> extension programming to my sources directory. The thing is, even then, I
> get a bunch of unresolved external symbols when I try to compile my program
> to a DLL file. I'm using Visual Studio 2010. I don't think I've forgotten
> anything, not after re-reading the installation and module sections of the
> documentation a few times. The errors are as follow:
> 
> Error	24	error LNK2001: símbolo externo "void __cdecl Py::_XINCREF(struct
> _object *)" (?_XINCREF@Py@@YAXPEAU_object@@@Z) sin
> resolver	C:\...\cxxsupport.obj	Hephestos Hammer Error	21	error 
LNK2001:
> símbolo externo "void __cdecl Py::_XDECREF(struct _object *)"
> (?_XDECREF@Py@@YAXPEAU_object@@@Z) sin
> resolver	C:\...\cxxsupport.obj	Hephestos Hammer Error	41	error 
LNK2001:
> símbolo externo "struct _typeobject * __cdecl Py::_Type_Type(void)"
> (?_Type_Type@Py@@YAPEAU_typeobject@@XZ) sin
> resolver	C:\...\cxx_extensions.obj	Hephestos Hammer Error	39	error 
LNK2001:
> símbolo externo "struct _object * __cdecl Py::_None(void)"
> (?_None@Py@@YAPEAU_object@@XZ) sin
> resolver	C:\...\cxx_extensions.obj	Hephestos Hammer Error	36	error 
LNK2001:
> símbolo externo "struct _object * __cdecl Py::_Exc_TypeError(void)"
> (?_Exc_TypeError@Py@@YAPEAU_object@@XZ) sin
> resolver	C:\...\cxxsupport.obj	Hephestos Hammer Error	42	error 
LNK2001:
> símbolo externo "struct _object * __cdecl Py::_Exc_RuntimeError(void)"
> (?_Exc_RuntimeError@Py@@YAPEAU_object@@XZ) sin
> resolver	C:\...\cxx_extensions.obj	Hephestos Hammer Error	47	error 
LNK2001:
> símbolo externo "char * __cdecl Py::__Py_PackageContext(void)"
> (?__Py_PackageContext@Py@@YAPEADXZ) sin
> resolver	C:\...\cxx_extensions.obj	Hephestos Hammer Error	27	error 
LNK2001:
> símbolo externo "bool __cdecl Py::_Unicode_Check(struct _object *)"
> (?_Unicode_Check@Py@@YA_NPEAU_object@@@Z) sin
> resolver	C:\...\cxxsupport.obj	Hephestos Hammer Error	30	error 
LNK2001:
> símbolo externo "bool __cdecl Py::_Type_Check(struct _object *)"
> (?_Type_Check@Py@@YA_NPEAU_object@@@Z) sin
> resolver	C:\...\cxxsupport.obj	Hephestos Hammer Error	31	error 
LNK2001:
> símbolo externo "bool __cdecl Py::_String_Check(struct _object *)"
> (?_String_Check@Py@@YA_NPEAU_object@@@Z) sin
> resolver	C:\...\cxxsupport.obj	Hephestos Hammer Error	34	error 
LNK2001:
> símbolo externo "bool __cdecl Py::_List_Check(struct _object *)"
> (?_List_Check@Py@@YA_NPEAU_object@@@Z) sin
> resolver	C:\...\cxxsupport.obj	Hephestos Hammer Error	48	error 
LNK2001:
> símbolo externo "bool __cdecl Py::_Int_Check(struct _object *)"
> (?_Int_Check@Py@@YA_NPEAU_object@@@Z) sin
> resolver	C:\...\thirdeye.obj	Hephestos Hammer Error	46	error 
LNK2001:
> símbolo externo "bool __cdecl Py::_Dict_Check(struct _object *)"
> (?_Dict_Check@Py@@YA_NPEAU_object@@@Z) sin
> resolver	C:\...\cxx_extensions.obj	Hephestos Hammer
> 
> The Spanish bits just mean "Unresolved external symbol". I've also skipped
> some duplicated errors since I have to copy each one manually and there are
> some which are repeated up to three times in different .obj files. I don't
> know how to fix them, since I don't find anywhere the functions are
> defined. For example, I can track the Py::_XINCREF and Py::_XDECREF errors
> to the Objects.hxx file, where those functions are called, but none of the
> other files have a definition of said functions, it seems... Am I missing
> something?

The missing symbols are in IndirectPythonInterface.cxx
It is not optional.

You can generate a windows make file using the following:

c:\python27\python.exe setup_makefile.py win32 example.mak

change the path to python to suit your needs.

If you have nmake on the path you can then build the examples

nmake -f example.make

Barry

Batch of unsolved external symbols

[Ilçeren I. Ç. <il...@ho...>]

Good afternoon,I've came across this PyCXX project just recently as I was looking for information on how to make extension modules for Python so I could hook up some C++ code to a Python back-end. I've used the "example.cxx" file heavily for reference in order to code the interface with Python using an ExtensionModule, added the path to the installation folder for the header inclusion, included the necessary headers at the start of my code and added the three PyCXX code files required for extension programming to my sources directory.
The thing is, even then, I get a bunch of unresolved external symbols when I try to compile my program to a DLL file. I'm using Visual Studio 2010. I don't think I've forgotten anything, not after re-reading the installation and module sections of the documentation a few times. The errors are as follow:

Error	24	error LNK2001: símbolo externo "void __cdecl Py::_XINCREF(struct _object *)" (?_XINCREF@Py@@YAXPEAU_object@@@Z) sin resolver	C:\...\cxxsupport.obj	Hephestos Hammer
Error	21	error LNK2001: símbolo externo "void __cdecl Py::_XDECREF(struct _object *)" (?_XDECREF@Py@@YAXPEAU_object@@@Z) sin resolver	C:\...\cxxsupport.obj	Hephestos Hammer
Error	41	error LNK2001: símbolo externo "struct _typeobject * __cdecl Py::_Type_Type(void)" (?_Type_Type@Py@@YAPEAU_typeobject@@XZ) sin resolver	C:\...\cxx_extensions.obj	Hephestos Hammer
Error	39	error LNK2001: símbolo externo "struct _object * __cdecl Py::_None(void)" (?_None@Py@@YAPEAU_object@@XZ) sin resolver	C:\...\cxx_extensions.obj	Hephestos Hammer
Error	36	error LNK2001: símbolo externo "struct _object * __cdecl Py::_Exc_TypeError(void)" (?_Exc_TypeError@Py@@YAPEAU_object@@XZ) sin resolver	C:\...\cxxsupport.obj	Hephestos Hammer
Error	42	error LNK2001: símbolo externo "struct _object * __cdecl Py::_Exc_RuntimeError(void)" (?_Exc_RuntimeError@Py@@YAPEAU_object@@XZ) sin resolver	C:\...\cxx_extensions.obj	Hephestos Hammer
Error	47	error LNK2001: símbolo externo "char * __cdecl Py::__Py_PackageContext(void)" (?__Py_PackageContext@Py@@YAPEADXZ) sin resolver	C:\...\cxx_extensions.obj	Hephestos Hammer
Error	27	error LNK2001: símbolo externo "bool __cdecl Py::_Unicode_Check(struct _object *)" (?_Unicode_Check@Py@@YA_NPEAU_object@@@Z) sin resolver	C:\...\cxxsupport.obj	Hephestos Hammer
Error	30	error LNK2001: símbolo externo "bool __cdecl Py::_Type_Check(struct _object *)" (?_Type_Check@Py@@YA_NPEAU_object@@@Z) sin resolver	C:\...\cxxsupport.obj	Hephestos Hammer
Error	31	error LNK2001: símbolo externo "bool __cdecl Py::_String_Check(struct _object *)" (?_String_Check@Py@@YA_NPEAU_object@@@Z) sin resolver	C:\...\cxxsupport.obj	Hephestos Hammer
Error	34	error LNK2001: símbolo externo "bool __cdecl Py::_List_Check(struct _object *)" (?_List_Check@Py@@YA_NPEAU_object@@@Z) sin resolver	C:\...\cxxsupport.obj	Hephestos Hammer
Error	48	error LNK2001: símbolo externo "bool __cdecl Py::_Int_Check(struct _object *)" (?_Int_Check@Py@@YA_NPEAU_object@@@Z) sin resolver	C:\...\thirdeye.obj	Hephestos Hammer
Error	46	error LNK2001: símbolo externo "bool __cdecl Py::_Dict_Check(struct _object *)" (?_Dict_Check@Py@@YA_NPEAU_object@@@Z) sin resolver	C:\...\cxx_extensions.obj	Hephestos Hammer

The Spanish bits just mean "Unresolved external symbol". I've also skipped some duplicated errors since I have to copy each one manually and there are some which are repeated up to three times in different .obj files. I don't know how to fix them, since I don't find anywhere the functions are defined. For example, I can track the Py::_XINCREF and Py::_XDECREF errors to the Objects.hxx file, where those functions are called, but none of the other files have a definition of said functions, it seems... Am I missing something?

Kind regards,Ilceren
 		 	   		  

Re: Py::Object( NULL ) and PySequence_DelItem() support

[Barry S. <ba...@ba...>]

This is proving harder to change then I expected.

This is because Py::Object( NULL ) most work and be assignable etc.

I came across a case where I needed the ability to return a NULL as 
Py:Object() which is why the Python 3 has that API.

Until there are more use cases to consider I want to leave the code as
it is.

Barry


On Friday 18 September 2015 09:47:05 Barry Scott wrote:
> Thanks for the feedback. I hope to have something to test next week.
> 
> Barry
> 
> > On 12 Sep 2015, at 20:59, Vivian De Smedt <vd...@gm...> wrote:
> > 
> > Hi Barry,
> > 
> > I understand and share your concerns, backward compatibility is important.
> > 
> > Personally I prefer you second solution based on Py::ObjectAllowNull.
> > I think construct object that could accept NULL should be rare and easily
> > identifiable in a code.
> > 
> > Best regards,
> > Vivian.
> > 
> > 
> > 
> > 
> > Le mar. 1 sept. 2015 à 18:13, Vivian De Smedt <vd...@gm...
> > <mailto:vd...@gm...>> a écrit : Hi Barry,
> > 
> > Thanks for your detailed analysis. I'm currently in Holidays in Scotland.
> > I'll read your mail carefully next week.
> > 
> > Best regards,
> > Vivian
> > 
> > 
> > On Tuesday, September 1, 2015, Barry Scott <ba...@ba...
> > <mailto:ba...@ba...>> wrote: Vivian,
> > 
> > I think I have 2 patches to work on from you.
> > 
> > * Add the ability to react to PySequence_DelItem()
> > * Make py2 and py3 handling of Py::Object( NULL ) consistent
> > 
> > I was worried that someone might be relying on being able to use
> > detect the NULL value in sequence_ass_item(). Given that Py2 cannot
> > call the sequence_ass_item() because of the NULL check. I will take
> > your patch to all sequence_ass_del(). It is a nice API.
> > 
> > My feeling that it should be possible to create Py::Object from NULL.
> > But there are clearly cases when its not a reasonable thing to do.
> > 
> > I wondering what the API should look like.
> > 
> > Adding another bool to the c’tor is possible.
> > 
> > Py::Object( PyObject *, bool owner, bool accept_null );
> > Then have validate() change behaviour on accept_null.
> > 
> > Or use two classes:
> > 
> > * a Py::Object which Allows Null
> > * a Py::Object which Disallows Null
> > 
> > Take the current Py::Object and rename to Py::ObjectAllowNull.
> > 
> > class Py::ObjectAllowNull;
> > with validate() { return true; }
> > add new predicate hasObject() { return p != NULL; }
> > 
> > Then derive Py::Object from Py::ObjectAllowNull.
> > 
> > class Py::Object(Py::ObjectALllowNull);
> > with validate() { return p!=NULL; }
> > 
> > All the existing classes like Py::Int, Py::List derive from Py::Object
> > that rejects Null.
> > 
> > With the accept_null c’tor it would be possible to have a Py::Int that
> > accepts NULL.
> > 
> > Any thoughts?
> > 
> > Barry

Re: Py::Object( NULL ) and PySequence_DelItem() support

[Barry S. <ba...@ba...>]

Thanks for the feedback. I hope to have something to test next week.

Barry


> On 12 Sep 2015, at 20:59, Vivian De Smedt <vd...@gm...> wrote:
> 
> Hi Barry,
> 
> I understand and share your concerns, backward compatibility is important.
> 
> Personally I prefer you second solution based on Py::ObjectAllowNull.
> I think construct object that could accept NULL should be rare and easily identifiable in a code.
> 
> Best regards,
> Vivian.
> 
> 
> 
> 
> Le mar. 1 sept. 2015 à 18:13, Vivian De Smedt <vd...@gm... <mailto:vd...@gm...>> a écrit :
> Hi Barry,
> 
> Thanks for your detailed analysis. I'm currently in Holidays in Scotland. I'll read your mail carefully next week.
> 
> Best regards,
> Vivian
> 
> 
> On Tuesday, September 1, 2015, Barry Scott <ba...@ba... <mailto:ba...@ba...>> wrote:
> Vivian,
> 
> I think I have 2 patches to work on from you.
> 
> * Add the ability to react to PySequence_DelItem()
> * Make py2 and py3 handling of Py::Object( NULL ) consistent
> 
> I was worried that someone might be relying on being able to use
> detect the NULL value in sequence_ass_item(). Given that Py2 cannot
> call the sequence_ass_item() because of the NULL check. I will take
> your patch to all sequence_ass_del(). It is a nice API.
> 
> My feeling that it should be possible to create Py::Object from NULL.
> But there are clearly cases when its not a reasonable thing to do.
> 
> I wondering what the API should look like.
> 
> Adding another bool to the c’tor is possible.
> 
> Py::Object( PyObject *, bool owner, bool accept_null );
> Then have validate() change behaviour on accept_null.
> 
> Or use two classes:
> 
> * a Py::Object which Allows Null
> * a Py::Object which Disallows Null
> 
> Take the current Py::Object and rename to Py::ObjectAllowNull.
> 
> class Py::ObjectAllowNull;
> with validate() { return true; }
> add new predicate hasObject() { return p != NULL; }
> 
> Then derive Py::Object from Py::ObjectAllowNull.
> 
> class Py::Object(Py::ObjectALllowNull);
> with validate() { return p!=NULL; }
> 
> All the existing classes like Py::Int, Py::List derive from Py::Object that rejects Null.
> 
> With the accept_null c’tor it would be possible to have a Py::Int that accepts NULL.
> 
> Any thoughts?
> 
> Barry
> 

Py::Object( NULL ) and PySequence_DelItem() support

[Barry S. <ba...@ba...>]

Vivian,

I think I have 2 patches to work on from you.

* Add the ability to react to PySequence_DelItem()
* Make py2 and py3 handling of Py::Object( NULL ) consistent

I was worried that someone might be relying on being able to use
detect the NULL value in sequence_ass_item(). Given that Py2 cannot
call the sequence_ass_item() because of the NULL check. I will take
your patch to all sequence_ass_del(). It is a nice API.

My feeling that it should be possible to create Py::Object from NULL.
But there are clearly cases when its not a reasonable thing to do.

I wondering what the API should look like.

Adding another bool to the c’tor is possible.

Py::Object( PyObject *, bool owner, bool accept_null );
Then have validate() change behaviour on accept_null.

Or use two classes:

* a Py::Object which Allows Null
* a Py::Object which Disallows Null

Take the current Py::Object and rename to Py::ObjectAllowNull.

class Py::ObjectAllowNull;
with validate() { return true; }
add new predicate hasObject() { return p != NULL; }

Then derive Py::Object from Py::ObjectAllowNull.

class Py::Object(Py::ObjectALllowNull);
with validate() { return p!=NULL; }

All the existing classes like Py::Int, Py::List derive from Py::Object that rejects Null.

With the accept_null c’tor it would be possible to have a Py::Int that accepts NULL.

Any thoughts?

Barry

PiCxx

[π <sun...@gm...>]

Hello everyone,

Last winter I did a complete (from the ground up) rewrite of PyCXX, making full use of modern design patterns allowed by C++11.

It started out as an adventure and a vehicle with which to explore/learn C++, and ended up consuming several months and relapsing strain injuries from which I'm still struggling.

I am happy with the result, and welcome improvements and suggestions, although my main focus is currently elsewhere.

PiCxx currently only supports Python3; I did not find the energy to add Python2 support, although it would be small amount of work in relation to the project.

My gratitude to those that guided me, notably Barry for writing the original, Yhg1s on IRC for patiently pulling me through the necessary CPython internals, and a couple of splendid engineers on Stack Overflow.

Shortly after completing work, I learned that Apple are planning to open source Swift, which is a truly modern language that gives a level of flexibility close (enough) to Python, at speeds close (enough) to C/C++.  I've been planning to integrate Python into JUCE (a multi-platform C++ framework), and Swift may prove to be a better solution, so as far as my own purpose is concerned I may have wasted a noticeable chunk of my life-force spinning my wheels.  But maybe it is of some interest to another. And maybe I will in the future use it myself.

PiCxx is available here: https://github.com/p-i-/PiCxx <https://github.com/p-i-/PiCxx>

π

Re: can't import most modules (but no exception)

[Barry S. <ba...@ba...>]

Py::Module canot import a module. It can only hold a ref to an already 
imported module.

Becuase of the complexity of import in python its easiest to use 
PyRun_SimpleString.

PyRun_SimpleString(  "import foo.bar" );

Barry
p.s. I did reply a while ago but sf.net's FUBAR dropped the mail.


On 01/07/2015 21:37, David Fox wrote:
> I’m trying to take the example from section 1.3 of the python 
> embedding documentation 
> (https://docs.python.org/3/extending/embedding.html#pure-embedding), 
> with usage
>
> Usage: call pythonfile funcname [args]
>
>
> and rewrite it using CXX (with some extra diagnostics, and modified to 
> use str() so it handles functions which return anything, not just long):
>
> #include <Python.h>
> #include <CXX/Objects.hxx>
> #include <string>
> #include <iostream>
>
> int
> main(int argc, char *argv[])
> {
>     int i;
>
>     if (argc < 3) {
>         fprintf(stderr,"Usage: call pythonfile funcname [args]\n");
>         return 1;
>     }
>
>     std::string name(argv[1]);
>
>     std::string funcname(argv[2]);
>     Py_Initialize();
>     std::string status;
>     try {
>         Py::Module module(name);
>
>         std::cout << module.repr() << '\n';
>         if (!module.isTrue()) {
>             std::cout << "module was null/false\n";
>             throw Py::Exception();
>         }
>         status = "finding function " + funcname;
>         if (!module.hasAttr(funcname)) {
>             std::cout << "function " << funcname + " not found\n";
>             throw Py::Exception();
>         }
>         Py::Object obj = module.getAttr(funcname);
>         if (!obj.isTrue()) {
>             std::cout << "function was null/false\n";
>             throw Py::Exception();
>         }
>         if (!obj.isCallable()) {
>             std::cout << "got function but not callable?\n";
>             throw Py::Exception();
>         }
>         status = "casting function " + funcname;
>         Py::Callable func(obj);
>         status = "creating arg tuple";
>         Py::Tuple fargs(argc - 3);
>         status = "filling arg tuple";
>         for (i = 0; i < argc - 3; ++i) {
>             fargs[i] = Py::Long(atoi(argv[i + 3]));
>         }
>         status = "calling";
>         Py::Object res(func.apply(fargs));
>         //        status = "converting to long";
>         //        Py::Long v(res);
>         std::cout << "Result of call: " << res.str() << '\n';
>     } catch (const Py::Exception &) {
>         std::cout << "call failed with status " << status << '\n';
>     }
>
>     Py_Finalize();
>     return 0;
> }
>
>
> The original version compiles and runs correctly.   For example
>
> $ ./call math exp 2
> Result of call: 7
>
> $ PYTHONPATH=`pwd` ./call multiply multiply 2 3
> Will compute 2 times 3
> Result of call: 6
>
> My CXX version works as long as the module I specify is sys or os:
>
> ./pure_cxx sys exc_info
>  <module 'sys' (built-in)>
>  Result of call: (None, None, None)
>
> $ ./pure_cxx os getcwd
> <module 'os' from 
> '/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/os.py'>
> Result of call: /Users/dfox/python/test/embed
>
> but if I use math or a custom module, it fails to find the function in 
> the given module:
>
>     $ ./pure_cxx math exp 2
>     <module 'math'>
>     function exp not found
>     call failed with status finding function exp
>
>
>     $ PYTHONPATH=`pwd` ./pure_cxx multiply multiply 2 3
>     <module 'multiply'>
>     function multiply not found
>     call failed with status finding function multiply
>
>
> Oddly, it doesn’t raise an exception when I create the Py::Module, but 
> that module seems to be a dummy module, with the right name and a few 
> other attributes all with value None, as can be seen by converting the 
> module to a Py::List and iterating over it:
>
> 0  __name__ : math
>
> 1  __loader__ : None
>
> 2  __spec__ : None
>
> 3  __package__ : None
>
> 4  __doc__ : None
>
>
> Any idea what I’m doing wrong?
>
>
> ------------------------------------------------------------------------------
> Don't Limit Your Business. Reach for the Cloud.
> GigeNET's Cloud Solutions provide you with the tools and support that
> you need to offload your IT needs and focus on growing your business.
> Configured For All Businesses. Start Your Cloud Today.
> https://www.gigenetcloud.com/
>
>
> _______________________________________________
> CXX-Users mailing list
> CXX...@li...
> https://lists.sourceforge.net/lists/listinfo/cxx-users

can't import most modules (but no exception)

[David F. <Dav...@hu...>]

I’m trying to take the example from section 1.3 of the python embedding documentation (https://docs.python.org/3/extending/embedding.html#pure-embedding), with usage


Usage: call pythonfile funcname [args]

and rewrite it using CXX (with some extra diagnostics, and modified to use str() so it handles functions which return anything, not just long):

#include <Python.h>
#include <CXX/Objects.hxx>
#include <string>
#include <iostream>

int
main(int argc, char *argv[])
{
    int i;

    if (argc < 3) {
        fprintf(stderr,"Usage: call pythonfile funcname [args]\n");
        return 1;
    }

    std::string name(argv[1]);

    std::string funcname(argv[2]);
    Py_Initialize();
    std::string status;
    try {
        Py::Module module(name);

        std::cout << module.repr() << '\n';
        if (!module.isTrue()) {
            std::cout << "module was null/false\n";
            throw Py::Exception();
        }

        status = "finding function " + funcname;
        if (!module.hasAttr(funcname)) {
            std::cout << "function " << funcname + " not found\n";
            throw Py::Exception();
        }
        Py::Object obj = module.getAttr(funcname);
        if (!obj.isTrue()) {
            std::cout << "function was null/false\n";
            throw Py::Exception();
        }
        if (!obj.isCallable()) {
            std::cout << "got function but not callable?\n";
            throw Py::Exception();
        }
        status = "casting function " + funcname;
        Py::Callable func(obj);
        status = "creating arg tuple";
        Py::Tuple fargs(argc - 3);
        status = "filling arg tuple";
        for (i = 0; i < argc - 3; ++i) {
            fargs[i] = Py::Long(atoi(argv[i + 3]));
        }
        status = "calling";
        Py::Object res(func.apply(fargs));
        //        status = "converting to long";
        //        Py::Long v(res);
        std::cout << "Result of call: " << res.str() << '\n';
    } catch (const Py::Exception &) {
        std::cout << "call failed with status " << status << '\n';
    }

    Py_Finalize();
    return 0;
}


The original version compiles and runs correctly.   For example

$ ./call math exp 2
Result of call: 7

$ PYTHONPATH=`pwd` ./call multiply multiply 2 3
Will compute 2 times 3
Result of call: 6

My CXX version works as long as the module I specify is sys or os:

./pure_cxx sys exc_info
 <module 'sys' (built-in)>
 Result of call: (None, None, None)

$ ./pure_cxx os getcwd
<module 'os' from '/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/os.py'>
Result of call: /Users/dfox/python/test/embed

but if I use math or a custom module, it fails to find the function in the given module:

$ ./pure_cxx math exp 2
<module 'math'>
function exp not found
call failed with status finding function exp

$ PYTHONPATH=`pwd` ./pure_cxx multiply multiply 2 3
<module 'multiply'>
function multiply not found
call failed with status finding function multiply

Oddly, it doesn’t raise an exception when I create the Py::Module, but that module seems to be a dummy module, with the right name and a few other attributes all with value None, as can be seen by converting the module to a Py::List and iterating over it:


0  __name__ : math

1  __loader__ : None

2  __spec__ : None

3  __package__ : None

4  __doc__ : None

Any idea what I’m doing wrong?

Re: PyCXX V6.2.6 Released

[Barry S. <ba...@ba...>]

> On 5 Jan 2015, at 23:41, π <sun...@gm...> wrote:
> 
> Hello PyCXX people,
> 
> I recently stumbled upon what I believed to be a minor PyCXX bug regarding allocation for new style classes.
> 
> http://stackoverflow.com/questions/27662074/how-to-tidy-fix-pycxxs-creation-of-new-style-python-extension-class

I'll have to check the memory alloc bug mentioned. If you can isolate the patch I'd appreciate it.

There is a lot of clutter in the SO articles.

> 
> The link proposes a fix. Amazingly, there is some anonymous engineer on SO who appears to understand every line of the PyCXX source code.
> My hat off to this individual! The technical complexity of this library has really stretched my mental capacity to its limits.

If you understand the python internals I think the PyCXX is not so hard to understand.

> A second concern I have is regarding...
> 
> http://sourceforge.net/p/cxx/code/HEAD/tree/trunk/CXX/Src/Python3/cxx_extensions.cxx
> 
> extern "C" PyObject *str_handler( PyObject *self )
> {
>    try
>    {
>        PythonExtensionBase *p = getPythonExtensionBase( self );
>        return new_reference_to( p->str() );
>    }
>    catch( Py::Exception & )
>    {
>        return NULL;    // indicate error
>    }
> }
> 
> There are about 50 functions of the same format and my concern applies to all of them.
> 
> What if somewhere in the p->str() C++ code an exception happens that is not of type Py::Exception ?

You mean a C++ exception?

That would be a bug in the C++ code which needs to map the C++ exception into a Python Exception.

> I don't think it will be caught.
> 
> Shouldn't there be an extra 'catch(...)' to catch the remaining errors?
> 
> I'm not too sure about the second one, the first one does appear to be a bug.
> 

Barry

Re: PyCXX V6.2.6 Released

[π <sun...@gm...>]

Hello PyCXX people,

I recently stumbled upon what I believed to be a minor PyCXX bug regarding allocation for new style classes.

http://stackoverflow.com/questions/27662074/how-to-tidy-fix-pycxxs-creation-of-new-style-python-extension-class

The link proposes a fix. Amazingly, there is some anonymous engineer on SO who appears to understand every line of the PyCXX source code.
My hat off to this individual! The technical complexity of this library has really stretched my mental capacity to its limits.

A second concern I have is regarding...

 http://sourceforge.net/p/cxx/code/HEAD/tree/trunk/CXX/Src/Python3/cxx_extensions.cxx

extern "C" PyObject *str_handler( PyObject *self )
{
    try
    {
        PythonExtensionBase *p = getPythonExtensionBase( self );
        return new_reference_to( p->str() );
    }
    catch( Py::Exception & )
    {
        return NULL;    // indicate error
    }
}

There are about 50 functions of the same format and my concern applies to all of them.

What if somewhere in the p->str() C++ code an exception happens that is not of type Py::Exception ?

I don't think it will be caught.

Shouldn't there be an extra 'catch(...)' to catch the remaining errors?

I'm not too sure about the second one, the first one does appear to be a bug.

π

> On 4 Jan 2015, at 17:03, Barry Scott <ba...@ba...> wrote:
> 
> Version 6.2.6 (4-Jan-2015)
> * Fix various compiler warning.
> * Fix incorrect use of NULL.
> * Fix conversion from (const char *) to (char *) is deprecated warning.
> 
> Download from https://sourceforge.net/projects/cxx/
> 
> Barry
> 
> 
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming! The Go Parallel Website,
> sponsored by Intel and developed in partnership with Slashdot Media, is your
> hub for all things parallel software development, from weekly thought
> leadership blogs to news, videos, case studies, tutorials and more. Take a
> look and join the conversation now. http://goparallel.sourceforge.net
> _______________________________________________
> CXX-Users mailing list
> CXX...@li...
> https://lists.sourceforge.net/lists/listinfo/cxx-users

PyCXX V6.2.6 Released

[Barry S. <ba...@ba...>]

Version 6.2.6 (4-Jan-2015)
* Fix various compiler warning.
* Fix incorrect use of NULL.
* Fix conversion from (const char *) to (char *) is deprecated warning.

Download from https://sourceforge.net/projects/cxx/

Barry

Re: simple2 demo seems to be buggy

[Barry S. <ba...@ba...>]

On 22 Oct 2014, at 23:22, π <sun...@gm...> wrote:

> I've been playing around with PyCXX, slowly figuring it out.
> 
> I've got all of the demos but one working: simple, example, range, iter, but not simple2.


simple2 was written to test and fix bugs raise around 2008.

If you are not doing so I would recommend that you checkout the svn repo for pycxx and look at log messages related to code you are investigating.

These seems to be the interesting svn logs:

r160 | barry-scott | 2008-12-01 19:56:37 +0000 (Mon, 01 Dec 2008) | 2 lines

can call methods on new style classes
but dir(new_style_class) does not return a list of methods
------------------------------------------------------------------------
r159 | barry-scott | 2008-12-01 17:40:54 +0000 (Mon, 01 Dec 2008) | 2 lines

Add debug functions under the PYCXX_DEBUG define

------------------------------------------------------------------------
r157 | barry-scott | 2008-11-28 12:12:30 +0000 (Fri, 28 Nov 2008) | 2 lines

Fix for Sourceforge bug 2338117

------------------------------------------------------------------------
r141 | barry-scott | 2008-10-07 21:46:41 +0100 (Tue, 07 Oct 2008) | 2 lines

Add test case for problem with using PythonExtension type as  dictionary key



> 
> The following code shows the problem, I think:
> 
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
> 
> import simple2
> import sys
> 
> print( "---" )
> print( id(simple2.xxx.second) ) # 4299353944
> print( id(simple2.xxx.second) ) # 4299353944
> print( id(simple2.xxx.second) ) # 4299353944
> print( "---" )
> 
> a,b,c = simple2.xxx.second, simple2.xxx.second, simple2.xxx.second
> print( id(a) ) # 4299353944
> print( id(b) ) # 4299513704
> print( id(c) ) # 4299214952
> print( "---" )
> print( id(simple2.xxx.second) ) # 4299435416
> print( id(simple2.xxx.second) ) # 4299435416
> print( id(simple2.xxx.second) ) # 4299435416
> print( "---" )
> 
> p = simple2.xxx.second
> q = simple2.xxx.second
> 
> print ( id(p) ) # 4299435416
> print ( id(q) ) # 4299521272
> 
> print( id(simple2.xxx.second) ) # 4299516904
> print( id(simple2.xxx.second) ) # 4299516904
> print( "---" )
> 
> print ( "Making dict m..." )
> m = {
>     simple2.xxx.first  : 1,
>     simple2.xxx.second : 2,
>     simple2.xxx.third  : 3
> }
> 
> print( "---" )
> print( id(simple2.xxx.second) )  # 4299381256
> print( id(simple2.xxx.second) )  # 4299381256
> print( "---" )
> 
> print ( "... done!" )
> 
> v = m[ simple2.xxx.second ]
> 
> #Traceback (most recent call last):
> #    File "./py/test_simple2.py", line 53, in <module>
> #    v = m[ simple2.xxx.second ]
> #
> #did EnumString< xxx_t >::EnumString()
> #KeyError: <xxx.second>
> #-1
> #Program ended with exit code: 0
> 
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
> 
> It looks as though every time we assign simple2.xxx.second to some variable, it creates a new instance rather than giving just a new reference.
> 
> And that in defining m = {…}, a new instance is also created.
> 
> If instead we were to do:
> 
> - - - - - - - - - - - - -
> u = simple2.xxx.second
> m = {
>     simple2.xxx.first  : 1,
>     u : 2,
>     simple2.xxx.third  : 3
> }
> v = m[ u ]
> - - - - - - - - - - - - -
> 
> … this works.

The point of the test is the use an object not an python interned int.

> 
> The same problem doesn't seem to occur with the root object:
> - - - - - - - - - - - - -
> print( id(simple2.xxx) )
> print( id(simple2.xxx) )
>       
> x1 = simple2.xxx
> x2 = simple2.xxx
> print( id(x1) )
> print( id(x2) )
> - - - - - - - - - - - - -
> The above produces the same id all 4 times.

equality does not require id() to be the same.


> 
> I think the problem is here:
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
>     // if the given string has an associated T object, return that T object
>     Py::Object getattr( const char *_name ) override
>     {
>         std::string name( _name );
>         T value;
> 
>         if( name == "__methods__" )
>             return Py::List();
> 
>         if( name == "__members__" )
>             return memberList( static_cast<T>( 0 ) );
> 
>         if( toEnum( name, value ) ) // <-- sets value
>             return Py::asObject( new pysvn_enum_value<T>( value ) ); // <— HERE

And no where is there an instance of pysvn_enum_value that can be returned.
So a new instance must be made.

> 
>         return this->getattr_methods( _name );    
>     }
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
> 
> Where it can be seen clearly that getattr returns a newly created Python object.
> 
> In fact I think that the only time we ever get the same id returned consecutively is due to coincidence: the first object gets destructed, and another object gets created. If this happen straight away, maybe the Python runtime reuses the ID of the object that just got destroyed.

No idea on the id() reuse. You would have to read the python sources.

> 
> Although I still don't fully grok this simple2 example (I don't find it very simple at all), my guess is that to fix this it would be necessary to create a pysvn_enum_value<xxx_t> py-extension-object for each xxx_t enum value, and bind them together using ‘map’ upon creation of simple.xxx (i.e. in it’s init_type).  Then whenever e.g. xxx.simple2.second is encountered, get_attr can return a non-owned reference to the corresponding pysvn_enum_value<xxx_t>.

Its based on a real world problem that I hit in pysvn. The point of the test is to prevent a regression of the fix made 6 years ago.

> 
> Another thing I notice is that several functions contain:
> 
>     static EnumString<T> enum_map;
> 
> This doesn't look like a good idea, surely there should only be one map created.  Maybe it should be a static public variable inside EnumString?

Correct, however, the code in pysvn is not this shape.

You might find the pysvn code interesting to read as it is a major user of pycxx.

Barry

> 
> π
> ------------------------------------------------------------------------------
> _______________________________________________
> CXX-Users mailing list
> CXX...@li...
> https://lists.sourceforge.net/lists/listinfo/cxx-users

Re: ExtensionType vs ExtensionOldType

[Barry S. <ba...@ba...>]

On 14 Oct 2014, at 02:00, π <sun...@gm...> wrote:

> Hello PyCXX people,
> 
> I am very glad to find this framework,  I have been spending some time digging through the source code attempting to understand how it does its magic.

You need a very deep understanding of how C python 2 and C python 3 works internally.

The idea behind PyCXX is that you can create modules in C++ with needing to understand all that detail.

> 
> I'm confused by the existence of new-style classes and old-style classes.
> 
> For example, in test_simple.py, both of these are tested?

It is a *test* and the tests attempt to provide coverage of all the supported API. Python 3 tests are far better then python 2.

> 
> What are these entities?
> 
> Did someone create a new mechanism for wrapping C++ classes as Python Types that was not backwards compatible (whatever that may mean), and hence both are now coexisting in the CodeBase?

CPython has two ways to implement a Class from C. new-style-classes is the replacement for the older API.

For new code use the new-style classes gives you more options.

> π
> 
> PS looking through the source code, I notice that vast amount of compacting is possible through preprocessor macros and Variadic Templates.  I'm in the process of refactoring, mainly so that I can gain some familiarity with the CodeBase and present certain sections in a way that my brain can parse more easily.  I've uploaded some of my changes to http://mathpad.wikidot.com/pycxx — the links at the bottom of this page go to various modifications.  

preprocessor macros are often bad news for the maintenance of code. You can end up with non-obvious bugs that require you to know a lot about the macros to review changes to the code.

The uses of templates in PyCXX as a type safe wrapper around a non-template core is for binary code efficiency reasons. The old PyCXX code used templates only and the code size of the resulting binaries for huge.

If you have patches that improve the code I will take them, but not if it loses binary code size efficiency or makes the source code fragile.

Barry

simple2 demo seems to be buggy

[π <sun...@gm...>]

I've been playing around with PyCXX, slowly figuring it out.

I've got all of the demos but one working: simple, example, range, iter, but not simple2.

The following code shows the problem, I think:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 

import simple2
import sys

print( "---" )
print( id(simple2.xxx.second) ) # 4299353944
print( id(simple2.xxx.second) ) # 4299353944
print( id(simple2.xxx.second) ) # 4299353944
print( "---" )

a,b,c = simple2.xxx.second, simple2.xxx.second, simple2.xxx.second
print( id(a) ) # 4299353944
print( id(b) ) # 4299513704
print( id(c) ) # 4299214952
print( "---" )
print( id(simple2.xxx.second) ) # 4299435416
print( id(simple2.xxx.second) ) # 4299435416
print( id(simple2.xxx.second) ) # 4299435416
print( "---" )

p = simple2.xxx.second
q = simple2.xxx.second

print ( id(p) ) # 4299435416
print ( id(q) ) # 4299521272

print( id(simple2.xxx.second) ) # 4299516904
print( id(simple2.xxx.second) ) # 4299516904
print( "---" )

print ( "Making dict m..." )
m = {
    simple2.xxx.first  : 1,
    simple2.xxx.second : 2,
    simple2.xxx.third  : 3
}

print( "---" )
print( id(simple2.xxx.second) )  # 4299381256
print( id(simple2.xxx.second) )  # 4299381256
print( "---" )

print ( "... done!" )

v = m[ simple2.xxx.second ]

#Traceback (most recent call last):
#    File "./py/test_simple2.py", line 53, in <module>
#    v = m[ simple2.xxx.second ]
#
#did EnumString< xxx_t >::EnumString()
#KeyError: <xxx.second>
#-1
#Program ended with exit code: 0

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 

It looks as though every time we assign simple2.xxx.second to some variable, it creates a new instance rather than giving just a new reference.

And that in defining m = {…}, a new instance is also created.

If instead we were to do:

- - - - - - - - - - - - -
u = simple2.xxx.second
m = {
    simple2.xxx.first  : 1,
    u : 2,
    simple2.xxx.third  : 3
}
v = m[ u ]
- - - - - - - - - - - - -

… this works.

The same problem doesn't seem to occur with the root object:
- - - - - - - - - - - - -
print( id(simple2.xxx) )
print( id(simple2.xxx) )
      
x1 = simple2.xxx
x2 = simple2.xxx
print( id(x1) )
print( id(x2) )
- - - - - - - - - - - - -
The above produces the same id all 4 times.

I think the problem is here:
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
    // if the given string has an associated T object, return that T object
    Py::Object getattr( const char *_name ) override
    {
        std::string name( _name );
        T value;

        if( name == "__methods__" )
            return Py::List();

        if( name == "__members__" )
            return memberList( static_cast<T>( 0 ) );

        if( toEnum( name, value ) ) // <-- sets value
            return Py::asObject( new pysvn_enum_value<T>( value ) ); // <— HERE

        return this->getattr_methods( _name );    
    }
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 

Where it can be seen clearly that getattr returns a newly created Python object.

In fact I think that the only time we ever get the same id returned consecutively is due to coincidence: the first object gets destructed, and another object gets created. If this happen straight away, maybe the Python runtime reuses the ID of the object that just got destroyed.

Although I still don't fully grok this simple2 example (I don't find it very simple at all), my guess is that to fix this it would be necessary to create a pysvn_enum_value<xxx_t> py-extension-object for each xxx_t enum value, and bind them together using ‘map’ upon creation of simple.xxx (i.e. in it’s init_type).  Then whenever e.g. xxx.simple2.second is encountered, get_attr can return a non-owned reference to the corresponding pysvn_enum_value<xxx_t>.

Another thing I notice is that several functions contain:

    static EnumString<T> enum_map;

This doesn't look like a good idea, surely there should only be one map created.  Maybe it should be a static public variable inside EnumString?

π

ExtensionType vs ExtensionOldType

[π <sun...@gm...>]

Hello PyCXX people,

I am very glad to find this framework,  I have been spending some time digging through the source code attempting to understand how it does its magic.

I'm confused by the existence of new-style classes and old-style classes.

For example, in test_simple.py, both of these are tested?

What are these entities?

Did someone create a new mechanism for wrapping C++ classes as Python Types that was not backwards compatible (whatever that may mean), and hence both are now coexisting in the CodeBase?

π

PS looking through the source code, I notice that vast amount of compacting is possible through preprocessor macros and Variadic Templates.  I'm in the process of refactoring, mainly so that I can gain some familiarity with the CodeBase and present certain sections in a way that my brain can parse more easily.  I've uploaded some of my changes to http://mathpad.wikidot.com/pycxx — the links at the bottom of this page go to various modifications.  

Re: Declaring attributes in PyCXX

[Barry S. <ba...@ba...>]

On 7 Apr 2014, at 12:42, Burak Kahraman <bur...@la...> wrote:

> Hi Barry,
> 
> I just started to write my own Python extensions and immediately switched to your library because of several positive posts on stackoverflow.
> I really like it so far, but I have a very basic problem. I added my attributes by overriding getattro() as in the examples, but the thing is I want to see these attributes when using dir() on my object. So I have to declare these somehow.
> With the regular functions I could create a static array of PyGetSetDef() (defining the attribute name and getter and setter functions) and assign it to tp_getset.
> This doesn't work in my subclass of ExtensionModule, as I don't have access to the protected variable PythonType::tp_getset.
> I also started a topic on stackoverflow without any luck yet :-\
> 
> Thanks for your great work and it would be nice to hear back from you.
> 

The trick is that python asks for the value of __members__.

In your code return a list of the names you want to see in dir(your_obj).

This is a cut down version of what I did in pysvn code to support dir().

Py::Object pysvn_client::getattr( const char *_name )
{
    std::string name( _name );

    if( name == "___members__" )
    {
        Py::List members;

        members.append( "callback_get_login" );
        members.append( "callback_notify" );

        return members;
    }

    if( name == "callback_get_login" )
        return m_context.m_pyfn_GetLogin;

    return getattr_default( _name );
}

Barry





> Best,
> Burak
> 
> -- 
> Burak Kahraman
> Software Developer
> 
> Laubwerk GmbH
> August-Bebel-Str. 27
> 14482 Potsdam
> Germany
> 
> Phone +49 (0)30 / 609 86 48-01
> Fax +49 (0)30 / 609 86 48-09
> 
> www.laubwerk.com
> facebook.com/laubwerk 		

PyCXX V6.2.5 Released

[Barry S. <ba...@ba...>]

Version 6.2.5 (18-August-2013)

* Fix build issue with GCC 4.2.1 on FreeBSD and Mac OS X (stop python defining isspace as a macro).
* Remove support for python 3.1 (API's are unstable).
* Add Python 3.3 support.
* Patch from Michael Droettboom to fix compilation issues.
* Patch from Michael Droettboom to add buffer interface for python3.

Patches for matplotlib Python 3.x support

[Michael D. <md...@st...>]

I'm not sure what the procedure for submitting patches to PyCXX is, but 
I wanted to bring to your attention some patches that are required to 
successfully build matplotlib on Python 3.x.  We currently ship our own 
copy of PyCXX with these patches, but we'd like to stop doing so in the 
future, of course.

This is to support the buffer interface:

https://sourceforge.net/tracker/?func=detail&aid=3590190&group_id=3180&atid=303180

This is to use Py_hash_t rather than long where appropriate, because 
sizeof(long) != sizeof(long *) on some platforms:

https://sourceforge.net/tracker/?func=detail&aid=3590194&group_id=3180&atid=303180

And lastly, and least importantly, a simple patch to suppress a compiler 
warning about compare_handler being defined but not used:

https://sourceforge.net/tracker/?func=detail&aid=3590198&group_id=3180&atid=303180

Cheers,
Mike

PyCXX V6.2.4 released

[Barry S. <ba...@ba...>]

Version 6.2.4 (3-March-2012)

* Fix problem compiling against clang compiler used with Mac OS X 10.7
* Fix segv when using getting the type() of extension objects
* Support PEP 3149
* Support Python 3.2
* For python 3.1 and onwards use the Capsule API - thanks to Michael Droettboom
* Fixed memory leak - thanks to Vivian De Smedt
* Improve setup_makefile.py to use distutils to find paths

Re: the performance of pycxx

[Barry S. <ba...@ba...>]

On 22 Nov 2011, at 03:39, <las...@si...> <las...@si...> wrote:

>     Py::Object fib (const Py::Tuple &args){
>         args.verify_length(1);
>         Py::Long res = args[0];
>         if (res<=Py::Long(1))  return Py::Long(1);
>         else{
>             Py::Tuple a(1), b(1); a[0]=Py::Long(res-2); b[0]=Py::Long(res-1);
>             return fib(a)+fib(b);
>         }
>     }


Following on from what Paul said:

Why use Python objects all the time if the goal is speed?

You have transliterated the pure Python code into C++
with all the costs that implies and none of Python's optimizations...
Of course its slow.

Do not use Py::Long when what you want for speed is int.

Create one function that computes the fib using pure C++.

Then have a second function that drives is from python.
That will avoid the costs of translating to and from the python
world.

Barry

RE: the performance of pycxx

[Paul K. <Pau...@ni...>]

>> I conclude that pycxx is only for the aim "to write python exteison in C++ easily", but not for the aim "to speed up the original pure python program". Am I right?

Not really. You can make a Python program run faster if you translate a resource-heavy Python function into C++, wrap it in a Python wrapper, then call it from Python. But you have to arrange to call it only once, not thousands of times, because the speedup you get from translating the code into C++ will be completely lost if you have to pay the pycxx overhead thousands of times. The problem is not the speed of the translated code. The problem is the number of times you are calling it.

When your code computes fib(10) and returns the value 89, the function fib() gets called 177 times. That's okay. But when your code computes fib(30) and returns the value 1,346,269 that is not okay, because the function fib() gets called 2,692,537 times. It doesn't matter how much faster the C++ verion of fib() is, because any saving is dwarfed by the huge function call overhead. Your C++ code is calling the Python function, which in turn is calling back to the C++ function. That is very expensive, and you are doing it millions of times. Why is your C++ function calling back to the Python function? The whole point of the exercise is to do the computation in C++, because that is the only way you are going to get a speedup.

There are two ways to eliminate the problem.

1. Have your C++ function fib() call a C++ function cfib(), not the Python function fib(). Do the computation entirely in C++ and only return the value to Python when you are done. Don't route intermediate results via the pycxx interface. Like this:

        if (res<=Py::Long(1))  return Py::Long(1);
        else{
            return Py::Long(cfib(res-2)+cfib(res-1));
        }

2. Rewrite fib() as a nonrecursive function. Here's a sample Python function that runs in linear time and caches its results so that repeated calls are free:

def fibseries(n, seed=[0,1]):
        while n > len(seed) - 2:
          seed.append(sum(seed[-2:]))
        return seed

def fib(n, series=[0,1]):
        series = fibseries(n, series)
        return series[n+1]

Implement that in C++ (which admittedly is a bit challenging) and you should see a modest improvement. But again, you must implement both functions in pure C++. No calling back to Python allowed.

If your recursive C++ code is deliberately calling back to Python because there is some aspect of the algorithm you are finding hard to implement in C++ (integers bigger than 2**31-1, perhaps?), then your code is not a good candidate for migration to C++. That doesn't mean you can't get a speedup by migrating Python code to C++. You can. But not if your C++ code cheats by calling back into Pytnon.

The information contained in this e-mail is confidential and may be privileged. It may be read, copied and used only by the intended recipient. If you have received it in error, please contact the sender immediately by return e-mail. Please delete this e-mail and do not disclose its contents to any person. NIBC Holding N.V. nor its subsidiaries accept liability for any errors, omissions, delays of receipt or viruses in the contents of this message which arise as a result of e-mail transmission. NIBC Holding N.V. (Chamber of commerce nr. 27282935), NIBC Bank N.V. (Chamber of commerce nr. 27032036) and NIBC Investment Management N.V. (Chamber of commerce nr. 27253909) all have their corporate seat in The Hague, The Netherlands.

De informatie in dit e-mailbericht is vertrouwelijk en uitsluitend bestemd voor de geadresseerde. Wanneer u dit bericht per abuis ontvangt, gelieve onmiddellijk contact op te nemen met de afzender per kerende e-mail. Wij verzoeken u dit e-mailbericht te Vernietigen en de inhoud ervan aan niemand openbaar te maken. NIBC Holding N.V. noch haar dochterondernemingen aanvaarden enige aansprakelijkheid voor onjuiste, onvolledige dan wel ontijdige overbrenging van de inhoud van een verzonden e-mailbericht, noch voor door haar daarbij overgebrachte virussen. NIBC Holding N.V. (KvK nr. 27282935), NIBC Bank N.V. (KvK nr. 27032036) en NIBC Investment Management N.V. (KvK nr. 27253909) zijn statutair gevestigd te Den Haag, Nederland.