matplotlib-devel

Re: [matplotlib-devel] Experiments in removing/replacing PyCXX

[Damon M. <dam...@gm...>]

On Mon, Dec 3, 2012 at 12:12 PM, Chris Barker - NOAA Federal
<chr...@no...> wrote:
> generated code is ugly and hard to maintain, it is not designed to be
> human-readable, and we wouldn't get the advantages of bug-fixes
> further development in Cython.

As far as I'm concerned, this is an argument against Cython.

I've had to touch the C/C++/ObjC codebase. It was not automatically
generated by Cython and it's not that hard to read. There's almost
certainly a C/C++/ObjC expert around to help out. There's almost
certainly Cython experts to help out, too. There is almost certainly
*not* an expert in Cython-generated C code that is hard to read.

I vote raw Python/C API. Managing reference counters is not the
mundane task pythonistas make it out to be, in my opinion. If you know
ObjC, you've had to do your own reference counting. If you know C,
you've had to do your own memory management. If you know C++, you've
had to do your own new/delete (or destructor) management. I agree not
having to worry about reference counting is nice positive, but I don't
think it outweighs the negatives.

It seems to me that Cython is a 'middle-man' tool, with the added
downside of hard-to-maintain under-code.

-- 
Damon McDougall
http://www.damon-is-a-geek.com
Institute for Computational Engineering Sciences
201 E. 24th St.
Stop C0200
The University of Texas at Austin
Austin, TX 78712-1229

Re: [matplotlib-devel] Experiments in removing/replacing PyCXX

[Ryan M. <rm...@gm...>]

On Tue, Dec 4, 2012 at 4:07 PM, Damon McDougall
<dam...@gm...>wrote:

> On Mon, Dec 3, 2012 at 12:12 PM, Chris Barker - NOAA Federal
> <chr...@no...> wrote:
> > generated code is ugly and hard to maintain, it is not designed to be
> > human-readable, and we wouldn't get the advantages of bug-fixes
> > further development in Cython.
>
> As far as I'm concerned, this is an argument against Cython.
>
> I've had to touch the C/C++/ObjC codebase. It was not automatically
> generated by Cython and it's not that hard to read. There's almost
> certainly a C/C++/ObjC expert around to help out. There's almost
> certainly Cython experts to help out, too. There is almost certainly
> *not* an expert in Cython-generated C code that is hard to read.
>

You've had to touch the C/C++/ObjC because that's the only source that
exists; in this case that's the C *is* the implementation of the wrapper.

 If we go Cython, the cython source is all that is maintained. It may be
useful to glance at generated code, but no-one should be tweaking it by
hand--the Cython source, and only the Cython source, represents the
implementation of the wrapper.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma

Re: [matplotlib-devel] Experiments in removing/replacing PyCXX

[Eric F. <ef...@ha...>]

On 2012/12/04 12:07 PM, Damon McDougall wrote:
> On Mon, Dec 3, 2012 at 12:12 PM, Chris Barker - NOAA Federal
> <chr...@no...> wrote:
>> generated code is ugly and hard to maintain, it is not designed to be
>> human-readable, and we wouldn't get the advantages of bug-fixes
>> further development in Cython.
>
> As far as I'm concerned, this is an argument against Cython.

Nonsense.  It is an argument against the idea of maintaining the 
generated code directly, rather than maintaining the cython source code 
and regenerating the C code as needed.  That idea never made any sense 
in the first place.  I doubt that anyone follows it.  Chris already 
pointed this out.  Would you maintain the assembly code generated by 
your C++ compiler?  Do you consider the fact that this is unreadable and 
unmaintainable a reason to avoid using that compiler, and instead to 
code directly in assembly?

>
> I've had to touch the C/C++/ObjC codebase. It was not automatically
> generated by Cython and it's not that hard to read. There's almost
> certainly a C/C++/ObjC expert around to help out. There's almost
> certainly Cython experts to help out, too. There is almost certainly
> *not* an expert in Cython-generated C code that is hard to read.
>

There doesn't need to be.

> I vote raw Python/C API. Managing reference counters is not the
> mundane task pythonistas make it out to be, in my opinion. If you know
> ObjC, you've had to do your own reference counting. If you know C,
> you've had to do your own memory management. If you know C++, you've
> had to do your own new/delete (or destructor) management. I agree not
> having to worry about reference counting is nice positive, but I don't
> think it outweighs the negatives.

You have completely misrepresented the negatives.

>
> It seems to me that Cython is a 'middle-man' tool, with the added
> downside of hard-to-maintain under-code.
>

Please, if you don't use Cython yourself, and therefore don't know it 
well, refrain from these sorts of criticisms.  In normal cython use, one 
*never* modifies the code it generates.  In developing with cython, one 
*might* read this code to find out what is going on, and especially to 
find out whether one inadvertently triggered a call to the python API by 
forgetting to declare a variable, for example.  This is pretty easy, 
because the comments in the generated code show exactly which source 
line has generated each chunk of generated code.  Context is included. 
It is very nicely done.

Eric

Re: [matplotlib-devel] Experiments in removing/replacing PyCXX

[Michael D. <md...@st...>]

I think this has been a very helpful and useful discussion.

I'm going to attempt to summarize this discussion and propose some ways 
forward here.

The impetus for this discussion is that PyCXX seems to be not adequately 
maintained.  It is difficult to build matplotlib with "vanilla" PyCXX in 
certain configurations.  (This history sort of predates this thread).

So we have some options:

1) One way forward is to offer to take ownership of the PyCXX project.  
(I'm not using the "f" word here... I'd much prefer to just become more 
involved upstream somehow).  I don't think this would be considerable 
additional work, as most of that work has been done in matplotlib for 
some time anyway.  To the extent that it needs new features, it would be 
killer to add support for Numpy so Numpy no longer required manual 
reference counting.  I had initially dismissed this approach, as I seem 
to be in the minority in liking PyCXX -- I happen to think it's 
fundamentally an extremely good approach to the problem: it helps with 
reference counting errors, but otherwise mostly stays out of the way.  
But I'd like to remove any one person as a bottleneck by choosing 
something that's more preferred all around.

2) Move to a different wrapping mechanism of some sort.  While Cython is 
the clear choice for a third-party Python/C wrapping tool, it seems to 
be polarizing.  (I won't attempt to repeat or summarize, but I think 
good points have been made on either side of the argument).  I think 
it's ok to allow Cython to be used in matplotlib, given that we include 
both the Cython source and the generated C in the source repository such 
that matplotlib can be built without Cython installed.  There are many 
other projects doing this that can provide best practices for us.  I 
don't think, however, that we can or should require that all wrapping is 
done with Cython.  I think we should allow raw Python/C API where it is 
most appropriate (and that is mainly in the case of wrapping third-party 
libraries, such as the png module and the macosx module which is already 
raw Python/C API).  What I wouldn't want to see is the use of more than 
one wrapping tool, if only for reasons of proliferation of 
dependencies.  (I count the Python/C API as "free" since it's always 
available anyway).  I haven't seen in this discussion anyone really 
pushing for any of the alternatives (SWIG, Boost.Python, etc.) in any event.

Note also, the goal is to deal with the PyCXX "problem", not rewrite 
large chunks of our existing and well-tested C/C++ code base in Cython, 
unless someone sees a real clear benefit to doing that for a particular 
module and is highly motivated to do the work.  This is primarily about 
refactoring the code so that the interface layer between Python and C is 
separated and then replaced with either Cython or raw Python/C API using 
the most appropriate tool for the job.

3) Any other options...?

Cheers,
Mike

On 12/04/2012 05:33 PM, Eric Firing wrote:
> On 2012/12/04 12:07 PM, Damon McDougall wrote:
>> On Mon, Dec 3, 2012 at 12:12 PM, Chris Barker - NOAA Federal
>> <chr...@no...> wrote:
>>> generated code is ugly and hard to maintain, it is not designed to be
>>> human-readable, and we wouldn't get the advantages of bug-fixes
>>> further development in Cython.
>> As far as I'm concerned, this is an argument against Cython.
> Nonsense.  It is an argument against the idea of maintaining the
> generated code directly, rather than maintaining the cython source code
> and regenerating the C code as needed.  That idea never made any sense
> in the first place.  I doubt that anyone follows it.  Chris already
> pointed this out.  Would you maintain the assembly code generated by
> your C++ compiler?  Do you consider the fact that this is unreadable and
> unmaintainable a reason to avoid using that compiler, and instead to
> code directly in assembly?
>
>> I've had to touch the C/C++/ObjC codebase. It was not automatically
>> generated by Cython and it's not that hard to read. There's almost
>> certainly a C/C++/ObjC expert around to help out. There's almost
>> certainly Cython experts to help out, too. There is almost certainly
>> *not* an expert in Cython-generated C code that is hard to read.
>>
> There doesn't need to be.
>
>> I vote raw Python/C API. Managing reference counters is not the
>> mundane task pythonistas make it out to be, in my opinion. If you know
>> ObjC, you've had to do your own reference counting. If you know C,
>> you've had to do your own memory management. If you know C++, you've
>> had to do your own new/delete (or destructor) management. I agree not
>> having to worry about reference counting is nice positive, but I don't
>> think it outweighs the negatives.
> You have completely misrepresented the negatives.
>
>> It seems to me that Cython is a 'middle-man' tool, with the added
>> downside of hard-to-maintain under-code.
>>
> Please, if you don't use Cython yourself, and therefore don't know it
> well, refrain from these sorts of criticisms.  In normal cython use, one
> *never* modifies the code it generates.  In developing with cython, one
> *might* read this code to find out what is going on, and especially to
> find out whether one inadvertently triggered a call to the python API by
> forgetting to declare a variable, for example.  This is pretty easy,
> because the comments in the generated code show exactly which source
> line has generated each chunk of generated code.  Context is included.
> It is very nicely done.
>
> Eric
>
> ------------------------------------------------------------------------------
> LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
> Remotely access PCs and mobile devices and provide instant support
> Improve your efficiency, and focus on delivering more value-add services
> Discover what IT Professionals Know. Rescue delivers
> http://p.sf.net/sfu/logmein_12329d2d
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Experiments in removing/replacing PyCXX

[Nicolas R. <Nic...@in...>]

Just for completeness, there is also ctypes. I wrapped the freetype library (http://code.google.com/p/freetype-py/) using it and it is quite easy (and boring). But this only works for C (not C++).

Nicolas



On Dec 6, 2012, at 18:06 , Michael Droettboom wrote:

> I think this has been a very helpful and useful discussion.
> 
> I'm going to attempt to summarize this discussion and propose some ways 
> forward here.
> 
> The impetus for this discussion is that PyCXX seems to be not adequately 
> maintained.  It is difficult to build matplotlib with "vanilla" PyCXX in 
> certain configurations.  (This history sort of predates this thread).
> 
> So we have some options:
> 
> 1) One way forward is to offer to take ownership of the PyCXX project.  
> (I'm not using the "f" word here... I'd much prefer to just become more 
> involved upstream somehow).  I don't think this would be considerable 
> additional work, as most of that work has been done in matplotlib for 
> some time anyway.  To the extent that it needs new features, it would be 
> killer to add support for Numpy so Numpy no longer required manual 
> reference counting.  I had initially dismissed this approach, as I seem 
> to be in the minority in liking PyCXX -- I happen to think it's 
> fundamentally an extremely good approach to the problem: it helps with 
> reference counting errors, but otherwise mostly stays out of the way.  
> But I'd like to remove any one person as a bottleneck by choosing 
> something that's more preferred all around.
> 
> 2) Move to a different wrapping mechanism of some sort.  While Cython is 
> the clear choice for a third-party Python/C wrapping tool, it seems to 
> be polarizing.  (I won't attempt to repeat or summarize, but I think 
> good points have been made on either side of the argument).  I think 
> it's ok to allow Cython to be used in matplotlib, given that we include 
> both the Cython source and the generated C in the source repository such 
> that matplotlib can be built without Cython installed.  There are many 
> other projects doing this that can provide best practices for us.  I 
> don't think, however, that we can or should require that all wrapping is 
> done with Cython.  I think we should allow raw Python/C API where it is 
> most appropriate (and that is mainly in the case of wrapping third-party 
> libraries, such as the png module and the macosx module which is already 
> raw Python/C API).  What I wouldn't want to see is the use of more than 
> one wrapping tool, if only for reasons of proliferation of 
> dependencies.  (I count the Python/C API as "free" since it's always 
> available anyway).  I haven't seen in this discussion anyone really 
> pushing for any of the alternatives (SWIG, Boost.Python, etc.) in any event.
> 
> Note also, the goal is to deal with the PyCXX "problem", not rewrite 
> large chunks of our existing and well-tested C/C++ code base in Cython, 
> unless someone sees a real clear benefit to doing that for a particular 
> module and is highly motivated to do the work.  This is primarily about 
> refactoring the code so that the interface layer between Python and C is 
> separated and then replaced with either Cython or raw Python/C API using 
> the most appropriate tool for the job.
> 
> 3) Any other options...?
> 

Re: [matplotlib-devel] Experiments in removing/replacing PyCXX

[Michael D. <md...@st...>]

I think this has been a very helpful and useful discussion.

I'm going to attempt to summarize this discussion and propose some ways 
forward here.

The impetus for this discussion is that PyCXX seems to be not adequately 
maintained.  It is difficult to build matplotlib with "vanilla" PyCXX in 
certain configurations.  (This history sort of predates this thread).

So we have some options:

1) One way forward is to offer to take ownership of the PyCXX project. 
(I'm not using the "f" word here... I'd much prefer to just become more 
involved upstream somehow).  I don't think this would be considerable 
additional work, as most of that work has been done in matplotlib for 
some time anyway.  To the extent that it needs new features, it would be 
killer to add support for Numpy so Numpy no longer required manual 
reference counting.  I had initially dismissed this approach, as I seem 
to be in the minority in liking PyCXX -- I happen to think it's 
fundamentally an extremely good approach to the problem: it helps with 
reference counting errors, but otherwise mostly stays out of the way. 
But I'd like to remove any one person as a bottleneck by choosing 
something that's more preferred all around.

2) Move to a different wrapping mechanism of some sort.  While Cython is 
the clear choice for a third-party Python/C wrapping tool, it seems to 
be polarizing.  (I won't attempt to repeat or summarize, but I think 
good points have been made on either side of the argument).  I think 
it's ok to allow Cython to be used in matplotlib, given that we include 
both the Cython source and the generated C in the source repository such 
that matplotlib can be built without Cython installed.  There are many 
other projects doing this that can provide best practices for us.  I 
don't think, however, that we can or should require that all wrapping is 
done with Cython.  I think we should allow raw Python/C API where it is 
most appropriate (and that is mainly in the case of wrapping third-party 
libraries, such as the png module and the macosx module which is already 
raw Python/C API).  What I wouldn't want to see is the use of more than 
one wrapping tool, if only for reasons of proliferation of dependencies. 
  (I count the Python/C API as "free" since it's always available 
anyway).  I haven't seen in this discussion anyone really pushing for 
any of the alternatives (SWIG, Boost.Python, etc.) in any event.

Note also, the goal is to deal with the PyCXX "problem", not rewrite 
large chunks of our existing and well-tested C/C++ code base in Cython, 
unless someone sees a real clear benefit to doing that for a particular 
module and is highly motivated to do the work.  This is primarily about 
refactoring the code so that the interface layer between Python and C is 
separated and then replaced with either Cython or raw Python/C API using 
the most appropriate tool for the job.

3) Any other options...?

Cheers,
Mike

On 12/04/2012 05:33 PM, Eric Firing wrote:
> On 2012/12/04 12:07 PM, Damon McDougall wrote:
>> On Mon, Dec 3, 2012 at 12:12 PM, Chris Barker - NOAA Federal
>> <chr...@no...> wrote:
>>> generated code is ugly and hard to maintain, it is not designed to be
>>> human-readable, and we wouldn't get the advantages of bug-fixes
>>> further development in Cython.
>> As far as I'm concerned, this is an argument against Cython.
> Nonsense.  It is an argument against the idea of maintaining the
> generated code directly, rather than maintaining the cython source code
> and regenerating the C code as needed.  That idea never made any sense
> in the first place.  I doubt that anyone follows it.  Chris already
> pointed this out.  Would you maintain the assembly code generated by
> your C++ compiler?  Do you consider the fact that this is unreadable and
> unmaintainable a reason to avoid using that compiler, and instead to
> code directly in assembly?
>
>> I've had to touch the C/C++/ObjC codebase. It was not automatically
>> generated by Cython and it's not that hard to read. There's almost
>> certainly a C/C++/ObjC expert around to help out. There's almost
>> certainly Cython experts to help out, too. There is almost certainly
>> *not* an expert in Cython-generated C code that is hard to read.
>>
> There doesn't need to be.
>
>> I vote raw Python/C API. Managing reference counters is not the
>> mundane task pythonistas make it out to be, in my opinion. If you know
>> ObjC, you've had to do your own reference counting. If you know C,
>> you've had to do your own memory management. If you know C++, you've
>> had to do your own new/delete (or destructor) management. I agree not
>> having to worry about reference counting is nice positive, but I don't
>> think it outweighs the negatives.
> You have completely misrepresented the negatives.
>
>> It seems to me that Cython is a 'middle-man' tool, with the added
>> downside of hard-to-maintain under-code.
>>
> Please, if you don't use Cython yourself, and therefore don't know it
> well, refrain from these sorts of criticisms.  In normal cython use, one
> *never* modifies the code it generates.  In developing with cython, one
> *might* read this code to find out what is going on, and especially to
> find out whether one inadvertently triggered a call to the python API by
> forgetting to declare a variable, for example.  This is pretty easy,
> because the comments in the generated code show exactly which source
> line has generated each chunk of generated code.  Context is included.
> It is very nicely done.
>
> Eric
>
> ------------------------------------------------------------------------------
> LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
> Remotely access PCs and mobile devices and provide instant support
> Improve your efficiency, and focus on delivering more value-add services
> Discover what IT Professionals Know. Rescue delivers
> http://p.sf.net/sfu/logmein_12329d2d
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Experiments in removing/replacing PyCXX

[Eric F. <ef...@ha...>]

On 2012/12/06 7:16 AM, Michael Droettboom wrote:
> I think this has been a very helpful and useful discussion.
>
> I'm going to attempt to summarize this discussion and propose some ways
> forward here.
>
> The impetus for this discussion is that PyCXX seems to be not adequately
> maintained.  It is difficult to build matplotlib with "vanilla" PyCXX in
> certain configurations.  (This history sort of predates this thread).
>
> So we have some options:
>
> 1) One way forward is to offer to take ownership of the PyCXX project.
> (I'm not using the "f" word here... I'd much prefer to just become more
> involved upstream somehow).  I don't think this would be considerable
> additional work, as most of that work has been done in matplotlib for
> some time anyway.  To the extent that it needs new features, it would be
> killer to add support for Numpy so Numpy no longer required manual
> reference counting.  I had initially dismissed this approach, as I seem
> to be in the minority in liking PyCXX -- I happen to think it's
> fundamentally an extremely good approach to the problem: it helps with
> reference counting errors, but otherwise mostly stays out of the way.
> But I'd like to remove any one person as a bottleneck by choosing
> something that's more preferred all around.
>
> 2) Move to a different wrapping mechanism of some sort.  While Cython is
> the clear choice for a third-party Python/C wrapping tool, it seems to
> be polarizing.  (I won't attempt to repeat or summarize, but I think
> good points have been made on either side of the argument).  I think
> it's ok to allow Cython to be used in matplotlib, given that we include
> both the Cython source and the generated C in the source repository such
> that matplotlib can be built without Cython installed.  There are many
> other projects doing this that can provide best practices for us.  I
> don't think, however, that we can or should require that all wrapping is
> done with Cython.  I think we should allow raw Python/C API where it is
> most appropriate (and that is mainly in the case of wrapping third-party
> libraries, such as the png module and the macosx module which is already
> raw Python/C API).  What I wouldn't want to see is the use of more than
> one wrapping tool, if only for reasons of proliferation of dependencies.
>    (I count the Python/C API as "free" since it's always available
> anyway).  I haven't seen in this discussion anyone really pushing for
> any of the alternatives (SWIG, Boost.Python, etc.) in any event.
>
> Note also, the goal is to deal with the PyCXX "problem", not rewrite
> large chunks of our existing and well-tested C/C++ code base in Cython,
> unless someone sees a real clear benefit to doing that for a particular
> module and is highly motivated to do the work.  This is primarily about
> refactoring the code so that the interface layer between Python and C is
> separated and then replaced with either Cython or raw Python/C API using
> the most appropriate tool for the job.
>
> 3) Any other options...?
>
> Cheers,
> Mike

Mike,

That is an excellent summary.  The options actually are not mutually 
exclusive; perhaps what we are considering is more in the line of 
nudging evolution in one direction or the other, not pushing for rapid 
extinction of a species.

Regarding PyCXX, I respect your opinion that it is a good match for what 
it does.  To the limited extent that I can work with C++ at all, I don't 
have any problem with its use in mpl.  I do share the concern about 
depending heavily on it, given that problems with it have cropped up, 
and you have been the only one willing and able to deal with those 
problems.  Since PyCXX is a pure C++ construct, perhaps other C++ 
gurus--and it seems that we now have more than previously--would be 
willing to take a closer look at it, and reconsider whether they can 
relieve the single-person-bottleneck problem.

There is always a tradeoff between going to a higher-level language or 
library versus sticking with lower levels.  Personally, I like C over 
C++ because the former is simple enough that I can generally figure out 
what is going on; but I like Cython over raw Python/C API because its 
internal complexity allows an external simplicity, hiding all sorts of 
things I really don't want to have to think about.  Going to higher 
levels always brings the risk of dependency on a complex system, whether 
it be Cython or PyCXX or Agg, or even the C/C++ compiler itself. The 
cost/benefit ratios of such tradeoffs vary greatly with the situation, 
and from person to person, depending on training, experience, and 
personal quirks.  So we just have to keep looking for the balance that 
is appropriate to the task, the times, and the people at hand.  Your 
summary nicely facilitates that balancing act.

Eric

Re: [matplotlib-devel] Experiments in removing/replacing PyCXX

[Ian T. <ian...@gm...>]

Yes, an excellent summary and neatly bringing us back to the crux of the
matter.

For completeness I should say that I wouldn't use SWIG.  I used it about 5
years ago to wrap some C++ for Python and other languages.  Initially it
was very useful, but eventually the default mapping between C++ and Python
types and the default handling of object lifetimes weren't quite what I
wanted and I found myself writing a lot of extra configuration code to coax
it back in line.  In the end I went back to the Python/C API.  Perhaps its
aim of targetting many different languages means it isn't suited for our
language of interest.  It doesn't support Numpy arrays anyway.

I would like to be able to recommend Boost.Python, but I have never used
it.  I have used some Boost modules and found all to be well-designed and
actively maintained.  However, it currently doesn't support Numpy arrays
(although it is an active area of work) and it appears that there are
difficulties building it with anything other than the default build tool
BJam leading to concerns over portability.

Although my preference, in the absence of PyCXX, for wrapping our larger
C/C++ modules is to use the Python/C API rather than Cython, I have been
persuaded that there is a place for Cython in matplotlib.  The ability to
improve the performance of small sections of Python code in a simple and
localised manner seems very useful, and would be a good starting point for
speeding up areas of code that a number of users are frustrated by.  Given
the number of people who would like to see it used in matplotlib, I think
it is inevitable that we eventually will.

I hadn't really considered the option of adding Numpy arrays to PyCXX.
I've taken a quick look at the existing code and whilst I don't think it is
a trivial task it looks well worth investigating - the existing code is
well organised and quite well documented.  If two or more of us were
prepared to make the Numpy additions to PyCXX and provide ongoing
maintenance, we would address the deficiencies of the current solution and
remove the single-person bottleneck.  But I am not sure where this leaves
us as I a now advocating use of Cython to some extent and hence we would
have two wrapping tools.  Should we reject Cython + improved PyCXX on these
grounds and revert to Cython + Python/C API?

Ian

Re: [matplotlib-devel] Experiments in removing/replacing PyCXX

[Jason G. <jas...@cr...>]

On 11/29/12 10:59 AM, Michael Droettboom wrote:
> I've not had
> much luck with Cython for this kind of thing in the past, but I know it
> is popular.

I'm curious about what problems you've run into and how long it was.  In 
the past, Cython hasn't supported C++ very well, but the situation has 
greatly improved recently.  See 
http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html for some 
details.

Thanks,

Jason

Re: [matplotlib-devel] Experiments in removing/replacing PyCXX

[Michael D. <md...@st...>]

For the PNG extension specifically, it was creating callbacks that can 
be called from C and the setjmp magic that libpng requires.  I think 
it's possible to do it, but I was surprised at how non-obvious those 
pieces of Cython were.  I was really hoping by creating this experiment 
that a Cython expert would step up and show the way ;)

The Agg backend has more C++-specific challenges, particularly 
instantiating very complex template expressions -- but I haven't really 
followed that on through.

Mike

On 11/30/2012 09:13 AM, Jason Grout wrote:
> On 11/29/12 10:59 AM, Michael Droettboom wrote:
>> I've not had
>> much luck with Cython for this kind of thing in the past, but I know it
>> is popular.
> I'm curious about what problems you've run into and how long it was.  In
> the past, Cython hasn't supported C++ very well, but the situation has
> greatly improved recently.  See
> http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html for some
> details.
>
> Thanks,
>
> Jason
>
>
> ------------------------------------------------------------------------------
> Keep yourself connected to Go Parallel:
> TUNE You got it built. Now make it sing. Tune shows you how.
> http://goparallel.sourceforge.net
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Experiments in removing/replacing PyCXX

[Jason G. <jas...@cr...>]

On 12/1/12 12:36 PM, Michael Droettboom wrote:
> I was really hoping by creating this experiment
> that a Cython expert would step up and show the way ;)

There are a lot more Cython experts on the Cython mailing list ;).

Thanks,

Jason

Re: [matplotlib-devel] Experiments in removing/replacing PyCXX

[Michael D. <md...@st...>]

For point of comparison, my branch now has a Cython and C++ version of 
the same thing.

Here's the Cython version:

https://github.com/mdboom/matplotlib/blob/no_cxx/src/_png.pyx

Here's the C++ version:

https://github.com/mdboom/matplotlib/blob/no_cxx/src/_png_wrap.cpp

Some interesting things to note:

The Cython version isn't that much shorter than the C++ version. It 
mostly consists of declarations.  These declarations aren't exact 
matches to what one would find in the header file(s) because Cython 
doesn't support exact-width data types etc.  The Cython documentation 
says "not to worry", but I do wonder how well this will work across 
different architectures etc.  I'm not sure why some of the Python/C API 
calls I needed were not defined in Cython's include wrappers.

The Cython extension only builds with "-fpermissive" because I can't 
seem to get the casts and const coercions to work.  Maybe there's a 
simple solution...

The exception handling in the png_core.cpp file will need to be updated 
because Cython only supports handling built-in C++ exceptions (and 
subclasses), and the Cython custom exception handler doesn't provide a 
way to get at the exception object that was thrown.

It seems that a lot of things pass through the Cython compiler, but then 
fail in the C compiler -- you then have to wade through the generated C 
code to figure out what's going wrong.  This reminds me of the bad old 
days of C++ when the error messages generated would be dozens of lines 
long and rather inscrutable.

Once things compiled, due to my own mistake, calling the function 
segfaulted.  Debugging that segfault in gdb required, again, wading 
through the generated code.  Using gdb on hand-written code is *much* nicer.

So, it's probably clear that I'm not much of a fan of this approach, but 
I am trying to find something that the whole community around matplotlib 
finds easier and more accessible so that the C/C++ experts among use are 
not exclusively burdened to maintain this part of the code base.  I 
would be interested to see what others think now that we have an 
apples-to-apples comparison.

Cheers,
Mike

On 12/01/2012 01:36 PM, Michael Droettboom wrote:
> For the PNG extension specifically, it was creating callbacks that can
> be called from C and the setjmp magic that libpng requires.  I think
> it's possible to do it, but I was surprised at how non-obvious those
> pieces of Cython were.  I was really hoping by creating this experiment
> that a Cython expert would step up and show the way ;)
>
> The Agg backend has more C++-specific challenges, particularly
> instantiating very complex template expressions -- but I haven't really
> followed that on through.
>
> Mike
>
> On 11/30/2012 09:13 AM, Jason Grout wrote:
>> On 11/29/12 10:59 AM, Michael Droettboom wrote:
>>> I've not had
>>> much luck with Cython for this kind of thing in the past, but I know it
>>> is popular.
>> I'm curious about what problems you've run into and how long it was.  In
>> the past, Cython hasn't supported C++ very well, but the situation has
>> greatly improved recently.  See
>> http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html for some
>> details.
>>
>> Thanks,
>>
>> Jason
>>
>>
>> ------------------------------------------------------------------------------
>> Keep yourself connected to Go Parallel:
>> TUNE You got it built. Now make it sing. Tune shows you how.
>> http://goparallel.sourceforge.net
>> _______________________________________________
>> Matplotlib-devel mailing list
>> Mat...@li...
>> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>
> ------------------------------------------------------------------------------
> Keep yourself connected to Go Parallel:
> INSIGHTS What's next for parallel hardware, programming and related areas?
> Interviews and blogs by thought leaders keep you ahead of the curve.
> http://goparallel.sourceforge.net
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Experiments in removing/replacing PyCXX

[Ian T. <ian...@gm...>]

I vote for using the raw Python/C API.  I've written a couple of PyCXX
extensions and whilst it is mostly convenient, PyCXX doesn't support the
use of numpy arrays so for them you have to use the Python/C API.  This
means dealing with the reference counting yourself for numpy arrays;
extending this to do the reference counting for all python objects is not
onerous.  Dealing with object lifetimes is bread-and-butter work for C/C++
developers.

I have never used Cython, but to me the code looks like an inelegant
combination of Python, C/C++ and some Cython-specific stuff.  I can see the
advantage of this approach for small sections of code, but I have strong
reservations about using it for complicated modules that have extensive use
of templated code and/or Standard Template Library collections (mpl has
examples of both of these).

I agree that Cython opens us up to a larger body of contributors, but I
don't think that this is necessarily a good thing.  I think this really
means opens us up to a larger body of Python/Cython contributors, and is a
view expressed from the Python side of the fence and has the wrong
emphasis.  I am primarily a C++ developer is a sea of Python developers,
and rather than encourage other Python contributors to dip their toes into
C/C++ via Cython I think we should be encouraging C/C++ contributors to do
what they do best.  We only need a few C/C++ developers if we allow them to
use their skills in their preferred way, and they are used to interfacing
to legacy APIs and dealing with object lifetimes.

OK, cards on the table.  If we wanted to switch all of our PyCXX modules to
use the raw Python/C API, I would happily take on some of the burden for
making the changes and ongoing maintenance of such modules.  Particularly
if, in return, I get some help with my sometimes substandard Python!  If we
go down the Cython route I couldn't make this offer; would our many Cython
advocates take on the responsibility of changing and maintaining my C++
code in this scenario?

Ian Thomas

Re: [matplotlib-devel] Experiments in removing/replacing PyCXX

[Michael D. <md...@st...>]

On 12/03/2012 04:07 AM, Ian Thomas wrote:
> I vote for using the raw Python/C API.  I've written a couple of PyCXX 
> extensions and whilst it is mostly convenient, PyCXX doesn't support 
> the use of numpy arrays so for them you have to use the Python/C API.  
> This means dealing with the reference counting yourself for numpy 
> arrays; extending this to do the reference counting for all python 
> objects is not onerous.  Dealing with object lifetimes is 
> bread-and-butter work for C/C++ developers.

That matches my experience quite well.

> I have never used Cython, but to me the code looks like an inelegant 
> combination of Python, C/C++ and some Cython-specific stuff.  I can 
> see the advantage of this approach for small sections of code, but I 
> have strong reservations about using it for complicated modules that 
> have extensive use of templated code and/or Standard Template Library 
> collections (mpl has examples of both of these).

Even for C libraries like libpng, which requires use of C function 
callbacks for some things, Cython is more convoluted, particularly when 
things go wrong and require debugging.  (Running gdb over generated 
Cython code is not fun!)  And in my view, writing code like that 
requires a pretty deep understanding of the Python/C API, C itself, and 
the rather complex transformations that Cython performs.  Writing 
directly to the Python/C API only requires knowledge of the first two.  
And there's a large body of books/tutorials/debuggers/tools for C that 
don't really have equivalents for Cython.

>
> I agree that Cython opens us up to a larger body of contributors, but 
> I don't think that this is necessarily a good thing.  I think this 
> really means opens us up to a larger body of Python/Cython 
> contributors, and is a view expressed from the Python side of the 
> fence and has the wrong emphasis.  I am primarily a C++ developer is a 
> sea of Python developers, and rather than encourage other Python 
> contributors to dip their toes into C/C++ via Cython I think we should 
> be encouraging C/C++ contributors to do what they do best.  We only 
> need a few C/C++ developers if we allow them to use their skills in 
> their preferred way, and they are used to interfacing to legacy APIs 
> and dealing with object lifetimes.

I think Cython is well suited to writing new algorithmic code to speed 
up hot spots in Python code.  I don't think it's as well suited as glue 
between C and Python -- that was not a main goal of the original Pyrex 
project, IIRC.  It feels kind of tacked on and not a very good fit to 
the problem.  Most of the work to remove PyCXX use in matplotlib is 
either wrapping third-party libraries (where Cython doesn't really 
shine), or wrapping C/C++ code in our own tree that's already 
well-tested and vetted, and I wouldn't propose rewriting that in 
Cython.  I'm only really considering rewriting the Python-to-C interface 
layer.

>
> OK, cards on the table.  If we wanted to switch all of our PyCXX 
> modules to use the raw Python/C API, I would happily take on some of 
> the burden for making the changes and ongoing maintenance of such 
> modules.  Particularly if, in return, I get some help with my 
> sometimes substandard Python!  If we go down the Cython route I 
> couldn't make this offer; would our many Cython advocates take on the 
> responsibility of changing and maintaining my C++ code in this scenario?

That's a good way to look at this.  I was definitely hoping that moving 
to Cython might open us up to more developers, but at the end of the 
day, the chosen tool should be the one preferred by those doing the 
work.  Maybe rather than asking "if we switched to using Cython, would 
more participate", I should be asking "among those that can participate 
in removing the PyCXX dependency, what is the preferred approach?"

Cheers,
Mike

Re: [matplotlib-devel] Experiments in removing/replacing PyCXX

[Eric F. <ef...@ha...>]

On 2012/12/03 4:54 AM, Michael Droettboom wrote:
> I think Cython is well suited to writing new algorithmic code to speed
> up hot spots in Python code.  I don't think it's as well suited as glue
> between C and Python -- that was not a main goal of the original Pyrex
> project, IIRC.  It feels kind of tacked on and not a very good fit to
> the problem.

Not entirely relevant to the PyCXX discussion, but to avoid misleading 
others reading this discussion, I must strongly disagree with your 
assertion about Cython's usefulness for wrapping C libraries or small 
chunks of C.  I think this has always been a primary function of Cython 
and Pyrex, as far back as I have been aware of them.  I wrote the raw 
interface to our contouring code, and I have written cython interfaces 
to various chunks of C outside of mpl; and cython makes it much easier 
for a non-professional programmer such as myself.  So I am not arguing 
that Cython should be the choice for removing PyCXX, but for 
non-wizards, it can work very well as glue.  It is much more 
approachable than any alternative of which I am aware.  For Fortran, of 
course, f2py plays this glue code generation role.

Eric