Re: [Matplotlib-users] Segmentation fault under Agg

[John H. <jd...@gm...>]

On Wed, Sep 23, 2009 at 10:15 PM, John Hunter <jd...@gm...> wrote:
> On Wed, Sep 23, 2009 at 6:47 PM, Mike Nicolls <mic...@sr...> wrote:
>> I've encountered a segmentation fault using the Agg backend when making
>> and saving a png plot.  The segmentation fault does not seem to occur
>> under different backends (e.g. Cairo).
>>
>> The sequence of commands amounts to making a plot, setting the limits of
>> the plot, then saving the plot as a png using savefig.  If the value of
>> one of the points is too large, a segmentation fault is generated.
>> Obviously the simple work-around is to not plot the out-of-bounds
>> points, but the segmentation fault is probably not the desired behavior!
>>
>> Below is an example, which generates a segmentation fault on matplotlib
>> versions 0.99.0 (run on OS X 10.5.8) and 0.98.5.2 (run on Fedora 11 and
>> CentOS 5.3).
>>
>> Can others reproduce this issue?
>
> Thanks for the report -- I can confirm this on OSX and linux on svn HEAD
>
> I've made minor tweaks to your test case to make it deterministic::

I applied the following patch to the branch and head -- it feels like there
would be a more elegant way to handle it, so I'd like Michael at least to
take a look at it::

home:~/mpl> svn diff -rPREV:HEAD src/_backend_agg.cpp
Index: src/_backend_agg.cpp
===================================================================
--- src/_backend_agg.cpp (revision 7820)
+++ src/_backend_agg.cpp (revision 7821)
@@ -608,6 +608,14 @@

x = (double)(int)x; y = (double)(int)y;

+ // if x or y is close to the width or height, the filled
+ // region could be inside the boundary even if the center is
+ // out. But at some point we need to cull these points
+ // because they can create segfaults of they overflow; eg
+ //
https://sourceforge.net/tracker/?func=detail&aid=2865490&group_id=80706&atid=560720
+ if (fabs(x)>(2*width)) continue;
+ if (fabs(y)>(2*height)) continue;
+
pixfmt_amask_type pfa(pixFmt, alphaMask);
amask_ren_type r(pfa);
amask_aa_renderer_type ren(r);
@@ -629,6 +637,14 @@

x = (double)(int)x; y = (double)(int)y;

+ // if x or y is close to the width or height, the filled
+ // region could be inside the boundary even if the center is
+ // out. But at some point we need to cull these points
+ // because they can create segfaults of they overflow; eg
+ //
https://sourceforge.net/tracker/?func=detail&aid=2865490&group_id=80706&atid=560720
+ if (fabs(x)>(2*width)) continue;
+ if (fabs(y)>(2*height)) continue;
+
if (face.first) {
rendererAA.color(face.second);
sa.init(fillCache, fillSize, x, y);