matplotlib-devel

[matplotlib-devel] Planning for 1.3.0

[Michael D. <md...@st...>]

Congrats to everyone on a successful 1.2.1 -- there was a relatively 
small influx of bug reports following it -- perhaps a sign of improving 
quality?

In any event, as Phil Elson likes to repeatedly point out (<wink>), we 
have a great deal of awesome new features in master that it would be 
great to get out.  As for time frame, I think if we aim for a 1.3.0rc1 
of May 27, that's within a good time frame to get something out in time 
for Scipy.  That will put us about 8 months past 1.2.0, which is not far 
off from my eventual goal of 6 month releases once things get 
streamlined.  We can, of course, adjust the date as necessary as we go 
along.

So... let the bug triaging begin!  As there are lot of new features in 
the queue, I think it's important to distinguish the "nice to have" from 
the "must have".  I've created a new milestone "1.3.x blocker" that we 
should use for critical bugs that should hold up the release.  All other 
new features that are looking close to ready for 1.3.x should be 
milestoned 1.3.x, and as we get closer, we can punt those on to 1.4.x.  
Simple bugs that apply to 1.2.x as well as master should be milestoned 
as "1.2.x" and merged onto the "v1.2.x" branch (as we've been doing), as 
I still think we should reserve the right to make another 1.2.2 bugfix 
release if necessary.

There are a couple major ongoing projects that I think we'll need to get 
to a steady interim state before we can make another release.

   MEP10: Documentation

      We already have a number of improved docstrings, and better 
organization throughout.  I don't think we need to finish all of this 
work before the next release, but we should get it back into shape so 
that the doc build has fewer warnings (#1896) and the LaTeX build works 
again (#1836).

   MEP12: Reorganize and cleanup examples

      Again, I think a lot of great work has already been done, and we 
don't necessarily have to wait until it's done.

For both of the above, it would be nice to divide the work up so the 
leaders of those projects are less individually burdened.  Nelle and 
Tony, if you know of any critical blockers or loose ends that should be 
tied up before a release should be made, please make issues for them and 
milestone them as "1.3.x blocker".

Cheers,
Mike

Re: [matplotlib-devel] Planning for 1.3.0

[Phil E. <pel...@gm...>]

Great news! A lot of fantastic work has been done by a whole host of people
to go into this release. It's exciting stuff!

May 27th sounds like a sensible target to me. As you know, I'm an advocate
of releasing often - the more frequently we make a release, the less we
will have the "impending release rush" that can really hamper the whole
release process.

Cheers,




On 11 April 2013 17:38, Michael Droettboom <md...@st...> wrote:

> Congrats to everyone on a successful 1.2.1 -- there was a relatively
> small influx of bug reports following it -- perhaps a sign of improving
> quality?
>
> In any event, as Phil Elson likes to repeatedly point out (<wink>), we
> have a great deal of awesome new features in master that it would be
> great to get out.  As for time frame, I think if we aim for a 1.3.0rc1
> of May 27, that's within a good time frame to get something out in time
> for Scipy.  That will put us about 8 months past 1.2.0, which is not far
> off from my eventual goal of 6 month releases once things get
> streamlined.  We can, of course, adjust the date as necessary as we go
> along.
>
> So... let the bug triaging begin!  As there are lot of new features in
> the queue, I think it's important to distinguish the "nice to have" from
> the "must have".  I've created a new milestone "1.3.x blocker" that we
> should use for critical bugs that should hold up the release.  All other
> new features that are looking close to ready for 1.3.x should be
> milestoned 1.3.x, and as we get closer, we can punt those on to 1.4.x.
> Simple bugs that apply to 1.2.x as well as master should be milestoned
> as "1.2.x" and merged onto the "v1.2.x" branch (as we've been doing), as
> I still think we should reserve the right to make another 1.2.2 bugfix
> release if necessary.
>
> There are a couple major ongoing projects that I think we'll need to get
> to a steady interim state before we can make another release.
>
>    MEP10: Documentation
>
>       We already have a number of improved docstrings, and better
> organization throughout.  I don't think we need to finish all of this
> work before the next release, but we should get it back into shape so
> that the doc build has fewer warnings (#1896) and the LaTeX build works
> again (#1836).
>
>    MEP12: Reorganize and cleanup examples
>
>       Again, I think a lot of great work has already been done, and we
> don't necessarily have to wait until it's done.
>
> For both of the above, it would be nice to divide the work up so the
> leaders of those projects are less individually burdened.  Nelle and
> Tony, if you know of any critical blockers or loose ends that should be
> tied up before a release should be made, please make issues for them and
> milestone them as "1.3.x blocker".
>
> Cheers,
> Mike
>
>
> ------------------------------------------------------------------------------
> Precog is a next-generation analytics platform capable of advanced
> analytics on semi-structured data. The platform includes APIs for building
> apps and a phenomenal toolset for data science. Developers can use
> our toolset for easy data analysis & visualization. Get a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>

Re: [matplotlib-devel] Planning for 1.3.0

[Derek H. <de...@as...>]

On 11.04.2013, at 6:38PM, Michael Droettboom <md...@st...> wrote:

> Congrats to everyone on a successful 1.2.1 -- there was a relatively 
> small influx of bug reports following it -- perhaps a sign of improving 
> quality?

Thanks and congratulations to everyone involved as well; I've built 1.2.1 on MacOS X with fink for 
Python2.6-3.2 without any failures in the test suite!
I did run into a problem though that has actually existed since the first 1.2 release - with the MacOSX
backend line plots of somewhat larger arrays with significant "high-frequency" power had extremely
degraded, e.g. something like

x = np.linspace(0,10,1.e6)
y = np.cos(x)+0.2*np.sin(x*3.e3)+0.1*np.cos(x*4.e4)*np.random.rand(1.e6)
plt.plot(x, y)

would display within less than 2 seconds with 1.1.1, but with 1.2.x you literally have to wait minutes,
and it takes similarly long to zoom in as long as you have a substantial part of the line in the window.

I found in the current HEAD (9e477b3) this has finally been fixed - thanks for that as well, whatever
the problem was, but now in the 1.3 branch the _macosx backend has been altogether disabled!
I verified after removing that RuntimeError from _macosx.m that the backend still works and is indeed
up to its old speed; but if that change stays in, it won't be usable from non-framework Python installs
like the fink ones. 
Personally, I am aware of the problems with the missing window manager control, and occasionally
am annoyed by hunting for a plot window that has sneaked somewhere underneath other windows,
but with that in mind I still prefer the MacOSX backend to any of the others, and I would suggest to
leave it at a warning rather than an error, so users can still decide for themselves if they want to put
up with the possible troubles.

Cheers,
							Derek

Re: [matplotlib-devel] Planning for 1.3.0

[Michiel de H. <mjl...@ya...>]

Hi Derek,

The slow speed for long paths like the one in your example was due to a limitation to Quartz itself. This was solved by breaking the path up into subpaths of up to 100 points. But you mentioned that releases before 1.2 were not slow (and I verified this with matplotlib 1.1.1), suggesting that something else is going on. Can you check which change between 1.1.1 and 1.2 is causing the slowdown for your example?

Best,
-Michiel.

--- On Fri, 4/12/13, Derek Homeier <de...@as...> wrote:

> From: Derek Homeier <de...@as...>
> Subject: Re: [matplotlib-devel] Planning for 1.3.0
> To: "mat...@li... list" <mat...@li...>
> Date: Friday, April 12, 2013, 5:16 PM
> On 11.04.2013, at 6:38PM, Michael
> Droettboom <md...@st...>
> wrote:
> 
> > Congrats to everyone on a successful 1.2.1 -- there was
> a relatively 
> > small influx of bug reports following it -- perhaps a
> sign of improving 
> > quality?
> 
> Thanks and congratulations to everyone involved as well;
> I've built 1.2.1 on MacOS X with fink for 
> Python2.6-3.2 without any failures in the test suite!
> I did run into a problem though that has actually existed
> since the first 1.2 release - with the MacOSX
> backend line plots of somewhat larger arrays with
> significant "high-frequency" power had extremely
> degraded, e.g. something like
> 
> x = np.linspace(0,10,1.e6)
> y =
> np.cos(x)+0.2*np.sin(x*3.e3)+0.1*np.cos(x*4.e4)*np.random.rand(1.e6)
> plt.plot(x, y)
> 
> would display within less than 2 seconds with 1.1.1, but
> with 1.2.x you literally have to wait minutes,
> and it takes similarly long to zoom in as long as you have a
> substantial part of the line in the window.
> 
> I found in the current HEAD (9e477b3) this has finally been
> fixed - thanks for that as well, whatever
> the problem was, but now in the 1.3 branch the _macosx
> backend has been altogether disabled!
> I verified after removing that RuntimeError from _macosx.m
> that the backend still works and is indeed
> up to its old speed; but if that change stays in, it won't
> be usable from non-framework Python installs
> like the fink ones. 
> Personally, I am aware of the problems with the missing
> window manager control, and occasionally
> am annoyed by hunting for a plot window that has sneaked
> somewhere underneath other windows,
> but with that in mind I still prefer the MacOSX backend to
> any of the others, and I would suggest to
> leave it at a warning rather than an error, so users can
> still decide for themselves if they want to put
> up with the possible troubles.
> 
> Cheers,
>            
>            
>     Derek
> 
> 
> ------------------------------------------------------------------------------
> Precog is a next-generation analytics platform capable of
> advanced
> analytics on semi-structured data. The platform includes
> APIs for building
> apps and a phenomenal toolset for data science. Developers
> can use
> our toolset for easy data analysis & visualization. Get
> a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
> 

Re: [matplotlib-devel] Planning for 1.3.0

[Derek H. <de...@as...>]

Hi Michiel,

On 13.04.2013, at 1:30AM, Michiel de Hoon wrote:

> The slow speed for long paths like the one in your example was due to a limitation to Quartz itself. This was solved by breaking the path up into subpaths of up to 100 points. But you mentioned that releases before 1.2 were not slow (and I verified this with matplotlib 1.1.1), suggesting that something else is going on. Can you check which change between 1.1.1 and 1.2 is causing the slowdown for your example?

It's the passing of set_dpi (commit 6533674) - that's still unchanged in master,
but I don't see any speed penalty compared to 1.1.1 any more. I don't know if
the change you mentioned above completely fixed this or just made up for it
by speeding it up otherwise…
I have just merged all updates to backend_maxosx.py and _macosx.m back
into 1.2.1, and this seems to solve the issue and passes all tests as well.

Cheers,
					Derek

Re: [matplotlib-devel] Planning for 1.3.0

[Michiel de H. <mjl...@ya...>]

Hi Derek,

That is good to hear.
The slowdown was caused by the performance of Quartz itself, but it depends strongly on the line width. In your example, the plot appears immediately if you use linewidth=0.9, but (with matplotlib 1.2.1) takes minutes to appear if you use linewidth=1.0. The change in set_dpi caused the line width actually used for drawing to increase slightly. The increase was very small, but big enough to trigger the ultraslow behavior of Quartz. As I mentioned, we solved this by breaking up the path into many subpaths, which solved the problem (without having to change set_dpi back).
Anyway, if I understand your mail correctly, the problem has been fixed in HEAD. Is the 1.3 branch also OK now? In your first post you mentioned that there was some RuntimeError.

Best,
-Michiel.

--- On Sat, 4/13/13, Derek Homeier <de...@as...> wrote:

> From: Derek Homeier <de...@as...>
> Subject: Re: [matplotlib-devel] Planning for 1.3.0
> To: "matplotlib development list" <mat...@li...>
> Date: Saturday, April 13, 2013, 9:03 AM
> Hi Michiel,
> 
> On 13.04.2013, at 1:30AM, Michiel de Hoon wrote:
> 
> > The slow speed for long paths like the one in your
> example was due to a limitation to Quartz itself. This was
> solved by breaking the path up into subpaths of up to 100
> points. But you mentioned that releases before 1.2 were not
> slow (and I verified this with matplotlib 1.1.1), suggesting
> that something else is going on. Can you check which change
> between 1.1.1 and 1.2 is causing the slowdown for your
> example?
> 
> It's the passing of set_dpi (commit 6533674) - that's still
> unchanged in master,
> but I don't see any speed penalty compared to 1.1.1 any
> more. I don't know if
> the change you mentioned above completely fixed this or just
> made up for it
> by speeding it up otherwise…
> I have just merged all updates to backend_maxosx.py and
> _macosx.m back
> into 1.2.1, and this seems to solve the issue and passes all
> tests as well.
> 
> Cheers,
>            
>         Derek
> 
> 
> ------------------------------------------------------------------------------
> Precog is a next-generation analytics platform capable of
> advanced
> analytics on semi-structured data. The platform includes
> APIs for building
> apps and a phenomenal toolset for data science. Developers
> can use
> our toolset for easy data analysis & visualization. Get
> a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
> 

Re: [matplotlib-devel] Planning for 1.3.0

[Derek H. <de...@as...>]

Hi Michiel,

> That is good to hear.
> The slowdown was caused by the performance of Quartz itself, but it depends strongly on the line width. In your example, the plot appears immediately if you use linewidth=0.9, but (with matplotlib 1.2.1) takes minutes to appear if you use linewidth=1.0. The change in set_dpi caused the line width actually used for drawing to increase slightly. The increase was very small, but big enough to trigger the ultraslow behavior of Quartz. As I mentioned, we solved this by breaking up the path into many subpaths, which solved the problem (without having to change set_dpi back).
> Anyway, if I understand your mail correctly, the problem has been fixed in HEAD. Is the 1.3 branch also OK now? In your first post you mentioned that there was some RuntimeError.

I saw a couple of warnings with Friday's checkout on 10.8, but the current one seems to
work fine (now on 10.7 however…). I've run the full test suite and only had three failures
in test_font_styles (basically all created fonts look like 'light'/'condensed').
The same with python3.2 after I upgraded pyparsing, only at the end of 'setup.py install'
there was an additional error, but this did not seem to affect the install (appended below).

The RuntimeError was enforced by the #ifdef WITH_NEXT_FRAMEWORK check that
does not allow to use the backend at all, so I had to change this to a RuntimeWarning
to be able to test the backend in the 1.3 branch.

Cheers,
					Derek

--
Processing matplotlib-1.3.x-py3.2-macosx-10.7-x86_64.egg
creating /Users/derek/lib/python3.2/site-packages/matplotlib-1.3.x-py3.2-macosx-10.7-x86_64.egg
Extracting matplotlib-1.3.x-py3.2-macosx-10.7-x86_64.egg to /Users/derek/lib/python3.2/site-packages
Adding matplotlib 1.3.x to easy-install.pth file

Installed /Users/derek/lib/python3.2/site-packages/matplotlib-1.3.x-py3.2-macosx-10.7-x86_64.egg
Processing dependencies for matplotlib==1.3.x
Traceback (most recent call last):
 File "setup.py", line 228, in <module>
   'KnownFailure = matplotlib.testing.noseclasses:KnownFailure'
 File "/sw/lib/python3.2/distutils/core.py", line 148, in setup
   dist.run_commands()
 File "/sw/lib/python3.2/distutils/dist.py", line 917, in run_commands
   self.run_command(cmd)
 File "/sw/lib/python3.2/distutils/dist.py", line 936, in run_command
   cmd_obj.run()
 File "/sw/lib/python3.2/site-packages/setuptools/command/install.py", line 73, in run
   self.do_egg_install()
 File "/sw/lib/python3.2/site-packages/setuptools/command/install.py", line 101, in do_egg_install
   cmd.run()
 File "/sw/lib/python3.2/site-packages/setuptools/command/easy_install.py", line 358, in run
   self.easy_install(spec, not self.no_deps)
 File "/sw/lib/python3.2/site-packages/setuptools/command/easy_install.py", line 582, in easy_install
   return self.install_item(None, spec, tmpdir, deps, True)
 File "/sw/lib/python3.2/site-packages/setuptools/command/easy_install.py", line 634, in install_item
   self.process_distribution(spec, dist, deps)
 File "/sw/lib/python3.2/site-packages/setuptools/command/easy_install.py", line 686, in process_distribution
   [requirement], self.local_index, self.easy_install
 File "/sw/lib/python3.2/site-packages/pkg_resources.py", line 586, in resolve
   dist = best[req.key] = env.best_match(req, self, installer)
 File "/sw/lib/python3.2/site-packages/pkg_resources.py", line 829, in best_match
   for dist in self[req.key]:
 File "/sw/lib/python3.2/site-packages/pkg_resources.py", line 799, in __getitem__
   _sort_dists(dists)
 File "/sw/lib/python3.2/site-packages/pkg_resources.py", line 2613, in _sort_dists
   tmp.sort()
TypeError: unorderable types: NoneType() < str()

Re: [matplotlib-devel] Planning for 1.3.0

[Michiel de H. <mjl...@ya...>]

Hi Derek,

--- On Sun, 4/14/13, Derek Homeier <de...@as...> wrote:
> The RuntimeError was enforced by the #ifdef
> WITH_NEXT_FRAMEWORK check that
> does not allow to use the backend at all, so I had to change
> this to a RuntimeWarning
> to be able to test the backend in the 1.3 branch.

This RuntimeError is there for a reason: If your Python is not installed as  a framework, the backend will not work correctly (and if you ignore the RuntimeError, you won't know if any problems you encounter are real bugs, or simply due to your Python not being installed as a framework).

Best,
-Michiel

Re: [matplotlib-devel] Planning for 1.3.0

[Derek H. <de...@as...>]

Hi Michiel,

> This RuntimeError is there for a reason: If your Python is not installed as  a framework, the backend will not work correctly (and if you ignore the RuntimeError, you won't know if any problems you encounter are real bugs, or simply due to your Python not being installed as a framework).

I have used the MacOSX backend ever since I started working with Python,
and there only was a warning for the last 3 years or so. Other than my plot
windows evading the control of Exposé/Application switcher I have never
noticed any problems with this. Thus my request in my initial post to leave it
as a mere warning, since I'd think people can switch on their own decision
if they do not like the interaction (or lack thereof) with the window manager.
Otherwise I would have to grudgingly switch to another backend (seems now
that I could live with QT4Agg-Quartz or the like though), since installation as
as framework is not an option with Fink.
Of course if there are any other possible negative effects besides the window
handling, I'd take your point.

Best,
						Derek

Re: [matplotlib-devel] Planning for 1.3.0

[Michiel de H. <mjl...@ya...>]

Hi Derek,

--- On Sun, 4/14/13, Derek Homeier <de...@as...> wrote:
> Of course if there are any other possible negative effects
> besides the window handling, I'd take your point.

Several bugs have been reported in the past that turned out to be due to Python not being installed as a framework. For example, the file selection window when saving a figure doesn't respond. This has been a major hassle, since each of those bug reports take time to investigate before realizing that it is due to the Python installation.

Best,
-Michiel.

Re: [matplotlib-devel] Planning for 1.3.0

[Derek H. <de...@as...>]

Hi Michiel,

On 15.04.2013, at 6:03AM, Michiel de Hoon <mjl...@ya...> wrote:

> --- On Sun, 4/14/13, Derek Homeier <de...@as...> wrote:
>> Of course if there are any other possible negative effects
>> besides the window handling, I'd take your point.
> 
> Several bugs have been reported in the past that turned out to be due to Python not being installed as a framework. For example, the file selection window when saving a figure doesn't respond. This has been a major hassle, since each of those bug reports take time to investigate before realizing that it is due to the Python installation.

OK, that is a valid reason! I vaguely remember some problems with that in the past,
though haven't experienced any of that in a long time (just tested 'Save File', and I've
been regularly using 'Configure Subplots'). But this is probably a case where a
Warning might not keep all users from filing a bug. :-(
The QT4Agg backend has its ups as well, though there still seem to be some quirks,
too (e.g. the zoom rectangle only shows up with the left and lower border); I will probably
become a fan of the line configuration tool that allows you to switch back to linear from a
log axis scale (in the command line there seems to be no return from plt.semilogy() or
plt.loglog())!

Cheers,
						Derek

Re: [matplotlib-devel] Planning for 1.3.0

[Michiel de H. <mjl...@ya...>]

Hi Derek,

Can you perhaps ask the Fink developers to provide a framework installation of Python? Most matplotlib users who ran into framework-related bugs were Fink users.

Best,
-Michiel.

--- On Mon, 4/15/13, Derek Homeier <de...@as...> wrote:

> From: Derek Homeier <de...@as...>
> Subject: Re: [matplotlib-devel] Planning for 1.3.0
> To: "matplotlib development list" <mat...@li...>
> Date: Monday, April 15, 2013, 11:00 AM
> Hi Michiel,
> 
> On 15.04.2013, at 6:03AM, Michiel de Hoon <mjl...@ya...>
> wrote:
> 
> > --- On Sun, 4/14/13, Derek Homeier <de...@as...>
> wrote:
> >> Of course if there are any other possible negative
> effects
> >> besides the window handling, I'd take your point.
> > 
> > Several bugs have been reported in the past that turned
> out to be due to Python not being installed as a framework.
> For example, the file selection window when saving a figure
> doesn't respond. This has been a major hassle, since each of
> those bug reports take time to investigate before realizing
> that it is due to the Python installation.
> 
> OK, that is a valid reason! I vaguely remember some problems
> with that in the past,
> though haven't experienced any of that in a long time (just
> tested 'Save File', and I've
> been regularly using 'Configure Subplots'). But this is
> probably a case where a
> Warning might not keep all users from filing a bug. :-(
> The QT4Agg backend has its ups as well, though there still
> seem to be some quirks,
> too (e.g. the zoom rectangle only shows up with the left and
> lower border); I will probably
> become a fan of the line configuration tool that allows you
> to switch back to linear from a
> log axis scale (in the command line there seems to be no
> return from plt.semilogy() or
> plt.loglog())!
> 
> Cheers,
>            
>            
> Derek
> 
> 
> ------------------------------------------------------------------------------
> Precog is a next-generation analytics platform capable of
> advanced
> analytics on semi-structured data. The platform includes
> APIs for building
> apps and a phenomenal toolset for data science. Developers
> can use
> our toolset for easy data analysis & visualization. Get
> a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
> 

Re: [matplotlib-devel] Planning for 1.3.0

[Derek H. <de...@as...>]

Hi Michiel,

On 16.04.2013, at 12:03AM, Michiel de Hoon <mjl...@ya...> wrote:

> Can you perhaps ask the Fink developers to provide a framework installation of Python? Most matplotlib users who ran into framework-related bugs were Fink users.

I've already looked for that in the list archives and it seems this topic comes up about once a
year when some other package broke with the non-framework build. Changing the build does
not seem a particular problem, but was always declined as it would mean all (or a large number)
of the other Python-dependent packages would have to be fixed at the same time.
But I can of course bring this up for discussion again pointing out that the macosx backend support
is going to be discontinued.

Cheers,
						Derek