Menu
▾
▴
matplotlib-devel
|
From: Brian G. <ell...@gm...> - 2009-07-16 17:04:55
|
Hello all, [sent to mpl-dev, enthought-dev and ipython-dev] This summer, we are doing some major refactoring of IPython's core. One of the things I am working on is changing how IPython's works with GUI toolkits. These changes will have a significant impact (hopefully for the better) on your project, so I wanted to open a discussion about this issue. Here is the current situation: currently, IPython uses threads to allow GUI event loops. This code lives in IPython.Shell and is extremely subtle, hard to maintain and fragile. Fernando and John Hunter have done a fantastic job in developing this code, but in the long run we need a more robust approach. Here is the proposal: Python has an obscure hook called PyOS_InputHook. By using this hook, GUI toolkits can interleave their event loop with a command line program *without threads*. Even though PyOS_InputHook is not well known, this is how Python's built-in integration with Tk works. The good news is that other GUI toolkits are starting to support PyOS_InputHook: * PyGTK 2.15.1 has this. * The mpl MacOSX backend works this way * Recent versions of PyQT 4 have this. * I am working with Robin Dunn to implement this in wxPython 2.8 and 2.9 Bottom line: once people are using these recent/upcoming versions of the GUI toolkits, IPython will no longer need to maintain the code in Shell.py and IPython won't need to have -pylab/-wthread/etc options. So, how does affect your project? * People will be able to use your project interactive from the regular python prompt. * You will need to make small changes to your GUI toolkits initialization code. * All of us will need to coordinate version transitions to make sure that there is a clean transition to this new approach. * I need help testing the new approach (especially with wxPython) to make sure that your project actually works with the new approach. What needs to be done at this point? * I would like to discuss how the transition should be made in terms of versions. * I need help testing this new approach in the various toolkits - especially with wx. * I want to see if there are other issues related to this that I am missing. Cheers, Brian |
|
Re: [matplotlib-devel] [IPython-dev] IPython proposal: getting rid
of "ipython -pylab\-wthread\etc."
From: Gael V. <gae...@no...> - 2009-07-16 17:21:05
|
On Thu, Jul 16, 2009 at 10:04:33AM -0700, Brian Granger wrote: > So, how does affect your project? > * People will be able to use your project interactive from the regular > python prompt. > * You will need to make small changes to your GUI toolkits initialization > code. > * All of us will need to coordinate version transitions to make sure that > there is a clean transition to this new approach. > * I need help testing the new approach (especially with wxPython) to make > sure that your project actually works with the new approach. Would it be possible for IPython to expose both solutions for a while (say 6 month to a year), so that we all have time to adapt? Ga |
|
From: Michael D. <md...@st...> - 2009-07-16 17:32:08
|
This is a great project and I'm sure we're all looking forward to having "just one way to do it". However, do not overestimate how up-to-date these packages will be, particularly in managed environments. For instance the RHEL4 boxes we run at my employer still have pygtk 2.4. (Yes, that is 4 years old!) It would be unfortunate if our users couldn't update matplotlib without the pain of recompiling a large part of the gtk stack underneath. So just a plea to keep the old code paths working -- perhaps surrounded in big flashing "REMOVE ME LATER" comments. I understand that maintaining code that fewer and fewer users will be running is like a time bomb. Maybe we could raise a deprecation warning if a user has an old version of a toolkit, so at least when the bomb finally goes off the user has a first guess as to why. But I think dropping all support for these older versions in one step would be a mistake. Cheers, Mike Brian Granger wrote: > Hello all, > > [sent to mpl-dev, enthought-dev and ipython-dev] > > This summer, we are doing some major refactoring of IPython's core. > One of the things I am working on is changing how IPython's works with > GUI toolkits. These changes will have a significant impact (hopefully > for the better) on your project, so I wanted to open a discussion > about this issue. > > Here is the current situation: currently, IPython uses threads to > allow GUI event loops. This code lives in IPython.Shell and is > extremely subtle, hard to maintain and fragile. Fernando and John > Hunter have done a fantastic job in developing this code, but in the > long run we need a more robust approach. > > Here is the proposal: Python has an obscure hook called > PyOS_InputHook. By using this hook, GUI toolkits can interleave their > event loop with a command line program *without threads*. Even though > PyOS_InputHook is not well known, this is how Python's built-in > integration with Tk works. The good news is that other GUI toolkits > are starting to support PyOS_InputHook: > > * PyGTK 2.15.1 has this. > * The mpl MacOSX backend works this way > * Recent versions of PyQT 4 have this. > * I am working with Robin Dunn to implement this in wxPython 2.8 and 2.9 > > Bottom line: once people are using these recent/upcoming versions of > the GUI toolkits, IPython will no longer need to maintain the code in > Shell.py and IPython won't need to have -pylab/-wthread/etc options. > > So, how does affect your project? > > * People will be able to use your project interactive from the regular > python prompt. > * You will need to make small changes to your GUI toolkits > initialization code. > * All of us will need to coordinate version transitions to make sure > that there is a clean transition to this new approach. > * I need help testing the new approach (especially with wxPython) to > make sure that your project actually works with the new approach. > > What needs to be done at this point? > > * I would like to discuss how the transition should be made in terms > of versions. > * I need help testing this new approach in the various toolkits - > especially with wx. > * I want to see if there are other issues related to this that I am > missing. > > Cheers, > > Brian > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------------ > Enter the BlackBerry Developer Challenge > This is your chance to win up to $100,000 in prizes! For a limited time, > vendors submitting new applications to BlackBerry App World(TM) will have > the opportunity to enter the BlackBerry Developer Challenge. See full prize > details at: http://p.sf.net/sfu/Challenge > ------------------------------------------------------------------------ > > _______________________________________________ > Matplotlib-devel mailing list > Mat...@li... > https://lists.sourceforge.net/lists/listinfo/matplotlib-devel > -- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA |
|
From: Brian G. <ell...@gm...> - 2009-07-16 18:06:31
|
This is a great project and I'm sure we're all looking forward to having > "just one way to do it". > > However, do not overestimate how up-to-date these packages will be, > particularly in managed environments. For instance the RHEL4 boxes we run > at my employer still have pygtk 2.4. (Yes, that is 4 years old!) It would > be unfortunate if our users couldn't update matplotlib without the pain of > recompiling a large part of the gtk stack underneath. > I am fully aware that some environments upgrade things like GUI toolkits over very long time scales. More important - there isn't a released version of wx that has these capabilities in it. It will be a long time before everyone is running the needed versions of wx/gtk/qt. In the meantime, we will do everything we can to make sure that people running older versions of GUI toolkits can continue to use IPython (and mpl). On the mpl side of things, there will probably have to be some changes to the GUI toolkit initialization code that detects which way things are being run (old/new) and then does the right thing. If we coordinate this well, that should allow newer versions of mpl to work with both new and old versions of IPython and the GUI toolkits. However *at some point*, we will have to make a clean break. > So just a plea to keep the old code paths working -- perhaps surrounded in > big flashing "REMOVE ME LATER" comments. I understand that maintaining code > that fewer and fewer users will be running is like a time bomb. This is especially true of the threaded Shells in IPython. Honestly, even with thousands of people using it (as is currently the case) it is a time bomb. > Maybe we could raise a deprecation warning if a user has an old version of > a toolkit, so at least when the bomb finally goes off the user has a first > guess as to why. But I think dropping all support for these older versions > in one step would be a mistake. > Yes, there will definitely have to be some sort of transition. One possibility is that during the transition, the old threaded Shells would slowly loose features that become impossible to maintain (such as interrupt handling in the threaded shells). With our very limited IPython manpower, we have a very fine line to walk between moving forward and maintaining backwards compatibility. Cheers, Brian > Cheers, > Mike > > Brian Granger wrote: > >> Hello all, >> >> [sent to mpl-dev, enthought-dev and ipython-dev] >> >> This summer, we are doing some major refactoring of IPython's core. One >> of the things I am working on is changing how IPython's works with GUI >> toolkits. These changes will have a significant impact (hopefully for the >> better) on your project, so I wanted to open a discussion about this issue. >> >> Here is the current situation: currently, IPython uses threads to allow >> GUI event loops. This code lives in IPython.Shell and is extremely subtle, >> hard to maintain and fragile. Fernando and John Hunter have done a >> fantastic job in developing this code, but in the long run we need a more >> robust approach. >> >> Here is the proposal: Python has an obscure hook called PyOS_InputHook. >> By using this hook, GUI toolkits can interleave their event loop with a >> command line program *without threads*. Even though PyOS_InputHook is not >> well known, this is how Python's built-in integration with Tk works. The >> good news is that other GUI toolkits are starting to support PyOS_InputHook: >> >> * PyGTK 2.15.1 has this. >> * The mpl MacOSX backend works this way >> * Recent versions of PyQT 4 have this. >> * I am working with Robin Dunn to implement this in wxPython 2.8 and 2.9 >> >> Bottom line: once people are using these recent/upcoming versions of the >> GUI toolkits, IPython will no longer need to maintain the code in Shell.py >> and IPython won't need to have -pylab/-wthread/etc options. >> So, how does affect your project? >> >> * People will be able to use your project interactive from the regular >> python prompt. >> * You will need to make small changes to your GUI toolkits initialization >> code. >> * All of us will need to coordinate version transitions to make sure that >> there is a clean transition to this new approach. >> * I need help testing the new approach (especially with wxPython) to make >> sure that your project actually works with the new approach. >> >> What needs to be done at this point? >> >> * I would like to discuss how the transition should be made in terms of >> versions. >> * I need help testing this new approach in the various toolkits - >> especially with wx. >> * I want to see if there are other issues related to this that I am >> missing. >> >> Cheers, >> >> Brian >> ------------------------------------------------------------------------ >> >> >> ------------------------------------------------------------------------------ >> Enter the BlackBerry Developer Challenge This is your chance to win up to >> $100,000 in prizes! For a limited time, vendors submitting new applications >> to BlackBerry App World(TM) will have >> the opportunity to enter the BlackBerry Developer Challenge. See full >> prize details at: http://p.sf.net/sfu/Challenge >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Matplotlib-devel mailing list >> Mat...@li... >> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel >> >> > > -- > Michael Droettboom > Science Software Branch > Operations and Engineering Division > Space Telescope Science Institute > Operated by AURA for NASA > > |
|
Re: [matplotlib-devel] [IPython-dev] IPython proposal: getting rid
of "ipython -pylab\-wthread\etc."
From: Brian G. <ell...@gm...> - 2009-07-16 17:38:59
|
On Thu, Jul 16, 2009 at 10:20 AM, Gael Varoquaux < gae...@no...> wrote: > On Thu, Jul 16, 2009 at 10:04:33AM -0700, Brian Granger wrote: > > So, how does affect your project? > > > * People will be able to use your project interactive from the regular > > python prompt. > > * You will need to make small changes to your GUI toolkits > initialization > > code. > > * All of us will need to coordinate version transitions to make sure > that > > there is a clean transition to this new approach. > > * I need help testing the new approach (especially with wxPython) to > make > > sure that your project actually works with the new approach. > > Would it be possible for IPython to expose both solutions for a while > (say 6 month to a year), so that we all have time to adapt? > Yes and no. In the short term, this is definitely possible. Also, as Ville mentions, the new approach will also work with *any* older version of IPython. However, one of the main things that needs to be refactored in IPython is the threading model. This is needed for a variety of reasons, but the bottom line is this - a number of projects are using IPython as if it were thread safe (including IPython itself) - but it is not. When I fix these things (which will be soon), I am not sure (I am doubtful) that the threaded Shell.py code will continue to work. But, this is something that I can look into. But, at some point, we will have to say "if you want the old threaded shells, you will have to use an older version of IPython". We just have to figure out what the transition looks like. Cheers, Brian > Gaël > |
|
Re: [matplotlib-devel] [IPython-dev] IPython proposal: getting rid
of "ipython -pylab\-wthread\etc."
From: Gael V. <gae...@no...> - 2009-07-16 17:41:45
|
On Thu, Jul 16, 2009 at 10:38:56AM -0700, Brian Granger wrote: > But, at some point, we will have to say "if you want the old threaded > shells, you will have to use an older version of IPython". We just have > to figure out what the transition looks like. As long as the packages sitting outside of IPython have a way of dealing with both reasonnably easily that fine. What we don't want to happen is having confusing errors or instructions depending on the version of IPython people use. As Michael points out, there will be people using the old model for a long time. Ga |
|
Re: [matplotlib-devel] [IPython-dev] IPython proposal: getting rid
of "ipython -pylab\-wthread\etc."
From: Gael V. <gae...@no...> - 2009-07-16 17:39:33
|
On Thu, Jul 16, 2009 at 08:28:11PM +0300, Ville M. Vainio wrote: > On Thu, Jul 16, 2009 at 8:20 PM, Gael > Varoquaux<gae...@no...> wrote: > > Would it be possible for IPython to expose both solutions for a while > > (say 6 month to a year), so that we all have time to adapt? > You can start adapting already, as indicated in previous post. Just > adjust the embedding code, and any old version of ipython will work. So, ... I am dumb, and I haven't been following things well enough (not enough time). Concretely, what do I need to do to be able to launch IPython, and pop up a wx dialog not blocking IPython with the new technology? Gaël |
|
Re: [matplotlib-devel] [IPython-dev] IPython proposal: getting rid
of "ipython -pylab\-wthread\etc."
From: Brian G. <ell...@gm...> - 2009-07-16 17:45:15
|
Good question Gael, > So, ... I am dumb, and I haven't been following things well enough (not > enough time). Concretely, what do I need to do to be able to launch > IPython, and pop up a wx dialog not blocking IPython with the new > technology? > If you have a patched version of wx (that Robin and I are working on) you just do the following (from either ipython or python!): >>> import wx >>> app = wx.IApp(clearSigInt=False) # for interactive app then you are off to the races (you don't even need to start the mainloop) With current versions of mpl, you can do: >>> import wx >>> app = wx.IApp(clearSigInt=False) >>> from matplotlib import pyplot as plt >>> plt.interactive(True) the everything just works... Obviously, the creation of the IApp object should be put into the GUI toolkit initialization code in mpl/mayavi/etc. I will work with Robin to get a version of wx posted that has the patched applied so people can begin to test this. Cheers, Brian |
|
Re: [matplotlib-devel] [IPython-dev] IPython proposal: getting rid
of "ipython -pylab\-wthread\etc."
From: Gael V. <gae...@no...> - 2009-07-16 17:53:13
|
On Thu, Jul 16, 2009 at 10:45:04AM -0700, Brian Granger wrote: > Obviously, the creation of the IApp object should be put into the GUI > toolkit initialization code in mpl/mayavi/etc. I will work with Robin to > get a version of wx posted that has the patched applied so people can > begin to test this. OK, that's great. It enlightens me. So, it seems you guys are on the right track. I'll just wait for the new versions of wx to come out. You are doing great work on that. One piece of warning: Wx has a terrible upgrade path. Upgrading it breaks stuff. As a consequence, people don't upgrade it often, and my experience is that we have to maintain compatibility with old version for years (at least 3 years: in Ubuntu, latest release, installing SPE drags along wxPython 2.6, and thus people find themselves importing this version without realizing it). Ga |
|
Re: [matplotlib-devel] [IPython-dev] IPython proposal: getting rid
of "ipython -pylab\-wthread\etc."
From: Brian G. <ell...@gm...> - 2009-07-16 18:07:26
|
> OK, that's great. It enlightens me. So, it seems you guys are on the > right track. I'll just wait for the new versions of wx to come out. You > are doing great work on that. > Thanks. > One piece of warning: Wx has a terrible upgrade path. Upgrading it breaks > stuff. As a consequence, people don't upgrade it often, and my experience > is that we have to maintain compatibility with old version for years (at > least 3 years: in Ubuntu, latest release, installing SPE drags along > wxPython 2.6, and thus people find themselves importing this version > without realizing it). > This is good to know. Cheers, Brian > > Gaël > |
|
Re: [matplotlib-devel] [IPython-dev] IPython proposal: getting rid
of "ipython -pylab\-wthread\etc."
From: Brian G. <ell...@gm...> - 2009-07-16 19:06:05
|
Yes, this is also a possibility that we need to pursue as it would greatly help the transition. I do have a Cython prototype that works. But, we are currently testing all of this on wx trunk. I am not sure if what we are doing will work on previous versions. I plan on looking into this. Cheers, Brian On Thu, Jul 16, 2009 at 11:11 AM, Ville M. Vainio <viv...@gm...>wrote: > On Thu, Jul 16, 2009 at 9:06 PM, Brian Granger<ell...@gm...> > wrote: > > > I am fully aware that some environments upgrade things like GUI toolkits > > over very long time scales. More important - there isn't a released > version > > of wx that has these capabilities in it. It will be a long time before > > I thought you were able to make wx work with a cython-based > PyOS_InputHook extension. Possibly a C-based extension could fix this > issue for users of old wx versions? > > -- > Ville M. Vainio > http://tinyurl.com/vainio > |
|
Re: [matplotlib-devel] [IPython-dev] IPython proposal: getting rid
of "ipython -pylab\-wthread\etc."
From: Brian G. <ell...@gm...> - 2009-07-16 19:13:12
|
I tried ctypes first, but with no luck. Here is what I tried: >>> import readline >>> import ctypes >>> ctypes.pythonapi.PyOS_InputHook <_FuncPtr object at 0x76420> >>> def my_callback(): ... print "In am here" ... return 0 ... >>> cbf = ctypes.CFUNCTYPE(ctypes.c_int)(my_callback) >>> cbf <CFunctionType object at 0x769d0> >>> ctypes.pythonapi.PyOS_InputHook = cbf Do you see anything wrong with this? I should probably try it again to see if I can make it work though. Cheers, Brian On Thu, Jul 16, 2009 at 11:35 AM, Robert Kern <rob...@gm...> wrote: > On 2009-07-16 13:11, Ville M. Vainio wrote: > > On Thu, Jul 16, 2009 at 9:06 PM, Brian Granger<ell...@gm...> > wrote: > > > >> I am fully aware that some environments upgrade things like GUI toolkits > >> over very long time scales. More important - there isn't a released > version > >> of wx that has these capabilities in it. It will be a long time before > > > > I thought you were able to make wx work with a cython-based > > PyOS_InputHook extension. Possibly a C-based extension could fix this > > issue for users of old wx versions? > > Or even just a bit of ctypes? > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma > that is made terrible by our own mad attempt to interpret it as though it > had > an underlying truth." > -- Umberto Eco > > _______________________________________________ > IPython-dev mailing list > IPy...@sc... > http://mail.scipy.org/mailman/listinfo/ipython-dev > |
|
Re: [matplotlib-devel] [IPython-dev] IPython proposal: getting rid
of "ipython -pylab\-wthread\etc."
From: Brian G. <ell...@gm...> - 2009-07-16 19:27:19
|
OK, I just found this thread that shows how to set PyOS_InputHook from ctypes: http://osdir.com/ml/python.ctypes/2006-12/msg00045.html I tried it and it works fine. This will simplify the transition a lot. I will develop some pure ctypes prototypes for the various toolkits and then folks can begin to try this out. Cheers, Brian On Thu, Jul 16, 2009 at 12:13 PM, Brian Granger <ell...@gm...>wrote: > I tried ctypes first, but with no luck. Here is what I tried: > > >>> import readline > >>> import ctypes > >>> ctypes.pythonapi.PyOS_InputHook > <_FuncPtr object at 0x76420> > >>> def my_callback(): > ... print "In am here" > ... return 0 > ... > >>> cbf = ctypes.CFUNCTYPE(ctypes.c_int)(my_callback) > >>> cbf > <CFunctionType object at 0x769d0> > >>> ctypes.pythonapi.PyOS_InputHook = cbf > > Do you see anything wrong with this? > > I should probably try it again to see if I can make it work though. > > Cheers, > > Brian > > > On Thu, Jul 16, 2009 at 11:35 AM, Robert Kern <rob...@gm...>wrote: > >> On 2009-07-16 13:11, Ville M. Vainio wrote: >> > On Thu, Jul 16, 2009 at 9:06 PM, Brian Granger<ell...@gm...> >> wrote: >> > >> >> I am fully aware that some environments upgrade things like GUI >> toolkits >> >> over very long time scales. More important - there isn't a released >> version >> >> of wx that has these capabilities in it. It will be a long time before >> > >> > I thought you were able to make wx work with a cython-based >> > PyOS_InputHook extension. Possibly a C-based extension could fix this >> > issue for users of old wx versions? >> >> Or even just a bit of ctypes? >> >> -- >> Robert Kern >> >> "I have come to believe that the whole world is an enigma, a harmless >> enigma >> that is made terrible by our own mad attempt to interpret it as though it >> had >> an underlying truth." >> -- Umberto Eco >> >> _______________________________________________ >> IPython-dev mailing list >> IPy...@sc... >> http://mail.scipy.org/mailman/listinfo/ipython-dev >> > > |
|
From: Brian G. <ell...@gm...> - 2009-07-16 20:36:32
|
After playing a bit more, I have a ctypes based prototype that can set PyOS_InputHook in an appropriate way for wx 2.9 (trunk). I think it is possible to have a ctypes based version that would work for all the major GUI toolkits that we could ship with IPython until the transition is over (i.e., everyone is using a GUI toolkits recent enough). BUT. I need to begin to narrow the GUI toolkits and versions that we will support. The total number of possibilities (for testing, etc.) is approximately: (number of OSs~3) x (number of GUI toolkits~qt4/wx/gtk) x (number of supported versions of each GUI) Because the first 2 terms are pretty fixed, I want to bring sanity to the picture by keeping the 3rd term as small as possible. So, here is a question. What versions of what GUI toolkits do we realistically need to support moving forward (today and beyond)? By this, I mean that: "unsupported" = older versions of GUIs toolkits that will have to use older versions of IPython. "supported" = versions of GUIS that will be able to use the PyOS_InputHook approach in newer releases of IPython. Cheers, Brian |
|
Re: [matplotlib-devel] [IPython-dev] IPython proposal: getting rid
of "ipython -pylab\-wthread\etc."
From: Gael V. <gae...@no...> - 2009-07-16 21:31:57
|
On Thu, Jul 16, 2009 at 01:36:21PM -0700, Brian Granger wrote: > What versions of what GUI toolkits do we realistically need to support > moving forward (today and beyond)? By this, I mean that: > "unsupported" = older versions of GUIs toolkits that will have to use > older versions of IPython. > "supported" = versions of GUIS that will be able to use the > PyOS_InputHook approach in newer releases of IPython. For Wx, we would obviously need to support 2.8. We can toy with the idea of not supporting 2.6, I believe. A lot of people are starting to feel that 2.6 is depreciated. Ga |
|
Re: [matplotlib-devel] [IPython-dev] IPython proposal: getting rid
of "ipython -pylab\-wthread\etc."
From: Brian G. <ell...@gm...> - 2009-07-16 21:34:08
|
> For Wx, we would obviously need to support 2.8. We can toy with the idea > of not supporting 2.6, I believe. A lot of people are starting to feel > that 2.6 is depreciated. > Yes, 2.8 is latest stable, so we should support it. How many people would complain if 2.6 were not supported? I take it that all of ETS is 2.8 based? Cheers, Brian > > Gaël > |
|
Re: [matplotlib-devel] [IPython-dev] IPython proposal: getting rid
of "ipython -pylab\-wthread\etc."
From: Gael V. <gae...@no...> - 2009-07-16 21:34:59
|
On Thu, Jul 16, 2009 at 02:34:01PM -0700, Brian Granger wrote: > Yes, 2.8 is latest stable, so we should support it. How many people would > complain if 2.6 were not supported? I take it that all of ETS is 2.8 > based? ETS works better on wx 2.8, but it also works on 2.6, with small bugs. Ga |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X