Menu
▾
▴
matplotlib-users
|
From: Josef K. <jk...@la...> - 2008-09-16 15:06:49
|
Hi folks, I would like to save preliminary figures for later processing and refinement with matplotlib. Is there a way to save or pickle a figure object and later reload it. Matlab has a feature like that and and I was wondering if matplotlib has it too. Thanks a lot, Josef -- Josef Koller (TSPA/Correspondence) Space Science and Applications, ISR-1, MS D466 Los Alamos National Laboratory, NM 87545-1663 Phone: (505) 665 3399 http://www.koller.info Fax: (505) 665 7395 email jk...@la... |
|
From: John H. <jd...@gm...> - 2008-09-16 18:49:11
|
On Tue, Sep 16, 2008 at 5:06 PM, Josef Koller <jk...@la...> wrote: > Hi folks, > I would like to save preliminary figures for later processing and > refinement with matplotlib. Is there a way to save or pickle a figure > object and later reload it. Matlab has a feature like that and and I was > wondering if matplotlib has it too. No, it doesn't exist. We've taken a stab at it once or twice, but have been stymied because we make extensive use of a python extension libray CXX, and these objects have resisted our attempts to pickle them. With our recent transforms refactoring, which removes the hairiest CXX dependency, it may be worth taking another look, but noone is currently working on it. JDH |
|
From: Eric F. <ef...@ha...> - 2008-09-16 19:04:23
|
John Hunter wrote:
> On Tue, Sep 16, 2008 at 5:06 PM, Josef Koller <jk...@la...> wrote:
>> Hi folks,
>> I would like to save preliminary figures for later processing and
>> refinement with matplotlib. Is there a way to save or pickle a figure
>> object and later reload it. Matlab has a feature like that and and I was
>> wondering if matplotlib has it too.
>
> No, it doesn't exist. We've taken a stab at it once or twice, but
> have been stymied because we make extensive use of a python extension
> libray CXX, and these objects have resisted our attempts to pickle
> them. With our recent transforms refactoring, which removes the
> hairiest CXX dependency, it may be worth taking another look, but
> noone is currently working on it.
My sense, based on very little experience, is that pickles of
complicated objects are very fragile, so even if we could pickle
figures, I fear it might cause more trouble ("I can't load this
absolutely critical figure I pickled 6 months ago") than it would be worth.
Eric
|
|
From: Ted D. <ted...@jp...> - 2008-09-16 20:38:25
|
We have some experience maintaining persistent object storage over long
periods of time. The best solution we've found is to do something like
this:
- create a read/write method on each class. Every class that needs to be
stored must have this. This includes class you would store (eg Figure) and
things that are member variables of those classes.
- Each class stores a version number along with it's data which represents
the version of the persistent representation for that class. So each class
has its own, internal versioning scheme that represents a specific set of
variables with specific types.
- The read method on each class must check the version number and then read
the appropriate data for that version of itself. Whenever the persistent
representation of the class changes (usually if the member variables
change), you increment the version number. Implicit in this is that if you
change the member variables of a class, the class read method must be able
to convert the variables that existed in the older version of itself into
the new member variables (since that's what the new methods on that class
will be using)
FYI It is possible to use pickle to do this but you can't rely on pickle to
automatically save the member dictionary. You need to implement
__getstate__ and __setstate__ and have them incorporate a version number in
the dictionary they return. In addition, you shouldn't blindly save every
member variable. If member variables can be constructed in terms of other
data, it may be better to store that data and then reconstruct the member
variables in the __setstate__ method.
Using this type of system, you get a hierarchy of objects that each have
their own, internal versioning system. This lets you make changes to a
single class, increment it's version, and update its save/load methods and
it won't affect any other part of the system and still retains backwards
reading capability.
Ted
> -----Original Message-----
> From: mat...@li...
> [mailto:mat...@li...] On Behalf Of
> Eric Firing
> Sent: Tuesday, September 16, 2008 7:04 PM
> To: John Hunter
> Cc: Josef Koller; mat...@li...
> Subject: Re: [Matplotlib-users] save or pickle figure object
>
> John Hunter wrote:
> > On Tue, Sep 16, 2008 at 5:06 PM, Josef Koller <jk...@la...>
> wrote:
> >> Hi folks,
> >> I would like to save preliminary figures for later processing and
> >> refinement with matplotlib. Is there a way to save or pickle a
> figure
> >> object and later reload it. Matlab has a feature like that and and I
> was
> >> wondering if matplotlib has it too.
> >
> > No, it doesn't exist. We've taken a stab at it once or twice, but
> > have been stymied because we make extensive use of a python extension
> > libray CXX, and these objects have resisted our attempts to pickle
> > them. With our recent transforms refactoring, which removes the
> > hairiest CXX dependency, it may be worth taking another look, but
> > noone is currently working on it.
>
> My sense, based on very little experience, is that pickles of
> complicated objects are very fragile, so even if we could pickle
> figures, I fear it might cause more trouble ("I can't load this
> absolutely critical figure I pickled 6 months ago") than it would be
> worth.
>
> Eric
>
> -----------------------------------------------------------------------
> --
> This SF.Net email is sponsored by the Moblin Your Move Developer's
> challenge
> Build the coolest Linux based applications with Moblin SDK & win great
> prizes
> Grand prize is a trip for two to an Open Source event anywhere in the
> world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
> Checked by AVG - http://www.avg.com
> Version: 8.0.169 / Virus Database: 270.6.21/1674 - Release Date:
> 9/16/2008 8:15 AM
|
|
From: B C. <clo...@ya...> - 2008-09-17 13:05:03
|
Josef, I too have been interested in such a feature for matplotlib and have made some (albeit lame) stabs at finding a solution. I started a project on google code that has some very limited capacity to save line plots and the necessary data arrays from matplotlib into an hdf5 file for later processing. It is by no means a complete solution but may serve as a VERY rough model to add this functionality. It was my hope that if you could capture the underlying keyword arguments necessary to recreate a plot then you might be able to simply recreate the original figure assuming the correct interpreter functions are established. Anyway, if you are interested please take a look to see if there may be something useful. From the perspective of the IPython users out there, a question to you: is there a good way to do achieve this feature with some of the logging commands? http://code.google.com/p/subplot/ Cheers, Brian -ps excuse the name of the project--I lacked creative drive at the time of naming. --- On Tue, 9/16/08, John Hunter <jd...@gm...> wrote: From: John Hunter <jd...@gm...> Subject: Re: [Matplotlib-users] save or pickle figure object To: "Josef Koller" <jk...@la...> Cc: mat...@li... Date: Tuesday, September 16, 2008, 7:49 PM On Tue, Sep 16, 2008 at 5:06 PM, Josef Koller <jk...@la...> wrote: > Hi folks, > I would like to save preliminary figures for later processing and > refinement with matplotlib. Is there a way to save or pickle a figure > object and later reload it. Matlab has a feature like that and and I was > wondering if matplotlib has it too. No, it doesn't exist. We've taken a stab at it once or twice, but have been stymied because we make extensive use of a python extension libray CXX, and these objects have resisted our attempts to pickle them. With our recent transforms refactoring, which removes the hairiest CXX dependency, it may be worth taking another look, but noone is currently working on it. JDH ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Matplotlib-users mailing list Mat...@li... https://lists.sourceforge.net/lists/listinfo/matplotlib-users |
|
From: Anthony F. <ant...@gm...> - 2008-09-16 20:12:34
|
On Tue, Sep 16, 2008 at 3:06 PM, Josef Koller <jk...@la...> wrote: > Hi folks, > I would like to save preliminary figures for later processing and > refinement with matplotlib. Is there a way to save or pickle a figure > object and later reload it. Matlab has a feature like that and and I was > wondering if matplotlib has it too. > > Thanks a lot, > > Josef As you've already been told, you can't pickle/shelve mpl objects. Our solution to this is to have a native python shadow object that contains all the bits and pieces needed to create a figure, and always build the plots from these shadow objects. This gives us the advantage of being able to shelve the shadow objects and rebuild the figures later. Of course, we're doing this in context of a large program using the mpl api, so if you're just using pylab, you're a bit SOL. A> |
|
From: Sebastien B. <hep...@gm...> - 2008-09-16 20:18:44
|
Hi Anthony, > As you've already been told, you can't pickle/shelve mpl objects. Our > solution to this is to have a native python shadow object that > contains all the bits and pieces needed to create a figure, and always > build the plots from these shadow objects. This gives us the > advantage of being able to shelve the shadow objects and rebuild the > figures later. this is rather interesting ! any code to point to ? cheers, sebastien. -- ################################### # Dr. Sebastien Binet # Lawrence Berkeley National Lab. # 1 Cyclotron Road # Berkeley, CA 94720 ################################### |
|
From: Anthony F. <ant...@gm...> - 2008-09-16 20:30:28
|
On Tue, Sep 16, 2008 at 8:17 PM, Sebastien Binet <hep...@gm...> wrote: > Hi Anthony, > >> As you've already been told, you can't pickle/shelve mpl objects. Our >> solution to this is to have a native python shadow object that >> contains all the bits and pieces needed to create a figure, and always >> build the plots from these shadow objects. This gives us the >> advantage of being able to shelve the shadow objects and rebuild the >> figures later. > > this is rather interesting ! > any code to point to ? Hi Sebastien, I'll see what I can extract from the code tomorrow at work. It's pretty modular so I should be able to point to something. A> |
|
From: Eric F. <ef...@ha...> - 2008-09-17 12:02:05
|
Ted Drain wrote:
> We have some experience maintaining persistent object storage over long
> periods of time. The best solution we've found is to do something like
> this:
>
> - create a read/write method on each class. Every class that needs to be
> stored must have this. This includes class you would store (eg Figure) and
> things that are member variables of those classes.
>
> - Each class stores a version number along with it's data which represents
> the version of the persistent representation for that class. So each class
> has its own, internal versioning scheme that represents a specific set of
> variables with specific types.
>
> - The read method on each class must check the version number and then read
> the appropriate data for that version of itself. Whenever the persistent
> representation of the class changes (usually if the member variables
> change), you increment the version number. Implicit in this is that if you
> change the member variables of a class, the class read method must be able
> to convert the variables that existed in the older version of itself into
> the new member variables (since that's what the new methods on that class
> will be using)
>
> FYI It is possible to use pickle to do this but you can't rely on pickle to
> automatically save the member dictionary. You need to implement
> __getstate__ and __setstate__ and have them incorporate a version number in
> the dictionary they return. In addition, you shouldn't blindly save every
> member variable. If member variables can be constructed in terms of other
> data, it may be better to store that data and then reconstruct the member
> variables in the __setstate__ method.
>
> Using this type of system, you get a hierarchy of objects that each have
> their own, internal versioning system. This lets you make changes to a
> single class, increment it's version, and update its save/load methods and
> it won't affect any other part of the system and still retains backwards
> reading capability.
>
> Ted
Sounds good--for some applications--but I would strongly oppose adding
this additional level of complexity to mpl. It's just not worth it. If
you want to be able to work with a plot, then generate it with a script,
and save the data and the script. That is the user's responsibility,
not mpl's.
Unless mpl is taken over by a cadre of full-time professional
programmers, we have to try to keep it accessible to people who can only
work on it sporadically. That means we need to try to keep it
simple--indeed, to work on simplifying it and cleaning up the rough
edges, and to work on maintaining a design that makes it easy to improve
the real plotting capabilities and ease-of-use.
Eric
>
>> -----Original Message-----
>> From: mat...@li...
>> [mailto:mat...@li...] On Behalf Of
>> Eric Firing
>> Sent: Tuesday, September 16, 2008 7:04 PM
>> To: John Hunter
>> Cc: Josef Koller; mat...@li...
>> Subject: Re: [Matplotlib-users] save or pickle figure object
>>
>> John Hunter wrote:
>>> On Tue, Sep 16, 2008 at 5:06 PM, Josef Koller <jk...@la...>
>> wrote:
>>>> Hi folks,
>>>> I would like to save preliminary figures for later processing and
>>>> refinement with matplotlib. Is there a way to save or pickle a
>> figure
>>>> object and later reload it. Matlab has a feature like that and and I
>> was
>>>> wondering if matplotlib has it too.
>>> No, it doesn't exist. We've taken a stab at it once or twice, but
>>> have been stymied because we make extensive use of a python extension
>>> libray CXX, and these objects have resisted our attempts to pickle
>>> them. With our recent transforms refactoring, which removes the
>>> hairiest CXX dependency, it may be worth taking another look, but
>>> noone is currently working on it.
>> My sense, based on very little experience, is that pickles of
>> complicated objects are very fragile, so even if we could pickle
>> figures, I fear it might cause more trouble ("I can't load this
>> absolutely critical figure I pickled 6 months ago") than it would be
>> worth.
>>
>> Eric
>>
>> -----------------------------------------------------------------------
>> --
>> This SF.Net email is sponsored by the Moblin Your Move Developer's
>> challenge
>> Build the coolest Linux based applications with Moblin SDK & win great
>> prizes
>> Grand prize is a trip for two to an Open Source event anywhere in the
>> world
>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>> _______________________________________________
>> Matplotlib-users mailing list
>> Mat...@li...
>> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>> Checked by AVG - http://www.avg.com
>> Version: 8.0.169 / Virus Database: 270.6.21/1674 - Release Date:
>> 9/16/2008 8:15 AM
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
|
|
From: Ted D. <ted...@jp...> - 2008-09-17 12:56:00
|
I agree completely - I was just pointing that it is possible. I think what
people might not be aware of is that it's really an all or nothing
proposition. You either jump in completely and pay the large cost to handle
this in a maintainable, scalable way or don't do it at all. All of the
"quick and easy" solutions have too many problems and aren't really
maintainable.
Ted
> -----Original Message-----
> From: Eric Firing [mailto:ef...@ha...]
> Sent: Wednesday, September 17, 2008 12:01 PM
> To: Ted Drain
> Cc: mat...@li...
> Subject: Re: [Matplotlib-users] save or pickle figure object
>
> Ted Drain wrote:
> > We have some experience maintaining persistent object storage over
> long
> > periods of time. The best solution we've found is to do something
> like
> > this:
> >
> > - create a read/write method on each class. Every class that needs
> to be
> > stored must have this. This includes class you would store (eg
> Figure) and
> > things that are member variables of those classes.
> >
> > - Each class stores a version number along with it's data which
> represents
> > the version of the persistent representation for that class. So each
> class
> > has its own, internal versioning scheme that represents a specific
> set of
> > variables with specific types.
> >
> > - The read method on each class must check the version number and
> then read
> > the appropriate data for that version of itself. Whenever the
> persistent
> > representation of the class changes (usually if the member variables
> > change), you increment the version number. Implicit in this is that
> if you
> > change the member variables of a class, the class read method must be
> able
> > to convert the variables that existed in the older version of itself
> into
> > the new member variables (since that's what the new methods on that
> class
> > will be using)
> >
> > FYI It is possible to use pickle to do this but you can't rely on
> pickle to
> > automatically save the member dictionary. You need to implement
> > __getstate__ and __setstate__ and have them incorporate a version
> number in
> > the dictionary they return. In addition, you shouldn't blindly save
> every
> > member variable. If member variables can be constructed in terms of
> other
> > data, it may be better to store that data and then reconstruct the
> member
> > variables in the __setstate__ method.
> >
> > Using this type of system, you get a hierarchy of objects that each
> have
> > their own, internal versioning system. This lets you make changes to
> a
> > single class, increment it's version, and update its save/load
> methods and
> > it won't affect any other part of the system and still retains
> backwards
> > reading capability.
> >
> > Ted
>
> Sounds good--for some applications--but I would strongly oppose adding
> this additional level of complexity to mpl. It's just not worth it.
> If
> you want to be able to work with a plot, then generate it with a
> script,
> and save the data and the script. That is the user's responsibility,
> not mpl's.
>
> Unless mpl is taken over by a cadre of full-time professional
> programmers, we have to try to keep it accessible to people who can
> only
> work on it sporadically. That means we need to try to keep it
> simple--indeed, to work on simplifying it and cleaning up the rough
> edges, and to work on maintaining a design that makes it easy to
> improve
> the real plotting capabilities and ease-of-use.
>
> Eric
>
>
> >
> >> -----Original Message-----
> >> From: mat...@li...
> >> [mailto:mat...@li...] On Behalf Of
> >> Eric Firing
> >> Sent: Tuesday, September 16, 2008 7:04 PM
> >> To: John Hunter
> >> Cc: Josef Koller; mat...@li...
> >> Subject: Re: [Matplotlib-users] save or pickle figure object
> >>
> >> John Hunter wrote:
> >>> On Tue, Sep 16, 2008 at 5:06 PM, Josef Koller <jk...@la...>
> >> wrote:
> >>>> Hi folks,
> >>>> I would like to save preliminary figures for later processing and
> >>>> refinement with matplotlib. Is there a way to save or pickle a
> >> figure
> >>>> object and later reload it. Matlab has a feature like that and and
> I
> >> was
> >>>> wondering if matplotlib has it too.
> >>> No, it doesn't exist. We've taken a stab at it once or twice, but
> >>> have been stymied because we make extensive use of a python
> extension
> >>> libray CXX, and these objects have resisted our attempts to pickle
> >>> them. With our recent transforms refactoring, which removes the
> >>> hairiest CXX dependency, it may be worth taking another look, but
> >>> noone is currently working on it.
> >> My sense, based on very little experience, is that pickles of
> >> complicated objects are very fragile, so even if we could pickle
> >> figures, I fear it might cause more trouble ("I can't load this
> >> absolutely critical figure I pickled 6 months ago") than it would be
> >> worth.
> >>
> >> Eric
> >>
> >> --------------------------------------------------------------------
> ---
> >> --
> >> This SF.Net email is sponsored by the Moblin Your Move Developer's
> >> challenge
> >> Build the coolest Linux based applications with Moblin SDK & win
> great
> >> prizes
> >> Grand prize is a trip for two to an Open Source event anywhere in
> the
> >> world
> >> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> >> _______________________________________________
> >> Matplotlib-users mailing list
> >> Mat...@li...
> >> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
> >> Checked by AVG - http://www.avg.com
> >> Version: 8.0.169 / Virus Database: 270.6.21/1674 - Release Date:
> >> 9/16/2008 8:15 AM
> >
> >
> > ---------------------------------------------------------------------
> ----
> > This SF.Net email is sponsored by the Moblin Your Move Developer's
> challenge
> > Build the coolest Linux based applications with Moblin SDK & win
> great prizes
> > Grand prize is a trip for two to an Open Source event anywhere in the
> world
> > http://moblin-contest.org/redirect.php?banner_id=100&url=/
> > _______________________________________________
> > Matplotlib-users mailing list
> > Mat...@li...
> > https://lists.sourceforge.net/lists/listinfo/matplotlib-users
|
|
From: Friedrich H. <fri...@gm...> - 2008-09-17 13:10:09
|
On Wed, Sep 17, 2008 at 12:55:40PM -0700, Ted Drain wrote: > I agree completely - I was just pointing that it is possible. I think what > people might not be aware of is that it's really an all or nothing > proposition. You either jump in completely and pay the large cost to handle > this in a maintainable, scalable way or don't do it at all. All of the > "quick and easy" solutions have too many problems and aren't really > maintainable. Here is my (easy and maintainable) way to handle the versions for my graphics: * I write the data creation in a python script * I write the creation of the mpl graphics in a script (most the same) * I manage these python file with a version-system (eg mercurial) So I have different versions for my mpl graphics and I can modify the graphic any time (with a new python run). If the data creation takes too long, I could save the data in an extra file (eg pickle file) and versioning the pickled file. So I dont need an extra way to reedit a mpl graphic. By, Friedrich |
