matplotlib-devel

[matplotlib-devel] Add some caching to texmanager.py?

[Fernando P. <Fer...@co...>]

Hi all,

I just had this crash in mpl:

/usr/lib/python2.3/site-packages/matplotlib/texmanager.py in 
get_dvipng_version(self)
     299     def get_dvipng_version(self):
     300         if self.dvipngVersion is not None: return 
self.dvipngVersion
--> 301         sin, sout = os.popen2('dvipng --version')
     302         for line in sout.readlines():
     303             if line.startswith('dvipng '):

/usr/lib/python2.3/os.py in popen2(cmd, mode, bufsize)
     614         def popen2(cmd, mode="t", bufsize=-1):
     615             import popen2
--> 616             stdout, stdin = popen2.popen2(cmd, bufsize)
     617             return stdin, stdout
     618         __all__.append("popen2")

/usr/lib/python2.3/popen2.py in popen2(cmd, bufsize, mode)
     145         specified, it sets the buffer size for the I/O pipes. 
The file objects
     146         (child_stdout, child_stdin) are returned."""
--> 147         inst = Popen3(cmd, False, bufsize)
     148         return inst.fromchild, inst.tochild
     149

/usr/lib/python2.3/popen2.py in __init__(self, cmd, capturestderr, bufsize)
      40         if capturestderr:
      41             errout, errin = os.pipe()
---> 42         self.pid = os.fork()
      43         if self.pid == 0:
      44             # Child

OSError: [Errno 12] Cannot allocate memory


This was in a long-running session with very large memory allocations, 
but it dawned on me that get_dvipng_version(self) should cache its 
return value.  There's no point in forcing a popen() call every single 
time, is there?

I don't know the texmanager code, so I don't want to touch it myself. 
But if the idea is OK, a simple property tag is enough to cache:


In [3]: def cached():
    ...:     try:
    ...:         return cached.cachedvalue
    ...:     except AttributeError:
    ...:         cached.cachedvalue=42
    ...:         return cached.cachedvalue
    ...:

In [4]: cached()
Out[4]: 42

I also just saw pylab crash when a user was trying to run with $PWD 
being something he didn't have write access to.  Are there any checks in 
the code to fall back to /tmp or something sensible if texmanager can't 
write the temp files it needs?  Sorry for not giving a traceback, but I 
only saw this on someone else's screen while helping them, and for some 
odd reason it's not happening on my box.

Cheers,

f

Re: [matplotlib-devel] Add some caching to texmanager.py?

[John H. <jdh...@ac...>]

>>>>> "Fernando" == Fernando Perez <Fer...@co...> writes:

    Fernando> This was in a long-running session with very large
    Fernando> memory allocations, but it dawned on me that
    Fernando> get_dvipng_version(self) should cache its return value.
    Fernando> There's no point in forcing a popen() call every single
    Fernando> time, is there?

It already does cache the version

    def get_dvipng_version(self):
        if self.dvipngVersion is not None: return self.dvipngVersion
        sin, sout = os.popen2('dvipng --version')
        ...snip...

I think the reason it is failing in your case is that is each renderer
creation creates a new TexManager instance.  The cache should be on a
class basis and not on a per instance basis (slaps self on head).
Moving

        self.dvipngVersion = None

to the class level and out of __init__ should suffice.  I made this
change, and similar changes in texmanager and backend_agg, so that
caching is shared between instances.  Update from CVS and let me know
if everything still works :-)

    Fernando> I also just saw pylab crash when a user was trying to
    Fernando> run with $PWD being something he didn't have write
    Fernando> access to.  Are there any checks in the code to fall
    Fernando> back to /tmp or something sensible if texmanager can't
    Fernando> write the temp files it needs?  Sorry for not giving a
    Fernando> traceback, but I only saw this on someone else's screen
    Fernando> while helping them, and for some odd reason it's not
    Fernando> happening on my box.

This is probably failing on the tex/latex temporary files.  I spent
some time initially trying to figure out how to get these to go into
~/.tex.cache but didn't succeed.  If anyone knows how to direct
tex/latex to put the various *.aux, *.log, etc, files in a specified
directory, pass it my way.

Thanks,
JDH

Re: [matplotlib-devel] Add some caching to texmanager.py?

[Fernando P. <Fer...@co...>]

John Hunter wrote:
>>>>>>"Fernando" == Fernando Perez <Fer...@co...> writes:
> 
> 
>     Fernando> This was in a long-running session with very large
>     Fernando> memory allocations, but it dawned on me that
>     Fernando> get_dvipng_version(self) should cache its return value.
>     Fernando> There's no point in forcing a popen() call every single
>     Fernando> time, is there?
> 
> It already does cache the version
> 
>     def get_dvipng_version(self):
>         if self.dvipngVersion is not None: return self.dvipngVersion
>         sin, sout = os.popen2('dvipng --version')
>         ...snip...
> 
> I think the reason it is failing in your case is that is each renderer
> creation creates a new TexManager instance.  The cache should be on a
> class basis and not on a per instance basis (slaps self on head).
> Moving
> 
>         self.dvipngVersion = None
> 
> to the class level and out of __init__ should suffice.  I made this
> change, and similar changes in texmanager and backend_agg, so that
> caching is shared between instances.  Update from CVS and let me know
> if everything still works :-)

swamped at the moment, but I'll let you know if I see problems when I 
get a chance.

>     Fernando> I also just saw pylab crash when a user was trying to
>     Fernando> run with $PWD being something he didn't have write
>     Fernando> access to.  Are there any checks in the code to fall
>     Fernando> back to /tmp or something sensible if texmanager can't
>     Fernando> write the temp files it needs?  Sorry for not giving a
>     Fernando> traceback, but I only saw this on someone else's screen
>     Fernando> while helping them, and for some odd reason it's not
>     Fernando> happening on my box.
> 
> This is probably failing on the tex/latex temporary files.  I spent
> some time initially trying to figure out how to get these to go into
> ~/.tex.cache but didn't succeed.  If anyone knows how to direct
> tex/latex to put the various *.aux, *.log, etc, files in a specified
> directory, pass it my way.

I don't really know, sorry.  My texpertise is pretty limited, as you know :)

Thanks for the fixes!

f

Re: [matplotlib-devel] Add some caching to texmanager.py?

[Stephen W. <ste...@cs...>]

John Hunter wrote:

>This is probably failing on the tex/latex temporary files.  I spent
>some time initially trying to figure out how to get these to go into
>~/.tex.cache but didn't succeed.  If anyone knows how to direct
>tex/latex to put the various *.aux, *.log, etc, files in a specified
>directory, pass it my way.
>  
>
 From "man tex" on FC3, the section on environment variables:

       TEXMFOUTPUT
              Normally,  TeX  puts its output files in the current 
directory.
              If any output file cannot be opened there, it tries to 
open  it
              in  the  directory  specified in the environment variable 
TEXM-
              FOUTPUT.  There is no default value  for  that  
variable.   For
              example,  if you say tex paper and the current directory 
is not
              writable, if TEXMFOUTPUT has the value /tmp,  TeX  
attempts  to
              create  /tmp/paper.log  (and  /tmp/paper.dvi,  if any 
output is
              produced.)

Re: [matplotlib-devel] Add some caching to texmanager.py?

[John H. <jdh...@ac...>]

>>>>> "Stephen" == Stephen Walton <ste...@cs...> writes:

    Stephen>        TEXMFOUTPUT Normally, TeX puts its output files in
    Stephen> the current directory.  If any output file cannot be
    Stephen> opened there, it tries to open it in the directory
    Stephen> specified in the environment variable TEXM- FOUTPUT.
    Stephen> There is no default value for that variable.  For
    Stephen> example, if you say tex paper and the current directory
    Stephen> is not writable, if TEXMFOUTPUT has the value /tmp, TeX
    Stephen> attempts to create /tmp/paper.log (and /tmp/paper.dvi, if
    Stephen> any output is produced.)

Hmm, that looks familiar -- I think I did come across that.  But this
will be a bit of a pain to set from matplotlib, since we are making
calls to os.popen.  We might have to do something like

  os.popen('export TEXMFOUTPUT=/some/path; run some command')

and then deal with platform and shell dependent differences in how to
set environment variables.  Is there a better way?  Me thinks it is
better to make this a FAQ and advise people working in non-writable
dirs to set this var themselves.

Thanks for the help...

JDH

Re: [matplotlib-devel] Add some caching to texmanager.py?

[Stephen W. <ste...@cs...>]

John Hunter wrote:

>We might have to do something like
>
>  os.popen('export TEXMFOUTPUT=/some/path; run some command')
>
>and then deal with platform and shell dependent differences in how to
>set environment variables.  Is there a better way?  Me thinks it is
>better to make this a FAQ and advise people working in non-writable
>dirs to set this var themselves.
>
I agree with this last.  On Unix, you could do

os.popen('/bin/sh -c "export TEXMFOUTPUT=/some/path; run some command"')

and be pretty sure it would work on all systems.  But Windows would be a 
pain.  I agree with the FAQ solution.  This seems like a very rare 
problem to me, to be honest.

Re: [matplotlib-devel] Add some caching to texmanager.py?

[Fernando P. <Fer...@co...>]

Stephen Walton wrote:
> John Hunter wrote:
> 
> 
>>We might have to do something like
>>
>> os.popen('export TEXMFOUTPUT=/some/path; run some command')
>>
>>and then deal with platform and shell dependent differences in how to
>>set environment variables.  Is there a better way?  Me thinks it is
>>better to make this a FAQ and advise people working in non-writable
>>dirs to set this var themselves.
>>
> 
> I agree with this last.  On Unix, you could do
> 
> os.popen('/bin/sh -c "export TEXMFOUTPUT=/some/path; run some command"')
> 
> and be pretty sure it would work on all systems.  But Windows would be a 
> pain.  I agree with the FAQ solution.  This seems like a very rare 
> problem to me, to be honest.

It may be simpler than that:

planck[~/test]> cat env.py
#!/usr/bin/env python
import os

print 'before'
os.system('env | grep TEXMFOUTPUT')

os.environ['TEXMFOUTPUT'] = '/some/path'

print 'after'
os.system('env | grep TEXMFOUTPUT')
planck[~/test]> ./env.py
before
after
TEXMFOUTPUT=/some/path
planck[~/test]>

Cheers,

f

Re: [matplotlib-devel] Add some caching to texmanager.py?

[Stephen W. <ste...@cs...>]

Fernando Perez wrote:

> It may be simpler than that:

So os.environ does change the environment under Python and its 
children.  The remaining question is what should TEXMFOUTPUT be?  
/home/user/.tex.cache will not work across all platforms.

Re: [matplotlib-devel] Add some caching to texmanager.py?

[Fernando P. <Fer...@co...>]

Stephen Walton wrote:
> Fernando Perez wrote:
> 
> 
>>It may be simpler than that:
> 
> 
> So os.environ does change the environment under Python and its 
> children.  The remaining question is what should TEXMFOUTPUT be?  
> /home/user/.tex.cache will not work across all platforms.

Well, it could be something like $HOME/.tex.cache, where $HOME can be 
determined via a routine like the below (this is what ipython uses to 
try and guess a sensible value for $HOME):

#----------------------------------------------------------------------------
class HomeDirError(Error):
     pass

def get_home_dir():
     """Return the closest possible equivalent to a 'home' directory.

     We first try $HOME.  Absent that, on NT it's $HOMEDRIVE\$HOMEPATH.

     Currently only Posix and NT are implemented, a HomeDirError 
exception is
     raised for all other OSes. """ #'

     try:
         return os.environ['HOME']
     except KeyError:
         if os.name == 'posix':
             raise HomeDirError,'undefined $HOME, IPython can not proceed.'
         elif os.name == 'nt':
             # For some strange reason, win9x returns 'nt' for os.name.
             try:
                 return 
os.path.join(os.environ['HOMEDRIVE'],os.environ['HOMEPATH'])
             except:
                 try:
                     # Use the registry to get the 'My Documents' folder.
                     import _winreg as wreg
                     key = wreg.OpenKey(wreg.HKEY_CURRENT_USER,
 
"Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders")
                     homedir = wreg.QueryValueEx(key,'Personal')[0]
                     key.Close()
                     return homedir
                 except:
                     return 'C:\\'
         elif os.name == 'dos':
             # Desperate, may do absurd things in classic MacOS. May 
work under DOS.
             return 'C:\\'
         else:
             raise HomeDirError,'support for your operating system not 
implemented.'

Cheers,

f

Re: [matplotlib-devel] Add some caching to texmanager.py?

[Stephen W. <ste...@cs...>]

Fernando Perez wrote:

> Well, it could be something like $HOME/.tex.cache, where $HOME can be 
> determined via a routine like the below (this is what ipython uses to 
> try and guess a sensible value for $HOME):

I *like* it.

Re: [matplotlib-devel] Add some caching to texmanager.py?

[Fernando P. <Fer...@co...>]

Stephen Walton wrote:
> Fernando Perez wrote:
> 
> 
>>Well, it could be something like $HOME/.tex.cache, where $HOME can be 
>>determined via a routine like the below (this is what ipython uses to 
>>try and guess a sensible value for $HOME):
> 
> 
> I *like* it.

Though I'd personally vote for matplotlib holding $HOME/.matplotlib/ as a 
directory, and putting in there a tex.cache dir, the matplotlibrc file, and 
anything else it may need in the future.

I *really* don't like the idea that matplotlib will begin to put a bunch of 
differently named things under $HOME with various .foo names.  Ipython also 
started with .ipythonrc, and I quickly moved to an .ipython/ directory, where 
I stuff anything I need.  It's future-proof, clean, and gives an easy way for 
users to clean up if needed, without having to guess 'how many dot 
files/directories does this thing create'?

So if you were to ask for my opinion, I'd vote +100 on matplotlib moving to a 
single directory for holding *ALL* user and configuration data, which would 
default to $HOME/.matplotlib, and which could be overridden if the 
$MATPLOTLIBDIR environment variable is defined.  That's exactly how ipython 
works, so it must be the perfect solution :)

Cheers,

f

Re: [matplotlib-devel] Add some caching to texmanager.py?

[Darren D. <dd...@co...>]

On Wednesday 22 June 2005 06:35 pm, Fernando Perez wrote:
> Stephen Walton wrote:
> > Fernando Perez wrote:
> >>Well, it could be something like $HOME/.tex.cache, where $HOME can be
> >>determined via a routine like the below (this is what ipython uses to
> >>try and guess a sensible value for $HOME):
> >
> > I *like* it.
>
> Though I'd personally vote for matplotlib holding $HOME/.matplotlib/ as a
> directory, and putting in there a tex.cache dir, the matplotlibrc file, and
> anything else it may need in the future.

I second that notion. In fact, I have been meaning to suggest this on the list 
for some time now. Thank for reminding me.

> I *really* don't like the idea that matplotlib will begin to put a bunch of
> differently named things under $HOME with various .foo names.  Ipython also
> started with .ipythonrc, and I quickly moved to an .ipython/ directory,
> where I stuff anything I need.  It's future-proof, clean, and gives an easy
> way for users to clean up if needed, without having to guess 'how many dot
> files/directories does this thing create'?
>
> So if you were to ask for my opinion, I'd vote +100 on matplotlib moving to
> a single directory for holding *ALL* user and configuration data, which
> would default to $HOME/.matplotlib, and which could be overridden if the
> $MATPLOTLIBDIR environment variable is defined.  That's exactly how ipython
> works, so it must be the perfect solution :)

+101

Darren

Re: [matplotlib-devel] Add some caching to texmanager.py?

[John H. <jdh...@ac...>]

>>>>> "Fernando" == Fernando Perez <Fer...@co...> writes:

    Fernando> Stephen Walton wrote:
    >> Fernando Perez wrote:
    >>> Well, it could be something like $HOME/.tex.cache, where $HOME
    >>> can be determined via a routine like the below (this is what
    >>> ipython uses to try and guess a sensible value for $HOME):
    >> I *like* it.

    Fernando> Though I'd personally vote for matplotlib holding
    Fernando> $HOME/.matplotlib/ as a directory, and putting in there
    Fernando> a tex.cache dir, the matplotlibrc file, and anything
    Fernando> else it may need in the future.

I am in the process of moving all of matplotlib's config files and
outputs to HOME/.matplotlib.

I want to issue a warning to users if I find a .matplotlibrc file (the
new name is matplotlibrc (no hidden "dot") and the default location is
HOME/.matplotlib/matplotlibrc.  As before, you can put an rc file in
the current dir.

I am using warnings.warn to issue the warning

    home = get_home()
    oldname = os.path.join( home, '.matplotlibrc')
    if os.path.exists(oldname):
        warnings.warn('Old rc filename "%s" found and ignored; new default rc file name is HOME/.matplotlib/matplotlibrc"'%oldname)

But this results in a warning that looks like this

  > python ~/python/projects/matplotlib/examples/subplot_demo.py
  /usr/lib/python2.4/site-packages/matplotlib/__init__.py:737: UserWarning: Old rc filename "/home/jdhunter/.matplotlibrc" found and ignored; new default rc file name is HOME/.matplotlib/matplotlibrc"
    warnings.warn('Old rc filename "%s" found and ignored; new default rc file name is HOME/.matplotlib/matplotlibrc"'%oldname)

which I find hard to read.  I can add a stacklevel=0 argument to warn
which is slightly more legible

  > python ~/python/projects/matplotlib/examples/subplot_demo.py
  /usr/lib/python2.4/warnings.py:41: UserWarning: Old rc filename "/home/jdhunter/.matplotlibrc" found and ignored; new default rc file name is HOME/.matplotlib/matplotlibrc"
    lineno = caller.f_lineno 

but still has the annoying "lineno = caller.f_lineno"

what I am really after is a simple print >> sys.stderr.  Is there
anything wrong with using sys.stderr for this kind of thing, error
mavens?  I know when we discussed this many moons ago we converged on
using exceptions and the warnings module everywhere.  But now I am a
bit confounded by the inability to make the warnings print pretty
messages that the typical user will read and understand.  Or is there
a way to tell warning.warn to just print the error message with no
lineno/traceback info?

JDH

Re: [matplotlib-devel] Add some caching to texmanager.py?

[Fernando P. <Fer...@co...>]

John Hunter wrote:
>>>>>>"Fernando" == Fernando Perez <Fer...@co...> writes:
> 
> 
>     Fernando> Stephen Walton wrote:
>     >> Fernando Perez wrote:
>     >>> Well, it could be something like $HOME/.tex.cache, where $HOME
>     >>> can be determined via a routine like the below (this is what
>     >>> ipython uses to try and guess a sensible value for $HOME):
>     >> I *like* it.
> 
>     Fernando> Though I'd personally vote for matplotlib holding
>     Fernando> $HOME/.matplotlib/ as a directory, and putting in there
>     Fernando> a tex.cache dir, the matplotlibrc file, and anything
>     Fernando> else it may need in the future.
> 
> I am in the process of moving all of matplotlib's config files and
> outputs to HOME/.matplotlib.

Great!

> what I am really after is a simple print >> sys.stderr.  Is there
> anything wrong with using sys.stderr for this kind of thing, error
> mavens?  I know when we discussed this many moons ago we converged on
> using exceptions and the warnings module everywhere.  But now I am a
> bit confounded by the inability to make the warnings print pretty
> messages that the typical user will read and understand.  Or is there
> a way to tell warning.warn to just print the error message with no
> lineno/traceback info?

Dunno, sorry.   In ipython, I have a trivial warn() routine which is just a 
print >> sys.stderr wrapper.  I've never used the stdlib's warnings module.

Cheers,

f

Re: [matplotlib-devel] Add some caching to texmanager.py?

[Nicholas Y. <su...@su...>]

On Wed, 2005-06-22 at 11:45 -0600, Fernando Perez wrote:
> os.environ['TEXMFOUTPUT'] = '/some/path'

According to the online docs
(http://docs.python.org/lib/os-procinfo.html) setting os.environ isn't
safe/available for all platforms.  You can use the subprocess module to
set the environment of a subprocess under python 2.4 but I don't think
there's a simple way to do this and capture the output for earlier
versions.

Nick

Re: [matplotlib-devel] Add some caching to texmanager.py?

[Andrew S. <str...@as...>]

On Jun 21, 2005, at 1:09 PM, John Hunter wrote:

>     Fernando> I also just saw pylab crash when a user was trying to
>     Fernando> run with $PWD being something he didn't have write
>     Fernando> access to.  Are there any checks in the code to fall
>     Fernando> back to /tmp or something sensible if texmanager can't
>     Fernando> write the temp files it needs?  Sorry for not giving a
>     Fernando> traceback, but I only saw this on someone else's screen
>     Fernando> while helping them, and for some odd reason it's not
>     Fernando> happening on my box.
>
> This is probably failing on the tex/latex temporary files.  I spent
> some time initially trying to figure out how to get these to go into
> ~/.tex.cache but didn't succeed.  If anyone knows how to direct
> tex/latex to put the various *.aux, *.log, etc, files in a specified
> directory, pass it my way.

I get this when mpl complains about not being able to create a 
.ttffont.cache (or whatever it's called) file.  I'll submit a bug on 
the tracker later.

Cheers!
Andrew

Re: [matplotlib-devel] Add some caching to texmanager.py?

[Fernando P. <Fer...@co...>]

Nicholas Young wrote:
> On Wed, 2005-06-22 at 11:45 -0600, Fernando Perez wrote:
> 
>>os.environ['TEXMFOUTPUT'] = '/some/path'
> 
> 
> According to the online docs
> (http://docs.python.org/lib/os-procinfo.html) setting os.environ isn't
> safe/available for all platforms.  You can use the subprocess module to
> set the environment of a subprocess under python 2.4 but I don't think
> there's a simple way to do this and capture the output for earlier
> versions.

Well, after reading that I get that os.environ _is_ writable everywhere, it's 
just that it may leak memory in OSX/BSD.  What's not always available is the 
putenv() call, but python will find its way around it if needed.

Since this would be a once-only call, I think that leaking a few bytes is an 
acceptable price to pay to prevent a crash if the user happens to be 
positioned on a non-writable dir.

Cheers,

f

Re: [matplotlib-devel] Add some caching to texmanager.py?

[Nicholas Y. <su...@su...>]

On Thu, 2005-06-23 at 10:58 -0600, Fernando Perez wrote:
> Nicholas Young wrote:
> > On Wed, 2005-06-22 at 11:45 -0600, Fernando Perez wrote:
> > 
> >>os.environ['TEXMFOUTPUT'] = '/some/path'
> > 
> > 
> > According to the online docs
> > (http://docs.python.org/lib/os-procinfo.html) setting os.environ isn't
> > safe/available for all platforms.  You can use the subprocess module to
> > set the environment of a subprocess under python 2.4 but I don't think
> > there's a simple way to do this and capture the output for earlier
> > versions.
> 
> Well, after reading that I get that os.environ _is_ writable everywhere, it's 
> just that it may leak memory in OSX/BSD.  What's not always available is the 
> putenv() call, but python will find its way around it if needed.

To quote "If putenv() is not provided, this mapping may be passed to the
appropriate process-creation functions to cause child processes to use a
modified environment.".  To me this implies that you have to pass
os.environ to a process-creation function supporting the env keyword
(os.execve, etc.) none of which seem to support capturing output.  On
the other hand does anyone actually run mpl on a platform which doesn't
support putenv?

> Since this would be a once-only call, I think that leaking a few bytes is an 
> acceptable price to pay to prevent a crash if the user happens to be 
> positioned on a non-writable dir.

After reading an online copy of the freebsd man putenv it actually reads
"Successive calls to setenv() or putenv() assigning a differently sized
value to the same name will result in a memory leak." so setting this
once wouldn't be a problem.

So I was wrong and setting os.environ seems reasonable in this case -
but it seems sensible to be aware of the potential for causing problems
if anyone trys to get mpl working on an odd platform.

Nick