matplotlib-users — Discussion related to using matplotlib

Re: [Matplotlib-users] Mathtext doesn't work on the PS backend

[Paul B. <ba...@st...>]

Serge Barral wrote:

>I'm a new user of matlibplot, and have difficulties in making mathtext to work with the PS backend. Instead of the math symbols, the raw TeX expression is displayed. Mathtext works fine on the GUI backends and on raster images output, though.
>
>The following quote from matplotlib web page made suggests that mathtext should work for PS output:
>"mathtext now embeds the TrueType computer modern fonts into the PS file, so what you see on the screen should be what you get on paper"
>Did I misunderstood?
>  
>
What version of matplotlib are you using?  Versions earlier than 0.60.2 
may not work.

I just uploaded improved TrueType Font (TTF) support for PS last night, 
so the version on CVS should fix this. Please give it a try and let me 
know if you have any problems.

>I am on debian unstable and initially tried Vittorio's packages. Thinking that this could be the problem, I resolved to install matplotlib from sources, but the result is the same.
>I did check that BUILD_FT2FONT=1 is set.
>  
>
 -- Paul

-- 
Paul Barrett, PhD      Space Telescope Science Institute
Phone: 410-338-4475    ESS/Science Software Branch
FAX:   410-338-4767    Baltimore, MD 21218

[Matplotlib-users] Mathtext doesn't work on the PS backend

[Serge B. <se...@op...>]

I'm a new user of matlibplot, and have difficulties in making mathtext to work with the PS backend. Instead of the math symbols, the raw TeX expression is displayed. Mathtext works fine on the GUI backends and on raster images output, though.

The following quote from matplotlib web page made suggests that mathtext should work for PS output:
"mathtext now embeds the TrueType computer modern fonts into the PS file, so what you see on the screen should be what you get on paper"
Did I misunderstood?

I am on debian unstable and initially tried Vittorio's packages. Thinking that this could be the problem, I resolved to install matplotlib from sources, but the result is the same.
I did check that BUILD_FT2FONT=1 is set.

Thank you in advance for any help,

Serge

[Matplotlib-users] Widgets

[Stephen W. <ste...@cs...>]

As a new user of matplotlib, I'm surprised I haven't seen this
mentioned, offhand, in the mailing list archive.  I am not seeing the
redraw or close widgets on the plot windows I produce.  The platform is
Fedora Core 1, although I built on RHEL3 because matplotlib-0.60-2 won't
build on FC1 (some problem with tk-devel, apparently).

-- 
Stephen Walton <ste...@cs...>
Dept. of Physics & Astronomy, Cal State Northridge

Re: [Matplotlib-users] Problems compiling matplotlib under OS X

[John H. <jdh...@ac...>]

>>>>> "Kenneth" == Kenneth McDonald <ken...@sb...> writes:

    Kenneth> The easiest way to describe this is to repeat the info
    Kenneth> that's been posted into the "macpython" mailing
    Kenneth> list--sorry if you've already seen it. According to one
    Kenneth> of the people who helped me on the macpython list, the
    Kenneth> problem I'm having is likely caused by matplotlib using a
    Kenneth> custom method of finding Tcl/Tk headers, rather than by a
    Kenneth> Python/OS X interaction problem, which is why I'm
    Kenneth> bringing the topic over to this list.

The file I attached in my last email had an error in it.  It would
probably work for darwin, but fail for linux.  Here is the correct
file

http://matplotlib.sf.net/setupext.py

JDH

Re: [Matplotlib-users] Matplotlib and pygtk

[John H. <jdh...@ac...>]

>>>>> "MWallis" == MWallis  <mw...@sw...> writes:


    MWallis> This has been my first time using matplotlib and it has
    MWallis> not been entirely successful .  I was asked to develop a
    MWallis> GUI using Python and omniORB to connect as a client to a
    MWallis> server and request a data stream which they wanted me to
    MWallis> display.  There is a considerable amount of data that is
    MWallis> generated and I am not sure that Python was going to be
    MWallis> able to handle it.  I developed the app in Python and was
    MWallis> able to connect to the server using omniORB and register
    MWallis> a callback and receive the data.  This worked fine.  I
    MWallis> then created the GUI using matplotlib and pygtk.  The
    MWallis> display included 2 spectrograms a pan display and 2 bar
    MWallis> charts.  After data of the appropriate type was received
    MWallis> they wanted the displays to be updated dynamically.  I
    MWallis> used threading but it still seemed that after awhile the
    MWallis> GUI would lock up.  Are there any examples available that
    MWallis> embed multiple dynamically updated figures using gtk? I'm
    MWallis> afraid that if I can't get this fixed they will write
    MWallis> python and matplotlib off and refuse to use it again.

Hi Melissa,

I have little experience with threads and try to avoid using them
directly - I'll leave that to the pros.  I suggest another approach -
a useful pygtk trick is to use the idle manager to avoid calls when
gtk is busy.  In the script below, new axes images are created and
drawn only when gtk is idle.  It doesn't do you any good to force feed
gtk data it can't handle, so only make calls through the idle handler.

I simulate a data event generator that calls one of two image axes.
On my system, I get about 10 frames per second with the script below
using gtkagg (gtkagg will probably be a little faster than gtk, which
uses string methods to render image data).

As for your clients giving up on us, remind them that image handling
in matplotlib has gotten an order of magnitude faster in the last 5
months and that further enhancements for dynamic updating are in the
works.  One important one will be the ability to render only selective
parts of the canvas (eg, just redraw the axes image portion rather
than the entire canvas).  My guess is that this change alone would
roughly double the performance of gtkagg for dynamic images.  Other
optimizations specifically for updating image data would provide
further performance enhancements, since this is a simple agg->agg
rendering buffer transfer.

Also, although matplotlib-0.60.2 was just released, a bug that affects
image data with data extent set (eg specgram data) was recently fixed
in CVS.  You may want to work with
http://nitace.bsd.uchicago.edu:8080/files/share/matplotlib-0.60.3a.tar.gz.
It's not a critical update as the bug is only exposed in the following
combination: 1) image data with extent set, 2) image.origin = 'lower'
and 3) you have changed the axes limits from their defaults.  May seem
arcane, but nonetheless, Andrew Straw managed to find it within about
6 minutes of my CVS checkin.

Let me know if this helps,
JDH

#!/usr/bin/env python
"""
An animated image
"""
import sys, time, os, gc
from matplotlib import rcParams

from matplotlib.matlab import *
import gtk

# if hold is on the axes images will accumulate and your performance
# will tank!
rc('axes', hold=False)

class HandleDraws:
    drawing_idle_id = 0
    shape = 100,100  # image size
    cnt = 0
    def __init__(self):
        self.fig = figure(1)
        self.a1 = subplot(211)
        self.a2 = subplot(212)
        
    def idle_update(self, *args):
        'only call a draw if gtk is idle'
        if self.cnt==0: self.tstart = time.time()
        draw()
        self.drawing_idle_id = 0
        self.cnt += 1
        if self.cnt>=50:
            print 'FPS', self.cnt/(time.time() - self.tstart)
            sys.exit()
            
        return False

    def update1(self, data):
        if self.drawing_idle_id == 0:
            self.a1.imshow(data, interpolation='nearest')
            self.drawing_idle_id = gtk.idle_add(self.idle_update)
        else: print 'dropping frame for axes 1'

    def update2(self, data):
        if self.drawing_idle_id == 0:
            self.a2.imshow(data, interpolation='nearest')
            self.drawing_idle_id = gtk.idle_add(self.idle_update)
        else: print 'dropping frame for axes 2'

handler = HandleDraws()

def generate_events(*args):
    data = rand(100,100)

    # randomly pick which axes to update
    if rand()>0.5: handler.update1(data)
    else:          handler.update2(data)
    return True

cnt = 0

gtk.timeout_add(10, generate_events)
show()

Re: [Matplotlib-users] map projections

[John H. <jdh...@ac...>]

>>>>> "Jeff" == Jeff Whitaker <js...@fa...> writes:

    Jeff> I'm pretty sure it's not the data.  I only see this when I
    Jeff> use imshow, not pcolor.

OK, I'll get this sorted out.  Thanks for letting me know.

    Jeff> Sure.  I've reworked the example a bit - now I read in a
    Jeff> regular lat/lon grid from a pickle and use numarray's spline
    Jeff> interpolation function to interpolate to the native
    Jeff> projection grid.  Here's the modified example:

It looks very nice.  Perry Greenfield has provided a nice framework
with matplotlib.colors.LinearSegmentedColormap to define new
colormaps.  You can create new colormaps fairly easy by following the
example of jet in matplotlib.cm.  It would be very nice if you could
add some of the common cartographic colormaps.  If you do get the time
to do so when working on your mapping code, here are the steps

  * define your rgb linear segments in matplotlib.cm, following the lead
    of the _jet_data dictionary in that module

  * add an entry to the datad dictionary in that module which maps rc
    string names for your color map to the dictionary you just
    defined.

  * instantiate a single instance of your colormap in cm, following
    the example
    
    jet =  colors.LinearSegmentedColormap('jet',  _jet_data, LUTSIZE)

  * add a matplotlib.matlab function which has the same name as your
    colormap, following the example of matplotlib.matlab.jet.

Now anyone can use the colormap interactively from the shell, by
setting it as the default image.cmap in rc, etc.

JDH

Re: [Matplotlib-users] debian

[John H. <jdh...@ac...>]

>>>>> "Charles" == Charles R Twardy <cha...@in...> writes:

    Charles> I can't see the matplotlib package with "apt-get".  I
    Charles> have these lines in my sources.list and I've rerun
    Charles> "apt-get update".

    Charles> # Matplotlib deb http://mentors.debian.net/debian
    Charles> unstable main contrib non-free deb-src
    Charles> http://mentors.debian.net/debian unstable main contrib
    Charles> non-free

    Charles> I can see matplotlib-doc, but not matplotlib. Any ideas?

Hi Charles, I think messages about the debian distribution should be
sent to the devel list; I'm not sure Vittorio reads the users list.
I've CCd him on this email.

Cheers,
JDH

Re: [Matplotlib-users] python crash

[John H. <jdh...@ac...>]

>>>>> "Brett" == Brett Calcott <br...@co...> writes:

    Brett> win32, xp, python 2.3.3 matplotlib: __version__ = '0.60.2'
    Brett> __revision__ = '$Revision: 1.64 $'

    Brett> Any attempt to save ps: savefig('fig.ps')

    Brett> gives this:

    Brett> Fatal Python error: PyEval_RestoreThread: NULL tstate

    Brett> This application has requested the Runtime to terminate it
    Brett> in an unusual way.  Please contact the application's
    Brett> support team for more information.

    Brett> The postscript file *is* actually written.

    Brett> What more info do you need to track this?

Bizarre - on virtually the same platform (windows xp, 0.60.2,
enthought python 2.3.3) I can save PS either directly

  > python simple_plot.py -dPS

or from the GUI by clicking save and using a ps or eps extension with
no troubles, warnings or errors.

As for more information: how are you running matplotlib when you try
to save ps (ie from a GUI window, from an IDE, by setting ps as the
backend in rc, interactively from the python shell)?  Do you still get
the same error when calling a script from the command shell with -dPS
as above?  Which matplotlib windows installer are you using:

  matplotlib-0.60.2-numarray0.9-win32-py2.3.exe
  matplotlib-0.60.2-numarray1.0-win32-py2.3.exe 
  matplotlib-0.60.2.win32-py2.3.exe?

What is your numerix setting in
c:\python23\share\matplotlib\.matplotlibrc, or wherever you keep your
rc file?  Do you get this problem when you try to save png with the
agg backend?

After gathering the info above, I suggest removing
c:\python23\lib\site-packages\matplotlib and
c:\python23\share\matplotlib and doing a clean install.

From your message (PyEval_RestoreThread: NULL), I'm guessing you are
running from an IDE (idle?)  with the tkagg backend and are getting a
mainloop conflict.  Have you read
http://matplotlib.sourceforge.net/faq.html#FREEZE?

Good luck!  JDH

[Matplotlib-users] debian

[Charles R. T. <cha...@in...>]

I can't see the matplotlib package with "apt-get".
I have these lines in my sources.list and I've rerun "apt-get update".

# Matplotlib
deb http://mentors.debian.net/debian unstable main contrib non-free
deb-src http://mentors.debian.net/debian unstable main contrib non-free

I can see matplotlib-doc, but not matplotlib. Any ideas?

-Charles

-- 
Charles R. Twardy                    www.csse.monash.edu.au/~ctwardy
Monash University                    sarbayes.org
Computer Sci. & Software Eng.        +61(3) 9905 5823 (w)  5146 (fax)

Allow the president to invade a neighboring nation, whenever he shall
deem it necessary to repel an invasion, ...  and you allow him to make
war at pleasure.                               --Abraham Lincoln

Re: [Matplotlib-users] map projections

[Philip A. <pa...@eo...>]

Jeff  Whitaker writes:

 > When I get some time I'd like to wrap some of this in some
 > easy-to-use functions, maybe with an interface similar to the
 > matlab mapping toolbox.

That would be very nice -- I think the matlab mapping/colorscaling
examples at

http://www.mathworks.com/company/newsletters/digest/nov02/earth_pt4.html
http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectType=file&objectId=2611

represent the vast majority of things that people would want to do
with map/image overlays.

Regards, Phil

Re: [Matplotlib-users] map projections

[Jeff W. <js...@fa...>]

On Wed, 14 Jul 2004, John Hunter wrote:

>>>>>> "Jeff" == Jeff Whitaker <js...@fa...> writes:
>
> I noticed that the lat/lon lines don't precisely agree with the
> colormap, eg around the Aleutian Islands the light blue is not
> perfectly registered with the black lines.  Should I be concerned that
> this reflects a problem in matplotlib, or is this kind of error
> standard in the data you've provided?  I think this is related to the
> pixel border that appears around some images, and is magnified when
> interpolation is used because the top and right borders are not
> defined when interpolating.  I'll continue to look into this.

I'm pretty sure it's not the data.  I only see this when I use imshow, not 
pcolor.

>
> Would it be OK if I used this example on the web page?  I like it!
>

Sure.  I've reworked the example a bit - now I read in a regular lat/lon 
grid from a pickle and use numarray's spline interpolation function to 
interpolate to the native projection grid.  Here's the modified example:

from matplotlib.matlab import *
from matplotlib.collections import LineCollection 
from proj import Proj
import numarray, cPickle
from numarray import nd_image

# example to demonstrate plotting data on a map projection.
# requires numarray, proj module (which in turn requires 
# proj command from http://proj.maptools.org)

# set up map projection parameters (lambert conformal conic,
# standard parallels at 50 deg N, center longitued 107 deg W.
params = {}
params['proj'] = 'lcc'
params['R'] = 63712000
params['lat_1'] = 50
params['lat_2'] = 50
params['lon_0'] = -107
proj = Proj(params)
llcornerx, llcornery = proj(-145.5,1.)
xmax=11297266.68; ymax=8959901.16
params['x_0'] = -llcornerx # add cartesian offset so lower left corner = (0,0)
params['y_0'] = -llcornery
# create a Proj instance for desired map.
proj = Proj(params)

# define grid (nx x ny regularly spaced native projection grid)
nx = 349; ny = 277 
dx = xmax/(nx-1); dy = ymax/(ny-1)
xgrid = dx*numarray.indices((ny,nx))[1,:,:]
ygrid = dy*numarray.indices((ny,nx))[0,:,:]
# compute lons, lats of regular projection grid.
lonout, latout = proj(xgrid, ygrid, inverse=True)
# make sure lons are between 0 and 360
lonout = numarray.where(lonout < 0, lonout+360, lonout)
latout = latout+90

# read in topo data from pickle (on a regular lat/lon grid)
topodict = cPickle.load(open('etopo20.pickle','rb')) 
lons = topodict['lons']
lats = topodict['lats']
topoin = topodict['topo']

# find coordinates of native projection grid.
xcoords = (len(lons)-1)*(lonout-lons[0])/(lons[-1]-lons[0])
ycoords = (len(lats)-1)*(latout-lats[0])/(lats[-1]-lats[0])
coords = [ycoords,xcoords]
# interpolate to projection grid using numarray.nd_image spline filter.
topodat = numarray.nd_image.map_coordinates(topoin,coords,mode='nearest')

ax = subplot(111) 
# use imshow rather than pcolor for speed 
# set the default params for imshow 
rc('image', origin='lower', cmap='jet') 
im = ax.imshow(topodat, interpolation='nearest',extent=(0, xmax, 0, ymax)) 
#pcolor(xgrid, ygrid, topodat, shading='flat')

# read in coastline data from pickle.
wcl = cPickle.load(open('wcl.pickle','rb'))
ind = wcl['segment_index']; lons = wcl['lons']; lats = wcl['lats']
# transform coastline segements to map projection coordinates. 
xs, ys = proj(lons,lats)
# a sequence of xy tuples 
segments = [zip(xs[i0:i1], ys[i0:i1]) for i0, i1 in zip(ind[:-1], ind[1:])]

# line collection
collection = LineCollection(
     segments,
     colors       = ( (0,0,0,1), ), # black
     linewidths   = (1.5,),
     antialiaseds = (1,),)  # turn off aa for speed

ax.add_collection(collection) 
corners = (min(xs), min(ys)), (max(xs), max(ys)) 
ax.update_datalim( corners ) 
axis([0, xmax, 0, ymax]) 
ax.set_xticks([]) # no ticks 
ax.set_yticks([]) 
title('20-minute Topography: Lambert Conformal Conic Projection') 
show()

I've tarred up all the pickles and scripts to run this at 
http://whitaker.homeunix.org/~jeff/plotmap.tar.gz.  The resulting image is 
at http://whitaker.homeunix.org/~jeff/plotmap.png. When I get some time 
I'd like to wrap some of this in some easy-to-use functions, maybe with an 
interface similar to the matlab mapping toolbox.

Thanks for all your help!

-Jeff

  -- 
Jeffrey S. Whitaker         Phone : (303)497-6313
NOAA/OAR/CDC  R/CDC1        FAX   : (303)497-6449
325 Broadway                Web   : http://www.cdc.noaa.gov/~jsw
Boulder, CO, USA 80305-3328 Office: Skaggs Research Cntr 1D-124

[Matplotlib-users] python crash

[Brett C. <br...@co...>]

win32, xp, python 2.3.3
matplotlib:
__version__  = '0.60.2'
__revision__ = '$Revision: 1.64 $'

Any attempt to save ps:
savefig('fig.ps')

gives this:

Fatal Python error: PyEval_RestoreThread: NULL tstate

This application has requested the Runtime to terminate it in an unusual  
way.
Please contact the application's support team for more information.

The postscript file *is* actually written.

What more info do you need to track this?


-- 
Brett Calcott
Philosophy Program, RSSS, ANU
Canberra, ACT 0200, AUSTRALIA

[Matplotlib-users] Problems compiling matplotlib under OS X

[Kenneth M. <ken...@sb...>]

The easiest way to describe this is to repeat the info that's been
posted into the "macpython" mailing list--sorry if you've already
seen it. According to one of the people who helped me on the
macpython list, the problem I'm having is
likely caused by matplotlib using a custom method of finding
Tcl/Tk headers, rather than by a Python/OS X interaction problem,
which is why I'm bringing the topic over to this list.

Many thanks for any help you can offer!

Here's what has gone before, consisting of questions from me and
answers from other people. I've done some editing to shorten things up.

>>>> I'm attempting to compile matplotlib, and get messages saying that
>>>> it can't find the headers for Tcl/Tk. They exist, they just happen 
>>>> to
>>>> be in the Frameworks directories for Tcl and for Tk. I know I can
>>>> get this to work by hacking (setting up path variables, putting 
>>>> symbolic
>>>> links in directories, or some such), but aside from the fact that 
>>>> that's
>>>> a pain and ugly, it doesn't solve the more general problem; if 
>>>> header
>>>> files are supposed to be in Frameworks directories (for example, I
>>>> found my Tk header files in 
>>>> /Library/Frameworks/Tk.framework/Headers),
>>>> what is the best way to set up OS X so that they will be available 
>>>> to link
>>>> against?

>>> Short answer:  Whatever you're trying to build is (incorrectly) not 
>>> using tclConfig.sh (they should allow the user to choose which one, 
>>> but should default to /usr/lib/tclConfig.sh I guess).  This is 
>>> equivalent to not using distutils.  It probably won't be a problem 
>>> in the Mac OS X future however, because...
>>>
>>> Apple's latest strategy for unixy stuff (at least for Python) seems 
>>> to be a hybrid approach that should please almost anyone:
>>> 	(1) the actual dylib lives in /usr/lib and has a mach-o id pointing 
>>> to /usr/lib
>>> 	(2) the framework has symlinks to /usr/lib for its dylib
>>> 	(3) the headers live in the framework
>>> 	(4) /usr/include has appropriate symlinks into the framework
>>>
>>> [I can say this without breaking NDA because Apple has publicly 
>>> released their sources for Python in Darwin 8.0.0b1]
>>
>> After reading Bob's message, I looked into /usr/lib and found that 
>> while
>> it contained a tclConfig.sh, there was no tkConfig.sh. A search of the
>> file system revealed a tkConfig.sh in 
>> /Library/Framewords/Tk.framework, so
>> I copied it there.
>>
>> While I suspect this was a Good Thing To Do, it still hasn't solved
>> the problem of matplotlib's setup.py not being able to find the tcl/tk
>> headers. Here's my attempt at building and the resultant message:
>>
>> 	ken% python setup.py build
>> 	GTKAgg requires pygtk
>> 	cannot find tcl/tk headers. giving up.
>>
>> The message about "GTKAgg" is expected and should be ignored. 
>> Attempting
>> a "python setup.py -v build" gave exactly the same error messages. 
>> Unfortunately
>> the error message about the header, while to the point, isn't 
>> terribly helpful :-)
>>
>> matplotlib is packaged using distutil, and presumably distutil isn't 
>> getting
>> the info it needs to figure out where the headers are. I've started 
>> reading
>> about distutil, but am a novice at it (and it's a fairly involved 
>> package, as
>> well), so if others could offer suggestions as to how to track 
>> down/fix this
>> problem, it would be a real help.
>>
> distutils is only good at finding Python's compiler/linker settings 
> and Python's headers.
> matplotlib has custom code to find Tcl/Tk in its setup.py, and that 
> custom code is not doing the right thing.

Re: [Matplotlib-users] matplotlib-0.60.2 - Memory Leaks

[John H. <jdh...@ac...>]

>>>>> "Matthew" == Matthew O'Connnor <mat...@ca...> writes:

    Matthew> Hi, I was wondering if anybody else was seeing memory
    Matthew> leaks in the latest version (0.60.2).

    Matthew> I'm using the Agg backend to produce images for web
    Matthew> pages.

    Matthew> The release notes for 0.60.2 indicate that memory leaks
    Matthew> have been fixed.  Are there any known leaks left?

Yes, when I say memory leaks were fixed, I mean I identified and fixed
some, but am not claiming that all are fixed.  There are known memory
leaks.  For example, there is a small leak in freetype2 that was fixed
in freetype CVS but may not be released yet.  There appears to be a
leak in _backend_agg's write_png method, but I haven't traced this
sufficiently to find if it is in my code or in libpng or in zlib.

The whole memory leak situation has continued to improve as I've
migrated almost all of the extension code to pycxx, and I have a
number of unit tests to test leaks in various components.  For
example, there are no detectable leaks in the transforms module.

Below is a prototypical script that I use to diagnose and report
memory leaks.  Please use something like it when reporting leaks so I
get an idea of the magnitude of the problem (ie bytes per figure).
Also please provide your platform, matplotlib version, backend and as
much information about the versions of the libraries you are using:
freetype, png and zlib.  It would also help if you posted code so I
could look for any obvious coding errors vis-a-vis matplotlib.

Running the code below, I'm getting about 400 bytes / per / figure
(matplotlib-0.60.2 / agg on linux with freetype 2.17, libpng1.2 and
libz 1.1.4).  I know leaks of this size can cause troubles on an
application server, and in animated figures, and will continue to try
and track them down.  As I noted above, some of this results from the
libraries on which matplotlib depends, and so may be system and
version dependent.  If your numbers are much worse, it may be that you
are doing something wrong, ie, not properly closing, reusing figure,
etc.

Let me know what you find,
JDH

import os, sys, time
import matplotlib
matplotlib.use('Agg')
from matplotlib.matlab import *


def report_memory(i):
    pid = os.getpid()
    a2 = os.popen('ps -p %d -o rss,sz' % pid).readlines()
    print i, '  ', a2[1],
    return int(a2[1].split()[1])

N = 100
for i in range(N):
    ind = arange(100)
    xx = rand(len(ind), 1)
    figure(1)
    plot(ind, xx[:,0])
    savefig('tmp%d.png' % i, dpi = 75)
    close(1)

    val = report_memory(i)
    if i==1: start = val

end = val
print 'Average memory consumed per loop: %1.4fk bytes\n' % ((end-start)/float(N))

Re: [Matplotlib-users] map projections

[John H. <jdh...@ac...>]

>>>>> "Jeff" == Jeff Whitaker <js...@fa...> writes:

    Jeff> John: I found that if I just call proj with all the lats and
    Jeff> lons at once (instead of once for each segment) I can speed
    Jeff> it up tremendously. Here's what I tried, using the new
    Jeff> LineCollection snippets you sent me, and the updated
    Jeff> matplotlib snapshot:

Yep - very good idea.  Your mistake with the line collection is how
you define the segments.  From matplotlib.collections.LineCollection
documentation

    segments is a sequence of ( line0, line1, line2), where linen =
    (x0, y0), (x1, y1), ... (xm, ym).  Each line can be a different
    length

That is, it is a sequence of sequences of xy tuples.  When you write
zip(xs.tolist(),ys.tolist()), you have a sequence of xy tuples.  If
you plotted this, you would have one giant, connected line, which is
not what you want.  You want a series of disconnected lines.  Thus you
need to keep track of the indices where each separate segment starts,
and split out the segments, as in the code below.

For future reference, you may also want to use PolyCollections if you
want to generate a map and you have a bunch of segments defined by
sequences of xy tuples with associated face colors.  

There was no significant difference between using bilinear
interpolation with antialiased drawing vs nearest neighbor
interpolation w/o aa, so I turned both back on.

And don't forget to try the new toolbar....

I noticed that the lat/lon lines don't precisely agree with the
colormap, eg around the Aleutian Islands the light blue is not
perfectly registered with the black lines.  Should I be concerned that
this reflects a problem in matplotlib, or is this kind of error
standard in the data you've provided?  I think this is related to the
pixel border that appears around some images, and is magnified when
interpolation is used because the top and right borders are not
defined when interpolating.  I'll continue to look into this.

Would it be OK if I used this example on the web page?  I like it!

Enjoy,
JDH

import cPickle
from matplotlib.matlab import *
from matplotlib.collections import LineCollection 
from proj import Proj
import Numeric

# standard parallels at 50 deg N, center longitued 107 deg W.
params = {}
params['proj'] = 'lcc'
params['R'] = 63712000
params['lat_1'] = 50
params['lat_2'] = 50
params['lon_0'] = -107
proj = Proj(params)
llcornerx, llcornery = proj(-145.5,1.)
params['x_0'] = -llcornerx # add cartesian offset so lower left corner = (0,0)
params['y_0'] = -llcornery
# create a Proj instance for desired map.
proj = Proj(params)

# set the default params for imshow
rc('image', origin='lower', cmap='jet')
ax = subplot(111)
nx = 349; ny = 277
dx = 32463.41; dy = 32463.41
xmax = (nx-1)*dx; ymax = (ny-1)*dy # size of domain to plot

C = cPickle.load( file('topodata.pickle','rb') )
im = ax.imshow(C, interpolation='bilinear',
               extent=(0, xmax, 0, ymax))

# ind is the index for the start of a new group
lons = []; lats = []; ind = []
i = 0  # the current ind
for line in file('wcl.txt'):
    if line.startswith('# -b'):
        ind.append(i)
        continue
    
    # lon/lat
    lon, lat = [float(val) for val in line.split('\t')]
    lons.append(lon)
    lats.append(lat)
    i += 1

xs, ys = proj(Numeric.array(lons),Numeric.array(lats))
#a sequence of xy tuples
segments = [zip(xs[i0:i1], ys[i0:i1]) for i0, i1 in zip(ind[:-1], ind[1:])]


collection = LineCollection(
    segments,     
    colors       = ( (0,0,0,1), ), # black              
    linewidths   = (1.5,),                              
    antialiaseds = (1,),)  # turn off aa for speed     
ax.add_collection(collection)
corners = (min(xs), min(ys)), (max(xs), max(ys))
ax.update_datalim( corners )
axis([0, xmax, 0, ymax])
ax.set_xticks([]) # no ticks
ax.set_yticks([])
title('Lambert Conformal Conic Projection')                  
#savefig('test')
show()      

Re: [Matplotlib-users] map projections

[Jeff W. <js...@fa...>]

On Tue, 13 Jul 2004, John Hunter wrote:

>
> OK, made some headway here.
>
....
>
> For the line part of the map, I extended the
> matplotlib.collections.LineCollection class to handle a sequence of
> lines, where each line is defined by a list of tuples (x0,y0), (x1,
> y1), ...  Thus all of your lines are handled by a single object,
> rather than having 1800+ separate line objects created in plot.
> Again, no python loops required.
>
> In the current form, the code takes about 1.15s to run on my machine
> and is about 30x faster than the original code you posted which
> includes the data loading part.  Nonetheless, the matplotlib part is
> much faster too, as you'll see when you interact with the data.
>

John:  I found that if I just call proj with all the lats and lons at 
once (instead of once for each segment) I can speed it up tremendously. 
Here's what I tried, using the new LineCollection snippets you sent me, 
and the updated matplotlib snapshot:


from matplotlib.matlab import *
from matplotlib.collections import LineCollection 
from proj import Proj
import Numeric, time

# open file with coastline data (from world coastline database).
wcl = open('wcl.txt')

# set up map projection parameters (lambert conformal conic,
# standard parallels at 50 deg N, center longitued 107 deg W.
params = {}
params['proj'] = 'lcc'
params['R'] = 63712000
params['lat_1'] = 50
params['lat_2'] = 50
params['lon_0'] = -107
proj = Proj(params)
llcornerx, llcornery = proj(-145.5,1.)
params['x_0'] = -llcornerx # add cartesian offset so lower left corner = (0,0)
params['y_0'] = -llcornery
# create a Proj instance for desired map.
proj = Proj(params)

# set the default params for imshow 
rc('image', origin='lower', cmap='jet')

ax = subplot(111) 
nx = 349; ny = 277 
dx = 32463.41; dy = 32463.41 
xmax = (nx-1)*dx; ymax = (ny-1)*dy # size of domain to plot

t1 = time.clock()
segnum = 0
lons = []
lats = []
for line in wcl:
     splitline = line.split()
     if splitline[0] != '#':
         lon = float(splitline[0])
 	lat = float(splitline[1])
 	lons.append(lon)
 	lats.append(lat)
xs, ys = proj(Numeric.array(lons),Numeric.array(lats))
minx, maxx = min(xs), max(xs) 
miny, maxy = min(ys), max(ys) 
xypts = zip(xs.tolist(),ys.tolist())

wcl.close() # close coastline file
# all args are sequences, length 1 in case of linewidths and 
# antialiased 
collection = LineCollection(segments     = xypts,
                             colors       = ( (0,0,0,1), ), # black
                             linewidths   = (1.5,),
                             antialiaseds = (0,),)  # turn off aa for speed 
ax.add_collection(collection) 
ax.update_datalim( ((minx, miny), (maxx, maxy)))

axis([0, xmax, 0, ymax]) 
ax.set_xticks([]) # no ticks 
ax.set_yticks([]) 
title('Lambert Conformal Conic Projection') 
t2 = time.clock()
print t2-t1
show()

But when I run it, I get the following error:

Traceback (most recent call last):
   File 
"/sw/lib/python2.3/site-packages/matplotlib/backends/backend_gtkagg.py", 
line 75, in callback
     self.draw()
   File 
"/sw/lib/python2.3/site-packages/matplotlib/backends/backend_gtkagg.py", 
line 42, in draw
     agg.draw()
   File 
"/sw/lib/python2.3/site-packages/matplotlib/backends/backend_agg.py", line 
291, in draw
     self.figure.draw(self.renderer)
   File "/sw/lib/python2.3/site-packages/matplotlib/figure.py", line 236, 
in draw
     for a in self.axes: a.draw(renderer)
   File "/sw/lib/python2.3/site-packages/matplotlib/axes.py", line 668, in 
draw
     c.draw(renderer)
   File "/sw/lib/python2.3/site-packages/matplotlib/collections.py", line 
286, in draw
     self._transOffset)
TypeError: CXX: type error.


-Jeff

  -- 
Jeffrey S. Whitaker         Phone : (303)497-6313
NOAA/OAR/CDC  R/CDC1        FAX   : (303)497-6449
325 Broadway                Web   : http://www.cdc.noaa.gov/~jsw
Boulder, CO, USA 80305-3328 Office: Skaggs Research Cntr 1D-124

[Matplotlib-users] matplotlib-0.60.2 - Memory Leaks

[Matthew O'C. <mat...@ca...>]

Hi,

I was wondering if anybody else was seeing memory leaks in the latest
version (0.60.2).

I'm using the Agg backend to produce images for web pages.

The release notes for 0.60.2 indicate that memory leaks have been fixed.
Are there any known leaks left?

Cheers,

Matthew O'Connor

Re: [Matplotlib-users] map projections

[Jeff W. <js...@fa...>]

On Tue, 13 Jul 2004, John Hunter wrote:

John:

>
> OK, made some headway here.
>
> When profiling for performace, I often turn off the GUI so my numbers
> aren't influenced by the user interface
>
> > /usr/local/lib/python/profile.py plotmap.py -dAgg > prof.out
>
> Here are the profile results sorted by cumulative time for your script
> - only the most expensive functions are shown
>
>   7.590  proj.py:43(__call__)
>   7.080  os.py:611(popen2)
>   7.050  popen2.py:31(__init__)
>   7.050  popen2.py:143(popen2)
>   2.610  matlab.py:1305(savefig)
>
> On my system, over half the time is spent running your popen process.
> Can't help you with this one :-).  To get better numbers and focus on
> matplotlib, I cached the xy points that the popen process is
> generating to a pickled list.

OK - looks like I have to work on the proj module. I probably should make 
it a C extension that calls the proj library directly instead of 
communicating with the proj application via pipes.

>
> You're using pcolor to generate the colormap.  pcolor uses
> collections, which is already pretty fast (agg implements collection
> drawing in extension code, no python loops needed to make a pcolor).
> However, you can get better performace still from imshow with the data
> limits set.  This will approximately double the performance of the
> image part of the map.  I had to make some changes to matplotlib (see
> below) because image origin wasn't playing nicely with image extent -
> this also fixes Andrew's bug.
>
> For the line part of the map, I extended the
> matplotlib.collections.LineCollection class to handle a sequence of
> lines, where each line is defined by a list of tuples (x0,y0), (x1,
> y1), ...  Thus all of your lines are handled by a single object,
> rather than having 1800+ separate line objects created in plot.
> Again, no python loops required.
>
> In the current form, the code takes about 1.15s to run on my machine
> and is about 30x faster than the original code you posted which
> includes the data loading part.  Nonetheless, the matplotlib part is
> much faster too, as you'll see when you interact with the data.
>
> I'm including a link to a matplotlib snapshot below which includes the
> required changes.  As a bonus, you can try out the new navigation
> toolbar (in progress, only works w/ gtk and gtkagg).  It includes a
> view limits stack, hand pan, and zoom to rectangle.  Much nicer for
> map navigation.  And with the changes, you can actually interact with
> your data with reasonable performance.  I need to add some more
> features to the toolbar, but give it a test drive and let me know if
> you have suggestions.  Set 'toolbar: toolbar2' in matplotlibrc

Excellent!  I'll do some more work on it and repost a (hopefully) faster 
version.

Thanks for the quick response.

-Jeff


-- 
Jeffrey S. Whitaker         Phone  : (303)497-6313
Meteorologist               FAX    : (303)497-6449
NOAA/OAR/CDC  R/CDC1        Email  : Jef...@no...
325 Broadway                Web    : www.cdc.noaa.gov/~jsw
Boulder, CO, USA 80303-3328 Office : Skaggs Research Cntr 1D-124

Re: [Matplotlib-users] map projections

[John H. <jdh...@ac...>]

OK, made some headway here.

When profiling for performace, I often turn off the GUI so my numbers
aren't influenced by the user interface

 > /usr/local/lib/python/profile.py plotmap.py -dAgg > prof.out
 
Here are the profile results sorted by cumulative time for your script
- only the most expensive functions are shown

   7.590  proj.py:43(__call__)
   7.080  os.py:611(popen2)
   7.050  popen2.py:31(__init__)
   7.050  popen2.py:143(popen2)
   2.610  matlab.py:1305(savefig)

On my system, over half the time is spent running your popen process.
Can't help you with this one :-).  To get better numbers and focus on
matplotlib, I cached the xy points that the popen process is
generating to a pickled list.

You're using pcolor to generate the colormap.  pcolor uses
collections, which is already pretty fast (agg implements collection
drawing in extension code, no python loops needed to make a pcolor).
However, you can get better performace still from imshow with the data
limits set.  This will approximately double the performance of the
image part of the map.  I had to make some changes to matplotlib (see
below) because image origin wasn't playing nicely with image extent -
this also fixes Andrew's bug.

For the line part of the map, I extended the
matplotlib.collections.LineCollection class to handle a sequence of
lines, where each line is defined by a list of tuples (x0,y0), (x1,
y1), ...  Thus all of your lines are handled by a single object,
rather than having 1800+ separate line objects created in plot.
Again, no python loops required.

In the current form, the code takes about 1.15s to run on my machine
and is about 30x faster than the original code you posted which
includes the data loading part.  Nonetheless, the matplotlib part is
much faster too, as you'll see when you interact with the data.

I'm including a link to a matplotlib snapshot below which includes the
required changes.  As a bonus, you can try out the new navigation
toolbar (in progress, only works w/ gtk and gtkagg).  It includes a
view limits stack, hand pan, and zoom to rectangle.  Much nicer for
map navigation.  And with the changes, you can actually interact with
your data with reasonable performance.  I need to add some more
features to the toolbar, but give it a test drive and let me know if
you have suggestions.  Set 'toolbar: toolbar2' in matplotlibrc

Thanks much for the fink package - I'll continue to point OS X users
to your site!

JDH

### matplotlib shapshot:
    http://nitace.bsd.uchicago.edu:8080/files/share/matplotlib-0.60.3a.tar.gz

### Here is a link to the xy point data:
    http://nitace.bsd.uchicago.edu:8080/files/share/xypts.pickled 

### Here is the code snippet I used to generate it

xypts = []
for line in wcl:
    splitline = line.split()
    if splitline[0] == '#':
        segnum = segnum + 1
	if segnum > 1:
            # convert lon,lat to map coordinates x,y
            xys = zip(*proj(Numeric.array(lons),Numeric.array(lats)))
            xypts.append( xys)
pickle.dump(xypts, file('xypts.pickled', 'w') )


### Here is the modified plotmap script which uses imshow and line
### collections

import time
from matplotlib.matlab import *
from matplotlib.collections import LineCollection
from proj import Proj
import Numeric, cPickle

# set the default params for imshow
rc('image', origin='lower', cmap='jet')

ax = subplot(111)
nx = 349; ny = 277
dx = 32463.41; dy = 32463.41
xmax = (nx-1)*dx; ymax = (ny-1)*dy # size of domain to plot

datin = open('topodata.pickle','rb')
C = cPickle.load(datin)
# use imshow rather than pcolor for speed
im = ax.imshow(C, interpolation='nearest',
               extent=(0, xmax, 0, ymax))

xypts = cPickle.load(file('xypts.pickled', 'r') )
# all args are sequences, length 1 in case of linewidths and
# antialiased
collection = LineCollection(segments     = xypts,
                            colors       = ( (0,0,0,1), ), # black
                            linewidths   = (1.5,),
                            antialiaseds = (0,),  # turn off aa for speed
                            )

ax.add_collection(collection)

# you have to manually update the datalim; this is a bit ugly so I'll
# work on the interface
xs = [ x for thisline in xypts for x,y in thisline ]
ys = [ y for thisline in xypts for x,y in thisline ]
minx, maxx = min(xs), max(xs)
miny, maxy = min(ys), max(ys)
ax.update_datalim( ((minx, miny), (maxx, maxy)))


axis([0, xmax, 0, ymax])
ax.set_xticks([]) # no ticks
ax.set_yticks([])
title('40 km Topography - Lambert Conformal Conic Projection')
#savefig('test4')
show()

Re: [Matplotlib-users] new embedding_in_wx3 example

[John H. <jdh...@ac...>]

>>>>> "Andrew" == Andrew Straw <str...@as...> writes:

  Andrew> OK, I discovered a rather easy demonstration of what looks
  Andrew> to be a OK, (possibly the same) bug (adding 'ylim' to the
  Andrew> image_origin demo):

    from matplotlib.matlab import *
    x = arange(100.0); x.shape = 10,10

    subplot(211)
    title('blue should be up')
    imshow(x, origin='upper', interpolation='nearest')
    set(gca(),'ylim',(0,23))

    subplot(212)
    title('blue should be down')
    imshow(x, origin='lower', interpolation='nearest')
    set(gca(),'ylim',(0,23))

    #savefig('image_origin2')
    show()

I agree it looks like a bug.  Just to make sure we are on the same
page: what do you thing the correct behavior should be here?  Do you
think subplot (211) is correct?  

The question is: how should image origin interact with the axes
ylimits?  In this example, should both images fill the space from 0-10
on the y axes, regardless of the ylimit setting?  In the origin upper
case, should the blue strip be at 10 and the red strip at zero?  In
the origin lower case, should the blue be at zero and the red at 10?

Inquiring minds want to know.

Thanks,
JDH

Re: [Matplotlib-users] map projections

[Jeff W. <js...@fa...>]

On Tue, 13 Jul 2004, John Hunter wrote:

>
> I am having trouble running this - proj.py is making the system call
> 'stdin,stdout=os.popen2(cmd,mode='b')' where cmd is
>
> proj +R=63712000 +lon_0=-107 +proj=lcc +lat_1=50 +lat_2=50 -b
>
> but I don't have proj on my system.  Am I missing something?  Is this
> a tool I can easily install?

John:  Yes, it's easy to install.  You can grab it from 
http://proj.maptools.org.  A simple ./configure; make; make install worked 
for me.

>
> Also, while I have your ear, what version of matplotlib are you
> working with and what backend do you typically use?

I have 0.60.2 and usually use the default gtkagg backend.  I work mostly 
on OS X (I'm actually the fink package maintainer).

>
> Nice map, by the way!
>

Thanks - very nice software!

-Jeff

-- 
Jeffrey S. Whitaker         Phone : (303)497-6313
NOAA/OAR/CDC  R/CDC1        FAX   : (303)497-6449
325 Broadway                Web   : http://www.cdc.noaa.gov/~jsw
Boulder, CO, USA 80305-3328 Office: Skaggs Research Cntr 1D-124

Re: [Matplotlib-users] map projections

[John H. <jdh...@ac...>]

>>>>> "Jeff" == Jeff Whitaker <js...@fa...> writes:

    Jeff> Hi:

    Jeff> I've hacked together some scripts that let me plot data on
    Jeff> top of world maps with different map projections in
    Jeff> matplotlib.  It's quite slow, but it works!  There's an
    Jeff> example at
    Jeff> http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/plotmap.py
    Jeff> (the plot should look like
    Jeff> http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/plotmap.png).
    Jeff> To run the example you'll need the data files
    Jeff> http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/wcl.txt
    Jeff> (world coastlines) and
    Jeff> http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/topodata.pickle
    Jeff> (data to plot), as well as a module I wrote to compute
    Jeff> cartographic transformations using proj4
    Jeff> http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/python/proj.py.


    Jeff> If anyone knows of a faster way to do this, I'd love to hear
    Jeff> about it.

I am having trouble running this - proj.py is making the system call
'stdin,stdout=os.popen2(cmd,mode='b')' where cmd is

 proj +R=63712000 +lon_0=-107 +proj=lcc +lat_1=50 +lat_2=50 -b

but I don't have proj on my system.  Am I missing something?  Is this
a tool I can easily install?

Also, while I have your ear, what version of matplotlib are you
working with and what backend do you typically use?

Nice map, by the way!

JDH

[Matplotlib-users] map projections

[Jeff W. <js...@fa...>]

Hi:

I've hacked together some scripts that let me plot data on top of world 
maps with different map projections in matplotlib.  It's quite slow, but 
it works!  There's an example at 
http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/plotmap.py (the plot 
should look like 
http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/plotmap.png).  To run 
the example you'll need the data files 
http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/wcl.txt (world 
coastlines) and 
http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/topodata.pickle (data to 
plot), as well as a module I wrote to compute cartographic transformations 
using proj4 
http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/python/proj.py.


If anyone knows of a faster way to do this, I'd love to hear about it.

-Jeff




-- 
Jeffrey S. Whitaker         Phone : (303)497-6313
NOAA/OAR/CDC  R/CDC1        FAX   : (303)497-6449
325 Broadway                Web   : http://www.cdc.noaa.gov/~jsw
Boulder, CO, USA 80305-3328 Office: Skaggs Research Cntr 1D-124

Re: [Matplotlib-users] new embedding_in_wx3 example

[Andrew S. <str...@as...>]

John Hunter wrote:

>
>    Andrew> John+Perry+whoever: the only matplotlib bug I found this
>    Andrew> time is worked around by the line
>
>    >> rcParams['image.origin'] = 'upper' # 'lower': nav toolbar
>    >> problem -ADS
>
>    Andrew> We should fix that at some point.
>
>What was the bug?  I didn't notice any problems with lower.
>  
>
OK, I discovered a rather easy demonstration of what looks to be a 
(possibly the same) bug (adding 'ylim' to the image_origin demo):

from matplotlib.matlab import *

x = arange(100.0); x.shape = 10,10


subplot(211)
title('blue should be up')
imshow(x, origin='upper', interpolation='nearest')
set(gca(),'ylim',(0,23))

subplot(212)
title('blue should be down')
imshow(x, origin='lower', interpolation='nearest')
set(gca(),'ylim',(0,23))

#savefig('image_origin2')
show()

Re: [Matplotlib-users] new embedding_in_wx3 example

[Andrew S. <as...@ca...>]

John Hunter wrote:

>    Andrew> John+Perry+whoever: the only matplotlib bug I found this
>    Andrew> time is worked around by the line
>
>    >> rcParams['image.origin'] = 'upper' # 'lower': nav toolbar
>    >> problem -ADS
>
>    Andrew> We should fix that at some point.
>
>What was the bug?  I didn't notice any problems with lower.
>  
>
Try vertically zooming and panning with the matplotlib toolbar -- you'll 
see that the overlay points move opposite from the data.  Maybe it's my 
head that's on wrong, though. :)

>Updates are in CVS - thanks for the submission.
>  
>

Thanks for the suggestions.  You'll make a respectable (matplotlib) 
programmer out of me yet!

Cheers!
Andrew