opengrads-devel — OpenGrADS development issues, with focus on interfaces and extensions

Re: [Opengrads-devel] mcidas .f90 code from ncdc

[Arlindo da S. <da...@al...>]

Mike,

  Jeff Whitaker just patched basemaps so that we can handle any GOES images.
I'll get back back to pygrads and images sometime next week after I get the
current set of binaries relesead on sf.net.

  The main question in my mind is: 1) should we handle images entirely on
the python end? or 2) Teach GrADS to read images (just another data file),
and display them with a pixmap extension. The problem with 2) is that GrADS
does not understand too many map projections, and we could make much faster
progress in 1) than in 2). The problem with 1) is that matplotlib is
*slow*. However, these 2 approaches are not exclusive.

Another trick that may help integrating GrADS graphics and python is to
write a small function that reads the gx metafile from memory and draws it
on the matplotlib canvas. This would give us the speed of GrADS graphics
(I'm expecting that rendering the gx meta is fairly quick as matplotlib uses
cairo underneath). And if we translate the metafile (x,y) coordinates back
into lat/lon, then we can use matplotlib projections to warp the plots on
the screen, on any basemap supported map transformation. One by-product of
this is to write out the lat/lon gx metafile to disk as a shapefile. (In
fact, matplotlib/basemap can draw shapefiles --- this is the way the draw
the coastlines).

Here is a proposed roadmap:

1) Write GaIL.py as an extension of PIL which not only reads the images (as
PIL does), but collects the geo-referencing as well. Initial formats to be
supported would be AREA, the NetCDF files you sent me the other day (not the
atrocity written out by AreaToNC.jar), geotiff, and PNG/JPEG/GIF provided
that a KML kind of metadata is provided (much like a "ctl" for images).

2) Integrate GaIL into GaLab so that we can handle these and contour/image
plots transparently. (A call to GaIL.open() would automatically set basemap
to match the image coordinates).

3) Write ga.show() which would render the gx metafile in memory on the
current matplotlib figure, using the current basemap settings. These would
give us the speed of GrADS basic graphics, and the ability to handle sat
images.

4) Bring back "set gxout pixmap" as an extension. By providing an
appropriate DDF GOES images in NetCDF could be read in as a regular grads
variable, and displayed on the screen, and contour plots supperimposed. At
this point the 100 color limit will get on the way.

5) The "set clevs"/"set ccols" way of associating colors with contour
intervals is very awkward when you are dealing with a continuous color
scale. The notion of a colormap and scaling will need to be implement. One
may be able to handle this as an extension. Perhaps we should wait until v2
is out doing 4) and 5).

  Any comments greatly appreciated. Any body willing to help with any of
that?

  Regards,

        Arlindo



















On Jan 25, 2008 11:39 AM, Mike Fiorino <mfi...@gm...> wrote:

> hi arlindo,
>
> good news, got code from ncdc that seems to work area files...
>
> see:
>
> ftp://ftp.nhc.noaa.gov/users/fiorino/ncdc_mcidas.tgz
>
> the tarball is what i got from ken knapp at ncdc and includes a sample
> data file.
>
> it was orginally built under lahey fortran, but i got g95 to work...
> i've asked ken for the code that creates those nice .nc files too...
>
> i had to make a couple of small hacks,  an array was not declared as
> allocatable and the usual problem with non-standard i/o -- had to use
> the compiler specific access='stream' option in the open the file for
> access like in C.
>
> seems like a good candidate for your extensions...
>
> thanks best /r mike
>
>
>


-- 
Arlindo da Silva
da...@al...

Re: [Opengrads-devel] my Solaris build

[Arlindo da S. <da...@al...>]

On Jan 26, 2008 11:44 AM, Don Hooper <ho...@co...> wrote:

> Arlindo,
>
> The make check still craps out immediately in pytests, saying:
> Traceback (most recent call last):
>  File "./grads_tests.py", line 5, in ?
>    from optparse      import OptionParser   # Command-line args
> ImportError: No module named optparse
>

Yes, a lot of these modules come standard in python 2.3 and later. I should
add a test for the python version. Up to now wasn't quite sure where the
divide was, but it appears that 2.3 is a hard requirement.  (The only reason
why I wrote the test suite in python is because I could do it quickly with
PyUnit.)


>
> "gmake check" does nothing in the extensions directory.


Unfortunately I am yet to write a test suite for the extensions, although it
is badly needed.


> The good
> news, however, is the 'hello' command works, as well as the queries
> on userdefined stuff.  Also, the gex and bin tarballs are different
> now.


This is great! I am using dlopen() to load the .so into GrADS as needed.
Since dlopen() was invented at Sun it is not surprising that it works on
solaris, but it is always good to know that it actually works.


> You can find them at:
>        ftp://ftp.cdc.noaa.gov/public/dhooper
>

I am having some trouble connecting to it. I'll try again later.


>
> And yes, --without-nc4 yielded satisfactory results.  There's no
> gxyat, of course.  When I tried to build cairo, it insisted on
> using a package configuration tool, which didn't find the X11
> back-end.  I guess the /usr/X11 convention wasn't something it
> wanted to try....  It also lost big time on the font stuff.
>

After this round of builds I'll add cairo to the supplibs. The reason I
haven't is because of the additional dependencies it brings. Of these only
the font stuff is required by cairo --- kind of unfortunate because GrADS
uses none of it. However, cairo could be a good alternative to replace
gxX.cin v2: it has a better rendering than X, handles fonts and
transparencies
and has been ported natively to just about anywhere. And because firefox and
gnome uses it we know that it will continue to be maintained. So, the first
step to explore it as a backend for grads is to add it to the supplibs.

  Thank you,

      Arlindo

-- 
Arlindo da Silva
da...@al...

Re: [Opengrads-devel] Solaris build

[Arlindo da S. <da...@al...>]

On Jan 26, 2008 12:50 AM, Don Hooper <ho...@co...> wrote:

> Arlindo,
>
> I'm very confused.  The configure output doesn't mention netCDF4,
> yet it insists on trying to build it anyway.


Strange. When I type 'configure --help' I get

---cut---
Optional Packages:
  --with-PACKAGE[=ARG]    use PACKAGE [ARG=yes]
  --without-PACKAGE       do not use PACKAGE (same as --with-PACKAGE=no)
  --with-readline         command line editing via readline
  --with-printim          printim command - image output using GD
  --with-lats             LATS interface for generating GRIB and netCDF
files
  --with-gui              Athena X11 widget-based GUI
  --with-nc               gradsnc executable, with netCDF-3 support
  --with-nc4              gradsnc executable, with netCDF-4 support
  --with-dods             gradsdods executable, with netCDF and OPeNDAP
                          support
---cut---


> Beyond that, I'm
> not sure what you mean by "all other extensions should build"; I
> saw nothing but the usual GrADS binaries build.  I didn't see gxyat
> at all.  The bin-dist tarball and the gex-dist tarball are identical.
> I must really be missing something.  The make check was odd, too.  It
> only ran in pytests and failed utterly there.  It claimed there was
> nothing to do for target "check" in src.  I'm very confused....
>

OK, you may have found a bug in our 'make dist' which creates the src tar
ball. The extensions/ directory, I forgot, is not included with that (we
wanted to retain a vanilla COLA look and feel and handle extensions
separately). We have a module "Grads" which adds the extensions/ directory
to it.

> Also, I don't have a sf username.
OK, if you ever get one let me know I'll add you as a developer (the
sf.netregistration URL is
https://sourceforge.net/account/registration/). In the mean time, your best
bet is to check anonimously the module Grads from the repo (notice capital
"G"):


cvs -d:pserver:ano...@op...:/cvsroot/opengrads
login

cvs -z3 -d:pserver:ano...@op...:/cvsroot/opengrads
co -P *Grads*

*See whether building with this would work. I am more concerned about you
being able to have "make check" working as it has been extremely useful
detecting problems in the binaries. BTW, it only requires plain python, but
it may not work with a very old version. Enter "python" and check the
version; v2.3 and newer should be fine.*

*   Thanks,*

*      Arlindo*

**

**

**

**




>
> -Hoop
>
> > From arl...@gm...  Fri Jan 25 20:58:48 2008
>  >
> > On Jan 25, 2008 7:35 PM, Don Hooper <ho...@co...> wrote:
> >
> > > Arlindo,
> > >
> > > I'm not sure just what I've accomplished.  I built your supplibs
> tarball
> > > (the trick was not only GNU's make, but getting it to use bash, as in:
> > >        gmake SHELL=/bin/bash install
> > > ).  Then, I could build everything in your GrADS tarball
> >
> > This is great! NetCDF4 is just an experimental version to see whether it
> is
> > ready for prime time. So far gradsnc4 has been passing all tests in all
> > platforms.
> >
> > I just posted the frozen grads 1.9.0-rc1 sources on
> >
> >      http://opengrads.org/pre-rel
> >
> > Untar the sources alongside the supplibs then
> >
> > ./configure   (all should be enabled except wi and gradsnc4)
> > make
> > make check  (to run the test suite)
> > make bin-dist  (create binary tarball)
> > make gex-dist  (for the extension tarball)
> >
> > Unless you have cairo around, gxyat may not build but all other
> extensions
> > should build (make sure g77 is present). Check the documentation at the
> > wiki. I'm slowly completing the documentation but would appreciate any
> > feedback on what is there. You should have a login for the wiki. If you
> see
> > anything needs editing feel free to jump in and make corrections (you
> need
> > to be logged on to be able to edit it).
> >
> > > except the nc4
> > > version, as your supplibs tarball didn't seem to have code for netcdf4
> > > libraries, etc.
> >
> > Strange, I must have messed up making the src tarball. I'll refresh the
> > sources. However, if you send me your sf.net username I can give you
> commit
> > privileges to the repository so you check a fresh copy yourself.
> >
> > BTW, in the supplibs page at the wiki I am maintaining a section on
> platform
> > specific notes. I'd be interested in having some notes on your tweaks
> for
> > the solaris build.
> >
> > > I guess I need to download more source to do any of
> > > the extensions, like PyGrADS, right?
> >
> > Not really. PyGrADS is 100% python, the core package even works under
> jython
> > with the java vm.  For taking full advantage of it you must have a
> capable
> > python installation. Basically, make sure you have Matplotlib with the
> > basemap toolkit installed. I have the latest pygrads tarball in the same
> > pre-rel site above. Untar and take a look at the README. Installation is
> > really easy, and the minimum requirement is python. If you have
> matplotlib
> > then you can use it for graphics; if you have numpy you can do EOFs and
> > least square estimation. In the very least you should be able to use it
> to
> > script grads. See the examples directory.
> >
> > BTW, I am making a somewhat artificial distinction between "extensions"
> and
> > "interfaces". Extensions are user defined commands and user defined
> > functions implemented as shared objects; so it seats below grads.
> Interfaces
> > are basically written at the host interpreted language (perl, python,
> even
> > IDL), so it seats above grads.
> >
> > So, whenever you have your solaris binaries I'd be glad to post them on
> > sf.net. Just make sure they pass the test suite (make check). And feel
> free
> > to introduce additional test cases --- I just did the bare minimum.
> >
> > Thank you very much for the help with the solaris build.
> >
> >   Best Regards,
> >
> >       Arlindo
> > --
> > Arlindo da Silva
> > da...@al...
>



-- 
Arlindo da Silva
da...@al...

Re: [Opengrads-devel] Solaris build

[Arlindo da S. <da...@al...>]

On Jan 25, 2008 7:35 PM, Don Hooper <ho...@co...> wrote:

> Arlindo,
>
> I'm not sure just what I've accomplished.  I built your supplibs tarball
> (the trick was not only GNU's make, but getting it to use bash, as in:
>        gmake SHELL=/bin/bash install
> ).  Then, I could build everything in your GrADS tarball


This is great! NetCDF4 is just an experimental version to see whether it is
ready for prime time. So far gradsnc4 has been passing all tests in all
platforms.

I just posted the frozen grads 1.9.0-rc1 sources on

     http://opengrads.org/pre-rel

Untar the sources alongside the supplibs then

./configure   (all should be enabled except wi and gradsnc4)
make
make check  (to run the test suite)
make bin-dist  (create binary tarball)
make gex-dist  (for the extension tarball)

Unless you have cairo around, gxyat may not build but all other extensions
should build (make sure g77 is present). Check the documentation at the
wiki. I'm slowly completing the documentation but would appreciate any
feedback on what is there. You should have a login for the wiki. If you see
anything needs editing feel free to jump in and make corrections (you need
to be logged on to be able to edit it).



> except the nc4
> version, as your supplibs tarball didn't seem to have code for netcdf4
> libraries, etc.


Strange, I must have messed up making the src tarball. I'll refresh the
sources. However, if you send me your sf.net username I can give you commit
privileges to the repository so you check a fresh copy yourself.

BTW, in the supplibs page at the wiki I am maintaining a section on platform
specific notes. I'd be interested in having some notes on your tweaks for
the solaris build.


> I guess I need to download more source to do any of
> the extensions, like PyGrADS, right?


Not really. PyGrADS is 100% python, the core package even works under jython
with the java vm.  For taking full advantage of it you must have a capable
python installation. Basically, make sure you have Matplotlib with the
basemap toolkit installed. I have the latest pygrads tarball in the same
pre-rel site above. Untar and take a look at the README. Installation is
really easy, and the minimum requirement is python. If you have matplotlib
then you can use it for graphics; if you have numpy you can do EOFs and
least square estimation. In the very least you should be able to use it to
script grads. See the examples directory.

BTW, I am making a somewhat artificial distinction between "extensions" and
"interfaces". Extensions are user defined commands and user defined
functions implemented as shared objects; so it seats below grads. Interfaces
are basically written at the host interpreted language (perl, python, even
IDL), so it seats above grads.

So, whenever you have your solaris binaries I'd be glad to post them on
sf.net. Just make sure they pass the test suite (make check). And feel free
to introduce additional test cases --- I just did the bare minimum.

Thank you very much for the help with the solaris build.

  Best Regards,

      Arlindo

-- 
Arlindo da Silva
da...@al...

Re: [Opengrads-devel] Windows build

[Arlindo da S. <da...@al...>]

 All,

  I once more refreshed to fix a bug reported by one of my testers. The
extensions now seem to run out of the box. Except for fixing some links in
the Getting_Started.html file,
I believe this is ready for community testing. So I'll post it and announce
at gradsusr over the weekend. BTW, it was passes 100% of the test suite.

  If any of you had a chance to play with it, let me know.

     Arlindo

-- 
Arlindo da Silva
da...@al...

Re: [Opengrads-devel] Final touches for v1.9.0-rc1

[Arlindo da S. <da...@al...>]

On Jan 23, 2008 1:14 PM, Wesley Ebisuzaki <Wes...@no...> wrote:

> Arlindo,
>
>     Diane Stokes is our AIX grads guru.
>

Diane,

   You can find GrADS bilding instructions here:

http://opengrads.org/wiki/index.php?title=Building_GrADS_from_Sources

    I'll post the final source tarball in a couple of days, or you can check
them out from CVS when I have the final tag. In the meantime you can already
start building the supplibs. Sources for the supplibs are available here:

     http://sourceforge.net/project/showfiles.php?group_id=161773

while building instructions are here:

http://opengrads.org/wiki/index.php?title=Supplemental_Libraries_%28Supplibs%29

 The idea is to make the supplibs and GrADS binaries available for the
community to test. The GrADS "configure" script will not work with older
versions of the supplibs.

  Let me know if you are able to help with this.

  Thank you,

        Arlindo

PS:  Let me know if you would like to be added to the opengrads project at
sourceforge. In this case I'd need your sf.net userid.






-- 
Arlindo da Silva
da...@al...

[Opengrads-devel] Final touches for v1.9.0-rc1

[Arlindo da S. <da...@al...>]

All,

   I have been delaying posting the v1.9.0-rc1 binaries until I had a fully
capable win32 build which included all the extensions and could be run
pygrads under a native python win32 build (non-cygwin). I'll put the final
touches on it tonight. I hoping to get to a point where anybody willing to
install cygwin could build Win32 GrADS.

FINAL CALL:  if you still have *bug fixes* that you like to report the
deadline is tomorrow 1/24, 5PM EST. I'd like to make sure we have a
consistent set of builds using exactly the same CVS tagged source code.

Once I issue the "grads-1_9_0-rc1" tag I'll refresh all the binaries that I
am able to produce (see the pre-rel directory for a list). There are 2
important omissions at this point:

- IBM AIX (very important because of NCEP; I'm cc'ing Wes on this)
- Sun solaris (Hoop used to be our Sun guy)

and a not-so-important omission, but in a "would be nice category"

- OpenBSD (I had an earlier NetBSD build).

(Darwin is a relative of OpenBSD, so our code base is BSD friendly already).


Are any of you able to help with any of these? Please let me know ASAP. You
will need to build the v2 supplibs and from there building grads proper
should be easy. The v2 supplibs are fully automated; if you are lucky you
simply type "make install" at the top and go brew some coffee/green tea.
I'll be available to help you with questions.

Except for Win32 (which is a bit trick and the user base not exactly kernel
tinkers), the "opengrads extension collection" will be released as a
separate package. So, I'll get back to the extensions later once the main
binaries are out.

Another way you can help is to try the current binaries under
http://opengrads.org/pre-rel.

  Thank you,

        Arlindo

-- 
Arlindo da Silva
da...@al...

Re: [Opengrads-devel] Windows build

[Arlindo da S. <da...@al...>]

All,

  I just refreshed

       http://opengrads.org/pre-rel/grads-1.9.0-rc.win32.zip<http://opengrads.org/pre-help>

including most of extensions: hello, libipc, re, libbjt, env, fish. Gxyat is
not there yet but will be by tomorrow morning. It should all work with no
setup: just "unzip and go". It should have all you need to run pygrads/gerl
under Win32, but I hold on that until I had a chance to test it.

  Please let me know if you had a chance to give this ago. It should take
just a few minutes of your time.  You can also unzip to a memory stick and
test it from there, no need to litter your hard disk.

            Arlindo

-- 
Arlindo da Silva
da...@al...

Re: [Opengrads-devel] geolocated sat data...

[Arlindo da S. <da...@al...>]

On Jan 23, 2008 8:48 AM, Mike Fiorino <mfi...@gm...> wrote:

> hi arlindo, see
>
> ftp://ftp.nhc.noaa.gov/users/fiorino/HURSAT-BASIN.NA.2006.06.27.21.beta.nc
>
> this comes from a big tropical cyclone satellite data set.  sdfopen
> works fine and i complained to the developer that the file did not have
> a time dimension (for templating ala xdfopen).  is this form useful for
> satellite data?
>

Actually, these are very nice files.  It uses 16 bits to hold the brightness
temperature, although is most instances 8-bits would be plenty.  Except for
efficiency issues, one should be able to read it into GrADS, with the proper
PDEF. Another nicety is that it uses Mercator rather than the
"Geostationary" projection and the bounding box is a stored as an attribute
in the file.

At this point "gxout grfill" is the best option to render this in GrADS. In
the original opengrads sources, I had introduced the option for "gxout
pixmap" which draws a single pixel at the center of the box rather than
doing a rectangle fill. We should be able to easily implement this as a udc.
Better yet, we need a "gxout image" much like in matplotlib, but this would
require changing the metafile format, something better left for v2; besides
Brian has plans for handling images in v2. The "pixmap" extension is a
reasonable compromise for v1.9.

However,  for ingesting it in Python I'd use the 100% Python NetCDF/OPeNDAP
reader that comes with basemap. I am thinking about subclassing
PIL.Imageand creating the GaIL class for handling satellite images in
pygrads.
Supported formats would be AREA,  this brand of netcdf files, Geotiff and
Tiff/JPEG/GIF (provided one prepares a ctl like file with the
geo-referencing when this is not there.)

In any event, it would be very helpful if we could get the software that
wrote these nc files. I haven't had much luck getting software out of NCDC.
I have made multiple requests for their Fortran AREA header without much
luck. Being at the NHC (and being Mike Fiorino) you may have better luck
than I.

At this point I am consumed with finalizing the Win32 build. I almost there.
Then I'll return to PyGrADS and finish up the satellite image handling.

          Arlindo

-- 
Arlindo da Silva
da...@al...

[Opengrads-devel] Windows build

[Arlindo da S. <da...@al...>]

Folks,

  For those of you with windows boxes, could help me out testing my latest
win32 build? Please download the grads-1.9.0-rc.win32.zip (ZIPO not tar.gz)
from

    http://opengrads.org/pre-help

Unzip this, go to the win32/ subdir and click on any of the grads binaries,
say gradshdf. It should simply "work", out of the box.

Wes: this should also work from a CD, memory stick.

  Let me know.

        Thanks,

            Arlindo

-- 
Arlindo da Silva
da...@al...

Re: [Opengrads-devel] PyGrads tarball

[Arlindo da S. <da...@al...>]

On Jan 4, 2008 10:36 AM, Arlindo da Silva <da...@al...> wrote:

> All,
>
>   I just placed a pygrads tarball on http://opengrads.org/pre-rel if you
> would like to give it a try. (So far tested on Mac OS X, Mandriva 2008 and
> Fedora 7).
>

I just replaced the tarball with v1.0.7 that includes bug fixes I did over
the weekend. I also included ganum_examples.py which exercises the NumPy
dependent portion of the package, in particular the EOF calculator and the
least square estimator.

Another important update is that I redesigned the class that represents a
grads field in python. GaField is now a subclass of the
numpy.ma.MaskedArrayclass and most of the algebraic operators have
been overloaded (some issues
with slicing still remains but it is not critical). This greatly simplifies
the syntax:

ts = ga.exp('ts')   # export 'ts' to python
ts = ts - 273 # change units in python
ga.imp('tc',ts)  # put it back into grads as defined variable "tc"

You can display it in GrADS:

ga("display tc")

or with Matplotlib:

ga.imshow(ts)

The "grid" attribute containing coordinate variables and the like are now
fairly hidden. Also, GaField are masked arrays, so it works pretty much as
in GrADS v2.

One last feature is that the grads object is now "callable", so these 2 are
equivalent:

ga.cmd("display ts")
ga("display ts")

I have not update the wiki, but the docstrings should be current.

            Arlindo





-- 
Arlindo da Silva
da...@al...

Re: [Opengrads-devel] Fwd: PyGrads tarball

[Arlindo da S. <da...@al...>]

On Jan 9, 2008 8:44 AM, Mike Fiorino <mfi...@gm...> wrote:

> dear arlindo,
>
> i'm back in the office after the holidays in CA and finally grabbed the
> latest pygrads...  i also install PIL..
>

How about the basemaps. Because  the map datasets it is a relatively large
tarball they do not include it in the standard Matplotlib, you need an
additional install. See the readme file.


>
> when i try running the examples:
>
> kishou.nhc.noaa.gov[W2]:/home/work/wxmap2/trunk/src/opengrads/pygrads-
> 1.0.6/examples
> 208 > ./galab_examples.py
> Traceback (most recent call last):
>  File "./galab_examples.py", line 149, in ?
>    ga = GaLab(Bin='gradsnc',Window=False,Echo=False)
> NameError: name 'GaLab' is not defined
>

This usually means that you do not have some of the dependencies.  I
designed the grads package in a way that when you import it, only imports
modules that *can* are imported; in retrospect, it is  perhaps  too quiet.
Here is how to diagnose the problem: start pygrads and at the ga-> prompt
type "import galab", the error messages should tell you what you are
missing.


>
> also in checked the tests/grads_tests.sh and it looks hardwired for a
> particular grads/src?
>

Yeah, "grads_tests.sh" is something Pat introduced specifically for the
"make check" in autotools. Run "grads_tests.py" instead.  Outside the GrADS
build environment you will need to specify the location of your grads
binaries. Type "grads_tests.py -h" for more info. However, these tests are
meant to test the grads binaries and only uses module gacore.py.


>
> some news from CA:
>
> bob drach says the lab is finally acting on converted lats to
> opensource; not sure when but at least it's in the works.


This is great news! Pat will finally be able to have a complete Fedora
package.


>
> bob was very interested in pygrads, especially passing data to/from
> grads/python.  bob also was surprised that you think perl is faster with
> numbers the python and numpy, his impression was that python was on par
> with other languages.


I read this in a couple of places, including a article implementing fractal
algorithms. Although to be taken with a grain of salt, I heard the same in
the PDL list as well. It would be nice to come up with a benchmark relevant
to our problem. I like Matplotlib, but performance is lousy compared to
GrADS.


>  i'm glad to see you getting into python, i have a
> lot of legacy grads scripts that i would like to convert over...
>

It would be nice to inline them. Something like:

def test(arg):
     return ga.inline(arg,
"""
function test(arg)
i = 3*arg
return i
""")

So,

>>> print test(7)
21

We just need a version of run that reads from stdin. (I am avoiding exchange
files for performance reasons). It may be less work than convert a bunch of
scripts...


>
> thanks and best regards, mike
>


Let me know if basemaps fixes your problem.

  Cheers!

       Arlindo

-- 
Arlindo da Silva
da...@al...

[Opengrads-devel] PyGrads tarball

[Arlindo da S. <da...@al...>]

All,

  I just placed a pygrads tarball on http://opengrads.org/pre-rel if you
would like to give it a try. (So far tested on Mac OS X, Mandriva 2008 and
Fedora 7).

% tar xvfz pygrads-1.0.6.tar.gz

and take a look at the README for installation info, requirements etc. You
you want to run the examples:

% cd pygrads-1.0.6/examples
% ./galab_examples.py

this will generate a bunch of .png files. If you have imagemagick installed,
you can see a "slide show" with the command:

% animate -delay 300 *.png

Notice that the "galab examples" require Matplotlib with the basemap
toolkit, see the README.

Pat: would you consider packaging pygrads for Fedora? It already has all the
requirements. I believe that if you enter

% python setup.py bdist_rpm

it will create the RPM for you, but you may need to tweak setup.py to
specify the dependencies.

    Arlindo

-- 
Arlindo da Silva
da...@al...

[Opengrads-devel] PyGrADS screenshots

[Arlindo da S. <da...@al...>]

All,
  Happy New Year, everyone!

  I just posted some PyGrADS screenshots on sf.net:

           http://sourceforge.net/project/screenshots.php?group_id=161773

Each of these figures takes just a few lines of code, e.g.,

        sat = Image.open("goes_fulldisk.jpg")
        ga.basemap('geos', opts=-75.)
        ga.imshow('duexttau',cmap=gacm.hot_l)
        title("Mineral Dust Optical Thickness")

I have added a derived classes named GaLab which interfaces GrADS to
PyLab/Matplotlib.  To install PyGrADS:

gacvs co -P contrib
cd contrib/interfaces/python
python setup.py install
(If you have installed a previous version, uninstall it first as "grads" is
now a python *package*, no longer a single module.)

if you have su privileges, or else specify --prefix at the *end* to install
on a local disk. You may want to run the examples:

cd examples/
./galab_examples.py
(go get a cup of coffee...)
animate -delay 300 *.png

In addition to python proper, you must have

NumPy
Matplotlib
Basemap (toolkit of Matplotlib)

Notice that SciPy is no longer required.

The PyGrADS doc on the wiki still needs update, though.

   Arlindo


-- 
Arlindo da Silva
da...@al...

Re: [Opengrads-devel] Win32 build

[Patrice D. <per...@fr...>]

On Thu, Dec 20, 2007 at 01:48:35PM -0500, Arlindo da Silva wrote:
> All,
> 
> a) On Win32, we make this library dynamic (libgradsc.dll) and link
> 
>       gcc -o gradsc main.o libgradsc.dll
> 
>    This libgradsc.dll is then used to build the dynamic extensions as well.

I am not very knowledgable in windows, but maybe libtool 
may help solving that issue more simply since there is this in 
libtool info manual:

    If symbols from your _executable_ are needed to satisfy unresolved
 references in a library you want to dlopen you will have to use the flag
 `-export-dynamic'.  You should use `-export-dynamic' while linking the
 executable that calls dlopen:

     burger$ libtool --mode=link gcc -export-dynamic -o hell-dlopener main.o
     burger$

It looks like it could solve the issue you are having.

Also you could use libltdl to abstract from the platform.

And of course I'll try to do my best to help ;)

--
Pat

[Opengrads-devel] Win32 build

[Arlindo da S. <da...@al...>]

All,

  In the past I used to handle the Win32 builds kind of on the side of
the general Unix/Linux builds. I'm trying now to bring it into the
general build framework so that anyone can perform these builds. As it
is right now, the supplibs build out of th box on cygwin and so does
the GrADS binaries. I'm now looking into having the "setup.exe"
packaging as being integrated into the makefile. More on that later.

  One pressing issue is to have the shared extensions working on
Win32. The good new is, I think I got it working. However,  giving to
the particular way Win32 does linking (Linux people usually hates
Windows for that), any shared library (DLL's in Windows parlance) must
have all its external dependencies resolved at DLL creation time,
rather than at run time (as in Linux/Unix). To make a long story
short, this can be accommodated with the following mods to the build
mechanism:

1) In grads.c, rename main() to grads_main();

2) Add a simple file main.c which simply calls grads_main:

int main(int argc, char **argv) {
    return grads_main(argc,argv);
}

The reason for this is because there is a circular dependency in
grads: the main program in grads.c contains functions that are called
by the other files. Another possibility would be to move these
functions from grads.c to a separate file. The main.c options seems
less intrusive to me.

3) In building the grads executable, say gradsc, I propose we first
create a library with all the .o's that would normally go into gradsc,
except for the main.o above, then

a) On Win32, we make this library dynamic (libgradsc.dll) and link

      gcc -o gradsc main.o libgradsc.dll

   This libgradsc.dll is then used to build the dynamic extensions as well.

b) On Linux/Unix, I propose we create a static libgradsc.a instead and then link

      gcc -o gradsc main.o libgradsc.a + $(LIBS)

 The reason for keeping the library static on Linux/Unix is to keep
the changes to a minimum. If you make it dynamic then the user may
need to set LD_LIBRARY_PATH, an inconvenience with no other benefit.
Of course, in practice we would do all of this with libtool.

   Does anybody have any thoughts/suggestions on this? Another
possibility is to handle the Win32 on a branch, which defeats the goal
of having it built by anyone (and makes maintenance harder).

   Pat: I may need your help with libtool/autotools.Does it seem doable to you?

   Happy Holidays to everyone!

       Arlindo

-- 
Arlindo da Silva
da...@al...

Re: [Opengrads-devel] v2

[Jennifer A. <jm...@co...>]

All,
The 'official' release of 1.9 from COLA will not be substantially  
different from what the opengrads builds are based on, so I see no  
reason for you to hold back in asking people to try the opengrads  
extensions. Pat is right, 1.9 is what's available to play with at the  
moment, but v2 will also be made public just as soon as it is ready.
Jennifer

--
Jennifer M. Adams
IGES/COLA
4041 Powder Mill Road, Suite 302
Calverton, MD 20705
jm...@co...

Re: [Opengrads-devel] v2

[Patrice D. <per...@fr...>]

On Tue, Dec 18, 2007 at 08:15:54AM -0500, Jennifer Adams wrote:
>
> user-defined shared objects. (Add those to the list above.) A new release 
> of v1.9 is a much lower priority than all the work for v2. I don't really 
> see why it matters all that much...

I may be missing something but v2 is not public, so 1.9 is the only thing 
we can play with?

--
Pat

Re: [Opengrads-devel] v2

[Arlindo da S. <da...@al...>]

On Dec 18, 2007 8:15 AM, Jennifer Adams <jm...@co...> wrote:
>
> ... Jennifer has promised ...
> I did no such thing. I am very careful to never promise anything related to
> the timing of GrADS work.
>

I apologize for the poor choice of words. I should have said that in
our last conversation you mentioned that it wouldn't be until early
December that you would have a chance to review the current grads
sources under the opengrads repo. Wishfull thinking on my part lead me
to state it in stronger terms.

> Any chance that you would be looking
> at the v1.9 sources in the coming weeks?I have put the patches in gadef()
> and gaprow() into v2, and I will also look at the --enable-dynamic-supplibs
> stuff and the hooks for the new user-defined shared objects. (Add those to
> the list above.) A new release of v1.9 is a much lower priority than all the
> work for v2. I don't really see why it matters all that much...

We have to agree to disagree on this one:

1) I understand and support COLA's commitment to have a v2 release in
a timely fashion, and I acknowledge your contractual commitment with
whoever is funding the effort. It is amazing how much you guys have
accomplished on a shoe string budget.

2) I believe the community could use a fresh build of the v1.9 series
with a usable gradsdods on a larger number of platforms, some bug
fixes, and win32 builds. This would also help iron out any issues with
the new supplibs and get them addressed before v2 is released.

3) From the point of view of the opengrads effort, I'd like to have
the python and perl interfaces exposed to a larger user base, and
hopefully recruit folks interested in further develop these interfaces
and connect to other perl/python efforts (e.g., image generation and
volumetric visualization.)

4) The shared library extensions, especially the stuff from B.-J.
Tsuang can be very useful to a number of people. Although some
extensions developed for v1.9 will require a major port for v2, some
may require only simple updates. Again, until we release this to a
broader user base we won't know of any glitches that the few of us
using this are yet to run into. Dealing with these issues in the
context of the stable v1.9 codebase would be simpler and will pave the
way for v2.

As you know, I do see a point for new v1.9 binaries before v2 is out.
With help from Pat, Gary and Mike we took the COLA v1.9Dev sources
applied a number of bug fixes and have refrained from new features
except for the absolutely necessary for the opengrads extensions. We
are at a point  where could benefit from feedback on our work from the
grads community at large.

Brian/Jennifer: here is the hard question: if you think it would be
sometime until you can have a chance to review our mods to v1.9Dev and
bless a final v1.9 release, can we go ahead and release what we have
as an un-official release candidate on gradsusr? With the usual
disclaimer that it is not the "final" release which will come directly
from COLA, and that COLA is not to be blamed for any issues with the
builds.  As I said many times I do not want to see what we are doing
as a fork but rather as a device to infuse community contributions in
the overall grads development.

  Cheers!

      Arlindo

-- 
Arlindo da Silva
da...@al...

[Opengrads-devel] v2

[Jennifer A. <jm...@co...>]

On Dec 18, 2007, at 12:12 AM, Arlindo da Silva wrote:

> ... Jennifer has promised ...
I did no such thing. I am very careful to never promise anything  
related to the timing of GrADS work.

> Jennifer: How is v2 coming along?
V2 is almost ready for alpha status. I am still testing the ensemble  
handling, the grib2 and new netcdf interfaces in a 64-bit linux  
environment, writing new descriptor files for data collections at  
COLA, patching up problems related to the switch to double precision,  
cleaning out the cruft, making sure the myriad of #IFDEFs are working  
they way they're supposed to, fixing problems with all the switches  
and args in grads.c, tweaking the configure script and makefiles,  
checking and updating the copyright,  looking at changes to the gadap  
library and how they affect GrADS, and updating the documentation.

> Any chance that you would be looking
> at the v1.9 sources in the coming weeks?
I have put the patches in gadef() and gaprow() into v2, and I will  
also look at the --enable-dynamic-supplibs stuff and the hooks for  
the new user-defined shared objects. (Add those to the list above.) A  
new release of v1.9 is a much lower priority than all the work for  
v2. I don't really see why it matters all that much...

Jennifer
--
Jennifer M. Adams
IGES/COLA
4041 Powder Mill Road, Suite 302
Calverton, MD 20705
jm...@co...

[Opengrads-devel] Documentation for pytests

[Arlindo da S. <da...@al...>]

All,

  I have added a wiki page documenting the python based tests:

  http://opengrads.org/wiki/index.php?title=PyTests:_GrADS_Test_Suite#Writing_unit_tests_in_python

Please send me feedback as to which this is sufficiently clear and
about the likely omissions. Unfortunately, working with pytests
requires a working knowledge of python and familiarity with basic OO
concepts such as subclassing, and PyUnit, Python's testing framework.
(If you know JUnit from Java will you feel right at home as PyUnit is
a rip off of Junit). In any event, please correct spelling and other
language issues directly on the wiki.

I can definitely use some help with additional test cases, specially
tests relating to station data and OPeNDAP datasets. Let me know if
anyone would like to help with any of that.

At this point I'll concentrate my efforts on documentation as Jennifer
has promised me to go over the our bug fixes in v1.9.0-rc1 for an
eventual release of the binaries sometime around December. (I hope she
meant December 2007 :-)

Jennifer: How is v2 coming along? Any chance that you would be looking
at the v1.9 sources in the coming weeks?

  Cheers!

     Arlindo

-- 
Arlindo da Silva
da...@al...

Re: [Opengrads-devel] files without license with an author

[Arlindo da S. <da...@al...>]

On Dec 17, 2007 11:33 AM, Wesley Ebisuzaki <Wes...@no...> wrote:
> Arlindo,
>
>    Too verbose for my tastes.  How about
>
>     12/2007  This code is placed into the public domain by the author,
> Wesley Ebisuzaki

This is fine, but I'd add the standard no warranty disclaimer:

 12/2007  This code is placed into the public domain by the author,
Wesley Ebisuzaki.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE

         Arlindo

>
>             Wesley
>
>
> Arlindo da Silva wrote:
> > On Dec 17, 2007 10:14 AM, Wesley Ebisuzaki <Wes...@no...> wrote:
> >
> >> Pat,
> >>
> >>     The fgutil.c was borrowed from some of my code that is now in the
> >> public domain.
> >> Consequently the code is in the public domain.
> >>
> >
> > OK, in this case I propose we include the following notice I adapted
> > from the wgrib site in file fgutils.c:
> >
> >
> > (c) 1996 Wesley Ebisuzaki
> >
> > This the software in this file is hereby placed into the public domain.  While
> > you could legally do anything you want with the code, telling
> > the world that you wrote it would be uncool.  Selling it would
> > be really uncool.  The code was written for the NCEP/NCAR Reanalysis
> > and may not work GRIB files from other sources.  The code,
> > as usual, is not warranted to be fit for any purpose what so ever.
> > Use at your own risk.
> >
> >     Any objections?
> >
> >         Arlindo
> >
> >
> >
> >>            Wesley Ebisuzaki
> >>
> >>
> >>
> >> Mike Fiorino wrote:
> >>
> >>> hi arlindo, no problem with me.. thanks /r mike
> >>>
> >>> On Dec 16, 2007 4:55 PM, Arlindo da Silva <da...@al...> wrote:
> >>>
> >>>
> >>>> Mike, Wes:
> >>>>
> >>>>    Do you authorize us to add the standard GPL notice on these files?
> >>>> Or else, which license would you prefer?
> >>>>
> >>>>        Arlindo
> >>>>
> >>>>
> >>>>
> >>>> ---------- Forwarded message ----------
> >>>> From: Patrice Dumas <per...@fr...>
> >>>> Date: Dec 14, 2007 9:47 AM
> >>>> Subject: [Opengrads-devel] files without license with an author
> >>>> To: ope...@li...
> >>>>
> >>>>
> >>>> Hello,
> >>>>
> >>>> I propose the attached patch for pcx11e.c, which adds the xlibemu
> >>>> license I found in ftp://ftp.delorie.com/pub/djgpp/contrib/xlibemu/.
> >>>> Ok to commit?
> >>>>
> >>>> There are still 3 files that have an author and no license:
> >>>>
> >>>> fgbds.c                         Mike Fiorino
> >>>> fgutil.c                        Wesley Ebisuzaki
> >>>> gstmp.c                         Mike Fiorino and ?
> >>>>
> >>>> --
> >>>> Pat
> >>>>
> >>>> -------------------------------------------------------------------------
> >>>> SF.Net email is sponsored by:
> >>>> Check out the new SourceForge.net Marketplace.
> >>>> It's the best place to buy or sell services
> >>>> for just about anything Open Source.
> >>>> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> >>>> _______________________________________________
> >>>> Opengrads-devel mailing list
> >>>> Ope...@li...
> >>>> https://lists.sourceforge.net/lists/listinfo/opengrads-devel
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Arlindo da Silva
> >>>> da...@al...
> >>>>
> >>>> -------------------------------------------------------------------------
> >>>> SF.Net email is sponsored by:
> >>>> Check out the new SourceForge.net Marketplace.
> >>>> It's the best place to buy or sell services
> >>>> for just about anything Open Source.
> >>>> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> >>>> _______________________________________________
> >>>> Opengrads-devel mailing list
> >>>> Ope...@li...
> >>>> https://lists.sourceforge.net/lists/listinfo/opengrads-devel
> >>>>
> >>>>
> >>>>
> >>
> >
> >
> >
> >
>
>



-- 
Arlindo da Silva
da...@al...

Re: [Opengrads-devel] files without license with an author

[Wesley E. <Wes...@no...>]

Arlindo,

   Too verbose for my tastes.  How about

    12/2007  This code is placed into the public domain by the author, 
Wesley Ebisuzaki

            Wesley

Arlindo da Silva wrote:
> On Dec 17, 2007 10:14 AM, Wesley Ebisuzaki <Wes...@no...> wrote:
>   
>> Pat,
>>
>>     The fgutil.c was borrowed from some of my code that is now in the
>> public domain.
>> Consequently the code is in the public domain.
>>     
>
> OK, in this case I propose we include the following notice I adapted
> from the wgrib site in file fgutils.c:
>
>
> (c) 1996 Wesley Ebisuzaki
>
> This the software in this file is hereby placed into the public domain.  While
> you could legally do anything you want with the code, telling
> the world that you wrote it would be uncool.  Selling it would
> be really uncool.  The code was written for the NCEP/NCAR Reanalysis
> and may not work GRIB files from other sources.  The code,
> as usual, is not warranted to be fit for any purpose what so ever.
> Use at your own risk.
>
>     Any objections?
>
>         Arlindo
>
>
>   
>>            Wesley Ebisuzaki
>>
>>
>>
>> Mike Fiorino wrote:
>>     
>>> hi arlindo, no problem with me.. thanks /r mike
>>>
>>> On Dec 16, 2007 4:55 PM, Arlindo da Silva <da...@al...> wrote:
>>>
>>>       
>>>> Mike, Wes:
>>>>
>>>>    Do you authorize us to add the standard GPL notice on these files?
>>>> Or else, which license would you prefer?
>>>>
>>>>        Arlindo
>>>>
>>>>
>>>>
>>>> ---------- Forwarded message ----------
>>>> From: Patrice Dumas <per...@fr...>
>>>> Date: Dec 14, 2007 9:47 AM
>>>> Subject: [Opengrads-devel] files without license with an author
>>>> To: ope...@li...
>>>>
>>>>
>>>> Hello,
>>>>
>>>> I propose the attached patch for pcx11e.c, which adds the xlibemu
>>>> license I found in ftp://ftp.delorie.com/pub/djgpp/contrib/xlibemu/.
>>>> Ok to commit?
>>>>
>>>> There are still 3 files that have an author and no license:
>>>>
>>>> fgbds.c                         Mike Fiorino
>>>> fgutil.c                        Wesley Ebisuzaki
>>>> gstmp.c                         Mike Fiorino and ?
>>>>
>>>> --
>>>> Pat
>>>>
>>>> -------------------------------------------------------------------------
>>>> SF.Net email is sponsored by:
>>>> Check out the new SourceForge.net Marketplace.
>>>> It's the best place to buy or sell services
>>>> for just about anything Open Source.
>>>> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
>>>> _______________________________________________
>>>> Opengrads-devel mailing list
>>>> Ope...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/opengrads-devel
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Arlindo da Silva
>>>> da...@al...
>>>>
>>>> -------------------------------------------------------------------------
>>>> SF.Net email is sponsored by:
>>>> Check out the new SourceForge.net Marketplace.
>>>> It's the best place to buy or sell services
>>>> for just about anything Open Source.
>>>> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
>>>> _______________________________________________
>>>> Opengrads-devel mailing list
>>>> Ope...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/opengrads-devel
>>>>
>>>>
>>>>         
>>     
>
>
>
>   

Re: [Opengrads-devel] files without license with an author

[Patrice D. <per...@fr...>]

On Mon, Dec 17, 2007 at 11:20:19AM -0500, Arlindo da Silva wrote:
> On Dec 17, 2007 10:14 AM, Wesley Ebisuzaki <Wes...@no...> wrote:
> > Pat,
> >
> >     The fgutil.c was borrowed from some of my code that is now in the
> > public domain.
> > Consequently the code is in the public domain.
> 
> OK, in this case I propose we include the following notice I adapted
> from the wgrib site in file fgutils.c:
> 
> 
> (c) 1996 Wesley Ebisuzaki
> 
> This the software in this file is hereby placed into the public domain.  While
> you could legally do anything you want with the code, telling
> the world that you wrote it would be uncool.  Selling it would
> be really uncool.  The code was written for the NCEP/NCAR Reanalysis
> and may not work GRIB files from other sources.  The code,
> as usual, is not warranted to be fit for any purpose what so ever.
> Use at your own risk.
> 
>     Any objections?

This looks good too.

--
Pat

Re: [Opengrads-devel] files without license with an author

[Patrice D. <per...@fr...>]

On Mon, Dec 17, 2007 at 10:14:02AM -0500, Wesley Ebisuzaki wrote:
> Pat,
> 
>     The fgutil.c was borrowed from some of my code that is now in the 
> public domain.
> Consequently the code is in the public domain.  
> 
>            Wesley Ebisuzaki

Thanks.

I'll commit the aattached patch unless somebody objects.

--
Pat