ccplot-general

python 2.7 instead of 2.6?

[Erik B. <eb...@fs...>]

Hi,

I followed the install directions for ccplot with success up to testing 
the install of PyNIO. My only deviation from the instructions was to 
(accidentally) use a slightly newer version of Python, 2.7, rather than 
the 2.6.2 that matches my gcc version. My setup:

Quad-core Intel Xeon CPU E5345 @ 2.33GHz CentOS 5 (only standard packages 
from standard repositories except "sshfs" from dag).

Python 2.4, the standard CentOS 5 version is in /usr/bin. Python 2.7 is in 
/usr/local/bin. By default, PATH=/usr/local/bin:/usr/bin

RPM installed software includes gcc version 4.1.2, libpng and 
freetype (including their devel files).

Installed in /usr/local are: Python 2.7, numpy (--fcompiler=gnu95 to match 
lapack/blas), matplotlib, basemap, and GEOS. I was able to test numpy with 
this command:
/usr/local/bin/python -c 'import numpy; numpy.test()'
Running unit tests for numpy
NumPy version 1.5.0b1
NumPy is installed in /usr/local/lib/python2.7/site-packages/numpy
Python version 2.7 (r27:82500, Aug 11 2010, 13:21:14) [GCC 4.1.2 20080704 
(Red Hat 4.1.2-48)]
nose version 0.11.4
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K................................................................................................................................................................................................................................K............................................................................................K......................K.................................................................................................................................................................................
 .............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 ..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Warning: 
divide by zero encountered in log
.......................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 2962 tests in 10.621s

OK (KNOWNFAIL=4)
[But the exit status of the test was zero.]

With those prerequisites installed, I untarred 
PyNIO-1.3.0b5.linux-x86_64-redhat-py262-gcc412.tar.gz to /usr/local, which 
made a /usr/local/lib/python2.6/site-packages directory, so I copied the 
files over to the /usr/local/lib/python2.7/site-packages/. I next ran the 
test listed in ccplot's "INSTALL" file. It failed:

/usr/local/bin/python -c "import Nio; print Nio.__version__"
Traceback (most recent call last):
   File "<string>", line 1, in <module>
   File "/usr/local/lib/python2.7/site-packages/PyNIO/Nio.py", line 145, in 
<module>
     _is_new_ma = get_integer_version(np.__version__) > 10004
   File "/usr/local/lib/python2.7/site-packages/PyNIO/Nio.py", line 138, in 
get_integer_version
     v = int(d[0]) * 10000 + int(d[1]) * 100 + int(d[2])
ValueError: invalid literal for int() with base 10: '0b1'

I tried changing the name of 
/usr/local/lib/python2.7/site-packages/PyNIO-1.3.0b5-py2.6.egg-info to 
PyNIO-1.3.0b5-py2.7.egg-info.

I haven't been able to get it to budge further. I wonder if I ought to 
pull 2.6.2 Python from the CVS tree. Do you have a suggestion?

Regards,

Erik Bryer
Krishnamurti Numerical Weather Prediction Lab
EOAS/Meteorology
Florida State University
Tallahassee, FL 32306-4520
850-644-9277
eb...@fs...

Re: python 2.7 instead of 2.6?

[Peter K. <pet...@wa...>]

Erik Bryer wrote:
> Hi,
> 
> I followed the install directions for ccplot with success up to testing
> the install of PyNIO. My only deviation from the instructions was to
> (accidentally) use a slightly newer version of Python, 2.7, rather than
> the 2.6.2 that matches my gcc version.My setup:
> 
> Quad-core Intel Xeon CPU E5345 @ 2.33GHz 	 (only standard
> packages from standard repositories except "sshfs" from dag).
> 
> Python 2.4, the standard CentOS 5 version is in /usr/bin. Python 2.7 is
> in /usr/local/bin. By default, PATH=/usr/local/bin:/usr/bin
> 
> RPM installed software includes gcc version 4.1.2, libpng and freetype
> (including their devel files).
> 
> Installed in /usr/local are: Python 2.7, numpy (--fcompiler=gnu95 to
> match lapack/blas), matplotlib, basemap, and GEOS. I was able to test
> numpy with this command:
> /usr/local/bin/python -c 'import numpy; numpy.test()'
> Running unit tests for numpy
> NumPy version 1.5.0b1
> NumPy is installed in /usr/local/lib/python2.7/site-packages/numpy
> Python version 2.7 (r27:82500, Aug 11 2010, 13:21:14) [GCC 4.1.2
> 20080704 (Red Hat 4.1.2-48)]
> nose version 0.11.4
> .......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K................................................................................................................................................................................................................................K............................................................................................K......................K...............................................................................................................................................................................
..
> 
> ............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
.
> 
> ..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Warning:
> divide by zero encountered in log
> .......................................................................................................................................................................................................................................
> 
> ----------------------------------------------------------------------
> Ran 2962 tests in 10.621s
> 
> OK (KNOWNFAIL=4)
> [But the exit status of the test was zero.]
> 
> With those prerequisites installed, I untarred
> PyNIO-1.3.0b5.linux-x86_64-redhat-py262-gcc412.tar.gz to /usr/local,
> which made a /usr/local/lib/python2.6/site-packages directory, so I
> copied the files over to the /usr/local/lib/python2.7/site-packages/. I
> next ran the test listed in ccplot's "INSTALL" file. It failed:
> 
> /usr/local/bin/python -c "import Nio; print Nio.__version__"
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "/usr/local/lib/python2.7/site-packages/PyNIO/Nio.py", line 145,
> in <module>
>     _is_new_ma = get_integer_version(np.__version__) > 10004
>   File "/usr/local/lib/python2.7/site-packages/PyNIO/Nio.py", line 138,
> in get_integer_version
>     v = int(d[0]) * 10000 + int(d[1]) * 100 + int(d[2])
> ValueError: invalid literal for int() with base 10: '0b1'
> 
> I tried changing the name of
> /usr/local/lib/python2.7/site-packages/PyNIO-1.3.0b5-py2.6.egg-info to
> PyNIO-1.3.0b5-py2.7.egg-info.
> 
> I haven't been able to get it to budge further. I wonder if I ought to
> pull 2.6.2 Python from the CVS tree. Do you have a suggestion?
> 
> Regards,
> 
> Erik Bryer
> Krishnamurti Numerical Weather Prediction Lab
> EOAS/Meteorology
> Florida State University
> Tallahassee, FL 32306-4520
> 850-644-9277
> eb...@fs...

Hi Erik,

The PyNIO binary should match exactly with the version of Python
installed. The easiest way would be to install Python 2.6.2. Precompiled
version of PyNIO for Python 2.7 is not yet available, and compiling it
with all the third-party libraries it needs would take a long time.

Regards,
Peter Kuma

Re: python 2.7 instead of 2.6?

[Erik B. <eb...@fs...>]

On Thu, 12 Aug 2010, Peter Kuma wrote:
> Hi Erik,
> The PyNIO binary should match exactly with the version of Python
> installed. The easiest way would be to install Python 2.6.2. Precompiled
> version of PyNIO for Python 2.7 is not yet available, and compiling it
> with all the third-party libraries it needs would take a long time.
> Regards,
> Peter Kuma

Hi Peter,

I cleared the cruft from all the software, and I compiled Python 2.6.2. I 
then re-extracted and recompiled the other software: numpy, matplotlib, 
and basemap. Things seemed to go well enough and any tests ran ok. 
However, when I installed PyNIO and typed '# /usr/local/bin/python -c 
"import Nio;"' the error was essentially identical. To debug, I typed '# 
/usr/local/bin/python -v -c "import Nio;"'. Here is the output just 
before the text of the error output I emailed you last time:
[...]
dlopen("/usr/local/lib/python2.6/lib-dynload/datetime.so", 2);
import datetime # dynamically loaded from 
/usr/local/lib/python2.6/lib-dynload/datetime.so
# /usr/local/lib/python2.6/site-packages/PyNIO/_xarray.pyc matches 
/usr/local/lib/python2.6/site-packages/PyNIO/_xarray.py
import _xarray # precompiled from 
/usr/local/lib/python2.6/site-packages/PyNIO/_xarray.pyc
Traceback (most recent call last):
   File "<string>", line 1, in <module>
   File "/usr/local/lib/python2.6/site-packages/PyNIO/Nio.py", line 145, in 
<module>
     _is_new_ma = get_integer_version(np.__version__) > 10004
[...]

I'm not sure what this tells us except it seems to be loading, then 
crashes.

This is a non-sequiter, but could this be numpy related? I only ask 
because I saw during compile on numpy (using --fcompiler=gnu95) this 
remark:
[...]
   FOUND:
     libraries = ['blas']
     library_dirs = ['/usr/lib64']
     language = f77
[...]
Yet the libraries seem dependent on gfortran, not f77 (an alias to g77).
# ldd libblas.so.3.0.3
         libm.so.6 => /lib64/libm.so.6 (0x00002b12ae958000)
         libgfortran.so.1 => /usr/lib64/libgfortran.so.1 
(0x00002b12aebdb000)
         libc.so.6 => /lib64/libc.so.6 (0x00002b12aee72000)
         /lib64/ld-linux-x86-64.so.2 (0x0000003a3f800000)

I've verified all my blas and lapack shared object files are dependent on 
libgfortran.

By the way, I had installed lapack-devel and blas-devel rpms before 
recompiling numpy. (I'm not using Atlas. From what I read at the Atlas 
site, it provides a speedier blas/lapack.) Another oddity when 
compiling numpy is:
[...]
customize GnuFCompiler
gnu: no Fortran 90 compiler found
gnu: no Fortran 90 compiler found
customize GnuFCompiler using config
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 
-Wall -Wstrict-prototypes -fPIC
[...]
There are other warnings of various kinds while building numpy. But again, 
the test in the README.txt runs with 4 failures of 2962 tests and 
$?=0.

Erik

Re: python 2.7 instead of 2.6?

[Peter K. <pet...@wa...>]

Erik Bryer wrote:
> Hi Peter,
> 
> I cleared the cruft from all the software, and I compiled Python 2.6.2.
> I then re-extracted and recompiled the other software: numpy,
> matplotlib, and basemap. Things seemed to go well enough and any tests
> ran ok. However, when I installed PyNIO and typed '#
> /usr/local/bin/python -c "import Nio;"' the error was essentially
> identical. To debug, I typed '# /usr/local/bin/python -v -c "import
> Nio;"'. Here is the output just before the text of the error output I
> emailed you last time:
> [...]
> dlopen("/usr/local/lib/python2.6/lib-dynload/datetime.so", 2);
> import datetime # dynamically loaded from
> /usr/local/lib/python2.6/lib-dynload/datetime.so
> # /usr/local/lib/python2.6/site-packages/PyNIO/_xarray.pyc matches
> /usr/local/lib/python2.6/site-packages/PyNIO/_xarray.py
> import _xarray # precompiled from
> /usr/local/lib/python2.6/site-packages/PyNIO/_xarray.pyc
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "/usr/local/lib/python2.6/site-packages/PyNIO/Nio.py", line 145,
> in <module>
>     _is_new_ma = get_integer_version(np.__version__) > 10004
> [...]
> 
> I'm not sure what this tells us except it seems to be loading, then
> crashes.
> 
> This is a non-sequiter, but could this be numpy related? I only ask
> because I saw during compile on numpy (using --fcompiler=gnu95) this
> remark:
> [...]
>   FOUND:
>     libraries = ['blas']
>     library_dirs = ['/usr/lib64']
>     language = f77
> [...]
> Yet the libraries seem dependent on gfortran, not f77 (an alias to g77).
> # ldd libblas.so.3.0.3
>         libm.so.6 => /lib64/libm.so.6 (0x00002b12ae958000)
>         libgfortran.so.1 => /usr/lib64/libgfortran.so.1
> (0x00002b12aebdb000)
>         libc.so.6 => /lib64/libc.so.6 (0x00002b12aee72000)
>         /lib64/ld-linux-x86-64.so.2 (0x0000003a3f800000)
> 
> I've verified all my blas and lapack shared object files are dependent
> on libgfortran.
> 
> By the way, I had installed lapack-devel and blas-devel rpms before
> recompiling numpy. (I'm not using Atlas. From what I read at the Atlas
> site, it provides a speedier blas/lapack.) Another oddity when compiling
> numpy is:
> [...]
> customize GnuFCompiler
> gnu: no Fortran 90 compiler found
> gnu: no Fortran 90 compiler found
> customize GnuFCompiler using config
> C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3
> -Wall -Wstrict-prototypes -fPIC
> [...]
> There are other warnings of various kinds while building numpy. But
> again, the test in the README.txt runs with 4 failures of 2962 tests and
> $?=0.
> 
> Erik

Erik,

This is a bug in PyNIO. The error happens because PyNIO expects numpy
version to be in the format x.x.x, where x are numbers. I presume you
are using the 1.5.0b1 version of numpy.
Attached is a patch for Nio.py, which can be applied by:

$ cd /usr/local/lib/python2.6/site-packages/PyNIO/
$ patch < /path/to/pynio.diff

(replacing /path/to/pynio.diff with the actual path)

Peter

/If that does not help or if you have any more questions I will not be
able to respond until Tuesday, because I will be on holiday./

Re: ccplot built (was Re: python 2.7 instead of 2.6?)

[Peter K. <pet...@wa...>]

Hi Erik,

Erik Bryer wrote:
> Numpy had installed arrayobject.h here:
> /usr/local/lib/python2.6/site-packages/numpy/core/include/numpy/.
> However, cctkmodule.c looks for arrayobject.h with:
> #include <numpy/arrayobject.h>
> 
> I noticed PyNIO was detected ok, and I thought maybe because it had a
> Nio.pth. So, I tried adding a .pth to add numpy to python's path. Python
> reported the new path (import sys;sys.path), but "build" still did not
> find arrayobject.h. I just changed the include statement to
> #include
> </usr/local/lib/python2.6/site-packages/numpy/core/include/numpy/arrayobject.h>
> 
> and ccplot built ok. It seems to be running ok too.

On CentOS numpy does not install its header files in the standard
locations such as /usr/include or /usr/local/include as on e.g. Debian.

> Out of curiosity: We have ccplot installed on our server; we noticed the
> client's X-server needs to be running in order to run ccplot on the
> server, and we wondered why this is.

This is because matplotlib imports some Gtk-specific modules by default,
which expect an X server to be available.

> Thanks for the help,
> 
> Erik

I have fixed both of these issues in ccplot 1.43. Thank you reporting them.

Peter