Building NCO is not so easy

Help
Anonymous
2011-02-03
2013-10-17
  • Anonymous - 2011-02-03

    Hello all.  I just spent three days attempting to build NCO and its dependencies on RHEL / CentOS 5.  I attempted with both the GNU compiler and the Intel Compiler.  I finally succeeded and hope others may avoid the pain with these steps.  Cheers.
    http://idolinux.blogspot.com/2011/02/nco-netcdf-operators-build-log.html

     
  • Charlie Zender

    Charlie Zender - 2011-02-06

    Compiling NCO (especially ncap2) builds character more often than it builds itself.
    Thanks for your scripts.
    Please submit patches to help simplify the build system!
    cz

     
  • Kyle Wilcox

    Kyle Wilcox - 2011-03-07

    Thanks Gavin, that got a few of us on the right page…. below is the script that finally got it built for us on RedHat5

      1
      2
      3
      4
      5
      6
      7
      8
      9
     10
     11
     12
     13
     14
     15
     16
     17
     18
     19
     20
     21
     22
     23
     24
     25
     26
     27
     28
     29
     30
     31
     32
     33
     34
     35
     36
     37
     38
     39
     40
     41
     42
     43
     44
     45
     46
     47
     48
     49
     50
     51
     52
     53
     54
     55
     56
     57
     58
     59
     60
     61
     62
     63
     64
     65
     66
     67
     68
     69
     70
     71
     72
     73
     74
     75
     76
     77
     78
     79
     80
     81
     82
     83
     84
     85
     86
     87
     88
     89
     90
     91
     92
     93
     94
     95
     96
     97
     98
     99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    #!/bin/bash
    INSTALL_PATH=/usr/local
    mkdir -p $INSTALL_PATH/src
    # ANTLR2
    APP=antlr-2.7.7
    ANTLR_PATH=$INSTALL_PATH/$APP
    rm -rf $ANTLR_PATH
    cd $INSTALL_PATH/src
    rm -rf antlr-2.7.7*
    wget http://www.antlr2.org/download/antlr-2.7.7.tar.gz
    tar xzf antlr-2.7.7.tar.gz ; cd antlr-2.7.7
    CC=gcc CXX='' ./configure \
    --prefix=$ANTLR_PATH \
    --disable-csharp \
    --disable-java \
    --disable-python 2>&1 | tee $APP.config
    make 2>&1 | tee $APP.make
    make install 2>&1 | tee $APP.install
    # UDUNITS
    APP=udunits-2.1.20
    UDUNITS_PATH=$INSTALL_PATH/$APP
    rm -rf $UDUNITS_PATH
    cd $INSTALL_PATH/src
    rm -rf udunits-2.1.20*
    wget ftp://ftp.unidata.ucar.edu/pub/udunits/udunits-2.1.20.tar.gz
    tar xzf udunits-2.1.20.tar.gz ; cd udunits-2.1.20
    CC=gcc CXX='' F77=gfortran ./configure \
    --prefix=$UDUNITS_PATH 2>&1 | tee $APP.config
    make 2>&1 | tee $APP.make
    make install 2>&1 | tee $APP.install
    # ZLIB
    APP=zlib-1.2.5
    ZLIB_PATH=$INSTALL_PATH/$APP
    rm -rf $ZLIB_PATH
    cd $INSTALL_PATH/src
    rm -rf zlib-1.2.5*
    wget http://zlib.net/zlib-1.2.5.tar.gz
    tar zxf zlib-1.2.5.tar.gz ; cd zlib-1.2.5
    CC=gcc FC=gfortran CXX='' ./configure \
    --prefix=$ZLIB_PATH | tee $APP.config
    make 2>&1 | tee $APP.make
    make install 2>&1 | tee $APP.install
    # HDF5
    APP=hdf5-1.8.6
    HDF5_PATH=$INSTALL_PATH/$APP
    rm -rf $HDF5_PATH
    cd $INSTALL_PATH/src
    rm -rf hdf5-1.8.6*
    wget http://www.hdfgroup.org/ftp/HDF5/hdf5-1.8.6/src/hdf5-1.8.6.tar.gz
    tar xzf hdf5-1.8.6.tar.gz ; cd hdf5-1.8.6
    CC=gcc FC=gfortran CXX='' ./configure \
    --prefix=$HDF5_PATH \
    --enable-fortran \
    --with-zlib=$ZLIB_PATH \
    --with-pic 2>&1 | tee $APP.config
    make 2>&1 | tee $APP.make
    make install 2>&1 | tee $APP.install
    # NetCDF4
    APP=netcdf-4.1.1
    NETCDF4_PATH=$INSTALL_PATH/$APP
    rm -rf $NETCDF4_PATH
    cd $INSTALL_PATH/src
    rm -rf netcdf-4.1.1*
    wget http://www.unidata.ucar.edu/downloads/netcdf/ftp/netcdf-4.1.1.tar.gz
    tar xzf netcdf-4.1.1.tar.gz ; cd netcdf-4.1.1
    CC=gcc FC=gfortran F77=gfortran CXX='' ./configure \
    --prefix=$NETCDF4_PATH \
    --enable-fortran \
    --enable-static \
    --enable-shared\
    --enable-f77 \
    --disable-cxx \
    --enable-netcdf4 \
    --with-hdf5=$HDF5_PATH \
    --with-zlib=$ZLIB_PATH \
    --with-pic 2>&1 | tee $APP.config
    make 2>&1 | tee $APP.make
    make install 2>&1 | tee $APP.install
    # NCO
    APP=nco-4.0.6
    NCO_PATH=$INSTALL_PATH/$APP
    rm -rf $NCO_PATH
    cd $INSTALL_PATH/src
    rm -rf nco-4.0.6*
    wget http://nco.sourceforge.net/src/nco-4.0.6.tar.gz
    tar xzf nco-4.0.6.tar.gz ; cd nco-4.0.6
    export LD_LIBRARY_PATH=$HDF5_PATH/lib:$LD_LIBRARY_PATH
    export PATH=$HDF5_PATH/bin:$PATH
    export LD_LIBRARY_PATH=$NETCDF4_PATH/lib:$LD_LIBRARY_PATH
    export PATH=$NETCDF4_PATH/bin:$PATH
    export LD_LIBRARY_PATH=$ANTLR_PATH/lib:$LD_LIBRARY_PATH
    export PATH=$ANTLR_PATH/bin:$PATH
    export LD_LIBRARY_PATH=$UDUNITS_PATH/lib:$LD_LIBRARY_PATH
    export PATH=$UDUNITS_PATH/bin:$PATH
    export LD_LIBRARY_PATH=$ZLIB_PATH/lib:$LD_LIBRARY_PATH
    export PATH=$ZLIB_PATH/bin:$PATH
    CC=gcc CXX='' \
    NETCDF_INC=$NETCDF4_PATH/include \
    NETCDF_LIB=$NETCDF4_PATH/lib \
    NETCDF4_ROOT=$NETCDF4_PATH \
    HDF5_LIB_DIR=$HDF5_PATH/lib \
    UDUNITS2_PATH=$UDUNITS_PATH \
    LDFLAGS="-L$ANTLR_PATH -lantlr \
    -lhdf5_hl -lhdf5 -L$NETCDF4_PATH/lib -lnetcdf" \
    CFLAGS="-I$HDF5_PATH/include \
    -L$HDF5_PATH/lib \
    -I$ANTLR_PATH/include \
    -L$ANTLR_PATH/lib" \
    CPPFLAGS="-I$HDF5_PATH/include \
    -L$HDF5_PATH/lib \
    -I$ANTLR_PATH/include \
    -L$ANTLR_PATH/lib" \
    ./configure \
    --prefix=$NCO_PATH \
    --disable-shared \
    --enable-netcdf-4 2>&1 | tee $APP.config
    make 2>&1 | tee $APP.make
    make install 2>&1 | tee $APP.install
    
     
  • Charlie Zender

    Charlie Zender - 2011-03-07

    Thanks for posting these build scripts!
    I have linked them to the NCO homepage in both the RPM and in the
    build-from-source sections:
    http://nco.sf.net#rpm
    http://nco.sf.net#bld
    cz

     
  • Kyle Wilcox

    Kyle Wilcox - 2011-11-18

    Updated to work with a RHEL6 or CentOS6 'minimal' install.
    Also added support for C++ and HDF4 reading in NetCDF.

    antlr-2.7.7
    udunits-2.1.24
    zlib-1.2.5
    hdf-4.2.6
    hdf5-1.8.8
    netcdf-4.1.3
    nco-4.0.8

      1
      2
      3
      4
      5
      6
      7
      8
      9
     10
     11
     12
     13
     14
     15
     16
     17
     18
     19
     20
     21
     22
     23
     24
     25
     26
     27
     28
     29
     30
     31
     32
     33
     34
     35
     36
     37
     38
     39
     40
     41
     42
     43
     44
     45
     46
     47
     48
     49
     50
     51
     52
     53
     54
     55
     56
     57
     58
     59
     60
     61
     62
     63
     64
     65
     66
     67
     68
     69
     70
     71
     72
     73
     74
     75
     76
     77
     78
     79
     80
     81
     82
     83
     84
     85
     86
     87
     88
     89
     90
     91
     92
     93
     94
     95
     96
     97
     98
     99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    123
    124
    125
    126
    127
    128
    129
    130
    131
    132
    133
    134
    135
    136
    137
    138
    139
    #!/bin/bash
    yum groupinstall base
    yum install -y gcc gcc-c++ gcc-gfortran curl-devel libxml2-devel bison flex wget make compat-gcc-34-g77 libjpeg-devel expat-devel
    INSTALL_PATH=/usr/local
    mkdir -p $INSTALL_PATH/src
    # ANTLR2
    APP=antlr-2.7.7
    ANTLR_PATH=/usr
    cd $INSTALL_PATH/src
    rm -rf antlr-2.7.7*
    wget http://www.antlr2.org/download/antlr-2.7.7.tar.gz
    tar xzf antlr-2.7.7.tar.gz ; cd antlr-2.7.7
    # Bug in antlr-2.7.7
    sed -i "13a #include <strings.h>" lib/cpp/antlr/CharScanner.hpp
    sed -i "14a #include <cstdio>" lib/cpp/antlr/CharScanner.hpp
    CC=gcc CXX='' ./configure \
    --prefix=$ANTLR_PATH \
    --disable-csharp \
    --disable-java \
    --disable-python 2>&1 | tee $APP.config
    make 2>&1 | tee $APP.make
    make install 2>&1 | tee $APP.install
    # UDUNITS
    APP=udunits-2.1.24
    UDUNITS_PATH=$INSTALL_PATH/$APP
    rm -rf $UDUNITS_PATH
    cd $INSTALL_PATH/src
    rm -rf udunits-2.1.24*
    wget ftp://ftp.unidata.ucar.edu/pub/udunits/udunits-2.1.24.tar.gz
    tar xzf udunits-2.1.24.tar.gz ; cd udunits-2.1.24
    CC=gcc CXX='' F77=gfortran ./configure \
    --prefix=$UDUNITS_PATH 2>&1 | tee $APP.config
    make 2>&1 | tee $APP.make
    make install 2>&1 | tee $APP.install
    # ZLIB
    APP=zlib-1.2.5
    ZLIB_PATH=$INSTALL_PATH/$APP
    rm -rf $ZLIB_PATH
    cd $INSTALL_PATH/src
    rm -rf zlib-1.2.5*
    wget http://zlib.net/zlib-1.2.5.tar.gz
    tar zxf zlib-1.2.5.tar.gz ; cd zlib-1.2.5
    CC=gcc FC=gfortran CXX='' ./configure \
    --prefix=$ZLIB_PATH | tee $APP.config
    make 2>&1 | tee $APP.make
    make install 2>&1 | tee $APP.install
    # HDF4
    APP=hdf-4.2.6
    HDF4_PATH=$INSTALL_PATH/$APP
    rm -rf $HDF4_PATH
    cd $INSTALL_PATH/src
    rm -rf hdf-4.2.6*
    wget http://www.hdfgroup.org/ftp/HDF/HDF_Current/src/hdf-4.2.6.tar.gz
    tar xzf hdf-4.2.6.tar.gz ; cd hdf-4.2.6
    # Bug in configure script, need to define EGREP variable here
    EGREP=egrep CC=gcc FC=gfortran F77=gfortran ./configure \
    --prefix=$HDF4_PATH \
    --disable-netcdf \
    --enable-fortran \
    --with-zlib=$ZLIB_PATH \
    --with-pic 2>&1 | tee $APP.config
    make 2>&1 | tee $APP.make
    make install 2>&1 | tee $APP.install
    # HDF5
    APP=hdf5-1.8.8
    HDF5_PATH=$INSTALL_PATH/$APP
    rm -rf $HDF5_PATH
    cd $INSTALL_PATH/src
    rm -rf hdf5-1.8.8*
    wget http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.8/src/hdf5-1.8.8.tar.gz
    tar xzf hdf5-1.8.8.tar.gz ; cd hdf5-1.8.8
    CC=gcc FC=gfortran CXX='' ./configure \
    --prefix=$HDF5_PATH \
    --enable-fortran \
    --with-zlib=$ZLIB_PATH \
    --with-pic 2>&1 | tee $APP.config
    make 2>&1 | tee $APP.make
    make install 2>&1 | tee $APP.install
    export LD_LIBRARY_PATH=$HDF4_PATH/lib:$LD_LIBRARY_PATH
    export PATH=$HDF4_PATH/bin:$PATH
    export LD_LIBRARY_PATH=$HDF5_PATH/lib:$LD_LIBRARY_PATH
    export PATH=$HDF5_PATH/bin:$PATH
    export LD_LIBRARY_PATH=$ANTLR_PATH/lib:$LD_LIBRARY_PATH
    export PATH=$ANTLR_PATH/bin:$PATH
    export LD_LIBRARY_PATH=$UDUNITS_PATH/lib:$LD_LIBRARY_PATH
    export PATH=$UDUNITS_PATH/bin:$PATH
    export LD_LIBRARY_PATH=$ZLIB_PATH/lib:$LD_LIBRARY_PATH
    export PATH=$ZLIB_PATH/bin:$PATH
    # NetCDF4
    APP=netcdf-4.1.3
    NETCDF4_PATH=$INSTALL_PATH/$APP
    rm -rf $NETCDF4_PATH
    cd $INSTALL_PATH/src
    rm -rf netcdf-4.1.3*
    wget http://www.unidata.ucar.edu/downloads/netcdf/ftp/netcdf-4.1.3.tar.gz
    tar xzf netcdf-4.1.3.tar.gz ; cd netcdf-4.1.3
    CPPFLAGS="-I$HDF5_PATH/include -I$ZLIB_PATH/include -I$HDF4_PATH/include" \
    LDFLAGS="-L$HDF5_PATH/lib -L$ZLIB_PATH/lib -L$HDF4_PATH/lib" \
    LIBS="-lmfhdf -lhdf5_hl -lhdf5 -lantlr -ldf -ljpeg -lz" \
    CXX=g++ CC=gcc FC=gfortran F77=gfortran ./configure \
    --prefix=$NETCDF4_PATH \
    --with-hdf5=$HDF5_PATH \
    --with-hdf4=$HDF4_PATH \
    --enable-fortran \
    --enable-static \
    --enable-shared \
    --enable-f77 \
    --enable-f90 \
    --enable-netcdf4 \
    --enable-hdf4 \
    --enable-cxx-4 \
    --with-pic 2>&1 | tee $APP.config
    make 2>&1 | tee $APP.make
    make install 2>&1 | tee $APP.install
    export LD_LIBRARY_PATH=$NETCDF4_PATH/lib:$LD_LIBRARY_PATH
    export PATH=$NETCDF4_PATH/bin:$PATH
    # NCO
    APP=nco-4.0.8
    NCO_PATH=$INSTALL_PATH/$APP
    rm -rf $NCO_PATH
    cd $INSTALL_PATH/src
    rm -rf nco-4.0.8*
    wget http://nco.sourceforge.net/src/nco-4.0.8.tar.gz
    tar xzf nco-4.0.8.tar.gz ; cd nco-4.0.8
    CC=gcc CXX='' \
    NETCDF_INC=$NETCDF4_PATH/include \
    NETCDF_LIB=$NETCDF4_PATH/lib \
    NETCDF4_ROOT=$NETCDF4_PATH \
    HDF5_LIB_DIR=$HDF5_PATH/lib \
    UDUNITS2_PATH=$UDUNITS_PATH \
    LDFLAGS="-L$ZLIB_PATH/lib -L$NETCDF4_PATH/lib  -L$HDF5_PATH/lib -L$HDF4_PATH/lib -L$ANTLR_PATH/lib -L$UDUNITS_PATH/lib" \
    CPPFLAGS="-I$ZLIB_PATH/include -I$NETCDF4_PATH/include -I$HDF5_PATH/include -I$HDF4_PATH/include -I$ANTLR_PATH/include -I$UDUNITS_PATH/include" \
    LIBS="-lmfhdf -lhdf5_hl -lhdf5 -lantlr -ldf -ljpeg -lz -ludunits2" \
    ./configure \
    --prefix=$NCO_PATH \
    --disable-shared \
    --enable-netcdf-4 2>&1 | tee $APP.config
    make 2>&1 | tee $APP.make
    make install 2>&1 | tee $APP.install
    

    Output of `/usr/local/netcdf-4.1.3/bin/nc-config -all`

    This netCDF 4.1.3 has been built with the following features:
      --cc        -> gcc
      --cflags    ->  -I/usr/local/testingNCO/netcdf-4.1.3/include -I/usr/local/testingNCO/hdf5-1.8.8/include -I/usr/local/testingNCO/zlib-1.2.5/include -I/usr/local/testingNCO/hdf-4.2.6/include
      --libs      -> -L/usr/local/testingNCO/netcdf-4.1.3/lib -lnetcdf
      --cxx       -> g++
      --has-c++   -> yes
      --fc        -> gfortran
      --fflags    -> -g -O2 -I/usr/local/testingNCO/netcdf-4.1.3/include
      --flibs     -> -L/usr/local/testingNCO/netcdf-4.1.3/lib -lnetcdff -lnetcdf
      --has-f77   -> yes
      --has-f90   -> yes
      --has-dap   -> yes
      --has-nc2   -> yes
      --has-nc4   -> yes
      --has-hdf5  -> yes
      --has-hdf4  -> yes
      --has-pnetcdf-> no
      --has-szlib ->
      --prefix    -> /usr/local/testingNCO/netcdf-4.1.3
      --includedir-> /usr/local/testingNCO/netcdf-4.1.3/include
      --version   -> netCDF 4.1.3
    
     
  • Anonymous - 2012-04-20

    anything special needed for CentOS 6.2…I'm 2 days in.

     
  • Daniel L. Wang

    Daniel L. Wang - 2012-04-20

    kwilcox's instructions for RHEL6 should work for CentOs 6.2.

    I'm curious, though, as to why the pre-built NCO in the EPEL repository is insufficient.
    There are RPMs in http://dl.fedoraproject.org/pub/epel/6/x86_64/ , and you can install EPEL as a repository in CentOS so that you can just "yum install nco" (directions at: http://fedoraproject.org/wiki/EPEL )

    I understand that you might want a more recent version than what's already out there and packaged (4.0.5), though.

     
  • bjell

    bjell - 2012-04-26

    Or, it appears that the epel nco was not compiled with support for NetCDF v4.  (If I understand correctly.)

    So, we have to recompile, from  source, with "-enable-netcdf-4".

    And, the CentOS 5 instructions above almost got me there…up to "make" for nco.

    gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../..   -I/usr/local/netcdf-4.1.1/include -I/usr/local/netcdf-4.1.1/include -I/usr/local/hdf
    5-1.8.8/include -L/usr/local/hdf5-1.8.8/lib -I/usr/local/antlr-2.7.7/include -L/usr/local/antlr-2.7.7/lib  -I/usr/include -I/usr/
    local/udunits-2.1.20/include  -I/usr/local/hdf5-1.8.8/include -L/usr/local/hdf5-1.8.8/lib -I/usr/local/antlr-2.7.7/include -L/usr
    /local/antlr-2.7.7/lib -std=c99 -D_BSD_SOURCE -D_POSIX_SOURCE -MT ncap.o -MD -MP -MF .deps/ncap.Tpo -c -o ncap.o ncap.c
    mv -f .deps/ncap.Tpo .deps/ncap.Po
    test -f ncap_yacc.c || /bin/sh ../../autobld/ylwrap ncap_yacc.y y.tab.c ncap_yacc.c y.tab.h ncap_yacc.h y.output ncap_yacc.output
    - byacc  -d -name-prefix=nco_yy
    byacc: e - line 96 of "/usr/local/src/nco-4.0.6/src/nco/ncap_yacc.y", syntax error
    %pure-parser
    ^
    make: ***  Error 1
    make: Leaving directory `/usr/local/src/nco-4.0.6/src/nco'
    make: ***  Error 1
    make: Leaving directory `/usr/local/src/nco-4.0.6/src'
    make: ***  Error 1
    make: Leaving directory `/usr/local/src/nco-4.0.6'
    make: ***  Error 2

    If you see this, byacc is the problem. Install bison (and bison -y will be used by default, if installed).
    (Not that I'm finished…stuck on not finding -lantler, though it's there… *sigh*)

     
  • Charlie Zender

    Charlie Zender - 2012-04-27

    As far as I know, the NCO homepage instructions for installing the base software needed to build NCO are up-to-date.
    They include installing bison and flex and some other libraries (http://nco.sf.net#bld):
    sudo yum install antlr antlr-c++-devel # ANTLR
    sudo yum install curl-devel libxml2-devel # DAP-prereqs
    sudo yum install libdap libdap-devel libnc-dap libnc-dap-devel # DAP
    sudo yum install bison flex gcc g++ # GNU toolchain
    sudo yum install gsl gsl-devel # GSL
    sudo yum install netcdf netcdf-devel # netCDF
    sudo yum install librx librx-devel # RX
    sudo yum install udunits udunits-devel # UDUnits

    Updates welcome, as the package names change occasionally…

     
  • Daniel L. Wang

    Daniel L. Wang - 2012-05-02

    Ah, I see. Looking at the build logs for EPEL (http://koji.fedoraproject.org/koji/packageinfo?packageID=2612), the latest nco build in their testing repo is 4.0.5, and though there is a successful build, the NCO build process couldn't detect netcdf4, so hence the problem.  Good to know.

    Somehow, the fedora NCO rpms build with netcdf4 just fine, but their build setup refuses to acknowledge netcdf4 on epel. Hmmm.

     
  • Charlie Zender

    Charlie Zender - 2012-05-04

    OK. I hope to have a new programmer on-board by this summer who can modify nco.spec to build on Fedora and RHEL/CentOS. TODO nco 1043.
    cz

     

Log in to post a comment.