Menu ▾ ▴

Configuring Bilder

Roopa Pundaleeka

Configuring Bilder

Required configuration information

Before running Bilder you need to tell it where its configuration information is. This is a directory. The value of the environment variable, BILDER_CONFDIR, is set to it. (E.g., BILDER_CONFDIR=/etc/bilder.)

Inside that directory, there must be at least two files.The first, bilderrc, defines a variable, PACKAGE_REPOS_FILE, that contains the name of the file containing the repositories to be searched for tarballs for packages to be built. E.g.,

PACKAGE_REPOS_FILE=${PACKAGE_REPOS_FILE:-"$BILDER_CONFDIR/numpkgssvn.txt"}

This follows the standard Bilder style, that no variable with a value is overwritten. This allows the person executing the build instruction to override any variable value on the command line, e.g., using env.

The Package Repos File, then contains the repos to be searched for packages, with the format:

$ cat numpkgssvn.txt 
##########
#
# File:    numpkgssvn.sh
#
# Purpose: List the package repos in the format,
#          subdir,method=URL
#          Where subdir is the desired location for the repo,
#          method = svn to get by svn, empty to get with wget
#          URL is the resource locator
#
# Version: $Id: numpkgssvn.txt 54 2012-04-08 13:52:09Z cary $
#
##########
PACKAGE_REPO: numpkgs,svn=https://ice.txcorp.com/svnrepos/code/numpkgs/trunk

Each line starting with PACKAGE_REPO: defines the subdir (in this case numpkgs) into which the packages are put, the method (in this case svn) for getting the packages, and after the equals sign, the URL for the directory containing all packages.

For the method (part between the command and the equals sign) of svn, this means that the svn repo will be checked out as empty,

svn co --depth=empty https://ice.txcorp.com/svnrepos/code/numpkgs/trunk numpkgs

and packages will be obtained by

svn up pkgname

in the numpkgs subdirectory.

Optional logic in bilderrc

It can happen that "hostname -f" does not give the fully qualified hostname for your machine. In this case, you can define FQHOSTNAME to contain that hostname.

You can also find the following three methods:

  • bilderGetAuxData defines how to get any auxiliary data needed by a package
  • bilderFinalAction defines a final action (like posting to a dashboard) to be undertaken at the end of a build run
  • signInstaller to sign any installers that you create during your build

Optional additional logic

You can provide specific logic in domainname files that also define default installation directories and such in files named with the domain name. Examples are seen in bilder/runnr. E.g., 

$ cat nersc.gov 
## ######################################################################
##
## File:    nersc.gov
##
## Purpose: Helper functions for setting variables and queues by domain
##
## Version: $Id: nersc.gov 5644 2012-04-02 13:35:02Z cary $
##
## /* vim: set filetype=sh : */
##
## ######################################################################
#
# Adjust the auxiliary names:
#   MAILSRVR, INSTALLER_HOST, INSTALLER_ROOTDIR, FQMAILHOST, BLDRHOSTID
#
runnrSetNamesByDomain() {
# Hosts for which FQMAILHOST is not obvious.  Also ensure that an
# install host name is set for all cases.
  case $UQHOSTNAME in
    cvrsvc[0-9]*)
      FQMAILHOST=carver.nersc.gov
      ;;
    dirac[0-9]*)
      FQMAILHOST=dirac.nersc.gov
      ;;
    freedom[0-9]*)
      FQMAILHOST=freedom.nersc.gov
      RUNNRSYSTEM=XT4
      ;;
    hopper[01][0-9]*)
      FQMAILHOST=hopper.nersc.gov
      RUNNRSYSTEM=XE6
      ;;
    nid[0-9]*)
      FQMAILHOST=franklin.nersc.gov
      RUNNRSYSTEM=XT4
      ;;
  esac
}
runnrSetNamesByDomain
cat >/dev/null <<EOF  ## (Block comment)
MODULES AT NERSC

This is an incomplete list of modules that have to be loaded on the machines that use modules.

FRANKLIN:
Currently Loaded Modulefiles:
1) modules/3.1.6.5
2) moab/5.2.5
3) torque/2.4.1b1-snap.200905131530
4) xtpe-barcelona
5) xtpe-target-cnl
6) MySQL/5.0.45
7) xt-service/2.1.50HDB_PS13A
8) xt-libc/2.1.50HDB_PS13A
9) xt-os/2.1.50HDB_PS13A
10) xt-boot/2.1.50HDB_PS13A
11) xt-lustre-ss/2.1.50HDB_PS13A_1.6.5
12) Base-opts/2.1.50HDB_PS13A
13) PrgEnv-gnu/2.1.50HDB_PS13A
14) xt-asyncpe/3.3
15) xt-pe/2.1.50HDB_PS13A
16) xt-mpt/3.5.0
17) xt-libsci/10.4.0
18) gcc/4.4.1
19) java/jdk1.6.0_07
20) python/2.6.2
21) subversion/1.6.4
22) szip/2.1

HOPPER:
Currently Loaded Modulefiles:
1) modules/3.1.6 9) xt-asyncpe/3.4
2) torque/2.4.1b1 10) PrgEnv-pgi/2.2.41
3) moab/5.3.4 11) xtpe-target-cnl
4) pgi/9.0.4 12) eswrap/1.0.5
5) xt-libsci/10.4.0 13) xtpe-shanghai
6) xt-mpt/3.5.0 14) gcc/4.3.3
7) xt-pe/2.2.41 15) java/jdk1.6.0_15
8) xt-sysroot/2.2.20090720 16) szip/2.1

CARVER:
bilder needs to find either a pgi or a gcc module in your modules list.
EOF

#
# Determine RUNNR_QTARGET, RUNNR_QUEUE, RUNNR_ACCOUNT, RUNNR_PPN
#
runnrSetQInfoByDomain() {
  RUNNR_QTARGET=${RUNNR_QTARGET:-"headnode"}
  local fqdn
  if ! fqdn=`hostname -f 2>/dev/null`; then
    fqdn=`hostname`
  fi
  case $SCRIPT_NAME in
    mkfcall | mkfcpkgs)
      RUNNR_ACCOUNT=${RUNNR_ACCOUNT:-"m681"}    # FACETS
      ;;
    mkvpall)
      RUNNR_ACCOUNT=${RUNNR_ACCOUNT:-"m778"}    # ComPASS
      ;;
    *)
      RUNNR_ACCOUNT=${RUNNR_ACCOUNT:-"m778"}    # ComPASS
      ;;
  esac
  RUNNR_QUEUE=${RUNNR_QUEUE:-"regular"}
  RUNNR_NCPUSVAR=mppwidth
}
runnrSetQInfoByDomain


#
# Set default options.  This has to be called after option parsing.
# Should set
#  CONTRIB_ROOTDIR    The root directory for common installations of tarballs
#  INSTALL_ROOTDIR    The root directory for common installations of repos
#  USERINST_ROOTDIR   The root directory for user installations (same for
#                     tarballs and repos)
#  INSTALL_SUBDIR_SFX Added to subdir (software, contrib, volatile, internal)
#                     to complete the installation dir
#  BUILD_ROOTDIR      Where builds are to take place
#  BILDER_ADDL_ARGS   Any additional args to pass to Bilder
#  MACHINEFILE        The machine file to use
#
setBilderHostVars() {
  #
  # Preliminary variables
  #   Determine the compiler and version for machinefile and namespacing
  #
  local compkey=`modulecmd bash list -t 2>&1 | grep PrgEnv | sed -e 's/^PrgEnv-//' -e 's?/.*??'`
  # echo compkey = $compkey
  if test -z "$compkey"; then
    local comp=
    for comp in pgi gcc gnu; do
      compkey=`module list -t 2>&1 | grep ^$comp | sed -e 's?/.*$??'`
      if test -n "$compkey"; then
        break
      fi
    done
  fi
  if test -z "$compkey"; then
    echo "Cannot determine the compkey.  Quitting."
    exit 1
  fi
  # echo "compkey = $compkey."
  case $compkey in
    gnu)   compkey=gcc;;
    path*) compkey=path;;
  esac
  echo compkey = $compkey
  local compver=`modulecmd bash list -t 2>&1 | grep ^$compkey | sed -e 's?^.*/??'`
  local majorminor=`echo $compver | sed -e "s/\(^[^\.]*\.[^\.]*\).*/\1/"`
  compver=$majorminor
  echo compver = $compver
  # echo "Quitting in nersc.gov."; exit

  # Set the installation and project subdirs
  CONTRIB_ROOTDIR=/project/projectdirs/facets
  if test -z "$PROJECT_INSTSUBDIR"; then
    echo "PROJECT_INSTSUBDIR not set.  Quitting."
    exit 1
  fi
  INSTALL_ROOTDIR=/project/projectdirs/$PROJECT_INSTSUBDIR
  local machinedir=$UQMAILHOST
  if test $UQMAILHOST = freedom; then
    machinedir=franklin
  fi
  CONTRIB_ROOTDIR=$CONTRIB_ROOTDIR/$machinedir
  USERINST_ROOTDIR=$INSTALL_ROOTDIR/$USER/$machinedir
  INSTALL_ROOTDIR=$INSTALL_ROOTDIR/$machinedir
  INSTALL_SUBDIR_SFX="-$compkey-$compver"

  # Set the build directory
  if test -n "$GSCRATCH"; then
    BUILD_ROOTDIR=${BUILD_ROOTDIR:-"$GSCRATCH/builds-${UQHOSTNAME}-$compkey"}
  elif test -n "$SCRATCH"; then
    BUILD_ROOTDIR=${BUILD_ROOTDIR:-"$SCRATCH/builds-${UQHOSTNAME}-$compkey"}
  fi

  # Add to BILDER_ARGS
  BILDER_ADDL_ARGS=-P

  # Set machine file
  case $machinedir in
    hopper | franklin) MACHINEFILE=${MACHINEFILE:-"cray.$compkey"};;
    *) MACHINEFILE=${MACHINEFILE:-"nersclinux.$compkey"};;
  esac
}

This file may also, as seen above, define the method, setBilderHostVars, which also can set the various variables defining where builds should take place, where installations should go, etc.

Related

Wiki: Running Bilder

MongoDB Logo MongoDB