Overview

This repository contains a copy of the ENES Coupling Benchmark v1.0.0 code produced as part of IS-ENES2 Coupling Technology Benchmarks project. The codebase has been modded to build and run on the Titan supercomputing cluster. The Coupling Benchmarks Report summarizes the initial results of the benchmark runs.

This repository contains a variety of helper codes:

1. Build script for ESMF on Titan: ./titan/build-scripts/build-esmf.sh
2. Build script for Coupling Benchmarks on Titan: ./titan/build-scripts/build-ecb.sh
3. Job script for submitting a Coupling Benchmark run using ESMF: ./titan/jobs/ecb-timing.sh
4. Python module for analyzing multiple benchmark scenarios: ./pyecb

Building on Titan

Building ESMF

It is of course possible to use your own ESMF build. If obliged, the build-esmf.sh script will build ESMF on Titan loading the appropriate modules, etc. Only two variables must be modified:

1. In build-esmf-env.sh, set $PREFIX to the installation directory for ESMF. This must exist somewhere beneath $MEMBERWORK.
2. In build-esmf-env.sh, set $ESMF_DIR to to the ESMF source directory.

Known Issues:

• Builds confirmed with Intel compilers only. There were build issues with PGI and GNU (I think) when comping the Coupling Benchmarks code.

Building the Coupling Benchmarks

It is recommended that the script build-ecb.sh is used for building the Coupling Benchmarks. The following environment variables should be modified.

1. In build-ecb.sh, set $ESMF_INSTALL_PREFIX to the ESMF installation directory below $MEMBERWORK.
2. In build-ecb.sh, set $ECB_DIR to the source code directory for the Coupling Benchmarks.
3. In build-ecb.sh, set $ECB_INSTALL_PREFIX to the installation directory for the Coupling Benchmarks somewehere below $MEMBERWORK.

With these environment variables modified, this script can be executed on a longin node to build the Coupling Benchmarks.

Running a Scenario on Titan

The ecb-timing.sh script can execute a single "scenario" using the Coupling Benchmark code. This job is configured to run the Very High Resolution (VHR) grid combination that causes the greatest slowdown with ESMF field exchanges (see page 16 in the Coupling Benchmarks Report).

The following environment variables should be modified:

1. In ecb-timing.sh, set $ECB_OUTDIR to the target output data directory to copy benchmark data files to following a benchmark run. This must be under $MEMBERWORK.
2. In ecb-timing.sh, set $ECB_CONFIG_USER to the source directory containing the ESMF namelist files describing the grids and decompositions for the scenario.
3. In ecb-timing.sh, set $ECB_INSTALL_PREFIX to the installation directory for the Coupling Benchmarks.

The following variables may be optionally modified:

1. In ecb-timing.sh, set $ID_SCENARIO to a unique identifier for the scenario (no convention - just for internal record keeping).
2. In ecb-timing.sh, set $SAMPLE_COUNT_NUMBER to the desired sample identifier. Sample directories are nested in the data output directory allowing multiple runs to be cataloged.
3. In ecb-timing.sh, set $ECB_NPROCS to the total number of processors.
4. In ecb-timing.sh, set nodes=* in #PBS -l walltime=00:30:00,nodes=216 to the total number of nodes (nprocs/16).

The scenarios themselves (i.e. decompositions and grid sizes) can be modified by changing namelist parameter values in the directory specified by $ECB_ESMF_INSTALL_PREFIX. A README is also in that directory and should be read before modifying these files. The files to modify are:

1. lonlat_config1.nml
2. lonlat_config2.nml
3. parallel_config.nml