Menu
▾
▴
Tree [967457] master / History
Jesper Baasch-Larsen
Jesper Larsen
Jesper Baasch-Larsen
Read Me
Model Validation Toolkit
------------------------
The Model Validation Toolkit (MVT) is a Python toolkit for validating ocean
models against observations. MVT is distributed under the GPL v.3 License,
see COPYING for the full license. Installation instructions can be found
in the INSTALL file. This README provides information to both users and
developers and may thus be somewhat technical.
MVT consists of a chain of 4 separate tools:
vvvvvvv +++ Prepare +++
vvvvv * Read in observations and put them into list of Observation objects
vvv * Save list of Observation objects to disk
v *
vvvvvvv +++ Extract +++
vvvvv * Read in list of Observation objects
vvv * Read in model hindcast/forecast data
v * Interpolate model to obs locs and save ModelObservation objects.
vvvvvvv +++ Bin data +++
vvvvv * Read in list of ModelObservation objects
vvv * Bin data along dimensions defined by user in config file
v * Save binned data to files
vvvvvvv +++ Statistics +++
vvvvv * Read in binned data
vvv * Calculate statistical measures on each bin
v * Save statistical measures to NetCDF file
Prepare
-------
The first tool reads in observations and convert them to a list of socalled
Observation objects. This list is saved to disk. The script is called by
typing:
$ validate_prepobs.py cfgfile outfile
where cfgfile is a configuration file for the script using the ConfigObj format
(http://configobj.voidspace.org) and outfile is the name of the output file
containing the list of Observation objects. The default output file format
is Python pickles. It is however also possible to store the files in JSON
format (see json.org) by applying the --json flag at the command line. This can
be used if you want to investigate or use the output using another programming
language since there are JSON parsers available for almost all programming
languages.
The configuration file specifies how to read in observation data. The snippet
below shows an example where data is read in from a comma separated
temperature/salinity data file from ICES.
[observations]
# The ObservationProviderICES class reads a CSV file containing CTD data
# from ICES and converts the data to Observation objects.
classpath = 'validation.providers.ices_provider.ObservationProviderICES'
# filename -- name of ICES input CSV file
# variables -- list of variable short names to extract from the file
filenames = '../data/ices/ices_200904c.csv'
variables = ['temp', 'salt']
# xmin/xmax -- minimum/maximum longitude (decimal degrees) to extract
# ymin/ymax -- minimum/maximum latitude (decimal degrees) to extract
# zmin/zmax -- minimum/maximum vertical (m) to extract
# tmin/tmax -- minimum/maximum time (YYYYMMDDhhmm) to extract
xmin = 12.5
xmax = 17.5
ymin = 54.5
ymax = 56.5
zmin = None
zmax = None
tmin = None
tmax = None
The classpath variable provides the (Python) path to a class which upon
initialization produces a list of Observation objects. All the variables
defined in the [observations] section are passed as keyword arguments to
this class initializer.
Technically the validate_prepobs script uses the classpath by doing something
corresponding to (using the above example):
from validation.providers.ices_provider import ObservationProviderICES \
as ObservationProvider
config = ConfigObj(cfgfile)
kwargs = config['observations']
obs_provider = ObservationProvider(**kwargs)
obs = obs_provider.data
where kwargs is a dictionary containing all the keys and values from the
[observation] section in the configuration file. The list of Observation
objects are stored in the "data" attribute of the ObservationProvider.
This means that you can easily switch to another ObservationProvider or develop
one yourself. To develop one yourself you should probably take a copy of the
ObservationProviderICES and use that as a template. The main requirement is
that it produces a list of Observation objects upon initialization and it
should also inherit from the DataProvider class (just as the
ObservationProvicerICES does).
Extract
-------
The next tool in the toolchain is used to extract model values at observation
points. The output is saved to disk as a list of socalled ModelObservation
objects. It is called by typing:
$ validate_extract.py cfgfile obsfile outfile
It uses information from a configuration file which looks like this:
[model]
# The ModelProviderNetCDF class reads data from a NetCDF file following the
# Climate and Forecast convention.
classpath = 'validation.providers.netcdf_provider.ModelProviderNetCDF'
# filenames -- NetCDF input files as a time ordered list of files or a string.
# If a string is given it will be globbed and sorted alphanumerically to
# provide a list of NetCDF files.
filenames = '../data/getm/full_dom.3D.salt-temp.3h.6nm.*.v004C.nc'
[interpolator]
# Interpolation setting
classpath = 'validation.interpolators.degradable_linear_interpolator.DegradableLinearInterpolator'
minpresent = 1
The model classpath specifies which ModelProvider class to use. It is the
responsibility of the ModelProvider class to read in model data. You should
never need to change the classpath value to anything else but the NetCDF
model provider. There are however still some limitations to the
Climate and Forecast Convention support in the provider.
The classpath in the interpolator section defines a class to use for the
interpolation of the model field to the observations. Currently a socalled
degradable linear interpolator is included (discussed below). This interpolator
is suitable for interpolating to measurements that have a spatial and temporal
resolution which is smaller than the model resolution. This is normally the case.
But if you for example want to validate your model against satellite observations
with a resolution coarser than the model setup you are using you should consider
implementing an alternative interpolator (e.g. some kind of averaging of the model
grid cells covered by the observation). Most users will however not need to
change the default interpolation method. The "minpresent" parameter specifies
how many model values we need around (in space and time) the observation point
to perform an interpolation. Alternatively you can specify a "maxmissing"
value.
The most important limitation is that the interpolation method in this release
supports only one dimensional coordinate variables. This means that you should
interpolate your vertical dimension to fixed z-coordinates using the gvc2zax
getm-utils tool. And you cannot use the validation tools on rotated or
curvilinear grids. It is however not a big task to add support for
multidimensional coordinate variables. And it would be a most welcome
contribution (Nudge, nudge, know what I mean? Nudge, nudge).
The interpolation method implemented in the ModelProviderNetCDF class extracts
the surrounding points for each Observation object. For a full field in space
and time this means that 8 points are extracted from the NetCDF file for
interpolation. You can however also provide model fields that are time series
at a specific point (the Observation objects must however have their x and y
coordinates set to "None") and 2-dimensional model fields (plus time). In the
latter case the z coordinate of the Observation objects must be "None".
The interpolation operator is a n-dimensional linear interpolator. For
interpolation in one dimension time it is equal to a linear interpolation,
bilinear for two dimensions and so on. It provides an optional graceful
degradation where it ignores a user specified maximum number of missing
values in the interpolation. The interpolation weights of the degraded
interpolation corresponds to those of the n-dimensional linear interpolator
just without the missing values.
MVT includes the Pure Python NetCDF Reader (pupynere) for reading NetCDF files.
This library supports only NetCDF 3 but it is easy to switch to another library
(there will however be some code that needs to be rewritten to support NetCDF
4). The reason for using pupynere is that it uses a memory map (mmap) for the
NetCDF file. This means that you have a buffer which is a map to the NetCDF
file content. When you request data you will therefore only read in data from
that specific position in the file. Other Python NetCDF libraries read in
records of data corresponding to an entire 2D slice. This is of course terribly
inefficient if you just need 4 points around an observation at a given
timestep (even though pupynere for other use cases are terribly slow compared
to other NetCDF libraries for Python).
Binning
-------
When you have succesfully run the validate_extract.py script you should have a
file containing the ModelObservation objects. You supply this file as input
argument together with a new configuration file to the binner script:
$ validate_binner.py cfgfile infile
This configuration file contains two sections. The first is a definition of
what we could call coordinate bins. See example below:
[bins]
[[x]]
units = "degrees_east"
data = "12.5,18.0,0.5"
[[y]]
units = "degrees_north"
data = "54.5,56.75,0.25"
[[z]]
units = "meter"
data = "0,65,5"
positive = 'down'
[[t]]
long_name = 'time'
units = 'days since 2009-04-01 00:00:00'
data = "0,11,1"
Here we have defined four coordinate bins. You must use x, y, z an t for the
spatio-temporal dimensions (custom dimensions providing support for for example
"time since analysis" will be provided in the next version of MVT). The units,
positive and long_name attributes are stored for later use in NetCDF output.
The second section contains a description of the products that the binner
should make.
[products]
[[SpaceTime]]
filename = 'tzyx.bins'
[[[bins]]]
[[[[t]]]]
[[[[z]]]]
[[[[y]]]]
[[[[x]]]]
[[Space]]
filename = 'zyx.bins'
[[[bins]]]
[[[[z]]]]
[[[[y]]]]
[[[[x]]]]
[[Vertical]]
filename = 'z.bins'
[[[bins]]]
[[[[z]]]]
[[Time]]
filename = 'tz.bins'
[[[bins]]]
[[[[t]]]]
[[[[z]]]]
The 2. level subsections ([[subsection]]) are simply names used for the
products. These section names are just used to provide you with an easy
identifier for a product but are not used by the binner script. The filename
is the output filename for a given product. The 3. level subsection [[[bins]]]
denotes which coordinate bins this product should bin the ModelObservation
objects into. The [Vertical] product in the example above would for example
only bin the ModelObservation objects vertically. It is adviced to use an
order of the coordinate bins identical to those in the example.
Statistics
----------
The last tool in the toolchain is the statistics tool. This is simply called
with:
$ validate_measures.py inpfiles
where inpfiles are the output files from the binning tool (the *.bins files
from the example above). The output is a NetCDF file for each of the input
files. These files are named as the input files but with the extension changed
to .nc.
Plotting
--------
It is the responsibility of the user to visualize the statistics. This can
be done interactively using for example ncview or pyncview. Or it can
be visualized using a plotting package. Please contribute by sharing
(somewhat) generic Python plotting scripts for the statistics.
Status
------
MVT is currently in alpha state. You should therefore expect errors, warts and
all.
Future
------
* Improve support for multidimensional coordinate variables (e.g. rotated grids).
* Running bins. It makes sense to add support for overlapping bins. This means
that ModelObservation values can be put into multiple bins for statistical
processing. One statistical measure for these running bins is for example
the running mean. But it is readily extended to other measures as well.
* General expressions in binner configuration file for "time since epoch",
cyclic times, grid displacement and more.
* Support for NetCDF 4 if need for it.
