Overview

wreport library and software overview

Definitions

• low level d/encoding
• physically-based data model: unique, high-level, physically-based data model (lon, lat, time, level, parameter, etc.)

Features provided by wreport

• C++ Library for encoding-decoding BUFR and CREX messages at low level
• It includes a simple command line BUFR dumper
• It has a minimal dependency on external libraries
• It cannot convert BUFR between different templates
• It can use tables in standard XML format as distributed by WMO (easy maintenance)

Core library features

• BUFR table loading and querying.
• An abstraction for BUFR variables, storing values with the correct number of significant digits (no artifacts introduced by floats or doubles), and with access to metadata (description, measurement unit, significant digits...) for each variable.
• Unit conversion, transparent unit conversion when working with variables, conversion functions between related code tables.
• Variables can be annotated using other variables, to store values that reference other values, such as quality control statistics or substituted values.
• Script to download and acquire the latest BUFR and CREX tables from WMO.

BUFR decoder features

The decoder supports BUFR editions 2, 3 and 4, both uncompressed and compressed.

C operators supported are:

• 1 change data width
• 2 change data scale
• 4 add associated field, limited to 32 bits of associated fields and to associated field significances 1, 2, and 6.
• 5 character data
• 6 local descriptor with explicit length
• 8 override length of character data
• 22 quality information matched via data present bitmap
• 23 substituted values matched via data present bitmap
• 24 first order statistical values matched via data present bitmap

BUFR encoder features

All features supported by the decoder are supported by the encoder, and tested via decode/reencode/decode runs.

The encoder however does not yet support compression of multisubset data sections.

CREX decoder features

The decoder supports CREX edition 1.

B, R and D opcodes are supported. Some C modifiers may be supported, but are untested due to lack of real-world samples.

CREX encoder features

All features supported by the decoder are supported by the encoder, and tested via decode/reencode/decode runs.

Command line tools

wrep: a tool to print the contents of BUFR and CREX messages.

Features provided by DB-All.e

DB-All.e can import observations from various sources into a unique, high-level, physically-based data model (lon, lat, time, level, parameter, etc.), where they can be manipulated at will and exported back to BUFR or CREX messages. In order to take advantage of the full capabilities of DB-All.e, the availability of a relational database, such as MySQL or sqlite, is required, however many elementary operations, namely the operations that process one message at a time, can be performed on data in plain files, thus not requiring to setup a database and link with a database library.

• It uses the Wreport library for BUFR and CREX encoding and decoding.
• It has a GUI for quickly navigating into observation datasets.
• It can import data from other formats, such as Binary COSMO AOF (ro) and Text with predefined csv structure (rw).
• It understands BUFR with different templates (e.g. those from MARS at ECMWF and some of the new WMO GTS templates).
• Using command line tools it can convert from those formats and write BUFR in WMO/GTS template.
• It can be interfaced to different database architectures for DB operations, e.g. MySQL, Sqlite, Oracle.
• It has a Fortran interface.

Command line tools

• dbatbl: a query tool for BUFR tables
• dbamsg: a tool to manipulate BFUR and CREX messages.
  • prints the contents of messages.
  • selects messages from a file.
  • convert to and from CSV, either raw messages or their physical interpretation.
  • convert data across different formats and/or message templates.
  • finds differences between message contents.
  • AOF messages can also be processed by most dbamsg functions, but they cannot be written.
• dbadb
  • database maintenance: create, dump, reset.
  • import and export of data from/to BUFR and CREX messages.

Fortran API

• Works with physically interpreted data, either in BUFR, CREX or AOF messages or in the DB-All.e database.
• Simple query syntax based on:
  • slicing the physical space by adding constraints on its dimensions (coordinate bounding boxes, minimum/maximum reference time, choice of vertical level, type of statistical processing, type of variable measured);
  • extra constraints based on ground station information or variable attributes (e.g. only data from stations higher than 1000m, or only values with confidence less than 50%).
• Comprehensive documentation.

Python bindings

Various parts of the C++ API are also exported as a Python module, which provides low-level message I/O and full database manipulation.

The basic Python bindings are used to build two more high level Python modules:

• volnd: exports physical data from a DB-All.e database into NumPy multidimensional Masked Arrays.
• rconvert: exports physical data from a DB-All.e database into multidimensional arrays for GNU R.

Graphical interface

provami is a graphical interface based on wxPython that can be used to browse the database, incrementally build queries, and perform import, export and simple manipulation.

On top of BUFR, CREX and CSV, provami can also export GNU R datasets.

Features provided by bufr2netcdf

bufr2netcdf is a tool for converting BUFR reports in the NetCDF format required for COSMO model data assimilation. The input for bufr2netcdf is BUFR format following a well-defined template (e.g. WMO reports). For more information see the page [CosmoNetcdfConversion]. If the input BUFR is not in the required template, DB-All.e dbamsg tool can be used to convert some kinds of input messages to their appropriate templates (see [TemplateConversion]) before converting the BUFR with bufr2netcdf.