Menu ▾ ▴

OdtbxTestingProcess

Anonymous

ODTBX Testing Process

The ODTBX Regression Test Suite is designed to be run by a developer on their development box as well as being exercised in an automated nightly run on a central server tha emails the lead developer the results.

This page describes the ODTBX testing process: the automated regression tests, the elements that support it, and how to manually run tests on the regression test machine or on a developer's own machine.

The sections follow a bit of an order: the first section, "Regression Tests" is for any developer who is writing or modifying a regression test to go into the automated suite. The remaining sections are useful for developers who have to know how it fits together or debug test harness failures.

To run the test, see the Regression Test Harness section below.

Regression Tests

Test Function Signature

Each regression test is expected to return a single failure indicator when it is called from the regression test harness. This return value is a scalar where failed=1 or failed=0. The test harness can call the test with no arguments, as in my_test(), or with a set of static arguments. 

function [failed] = my_test()

This is usually done when the ODTBX function under test is being driven by an external test function.

The convention so far has been that if an argument is required for a function to be run in "regression testing" mode then it is via a string, as in 

function [failed] = my_test('RegressionTest')

This is usually done when the ODTBX function itself has the regression or validation tests embedded inside.

Test execution

The test should perform all of its execution and evaluation of test criteria inside the test.

A developer can run an individual test just like the regression test harness via 

>>failed = my_test()
or
>>failed = my_test('RegressionTest')

Regression Test Harness

Overview & Structure

The regression suite harness, regressionTesting.m, contains all the functionality for the regression suite harness. This harness expects all the ODTBX and JAT paths to already be set up.

To run the regression suite, just run the function with no arguments 

>>regressionTesting

This harness records which tests passed, which failed, and which had execution errors (broken tests). The harness wraps each test with a MATLAB try-catch block to identify broken tests. These are output to the MATLAB console after all tests have been run.

Modes of Operation

The regression suite harness runs all tests twice. The first time all tests are run with all MATLAB paths (on the developer's machine or the regression test machine). The second time is all tests with the all of the MATLAB toolbox paths removed from the MATLAB path. This is done because some ODTBX functions operate differently depending if the Signal Processing Toolbox is installed. However, all ODTBX functions should operate/pass regardless of what toolboxes are installed. No users are currently required to have MATLAB toolboxes.

regressionTesting.m: Adding/Removing Tests

regressionTesting.m has a cell array at the top of the function that specifies the tests that should be run, testCases. testCases is a cell array list of test case ODTBX MATLAB function names and any required function arguments. They are run with one output argument, as in "failed = eval(testCases{i})". Each test is requried to return 0 if the case passed, and a 1 if the case failed.

To remove a test, simply remove or comment out the string of the test. To add a test, add the test name and any arguments to the list. Be sure to append new tests rather than inserting them into the list.

The tests are run in list order.

Automated Regression Test Description

This regressionTesting.m test harness is run each night via a cron job on the UNIX machine cypher at GSFC. The regressionTesting.m script is invoked by the regressionTest.sh shell script with additional arguments that enable a nightly email of the test results to be sent to a designated ODTBX developer. The ODTBX and JAT Subversion working copies that support the regression tests are updated before the tests are run. Additionally, Apache Maven is used to rebuild JAT before the tests are run.

The odtbxtest User

This section is outdated. It needs to be updated for the new server as of Dec 2009.

The ODTBX regression suite is run as a separate user, odtbxtest, on cypher.

Not only is this safer for the machine, but differences in the root cron job user environment settings were causing strange (and unrepeatable) behavior in the regression test suite. The strange behavior wasn't present for any other user running in a cron job or on any other development platform.

Mantis issues 186 and 37 chronicle some of the problems that led to the change from running as root.

The cron Job

This section is outdated. It needs to be updated for the new server as of Dec 2009.

The odtbxtest user cron job runs nightly to kick off the script in /usr/local/bin/odtbx.cron. This specifies a log file (currently /var/log/odtbx/odtbx.log), the email addresses of where to send the results, and executes the regressionTest.sh shell script.

The Files and Directories

This section is outdated. It needs to be updated for the new server as of Dec 2009.

On cypher, the directories that support the ODTBX regression test suite are located in /opt. The directories are working copies from the SVN repositories. (See the [RepositoryInstructions] for a refresher on these.)

  • jat
    • This is the working copy from the JAT repository /branches/ODTBX.
    • JAT is built here by Maven.
  • jat-lib
    • This is the working copy from the JAT repository /external_jar.
    • These are used by Maven to build JAT.
  • odtbx
    • This is the working copy from the ODTBX repository /trunk.
    • The code under test and the regression test suite code.
    • Additional log files are added under /opt/odtbx/Regression_Validation/logs.
  • odtbx-vendor
    • This is the working copy from the ODTBX repository /vendor.
    • The MICE support files for any ODTBX that requires them for regression testing.
  • subversion_config
    • Data required by the local Subversion client in order to access and update from the ODTBX and JAT repositories.

regressionTest.sh: The Shell Script

This section is outdated. It needs to be updated for the new server as of Dec 2009.

The regressionTest.sh script is a Bash shell script that does all the work before the tests can be run and then invokes MATLAB and the regressionTesting.m script. regressionTest.sh has optional arguments for setting the email and SMTP server of the test results recipient. It sets the log directory, currently /opt/odtbx/Regression_Validation/logs. It also establishes a log file to hold the results of the SVN update and Maven build before the regression tests are run, currently /opt/odtbx/Regression_Validation/logs/supporting_info.txt.

The SVN Update

regressionTest.sh then invokes an svn update on each of the four working copies listed above. Once the updates are complete, the svn info for the working copies are also run in order to capture the repository information.

The Maven Build

This section is outdated. It needs to be updated for the new server as of Dec 2009.

The nightly JAT build is performed with Apache Maven. In /opt/jat/, the JAT working copy, there is an additional maven directory that holds the Maven build setup. JAT doesn't currently follow the expected Maven (/opt/jat/maven/) layout of...

  • src
    • main
    • java
    • resources
    • test
  • target

...So additional directories and symbolic links are made under src/main/resources/jat/... and src/main/java over to the files in various /opt/jat locations in order for Maven to find the required JAT supporting resource and source files. This is somewhat complicated but will have to remain in place (and maintained) until JAT can be restructured in a Maven-friendly way. Many JAT classes have hardcoded assumptions on where to find their resource files. They expect those files to be in the same file/path location as the class file itself.

Maven places the results of its build (.class files and resource files) in the /opt/jat/maven/target/classes directory. The structure below this mirrors that of the JAT source code. Note, this is different than the typical Eclipse developer setup where the source, bytecode, and resource files are all together in the same directory structure. The ODTBX startup.m file expects this difference when invoked in regression testing mode.

Invoking MATLAB and startup.m

This section is outdated due to changes required to make the regression testing work with the Parallel Computing Toolbox. It needs to be updated for the new server as of Dec 2009.

The ODTBX startup.m has three modes of being invoked:

  1. As a developer (Where the developer has set their paths at the top of startup.m.)
  2. As an end-user install (Where there are no developer paths set at the top of startup.m and the installer has replaced a key word in the file with the user's installed path.)
  3. As a regression test run.

startup.m looks for the MATLAB workspace variable basePath. If it is present then startup.m assumes it is a regression test run and ignores any other configuration. regressionTest.sh invokes MATLAB and sets this variable before running startup.m. This is how startup.m can be pulled directly from the ODTBX SVN repository and run to support a regression suite without modification.

MATLAB is invoked by regressionTest.sh using the ability to pass commands directly to the MATLAB command interpreter. MATLAB is started with:

  • -nodisplay
  • set the basePath variable
  • add this basePath to the MATLAB path (to find startup.m)
  • run startup.m
  • then run regressionTesting.m with special arguments for sending emails
  • then to automatically quit after the tests are run.

Invoking regressionTesting.m

When running a nightly regression test, regressionTesting.m is invoked with arguments that:

  • set the log path to capture the results and create the email attachment report,
  • set the email(s) of the report recipient,
  • specify the SMTP server used to send the email report,
  • specifies a file that contains information to be appended to the outgoing email report.

The only differences between invoking regressionTesting.m with no arguments and invoking regressionTesting.m with the above arguments occur after the tests have been run.

The Logs

This section is outdated. It needs to be updated for the new server as of Dec 2009.

The ODTBX cron job holds a log file at /var/log/odtbx/odtbx.log. This is rotated by logrotate. This log contains a transcript of all the shell commands and the MATLAB session. If there is a failure in the regression test harness itself this is the first place to look... especially if the nightly email doesn't go out.

/opt/odtbx/Regression_Validation/logs This directory contains the logs that describe the summary of the test results and build information. This is created by regressionTesting.m and is what is sent out in the nightly email. Currently these tend to build up because there is no automated mechanism to erase them.

/opt/odtbx/Regression_Validation/logs/supporting_info.txt This file holds the svn info and Maven build data. The contents of this file are appended by regressionTesting.m to the outgoing email attachment. Usually if there is a prolem with the svn update or maven build then regressionTest.sh might exit early and MATLAB won't get run. The nightly email might not go out. Look here to see if this file was populated. It can give an idea if the failure occurred before MATLAB was invoked or not. It is overwritten each time the regressionTest.sh script is invoked.

Email

The formatting and sending of the regression test report email all occurs in regressionTesting.m. It uses the MATLAB built-in function sendmail. The subject has a quick summary: "All ODTBX Regression Test Cases Passed on: <date>" or "An ODTBX Regression Test Case Failed/Broke on: <date>". The log created by regressionTesting.m is attached.

Updating the email recipient: The email recipients could/should be updated in two places:

  1. In the /usr/local/bin/odtbx.cron file
  2. In the regressionTest.sh default email recipient.

Manual Tests on the Regression Machine

This section is outdated. It needs to be updated for the new server as of Dec 2009.

Here's some instructions that allow you to run the exact same ODTBX code that is used for the regression tests on the regression test machine.

Log into fftbwall then ssh over to cypher.

On cypher: 

cd /opt/odtbx

start matlab with: 

matlab -nodisplay

MATLAB will complain about a startup problem... don't worry.

In MATLAB, do: 

>>basePath = '/opt/odtbx/';
>>addpath(basePath);
>>startup();

Then check your matlab path... you should see the /opt/odtbx directories and JAT directories there.

Now you can run the code that is checked into cypher.

If you want to manually run the regression test suite on cypher, just like you would manually run the suite on your development box, then run 

>>regressionTesting

Navigation

[ForDevelopers] - Go back to the main development page

WikiStart - Go back to OD Toolbox Home

Related

Wiki: ForDevelopers
Wiki: Home
Wiki: RepositoryInstructions

MongoDB Logo MongoDB