Home

Setup

Once downloaded, unpack the compressed folder to see the structure of (the compiled) TestMP. You'll see a structure like this:

testmp/
|-- bin/      # contains the scripts for startup & maintenance.
|-- conf/     # contains the configuration file.
|-- data/     # contains the database files.
|-- lib/      # contains the TestMP client libraries.
|-- log/      # contains the generated log files.
|-- webapp/   # contains the wars of WebConsole and DataStore.
|-- README.md

It's not forced but recommended that you set the TESTMP_HOME environment variable, of which the value is the path to the testmp/, if you hope to launch TestMP outside such directory.

Before running TestMP, you also need to have Java Runtime Environment (Version 6 or the above) installed, and make sure the JAVA_HOME is set correctly.

Configuration

You will find most TestMP configurations in the conf/testmp.properties:

# The locale setting of webconsole and reports
locale=en_US

# The launching url of the datastore each for test case, data, and environment.
testCaseStoreUrl=http://localhost:10081/DataStore.do
testDataStoreUrl=http://localhost:10082/DataStore.do
testEnvStoreUrl=http://localhost:10083/DataStore.do

# The default automation service address
automationServiceUrl=http://localhost:8888

# The number of threads for running tasks.
executionThreadNum=10
# The timeout of task execution. 0 means no timeout.
executionTimeout=0

# The time gap (in seconds) between two refreshings of the task schedule.
scheduleRefreshingTimeGap=1800
# The maximum latency (in seconds) to trigger a scheduled task, or ignore it.
taskTriggerMaxLatency=600

# The default recipient list and subject of the test metrics report. 
testMetricsReportRecipients=
testMetricsReportSubject=

# The default recipient list and subject of the test env status report.
envStatusReportRecipients=
envStatusReportSubject=

# The default SMTP settings to sending the report.
smtpSettingHost=
smtpSettingPort=25
smtpSettingUser=
smtpSettingPass=
smtpSettingSTARTTLS=true

Currently TestMP supports UI languages of US English (the default) and Chinese. Change the locale to zh_CN if you want to change the language to Chinese.

While most of the other settings can be left to their default values, testCaseStoreUrl, testDataStoreUrl, testEnvStoreUrl may need to be modified if the default listening ports have been occupied or they are launched remotely.

It's also possible to make these urls the same to share only one datastore. But in practice it will not be efficient and may cause confusion.

Configurations in this file are default settings. A team or its member can override the value on the Web Console's 'Settings' window according to the personal needs.

Launch the TestMP

Now we're ready to startup the TestMP!

• On Windows

  cd TESTMP_HOME/bin
  startup.bat [port]
  

• On Linux / Mac

  cd TESTMP_HOME/bin
  ./startup.sh [port]
  

The argument port is optional. If it is not given, the Web Console will defaultly use 10080 as the listening port. Then you may see the output like this:

launching testCaseStore on 10081
launching testDataStore on 10082
launching testEnvStore on 10083
launching TestMP web console on 10080
...
2013-07-04 14:26:55.634:INFO:oejs.AbstractConnector:Started SelectChannelConnector@0.0.0.0:10080

which meanse the datastores for test case, data, and environment, and the web console are successfully and fully launched.

Open your favorite browser, enter "http://the_name_or_ip_of_your_host:10080" in the address bar and click Go. You see the the welcome page on the TestMP Web Console? Congratulations!

Add New Test Case

To add new test case, you can click the 'New' button on the Web Console's 'TestCase' tab, and input necessary information. Then by clicking the '*' column of the new case, you will see the generated code as follows:

• TestNG

  public class TestSomething {
  
      @Test(groups = { "group1", "group2" })
      @TestDoc(
          project = "project name", 
          name = "test case name", 
          description = "test case description")
      public void testSomething() {
          // test steps
      }
  }
  

• JUnit

  public class TestSomething extends TestSyncForJUnit {
  
      @TestDoc(
          project = "project name", 
          name = "test case name", 
          description = "test case description", 
          groups = { "group1", "group2" })
      @Test
      public void testSomething() throws Exception {
          // test steps
      }
  }
  

The values of @TestDoc depend on your provided when creating the case. Move the code to your automation project, and implement the test steps.

TestMP currently supports binding to TestNG or JUnit tests. For JUnit test, at least JUnit 4.9 is required. To bind automation test cases to TestMP, you need to put lib/*.jar on the class path of your automation project. If you use Maven to build your automation, an alternate way is to add the dependency below:

<dependency>
    <groupId>org.testmp</groupId>
    <artifactId>testsync</artifactId>
    <version>1.0.3</version>
</dependency>

Note: If you directly implement the test case in your automation project, and run it following the way below, it will also be automatically added to the Web Console.

Note: for TestMP v1.0.3, please download testsync-1.0.3.jar to replace lib/testsync-1.0.2.jar if you are going to add lib/*.jar to the classpath.

Run Test Case

Any updates of @TestDoc and the test result can be automatically synchronized to the Web Console. The only necessary step is passing the JVM arguments below when running test case:

Example:

-DupdateTestDocument=true -DupdateTestMeasures=true -DtestmpAddr=192.168.12.34

• updateTestDocument - whether to automatically update the test document.
• updateTestMeasures - whether to automatically update the test measures.
• runHistoryCapacity - the maxium number of test run records that are kept.
• testmpAddr - the remote host running TestMP.
• testCaseStoreAddr - the remote address of test case store (host:port).
• testDataStoreAddr - the remote address of test data store (host:port).

All these arugments have default values:

• updateTestDocument and updateTestMeasures default to false.
• runHistoryCapacity defaults to 30.
• testmpAddr defaults to localhost.
• TestMP infers testCaseStoreAddr and testDataStoreAddr from testmpAddr.

You only need to specify testmpAddr when it's running remotely. If test case store and test data store are running not on the same host as TestMP, or not listening to the default ports, then you should specify testCaseStoreAddr and testDataStoreAddr respectively instead of testmpAddr.

After the first run from the automation code, you can also launch the test case directly from the Web Console, by clicking the '*' column of each record, or by clicking the 'Run' button on the Web Console's 'TestCase' tab.

Such convenience needs some pre-configurations - the automation service. It is a server that accepts query parameters automation and action.

• action - there're 3 actions: 'query', 'launch', and 'cancel'

• automation - a list of the full name of test cases separated by ',' to act on.

Each response should return a string composed of '0' and '1', corresponding to the sequence specified in the query parameter automation. '1' means the test case is running, '0' means the reverse.

You can implement your own automation service, but TestMP provides one which you can learn the usage by running the command:

python3 $TESTMP_HOME/bin/automation_service.py -h

Example:

python3 $TESTMP_HOME/bin/automation_service.py -c "command"

For TestNG, the command could be:

java org.testng.TestNG -methods {a}

For JUnit, the command could be:

java org.testmp.sync.junit.RunTest -methods {a} /* To be done */

Note:

1. make sure the testng or testsync library and your automation project are in the class path.

2. {a} is a parameter holder to be populated by the automation service - the full name of test case (package.class.method)

Finally, remember to set the automation service url in conf/testmp.properties, or in the Web Console's 'Settings' window.

Evaluate Test Case

Open the Web Console, and select the 'Test Case' tab. Clicking the 'Filter' button to filter out the test cases that will be included in the test metrics report. Then clicking the 'Report' button will generate the report waiting for signoff, and there are metrics for each project:

• Total Tests - the number of test cases from this project.
• Groups (tags) - the groups involved in this test.
• Robustness - the distribution of quality robustness evaluated from the run history.
• Effectiveness - the number of real bugs and the number of false alarmings.
• Efficiency - the test run time and its volatility.
  
  

Mark each project as "Accept" or "Refuse" based on the metrics, and click the 'Signoff' button to get the final report. Now it's ready to send the report to the stakeholders. You can set the default settings of sending report in conf/testmp.properties, or on the Web Console's 'Settings' window.

Add Test Data

To add new test case, you can click the 'New' button on the Web Console's 'TestData' tab, and input necessary information.

• Name - the unique identifier of the test data.
• Properties - the json representation of the data properties.
• Parent - the parent data from which the properties are inherited.
• Tags - the groups to which the test data belongs.

Data Hierarchy

For automation code, the test data usually maps to the data model in the developer's code. Inheritance replationship exists between base model and sub models. In the real world, it's also true that many data share some properties, while having sepcial values to the other. TestMP takes such hierarchy into account when storing the test data, and facilitates the usage by merging the data and its parent automatically.

Use Test Data

It's very easy to use the test data in test case:

// Get test data by name
TestData td = TestData.get("test data name");

// Get property value
Object prop = td.getPropertyValue("property name");

// Get all properties
Map<String, Object> props = td.getProperties();

// Convert test data to the instance of class T
T o = td.convertTo(T.class);

// Get a list of test data belonging to some groups
String[] includedTags = new String[] { "tag1", "tag2" };
List<TestData> dataList1 = TestData.get(includedTags);

String excludedTags = new String[] { "tags" };
List<TestData> dataList2 = TestData.get(includedTags, excludedTags);

Manage Test Environment

Typically a test process involves several test environments at different stages including component (unit), integration, and production. And environment builds on one or more hosts.

You may have built or are going to create scripts to solve various environment-replated problems, like deploying packages, checking the build validity, configuring the environment, launching the automation, etc.

TestMP help you organize, schedule, visualize, and monitor these local / remote scripts, to facilitate the test environment & process management.

At first, there are some related terms:

• Environment - a concept about the hardware / software context in which the test is performed.
• Host - one or more devices that comprise the hardware part on which a test environment is built.
• Task - a concept about a group of executions on selected hosts belonging to an environment, usually with a certain purpose.
• Execution - a concept about a script / command running against a specified host, which will generate some results on the end.

Adding an environment to TestMP is intuitive: open the 'Test Environment' tab and click the 'New' button. Then input the environment's name and optionally give a url of the reference page on which the environment detail can be found.

Then you can define tasks for this environment: click the 'Tasks' icon of this environment record; on the pop-up window, click the 'New' button; Then input the task name, and optionally set the schedule if you hope it run repeatly and expectedly. The schedule syntax is unix-like but a little different.

By default, the schedule will take effect at most in 600 seconds. You can configure it in the conf/testmp.properties.

A task can have multiple executions at the same time: click the Executions icon of the task record to open the window. Each execution has the following fields:

• on/off - enable or disable such execution.
• Host - the host on which the execution is expected to run.
• Working Directory - the current directory for the execution.
• Command - the actual command to be executed.
• RetCode - the returned code of the execution's last run.
• Trace - the output of the last execution, which can be refreshed if the last run is still in progress.

When adding a new execution record, by default, its Host will be set to "localhost", the host on which TestMP is running. If you want it to be another, you'll need to firstly register that host.

Click the 'Hosts' button on the "Test Environment" tab. For each added new host, the hostname, username, and password are required to setup the SSH connection.

For connection failures, you may want to make sure the host administrator has enabled the "password" authentication method for its SSH configuration.

Besides scheduling the tasks, you can launch them at any time by clicking the 'Run' button of the task record.

By clicking the 'Report' button on the "Test Environment" tab, you will get a summary report about the status of all the test environments shown in list, just like below:

and you can send it as email to the people who are interested in this.

Project Members:

• Zhaowei (admin)