EvorobotExperimentTest

Prev: Programming a fitness function	Up: Home	Next: Creating custom sensors or motors

Table of contents

• Writing tests for evolutionary experiments
  • An example test
  • Custom tests in the configuration file

Writing tests for evolutionary experiments

In evolutionary robotics experiments, once the evolutionary process is finished, one generally needs to test the behaviour of the resulting agents. For example, it could be useful to test the best robots in the same condition they experienced during evolution, but for a greater number of trials. This would give a measure of performance that is less affected by chance. Moreover different experiments generally require specific tests of the robot behaviour and/or of the evolutionary process.

In FARSA, when using the Evoga and EvoRobotExperiment components, it is possible to implement test components and run them either from the total99 user interface or directly in batch mode. Tests must inherit from the AbstractTest class and are given access to the EvoRobotExperiment instance as well as to the Evoga object and so they can load genotypes, select which genotype to test, modify how the experiment is run and run the test. There are three ready-to-use tests, namely TestRandom, TestIndividual and TestCurrent, but you can write your own test classes in a plugin.

An example test

In this section we will show how to write a custom test. The purpouse of the test is to compute the performance of various evolved individuals under exactly the same conditions (i.e. the random number generator is reset before a new individual is tested). Here is the test class declaration:

:::C++
#include "farsaplugin.h"
#include "abstracttest.h"

class FARSA_PLUGIN_API TestBestsSameConditions : public farsa::AbstractTest
{
    Q_OBJECT
    FARSA_REGISTER_CLASS(AbstractTest)

public:
    TestBestsSameConditions();

    virtual void configure(farsa::ConfigurationParameters& params, QString prefix);
    static void describe(QString type);

    virtual void runTest();

private:
    unsigned int m_seed;
    QString m_resultBasename;
};

The AbstractTest class is a standard FARSA component with only one additional pure virtual function, runTest. This function is called when the test is run and should contain all the code needed to initialize the test environment, excute the test and possibly report the result of the test (e.g. writing results to a file or printing them through the Logger).

:::C++
TestBestsSameConditions::TestBestsSameConditions() :
    AbstractTest(),
    m_seed(1),
    m_resultBasename("testBestsSameConditionsS")
{
    m_menuText = "Test bests with same conditions";
    m_tooltip = "Tests the best individuals of all generations of a given seed and resets the random number generator before each generation, so that the random number sequence is the same for all tested individuals";
    m_iconFilename = QString();
}

The constructor of the TestBestsSameConditions class sets the value of three protected variables of the AbstractTest class. As their name implies, they are used to customize how the test is shown in the menu Tests of the total99 application.

:::C++
void TestBestsSameConditions::configure(farsa::ConfigurationParameters& params, QString prefix)
{
    farsa::AbstractTest::configure(params, prefix);

    m_seed = farsa::ConfigurationHelper::getUnsignedInt(params, prefix + "seed", m_seed);
    if (m_seed == 0) {
        farsa::ConfigurationHelper::throwUserConfigError(prefix + "seed", params.getValue(prefix + "seed"), "The seed value must not be 0");
    }
    m_resultBasename = farsa::ConfigurationHelper::getString(params, prefix + "resultBasename", m_resultBasename);
}

void TestBestsSameConditions::describe(QString type)
{
    farsa::AbstractTest::describe(type);

    Descriptor d = addTypeDescription(type, "Test bests with same conditions", "Tests the best individuals of all generations of a given seed and resets the random number generator before each generation, so that the random number sequence is the same for all tested individuals");
    d.describeInt("seed").def(1).limits(1, MaxInteger).help("The seed to test");
    d.describeString("resultBasename").def("testBestsSameConditionsS").help("The initial part of the name of the file to which fitness is saved", "The complete filename is <resultBasename><seed>.fit");
}

The configure() and describe() methods declare and read the two parameters needed by this example test. The test needs to know the value of the seed used during the evolution process (this is used to generate the name of the file with genotypes to test) and the name of the file to which results of the test are written.

:::C++
void TestBestsSameConditions::runTest()
{
    // Generating the name of the file from which we load genotypes
    const QString genomeFilename = "B0S" + QString::number(m_seed) + ".gen";
    component()->getGA()->loadGenotypes(genomeFilename);
    component()->getGA()->setSeed(m_seed);
    farsa::Logger::info(QString("TestBestsSameConditions - Loaded Genome for seed %1 from file %2").arg(m_seed).arg(genomeFilename));

    // Now generating the file to which we write results
    const QString outputFilename = m_resultBasename + QString::number(m_seed) + ".fit";
    QFile outputFile(outputFilename);
    if (!outputFile.open(QIODevice::WriteOnly | QIODevice::Truncate | QIODevice::Text)) {
        farsa::throwUserRuntimeError(QString("Cannot open file %1 for writing. Aborting").arg(outputFilename));
    }
    QTextStream outStream(&outputFile);

    // Now testing all individuals
    for (unsigned int i = 0; i < component()->getGA()->numLoadedGenotypes(); i++) {
        farsa::Logger::info(QString("TestBestsSameConditions - Start of the Test of Individual %1").arg(i));
        farsa::EvoRobotExperiment* exp = component()->getGA()->getEvoRobotExperiment();
        exp->setActivityPhase(farsa::EvoRobotExperiment::INTEST);
        exp->setNetParameters(component()->getGA()->getGenesForIndividual(i));
        // Resetting seed
        farsa::globalRNG->setSeed(m_seed);
        exp->initGeneration(i);
        exp->doAllTrialsForIndividual(0);
        exp->endGeneration(i);
        outStream << exp->getFitness() << "\n";
        farsa::Logger::info(QString("TestBestsSameConditions - End of the Test of Individual %1").arg(i));

        if (component()->getGA()->isStopped()) {
            break;
        }
    }

    farsa::Logger::info(QString("TestBestsSameConditions - Finished"));
}

The runTest() function is the core of the test class. The component() function of the AbstractTest class can be used to get the instance of the EvoRobotComponent class from which both the ga and the experiment can be retrieved: the former can be obtained using component()->getGA(), while the latter through the call component()->getGA()->getEvoRobotExperiment().

The function in the example first generates the name of the file with genotypes to read using the seed value and then reads the file using the Evoga::loadGenotypes() function. After that the file which will hold the results of the simulation is generated.

The last part of the function is a for cycle that sets up the test and runs it. As you can see there are direct calls to functions in the experiment like initGeneration(), doAllTrialsForIndividual() and endGeneration(). These are generally sufficient for generic tests like the one in this example. If you need a test that works with a specific experiment, you can call here all the experiment function (after casting the pointer returned by component()->getGA()->getEvoRobotExperiment() to the correct type).

The check on the value returned by component()->getGA()->isStopped() is needed to allow the user to interrupt a running test. It is possible to omit this check if the test runs quickly enough.

Custom tests in the configuration file

Once test components have been written as explained above, they should be defined in the configuration file so that they are listed in the Tests menu of total99 and can be executed in batch mode. An example configuration file is shown below (only the relevant portion of the file is shown):

:::INI

[TOTAL99]
mainComponent = Component

...

[Component]
type = EvoRobotComponent
testToRun = TEST:0

...

[Component/TEST:0]
type = TestOneBestIndividual
resultBasename = testS
seed = 100

Tests should be declared in groups named TEST:0, TEST:1, ... The testToRun parameter of the EvoRobotComponent component declares which test to run when total99 is executed in batch. If the parameter is missing, it defaults to the first test that is declared. All the tests that are declared this way in the configuration file will appear in the Tests menu. To execute a test in batch, make sure to set the testToRun parameter to the correct value, then use the runTest action (see this page for more information on how to start total99 in batch mode).