Test execution

Bubik tests are being executed from command line, using bubik script:

# unix
$ chmod a+x bubik
$ bubik -t<Tests> -g<groups> -e<envs> -r<reporters> -n<name>

# windows
C:\path\bubik> ruby bubik -t<Tests> -g<groups> -e<envs> -r<reporters> -n<name>

Command switches

--tests= (alias: -t) - coma separated list of tests to be executed

--groups= (alias: -g) - coma separated list of test groups to be executed

--reporters= (alias: -r) - coma separated list of reporters present results

--env= (alias: -e) - coma separated list of environments to run tests in

--name= (alias: -n) - name of this execution (extra information to be presented in reports)

Tests are assigned to groups using special comment #GROUPS:
if neither --tests nor --groups switch is present all tests will be executed
--reporters is optional, if not present 'console' will be used
--env is optional
--name is optional

EXAMPLES:
bubik --tests=GoogleSearch,DemoTest -rconsole 
bubik -gdemo -rconsole,db -epl
bubik -tGoogleSearch,DemoTest -rdb -name="Daily regression" -epl

Behind the scenes

When command is executed the following actions take place:

• For each test located in current project file Abc.rb is transformed to AbcScenario.rb
  • declaration of class extending Bubik::Scenario is added
  • comments #GROUPS: and #DESC: are transformed to body of 'init' method
  • lines defining test logic are transformed to body of 'run' method
  • instructions for execution logging are added
• Each test is instantiated and initialized (so that groups assignments are loaded)
• Given the criteria from --tests and --groups switches the instantiated tests are filtered out, or added to runner object
• Other parameters from command lines (envs, reporters, name) are passed to runner object
• runner.run method is called

Execution, Test, Variant

These three terms are crucial for understanding of handlers and reporters. Let's consider the following command run

bubik --tests=T1,T2 --env=pl,de --name="My Exec"

There are two test and two environment set. Additionally each test has 2 parameter sets (2 lines in CSV file) named 'normal_params' and 'boundary_params'. When command is executed users can expect output similar to below:

exec start: My Exec
    test start: T1
        variant start: normal_params pl
        variant end: normal_params pl, result: PASSED
        variant start: boundary_params pl
        variant end: boundary_params pl, result: PASSED
        variant start: normal_params de
        variant end: normal_params de, result: PASSED
        variant start: boundary_params de
        variant end: boundary_params de, result: PASSED
    test end: T1, result: PASSED
    test start: T2
        variant start: normal_params pl
        variant end: normal_params pl, result: PASSED
        variant start: boundary_params pl
        variant end: boundary_params pl, result: PASSED
        variant start: normal_params de
        variant end: normal_params de, result: PASSED
        variant start: boundary_params de
        variant end: boundary_params de, result: PASSED
    test end: T2, result: PASSED
exec end: My Exec, result PASSED

As it's presented on the listing above the hierarchy of entities that are subject to reporting is a tree of one EXECUTION that can consist of at least one TEST and each of them consists at least one VARIANT. Variant is a single test execution for given parameters set and environment.

Both reporters and handlers implement Bubik::Listener interface containing six methods:

def start_exec( name = "" )
def end_exec( duration, result, name, all_test_cnt, failed_test_cnt, exception_test_cnt )
def start_test( name )
def end_test( duration, result, name, all_variants_cnt, failed_variants_cnt, exception_variants_cnt )
def start_variant( name, env )
def end_variant( duration, result, name, errors_cnt, exceptions_cnt )

These are called at the start and end of each variant/test/execution allowing to perform any logic related to reporting or internal handler maintenance of objects used by tests while execution runs.

Is transformation Abc.rb to AbcScenario.rb necessary?

No, it is not. Test can be created as AbcScenario.rb file and if Abc.rb file is not present the Scenario file will be used without any modifications. One have to remember however that execution logging will not be that detailed, and groups assignments and description definition have to be handled manually in 'init' method.

This examples presents the transformation logic:

#GROUPS: a, b, c
#DESC: My fancy description

test_logic_step1
test_logic_step2
test_logic_step3

Listing above presents example Abc.rb file

# encoding: utf-8
class GoogleSearchScenario < Bubik::Scenario
    def run
        #GROUPS: demo, sremo, pipsko
        #DESC: Kaka demona!

        log( 'LINE: test_logic_step1' )
        test_logic_step1
        log( 'LINE: test_logic_step2' )
        test_logic_step2
        log( 'LINE: test_logic_step3' )
        test_logic_step3
    end

    def init
        groups("a", "b", "c")
        desc( "My fancy description" )
    end
end

Listing above presents transformation output: AbcScenario.rb