Recent changes to Test execution

WikiPage Test execution modified by Bartek Wilczek

Bartek Wilczek — Tue, 31 Jan 2012 18:40:43 -0000

--- v8 
+++ v9 
@@ -98,25 +98,26 @@
     test_logic_step3
 
 Listing above presents example Abc.rb file
-
+    
+    :::ruby
     # encoding: utf-8
     class GoogleSearchScenario < Bubik::Scenario
-	def run
-	    #GROUPS: demo, sremo, pipsko
-	    #DESC: Kaka demona!
+        def run
+            #GROUPS: demo, sremo, pipsko
+            #DESC: Kaka demona!
 
             log( 'LINE: test_logic_step1' )
             test_logic_step1
             log( 'LINE: test_logic_step2' )
             test_logic_step2
             log( 'LINE: test_logic_step3' )
             test_logic_step3
-	end	
-
-	def init
-		groups("a", "b", "c")
-		desc( "My fancy description" )
-	end
+        end	
+
+        def init
+            groups("a", "b", "c")
+            desc( "My fancy description" )
+        end
     end
 
 Listing above presents transformation output: AbcScenario.rb

WikiPage Test execution modified by Bartek Wilczek

Bartek Wilczek — Tue, 31 Jan 2012 18:38:47 -0000

--- v7 
+++ v8 
@@ -83,3 +83,40 @@
     def end_variant( duration, result, name, errors_cnt, exceptions_cnt )
 
 These are called at the start and end of each variant/test/execution allowing to perform any logic related to reporting or internal handler maintenance of objects used by tests while execution runs.
+
+### Is transformation Abc.rb to AbcScenario.rb necessary?
+
+No, it is not. Test can be created as AbcScenario.rb file and if Abc.rb file is not present the Scenario file will be used without any modifications. One have to remember however that execution logging will not be that detailed, and groups assignments and description definition have to be handled manually in 'init' method.
+
+This examples presents the transformation logic:
+
+    #GROUPS: a, b, c
+    #DESC: My fancy description
+
+    test_logic_step1
+    test_logic_step2
+    test_logic_step3
+
+Listing above presents example Abc.rb file
+
+    # encoding: utf-8
+    class GoogleSearchScenario < Bubik::Scenario
+	def run
+	    #GROUPS: demo, sremo, pipsko
+	    #DESC: Kaka demona!
+
+            log( 'LINE: test_logic_step1' )
+            test_logic_step1
+            log( 'LINE: test_logic_step2' )
+            test_logic_step2
+            log( 'LINE: test_logic_step3' )
+            test_logic_step3
+	end	
+
+	def init
+		groups("a", "b", "c")
+		desc( "My fancy description" )
+	end
+    end
+
+Listing above presents transformation output: AbcScenario.rb

WikiPage Test execution modified by Bartek Wilczek

Bartek Wilczek — Tue, 31 Jan 2012 18:03:11 -0000

--- v6 
+++ v7 
@@ -30,7 +30,7 @@
 
 When command is executed the following actions take place:
 
-- For each test located in current project file .test is transformed to .rb
+- For each test located in current project file Abc.rb is transformed to AbcScenario.rb
     - declaration of class extending Bubik::Scenario is added
     - comments #GROUPS: and #DESC: are transformed to body of 'init' method
     - lines defining test logic are transformed to body of 'run' method

WikiPage Test execution modified by Bartek Wilczek

Bartek Wilczek — Tue, 31 Jan 2012 08:01:17 -0000

--- v5 
+++ v6 
@@ -16,7 +16,7 @@
 **--name=** (alias: -n) - name of this execution (extra information to be presented in reports)
 
 Tests are assigned to groups using special comment #GROUPS: 
---tests or --groups switch is required (otherwise not tests will be executed)
+if neither --tests nor --groups switch is present all tests will be executed
 --reporters is optional, if not present 'console' will be used
 --env is optional
 --name is optional

WikiPage Test execution modified by Bartek Wilczek

Bartek Wilczek — Mon, 30 Jan 2012 20:46:49 -0000

--- v4 
+++ v5 
@@ -82,4 +82,4 @@
     def start_variant( name, env )
     def end_variant( duration, result, name, errors_cnt, exceptions_cnt )
 
-These are called at the start and end of each variant/test/execution allowing to perform any logic related to reporting or internal handler maintenance of object used by tests during execution time.
+These are called at the start and end of each variant/test/execution allowing to perform any logic related to reporting or internal handler maintenance of objects used by tests while execution runs.

WikiPage Test execution modified by Bartek Wilczek

Bartek Wilczek — Mon, 30 Jan 2012 20:45:42 -0000

--- v3 
+++ v4 
@@ -76,10 +76,10 @@
 Both reporters and handlers implement Bubik::Listener interface containing six methods:
 
     def start_exec( name = "" )
-    def end_exec( duration, result, name = "", all_test_cnt = 0, failed_test_cnt = 0, exception_test_cnt = 0 )
+    def end_exec( duration, result, name, all_test_cnt, failed_test_cnt, exception_test_cnt )
     def start_test( name )
-    def end_test( duration, result, name, all_variants_cnt = 0, failed_variants_cnt = 0, exception_variants_cnt = 0 )
-    def start_variant( name, env = "common" )
-    def end_variant( duration, result, name, errors_cnt = 0, exceptions_cnt = 0 )
+    def end_test( duration, result, name, all_variants_cnt, failed_variants_cnt, exception_variants_cnt )
+    def start_variant( name, env )
+    def end_variant( duration, result, name, errors_cnt, exceptions_cnt )
 
 These are called at the start and end of each variant/test/execution allowing to perform any logic related to reporting or internal handler maintenance of object used by tests during execution time.

WikiPage Test execution modified by Bartek Wilczek

Bartek Wilczek — Mon, 30 Jan 2012 20:44:08 -0000

--- v2 
+++ v3 
@@ -39,4 +39,47 @@
 - Given the criteria from --tests and --groups switches the instantiated tests are filtered out, or added to runner object
 - Other parameters from command lines (envs, reporters, name) are passed to runner object
 - runner.run method is called
- 
+
+### Execution, Test, Variant
+
+These three terms are crucial for understanding of handlers and reporters. Let's consider the following command run
+
+    bubik --tests=T1,T2 --env=pl,de --name="My Exec"
+
+There are two test and two environment set. Additionally each test has 2 parameter sets (2 lines in CSV file) named 'normal_params' and 'boundary_params'. When command is executed users can expect output similar to below:
+
+    exec start: My Exec
+        test start: T1
+            variant start: normal_params pl
+            variant end: normal_params pl, result: PASSED
+            variant start: boundary_params pl
+            variant end: boundary_params pl, result: PASSED
+            variant start: normal_params de
+            variant end: normal_params de, result: PASSED
+            variant start: boundary_params de
+            variant end: boundary_params de, result: PASSED
+        test end: T1, result: PASSED
+        test start: T2
+            variant start: normal_params pl
+            variant end: normal_params pl, result: PASSED
+            variant start: boundary_params pl
+            variant end: boundary_params pl, result: PASSED
+            variant start: normal_params de
+            variant end: normal_params de, result: PASSED
+            variant start: boundary_params de
+            variant end: boundary_params de, result: PASSED
+        test end: T2, result: PASSED
+    exec end: My Exec, result PASSED
+
+As it's presented on the listing above the hierarchy of entities that are subject to reporting is a tree of one EXECUTION that can consist of at least one TEST and each of them consists at least one VARIANT. Variant is a single test execution for given parameters set and environment.
+
+Both reporters and handlers implement Bubik::Listener interface containing six methods:
+
+    def start_exec( name = "" )
+    def end_exec( duration, result, name = "", all_test_cnt = 0, failed_test_cnt = 0, exception_test_cnt = 0 )
+    def start_test( name )
+    def end_test( duration, result, name, all_variants_cnt = 0, failed_variants_cnt = 0, exception_variants_cnt = 0 )
+    def start_variant( name, env = "common" )
+    def end_variant( duration, result, name, errors_cnt = 0, exceptions_cnt = 0 )
+
+These are called at the start and end of each variant/test/execution allowing to perform any logic related to reporting or internal handler maintenance of object used by tests during execution time.

WikiPage Test execution modified by Bartek Wilczek

Bartek Wilczek — Mon, 30 Jan 2012 20:23:04 -0000

--- v1 
+++ v2 
@@ -27,3 +27,16 @@
     bubik -tGoogleSearch,DemoTest -rdb -name="Daily regression" -epl
 
 ### Behind the scenes
+
+When command is executed the following actions take place:
+
+- For each test located in current project file .test is transformed to .rb
+    - declaration of class extending Bubik::Scenario is added
+    - comments #GROUPS: and #DESC: are transformed to body of 'init' method
+    - lines defining test logic are transformed to body of 'run' method
+    - instructions for execution logging are added
+- Each test is instantiated and initialized (so that groups assignments are loaded)
+- Given the criteria from --tests and --groups switches the instantiated tests are filtered out, or added to runner object
+- Other parameters from command lines (envs, reporters, name) are passed to runner object
+- runner.run method is called
+

WikiPage Test execution modified by Bartek Wilczek

Bartek Wilczek — Mon, 30 Jan 2012 20:06:34 -0000

Bubik tests are being executed from command line, using bubik script: # unix $ chmod a+x bubik $ bubik -t -g -e -r -n # windows C:\path\bubik> ruby bubik -t -g -e -r -n ### Command switches **--tests=** (alias: -t) - coma separated list of tests to be executed
**--groups=** (alias: -g) - coma separated list of test groups to be executed
**--reporters=** (alias: -r) - coma separated list of reporters present results
**--env=** (alias: -e) - coma separated list of environments to run tests in
**--name=** (alias: -n) - name of this execution (extra information to be presented in reports) Tests are assigned to groups using special comment #GROUPS: --tests or --groups switch is required (otherwise not tests will be executed) --reporters is optional, if not present 'console' will be used --env is optional --name is optional EXAMPLES: bubik --tests=GoogleSearch,DemoTest -rconsole bubik -gdemo -rconsole,db -epl bubik -tGoogleSearch,DemoTest -rdb -name="Daily regression" -epl ### Behind the scenes