Recent changes to Quality Guidelines

WikiPage Quality Guidelines modified by Andreas Mangold

Andreas Mangold — Thu, 03 Jan 2013 09:32:22 -0000

--- v6
+++ v7
@@ -7,7 +7,7 @@
 * __On first platform (Windows SAS 9.3 64 bit, OS settings English)__

     * __run example project test suite in overwrite-mode in English__
-Please note that the purpose of this project is the demonstration of SASUnit functionality to the end user (i.e. the SAS programmer who wants to test his programs) and to serve as a starting point for his own tests.
+Please note that the purpose of this project is the demonstration of SASUnit functionality to the end user (i.e. the SAS programmer who wants to test his programs) and to serve as a starting point for his own tests. When comparing boxplot results, please note that as always in report inspection there might be some small visual differences between the actual and the expected outcome. 

         * check test scenario page: 
             * All scenarios present? Check with scenario programs.
@@ -50,7 +50,7 @@
             * For each scenario
                 * All test cases per scenario present? Check with source code for each scenario!
                 * Log with exactly those errors and warnings expected from the various test cases? 
-                * Result as expected for each test case? Result must be red where indicated in test case description, otherwise green or white! 
+                * Result as expected for each test case? Result must be red where indicated in test case description, otherwise green or white! Test case 001 of scenario 007 (reportsasunit_emptyscn_test.sas) must fail although not stated in description.
                 * For each test case open test case detail page
                     * Result as expected for each test? Result must be red where indicated in test description, otherwise green (or white  for assertreport only)!
                     * Manually check for each occurrence of assertreport (white result)  whether results are as indicated in test description!

WikiPage Quality Guidelines modified by Andreas Mangold

Andreas Mangold — Thu, 03 Jan 2013 09:23:19 -0000

--- v5
+++ v6
@@ -44,7 +44,7 @@

         * check test scenario page: 
             * All scenarios present? Check with scenario programs.
-            * Result as expected for each scenario? Result must be red where indicated in test scenario description, otherwise green or white!
+            * Result as expected for each scenario? Result must be red where indicated in test scenario description, otherwise green or white! Test scenario number 001 must be red although not stated in the description. 
         * check test cases page
             * All scenarios present? Check with scenario programs. Every program ending in "_test.sas" in the saspgm/test folder is a scenario program. 
             * For each scenario
@@ -432,7 +432,7 @@
     * __test incremental build facility with example project__
         * Touch (save without changes) programs example/saspgm/nobs.sas and getvars_test.sas
         * Run example project test suite in non-overwrite-mode in English
-        * Check last run date/time on scenario overview page - must have been updated for nobs.sas and getvars.sas tests and unchanged for all other scenarios
+        * Check last run date/time on scenario overview page - must have been updated for nobs.sas and getvars.sas tests and unchanged for all other scenarios except for scenario tree1_test.sas which will always run.
         * Check generation date/time for test case page - must have been updated
         * Check generation date/time of a small sample of test case details pages of test cases for nobs.sas and getvars.sas - must have been updated
         * Check generation date/time of a small sample of test case details pages of other test cases - must be unchanged
@@ -471,3 +471,4 @@
         * __run doxygen for self test project__
             * compare to baseline
     *  "Compare to baseline" means: Compare the results on each platform to the results of the baseline (Windows 7 64 English / SAS 9.3 64 bit) above. Compare HTML reports using Firefox (latest version) under Microsoft Windows 7. 
+    * Please note that the sort order of test scenarios differs between Windows and Linux: Linux ignores the underscore when sorting scenario program names while windows doesn't.

WikiPage Quality Guidelines modified by Andreas Mangold

Andreas Mangold — Thu, 03 Jan 2013 08:57:09 -0000

--- v4
+++ v5
@@ -1,11 +1,14 @@
 Before releasing a new version, the following tests have to be run successfully. All deviances have to be documented.

+For all the following checks, use IE9, most current version, unless stated otherwise.
+
+For each run of SASUnit, make sure that all other SAS sessions have been close before. Warnings will appear in the logs otherwise. 
+           
 * __On first platform (Windows SAS 9.3 64 bit, OS settings English)__

     * __run example project test suite in overwrite-mode in English__
-       
-        * For all the following checks, use IE9, most current version, unless stated otherwise
-           
+Please note that the purpose of this project is the demonstration of SASUnit functionality to the end user (i.e. the SAS programmer who wants to test his programs) and to serve as a starting point for his own tests.
+
         * check test scenario page: 
             * All scenarios present? Check with scenario programs.
             * Result as expected for each scenario? Result must be red where indicated in test scenario description, otherwise green or white!
@@ -37,12 +40,13 @@
             * run_all.log must not contain any errors or warnings

     * __run self-test suite in overwrite-mode in English __
+Please note that the purpose of this test suite is to have a number of tests which check for correct functionality of SASUnit itself. This implies that some tests have to fail and that testing involves checking whether every test has the intended outcome. 

         * check test scenario page: 
             * All scenarios present? Check with scenario programs.
             * Result as expected for each scenario? Result must be red where indicated in test scenario description, otherwise green or white!
         * check test cases page
-            * All scenarios present? Check with scenario programs.
+            * All scenarios present? Check with scenario programs. Every program ending in "_test.sas" in the saspgm/test folder is a scenario program. 
             * For each scenario
                 * All test cases per scenario present? Check with source code for each scenario!
                 * Log with exactly those errors and warnings expected from the various test cases? 
@@ -152,26 +156,6 @@
   normal'>saspgm/test/pgmlib1
   saspgm/test/pgmlib2
-  
- 
- 
-  
-  Autoexec-program
-  for test scenarios
-  
-  
-  &g_autoexec
-  
-  
-  [empty]

WikiPage Quality Guidelines modified by Andreas Mangold

Andreas Mangold — Thu, 03 Jan 2013 08:38:23 -0000

--- v3
+++ v4
@@ -11,7 +11,7 @@
             * Result as expected for each scenario? Result must be red where indicated in test scenario description, otherwise green or white!
             * All scenario logs with exactly those errors and warnings expected from the various test cases? 
             * Links to test scenario details and programs functional? Check a small random sample!
-            * Correct popups on mouse hover for each column except duration? Check a small random sample!
+            * Correct tooltips on mouse hover for each column except duration? Check a small random sample!
             * Correct overall appearance? 
         * check test cases page
             * All scenarios present? Check with scenario programs.
@@ -19,17 +19,17 @@
             * Links to test case detail pages functional? Check for a small sample!
             * Links to units under test functional? Check for a small sample!
             * Links to logs functional? Check for a small sample!
-            * Correct popups on mouse hover for each column except duration? Check for a small sample!
+            * Correct tooltips on mouse hover for each column except duration? Check for a small sample!
             * Information and links on headers of test case details pages as in test cases page? Check for a small sample!
             * Correct overall appearance? 
         * check a sample of test case detail pages, at least two occurrences of every assertion type
         * Check page Units under Test 
-            * Popups on mouse over correct? 
+            * tooltips on mouse over correct? 
         * Check navigation tree 
             * Same scenarios, test cases and tests as on scenario report pages? Check with a sample!
-            * Proper popups on all levels of navigation tree below scenarios? Check with a sample!
+            * Proper tooltips on all levels of navigation tree below scenarios? Check with a sample!
             * Same units under test, test cases and tests as on units under test page? Check with a sample!
-            * Proper popups on all levels of navigation tree below units under test? Check with a sample!
+            * Proper tooltips on all levels of navigation tree below units under test? Check with a sample!
         * check main page of output: 
             * correct title with link to SF?
             * correct footnote with link to SF and correct version and build number?

WikiPage Quality Guidelines modified by Andreas Mangold

Andreas Mangold — Wed, 19 Dec 2012 11:01:44 -0000

--- v2
+++ v3
@@ -1,6 +1,6 @@
 Before releasing a new version, the following tests have to be run successfully. All deviances have to be documented.

-* __On first platform (Windows SAS 9.3 64 bit, OS settings German)__
+* __On first platform (Windows SAS 9.3 64 bit, OS settings English)__

     * __run example project test suite in overwrite-mode in English__

@@ -191,7 +191,7 @@
   none;border-bottom:solid #A3A3A3 1.0pt;border-right:solid #A3A3A3 1.0pt;
   padding:4.0pt 4.0pt 4.0pt 4.0pt'>
   bin/sasunit.9.3_64.windows.en.cfg
+  1.0pt;margin-left:0cm;line-height:normal'>bin/sasunit.9.3.windows.en.cfg
   
  
  
@@ -464,9 +464,9 @@
         * Open report of example project (Windows SAS 9.3 64 bit, OS settings German, SASUnit English) in Firefox (latest version) and compare with IE9
         * open doxygen for example project in Firefox (latest version) and compare with IE9

-* __On each of the following further platforms (in this order) repeat the following steps and compare to results of baseline (Windows 7 64 German / SAS 9.3 64 bit) above__
+* __On each of the following further platforms repeat the following steps__
     * __Platforms__
-        * __Windows 7 64 English / SAS 9.3 64 bit__
+        * __Windows 7 64 German / SAS 9.3 64 bit__
         * __Windows 7 64 English / SAS 9.3 32 bit__
         * __Windows 7 64 English / SAS 9.2 32 bit__
         * __Linux Ubuntu 12.04 LTS / SAS 9.3 64 bit__
@@ -486,3 +486,4 @@
             * compare to baseline
         * __run doxygen for self test project__
             * compare to baseline
+    *  "Compare to baseline" means: Compare the results on each platform to the results of the baseline (Windows 7 64 English / SAS 9.3 64 bit) above. Compare HTML reports using Firefox (latest version) under Microsoft Windows 7.

WikiPage Quality Guidelines modified by Andreas Mangold

Andreas Mangold — Mon, 17 Dec 2012 17:58:25 -0000

--- v1
+++ v2
@@ -0,0 +1,488 @@
+Before releasing a new version, the following tests have to be run successfully. All deviances have to be documented.
+
+* __On first platform (Windows SAS 9.3 64 bit, OS settings German)__
+   
+    * __run example project test suite in overwrite-mode in English__
+       
+        * For all the following checks, use IE9, most current version, unless stated otherwise
+           
+        * check test scenario page: 
+            * All scenarios present? Check with scenario programs.
+            * Result as expected for each scenario? Result must be red where indicated in test scenario description, otherwise green or white!
+            * All scenario logs with exactly those errors and warnings expected from the various test cases? 
+            * Links to test scenario details and programs functional? Check a small random sample!
+            * Correct popups on mouse hover for each column except duration? Check a small random sample!
+            * Correct overall appearance? 
+        * check test cases page
+            * All scenarios present? Check with scenario programs.
+            * Info and Links in header same as in scenario table? Check for a small sample!
+            * Links to test case detail pages functional? Check for a small sample!
+            * Links to units under test functional? Check for a small sample!
+            * Links to logs functional? Check for a small sample!
+            * Correct popups on mouse hover for each column except duration? Check for a small sample!
+            * Information and links on headers of test case details pages as in test cases page? Check for a small sample!
+            * Correct overall appearance? 
+        * check a sample of test case detail pages, at least two occurrences of every assertion type
+        * Check page Units under Test 
+            * Popups on mouse over correct? 
+        * Check navigation tree 
+            * Same scenarios, test cases and tests as on scenario report pages? Check with a sample!
+            * Proper popups on all levels of navigation tree below scenarios? Check with a sample!
+            * Same units under test, test cases and tests as on units under test page? Check with a sample!
+            * Proper popups on all levels of navigation tree below units under test? Check with a sample!
+        * check main page of output: 
+            * correct title with link to SF?
+            * correct footnote with link to SF and correct version and build number?
+            * correct overall appearance? 
+            * run_all.log must not contain any errors or warnings
+   
+    * __run self-test suite in overwrite-mode in English __
+       
+        * check test scenario page: 
+            * All scenarios present? Check with scenario programs.
+            * Result as expected for each scenario? Result must be red where indicated in test scenario description, otherwise green or white!
+        * check test cases page
+            * All scenarios present? Check with scenario programs.
+            * For each scenario
+                * All test cases per scenario present? Check with source code for each scenario!
+                * Log with exactly those errors and warnings expected from the various test cases? 
+                * Result as expected for each test case? Result must be red where indicated in test case description, otherwise green or white! 
+                * For each test case open test case detail page
+                    * Result as expected for each test? Result must be red where indicated in test description, otherwise green (or white  for assertreport only)!
+                    * Manually check for each occurrence of assertreport (white result)  whether results are as indicated in test description!
+        * Check page Units under Test 
+            * All units under test present, specified in at least one test scenario? Check with source code of test scenarios!
+            * All units under test correctly specified per program library and with test scenario? Check with source code of test scenarios!
+            * Results correct?
+        * Check main page of output, 
+            * run_all.log must not contain any errors or warnings
+            * must be as follows, all links must be functional:
+
+
+ 
+  
+  
+  
+ 
+ 
+  
+  
+  
+ 
+ 
+  
+  
+  
+ 
+ 
+  
+  
+  
+ 
+ 
+  
+  
+  
+ 
+ 
+  
+  
+  
+ 
+ 
+  
+  
+  
+ 
+ 
+  
+  
+  
+ 
+ 
+  
+  
+  
+ 
+ 
+  
+  
+  
+ 
+ 
+  
+  
+  
+ 
+ 
+  
+  
+  
+ 
+ 
+  
+  
+  
+ 
+ 
+  
+  
+  
+ 
+ 
+  
+  
+  
+ 
+ 
+  
+  
+  
+ 
+ 
+  
+  
+  
+ 
+ 
+  
+  
+  
+ 
+
+  Name
+  of project
+  
+  &g_project
+  
+  SASUnit
+  

+  Root
+  directory
+  
+  &g_root
+  
+  [check root
+  directory and link]
+  

+  Path
+  to test repository
+  
+  &g_target
+  
+  doc/sasunit/en
+  

+  Program
+  libraries
+  (macro
+  autocall paths)
+  
+  &g_sasautos
+  &g_sasautos1
+  &g_sasautos2
+  &g_sasautos3
+  
+  saspgm/sasunit
+  saspgm/test
+  saspgm/test/pgmlib1
+  saspgm/test/pgmlib2
+  

+  Autoexec-program
+  for test scenarios
+  
+  &g_autoexec
+  
+  [empty]
+  

+  SAS
+  configuration file for test scenarios
+  
+  &g_sascfg
+  
+  bin/sasunit.9.3_64.windows.en.cfg
+  

+  Folder
+  for test data
+  
+  &g_testdata
+  
+  dat
+  

+  Folder
+  for reference data
+  
+  &g_refdata
+  
+  dat
+  

+  Folder
+  for specification documents
+  
+  &g_doc
+  
+  doc/spec
+  

+  Path
+  to SASUnit macros
+  
+  &g_sasunit
+  
+  saspgm/sasunit
+  

+  SAS
+  log of reporting job
+  
+   
+  
+  doc/sasunit/en/run_all.log
+  

+  Platform
+  
+  &SYSCPL
+  
+  W64_7PRO
+  

+  SAS
+  Version
+  
+  &SYSVLONG4
+  
+  9.03[id
+  of maintenance release]
+  

+  User
+  ID
+  
+  &SYSUSERID
+  
+  [user
+  id of operator]
+  

+  SASUnit
+  Language
+  
+  SASUNIT_LANGUAGE
+  
+  en
+  

+  Number
+  of test scenarios
+  
+   
+  
+  [count]
+  

+  Number
+  of test cases
+  
+   
+  
+  [count]
+  

+  Number
+  of assertions
+  
+   
+  
+  [count]
+  
+
+* 
+       
+    * __run example project test suite in overwrite-mode in German__
+        * Compare to English version - differences must be only due to language and run date/time
+       
+    * __run self-test suite in overwrite-mode in German__
+        * Compare to English version - differences must be only due to language and run date/time
+       
+    * __test incremental build facility with example project__
+        * Touch (save without changes) programs example/saspgm/nobs.sas and getvars_test.sas
+        * Run example project test suite in non-overwrite-mode in English
+        * Check last run date/time on scenario overview page - must have been updated for nobs.sas and getvars.sas tests and unchanged for all other scenarios
+        * Check generation date/time for test case page - must have been updated
+        * Check generation date/time of a small sample of test case details pages of test cases for nobs.sas and getvars.sas - must have been updated
+        * Check generation date/time of a small sample of test case details pages of other test cases - must be unchanged
+           
+    * __run doxygen for example project__
+        * Check whether report has been created properly and all programs have been documented
+        * Check contents of navigation tree 
+        * Check complete documentation for a small sample of programs
+    * __run doxygen for self test project__
+        * Check whether report has been created properly and all programs have been documented
+           
+    * __Check rendering in Firefox (latest version)__
+        * Open report of example project (Windows SAS 9.3 64 bit, OS settings German, SASUnit English) in Firefox (latest version) and compare with IE9
+        * open doxygen for example project in Firefox (latest version) and compare with IE9
+   
+* __On each of the following further platforms (in this order) repeat the following steps and compare to results of baseline (Windows 7 64 German / SAS 9.3 64 bit) above__
+    * __Platforms__
+        * __Windows 7 64 English / SAS 9.3 64 bit__
+        * __Windows 7 64 English / SAS 9.3 32 bit__
+        * __Windows 7 64 English / SAS 9.2 32 bit__
+        * __Linux Ubuntu 12.04 LTS / SAS 9.3 64 bit__
+        * __Linux Suse 10.3 / SAS 9.2 32 bit__
+    * __Steps__
+        * __run example project test suite in overwrite-mode in English __
+            * compare to baseline: compare all main pages and a sample of the test case details pages
+        * __run self-test suite in overwrite-mode in English __
+            * compare to baseline: compare all main pages and a sample of the test case details pages
+        * __run example project test suite in overwrite-mode in German__
+            * compare to baseline: compare all main pages and a sample of the test case details pages
+        * __run self-test suite in overwrite-mode in German__
+            * compare to baseline: compare all main pages and a sample of the test case details pages
+        * __test incremental build facility with example project__
+            * same procedure as specified with baseline above
+        * __run doxygen for example project__
+            * compare to baseline
+        * __run doxygen for self test project__
+            * compare to baseline

WikiPage Quality Guidelines modified by Andreas Mangold

Andreas Mangold — Tue, 24 Jul 2012 11:53:00 -0000

+ Name + of project +	+ &g_project +	+ SASUnit +
+ Root + directory +	+ &g_root +	+ [check root + directory and link] +
+ Path + to test repository +	+ &g_target +	+ doc/sasunit/en +
+ Program + libraries + (macro + autocall paths) +	+ &g_sasautos + &g_sasautos1 + &g_sasautos2 + &g_sasautos3 +	+ saspgm/sasunit + saspgm/test + saspgm/test/pgmlib1 + saspgm/test/pgmlib2 +
+ Autoexec-program + for test scenarios +	+ &g_autoexec +	+ [empty] +
+ SAS + configuration file for test scenarios +	+ &g_sascfg +	+ bin/sasunit.9.3_64.windows.en.cfg +
+ Folder + for test data +	+ &g_testdata +	+ dat +
+ Folder + for reference data +	+ &g_refdata +	+ dat +
+ Folder + for specification documents +	+ &g_doc +	+ doc/spec +
+ Path + to SASUnit macros +	+ &g_sasunit +	+ saspgm/sasunit +
+ SAS + log of reporting job +	+ +	+ doc/sasunit/en/run_all.log +
+ Platform +	+ &SYSCPL +	+ W64_7PRO +
+ SAS + Version +	+ &SYSVLONG4 +	+ 9.03[id + of maintenance release] +
+ User + ID +	+ &SYSUSERID +	+ [user + id of operator] +
+ SASUnit + Language +	+ SASUNIT_LANGUAGE +	+ en +
+ Number + of test scenarios +	+ +	+ [count] +
+ Number + of test cases +	+ +	+ [count] +
+ Number + of assertions +	+ +	+ [count] +