PySys 0.9.3 Release Notes
=========================
PySys has been written and tested against Python 2.7 and higher. Support
for Python 3.x is included, where the 2to3.py script is run on install
time. See installation notes below for more details.
What's new in this release
--------------------------
- Added Darwin as a supported platform.
- Added the maker tag to the pysysproject file to allow specifiying a
custom test maker class, e.g. to create specific run templates etc.
See the pysysproject.xml file in pysys-examples for more information.
- The make option to pysys.py now accepts the testcase directory to be
specified to a value other than the current working directory.
Dependencies
------------
Running on windows requires installation of the pywin32 extensions written
by Mark Hammond (http://sourceforge.net/projects/pywin32). Running the
manual tester on unix systems requires the tcl/tk libraries to be installed
on the host machine and the Python version to be compiled with tcl/tk
support.
Windows Binary Installation
---------------------------
Installation on windows via the binary distribution bundle is performed by
downloading the PySys-X.Y.Z.win32.exe installer executable and running.
Note that a common error on windows is in the execution of the post-install
script, where an error of the form below is reported;
close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr
If obtained, right click the installer executable in an explorer window,
and select "Troubleshoot Compatability". Select "Try recommended settings",
and then "Start the program ...".
Windows/Unix Source Installation
--------------------------------
To install from source on unix or windows systems you should download the
source archive and perform the following (use winrar or winzip to unpack
on windows);
$ tar zxvpf PySys-X.Y.Z.tar.gz
$ cd PySys-X.Y.Z
$ python setup.py build
$ python setup.py install
To install on both windows and unix systems you may need to have root
privileges on the machine.
The 'pysys.py' launcher
-----------------------
PySys installs a launcher script 'pysys.py' as part of the installation
process to facilitate the management and execution of testcases. On unix
systems the script is installed into the Python binary directory, e.g.
/usr/local/bin, and is hence on the default user's path. On windows systems
the script is installed into the Scripts directory of the Python
installation, e.g. c:\Python24\Scripts\pysys.py, which is not by default on
the user's path. To run on windows systems the Scripts directory of the
Python installation should be added to the user's path to allow direct
execution of the script.
After installation, to see the available options to the pysys.py script use
$ pysys.py --help
The script takes four main top level command line options to it, namely
'run', 'print', 'make' and 'clean', which are used to run a set of testcases,
print the meta data for a set of testcases, make a new testcase directory
structure, or clean all testcase output. For more information on the further
options available to each add --help after the top level option, e.g.
$ pysys.py run --help
Running the samples
-------------------
PySys has a set of basic examples to demonstrate it's use for running
automated and manual testcases. The samples are distributed in a unix line
ending friendly tar.gz archive, and a windows line ending friendly zip file.
To unpack the tests on unix systems use
$ tar zxvpf PySys-examples.X.Y.Z.tar.gz
$ cd pysys-examples
To run the testcases, after changing directory to the testcases location,
perform
$ pysys.py run
Release History
---------------
0.9.1 to 0.9.2
--------------
- The method getNextAvailableTCPPort has been added to the
pysys.basetest.BaseTest class to allow users to allocate server TCP ports
in a robust manner.
- The unix and windows process helpers have been updated to fix handle leaks
(defect #11 "ProcessMonitor leaks file handles"), and to delete the stdin
queue when processes go away.
0.9.0 to 0.9.1
--------------
- Fixed issue with the determination of the overall test outcome due to the
incorrect use of the inbuilt sorted() function. The issue meant the test
outcome list was not correctly sorted based on precedent, leading to the
incorrect determination of the overall test outcome.
- Fixed issue in the pysys.basetest on handling FileNotFoundExceptions in
the assert* methods. The exception was not being caught, leading to
subsequent asserts in the test class not being performed.
0.8.1 to 0.9.0
--------------
- The PySys framework has been updated to be compliant with conversion to
Python 3.x with the 2to3.py conversion script. Installation on Python 3.x
is now supported via the source distribution bundle, where the 2to3.py
script is run automatically at install time. See details below for
installing the source distribution. A binary distribution installer for
windows will be included in a later release.
- There are now separate 32 and 64 bit binary distribution installers for
windows.
- On failure of the assertLineCount method, the log output now contains the
returned number and requested condition (tracker #3045931)
- Each assert method now takes an "assertMessage" parameter to be written
to the log output on execution (tracker #3045924). See test
PySys_internal_053 in the example testcases for example usage.
- Added the JUnitXMLResultsWriter to log test results in Apache Ant JUnit
XML format (one output file per test per cycle). This is useful for
integration into Continuous Integration build systems, e.g. TeamCity. The
TextResultsWriter and XMLResultsWriter now support the outputDir property
to specify the location to write the output files. See the pysys-examples
pysysproject.xml file for more details.
- Added the ability to run suites of pyunit tests wrapped up as a single
PySys test. This capability is exposed through the PyUnitTest class
contained in the pysys.unit.pyunit module. See the pysys-examples pyunit
tests for example usage.
- Fix to the unix process helper to correctly set the working directory of
child processes in the fork and exec.
- When running tests in parallel, a value of zero given for the
-n|--threads option to the run task of the pysys.py launcher, will set
the number of threads to the number of available CPUs.
0.7.6 to 0.8.1
--------------
- Updated the pysys.process.plat-win32.helper.ProcessWrapper module to
eliminate the use of threads to collect the stdout and stderr from the
process via pipes. The module now directly uses win32file.CreateFile to
create file objects to pass to the call to win32process.CreateProcess.
- Added the <formatters/> element to the pysysproject file. This allows
setting the format of the test output to stdout and the runlog in
accordance to the format specifiers in the python logging and time
modules. For examples of the use of this element, see the pysysproject
file included in the PySys examples.
- Logging of exceptions and failed asserts has been changed from info to
warn level (tracker #2784251).
- Added extra debug logging in pysys.utils.filegrep, and pysys.basetest
for when performing asserts against a line count in an input file
(tracker #2824758).
- The testcase output summary is now printed on termination of the test
run via a keyboard interrupt (tracker #2816212).
- The PySys project file now allows assignment of the project root
location to a variable which can then be used for later expansion within
the file. This allows the definition of project variables to include the
full path where this is required, e.g. XSL stylesheets which must use
the full path to the file rather than a relative path etc. Note that
modules within PySys can reference the project root location directly
using PROJECT.root (tracker #2795316).
- The pysys.baserunner class now passes the -X arguments into the test
summary writer setup action to allow logging of the user supplied extra
arguments(tracker #2814499). The pysys-log.xsl stylesheet used by the
XMLResultsWriter
has been updated to display this information in the test summary display.
- Fixed an issue where when the pysysproject file was missing, defaults
for the runner module and the test output summary writer were not being
set.
0.7.5 to 0.7.6
--------------
- Fixed a defect in the unix process helper module to correct a file
handle leak in the write end of the stdin pipe.
0.6.1 to 0.7.5
--------------
- Added the ability to run tests concurrently through the -n | --threads
option to the pysys launcher run target. Tests to be run are placed on a
request queue and processed by the designated number of worker threads.
The results of each test are then placed on a result queue, collated and
displayed in the order in which they would run serially. Depending on
the nature of the application under test, the recommended number of
threads to designate when using this option is no more than two times
the number of CPUs. Note also that care needs to be made when running
tests in parallel, so as to ensure no shared resources are accessed
in a non-atomic way, e.g using direct references to os.environ() in one
test when another test modifies the environment directly etc.
- The constructor to the pysys.baserunner.BaseRunner class was changed to
include the threads parameter, i.e.
def __init__(self, record, purge, cycle, mode, threads, outsubdir, descriptors, xargs)
This parameter is required for the runner to create the required
threadpool before running a set of tests in parallel. Any custom runner
classes extending the base runner will need to be updated to incorporate
this change.
- Removed module specific loggers from pysys in order to support running
tests in parallel. There is now a single logger used within the
framework, and which can be referenced directly from the pysys.constants
module. Attached to this logger are two handler types; one for logging
to stdout, and one for logging to the run
log file saved in the output subdirectory of each test. The sdtout
handler is set to only log to stdout from the main thread, whilst the
run log file handlers are set to log to the output subdirectory of a
test only on the creating thread.
- Added exception handling to the pysys.process.user module when trying to
stop all processes on destruction. When a background process takes
longer to stop than the default timeout period, the thrown
pysys.exceptions.ProcessTimeout exception was uncaught causing abnormal
exit from the test run.
0.6.0 to 0.6.1
--------------
- The clean target has been updated to accept the -a | --all command line
option to allow deleting all derived files produced when running a set
of testcases, i.e. both the testcase output subdirectory and any
compiled testclass modules.
- The waitForSignal method of the ProcessUser class, subclassed by both
the BaseTestand BaseRunner classes, has been updated to return a list of
match objects on invocation. By using tagged regular expressions in the
expr parameter of the method call, this allows retrieval of portions of
the matched data e.g. to extract expressions in the file to use later in
the validation routines.
- All references to pysys.constants.TRUE and pysys.constants.FALSE have
been replaced by the native Python True and False literals. The values
of the constants have been set to True and False respectively so as to
maintain backwards compatibility.
0.5.2 to 0.6.0
--------------
- The PySys test and PySys project files have been renamed by default from
.pysystest to pysystest.xm, and .pysysproject to pysysproject.xml
respectively. Backwards compatability is maintained for the previous
file naming convention, though this will be deprecated in a later
release; it is strongly advised that the new naming convention is
adopted. New tests made using the PySys launcher will by default use the
new naming convention. This change was made due to issues on Windows
systems displaying and recognising hidden files, and files without
specified extensions e.g. within the Eclipse framework, for display in
internet browsers etc.
- The clean mode of operation has been added to the pysys.py launcher. This
allows removal of testcase output subdirectories, e.g. before importing
into a source code control system. The -o option allows specifying the
output subdirectory name to be deleted, which defaults to the platform
identifier if not specified.
- The test output summary writer interface has been changed so that the
test output is written and updated during the test execution; previously
a call to the writer was only made on completion of the test run. This
allows monitoring the test output summary during the test execution to
monitor the run time status of the tests.
- Added the XMLFileResultsWriter class to the pysys.writer module. This
performs logging of the test output summary in an XML format, suitable
for display via XLST in a web browser. A simple XSL stylesheet is
included with the PySys distribution to provide better display in
internet browsers.
- Added the ability to specifiy custom test output summary writers in the
PySys project file via the <writer> tag. For an example see the
.pysysproject file in the pysys-examples distribution. Should no
<writer> be specified in the project file, the default
XMLFileResultsWriter will be used. Multiple writers may be specified in
the PySys project file.
- Added exception logging on parsing errors in the PySys project file, e.g.
when the file in badly formed due to invalid XML tokens etc.
- Added variable argument passing to the process.monitor.ProcessMonitor
class constructor so that operating specific arguments can be passed
into the class on instantiation. The wrapper method
pysys.basetest.BaseTest.startProcessMonitor has also been updated to
allow pass through of the variable arguments.
- The win32 process.monitor module has been changed so that on windows
systems the percentage CPU usage is not normalised by default by the
number of available processors, e.g. on a 4 core processor if 2 cores
were fully utilized the CPU usage was previously output as 50% - the
change means that the reported usage will now be 200% (a value of 100%
indicates that one core is fully utilitised). This makes the output
consistent with that reported on unix systems. Should the
previous behavior be required the numProcessors argument can be passed
to the pysys.basetest.BaseTest.startProcessMonitor method in order to
normalise the CPU usage statistics by the number of processors. On
windows systems the number of processors can be obtained from the
NUM_PROCESSORS environment variable.
- Added comments to the PySys Project file distributed with the example
testcases, to detail the possible configuration options.
0.5.1 to 0.5.2
--------------
- The lastgrep method has been added to pysys.utils.filegrep, and the
assertLastGrep method has been added to the BaseTest class. This allows
test validation to be performed based on regular expression matching on
the last line of an input file to the assertLastGrep method.
- The win32 process monitor has been modified to calculate the percentage
CPU usage statistics as a sum over all available processors. A CPU usage
of 100% represents the process fully utilising all available processors.
- The win32 process monitor now also logs the handle count of a process.
0.5.0 to 0.5.1
--------------
- Fixed a bug in pysys.process.user.ProcessUser destructor to explicitly
set the process list to null to allow process handles to be cleaned up
on destruction. This bug only seemed to be exhibited when the process
handle of a process returned in the startProcess() method was set as a
data attribute to an instance of the class. This handle was then both a
data attribute of the class, and was contained in a list data attribute
of the class. Under these conditions the handles were not being released
correctly.
- The print mode of the pysys.py launcher now supports printing out the
test user defined modes, and the printing out of tests that can be run
in a given mode.
0.4.0 to 0.5.0
--------------
- The OSFAMILY constant has been added to pysys.constants, and takes the
value 'windows' on all win32 operating systems, and 'unix' on sunos and
linux operating systems. The value of the OSFAMILY can be used within
the .pysysproject file using the osfamily attribute to the <property>
element. This allows capturing the value to be used in expansion of
other properties defined within the project file; see below for an
example usage. Should no value be set in a properties file, a default
value of "osfamily" is assumed.
- The .pysysproject file now allows explicitly setting the environment
value to be used in expansions via the environment attribute to the
<property> element; see
below for an example usage. Should no value be set in a properties file,
a default value of "env" is assumed (this allows for backwards compatibility).
- The .pysysproject file now takes the file attribute to the <property>
element. This allows properties to be read from file, where the
properties are specified in the name=value syntax, e.g.
<pysysproject>
<property environment="env"/>
<property osfamily="osfamily"/>
<property file="${osfamily}.properties" />
<property name="lib" value="${library}_${osfamily}_${version}_${env.USER}.so"/>
</pysysproject>
where the property file contains the following;
version=1.0
library=jstore${version}.jar
For more details, see testcase PySys_internal_002 in the
pysys-examples/internal area which demonstrates this.
- Fixed the issue of removing zero size files from the output subdirectory
on win32 platforms; was due to the stderr and stdout file handles not
being properly closed down. Updated the BaseRunner to attempt to remove
the zero sized files 3 times to try to avoid race conditions of stopped
background processes holding on to the file handles too long before dying.
- The win32 process helper now ensures the environment in which the
process runs is converted to unicode to avoid issues encountered with
running under certain locales.
0.3.5 to 0.4.0
--------------
- The pysys.process.ProcessUser class has been added to define an
interface to subclasses which use the underlying process helper classes.
Both the BaseTest and BaseRunner classes now extend this so as to
provide a common interface for process manipulation. A common paradigm
for creating extension modules to PySys is to create a helper class
which provides the methods for starting an interacting with the
application under test (AUT). These helper classes have a call back to
an instance of the ProcessUser so that it can make use of the high level
process methods. As both the BaseTest and BaseRunner classes are
instances of the ProcessUser, the extension module helper classes can be
used in extensions to both of these to allow the AUT to be started both
within a testcase, and within the runner.
- The method signature to the pysys.utils.filereplace replace method has
been changed to set the default value for the marker to the empty string
- Bugs fixes for cleaning up leakage of threads from the process helpers,
and file handle leakage from the base runner classes.
0.3.4 to 0.3.5
--------------
- Fixed a bug a testcase was not being marked as BLOCKED when unable to
start a process using the process helper module.
- Failure on the assertOrderedGrep now prints out the line the failure
occurred on.
0.3.3 to 0.3.4
--------------
- Fixed a bug where timedout processes started in the foreground were not
being stopped automatically at the end of the testcase.
0.3.2 to 0.3.3
--------------
- The default name of the PySys test descriptor file has been changed from
"descriptor.xml", to ".pysystest". This change is to maintain a consistent
naming convention across configuration files within the framework, e.g.
the project file ".pysysproject" denotes the project root and project
specific information, whilst a test file ".pysystest" denotes a testcase
within the project, and contains meta data for the test. Support for the
previous name is maintained, though it should be noted that testcases
created with the 'pysys.py make' command will have the new naming
convention used.
- The windows installer has been updated to add shortcuts to the
uninstaller, and to create a separate directory for the inclusion of
project extensions.
- The getInstanceCount method has been added to the
pysys.basetest.BaseTest class to reference count the number of named
processes started during a test run. The startProcess method of the
class adds a reference count to an internal dictionary
structure keyed on the displayName passed into the method to achieve
this.
- The writeProcess method has been added to the pysys.basetest.BaseTest
class to provide a wrapper around the write method of the underlying
process helper class. This wrapper perform a check on the running status
of the process prior to the write, and performs additional logging to
the run.log to audit the write.
- Fixed a bug in the replace method of the filereplace module, where the
method signature was missing the marker parameter
- Added support to the pysys project file to allow adding path locations
to the Python path. See the .pysysproject file in pysys-examples for
more detail.
0.3.1 to 0.3.2
--------------
- Release was superceded immediately by the 0.3.3 release. See release
notes for new features for 0.3.3 for more information.
0.3.0 to 0.3.1
--------------
- The process helper modules have been updated to allow the writing to the
stdin of a process via the write() method on the process handle.
- Several bug fixes have been applied to the unix process helper module.
- The pysys-examples/internal directory has been added to the examples
module. This will contain internal testcases for self testing the pysys
framework. These have been included in the distribution as examples of
the use of the framework.
- The pysys project file has been added into the framework to allow the
setting of project specific constants within the application. The
project file should be written to the base location of the project, with
the filename .pysysproject. The location of this file denotes the root
location of the project. For an example of the file see
pysys-examples/.pysysproject. Any name value properties
within the file will be set as data attributes of the pysys.Project
class, which is referenced in the pysys.constants module using the
variable PROJECT.
0.2.2 to 0.3.0
--------------
- Updates to the epydoc output for documenting the classes and modules of
pysys
- Addition of the pysys.py module for printing, running and making new
testcase directory structures. This allows a single distributed script
to be used to perform all functionality available from the console.
- Remove of the run method from the console launch helper.
0.2.1 to 0.2.2
--------------
- The suites element in the test descriptor has been renamed to groups.
This is to allow testcases in a single directory to be classified as a
single testsuite, and subsets thereof to be regarded as a group
- Minor bug fixes to the manual tester and process module
0.2.0 to 0.2.1
--------------
- The Manual Tester has been updated to support the <expectedresult>
element in the input xml file. This allows display of the expected
result for a manual step to be presented in the user interface. The
ability to optionally record a defect in the log output is also now
included.
- The createDescriptors method has been removed from the
pysys.launcher.console package and moved into pysys.launcher. This
allows the utility method to be used for other custom launchers.
0.1.7 to 0.2.0
--------------
- This release includes updates to the Python doc strings for automated
generation of epydoc using the Epydoc package
(http://epydoc.sourceforge.net). The installer now distributes the
generated epydoc in the site-packages/pysys-doc directory. For
windows installs a link to the epydoc and release notes is now added as
a link in the start menu items
- Added the setup() method to the BaseTest class to allow custom setup
actions to be performed prior to execution of a particular test case
- Fixed a bug where if the --type option was not supplied to
ConsoleMakeTestHelper as a command line option, the resulting descriptor
had type="None"
0.1.6 to 0.1.7
--------------
- The Manual Tester UI has been updated so that it can be resized, and is
easier to navigate through the tests.
- The BaseRunner start method now takes an optional list of result writer
class instances to perform test audit logging at the end of a test
cycle. This allows custom result writers to be passed to the runner to,
for example, write the results to a database, proprietary system etc
0.1.5 to 0.1.6
--------------
- Added the ability to differentiate between automated and manual
testcases using the test attribute to the pysystest element in the
testcase descriptor. If the attribute is not present the test will be
assumed to be an automated test. The runTest and printTest launch
helpers allow you to differentiate between automated and manual
tests using the --type command line argument. For more information see
the examples in pysys-examples
0.1.4 to 0.1.5
--------------
- Added support for the requirements traceability. This includes printing
requirements covered by a set of testcases, and running testcases which
cover a particular requirement id
0.1.3 to 0.1.4
--------------
- Added the ConsoleMakeTestHelper class to pysys.launcher.console. This
facilitates the creation of new testcase structures from the command
line. Updated pysys-examples/fibonacci to demonstrate the use of the
utility class
Download Latest Version
PySys-1.3.0.tar.gz (111.6 kB)
