PySys System Test Framework Activity
Brought to you by:
ben-spiller,
moraygrieve
Activity for PySys System Test Framework
-
Ben Spiller posted a comment on ticket #4
Replaced by https://github.com/pysys-test/pysys-test/issues/15
-
Inconsistent date format in process monitors
-
Moved to https://github.com/pysys-test/pysys-test/issues/6
-
filecopy module does not import pysys.exceptions
-
Not sure exactly what was seen; but things have moved on since 2009 so maybe fine now
-
OrderedGrep re-compiles regex every line
-
Yes re-compiling regex per line is not ideal. Will be fixed in upcoming 1.4.0 https://github.com/pysys-test/pysys-test/commit/559a420583dc669633574336eb3b3a96c48a6777
-
Test runner should highlight inconsistent tests
-
Not a clear enough description. There shouldn't be any inconsistent results
-
Installer cannot locate 64-bit Python installations
-
We're recommending use of standard pip/wheel installation in Python 1.4+
-
Support logging from background threads
-
Replaced by https://github.com/pysys-test/pysys-test/issues/5
-
Not sure if this is still valid; detectCore has a catch...warn handler around it now
-
quotePath in process/plat-win32/helper can get misled
-
Fix will be in 1.4.0
-
No hook for overiding the print summary logic in baserunner
-
create final tag (retrospectively) for 1.3.0 release (official release date 13 July 2018; includes setup.py fix for README)
-
sigh, have to delete tag again as needed a final (minor) setup.py change to make packaging work on windows
-
PySys System Test Framework released /pysys/1.3.0/pysys-release-README.txt
-
-
-
-
-
-
add short-term hack to allow README.rst to be located during bdist_wininst creation on windows and also sdist creation on linux, since it appears due to our non-standard directory layout the paths are different in both cases
-
create final tag (retrospectively) for 1.3.0 release (official release date 13 July 2018)
-
delete tag created wrongly
-
I noticed calling startProcessMonitor() was taking more than 2 minutes, and found the current implementation enumerates every single thread of every process, in a weird attempt to calculate "% Processor Time"; this is unnecessary as there's a Process performance counter for that, and also incorrect as it ignroes threads that the process may start after the process monitor was added;
-
fix a small problem introduced a couple of comits ago where writes of buffered output to stdout ended up going with the logger results in an extra "time INFO " prefix (and as a side-effect caused PYSYS_DISABLE_PRINT_REDIRECTOR to not work); this fix makes the buffered ouptut go through the exact same mechanism as the single threaded output, so safer and simpler
-
Fixed a really serious bug in which random log lines might not be written to run.log and/or stdout when running tests multi-threaded (as a result of an underlying python bug https://bugs.python.org/issue35185 - has been in pysys for ages/forever) - fixed with a workaround where we avoid calling add/removehandler for each test and use a single handler to multiplex with threadlocals; not only does this solve the problem but it ought also to be faster since it avoids grabbing process-global logg...
-
remove unused import ThreadedFileHandler
-
fix PySys_internal_091 print() test for linux utf-8 - interpretation of £ cant be easily verified in python 2 since its locale dependent, so only do that on python 3
-
minor improvement to internal testing - changing global os.environ is not safe (sometimes results in concurrency bugs when running in parallel; also means behaviour of later tests is affected by env changes made by earlier tests), so run that bit inside a nested testcase; from 002 simply remove as not used
-
rework colorama coloring support to avoid possibilty of infinite loop due to print() stdout redirection (shown by test 001); this is pretty robust now
-
inexperienced pysys users often use print() instead of self.log which 'appears' to work fine, but actually results in none of the information being stored in the run.log, making test failures extremely difficult to debug; now pysys will automatically redirect stdout print statements to the logger so they show up in run.log; I added an undocumented env var to disable this behaviour in case of the unlikely event that this causes problems for anyone (can remove this after it's had one or two rel...
-
Added pysys.writers.replaceIllegalXMLCharacters utility function, and use it to avoid XMLResultsWriter and JUnitXMLResultsWriter from generating invalid XML if run.log or outcome reason contained characters not permitted by XML (tested by 090);
-
add support for customizing the encoding used to write run.log files, and change the pysysexamples/pysysproject.xml file to use utf-8 for run.log files (existing pysys projects are not affected), which is a better default since it's the same across all test machines, and also allows pretty much any character to be encoded unlike the Windows default encoding which loses a lot of that information
-
add fileutils.deletedir and deprecate baserunner/writer.purgeDirectory since it's not great to have code copied+pasted in several places, better/more efficient to use standard shutil.rmtree than to roll our own custom implementation, and it should be up to the caller whether they want to ignore errors or not;
-
add support for statically configuring default file encodings for common file types e.g. <default-file-encoding pattern="*.xml" encoding="utf-8"/>, and do this for common utf-8 file types in the default project config; also make BaseTest.getDefaultFileEncoding() delegate to runner by default, so you can just override the runner one and have it affect both
-
add handy (+%d other failures) suffix to outcome reason to give a summary of how many other problems there were in addition to the reported outcome reason
-
Implement some significant (and btw, jolly tricky!) improvements to robustness of I18N character support, especially around logging, as some PySys users were experiencing loss of essential logging lines due to encoding issues. Specific changes:
-
fix PySys_internal_059 after stricter checking to avoid expr=None in r1039 (think the test probably never did what it was intended to until now)
-
add except block so that an exception in runner.cleanup/cycleCOmplete wouldn't prevent handleKbrdInt from successfully terminating the process
-
add assertion to provide a more helpful message if you forget to specify expr= when calling waitForSignal; people quite often seem to make that mistake (e.g. passing the expr as the second positional argument) and waitForSignal with expr='' is non-sensical so this assertion can't do any harm and should help mpeople avoid mnaking mistakes; also added example of a best-practice typical waitForSignal command to the pydoc
-
move baserunner cleanup() invocation into a finally block so it always executes even fi something else went wrong first, since failing to cleanup could be dangerous
-
improve error handling in runner - while debugging some failure modes recently I found several places where serious errors (e.g. failure to report a test outcome by a writer) might easily go unnoticed by the user, or be reported without information about the test id where the problem was caused; now exceptions from writers will always lead to a non-success exit code and a message at the end of the test run so usuers should always be aware if something went wrong that could lead to missing res...
-
print traceback if the console runtest gets an exception (e.g. from runner.start) since this should not happen unless something is seriously broken, and in that case you really need something to go on otherwise it's impossible to debug;
-
change version number to 1.3.1 and status to BETA, and add some logic to print the status (if not production) so it's easy to see if a non-final release of pysys is in use
-
fix PySys_internal_088 process test
-
Agreed. This is fixed in PySys 1.3.0 which allows you to configure one or more BaseSummaryResultsWriter subclasses in the configured set of writers. If no BaseSummaryResultsWriter subclasses are in the list of configured writers, the the default ConsoleSummaryResultsWriter is added. You can provide your own BaseSummaryResultsWriter in addition or instead of the default ConsoleSummaryResultsWriter.
-
Yes we should do that. The tricky bit is how to fix it without breaking existing projects. Do we need to add a project setting for it (woudl work, but feels like overkill to pollute the default project with that). Could maybe conditionalize on the requires-pysys version from the project, to allow older projects to work but newer ones to get the new behaviour? Suggestions welcome
-
I've fixed this (on a branch) at revision 1033. Expected to be in v1.3.1.
-
fix https://sourceforge.net/p/pysys/bugs/16/ - implement a simple and more reliable escaping scheme for startProcess() arguments on windows, and test coverage
-
reintroduce fix for cross-contamination of different run.py files - using exec() rather than the module import mechanism avoids unexpected interactions between symbols defined in different run.py files (and I suspect may also allow python to forget about testcases already executed in a way it couldn't when using the module import system); the tests show this approach does not regress existing behaviour and run.py source/line informatin is still available as it was before
-
back out exec() fix for namespacing issue for now, as this approach is causing some test failures
-
fix exec() to work with pysys3
-
fix race condition/bug in which symbols defined in one run.py could leak into another and introduce a dependency such that tests behave differently when run on their own (as a resut of all run.py modules having the same qualified name, "run") - to fix this, just use the standard exec() built-in (which is also a bit simpler) rather than the custom module importer which doesn't have any namespacing capability
-
add some more detailed doc for runner.testcomplete and cleanup after seeing some pysys users do some scary things that are not recommended liek using testComplete to checking test log files (which is too late to affect the outcome)
-
incldue traceback when reporting errors during cleanup functions as there's not much to go on otherwise
-
in response to user feedback, add a skipTest() method, since it apparently wasn't obvious to people how to skip tests and in the vacuum some hacky solutions were developing
-
add how to regarding log messages, based on some recent anti-patterns I saw in the userbase
-
define ENVSEPERATOR using os.pathsep rather than hardcoding; also I hope this will encourage people to use os.pathsep instead of that constant (which is mis-spelled :))
-
add IS_WINDOWS constant, as Windows is so different from other (Unix-y) OSes that a special-case is justified; I've noticed a lot of pysys users are doing error-prone matching against string literals (e.g. OSFAMILY=='windows', PLATFORM in ['win32','win64'] are common) or writing complex expressions using python's platform module; having a constant will reduce mistakes for these common cases;
-
add LIBRARY_PATH_ENV_VAR_NAME constant since small change like this make PySys easier to use, helping to abstract OS differences and make it easier to start processes with correct environment
-
fix v rare race condition in which newly created cycle dirs are deleted by the cycle=1 job - only affects multi-cycle and multi-threaded mode
-
ignoring python3 __pycache__ dirs when looking for testcases almost doubles the speed of walking over a big directory tree
-
in the recent python 2 fix to baserunner for unicode strings, downgrade gracefully to character replacement if there are any unrepresentable characters, since seeing something is a lot better than nothing
-
fix PySys_internal_085 on python2 to behave the same as python3 - pysys.out is expected to be in the default/preferred encoding
-
fix a small bug in the original design of the performance reproter API - it's really common to want to be able to access the runner to get at information such as what mode/config is in use; construct performance reporters after runner construction has completed to enable this; also needed to add **kwargs to performance reporter's constructor (this is generally a good idea for python APIs where subclassed constructors are likely) which we can use to pass in runner=; added a little hack (+test ...
-
experience suggests fractional latency numbers liek 0.0000123s are quite hard to read, so introduce NANO_SECONDS unit to give more easily readable output e.g. 10,230 ns
-
typo in test title
-
now we have testCycle available we can remove the previous hack, which caused rare failures (if hashes of different testobjs happened to collide); code is much neater now
-
add __str__ implementations for BaseTest and BaseRunner that provide a nice short human-friendly but unique (across multiple cycle) string for test objects and runners; could be useful for debugging purposes, and for naming externally created resources (e.g. docker images);
-
add missing docstring entry for basetest.descriptor
-
avoid internal flag __progressWritersEnabled getting shown up in xargs logged by testsummary.xml (it's not needed later, so no need to keep it in the xargs dict)
-
docstring change - fix some errors in perfreporter docstring, and add docuemntation for non-static/instance fields
-
add assert to give clearer error message if an invalid type is passed for a performance result
-
improve PySys_internal_065 perf reporter test to really test with integer, string and float inputs
-
Fix bug in which non-ASCII characters in test outcome reasons could prevent the test log being written to disk if executed in multi-threaded mode. Only affects Python 2. Added regresssion test 85 for this case
-
fix XMLResultsWriter and JUnitXMLResultsWriter to write XML with UTF-8 encoding (as almost all XML files do; this is also much more useful when aggregating test results than having files in whatever is the default on the OS where the test is running, e.g. iso8859-1 on windows); and also to put the encoding="utf-8" header into the XML files so it's explicit;
-
Added optional "errors" parameter to pycompat.openfile to allow easily enabling non-strict handling of unsupported characters for cases where it does not matter (e.g. logging file contents), without needing caller to special-case for python 2 where no encoding is performed and the parameter is not available in the open() method. ProcessUser.logFileContents() now uses errors='replace' since for diagnostic logging a best-effort approach to non-ASCII characters is most helpful.
-
startProcess was allowing ProcessError exceptions to be swallowed with only an INFO level message; for improved test robustenss and better outcome reasons, change to add BLOCKED outcome (and abort if configured to), unless ignoreExitStatus=True which indicates user didn't care about the process failing (since ignoreExitStatus=True if not overridden in project config this change preserves backwards compat, and since it's False in our example pysysproject config if gives safer behaviour for peo...
-
prepare release notes to accept new items for 1.3.1
-
change error handling when loading project - simplify logic to print python trace to stderr in all cases since it doesn't hurt and otherwise it's impossible to debug problems if your project XML has errors in it, since the exception messages (e.g. if you dont supply a required XML attribute there's often no clue in the exception message about which part of your config it was parsing when that happened)
-
create branch for post 1.3.0 minor fixes/improvements, pending review
-
create tag for final 1.3.0 release (dated 13th July)
-
delete 1.3.0 release tag as not used yet; will be replaced with final 1.3.0 tag with date 13th July 2018
-
set final 1.3.0 release date (now 13 July rather than 4 July, includes lastgrep info-level logging fix)
-
remove inadvertent info-level logging of contents in lastgrep
-
create tag for final PySys 1.3.0 release
-
update release version now 1.3 is ready to go
-
add support for the newly released python 3.7, since it works :)
-
last-minute testing shows were writing README.rst to the site packages dir with no pysys prefix; thats not great so for this release, put it into the pysys/ dir; when theres time to discuss we can find a better solution, probably by refactoring the source layout to match the conventions/expectations of distutils/setuptools which will give automatic correct handling of README.rst and other artifacts
-
add failonerror to build.xml so we notice if something fails; use longfile=posix for maximum compatibility as per ant recommendations (now we have some example paths >100 chars)
-
remove unused imports
-
print python version number in the usage screen, since in this new multi-python world it's really important to know what you're running (esp since you may need different pythons for different test projects)
-
non-functional refactoring change - move most of the pysys.py code out to console.py, which is closer to where the associated code for usage of run/make/clean/print is, and also is helpful given I've discovered users may need multiple copies of pysys.py for different python versions; also this change ensures that the pysys version that gets printed is really the pysys that is being used (in case the pysys.py on path and the site-packages used by your default python installation were out of sync)
-
add native line endings svn property
-
delete now all changes are on trunk
1 >
