[Pysys-commits] SF.net SVN: pysys:[1042] branches/post-1.3.0

[<ben...@us...>]

Revision: 1042
          http://sourceforge.net/p/pysys/code/1042
Author:   ben-spiller
Date:     2018-10-18 00:22:36 +0000 (Thu, 18 Oct 2018)
Log Message:
-----------
Implement some significant (and btw, jolly tricky!) improvements to robustness of I18N character support, especially around logging, as some PySys users were experiencing loss of essential logging lines due to encoding issues. Specific changes:
- implement fully safe and reliable logging of unicode strings (with ? replacements for any unsupported characters) that works regardless of what encoding is in use for stdout and run.log;
- fixed exception when logging unicode characters in Python 2 if a formatter was not configured in pysysproject.xml, by ensuring it is always stored as a unicode character string not a byte string (which used to happen in Python 2 if it was not mentioned in hte project config); ensures consistent (+sane +testable!) behaviour whether you've customized the format string or not; also added an assertion to check this assumption is never violated
- fixed logFileContents to more robustly handle files containing I18N/non-ASCII characters
- add runLogOutput parameter to processResult() method of BaseResultsWriter API so that writers such as the JUnitXMLResultsWriter can include the test output with no loss of unicode character information (and without relying on error-prone manual reading of run.log);
- introduced (non-public) _UnicodeSafeStreamWrapper to perform the complex gymnastics required to ensure all of the possible combinations of logging unicode characters and binary bytes in python 2 and 3 result in successful logging (with replacement characters if needed), regardless of the stdout encoding, getpreferredencoding and whether stdout is redirected or not; 
- add (non-public) _updateUnderlyingStream() method to ThreadedStreamHandler and streamFactory pattern to keep the slightly complex logic needed to handle reinitialization after a coloring formatter such as colorama rewrites sys.stdout contained to the ThreadedStreamHandler that originally set it up; 
added PySys_internal_086 to test this (which ended up having to be pretty complicated as there are a _lot_ of very tricky edge cases);
- change runner to use ThreadedStreamHandler not ThreadedFileHandler since the latter isn't really doing anything useful and we need more direct control over the stream to ensure we degrade gracefully if there are unprintable characters (ThreadedFileHandler is now redundant)
- add @deprecated and @undocumented for ThreadedStreamHandler/ThreadedFileHandler since they're not useful for PySys users (I'm certain no-one will be using them) and we don't really want to be tied to keeping them there and with same API - doing this will allow us to remove or change them in some future release but I'm not doing that now, to be very conservative (even though ThreadedFileHandler is actually no longer needed)
- made self.testFileHandlerStdoutBuffer handling more robust - StringIO can be used with characters or bytes but gives exceptions if you try to mix them, so to avoid nasty bugs where logging output disappears we need to consistently do the same thing (unicode characters) 
- release note updates

Modified Paths:
--------------
    branches/post-1.3.0/pysys/__init__.py
    branches/post-1.3.0/pysys/baserunner.py
    branches/post-1.3.0/pysys/constants.py
    branches/post-1.3.0/pysys/process/user.py
    branches/post-1.3.0/pysys/utils/logutils.py
    branches/post-1.3.0/pysys/utils/pycompat.py
    branches/post-1.3.0/pysys/writer/__init__.py
    branches/post-1.3.0/pysys-dist/pysys-release.txt
    branches/post-1.3.0/pysys-examples/internal/testcases/PySys_internal_068/run.py
    branches/post-1.3.0/pysys-examples/internal/utilities/resources/runpysys.py

Added Paths:
-----------
    branches/post-1.3.0/pysys-examples/internal/testcases/PySys_internal_086/
    branches/post-1.3.0/pysys-examples/internal/testcases/PySys_internal_086/Input/
    branches/post-1.3.0/pysys-examples/internal/testcases/PySys_internal_086/Input/NestedFail/
    branches/post-1.3.0/pysys-examples/internal/testcases/PySys_internal_086/Input/NestedFail/Input/
    branches/post-1.3.0/pysys-examples/internal/testcases/PySys_internal_086/Input/NestedFail/pysystest.xml
    branches/post-1.3.0/pysys-examples/internal/testcases/PySys_internal_086/Input/NestedFail/run.py
    branches/post-1.3.0/pysys-examples/internal/testcases/PySys_internal_086/Input/mypkg/
    branches/post-1.3.0/pysys-examples/internal/testcases/PySys_internal_086/Input/mypkg/__init__.py
    branches/post-1.3.0/pysys-examples/internal/testcases/PySys_internal_086/Input/mypkg/customfmt.py
    branches/post-1.3.0/pysys-examples/internal/testcases/PySys_internal_086/Input/pysysproject.xml
    branches/post-1.3.0/pysys-examples/internal/testcases/PySys_internal_086/Reference/
    branches/post-1.3.0/pysys-examples/internal/testcases/PySys_internal_086/Reference/pysys.out
    branches/post-1.3.0/pysys-examples/internal/testcases/PySys_internal_086/pysystest.xml
    branches/post-1.3.0/pysys-examples/internal/testcases/PySys_internal_086/run.py

This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.