#2 Error when catching standard outputs in UTF-8

[Olivier Mansion]

NoseXUnit doesn't seem to handle correctly standard outputs in UTF-8.

Here is the produced traceback when trying to write Chinese chars :

27-Apr-2009 13:23:52 File "build/bdist.aix-5.3/egg/nosexunit/core.py", line 199, in writeXmlOnStream
27-Apr-2009 13:23:52 stream.write('<system-err><![CDATA[%s]]></system-err>' % self.stderr)
27-Apr-2009 13:23:52 UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 643629: ordinal not in range(128)

To solve the issue, we plan following actions :
1. Try to call decode on outputs with sys.[stdout|stderr].encoding
2. If fails, decode in UTF-8 by replacing unknown chars by '?'

[S�bastien Sabl�]

Hi Olivier,

here is a patch which should solve the problem.

See you soon!

[S�bastien Sabl�]

Index: nosexunit/core.py

--- nosexunit/core.py (révision 897)
+++ nosexunit/core.py (copie de travail)
@@ -4,6 +4,8 @@
import time
import logging
import traceback
+import codecs
+import StringIO

import nosexunit.const as nconst
import nosexunit.tools as ntools
@@ -14,19 +16,30 @@
class StdRecorder:
'''Class to capture the standard outputs'''

- def __init__(self):
+ def __init__(self, stream):
'''Initialize with an empty record'''
self.rec = False
- self.record = ''
+ try:
+ self.encoding = stream.encoding
+ except AttributeError:
+ self.encoding = None
+ if self.encoding == None:
+ self.encoding = 'utf-8'
+ self.save = stream
+ self.reset()

def __getattr__(self, attr):
'''Call the functions on the output'''
return getattr(self.save, attr)

- def write(self, string):
+ def write(self, value):
'''Write on the record and on the standard output'''
- if self.rec: self.record += string
- return self.save.write(string)
+ if self.rec:
+ val = value
+ if not isinstance(val, unicode):
+ val = val.decode(self.encoding, 'replace')
+ self.record.write(val)
+ return self.save.write(value)

def start(self):
'''Start to record the stream given by constructor'''
@@ -38,23 +51,22 @@

def content(self):
'''Return the content of the record'''
- return self.record
+ return self.record.getvalue()

def reset(self):
'''Reset the recorder'''
- self.record = ''
+ self.record = StringIO.StringIO()

def end(self):
'''End the recorder'''
self.stop()

-class StdOutRecoder(StdRecorder):
+class StdOutRecorder(StdRecorder):
'''Class to record the standard output'''

def __init__(self):
'''Replace the sys.stdout output by this one'''
- StdRecorder.__init__(self)
- self.save = sys.stdout
+ StdRecorder.__init__(self, sys.stdout)
sys.stdout = self

def end(self):
@@ -67,8 +79,7 @@

def __init__(self):
'''Replace the sys.stderr output by this one'''
- StdRecorder.__init__(self)
- self.save = sys.stderr
+ StdRecorder.__init__(self, sys.stderr)
sys.stderr = self

def end(self):
@@ -182,7 +193,7 @@
if self.getNbrTestsFromKinds([nconst.TEST_SUCCESS, nconst.TEST_FAIL, nconst.TEST_ERROR, ]) > 0:
xname = self.getXmlName(folder)
xpath = self.getXmlPath(folder, xname)
- xfile = open(xpath, 'w')
+ xfile = codecs.open(xpath, 'w', encoding='utf-8')
self.writeXmlOnStream(xfile, xname)
xfile.close()

Index: nosexunit/plugin.py

--- nosexunit/plugin.py (révision 897)
+++ nosexunit/plugin.py (copie de travail)
@@ -249,7 +249,7 @@
# Store the start time
self.start = None
# Get a STDOUT recorder
- self.stdout = ncore.StdOutRecoder()
+ self.stdout = ncore.StdOutRecorder()
# Get a STDERR recorder
self.stderr = ncore.StdErrRecorder()

[Olivier Mansion]

Hi,

Thank you for the patch, I included it!

I had issues with writing unicode strings on standard and error outputs, so I change a little bit the StdRecorder.write method :

if isinstance(value, unicode):
if self.rec: self.record.write(value)
self.save.write(value.encode(self.encoding, 'replace'))
else:
if self.rec: self.record.write(value.decode(self.encoding, 'replace'))
return self.save.write(value)

Let me know if it correct your problem.

A test was added in test_NoseXUnit/unit/test_core/test_unicode.py

See you!