Docutils: Documentation Utilities

On 2011-11-25, Martin Bless wrote:
>>What happens with filenames with umlauts? Try e.g.

>>  $ rst2xml demo-grüße.rst

>>(with an empty file demo-grüße.rst).



> C:\Users\marble\kannweg>rst2pseudoxml --version
> rst2pseudoxml-script.py (Docutils 0.9 [repository], Python 2.7.2, on
> win32)

> C:\Users\marble\kannweg>rst2pseudoxml dornröschen.rst
> IOError: [Errno 2] No such file or directory: u'dornr\xf7schen.rst'
> Unable to open source file for reading ('dornröschen.rst'). Exiting.


This is in the DOS box, I suppose?
It seems consistent with the error in recognizing the cyrillic image file.

* What happens with `rst2xml demo-grüße.rst` issued in the Win... IDE?

* What happens if you put::
    
    import sys
    print sys.argv
  
  in a file dornröschen.py and run
  
    python dornröschen.py --title=tüdelig
    
  in the DOS box?


* Does `dir` in the dos box show the correct file names?
  Does this depend on whether the files are made/renamed from some Windows
  program or from the DOS box?
  
   


> Hhm, how can I be sure I'm really using the latest version of
> Docutils?

checking the svn version number?

(But I did not change the relevant behaviour recently, only the test.)


Günter

[Guenter Milde] wrote & schrieb:

>* What happens with `rst2xml demo-grüße.rst` issued in the Win... IDE?
>* What happens if you put::
>* Does `dir` in the dos box show the correct file names?
>  Does this depend on [..]

I have sent an email / a PM with of lot of screenshots to your address
given here. A copy of the email is available here as well:

http://mbless.de/4us/typo3-oo2rest/07-docutils-bug-hunting/

If you need more just tell me.

Martin

-- 
http://mbless.de

cheers,
--John Hurst
--5 Fran Court, Glen Waverley, VIC 3150                      ~ ~~~&#:
-- ajh@...  +61 3 9803 9346              _..___  ---____@...__
--(mob 0407 569 292)                             |_____[_|_________[__]_
-- http://www.ajhurst.org/~ajh                    oo oo  oo O--O--O o=o
--
-- Death takes but one life - the real suffering is for those left behind.

On 2011-11-26, JohnHurst wrote:

Dear JohnHurst,

I read the mail list through the gmane interface and all
I can see here in my newsreader is:

> [-- Skipped Type: application/octet-stream, Filename: docutilsfail.out --]
> [-- Type: text/plain, Encoding: 7bit --]
> 
> cheers,
> --John Hurst
> 
...

It seems the attachment's type is not recognized, defaulting to "binary"
and the attachement lost somewhere on the path.

Günter

On 2011-11-26, JohnHurst wrote:

> [-- Skipped Type: application/octet-stream, Filename: docutilsfail.out --]

Peter Funk sent me a copy of the output, so I could narrow down the problem
to a different format of the error reporting, e.g.:

Output::

             Problems with "csv-table" directive path:
             (2, \'No such file or directory\').

vs. Expected::

             Problems with "csv-table" directive path:
             [Errno 2] No such file or directory: u\'bogus.csv\'.

Maybe the default behaviour of str(<someErroInstance>) changed?
	     
With my current SVN version and Python versions 2.4, 2.5, 2.7, and 3.2,
all tests pass.

Which Python and Docutils versions do you use?

Günter

[Martin Bless] wrote & schrieb:

>[Guenter Milde] wrote & schrieb:
>
>>* What happens with `rst2xml demo-grüße.rst` issued in the Win... IDE?
>>* What happens if you put::
>>* Does `dir` in the dos box show the correct file names?
>>  Does this depend on [..]
>
>I have sent an email / a PM with of lot of screenshots to your address
>given here. A copy of the email is available here as well:
>
>http://mbless.de/4us/typo3-oo2rest/07-docutils-bug-hunting/

Now there is this PDF as well:

http://mbless.de/4us/typo3-oo2rest/07-docutils-bug-hunting/Re_%20Test%20suite%20failure.pdf


Martin

-- 
http://mbless.de

[Guenter Milde] wrote & schrieb:


>I read the mail list through the gmane interface and all
>I can see here in my newsreader is:

Me too. I'm reading the list "gmane.text.docutils.user" via nntp from
news.gmane.org. The attachment is there and comes through fine.

-- 
http://mbless.de

Proposed interactive test:
>   import sys
>   print sys.stdout.encode
>   print "Grüße".encode(sys.stdout.encode)
>   print "Grüße".encode(<your Windows encoding>)


TESTING

Windows 7 64bit, Dosbox:


E:\kannweg>python
Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit
(Intel)] on win
32
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys
>>> "grüße"
'gr\x81\xe1e'

>>> print "grüße"
grüße

>>> print sys.stdin.encoding
cp850

>>> print sys.stdout.encoding
cp850

>>> print "grüße".encode(sys.stdout.encoding)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\Python27\lib\encodings\cp850.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeDecodeError: 'ascii' codec can't decode byte 0x81 in position
2: ordinal not in range(128)

>>> print "grüße".encode('cp1252')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\Python27\lib\encodings\cp1252.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
UnicodeDecodeError: 'ascii' codec can't decode byte 0x81 in position
2: ordinal not in range(128)

>>> print "grüße".encode('utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0x81 in position
2: ordinal not in range(128)

>>> type("grüße")
<type 'str'>
>>> print "grüße".encode('latin-1')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0x81 in position
2: ordinal not in range(128)

>>> print "grüße".decode('cp850')
grüße

>>> print "grüße".decode('cp1252')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\Python27\lib\encodings\cp1252.py", line 15, in decode
    return codecs.charmap_decode(input,errors,decoding_table)
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position
2: character maps to <undefined>

>>> print "grüße".decode('utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\Python27\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x81 in position 2:
invalid start byte
>>>


This is most important I think:

>>> print "grüße".decode('cp850')
grüße

>>> print u'ÄÖÜäöüß'
ÄÖÜäöüß

>>> print u'ÄÖÜäöüß'.encode('cp850')
ÄÖÜäöüß

>>> print u'ÄÖÜäöüß'.encode('cp1252')
-Í_õ÷³¯

>>> print u'ÄÖÜäöüß'.encode('latin-1')
-Í_õ÷³¯


Martin

-- 
http://mbless.de

On 2011-11-30, Martin Bless wrote:

> Proposed interactive test:
>>   import sys
>>   print sys.stdout.encode
>>   print "Grüße".encode(sys.stdout.encode)
>>   print "Grüße".encode(<your Windows encoding>)

>>>> import sys
>>>> "grüße"
> 'gr\x81\xe1e'

>>>> print "grüße"
> grüße

As long as input and output encoding are the same, a binary
string seems to work fine. 

Also visible is, that the ü is not "latin-1" encoded (where it would be 
\xfc).

>>>> print sys.stdin.encoding
> cp850

>>>> print sys.stdout.encoding
> cp850



>>>> print "grüße".encode(sys.stdout.encoding)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "D:\Python27\lib\encodings\cp850.py", line 12, in encode
>     return codecs.charmap_encode(input,errors,encoding_map)
> UnicodeDecodeError: 'ascii' codec can't decode byte 0x81 in position
> 2: ordinal not in range(128)

Sorry, I forgot an important detail -- unicode strings:

   print u"Grüße".encode(sys.stdout.encode)
   print u"Grüße".encode(<your Windows encoding>)

otherwise, .encode() will first .decode() with the ASCII codec, which, of
course fails.


>>>> print "grüße".decode('cp850')
> grüße

>>>> print "grüße".decode('cp1252')
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "D:\Python27\lib\encodings\cp1252.py", line 15, in decode
>     return codecs.charmap_decode(input,errors,decoding_table)
> UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position
> 2: character maps to <undefined>

>>>> print "grüße".decode('utf-8')
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "D:\Python27\lib\encodings\utf_8.py", line 16, in decode
>     return codecs.utf_8_decode(input, errors, True)
> UnicodeDecodeError: 'utf8' codec can't decode byte 0x81 in position 2:
> invalid start byte

This shows that at least in the interactive mode, the sys.stdin encoding is
indeed cp850.


> This is most important I think:

>>>> print "grüße".decode('cp850')
> grüße

>>>> print u'ÄÖÜäöüß'
> ÄÖÜäöüß

>>>> print u'ÄÖÜäöüß'.encode('cp850')
> ÄÖÜäöüß

>>>> print u'ÄÖÜäöüß'.encode('cp1252')
> -Í_õ÷³¯

>>>> print u'ÄÖÜäöüß'.encode('latin-1')
> -Í_õ÷³¯

... as well as the sys.stdout.encoding.

What is the result of

>>>> print u'ÄÖÜäöüß'.encode('ascii', 'xmlcharrefreplace')

?

If you write

   u'ÄÖÜäöüß'.encode('cp1252')

to a file and open it with notepad (or similar), does it come out right?

Günter
Günter

[Guenter Milde] wrote & schrieb:

>What is the result of
>
>>>>> print u'ÄÖÜäöüß'.encode('ascii', 'xmlcharrefreplace')

Dosbox = Commandline = Shell:

E:\kannweg>python
Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

>>> print u'ÄÖÜäöüß'.encode('ascii','xmlcharrefreplace')
ÄÖÜäöüß

>>> print u'ÄÖÜäöüß'.encode('ascii','backslashreplace')
\xc4\xd6\xdc\xe4\xf6\xfc\xdf

>>> f2 = file('cp1252.txt','wb')
>>> f2.write(u'ÄÖÜäöüß'.encode('cp1252'))
>>> f2.close()

Yes, this file has the ususal Windows encoding. If opened by a Windows
application the Umlaute are correct: ÄÖÜäöüß


>>> f2 = file('cp850.txt','wb')
>>> f2.write(u'ÄÖÜäöüß'.encode('cp850'))
>>> f2.close()

If opened by a Windows editor the data is interpreted as CP1252 which
shows up as: Ž™š„”?á
If I tell the editor explicitely the source is in CP850 the result
shows up ok: ÄÖÜäöüß.


At the commandline in a Dosbox it's just the other way round:

E:\kannweg>type cp850.txt
ÄÖÜäöüß

E:\kannweg>type cp1252.txt
-Í_õ÷³¯

Summary: File behaviour is just as expected.


>If you write
>
>   u'ÄÖÜäöüß'.encode('cp1252')
>
>to a file and open it with notepad (or similar), does it come out right?

Yes.

-- 
http://mbless.de

On 2011-11-30, Martin Bless wrote:
> [Guenter Milde] wrote & schrieb:

>>What is the result of

>>>>>> print u'ÄÖÜäöüß'.encode('ascii', 'xmlcharrefreplace')

> Dosbox = Commandline = Shell:

...

>>>> print u'ÄÖÜäöüß'.encode('ascii','backslashreplace')
> \xc4\xd6\xdc\xe4\xf6\xfc\xdf

Fine. (See how this differs from the binary string in the last round -
Unicode code points for the "extended Latin chars" match with the upper
half of "latin1" and (mostly) cp1252.)


...

> Summary: File behaviour is just as expected.

OK, so back to Docutils.

Please test interactively with an empty file test.txt:

  rst2html --title=Grüße test.txt test.html

  rst2html --title=Grüße --output-encoding=cp1252 test.txt test1252.html

  rst2html --title=Grüße --output-encoding=utf8 test.txt test8.html

  rst2html --title=Grüße --output-encoding=ascii test.txt testA.html
  
and try the resulting html files in a browser (look at the title bar) and in
text editor (look at the <title>).

Günter

[Guenter Milde] wrote & schrieb:

>Please test interactively with an empty file test.txt:
>
>  rst2html --title=Grüße test.txt test.html

<?xml version="1.0" encoding="utf-8" ?>
[...]
<title>Gr³¯e</title>


>
>  rst2html --title=Grüße --output-encoding=cp1252 test.txt test1252.html

<?xml version="1.0" encoding="cp1252" ?>
[...]
<title>Gr³&#9600;e</title>



>
>  rst2html --title=Grüße --output-encoding=utf8 test.txt test8.html

<?xml version="1.0" encoding="utf8" ?>
[...]
<title>Gr³¯e</title>



>
>  rst2html --title=Grüße --output-encoding=ascii test.txt testA.html

<?xml version="1.0" encoding="ascii" ?>
[...]
<title>Gr&#179;&#9600;e</title>

The above is copied from the texteditor (notepad++).
All four look the same in the browsers title line but are the same way
wrong.

Martin

-- 
http://mbless.de

On 2011-11-30, Martin Bless wrote:
> [Guenter Milde] wrote & schrieb:

>>Please test interactively with an empty file test.txt:

...

>>  rst2html --title=Grüße --output-encoding=cp1252 test.txt test1252.html

><?xml version="1.0" encoding="cp1252" ?>
> [...]
><title>Gr³&#9600;e</title>

>>  rst2html --title=Grüße --output-encoding=utf8 test.txt test8.html

><?xml version="1.0" encoding="utf8" ?>
> [...]
><title>Gr³¯e</title>

>>  rst2html --title=Grüße --output-encoding=ascii test.txt testA.html

><?xml version="1.0" encoding="ascii" ?>
> [...]
><title>Gr&#179;&#9600;e</title>

> The above is copied from the texteditor (notepad++).
> All four look the same in the browsers title line but are the same way
> wrong.

It looks like "double-encoding": 


ß: 
  225 (ß in cp850)  -> 223 (ß in latin1, cp1252, and Unicode)
  
  223 (upper half block in cp850) -> 9600 (▀ upper half block in Unicode)
  
  9600 -> &#9600; (char code in HTML, if the document encoding does not
       	  	   provide the character)
		   
  The strange thing is how this changes to
  Character '¯' (175, 0xAF) 00AF	MACRON
  in the utf8 encoded example (test8.html). 
  
  Did you do some manual substitution?
  Does the browser title show a macron or an upper half block?
  Maybe the mail system replaced it somehow -- what character is in
  the file on your system?
  
ü:   
  154 (ü in cp850)  -> 252 (ü in latin1, cp1252, and Unicode)
  
  252 (³ in cp850) -> 9179 (³ in Unicode)


What happens if you set a different code page in the dos box?
(see e.g. http://www.daniweb.com/software-development/python/threads/316301)

Günter

[Guenter Milde] wrote & schrieb:
>  Did you do some manual substitution?

No.



>  Does the browser title show a macron or an upper half block?

I'm trying to answer these and more questions here:
http://mbless.de/4us/typo3-oo2rest/07-docutils-bug-hunting/Docutils,%20Windows%207%2064bit,%20Encodings,%202011-12-01.pdf

Using screenshots again to avoid other sources of error.

The result files are there as well:
http://mbless.de/4us/typo3-oo2rest/07-docutils-bug-hunting/

>What happens if you set a different code page in the dos box?
>(see e.g. http://www.daniweb.com/software-development/python/threads/316301)

As stated in the PDF as well:
"""
It doesn't make sense to change the codepage of a Dosbox because then
there will be errors in other areas.

http://www.google.de/search?q=python+windows+unicode+filenames
finds this and more:

http://stackoverflow.com/questions/3789924/python-os-walk-and-japanese-filename-crash
Tenor: Changing cp doesn't work. And indeed, I can switch to Unicode,
but then next Enter closes the dosbox.

This seems to be an important page:
http://wiki.python.org/moin/PrintFails
 
Cool, it seems I can use an environment variable
"PYTHONIOENCODING=cp850:backslashreplace" while
"PYTHONIOENCODING=ascii:strict" is implicit.
"""

Martin

-- 
http://mbless.de

Hi Günter (Milde),

I think I have detangled the encoding puzzle concerning concerning
Windows and the reported error. Here is a summary of my findings:

* This works:
   import locale

Hello Günter,

I think I have detangled the encoding puzzle concerning Windows and
the reported error. Here is a summary of my findings:

Interactive Session 1 (Dosbox)
==============================
E:\kannweg>python
Python 2.7.2 (default, Jun 12 2011, 15:08:59)
 [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" 
for more information.

01 >>> import locale
02 
03 >>> locale.getlocale()
04 (None, None)
05 
06 >>> locale.getdefaultlocale()
07 ('de_DE', 'cp1252')
08 
09 >>> locale.setlocale(locale.LC_ALL,'')
10 'German_Germany.1252'
11 
12 >>> locale.getlocale()
13 ('de_DE', 'cp1252')


[01] importing locale works
[03] Initially no locale is set.
[09] http://docs.python.org/library/locale.html
""" According to POSIX, a program which has not called
setlocale(LC_ALL, '') runs using the portable 'C' locale. Calling
setlocale(LC_ALL, '') lets it use the default locale as defined by the
LANG variable..."""
[12] Now locale.getlocale() returns something != None as value

Worth noting: the default encoding is always "windows" = 'cp1252'



Interactive Session 2 (Dosbox)
==============================

01 E:\kannweg>echo "" >empty.py
02 E:\kannweg>python -i empty.py grüße
03 >>> import sys
04 >>> import locale
05 >>> locale_encoding = locale.getdefaultlocale()[1]
06 >>> locale_encoding
07 'cp1252'
08 
09 >>> sys.argv[1]
10 'gr\xfc\xdfe'
11 
12 >>> sys.argv[1].decode(locale_encoding)
13 u'gr\xfc\xdfe'
14 
15 >>> print sys.argv[1].decode(locale_encoding)
16 grüße
17 
18 >>> u'grüße'
19 u'gr\xfc\xdfe'
20 
21 >>> 'grüße'
22 'gr\x81\xe1e'
23 
24 >>> print 'grüße'
25 grüße
26 
27 >>> print u'grüße'.encode('cp850')
28 grüße
29 
30 >>> 'grüße'.decode('cp850')
31 u'gr\xfc\xdfe'
32 
33 >>> print 'grüße'.decode('cp850')
34 grüße
35 
36 >>> print sys.argv[1]
37 gr³¯e
38 
39 >>> sys.argv[1].decode('cp850')
40 u'gr\xb3\u2580e'
41 
42 >>> print sys.argv[1].decode('cp850')
43 gr³¯e
44
45 >>> sys.stdin.encoding
46 'cp850'
47
48 >>> sys.stdout.encoding
49 'cp850'


Conclusions:

[15] The commandline argument 'grüße' = sys.argv[1]
is of type 'str' and has 'cp1252' encoding (!!!)

[24,33] 'grüße' entered at the interactive prompt has
'cp850' = sys.stdin.encoding.

u'grüße' will be correct.

str('grüße') will print correctly if the encoding
of the string is == sys.stdout.encoding


Interactive Session 3 (PythonWin)
=================================

# cyrillic filename
>>> fname = u'\u043a\u0430\u0440\u0442\u0438\u043d\u0430.jpg'

# create file works
>>> file(fname,'w').close()

# open file works
>>> f1 = file(fname)
>>> f1.close()

Note: File operations (and os.listdir(upath)!) handle unicode
filenames if the filename parameter 'fname' or path name
'upath' are of type unicode.



Suspected error in test suite
=============================

# $Id: test_dependencies.py 7220 2011-11-11 10:31:48Z milde $
lines 38..39:
        record = docutils.io.FileInput(source_path=recordfile,
 encoding=sys.getfilesystemencoding())

I suspect 'sys.getfilesystemencoding()' is completely wrong here,
as it has the value "mbcs" which is somehow Pythons way
of characterizing the method that windows uses to encode FILENAMES.
It seems to me that the 'encoding' parameter is going to be
applied to the file content.


I haven't really understood the test suite yet so I can't come
up with a patch.

My very summary at the end is:
==============================
- decode commandline parameters this way:
  sys.argv[1].decode(locale.getlocale()[1] or
locale.getdefaultlocale()[1])

- working with unicode strings is great

- non unicode strings you want to print have to have
  encoding 'sys.stdout.encoding'

- to work with unicode fileNAMES several parameters
  need to be of type unicode


HTH - hoping this helps ...

Martin

-- 
http://mbless.de

[Martin Bless] wrote & schrieb:

>Hi Günter (Milde),
>
>I think I have detangled the encoding puzzle concerning concerning
>Windows and the reported error. Here is a summary of my findings:
>
>* This works:
>   import locale


Sorry, that wasn't intended to be sent.

On 2011-12-03, Martin Bless wrote:

> I think I have detangled the encoding puzzle concerning Windows and
> the reported error. Here is a summary of my findings:

> Interactive Session 1 (Dosbox)
...

> [03] Initially no locale is set.
...
> Worth noting: the default encoding is always "windows" = 'cp1252'

I suppose this depends on your general Windows settings (Language,
Encoding).


> Interactive Session 2 (Dosbox)
>==============================

> 01 E:\kannweg>echo "" >empty.py
> 02 E:\kannweg>python -i empty.py grüße

...


> 09 >>> sys.argv[1]
> 10 'gr\xfc\xdfe'
> 11 
> 12 >>> sys.argv[1].decode(locale_encoding)
> 13 u'gr\xfc\xdfe'
...
> 27 >>> print u'grüße'.encode('cp850')
> 28 grüße
...
> 36 >>> print sys.argv[1]
> 37 gr³¯e
...
> 45 >>> sys.stdin.encoding
> 46 'cp850'
> 48 >>> sys.stdout.encoding
> 49 'cp850'

So the command line is "seen" by Python as an cp1252-encoded string
even if the stdin/stdout encoding is cp850!

> Conclusions:

> [15] The commandline argument 'grüße' = sys.argv[1]
> is of type 'str' and has 'cp1252' encoding (!!!)

> str('grüße') will print correctly if the encoding
> of the string is == sys.stdout.encoding

-> decoding the command line (sys.argv) shall use
   docutils.frontend.locale_encoding and not sys.stdin.encoding!
   
> Interactive Session 3 (PythonWin)
>=================================

> # cyrillic filename
>>>> fname = u'\u043a\u0430\u0440\u0442\u0438\u043d\u0430.jpg'

> # create file works
>>>> file(fname,'w').close()

> # open file works
>>>> f1 = file(fname)
>>>> f1.close()

> Note: File operations (and os.listdir(upath)!) handle unicode
> filenames if the filename parameter 'fname' or path name
> 'upath' are of type unicode.


> Suspected error in test suite
>=============================

> # $Id: test_dependencies.py 7220 2011-11-11 10:31:48Z milde $
> lines 38..39:
>         record = docutils.io.FileInput(source_path=recordfile,
>  encoding=sys.getfilesystemencoding())

> I suspect 'sys.getfilesystemencoding()' is completely wrong here,
> as it has the value "mbcs" which is somehow Pythons way
> of characterizing the method that windows uses to encode FILENAMES.
> It seems to me that the 'encoding' parameter is going to be
> applied to the file content.

True. I will remove the `encoding=...` option. Does this solve problem?
(If not, please post the output of just running test_dependencies.py.)


> My very summary at the end is:
>==============================
> - decode commandline parameters this way:
>   sys.argv[1].decode(locale.getlocale()[1] or
> locale.getdefaultlocale()[1])

> - working with unicode strings is great

great pain as soon as you need I/O, if you have to care for Python 2 and
3 as well as several OSs with different encoding settings...

> - non unicode strings you want to print have to have
>   encoding 'sys.stdout.encoding'

This is done for error/diagnostic output.

However, converted documents shall be (and are) encoded in the encoding
specified by the "output-encoding" config setting - independent on the
target stream (be it a file, the console or a pipe to another program).


Thanks for the tests. I will change the command line decoding.

Günter

[Guenter Milde] wrote & schrieb:

>-> decoding the command line (sys.argv) shall use
>   docutils.frontend.locale_encoding and not sys.stdin.encoding!

[...]
>True. I will remove the `encoding=...` option. Does this solve problem?
>(If not, please post the output of just running test_dependencies.py.)


There is one error left in test_dependencies.py.
And one in test_heuristics_latin1.py


==============================
Docutils version used:
==============================
Revision: 7247
Author: milde
Date: Montag, 5. Dezember 2011 22:20:43
Message: argv_encoding != sys.stdin.encoding

Thanks to Martin Bless for the report and tests.
----
Modified : /trunk/docutils/docutils/core.py
Modified : /trunk/docutils/test/test_command_line.py


==============================
Test: alltests.py, Win 7 64bit
==============================

E:\kannweg>python alltests.py
Testing Docutils 0.9 [repository] with Python 2.7.2 on 2011-12-06 at
09:47:04
[...]
........................................................
======================================================================
FAIL: test_dependencies (test_dependencies.RecordDependenciesTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\Repositories\docutils\test\test_dependencies.py", line 52,
in test_dependencies
    u'\u043a\u0430\u0440\u0442\u0438\u043d\u0430.jpg'])
AssertionError: Lists differ: [u'data/include.txt', u'data/r... !=
['data/include.txt', 'data/raw...

First differing element 3:
???????.jpg
\u043a\u0430\u0440\u0442\u0438\u043d\u0430.jpg

- [u'data/include.txt', u'data/raw.txt', u'some_image.png',
u'???????.jpg']
+ ['data/include.txt',
+  'data/raw.txt',
+  'some_image.png',
+  u'\u043a\u0430\u0440\u0442\u0438\u043d\u0430.jpg']

======================================================================
FAIL: test_heuristics_latin1 (test_io.InputTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\Repositories\docutils\test\test_io.py", line 81, in
test_heuristics_latin1
    self.assertEqual(input.successful_encoding, 'latin-1')
AssertionError: 'cp1252' != 'latin-1'

----------------------------------------------------------------------
Ran 1176 tests in 16.232s

FAILED (failures=2)
Elapsed time: 32.541 seconds


Thank you for fixing this!

Martin

-- 
http://mbless.de

On 2011-12-06, Martin Bless wrote:
> [Guenter Milde] wrote & schrieb:

>>-> decoding the command line (sys.argv) shall use
>>   docutils.frontend.locale_encoding and not sys.stdin.encoding!

> [...]
>>True. I will remove the `encoding=...` option. Does this solve problem?
>>(If not, please post the output of just running test_dependencies.py.)


> There is one error left in test_dependencies.py.
> And one in test_heuristics_latin1.py


>==============================
> Docutils version used:
>==============================
> Revision: 7247
> Author: milde
> Date: Montag, 5. Dezember 2011 22:20:43
> Message: argv_encoding != sys.stdin.encoding

> Thanks to Martin Bless for the report and tests.
> ----
> Modified : /trunk/docutils/docutils/core.py
> Modified : /trunk/docutils/test/test_command_line.py


>==============================
> Test: alltests.py, Win 7 64bit
>==============================

> E:\kannweg>python alltests.py
> Testing Docutils 0.9 [repository] with Python 2.7.2 on 2011-12-06 at
> 09:47:04
> [...]
> ........................................................
>======================================================================
> FAIL: test_dependencies (test_dependencies.RecordDependenciesTests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "D:\Repositories\docutils\test\test_dependencies.py", line 52,
> in test_dependencies
>     u'\u043a\u0430\u0440\u0442\u0438\u043d\u0430.jpg'])
> AssertionError: Lists differ: [u'data/include.txt', u'data/r... !=
> ['data/include.txt', 'data/raw...

> First differing element 3:
> ???????.jpg
> \u043a\u0430\u0440\u0442\u0438\u043d\u0430.jpg

> - [u'data/include.txt', u'data/raw.txt', u'some_image.png',
> u'???????.jpg']
> + ['data/include.txt',
> +  'data/raw.txt',
> +  'some_image.png',
> +  u'\u043a\u0430\u0440\u0442\u0438\u043d\u0430.jpg']

I forgot an important point: The record file is generated during the
Doctutils run and stores the names of files required to generate the
output document. (See
http://docutils.sourceforge.net/docs/user/config.html#record-dependencies
) This is why its content is encoded in the file system encoding. 

In order to prevent encoding errors while writing this file but still
have some meaningful and reversible content, the encoding-error-handler
is set to xmlcharrefreplace in the DependencyList class in utils.py. The
test case needs to read the file with these settings and compare the
content to a list of unicode strings.

Does it work with the following patch?

--- test_dependencies.py	(Revision 7246)
+++ test_dependencies.py	(Arbeitskopie)
@@ -35,8 +35,10 @@
         docutils.core.publish_file(
             destination=DocutilsTestSupport.DevNull(), **settings)
         settings['settings_overrides']['record_dependencies'].close()
+        # The record file contains filenames in the local file system encoding
         record = docutils.io.FileInput(source_path=recordfile,
-                                       encoding='utf8')
+                                       encoding=sys.getfilesystemencoding(),
+                                       error_handler='xmlcharrefreplace')
         return record.read().splitlines()
 
     def test_dependencies(self):



>======================================================================
> FAIL: test_heuristics_latin1 (test_io.InputTests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "D:\Repositories\docutils\test\test_io.py", line 81, in
> test_heuristics_latin1
>     self.assertEqual(input.successful_encoding, 'latin-1')
> AssertionError: 'cp1252' != 'latin-1'

This is a false positive. The heuristics should try the locale encoding
(if it is specified) before the "last ressort" latin-1.

Fixet in Revision 7248.


Thanks

Günter

[Guenter Milde] wrote & schrieb:

>I forgot an important point: The record file is generated during the
>Doctutils run and stores the names of files required to generate the
>output document. (See
>http://docutils.sourceforge.net/docs/user/config.html#record-dependencies
>) This is why its content is encoded in the file system encoding. 

Ok, I understand the reasoning. But reality is different. Using the
right unicode filename in Python file functions is totally ok and
works as expected.

Therefore the following patch would make good code bad:

>Does it work with the following patch?
>
>--- test_dependencies.py	(Revision 7246)
>+++ test_dependencies.py	(Arbeitskopie)
>@@ -35,8 +35,10 @@
>         docutils.core.publish_file(
>             destination=DocutilsTestSupport.DevNull(), **settings)
>         settings['settings_overrides']['record_dependencies'].close()
>+        # The record file contains filenames in the local file system encoding
>         record = docutils.io.FileInput(source_path=recordfile,
>-                                       encoding='utf8')
>+                                       encoding=sys.getfilesystemencoding(),
>+                                       error_handler='xmlcharrefreplace')
>         return record.read().splitlines()
> 
>     def test_dependencies(self):

I did, I tried it, just to be sure: It doesn't resolve the error.

BTW: Yes, the other errors is gone, only one left!

I suspect, Python's name for the Windows filesystemencoding 'mbcs' ist
just an indicator.

http://de.wikipedia.org/wiki/Multibyte_Character_Set

http://docs.python.org/howto/unicode.html says:

"""
For example, Mac OS X uses UTF-8 while Windows uses a configurable
encoding; on Windows, Python uses the name “mbcs” to refer to whatever
the currently configured encoding is.
"""

The reason for the remaining error in test_dependencies.py is most
probably the the Python file functions ARE NOT call with unicode
parameters but with string parameters.

Using the cyrillic filename which is correctly shown in windows I get:

>>> import os
>>> os.listdir( '.' )
['???????.jpg']
>>> os.listdir( u'.' )
[u'\u043a\u0430\u0440\u0442\u0438\u043d\u0430.jpg']

http://docs.python.org/library/os.html

"""
os.listdir(path)

    Return a list containing the names of the entries in the directory
given by path. The list is in arbitrary order. It does not include the
special entries '.' and '..' even if they are present in the
directory.

    Availability: Unix, Windows.

    Changed in version 2.3: On Windows NT/2k/XP and Unix, if path is a
Unicode object, the result will be a list of Unicode objects.
Undecodable filenames will still be returned as string objects.
"""

Martin

-- 
http://mbless.de

On 2011-12-06, Martin Bless wrote:
> [Guenter Milde] wrote & schrieb:

>>I forgot an important point: The record file is generated during the
>>Doctutils run and stores the names of files required to generate the
>>output document. (See
>>http://docutils.sourceforge.net/docs/user/config.html#record-dependencies
>>) This is why its content is encoded in the file system encoding. 

> Ok, I understand the reasoning. But reality is different. Using the
> right unicode filename in Python file functions is totally ok and
> works as expected.

However, this record file is not (only) intended for Python. The existing
use case is the "make" utility. And to write a unicode string to a file you
need to agree on an encoding. At least on Linux, using
sys.filesystemencoding is sensible, because then the paths in the
record-file can be used by any program without re-encoding.

> Therefore the following patch would make good code bad:

We have to differentiate between the code and the test!
The patch does not change the "real" code (i.e. the behaviour of Docutils).
It only brings the test in line with that behaviour.

The "offending" test was done to make sure that bug [ 3434355 ] is fixed.
(prevent errors recording non-ASCII filenames). The failure with
non-utf8-Windows may point to a problem with the behavior or be just a
false positive.

The correct behaviour is currently discussed in the docutils-devel
list. I invite you to join the thread.


>>Does it work with the following patch?

> I did, I tried it, just to be sure: It doesn't resolve the error.

Schade. What is the actual error/traceback with this patch?

...

> I suspect, Python's name for the Windows filesystemencoding 'mbcs' ist
> just an indicator.

> http://de.wikipedia.org/wiki/Multibyte_Character_Set

> http://docs.python.org/howto/unicode.html says:

> """
> For example, Mac OS X uses UTF-8 while Windows uses a configurable
> encoding; on Windows, Python uses the name “mbcs” to refer to whatever
> the currently configured encoding is.
> """

Thanks for this information. This means that we might need a test for 
sys.getfilesystemencoding == 'mcbs' and use the locale_encoding in this case.

However, this is a change to be done in utils.py in the first place
(if the file-system-encoding is to be used for the record file).

> The reason for the remaining error in test_dependencies.py is most
> probably the the Python file functions ARE NOT call with unicode
> parameters but with string parameters.

test_dependencies.py is testing whether the dependencies are correctly
written to a "record file". This includes testing that the intended
encoding and encoding-error-handler are used.

Thanks,

Günter

======================================================================
ERROR: test_invalid_raw_xml (test_writers.test_docutils_xml.DocutilsXMLTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File 
"/home/dcarson/Downloads/docutils/test/test_writers/test_docutils_xml.py", line 
181, in test_invalid_raw_xml
    result = publish_xml(settings, invalid_raw_xml_source)
  File 
"/home/dcarson/Downloads/docutils/test/test_writers/test_docutils_xml.py", line 
127, in publish_xml
    settings_overrides=settings)
  File "/home/dcarson/Downloads/docutils/docutils/core.py", line 418, in 
publish_string
    enable_exit_status=enable_exit_status)
  File "/home/dcarson/Downloads/docutils/docutils/core.py", line 666, in 
publish_programmatically
    output = pub.publish(enable_exit_status=enable_exit_status)
  File "/home/dcarson/Downloads/docutils/docutils/core.py", line 223, in publish
    output = self.writer.write(self.document, self.destination)
  File "/home/dcarson/Downloads/docutils/docutils/writers/__init__.py", line 77, 
in write
    self.translate()
  File "/home/dcarson/Downloads/docutils/docutils/writers/docutils_xml.py", line 
61, in translate
    self.document.walkabout(visitor)
  File "/home/dcarson/Downloads/docutils/docutils/nodes.py", line 173, in 
walkabout
    if child.walkabout(visitor):
  File "/home/dcarson/Downloads/docutils/docutils/nodes.py", line 165, in 
walkabout
    visitor.dispatch_visit(self)
  File "/home/dcarson/Downloads/docutils/docutils/nodes.py", line 1611, in 
dispatch_visit
    return method(node)
  File "/home/dcarson/Downloads/docutils/docutils/writers/docutils_xml.py", line 
171, in visit_raw
    col_num, line_num, node.astext())
TypeError: %d format: a number is required, not NoneType

======================================================================
ERROR: test_publish (test_writers.test_docutils_xml.DocutilsXMLTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File 
"/home/dcarson/Downloads/docutils/test/test_writers/test_docutils_xml.py", line 
155, in test_publish
    result = publish_xml(settings, source)
  File 
"/home/dcarson/Downloads/docutils/test/test_writers/test_docutils_xml.py", line 
127, in publish_xml
    settings_overrides=settings)
  File "/home/dcarson/Downloads/docutils/docutils/core.py", line 418, in 
publish_string
    enable_exit_status=enable_exit_status)
  File "/home/dcarson/Downloads/docutils/docutils/core.py", line 666, in 
publish_programmatically
    output = pub.publish(enable_exit_status=enable_exit_status)
  File "/home/dcarson/Downloads/docutils/docutils/core.py", line 223, in publish
    output = self.writer.write(self.document, self.destination)
  File "/home/dcarson/Downloads/docutils/docutils/writers/__init__.py", line 77, 
in write
    self.translate()
  File "/home/dcarson/Downloads/docutils/docutils/writers/docutils_xml.py", line 
60, in translate
    self.visitor = visitor = self.translator_class(self.document)
  File "/home/dcarson/Downloads/docutils/docutils/writers/docutils_xml.py", line 
110, in __init__
    self.xmlparser.setContentHandler(self.the_handle)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/sax/expatreader.py", line 
128, in setContentHandler
    self._reset_cont_handler()
  File "/usr/lib64/python2.7/site-packages/_xmlplus/sax/expatreader.py", line 
234, in _reset_cont_handler
    self._cont_handler.processingInstruction
AttributeError: 'NoneType' object has no attribute 
'ProcessingInstructionHandler'

======================================================================
ERROR: test_publish_indents (test_writers.test_docutils_xml.DocutilsXMLTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File 
"/home/dcarson/Downloads/docutils/test/test_writers/test_docutils_xml.py", line 
161, in test_publish_indents
    result = publish_xml(settings, source)
  File 
"/home/dcarson/Downloads/docutils/test/test_writers/test_docutils_xml.py", line 
127, in publish_xml
    settings_overrides=settings)
  File "/home/dcarson/Downloads/docutils/docutils/core.py", line 418, in 
publish_string
    enable_exit_status=enable_exit_status)
  File "/home/dcarson/Downloads/docutils/docutils/core.py", line 666, in 
publish_programmatically
    output = pub.publish(enable_exit_status=enable_exit_status)
  File "/home/dcarson/Downloads/docutils/docutils/core.py", line 223, in publish
    output = self.writer.write(self.document, self.destination)
  File "/home/dcarson/Downloads/docutils/docutils/writers/__init__.py", line 77, 
in write
    self.translate()
  File "/home/dcarson/Downloads/docutils/docutils/writers/docutils_xml.py", line 
60, in translate
    self.visitor = visitor = self.translator_class(self.document)
  File "/home/dcarson/Downloads/docutils/docutils/writers/docutils_xml.py", line 
110, in __init__
    self.xmlparser.setContentHandler(self.the_handle)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/sax/expatreader.py", line 
128, in setContentHandler
    self._reset_cont_handler()
  File "/usr/lib64/python2.7/site-packages/_xmlplus/sax/expatreader.py", line 
234, in _reset_cont_handler
    self._cont_handler.processingInstruction
AttributeError: 'NoneType' object has no attribute 
'ProcessingInstructionHandler'

======================================================================
ERROR: test_publish_newlines 
(test_writers.test_docutils_xml.DocutilsXMLTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File 
"/home/dcarson/Downloads/docutils/test/test_writers/test_docutils_xml.py", line 
167, in test_publish_newlines
    result = publish_xml(settings, source)
  File 
"/home/dcarson/Downloads/docutils/test/test_writers/test_docutils_xml.py", line 
127, in publish_xml
    settings_overrides=settings)
  File "/home/dcarson/Downloads/docutils/docutils/core.py", line 418, in 
publish_string
    enable_exit_status=enable_exit_status)
  File "/home/dcarson/Downloads/docutils/docutils/core.py", line 666, in 
publish_programmatically
    output = pub.publish(enable_exit_status=enable_exit_status)
  File "/home/dcarson/Downloads/docutils/docutils/core.py", line 223, in publish
    output = self.writer.write(self.document, self.destination)
  File "/home/dcarson/Downloads/docutils/docutils/writers/__init__.py", line 77, 
in write
    self.translate()
  File "/home/dcarson/Downloads/docutils/docutils/writers/docutils_xml.py", line 
60, in translate
    self.visitor = visitor = self.translator_class(self.document)
  File "/home/dcarson/Downloads/docutils/docutils/writers/docutils_xml.py", line 
110, in __init__
    self.xmlparser.setContentHandler(self.the_handle)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/sax/expatreader.py", line 
128, in setContentHandler
    self._reset_cont_handler()
  File "/usr/lib64/python2.7/site-packages/_xmlplus/sax/expatreader.py", line 
234, in _reset_cont_handler
    self._cont_handler.processingInstruction
AttributeError: 'NoneType' object has no attribute 
'ProcessingInstructionHandler'

======================================================================
ERROR: test_raw_xml (test_writers.test_docutils_xml.DocutilsXMLTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File 
"/home/dcarson/Downloads/docutils/test/test_writers/test_docutils_xml.py", line 
172, in test_raw_xml
    result = publish_xml(self.settings, raw_xml_source)
  File 
"/home/dcarson/Downloads/docutils/test/test_writers/test_docutils_xml.py", line 
127, in publish_xml
    settings_overrides=settings)
  File "/home/dcarson/Downloads/docutils/docutils/core.py", line 418, in 
publish_string
    enable_exit_status=enable_exit_status)
  File "/home/dcarson/Downloads/docutils/docutils/core.py", line 666, in 
publish_programmatically
    output = pub.publish(enable_exit_status=enable_exit_status)
  File "/home/dcarson/Downloads/docutils/docutils/core.py", line 223, in publish
    output = self.writer.write(self.document, self.destination)
  File "/home/dcarson/Downloads/docutils/docutils/writers/__init__.py", line 77, 
in write
    self.translate()
  File "/home/dcarson/Downloads/docutils/docutils/writers/docutils_xml.py", line 
60, in translate
    self.visitor = visitor = self.translator_class(self.document)
  File "/home/dcarson/Downloads/docutils/docutils/writers/docutils_xml.py", line 
110, in __init__
    self.xmlparser.setContentHandler(self.the_handle)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/sax/expatreader.py", line 
128, in setContentHandler
    self._reset_cont_handler()
  File "/usr/lib64/python2.7/site-packages/_xmlplus/sax/expatreader.py", line 
234, in _reset_cont_handler
    self._cont_handler.processingInstruction
AttributeError: 'NoneType' object has no attribute 
'ProcessingInstructionHandler'

----------------------------------------------------------------------
Ran 1187 tests in 13.744s

FAILED (errors=5)

On 2012-03-22, David L  Carson wrote:
>======================================================================
> ERROR: test_invalid_raw_xml (test_writers.test_docutils_xml.DocutilsXMLTestCase)
> ----------------------------------------------------------------------

...

> ----------------------------------------------------------------------
> Ran 1187 tests in 13.744s

> FAILED (errors=5)

All test run fine here (Docutils 0.9-svn on Linux).

Could you please report the OS and Docutils version?


Günter