Re: [Rest2web-develop] (pretty strange) encoding error with rest2web on a Mac

[Benoit <be...@ma...>]

Hello everybody

I've been investigating a bit deeper, from user point of view, the problem
I'm facing with this rest2web encoding issue on MacOSX.

I've created a single new file, using vim or textwrangler on MacOSX.
Here are the results.

1/ Latin1 file

- the file is encoded in Latin1 (confirmed by file(1)).
- in the file I've set:
    encoding: latin1
    output-encoding: utf-8

and when I run the Makefile, everything works fine (rest2web seems happy and
the HTML output looks great in Firefox)


2/ An UTF-8 file

- I've converted the exactly same file with iconv(1).
  iconv -f latin1 -t utf8 myfile.txt > myfile.txt.utf8
  mv myfile.txt.utf8 myfile.txt

- First attempt, I *DON'T* change the restindex:
        encoding: latin1  ## yes the file is now UTF-8!
        output-encoding: utf-8

=> results: rest2web seems happy and runs well, but the html output
has a double-encoding in UTF-8 (e.g. é -> Ã(c)).

- Second attempt, I fix the restindex:
    encoding: utf-8
    output-encoding: utf-8

=> results: rest2web stops with an encoding related error message.

$ make
rest2web version 0.5.1

[ skipping lots of files which are OK! ]

Processing "articles" directory.
Reading "articles/myfile.txt".
[err] Traceback (most recent call last):
[err]   File "/Users/benoit/local/rest2web-0.5.1/r2w.py", line 170, in
<module>
[err]     count = main(options, config)
[err]   File "/Users/benoit/local/rest2web-0.5.1/r2w.py", line 103, in main
[err]     return processor.walk()
[err]   File "/Users/benoit/local/rest2web-0.5.1/rest2web/restprocessor.py",
line 457, in walk
[err]     errorcheck = self.execute_safely(ProcessFile)
[err]   File "/Users/benoit/local/rest2web-0.5.1/rest2web/restprocessor.py",
line 218, in execute_safely
[err]     val =  function(*args, **keywargs)
[err]   File "/Users/benoit/local/rest2web-0.5.1/rest2web/restprocessor.py",
line 452, in ProcessFile
[err]     subdir=subdir)
[err]   File "/Users/benoit/local/rest2web-0.5.1/rest2web/restprocessor.py",
line 950, in process
[err]     doctitle=doctitle)
[err]   File "/Users/benoit/local/rest2web-0.5.1/rest2web/restutils.py",
line 183, in html_parts
[err]     writer_name='html', settings_overrides=overrides)
[err]   File "/Library/Python/2.5/site-packages/docutils/core.py", line 433,
in publish_parts
[err]     enable_exit_status=enable_exit_status)
[err]   File "/Library/Python/2.5/site-packages/docutils/core.py", line 614,
in publish_programmatically
[err]     output = pub.publish(enable_exit_status=enable_exit_status)
[err]   File "/Library/Python/2.5/site-packages/docutils/core.py", line 204,
in publish
[err]     self.settings)
[err]   File
"/Library/Python/2.5/site-packages/docutils/readers/__init__.py", line 68,
in read
[err]     self.input = self.source.read()
[err]   File "/Library/Python/2.5/site-packages/docutils/io.py", line 357,
in read
[err]     return self.decode(self.source)
[err]   File "/Library/Python/2.5/site-packages/docutils/io.py", line 124,
in decode
[err]     error_details))
[err] UnicodeError: Unable to decode input data.  Tried the following
encodings: 'utf-8'.
[err] (UnicodeDecodeError: 'utf8' codec can't decode bytes in position
1665-1666: invalid data)
[err]

Well, of course I  could use a latin1 encoding, but well, I would enjoy
utf-8!

Anyone a clue ?

Kind regards,

 -- Ben