docutils-users

[Docutils-users] Migration docutils 0.12 to 0.13

[Jean B. F. <web...@jb...>]

Hello,
I'm facing an issue with Trafficserver documentation [1] which doesn't
build with docutils 0.13.1

Error comes from a custom Inliner class which is defined in doc/conf.py
of Trafficserver project(starting line 163).
This allow to transform text like "(TS-XXXX)" in a link to
trafficserver's Jira

This custom Inliner used to work with docutils 0.12. It fails on 0.13.1
with following error:

  Exception occurred:
    File "conf.py", line 185, in __init__
      start_string_prefix=self.start_string_prefix,
  AttributeError: Inliner instance has no attribute 'start_string_prefix'

Since docutils 0.13, start_string_prefix isn't statically defined. We
have to call init_customiations for that.

First problem: one  must pass settings param when calling
init_customizations, but I can't find any format or structure for it.

Second problem: I tried a workaround, creating a InlinerSettings class
with minimal properties so that init_customizations could pass:

  class InlinerSettings:
    character_level_inline_markup=None
    pep_references=None
    rfc_references=None

But, then, I faced another error:

  Exception occurred:
    File
"/usr/lib/python2.7/dist-packages/docutils/parsers/rst/states.py", line
530, in init_customizations
      """ % args, re.VERBOSE | re.UNICODE),
  KeyError: 'non_unescaped_whitespace_escape_before'

It doesn't seem to be the right way to achieve it.

Third problem: since the above tries didn't worked, I'd a look on custom
directives and roles.
But, it does not seems to be allowed to define a role with a custom regex.
This would means I have to rewrite all "(TS-XXXX)" expression into ":TS:
XXXX".

Is there any way to achieve the migration in a compatible way with
docutils 0.12 *and* without rewriting the docuemntation ?

Cheers,
Jean Baptiste Favre

[1]: https://github.com/apache/trafficserver/tree/master/doc

Re: [Docutils-users] Migration docutils 0.12 to 0.13

[Guenter M. <mi...@us...>]

On 2016-12-31, Jean Baptiste Favre wrote:
> Hello,
> I'm facing an issue with Trafficserver documentation [1] which doesn't
> build with docutils 0.13.1

> Error comes from a custom Inliner class which is defined in doc/conf.py
> of Trafficserver project(starting line 163).
> This allow to transform text like "(TS-XXXX)" in a link to
> trafficserver's Jira

> This custom Inliner used to work with docutils 0.12. It fails on 0.13.1
> with following error:

>   Exception occurred:
>     File "conf.py", line 185, in __init__
>       start_string_prefix=self.start_string_prefix,
>   AttributeError: Inliner instance has no attribute 'start_string_prefix'

> Since docutils 0.13, start_string_prefix isn't statically defined. We
> have to call init_customiations for that.

Yes, this changed with the implementation of the long expected feature

  - Apply [ 103 ] Recognize inline markups without word boundaries.

and the new configuration setting "character_level_inline_markup".


> First problem: one  must pass settings param when calling
> init_customizations, but I can't find any format or structure for it.

> Second problem: I tried a workaround, creating a InlinerSettings class
> with minimal properties so that init_customizations could pass:

>   class InlinerSettings:
>     character_level_inline_markup=None
>     pep_references=None
>     rfc_references=None

> But, then, I faced another error:

...

"settings" is a an object returned from the option parser (optparse module).

There are others facing similar problems, e.g. Python distutils. 
There, I learned the trick is to use

  settings = frontend.OptionParser(components=(Parser,)).get_default_values()
  
  -- https://hg.python.org/cpython/rev/db09d760b965


> Third problem: since the above tries didn't worked, I'd a look on custom
> directives and roles.
> But, it does not seems to be allowed to define a role with a custom regex.
> This would means I have to rewrite all "(TS-XXXX)" expression into ":TS:
> XXXX".

This is the "minimal-invasive" approach, it would be work now but might save
issues later, as it depends less on Docutils internals.

> Is there any way to achieve the migration in a compatible way with
> docutils 0.12 *and* without rewriting the docuemntation ?

If the above example does not lead to a solution, you could also consider to
copy the definition of start_string_prefix ...
from states.py (importing utils.punctuation_chars first)

Günter

Re: [Docutils-users] Migration docutils 0.12 to 0.13

[David G. <go...@py...>]

On Sun, Jan 1, 2017 at 1:59 PM, Guenter Milde <mi...@us...> wrote:
> On 2016-12-31, Jean Baptiste Favre wrote:
>> Hello,
>> I'm facing an issue with Trafficserver documentation [1] which doesn't
>> build with docutils 0.13.1
>
>> Error comes from a custom Inliner class which is defined in doc/conf.py
>> of Trafficserver project(starting line 163).
>> This allow to transform text like "(TS-XXXX)" in a link to
>> trafficserver's Jira
>
>> This custom Inliner used to work with docutils 0.12. It fails on 0.13.1
>> with following error:
>
>>   Exception occurred:
>>     File "conf.py", line 185, in __init__
>>       start_string_prefix=self.start_string_prefix,
>>   AttributeError: Inliner instance has no attribute 'start_string_prefix'
>
>> Since docutils 0.13, start_string_prefix isn't statically defined. We
>> have to call init_customiations for that.
>
> Yes, this changed with the implementation of the long expected feature
>
>   - Apply [ 103 ] Recognize inline markups without word boundaries.
>
> and the new configuration setting "character_level_inline_markup".

This is another example of an API change coming back to bite us.

>> First problem: one  must pass settings param when calling
>> init_customizations, but I can't find any format or structure for it.
>
>> Second problem: I tried a workaround, creating a InlinerSettings class
>> with minimal properties so that init_customizations could pass:
>
>>   class InlinerSettings:
>>     character_level_inline_markup=None
>>     pep_references=None
>>     rfc_references=None
>
>> But, then, I faced another error:
>
> ...
>
> "settings" is a an object returned from the option parser (optparse module).
>
> There are others facing similar problems, e.g. Python distutils.
> There, I learned the trick is to use
>
>   settings = frontend.OptionParser(components=(Parser,)).get_default_values()
>
>   -- https://hg.python.org/cpython/rev/db09d760b965

I think in this case, this is a red herring (a false issue). It's not
the cause of the problem.

>> Third problem: since the above tries didn't worked, I'd a look on custom
>> directives and roles.
>> But, it does not seems to be allowed to define a role with a custom regex.
>> This would means I have to rewrite all "(TS-XXXX)" expression into ":TS:
>> XXXX".
>
> This is the "minimal-invasive" approach, it would be work now but might save
> issues later, as it depends less on Docutils internals.
>
>> Is there any way to achieve the migration in a compatible way with
>> docutils 0.12 *and* without rewriting the docuemntation ?
>
> If the above example does not lead to a solution, you could also consider to
> copy the definition of start_string_prefix ...
> from states.py (importing utils.punctuation_chars first)

Or we could fix our mistake. I think that's better.

David Goodger
<http://python.net/~goodger>

Re: [Docutils-users] Migration docutils 0.12 to 0.13

[Jean B. F. <web...@jb...>]

Hello,
I tried another workaround.
Instead of using a custom class, I overrided init_cutomizations method
using:

  class Inliner(BaseInliner):
    def __init(self):
      BaseInliner.__init__(self)

    def init_customizations(self, settings):
      BaseInliner.init_customizations(self, settings)

      issue_pattern = re.compile(u'''
        {start_string_prefix}
        TS-\d+
        {end_string_suffix}'''.format(
          start_string_prefix=self.start_string_prefix,
          end_string_suffix=self.end_string_suffix),
        re.VERBOSE | re.UNICODE)

      self.implicit_dispatch.append((issue_pattern, self.issue_reference))

I don't call explicitly init_customizations, but it's called: during
run, I still get the error:

  Exception occurred:
    File
"/usr/lib/python2.7/dist-packages/docutils/parsers/rst/states.py", line
530, in init_customizations
        """ % args, re.VERBOSE | re.UNICODE),
  KeyError: 'non_unescaped_whitespace_escape_before'
  The full traceback has been saved in /tmp/sphinx-err-wLtHoF.log, if
you want to report the issue to the developers.
  Please also report this if it was a user error, so that a better error
message can be provided next time.
  A bug report can be filed in the tracker at
<https://github.com/sphinx-doc/sphinx/issues>. Thanks!
  make[6]: *** [man] Error 1

What did I do wrong ?

Cheers,
Jean Baptiste Favre

On 01/01/2017 20:59, Guenter Milde wrote:
> On 2016-12-31, Jean Baptiste Favre wrote:
>> Hello,
>> I'm facing an issue with Trafficserver documentation [1] which doesn't
>> build with docutils 0.13.1
> 
>> Error comes from a custom Inliner class which is defined in doc/conf.py
>> of Trafficserver project(starting line 163).
>> This allow to transform text like "(TS-XXXX)" in a link to
>> trafficserver's Jira
> 
>> This custom Inliner used to work with docutils 0.12. It fails on 0.13.1
>> with following error:
> 
>>   Exception occurred:
>>     File "conf.py", line 185, in __init__
>>       start_string_prefix=self.start_string_prefix,
>>   AttributeError: Inliner instance has no attribute 'start_string_prefix'
> 
>> Since docutils 0.13, start_string_prefix isn't statically defined. We
>> have to call init_customiations for that.
> 
> Yes, this changed with the implementation of the long expected feature
> 
>   - Apply [ 103 ] Recognize inline markups without word boundaries.
> 
> and the new configuration setting "character_level_inline_markup".
> 
> 
>> First problem: one  must pass settings param when calling
>> init_customizations, but I can't find any format or structure for it.
> 
>> Second problem: I tried a workaround, creating a InlinerSettings class
>> with minimal properties so that init_customizations could pass:
> 
>>   class InlinerSettings:
>>     character_level_inline_markup=None
>>     pep_references=None
>>     rfc_references=None
> 
>> But, then, I faced another error:
> 
> ...
> 
> "settings" is a an object returned from the option parser (optparse module).
> 
> There are others facing similar problems, e.g. Python distutils. 
> There, I learned the trick is to use
> 
>   settings = frontend.OptionParser(components=(Parser,)).get_default_values()
>   
>   -- https://hg.python.org/cpython/rev/db09d760b965
> 
> 
>> Third problem: since the above tries didn't worked, I'd a look on custom
>> directives and roles.
>> But, it does not seems to be allowed to define a role with a custom regex.
>> This would means I have to rewrite all "(TS-XXXX)" expression into ":TS:
>> XXXX".
> 
> This is the "minimal-invasive" approach, it would be work now but might save
> issues later, as it depends less on Docutils internals.
> 
>> Is there any way to achieve the migration in a compatible way with
>> docutils 0.12 *and* without rewriting the docuemntation ?
> 
> If the above example does not lead to a solution, you could also consider to
> copy the definition of start_string_prefix ...
> from states.py (importing utils.punctuation_chars first)
> 
> Günter

Re: [Docutils-users] Migration docutils 0.12 to 0.13

[David G. <go...@py...>]

On Sun, Jan 1, 2017 at 4:17 PM, Jean Baptiste Favre
<web...@jb...> wrote:
> Hello,
> I tried another workaround.
> Instead of using a custom class, I overrided init_cutomizations method
> using:
>
>   class Inliner(BaseInliner):
>     def __init(self):
>       BaseInliner.__init__(self)
>
>     def init_customizations(self, settings):
>       BaseInliner.init_customizations(self, settings)
>
>       issue_pattern = re.compile(u'''
>         {start_string_prefix}
>         TS-\d+
>         {end_string_suffix}'''.format(
>           start_string_prefix=self.start_string_prefix,
>           end_string_suffix=self.end_string_suffix),
>         re.VERBOSE | re.UNICODE)
>
>       self.implicit_dispatch.append((issue_pattern, self.issue_reference))
>
> I don't call explicitly init_customizations, but it's called: during
> run, I still get the error:
>
>   Exception occurred:
>     File
> "/usr/lib/python2.7/dist-packages/docutils/parsers/rst/states.py", line
> 530, in init_customizations
>         """ % args, re.VERBOSE | re.UNICODE),
>   KeyError: 'non_unescaped_whitespace_escape_before'
>   The full traceback has been saved in /tmp/sphinx-err-wLtHoF.log, if
> you want to report the issue to the developers.
>   Please also report this if it was a user error, so that a better error
> message can be provided next time.
>   A bug report can be filed in the tracker at
> <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
>   make[6]: *** [man] Error 1
>
> What did I do wrong ?

You didn't do anything wrong that I can see. There was an internal
change to Docutils in revision 7942 that refactored some code, and a
side effect was to remove some attributes from the Inliner class (they
became locals instead). Your code depends on these class attributes.

Try applying the attached patch and let us know how it works. The
missing class attributes will be present as instance attributes, which
should be close enough.

Günter, I think the attached patch should also be rolled into a 0.13.2
bugfix release.

David Goodger
<http://python.net/~goodger>


> On 01/01/2017 20:59, Guenter Milde wrote:
>> On 2016-12-31, Jean Baptiste Favre wrote:
>>> Hello,
>>> I'm facing an issue with Trafficserver documentation [1] which doesn't
>>> build with docutils 0.13.1
>>
>>> Error comes from a custom Inliner class which is defined in doc/conf.py
>>> of Trafficserver project(starting line 163).
>>> This allow to transform text like "(TS-XXXX)" in a link to
>>> trafficserver's Jira
>>
>>> This custom Inliner used to work with docutils 0.12. It fails on 0.13.1
>>> with following error:
>>
>>>   Exception occurred:
>>>     File "conf.py", line 185, in __init__
>>>       start_string_prefix=self.start_string_prefix,
>>>   AttributeError: Inliner instance has no attribute 'start_string_prefix'
>>
>>> Since docutils 0.13, start_string_prefix isn't statically defined. We
>>> have to call init_customiations for that.
>>
>> Yes, this changed with the implementation of the long expected feature
>>
>>   - Apply [ 103 ] Recognize inline markups without word boundaries.
>>
>> and the new configuration setting "character_level_inline_markup".
>>
>>
>>> First problem: one  must pass settings param when calling
>>> init_customizations, but I can't find any format or structure for it.
>>
>>> Second problem: I tried a workaround, creating a InlinerSettings class
>>> with minimal properties so that init_customizations could pass:
>>
>>>   class InlinerSettings:
>>>     character_level_inline_markup=None
>>>     pep_references=None
>>>     rfc_references=None
>>
>>> But, then, I faced another error:
>>
>> ...
>>
>> "settings" is a an object returned from the option parser (optparse module).
>>
>> There are others facing similar problems, e.g. Python distutils.
>> There, I learned the trick is to use
>>
>>   settings = frontend.OptionParser(components=(Parser,)).get_default_values()
>>
>>   -- https://hg.python.org/cpython/rev/db09d760b965
>>
>>
>>> Third problem: since the above tries didn't worked, I'd a look on custom
>>> directives and roles.
>>> But, it does not seems to be allowed to define a role with a custom regex.
>>> This would means I have to rewrite all "(TS-XXXX)" expression into ":TS:
>>> XXXX".
>>
>> This is the "minimal-invasive" approach, it would be work now but might save
>> issues later, as it depends less on Docutils internals.
>>
>>> Is there any way to achieve the migration in a compatible way with
>>> docutils 0.12 *and* without rewriting the docuemntation ?
>>
>> If the above example does not lead to a solution, you could also consider to
>> copy the definition of start_string_prefix ...
>> from states.py (importing utils.punctuation_chars first)
>>
>> Günter

Re: [Docutils-users] Migration docutils 0.12 to 0.13

[David G. <go...@py...>]

Patch attached.

David Goodger
<http://python.net/~goodger>


On Mon, Jan 2, 2017 at 2:24 PM, David Goodger <go...@py...> wrote:
> On Sun, Jan 1, 2017 at 4:17 PM, Jean Baptiste Favre
> <web...@jb...> wrote:
>> Hello,
>> I tried another workaround.
>> Instead of using a custom class, I overrided init_cutomizations method
>> using:
>>
>>   class Inliner(BaseInliner):
>>     def __init(self):
>>       BaseInliner.__init__(self)
>>
>>     def init_customizations(self, settings):
>>       BaseInliner.init_customizations(self, settings)
>>
>>       issue_pattern = re.compile(u'''
>>         {start_string_prefix}
>>         TS-\d+
>>         {end_string_suffix}'''.format(
>>           start_string_prefix=self.start_string_prefix,
>>           end_string_suffix=self.end_string_suffix),
>>         re.VERBOSE | re.UNICODE)
>>
>>       self.implicit_dispatch.append((issue_pattern, self.issue_reference))
>>
>> I don't call explicitly init_customizations, but it's called: during
>> run, I still get the error:
>>
>>   Exception occurred:
>>     File
>> "/usr/lib/python2.7/dist-packages/docutils/parsers/rst/states.py", line
>> 530, in init_customizations
>>         """ % args, re.VERBOSE | re.UNICODE),
>>   KeyError: 'non_unescaped_whitespace_escape_before'
>>   The full traceback has been saved in /tmp/sphinx-err-wLtHoF.log, if
>> you want to report the issue to the developers.
>>   Please also report this if it was a user error, so that a better error
>> message can be provided next time.
>>   A bug report can be filed in the tracker at
>> <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
>>   make[6]: *** [man] Error 1
>>
>> What did I do wrong ?
>
> You didn't do anything wrong that I can see. There was an internal
> change to Docutils in revision 7942 that refactored some code, and a
> side effect was to remove some attributes from the Inliner class (they
> became locals instead). Your code depends on these class attributes.
>
> Try applying the attached patch and let us know how it works. The
> missing class attributes will be present as instance attributes, which
> should be close enough.
>
> Günter, I think the attached patch should also be rolled into a 0.13.2
> bugfix release.
>
> David Goodger
> <http://python.net/~goodger>
>
>
>> On 01/01/2017 20:59, Guenter Milde wrote:
>>> On 2016-12-31, Jean Baptiste Favre wrote:
>>>> Hello,
>>>> I'm facing an issue with Trafficserver documentation [1] which doesn't
>>>> build with docutils 0.13.1
>>>
>>>> Error comes from a custom Inliner class which is defined in doc/conf.py
>>>> of Trafficserver project(starting line 163).
>>>> This allow to transform text like "(TS-XXXX)" in a link to
>>>> trafficserver's Jira
>>>
>>>> This custom Inliner used to work with docutils 0.12. It fails on 0.13.1
>>>> with following error:
>>>
>>>>   Exception occurred:
>>>>     File "conf.py", line 185, in __init__
>>>>       start_string_prefix=self.start_string_prefix,
>>>>   AttributeError: Inliner instance has no attribute 'start_string_prefix'
>>>
>>>> Since docutils 0.13, start_string_prefix isn't statically defined. We
>>>> have to call init_customiations for that.
>>>
>>> Yes, this changed with the implementation of the long expected feature
>>>
>>>   - Apply [ 103 ] Recognize inline markups without word boundaries.
>>>
>>> and the new configuration setting "character_level_inline_markup".
>>>
>>>
>>>> First problem: one  must pass settings param when calling
>>>> init_customizations, but I can't find any format or structure for it.
>>>
>>>> Second problem: I tried a workaround, creating a InlinerSettings class
>>>> with minimal properties so that init_customizations could pass:
>>>
>>>>   class InlinerSettings:
>>>>     character_level_inline_markup=None
>>>>     pep_references=None
>>>>     rfc_references=None
>>>
>>>> But, then, I faced another error:
>>>
>>> ...
>>>
>>> "settings" is a an object returned from the option parser (optparse module).
>>>
>>> There are others facing similar problems, e.g. Python distutils.
>>> There, I learned the trick is to use
>>>
>>>   settings = frontend.OptionParser(components=(Parser,)).get_default_values()
>>>
>>>   -- https://hg.python.org/cpython/rev/db09d760b965
>>>
>>>
>>>> Third problem: since the above tries didn't worked, I'd a look on custom
>>>> directives and roles.
>>>> But, it does not seems to be allowed to define a role with a custom regex.
>>>> This would means I have to rewrite all "(TS-XXXX)" expression into ":TS:
>>>> XXXX".
>>>
>>> This is the "minimal-invasive" approach, it would be work now but might save
>>> issues later, as it depends less on Docutils internals.
>>>
>>>> Is there any way to achieve the migration in a compatible way with
>>>> docutils 0.12 *and* without rewriting the docuemntation ?
>>>
>>> If the above example does not lead to a solution, you could also consider to
>>> copy the definition of start_string_prefix ...
>>> from states.py (importing utils.punctuation_chars first)
>>>
>>> Günter

Re: [Docutils-users] Migration docutils 0.12 to 0.13

[Jean B. F. <web...@jb...>]

Hello David,

Thanks for your answer.
Unfortunatly, the patch doesn't help.

I still have a "KeyError: 'non_unescaped_whitespace_escape_before'" in
docutils/parsers/rst/states.py at line 533.
I guess it's the same kind of problem and I should redefine all local
attributes (from lines 655 to 687) as class attributes as you did in the
patch.

I'll test it asap and report,
Cheers,
Jean Baptiste


On 02/01/2017 21:24, David Goodger wrote:
> On Sun, Jan 1, 2017 at 4:17 PM, Jean Baptiste Favre
> <web...@jb...> wrote:
>> Hello,
>> I tried another workaround.
>> Instead of using a custom class, I overrided init_cutomizations method
>> using:
>>
>>   class Inliner(BaseInliner):
>>     def __init(self):
>>       BaseInliner.__init__(self)
>>
>>     def init_customizations(self, settings):
>>       BaseInliner.init_customizations(self, settings)
>>
>>       issue_pattern = re.compile(u'''
>>         {start_string_prefix}
>>         TS-\d+
>>         {end_string_suffix}'''.format(
>>           start_string_prefix=self.start_string_prefix,
>>           end_string_suffix=self.end_string_suffix),
>>         re.VERBOSE | re.UNICODE)
>>
>>       self.implicit_dispatch.append((issue_pattern, self.issue_reference))
>>
>> I don't call explicitly init_customizations, but it's called: during
>> run, I still get the error:
>>
>>   Exception occurred:
>>     File
>> "/usr/lib/python2.7/dist-packages/docutils/parsers/rst/states.py", line
>> 530, in init_customizations
>>         """ % args, re.VERBOSE | re.UNICODE),
>>   KeyError: 'non_unescaped_whitespace_escape_before'
>>   The full traceback has been saved in /tmp/sphinx-err-wLtHoF.log, if
>> you want to report the issue to the developers.
>>   Please also report this if it was a user error, so that a better error
>> message can be provided next time.
>>   A bug report can be filed in the tracker at
>> <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
>>   make[6]: *** [man] Error 1
>>
>> What did I do wrong ?
> 
> You didn't do anything wrong that I can see. There was an internal
> change to Docutils in revision 7942 that refactored some code, and a
> side effect was to remove some attributes from the Inliner class (they
> became locals instead). Your code depends on these class attributes.
> 
> Try applying the attached patch and let us know how it works. The
> missing class attributes will be present as instance attributes, which
> should be close enough.
> 
> Günter, I think the attached patch should also be rolled into a 0.13.2
> bugfix release.
> 
> David Goodger
> <http://python.net/~goodger>
> 
> 
>> On 01/01/2017 20:59, Guenter Milde wrote:
>>> On 2016-12-31, Jean Baptiste Favre wrote:
>>>> Hello,
>>>> I'm facing an issue with Trafficserver documentation [1] which doesn't
>>>> build with docutils 0.13.1
>>>
>>>> Error comes from a custom Inliner class which is defined in doc/conf.py
>>>> of Trafficserver project(starting line 163).
>>>> This allow to transform text like "(TS-XXXX)" in a link to
>>>> trafficserver's Jira
>>>
>>>> This custom Inliner used to work with docutils 0.12. It fails on 0.13.1
>>>> with following error:
>>>
>>>>   Exception occurred:
>>>>     File "conf.py", line 185, in __init__
>>>>       start_string_prefix=self.start_string_prefix,
>>>>   AttributeError: Inliner instance has no attribute 'start_string_prefix'
>>>
>>>> Since docutils 0.13, start_string_prefix isn't statically defined. We
>>>> have to call init_customiations for that.
>>>
>>> Yes, this changed with the implementation of the long expected feature
>>>
>>>   - Apply [ 103 ] Recognize inline markups without word boundaries.
>>>
>>> and the new configuration setting "character_level_inline_markup".
>>>
>>>
>>>> First problem: one  must pass settings param when calling
>>>> init_customizations, but I can't find any format or structure for it.
>>>
>>>> Second problem: I tried a workaround, creating a InlinerSettings class
>>>> with minimal properties so that init_customizations could pass:
>>>
>>>>   class InlinerSettings:
>>>>     character_level_inline_markup=None
>>>>     pep_references=None
>>>>     rfc_references=None
>>>
>>>> But, then, I faced another error:
>>>
>>> ...
>>>
>>> "settings" is a an object returned from the option parser (optparse module).
>>>
>>> There are others facing similar problems, e.g. Python distutils.
>>> There, I learned the trick is to use
>>>
>>>   settings = frontend.OptionParser(components=(Parser,)).get_default_values()
>>>
>>>   -- https://hg.python.org/cpython/rev/db09d760b965
>>>
>>>
>>>> Third problem: since the above tries didn't worked, I'd a look on custom
>>>> directives and roles.
>>>> But, it does not seems to be allowed to define a role with a custom regex.
>>>> This would means I have to rewrite all "(TS-XXXX)" expression into ":TS:
>>>> XXXX".
>>>
>>> This is the "minimal-invasive" approach, it would be work now but might save
>>> issues later, as it depends less on Docutils internals.
>>>
>>>> Is there any way to achieve the migration in a compatible way with
>>>> docutils 0.12 *and* without rewriting the docuemntation ?
>>>
>>> If the above example does not lead to a solution, you could also consider to
>>> copy the definition of start_string_prefix ...
>>> from states.py (importing utils.punctuation_chars first)
>>>
>>> Günter

Re: [Docutils-users] Migration docutils 0.12 to 0.13

[David G. <go...@py...>]

On Mon, Jan 2, 2017 at 3:26 PM, Jean Baptiste Favre
<web...@jb...> wrote:
> Hello David,
>
> Thanks for your answer.
> Unfortunatly, the patch doesn't help.
>
> I still have a "KeyError: 'non_unescaped_whitespace_escape_before'" in
> docutils/parsers/rst/states.py at line 533.

In future, please send a full traceback. We need to know the context
(what called what).

> I guess it's the same kind of problem and I should redefine all local
> attributes (from lines 655 to 687) as class attributes as you did in the
> patch.

I think I see what the problem is now. docutils/parsers/rst/states.py,
Inliner.init_customizations contains the following line::

    args.update(vars(self.__class__))

The "vars" function gets the __dict__ from the object's class. But
you're using a subclass, so vars returns the __dict__ of the subclass,
without the the superclass's attributes, which is what is needed. It's
a bit of metaprogramming gone wrong. It's not safe for subclassing,
which is a bug. It's already a bit of a kludgey shortcut. We could
replace it with::

    args.update(vars(Inliner))

That's a bit smelly, but I see it's done elsewhere in the code. Try
making that change (patch attached).

We may have to explicitly refer to all the class attributes. I'll think on it.

David Goodger

> I'll test it asap and report,
> Cheers,
> Jean Baptiste
>
>
> On 02/01/2017 21:24, David Goodger wrote:
>> On Sun, Jan 1, 2017 at 4:17 PM, Jean Baptiste Favre
>> <web...@jb...> wrote:
>>> Hello,
>>> I tried another workaround.
>>> Instead of using a custom class, I overrided init_cutomizations method
>>> using:
>>>
>>>   class Inliner(BaseInliner):
>>>     def __init(self):
>>>       BaseInliner.__init__(self)
>>>
>>>     def init_customizations(self, settings):
>>>       BaseInliner.init_customizations(self, settings)
>>>
>>>       issue_pattern = re.compile(u'''
>>>         {start_string_prefix}
>>>         TS-\d+
>>>         {end_string_suffix}'''.format(
>>>           start_string_prefix=self.start_string_prefix,
>>>           end_string_suffix=self.end_string_suffix),
>>>         re.VERBOSE | re.UNICODE)
>>>
>>>       self.implicit_dispatch.append((issue_pattern, self.issue_reference))
>>>
>>> I don't call explicitly init_customizations, but it's called: during
>>> run, I still get the error:
>>>
>>>   Exception occurred:
>>>     File
>>> "/usr/lib/python2.7/dist-packages/docutils/parsers/rst/states.py", line
>>> 530, in init_customizations
>>>         """ % args, re.VERBOSE | re.UNICODE),
>>>   KeyError: 'non_unescaped_whitespace_escape_before'
>>>   The full traceback has been saved in /tmp/sphinx-err-wLtHoF.log, if
>>> you want to report the issue to the developers.
>>>   Please also report this if it was a user error, so that a better error
>>> message can be provided next time.
>>>   A bug report can be filed in the tracker at
>>> <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
>>>   make[6]: *** [man] Error 1
>>>
>>> What did I do wrong ?
>>
>> You didn't do anything wrong that I can see. There was an internal
>> change to Docutils in revision 7942 that refactored some code, and a
>> side effect was to remove some attributes from the Inliner class (they
>> became locals instead). Your code depends on these class attributes.
>>
>> Try applying the attached patch and let us know how it works. The
>> missing class attributes will be present as instance attributes, which
>> should be close enough.
>>
>> Günter, I think the attached patch should also be rolled into a 0.13.2
>> bugfix release.
>>
>> David Goodger
>> <http://python.net/~goodger>
>>
>>
>>> On 01/01/2017 20:59, Guenter Milde wrote:
>>>> On 2016-12-31, Jean Baptiste Favre wrote:
>>>>> Hello,
>>>>> I'm facing an issue with Trafficserver documentation [1] which doesn't
>>>>> build with docutils 0.13.1
>>>>
>>>>> Error comes from a custom Inliner class which is defined in doc/conf.py
>>>>> of Trafficserver project(starting line 163).
>>>>> This allow to transform text like "(TS-XXXX)" in a link to
>>>>> trafficserver's Jira
>>>>
>>>>> This custom Inliner used to work with docutils 0.12. It fails on 0.13.1
>>>>> with following error:
>>>>
>>>>>   Exception occurred:
>>>>>     File "conf.py", line 185, in __init__
>>>>>       start_string_prefix=self.start_string_prefix,
>>>>>   AttributeError: Inliner instance has no attribute 'start_string_prefix'
>>>>
>>>>> Since docutils 0.13, start_string_prefix isn't statically defined. We
>>>>> have to call init_customiations for that.
>>>>
>>>> Yes, this changed with the implementation of the long expected feature
>>>>
>>>>   - Apply [ 103 ] Recognize inline markups without word boundaries.
>>>>
>>>> and the new configuration setting "character_level_inline_markup".
>>>>
>>>>
>>>>> First problem: one  must pass settings param when calling
>>>>> init_customizations, but I can't find any format or structure for it.
>>>>
>>>>> Second problem: I tried a workaround, creating a InlinerSettings class
>>>>> with minimal properties so that init_customizations could pass:
>>>>
>>>>>   class InlinerSettings:
>>>>>     character_level_inline_markup=None
>>>>>     pep_references=None
>>>>>     rfc_references=None
>>>>
>>>>> But, then, I faced another error:
>>>>
>>>> ...
>>>>
>>>> "settings" is a an object returned from the option parser (optparse module).
>>>>
>>>> There are others facing similar problems, e.g. Python distutils.
>>>> There, I learned the trick is to use
>>>>
>>>>   settings = frontend.OptionParser(components=(Parser,)).get_default_values()
>>>>
>>>>   -- https://hg.python.org/cpython/rev/db09d760b965
>>>>
>>>>
>>>>> Third problem: since the above tries didn't worked, I'd a look on custom
>>>>> directives and roles.
>>>>> But, it does not seems to be allowed to define a role with a custom regex.
>>>>> This would means I have to rewrite all "(TS-XXXX)" expression into ":TS:
>>>>> XXXX".
>>>>
>>>> This is the "minimal-invasive" approach, it would be work now but might save
>>>> issues later, as it depends less on Docutils internals.
>>>>
>>>>> Is there any way to achieve the migration in a compatible way with
>>>>> docutils 0.12 *and* without rewriting the docuemntation ?
>>>>
>>>> If the above example does not lead to a solution, you could also consider to
>>>> copy the definition of start_string_prefix ...
>>>> from states.py (importing utils.punctuation_chars first)
>>>>
>>>> Günter
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Docutils-users mailing list
> Doc...@li...
> https://lists.sourceforge.net/lists/listinfo/docutils-users
>
> Please use "Reply All" to reply to the list.

Re: [Docutils-users] Migration docutils 0.12 to 0.13

[Jean B. F. <web...@jb...>]

Still have the same error.
The full traceback is attached, sorry to have forgotten it before.

My subclass is as follow:

from docutils import nodes
from docutils.parsers.rst import states
from docutils.utils import unescape

# Customize parser.inliner in the only way that Sphinx supports.
# docutils.parsers.rst.Parser takes an instance of states.Inliner or a
# subclass but Sphinx initializes it from
# SphinxStandaloneReader.set_parser('restructuredtext') which is called
# from Publisher.set_components() and initializes the parser without
# arguments.

BaseInliner = states.Inliner
class Inliner(BaseInliner):

  def init_customizations(self, settings):
    BaseInliner.init_customizations(self, settings)

    issue_pattern = re.compile(u'''
      {start_string_prefix}
      TS-\d+
      {end_string_suffix}'''.format(
        start_string_prefix=self.start_string_prefix,
        end_string_suffix=self.end_string_suffix),
      re.VERBOSE | re.UNICODE)

    self.implicit_dispatch.append((issue_pattern, self.issue_reference))

  def issue_reference(self, match, lineno):
    text = match.group(0)

    rawsource = unescape(text, True)
    text = unescape(text, False)

    refuri = 'https://issues.apache.org/jira/browse/' + text

    return [nodes.reference(rawsource, text, refuri=refuri)]

states.Inliner = Inliner


Thanks,
Jean Baptiste

On 02/01/2017 22:53, David Goodger wrote:
> On Mon, Jan 2, 2017 at 3:26 PM, Jean Baptiste Favre
> <web...@jb...> wrote:
>> Hello David,
>>
>> Thanks for your answer.
>> Unfortunatly, the patch doesn't help.
>>
>> I still have a "KeyError: 'non_unescaped_whitespace_escape_before'" in
>> docutils/parsers/rst/states.py at line 533.
> 
> In future, please send a full traceback. We need to know the context
> (what called what).
> 
>> I guess it's the same kind of problem and I should redefine all local
>> attributes (from lines 655 to 687) as class attributes as you did in the
>> patch.
> 
> I think I see what the problem is now. docutils/parsers/rst/states.py,
> Inliner.init_customizations contains the following line::
> 
>     args.update(vars(self.__class__))
> 
> The "vars" function gets the __dict__ from the object's class. But
> you're using a subclass, so vars returns the __dict__ of the subclass,
> without the the superclass's attributes, which is what is needed. It's
> a bit of metaprogramming gone wrong. It's not safe for subclassing,
> which is a bug. It's already a bit of a kludgey shortcut. We could
> replace it with::
> 
>     args.update(vars(Inliner))
> 
> That's a bit smelly, but I see it's done elsewhere in the code. Try
> making that change (patch attached).
> 
> We may have to explicitly refer to all the class attributes. I'll think on it.
> 
> David Goodger
> 
>> I'll test it asap and report,
>> Cheers,
>> Jean Baptiste
>>
>>
>> On 02/01/2017 21:24, David Goodger wrote:
>>> On Sun, Jan 1, 2017 at 4:17 PM, Jean Baptiste Favre
>>> <web...@jb...> wrote:
>>>> Hello,
>>>> I tried another workaround.
>>>> Instead of using a custom class, I overrided init_cutomizations method
>>>> using:
>>>>
>>>>   class Inliner(BaseInliner):
>>>>     def __init(self):
>>>>       BaseInliner.__init__(self)
>>>>
>>>>     def init_customizations(self, settings):
>>>>       BaseInliner.init_customizations(self, settings)
>>>>
>>>>       issue_pattern = re.compile(u'''
>>>>         {start_string_prefix}
>>>>         TS-\d+
>>>>         {end_string_suffix}'''.format(
>>>>           start_string_prefix=self.start_string_prefix,
>>>>           end_string_suffix=self.end_string_suffix),
>>>>         re.VERBOSE | re.UNICODE)
>>>>
>>>>       self.implicit_dispatch.append((issue_pattern, self.issue_reference))
>>>>
>>>> I don't call explicitly init_customizations, but it's called: during
>>>> run, I still get the error:
>>>>
>>>>   Exception occurred:
>>>>     File
>>>> "/usr/lib/python2.7/dist-packages/docutils/parsers/rst/states.py", line
>>>> 530, in init_customizations
>>>>         """ % args, re.VERBOSE | re.UNICODE),
>>>>   KeyError: 'non_unescaped_whitespace_escape_before'
>>>>   The full traceback has been saved in /tmp/sphinx-err-wLtHoF.log, if
>>>> you want to report the issue to the developers.
>>>>   Please also report this if it was a user error, so that a better error
>>>> message can be provided next time.
>>>>   A bug report can be filed in the tracker at
>>>> <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
>>>>   make[6]: *** [man] Error 1
>>>>
>>>> What did I do wrong ?
>>>
>>> You didn't do anything wrong that I can see. There was an internal
>>> change to Docutils in revision 7942 that refactored some code, and a
>>> side effect was to remove some attributes from the Inliner class (they
>>> became locals instead). Your code depends on these class attributes.
>>>
>>> Try applying the attached patch and let us know how it works. The
>>> missing class attributes will be present as instance attributes, which
>>> should be close enough.
>>>
>>> Günter, I think the attached patch should also be rolled into a 0.13.2
>>> bugfix release.
>>>
>>> David Goodger
>>> <http://python.net/~goodger>
>>>
>>>
>>>> On 01/01/2017 20:59, Guenter Milde wrote:
>>>>> On 2016-12-31, Jean Baptiste Favre wrote:
>>>>>> Hello,
>>>>>> I'm facing an issue with Trafficserver documentation [1] which doesn't
>>>>>> build with docutils 0.13.1
>>>>>
>>>>>> Error comes from a custom Inliner class which is defined in doc/conf.py
>>>>>> of Trafficserver project(starting line 163).
>>>>>> This allow to transform text like "(TS-XXXX)" in a link to
>>>>>> trafficserver's Jira
>>>>>
>>>>>> This custom Inliner used to work with docutils 0.12. It fails on 0.13.1
>>>>>> with following error:
>>>>>
>>>>>>   Exception occurred:
>>>>>>     File "conf.py", line 185, in __init__
>>>>>>       start_string_prefix=self.start_string_prefix,
>>>>>>   AttributeError: Inliner instance has no attribute 'start_string_prefix'
>>>>>
>>>>>> Since docutils 0.13, start_string_prefix isn't statically defined. We
>>>>>> have to call init_customiations for that.
>>>>>
>>>>> Yes, this changed with the implementation of the long expected feature
>>>>>
>>>>>   - Apply [ 103 ] Recognize inline markups without word boundaries.
>>>>>
>>>>> and the new configuration setting "character_level_inline_markup".
>>>>>
>>>>>
>>>>>> First problem: one  must pass settings param when calling
>>>>>> init_customizations, but I can't find any format or structure for it.
>>>>>
>>>>>> Second problem: I tried a workaround, creating a InlinerSettings class
>>>>>> with minimal properties so that init_customizations could pass:
>>>>>
>>>>>>   class InlinerSettings:
>>>>>>     character_level_inline_markup=None
>>>>>>     pep_references=None
>>>>>>     rfc_references=None
>>>>>
>>>>>> But, then, I faced another error:
>>>>>
>>>>> ...
>>>>>
>>>>> "settings" is a an object returned from the option parser (optparse module).
>>>>>
>>>>> There are others facing similar problems, e.g. Python distutils.
>>>>> There, I learned the trick is to use
>>>>>
>>>>>   settings = frontend.OptionParser(components=(Parser,)).get_default_values()
>>>>>
>>>>>   -- https://hg.python.org/cpython/rev/db09d760b965
>>>>>
>>>>>
>>>>>> Third problem: since the above tries didn't worked, I'd a look on custom
>>>>>> directives and roles.
>>>>>> But, it does not seems to be allowed to define a role with a custom regex.
>>>>>> This would means I have to rewrite all "(TS-XXXX)" expression into ":TS:
>>>>>> XXXX".
>>>>>
>>>>> This is the "minimal-invasive" approach, it would be work now but might save
>>>>> issues later, as it depends less on Docutils internals.
>>>>>
>>>>>> Is there any way to achieve the migration in a compatible way with
>>>>>> docutils 0.12 *and* without rewriting the docuemntation ?
>>>>>
>>>>> If the above example does not lead to a solution, you could also consider to
>>>>> copy the definition of start_string_prefix ...
>>>>> from states.py (importing utils.punctuation_chars first)
>>>>>
>>>>> Günter
>>
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Docutils-users mailing list
>> Doc...@li...
>> https://lists.sourceforge.net/lists/listinfo/docutils-users
>>
>> Please use "Reply All" to reply to the list.
>>
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most 
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>
>>
>> _______________________________________________
>> Docutils-users mailing list
>> Doc...@li...
>> https://lists.sourceforge.net/lists/listinfo/docutils-users
>>
>> Please use "Reply All" to reply to the list.

Re: [Docutils-users] Migration docutils 0.12 to 0.13

[David G. <go...@py...>]

On Mon, Jan 2, 2017 at 5:03 PM, Jean Baptiste Favre
<web...@jb...> wrote:
> Still have the same error.
> The full traceback is attached, sorry to have forgotten it before.
>
> My subclass is as follow:
>
> from docutils import nodes
> from docutils.parsers.rst import states
> from docutils.utils import unescape
>
> # Customize parser.inliner in the only way that Sphinx supports.
> # docutils.parsers.rst.Parser takes an instance of states.Inliner or a
> # subclass but Sphinx initializes it from
> # SphinxStandaloneReader.set_parser('restructuredtext') which is called
> # from Publisher.set_components() and initializes the parser without
> # arguments.
>
> BaseInliner = states.Inliner
> class Inliner(BaseInliner):
>
>   def init_customizations(self, settings):
>     BaseInliner.init_customizations(self, settings)
>
>     issue_pattern = re.compile(u'''
>       {start_string_prefix}
>       TS-\d+
>       {end_string_suffix}'''.format(
>         start_string_prefix=self.start_string_prefix,
>         end_string_suffix=self.end_string_suffix),
>       re.VERBOSE | re.UNICODE)
>
>     self.implicit_dispatch.append((issue_pattern, self.issue_reference))
>
>   def issue_reference(self, match, lineno):
>     text = match.group(0)
>
>     rawsource = unescape(text, True)
>     text = unescape(text, False)
>
>     refuri = 'https://issues.apache.org/jira/browse/' + text
>
>     return [nodes.reference(rawsource, text, refuri=refuri)]
>
> states.Inliner = Inliner

That last line is the culprit. You're monkey-patching Docutils. Don't
do that. If something in Docutils breaks as a result, you can't
reasonably complain. If it worked before, it was just accidental.

I distilled the issue down, see the attached x.py & y.py. Put them on
your PYTHONPATH so y.py can import x.py. Run y.py. Note the output.
Now edit y.py and uncomment the commented line (the monkey patch). Now
run y.py again, and note the difference in the output. The monkey
patch messes up the namespace hierarchy, so x.A.run doesn't have
access to x.attr_x anymore.

Wasn't there another version of this code that worked in a different way?

Maybe you can monkey patch at the method level instead of the class
level. I.e. inject your methods into the existing Docutils Inliner
class, instead of replacing the class with your subclass. See z.py.
Caveat: monkey patching is dangerous. Even if this works now, I can't
guarantee that it will always work.

You're trying to do this through Sphinx, which isn't configurable, as
the comment in https://github.com/apache/trafficserver/blob/master/doc/conf.py
describes:

# Customize parser.inliner in the only way that Sphinx supports.
# docutils.parsers.rst.Parser takes an instance of states.Inliner or a
# subclass but Sphinx initializes it from
# SphinxStandaloneReader.set_parser('restructuredtext') which is called
# from Publisher.set_components() and initializes the parser without
# arguments.

Perhaps you should complain to Sphinx, and ask them to enable
configurability. If your code was using Docutils directly this
wouldn't be an issue (or would be less of an issue).

David Goodger
<http://python.net/~goodger>


> On 02/01/2017 22:53, David Goodger wrote:
>> On Mon, Jan 2, 2017 at 3:26 PM, Jean Baptiste Favre
>> <web...@jb...> wrote:
>>> Hello David,
>>>
>>> Thanks for your answer.
>>> Unfortunatly, the patch doesn't help.
>>>
>>> I still have a "KeyError: 'non_unescaped_whitespace_escape_before'" in
>>> docutils/parsers/rst/states.py at line 533.
>>
>> In future, please send a full traceback. We need to know the context
>> (what called what).
>>
>>> I guess it's the same kind of problem and I should redefine all local
>>> attributes (from lines 655 to 687) as class attributes as you did in the
>>> patch.
>>
>> I think I see what the problem is now. docutils/parsers/rst/states.py,
>> Inliner.init_customizations contains the following line::
>>
>>     args.update(vars(self.__class__))
>>
>> The "vars" function gets the __dict__ from the object's class. But
>> you're using a subclass, so vars returns the __dict__ of the subclass,
>> without the the superclass's attributes, which is what is needed. It's
>> a bit of metaprogramming gone wrong. It's not safe for subclassing,
>> which is a bug. It's already a bit of a kludgey shortcut. We could
>> replace it with::
>>
>>     args.update(vars(Inliner))
>>
>> That's a bit smelly, but I see it's done elsewhere in the code. Try
>> making that change (patch attached).
>>
>> We may have to explicitly refer to all the class attributes. I'll think on it.
>>
>> David Goodger
>>
>>> I'll test it asap and report,
>>> Cheers,
>>> Jean Baptiste
>>>
>>>

Re: [Docutils-users] Migration docutils 0.12 to 0.13

[Guenter M. <mi...@us...>]

On 2017-01-02, David Goodger wrote:
> On Sun, Jan 1, 2017 at 1:59 PM, Guenter Milde <mi...@us...> wrote:
>> On 2016-12-31, Jean Baptiste Favre wrote:
>>> Hello,
>>> I'm facing an issue with Trafficserver documentation [1] which doesn't
>>> build with docutils 0.13.1

>>> Error comes from a custom Inliner class which is defined in doc/conf.py
>>> of Trafficserver project(starting line 163).
...
>>> This custom Inliner used to work with docutils 0.12. It fails on 0.13.1
>>> with following error:

>>>   Exception occurred:
>>>     File "conf.py", line 185, in __init__
>>>       start_string_prefix=self.start_string_prefix,
>>>   AttributeError: Inliner instance has no attribute 'start_string_prefix'

>>> Since docutils 0.13, start_string_prefix isn't statically defined. We
>>> have to call init_customiations for that.

>> Yes, this changed with the implementation of the long expected feature

>>   - Apply [ 103 ] Recognize inline markups without word boundaries.

>> and the new configuration setting "character_level_inline_markup".

> This is another example of an API change coming back to bite us.

...

> Or we could fix our mistake. I think that's better.

I am not sure whether it was a mistake in this case:

The API change was discussed and the verdict was to use a switch to
solve the long-standing issue that makes reST difficult to use in
languages that don't use spaces as word delimiters.

__ https://sourceforge.net/p/docutils/patches/103/#f2f3

As a result, the `commit from 25 May 2016`__ 
implemented the `patch proposed by atsuo ishimoto`__ 

__ http://repo.or.cz/docutils.git/commitdiff/6839405e24a9a68e7cfd71643610d17115e6be2c
__ https://sourceforge.net/p/docutils/patches/_discuss/thread/e5337656/46b3/attachment/inline.patch.2

There is no easy way to implement 103 (new setting to allow character
level inline markup) without a change in the Inliner class. The
Inline.patterns becomes dependent on document settings.

Even if we define a static Inline.patterns default, using this default in
a 3rd party custom class inheriting from Inline may clash with the later
customization.

I see the following options:

a) revert the application of [ 103 ], reopening the
   "long-standing issue that makes reST difficult to use in languages
   that don't use spaces as word delimiters".
   
b) add documentation of this API change to the release notes in 0.13.2
   and help the OP to resolve the issue.

Günter

Re: [Docutils-users] Migration docutils 0.12 to 0.13

[Guenter M. <mi...@us...>]

Dear David,

On 2017-01-02, David Goodger wrote:

> Patch attached.

>> You didn't do anything wrong that I can see. There was an internal
>> change to Docutils in revision 7942 that refactored some code, and a
>> side effect was to remove some attributes from the Inliner class (they
>> became locals instead). Your code depends on these class attributes.

>> Try applying the attached patch and let us know how it works. The
>> missing class attributes will be present as instance attributes, which
>> should be close enough.

>> Günter, I think the attached patch should also be rolled into a 0.13.2
>> bugfix release.

Is the patch still "on the table"?

Also with this patch, the attributes `start_string_prefix`,
`end_string_suffix`, and `parts` are only available after calling
``inliner.init_customizations(document.settings)``. 

Is there a use-case for this scenario we want to support?
If not, we could define them as internal auxiliary variables after
the API change that came with implementation of [ 103 ].

That said, I am not against the patch. If including, I just propose a small
correction:


> Index: docutils/parsers/rst/states.py
>===================================================================
> --- docutils/parsers/rst/states.py	(revision 8008)
> +++ docutils/parsers/rst/states.py	(working copy)
> @@ -479,11 +479,13 @@
>                                   (punctuation_chars.closing_delimiters,
>                                    punctuation_chars.delimiters,
>                                    punctuation_chars.closers))
> +        self.start_string_prefix = start_string_prefix
> +        self.end_string_suffix = end_string_suffix

>          args = locals().copy()
>          args.update(vars(self.__class__))

> -        parts = ('initial_inline', start_string_prefix, '',
> +        parts = ('initial_inline', self.start_string_prefix, '',

No need to call the instance variable here, as we set it just some lines
above (overwriting an eventually existing instance).

>             [('start', '', self.non_whitespace_after, # simple start-strings
>               [r'\*\*',                # strong
>                r'\*(?!\*)',            # emphasis but not strong
> @@ -509,6 +511,7 @@
>               )
>              ]
>             )
> +        self.parts = parts

>          self.patterns = Struct(
>            initial=build_regexp(parts),


Thanks,

Günter

Re: [Docutils-users] Migration docutils 0.12 to 0.13

[David G. <go...@py...>]

On Tue, Jan 3, 2017 at 3:13 AM, Guenter Milde <mi...@us...> wrote:
> Dear David,
>
> On 2017-01-02, David Goodger wrote:
>
>> Patch attached.
>
>>> You didn't do anything wrong that I can see. There was an internal
>>> change to Docutils in revision 7942 that refactored some code, and a
>>> side effect was to remove some attributes from the Inliner class (they
>>> became locals instead). Your code depends on these class attributes.
>
>>> Try applying the attached patch and let us know how it works. The
>>> missing class attributes will be present as instance attributes, which
>>> should be close enough.
>
>>> Günter, I think the attached patch should also be rolled into a 0.13.2
>>> bugfix release.
>
> Is the patch still "on the table"?

Yes, but I'm looking at making more extensive changes first. The patch
that implemented the changes to support
settings.character_level_inline_markup really made a mess of things:

1. Inliner.__init__ is currently a no-op (code: just "pass"). That
smells bad. Why even have the method if it does nothing? Prior to the
patch (in, say, r7640), it initialized self.implicit_dispatch. It
can't do that completely now, but it can initialize an empty
self.implicit_dispatch with a docstring.

2. My coding style is to initialize all instance attributes in the
__init__ method of a class, for easy reference, and include an
attribute docstring for documentation. The Inliner class now breaks
this style. I'd like to fix this.

3. The Inliner.init_customizations method grew from 6 physical lines
of code to almost 140 lines. This was because all of the formerly
statically defined class attributes were brought in, to be defined
dynamically depending on settings.character_level_inline_markup. I
plan to refactor this huge method into a small init_customizations and
shunt all the definitions into another method.

4. That ``args.update(vars(Inliner))`` line is fragile, and should be replaced.

I'll work on all these and propose another patch.

> Also with this patch, the attributes `start_string_prefix`,
> `end_string_suffix`, and `parts` are only available after calling
> ``inliner.init_customizations(document.settings)``.

This may change back to the prior status.

> Is there a use-case for this scenario we want to support?
> If not, we could define them as internal auxiliary variables after
> the API change that came with implementation of [ 103 ].

I want to restore the attributes (formerly class attributes, maybe now
they have to be instance attributes, although maybe not) that were
part of the de-facto API prior to [103]. We've seen one example of
client code that depended on these, and there may be others.

> That said, I am not against the patch. If including, I just propose a small
> correction:

Incorporated into my working copy.

David Goodger
<http://python.net/~goodger>

Re: [Docutils-users] Migration docutils 0.12 to 0.13

[Jean B. F. <web...@jb...>]

Hello David,

On 03/01/2017 01:15, David Goodger wrote:
> On Mon, Jan 2, 2017 at 5:03 PM, Jean Baptiste Favre
> <web...@jb...> wrote:
>> Still have the same error.
>> The full traceback is attached, sorry to have forgotten it before.
>>
>> My subclass is as follow:
>>
>> from docutils import nodes
>> from docutils.parsers.rst import states
>> from docutils.utils import unescape
>>
>> # Customize parser.inliner in the only way that Sphinx supports.
>> # docutils.parsers.rst.Parser takes an instance of states.Inliner or a
>> # subclass but Sphinx initializes it from
>> # SphinxStandaloneReader.set_parser('restructuredtext') which is called
>> # from Publisher.set_components() and initializes the parser without
>> # arguments.
>>
>> BaseInliner = states.Inliner
>> class Inliner(BaseInliner):
>>
>>   def init_customizations(self, settings):
>>     BaseInliner.init_customizations(self, settings)
>>
>>     issue_pattern = re.compile(u'''
>>       {start_string_prefix}
>>       TS-\d+
>>       {end_string_suffix}'''.format(
>>         start_string_prefix=self.start_string_prefix,
>>         end_string_suffix=self.end_string_suffix),
>>       re.VERBOSE | re.UNICODE)
>>
>>     self.implicit_dispatch.append((issue_pattern, self.issue_reference))
>>
>>   def issue_reference(self, match, lineno):
>>     text = match.group(0)
>>
>>     rawsource = unescape(text, True)
>>     text = unescape(text, False)
>>
>>     refuri = 'https://issues.apache.org/jira/browse/' + text
>>
>>     return [nodes.reference(rawsource, text, refuri=refuri)]
>>
>> states.Inliner = Inliner
> 
> That last line is the culprit. You're monkey-patching Docutils. Don't
> do that. If something in Docutils breaks as a result, you can't
> reasonably complain. If it worked before, it was just accidental.

To make everything clear, I'm *not* the original author of this code,
but I completely agree with you :)

I'm "just" trying to fix trafficserver build against docutils 0.13.1 so
that trafficserver package isn't removed from Debian archive (stable
freeze is coming).
Right now, I only have 2 week left to make the build works in Debian.

> I distilled the issue down, see the attached x.py & y.py. Put them on
> your PYTHONPATH so y.py can import x.py. Run y.py. Note the output.
> Now edit y.py and uncomment the commented line (the monkey patch). Now
> run y.py again, and note the difference in the output. The monkey
> patch messes up the namespace hierarchy, so x.A.run doesn't have
> access to x.attr_x anymore.
> 
> Wasn't there another version of this code that worked in a different way?

I tried many things which never worked.

I finally get the build working with method level monkey patch as well
as your own patch Inliner8009.patch.

But, since a successful build requires docutils to be updated, I'll
suggest to trafficserver devs to change the way it works and use the
classic docutils role way, which will be more stable.

> Maybe you can monkey patch at the method level instead of the class
> level. I.e. inject your methods into the existing Docutils Inliner
> class, instead of replacing the class with your subclass. See z.py.
> Caveat: monkey patching is dangerous. Even if this works now, I can't
> guarantee that it will always work.
> You're trying to do this through Sphinx, which isn't configurable, as
> the comment in https://github.com/apache/trafficserver/blob/master/doc/conf.py
> describes:
> 
> # Customize parser.inliner in the only way that Sphinx supports.
> # docutils.parsers.rst.Parser takes an instance of states.Inliner or a
> # subclass but Sphinx initializes it from
> # SphinxStandaloneReader.set_parser('restructuredtext') which is called
> # from Publisher.set_components() and initializes the parser without
> # arguments.
> 
> Perhaps you should complain to Sphinx, and ask them to enable
> configurability. If your code was using Docutils directly this
> wouldn't be an issue (or would be less of an issue).

Which shouldn't be needed if trafficserver switch to classic docutils roles.

Thanks for your help and patience, I've learned a lot !

Cheers,
Jean Baptiste Favre

Re: [Docutils-users] Migration docutils 0.12 to 0.13

[David G. <go...@py...>]

On Tue, Jan 3, 2017 at 4:07 AM, Jean Baptiste Favre
<web...@jb...> wrote:
> I finally get the build working with method level monkey patch as well
> as your own patch Inliner8009.patch.
>
> But, since a successful build requires docutils to be updated, I'll
> suggest to trafficserver devs to change the way it works and use the
> classic docutils role way, which will be more stable.

Please give us a link to that discussion. I'd like to see how the decision goes.

Thanks,

David Goodger
<http://python.net/~goodger>

Re: [Docutils-users] Migration docutils 0.12 to 0.13

[jean B. F. <web...@jb...>]

On 04/01/2017 06:23, David Goodger wrote:
> On Tue, Jan 3, 2017 at 4:07 AM, Jean Baptiste Favre
> <web...@jb...> wrote:
>> I finally get the build working with method level monkey patch as well
>> as your own patch Inliner8009.patch.
>>
>> But, since a successful build requires docutils to be updated, I'll
>> suggest to trafficserver devs to change the way it works and use the
>> classic docutils role way, which will be more stable.
> 
> Please give us a link to that discussion. I'd like to see how the decision goes.
> 
> Thanks,
> 
> David Goodger
> <http://python.net/~goodger>
> 

Discusion started in december here:
https://mail-archives.apache.org/mod_mbox/trafficserver-dev/201612.mbox/%3c6...@jb...%3e

And continued in January here:
https://mail-archives.apache.org/mod_mbox/trafficserver-dev/201701.mbox/thread

Bug report is here:
https://issues.apache.org/jira/browse/TS-5107

And Github PR here:
https://github.com/apache/trafficserver/pull/1296

Regards,
Jean Baptiste

Re: [Docutils-users] Migration docutils 0.12 to 0.13

[Guenter M. <mi...@us...>]

Dear David,

>> Is the patch still "on the table"?

> Yes, but I'm looking at making more extensive changes first. The patch
> that implemented the changes to support
> settings.character_level_inline_markup really made a mess of things:

...

> I'll work on all these and propose another patch.

thank you

Günter