#520 SmartQuotes misclassifies an opening single quote before a converted ellipsis

[Jared Dillard]

When smart quotes and ellipsis conversion are enabled, a straight single quote before ... is converted to a closing quotation mark instead of an opening quotation mark.

Input:

Add secondary prompts ('...') to the sidebar.

Actual output:

Add secondary prompts (’…’) to the sidebar.

Expected output:

Add secondary prompts (‘…’) to the sidebar.

In the actual output, both quotes are U+2019 RIGHT SINGLE QUOTATION MARK. The opening quote should be U+2018 LEFT SINGLE QUOTATION MARK.

Minimal reproduction

from docutils.core import publish_parts

source = "Add secondary prompts ('...') to the sidebar.\n"
parts = publish_parts(
    source,
    writer_name="html5",
    settings_overrides={
        "smart_quotes": True,
        "language_code": "en",
    },
)
print(parts["fragment"].strip())

Output:

<p>Add secondary prompts (’…’) to the sidebar.</p>

Environment

• Python: 3.12.7
• Docutils: 0.22.3

Additional context

This was originally reported in Sphinx issue #9713. Docutils 0.17 fixed the separate quote-classification cases involving opening or separator characters such as [, -, and +. Those cases work with Docutils 0.22.3, but the ellipsis case above remains reproducible. I'm not sure if this is an issue worth fixing, but wanted to pass it along.

The behavior seems to be related to conversion order: ... becomes U+2026 HORIZONTAL ELLIPSIS before the neighboring quote is classified.

[Günter Milde]

Thanks for the report.

IMO, the issue is "SmartQuotes misclassifies an opening quote before non-ASCII punctuation when there are bracktes around".

Test sample:

The misclassification happens also in other cases:
  ('...') ("...") ('--') ("--") ('---') ("---") ASCII patterns

  ('…') ("…") ('–') ("–") ('—') ("—") ("—") Unicode punctuation


* No difference between single and double quotes
* No difference between smartquote patterns or direct Unicode input.

Word characters:
('a') ('b') ('c') ('D') ASCII
('ä') ('Ö') ('ß') ('λ') non-ASCII

ASCII punctuation and non-word characters:
('!') ('?') (';') (':') ('.') (',') ('/') ('#') ('+') ('-') ('*') ('=')
('^') ('"') ('%') ('$') but ('&')

non-ASCII punctuation:
('¡') ('¿') ('‧') ('‰') ('⁋') ('∀')


* ASCII punctuation (except the ampersand) is OK,
  non-ASCII punctuation is problematic.

OTOH, without brackets, everything is fine:
  '.' '&' '...' '--' '¡' '¿' '‧' '‰' '⁋'.

There are other misclassifications, e.g., the quote after an EM-DASH at the
end of a paragraph:

ASCII pattern ('---') ('---')

Unicode ('—') ('—')

I'm not sure if this is an issue worth fixing, [...]

Nor am I, "smartquotes" is intended as a simple helper and will always have its limitations (cf. caveats).