[eXist-commits] [eXist-db/exist] a2f89d: [bugfix] Treat an '@' in xqdoc prose as text rathe...

[Duncan P. <no...@gi...>]

  Branch: refs/heads/develop
  Home:   https://github.com/eXist-db/exist
  Commit: a2f89d3252a343d49d4308134a9b075e02bf9780
      https://github.com/eXist-db/exist/commit/a2f89d3252a343d49d4308134a9b075e02bf9780
  Author: Joe Wicentowski <jo...@gm...>
  Date:   2026-09-15 (Tue, 15 Sep 2026)

  Changed paths:
    M exist-core/src/main/antlr/org/exist/xquery/xqdoc/parser/XQDocParser.g
    M exist-core/src/test/java/org/exist/xquery/functions/inspect/InspectModuleTest.java

  Log Message:
  -----------
  [bugfix] Treat an '@' in xqdoc prose as text rather than a tag

inspect:inspect-module dropped everything from the first '@' in an xqdoc comment to the
end of that description, with no error raised -- the truncated text came back as if
complete. "Selects taxonomy[@type = "reign"] from the source." was returned as
"Selects taxonomy[".

Every consumer of xqdoc descriptions is affected, because inspect:inspect-module is the
ecosystem's only introspection entry point: published function documentation, editor
hover text and autocomplete descriptions all degrade at once, and because it fails
silently nobody notices until a sentence reads oddly.

The lexer's CHARS rule excludes '@', so prose can never contain one. What follows depends
on the next character: '@' plus alphanumerics lexes as TAG, and the parser's
( TAG ) => taggedContents predicate then opens a new xqdoc tag mid-sentence, filing the
rest of the paragraph under a tag nobody reads; a bare '@' emits AT, which no parser rule
accepted at all.

A real xqdoc tag only ever appears at the start of a comment line, immediately after the
leading " : ". The contents rule now tracks that position and treats '@' anywhere else as
ordinary text: a guarded alternative appends a mid-prose TAG, and a new alternative
appends a bare AT. A line-start TAG still matches no alternative, so contents ends there
and the outer predicate fires exactly as before.

Fixing this in the parser rather than the lexer is deliberate. Making TAG match only at
line start needs either mutable lexer state or '@' folded into CHARS, and the latter
creates a genuine CHARS/TAG ambiguity in a single-DFA ANTLR 2 lexer. The parser already
has the information -- it can see that only whitespace has been consumed since the line's
':' -- so no state has to be invented, and tokenization is unchanged, which keeps
@param/@return recognition from regressing through a lexer path that was not anticipated.

ANTLR reports no new class of warning. The existing "nondeterminism ... between alt N and
exit branch of block" warnings on this closure go from three to five only because the
closure gained two alternatives; the "Syntactic predicate ignored" warning belongs to the
untouched SIMPLE_COLON lexer rule.

InspectModuleTest.withAtSignInline was already present and @Ignore'd against this issue;
it is now enabled. Added alongside it: an '@' mid-description, a bare '@' with an email
address and a trailing '@', and a function whose @param and @return values themselves
contain '@' -- the last being the regression case, since it pins that tags are still
parsed as tags. Four of the seven fail without this change.

The '%' half of the issue is already resolved on develop: the existing onAnnotatedFunction
test inspects a %public %x:path(...) function and passes unchanged.

Closes https://github.com/eXist-db/exist/issues/1386

Co-Authored-By: Claude Opus 5 <no...@an...>


  Commit: 773d2dbb2035edd99014f4fcafde4e69f0daff6c
      https://github.com/eXist-db/exist/commit/773d2dbb2035edd99014f4fcafde4e69f0daff6c
  Author: Joe Wicentowski <jo...@gm...>
  Date:   2026-09-15 (Tue, 15 Sep 2026)

  Changed paths:
    M exist-core/src/main/antlr/org/exist/xquery/xqdoc/parser/XQDocParser.g
    M exist-core/src/main/java/org/exist/xquery/xqdoc/XQDocHelper.java
    M exist-core/src/test/java/org/exist/xquery/functions/inspect/InspectModuleTest.java

  Log Message:
  -----------
  [bugfix] Only recognize an xqdoc tag when it is one xqDoc defines

Raised in review: a line of prose can legitimately open with an '@', and the previous
commit's line-start rule was not enough on its own to tell "@home is where the heart is."
from "@since 1.0".

The consequence was worse than the line simply vanishing from the description. Every tag
that is not @param or @return goes into XQDocHelper's generic meta map, and
InspectFunctionHelper turns each of those into an element named after the tag:

    builder.startElement(new QName(meta.getKey(), XMLConstants.NULL_NS_URI), null);

So a prose line opening "@home ..." was removed from the description and re-emitted as
<home>, and "@2024 ..." produced an element named 2024 -- not a valid XML NCName -- with
no error raised anywhere.

Tag recognition now requires both the line-start position and membership in the set of
tags the xqDoc specification defines. Everything else at line start is prose, which is
what an author writing an email address or a year at the start of a line intends.

Measured before choosing a closed set: across all 304 XQuery source files in this
repository, the only line-start xqdoc tags in use are @see (94), @return (45), @author
(24), @param (21), and one @returns typo. Nothing here depends on an open tag set.

BEHAVIOR CHANGE worth noting for anyone relying on it: a custom tag such as @mycompany no
longer becomes queryable metadata on inspect:inspect-module -- it stays in the
description as prose. Nothing in this repository uses one. The single @returns typo now
surfaces in the description, where its author can see it, rather than as a <returns>
element nothing reads.

As a side effect the ANTLR nondeterminism warnings on this closure drop from five back to
three, matching the pre-change baseline exactly: the semantic predicate now disambiguates
the TAG and AT alternatives that previously conflicted with the loop exit.

Co-Authored-By: Claude Opus 5 <no...@an...>


  Commit: 645148a988809cc2c03791a32889a87aba8a84d3
      https://github.com/eXist-db/exist/commit/645148a988809cc2c03791a32889a87aba8a84d3
  Author: Joe Wicentowski <jo...@gm...>
  Date:   2026-09-15 (Tue, 15 Sep 2026)

  Changed paths:
    M exist-core/src/main/java/org/exist/xquery/xqdoc/XQDocHelper.java

  Log Message:
  -----------
  [bugfix] Align the xqdoc tag set with the specification and the shipped schema

The previous commit's tag set was assembled from memory and was wrong: it included
@example and @library, neither of which is an xqDoc tag. "library" appears in
extensions/xqdoc/xqdoc-1.0.xsd only as a module-type enumeration value (library vs main),
not as a comment directive.

Both authorities agree on exactly eight tags -- @author, @version, @param, @return,
@error, @deprecated, @see, @since:

  - the xqdoc-1.0.xsd shipped in this repository, whose "comment" complexType is a closed
    xsd:sequence of exactly those eight elements plus "description" (the xsd:any inside
    mixed-text permits embedded HTML within a value, not additional directives);
  - the specification at https://xqdoc.org/xqdoc_comments_doc.html, which lists the same
    eight.

That the schema's sequence is closed also answers the open question left by the previous
commit. A custom tag such as @mycompany was never representable in xqDoc: it could not
validate against xqdoc-1.0.xsd. So restricting recognition to these eight is not a
behavior regression for any conforming document -- it brings the parser into line with
the format it implements.

Note the specification does not explicitly require a tag to stand at the start of a
comment line; its examples all show one there, after the leading " : ". The line-start
rule this parser applies is a reading of those examples rather than a stated requirement.
What the specification does state -- "the beginning text (up to the first tag) is assumed
to be description text" -- is why mis-recognizing a word as a tag truncates the
description, and why getting the set right matters.

Co-Authored-By: Claude Opus 5 <no...@an...>


  Commit: 9e51fdf62f20913bb925a3b06aa9957c224f9d94
      https://github.com/eXist-db/exist/commit/9e51fdf62f20913bb925a3b06aa9957c224f9d94
  Author: Duncan Paterson <dun...@us...>
  Date:   2026-09-16 (Wed, 16 Sep 2026)

  Changed paths:
    M exist-core/src/main/antlr/org/exist/xquery/xqdoc/parser/XQDocParser.g
    M exist-core/src/main/java/org/exist/xquery/xqdoc/XQDocHelper.java
    M exist-core/src/test/java/org/exist/xquery/functions/inspect/InspectModuleTest.java

  Log Message:
  -----------
  Merge pull request #6718 from joewiz/fix/xqdoc-at-truncation


Compare: https://github.com/eXist-db/exist/compare/cbcd4be5f07a...9e51fdf62f20

To unsubscribe from these emails, change your notification settings at https://github.com/eXist-db/exist/settings/notifications