|
From: Duncan P. <no...@gi...> - 2026-10-13 19:52:08
|
Branch: refs/heads/develop Home: https://github.com/eXist-db/exist Commit: 4e0374e54b4fa65caca6d931110b83271dd8b77b https://github.com/eXist-db/exist/commit/4e0374e54b4fa65caca6d931110b83271dd8b77b Author: Joe Wicentowski <jo...@gm...> Date: 2026-10-13 (Tue, 13 Oct 2026) Changed paths: M exist-core/src/main/java/org/exist/dom/memtree/DocumentBuilderReceiver.java A exist-core/src/test/java/org/exist/dom/memtree/DocumentBuilderReceiverCDataTest.java Log Message: ----------- [bugfix] Preserve CDATA sections when parsing an XML request body request:get-data() silently turned every CDATA section in an XML request body into a plain text node, so the content came back escaped on the way out: "> Test <" inside a <script> block was stored as "> Test <". This is the remaining half of https://github.com/eXist-db/exist/issues/2081, open since 2018 -- the REST, WebDAV and XML-RPC paths were fixed earlier because they store the raw bytes and never take this route. A SAX parser reports the *content* of a CDATA section through characters() like any other text and marks the boundaries only through the LexicalHandler callbacks. DocumentBuilderReceiver implements LexicalHandler but had both boundary callbacks as explicit no-ops: public void startCDATA() { /* no-op */ } public void endCDATA() { /* no-op */ } so the boundaries were discarded and the content was built as text. Nothing downstream can recover the distinction once it is gone. SAXAdapter -- the sibling adapter into the same MemTreeBuilder, backing fn:parse-xml -- already did this correctly: set a flag on startCDATA, buffer characters() while it is set, flush through builder.cdataSection() on endCDATA. This applies the same pattern to DocumentBuilderReceiver, so the two adapters agree on the same bytes. Note the builder already had everything needed: memtree has CDATASectionImpl, MemTreeBuilder has cdataSection(), and DocumentBuilderReceiver even had a working cdataSection() Receiver method -- only the SAX entry points were stubbed out, so a Receiver-to-Receiver copy kept CDATA while a parse did not. The fix reaches every caller of request:get-data(), not just the one that surfaced it. The route by which it surfaced: a roaster-based app (eXide) PUTs an XML body, roaster's body:parse "xml" branch calls request:get-data(), and the CDATA is gone before any application code sees it. LIMITATION, stated so the fix is not read as more than it is: a CDATA section sandwiched between two text runs still coalesces into a single text node. MemTreeBuilder.cdataSection() appends to a preceding text node by design ("XML does not allow adjacent text nodes"), and SAXAdapter behaves identically -- so this is pre-existing builder behavior, not something introduced here, and the two adapters still agree. Issue #2081's case is unaffected: an element whose entire content is one CDATA section has no preceding text node to coalesce with. DocumentBuilderReceiverCDataTest covers the regression (a section must build as CDATA_SECTION_NODE), parity between the two adapters on identical bytes, that ordinary text still builds as a text node, and that no character is dropped around a section. Two of the four fail against the unfixed receiver; the parity one is the sharpest, since it fails even though SAXAdapter alone would pass. Closes https://github.com/eXist-db/exist/issues/2081 Co-Authored-By: Claude Opus 5 <no...@an...> Commit: fa6519d994d633fcf918d8ad573542c16de79c2c https://github.com/eXist-db/exist/commit/fa6519d994d633fcf918d8ad573542c16de79c2c Author: Joe Wicentowski <jo...@gm...> Date: 2026-10-13 (Tue, 13 Oct 2026) Changed paths: M exist-core/src/main/java/org/exist/storage/serializers/EXistOutputKeys.java M exist-core/src/main/java/org/exist/util/serializer/XMLWriter.java M exist-core/src/main/java/org/exist/xquery/util/SerializerUtils.java A exist-core/src/test/java/org/exist/util/serializer/PreserveCdataSerializationTest.java Log Message: ----------- [feature] Add exist:preserve-cdata so a stored CDATA section can be serialized back The second half of https://github.com/eXist-db/exist/issues/2081. With the request-parsing fix in the preceding commit a CDATA section now reaches storage intact -- and that exposed the other end: there was no way to get it back out. fn:serialize escaped it, so an editor that reads a document, shows it, and writes it back turned `>` into `>` inside <script> and <style> blocks on open, whatever the document looked like in the database. XDM has no CDATA node kind, so fn:serialize is specified to escape such content unless the containing element is named in cdata-section-elements. That is correct, and this does not change it. But it left eXist unable to hand back what it had stored: the DOM does keep CDATA nodes and eXist's own REST serializer emits them -- fetch a document over REST and the section is there -- that path was simply unreachable from fn:serialize. Naming elements in cdata-section-elements is not a substitute: it requires knowing them in advance and imposes CDATA on elements that never had it, which is the same silent rewriting this issue is about. exist:preserve-cdata is opt-in and defaults to false, so unqualified fn:serialize behavior is untouched. It follows exist:expand-xincludes, which is already an eXist-specific serialization parameter used for the same kind of need. Three changes are needed, and the third is easy to miss: - EXistOutputKeys gains PRESERVE_CDATA. - XMLWriter honors it at the two gates in startCdataSection()/endCdataSection() that otherwise suppress the delimiters whenever XDM serialization is active. The rest of the machinery was already in place: characters() writes raw while inside a section. - SerializerUtils.ExistParameterConvention must list it. That enum is a closed whitelist, and an eXist-namespace serialization parameter that is not in it is *silently discarded* rather than rejected -- so the first version of this change looked like a fix that did nothing. Worth a separate issue: a typo in an exist: serialization parameter today fails quietly. Verified end to end, not only in unit tests: an eXist built from this branch, with eXide asking for the parameter in db:load-document and sending its editor buffer as bytes, takes eXide's CDATA round-trip test from failing to passing. The same spec fails against a stock build. PreserveCdataSerializationTest pins both directions: the section is preserved with the parameter and escaped without it, explicit false() matches the default, it applies to in-memory as well as stored documents, and a document with no CDATA serializes identically either way. Two of its five cases fail without this change; the three asserting unchanged default behavior pass either way, which is the point. Co-Authored-By: Claude Opus 5 <no...@an...> Commit: a51dd586a730eaeaf998c4f8ffd80bda6fe7d05d https://github.com/eXist-db/exist/commit/a51dd586a730eaeaf998c4f8ffd80bda6fe7d05d Author: Joe Wicentowski <jo...@gm...> Date: 2026-10-13 (Tue, 13 Oct 2026) Changed paths: A exist-core/src/main/java/org/exist/dom/memtree/CDataSectionBuffer.java M exist-core/src/main/java/org/exist/dom/memtree/DocumentBuilderReceiver.java M exist-core/src/main/java/org/exist/dom/memtree/SAXAdapter.java Log Message: ----------- [refactor] Share one CDATA reassembly buffer between the two SAX handlers Review point from @duncdrum on #6729: the CDATA handling was duplicated across consumers. It was -- the preceding commit gave DocumentBuilderReceiver its own copy of the flag, buffer and three methods that SAXAdapter already had, character for character. Both classes build an in-memory DOM from SAX, and both need the same thing: a SAX parser reports the content of a CDATA section through characters() like any other text and marks the boundaries only through the LexicalHandler callbacks, so rebuilding the section means buffering between startCDATA() and endCDATA() and handing the result to MemTreeBuilder.cdataSection() in one piece. CDataSectionBuffer now holds that, and both handlers delegate to it. This is not only tidiness: these two adapters disagreeing about CDATA is exactly what issue #2081 was, and one of them silently lost sections for years while the other did not. Keeping the logic in one place is what stops them drifting apart again, and the class javadoc says so. No behavior change; 1,035 tests green across org.exist.dom.memtree.*, the serializer tests and the full src/test/xquery corpus. Co-Authored-By: Claude Opus 5 <no...@an...> Commit: 6c200fb8da7914298e8b45b2cf699f310a81aead https://github.com/eXist-db/exist/commit/6c200fb8da7914298e8b45b2cf699f310a81aead Author: Joe Wicentowski <jo...@gm...> Date: 2026-10-13 (Tue, 13 Oct 2026) Changed paths: M exist-core/src/main/java/org/exist/storage/serializers/EXistOutputKeys.java M exist-core/src/test/java/org/exist/util/serializer/PreserveCdataSerializationTest.java Log Message: ----------- [ignore] Use US spelling in the preserve-cdata documentation The project convention is US English; two Javadoc comments added with the preserve-cdata parameter used "behaviour". Comment-only change. Co-Authored-By: Claude Opus 5 <no...@an...> Commit: 787969d94b4985d75ea6bbc87941bd06938b5506 https://github.com/eXist-db/exist/commit/787969d94b4985d75ea6bbc87941bd06938b5506 Author: Joe Wicentowski <jo...@gm...> Date: 2026-10-13 (Tue, 13 Oct 2026) Changed paths: M exist-core/src/main/java/org/exist/xquery/CDATAConstructor.java M exist-core/src/test/java/org/exist/util/serializer/PreserveCdataSerializationTest.java Log Message: ----------- [bugfix] Scope preserve-cdata to CDATA sections that were in the source A CDATA section written in query source is an escaping convenience: it lets an author type < and > without entity references. XDM has no CDATA node kind, so what it contributes is character content, indistinguishable from the same characters typed directly. eXist was building a CDATA node for it anyway, which was invisible while serialization always escaped, but becomes visible under exist:preserve-cdata. That visibility is wrong in a way the test suites catch. With preserve-cdata on, serializing <style><![CDATA[ul > li { color:red; }]]></style> as HTML5 emitted the CDATA section into the output, where a browser reads it as literal CSS -- breaking the very case the parameter exists to serve. The W3C case Constr-ws-gencdata-1 fails the same way. Build a text node instead, so a CDATA node in eXist's trees means what preserve-cdata needs it to mean: a CDATA section that was present in a parsed document, not one an author typed into a query. Serialization is otherwise unchanged. Content is escaped either way, and cdata-section-elements still emits CDATA for a named element, because XMLWriter.characters applies that parameter to text nodes. Co-Authored-By: Claude Opus 5 <no...@an...> Commit: a99a8c6a4a90c2938d2394e1d3eddf85be0fd3f6 https://github.com/eXist-db/exist/commit/a99a8c6a4a90c2938d2394e1d3eddf85be0fd3f6 Author: Joe Wicentowski <jo...@gm...> Date: 2026-10-13 (Tue, 13 Oct 2026) Changed paths: M exist-core/src/main/java/org/exist/storage/serializers/EXistOutputKeys.java M exist-core/src/main/java/org/exist/util/serializer/XMLWriter.java M exist-core/src/main/java/org/exist/xquery/util/SerializerUtils.java M exist-core/src/test/java/org/exist/dom/persistent/PersistentDomTest.java M exist-core/src/test/java/org/exist/util/serializer/PreserveCdataSerializationTest.java M exist-core/src/test/java/org/exist/xquery/XPathQueryTest.java M exist-core/src/test/java/org/exist/xquery/XQueryTest.java Log Message: ----------- [feature] Default exist:preserve-cdata to on Requested by review on #6729: a long-standing bug is being fixed in 7, so implementations depending on the bugged behavior should change and everybody else should get the fixed behavior out of the box. This is a deliberate divergence from the serialization specification, which emits CDATA sections only for elements named in cdata-section-elements. It is on because the alternative is more surprising: retrieving a stored document already preserved its CDATA sections while returning the same document from a query did not, so a document round-tripped through an editor lost them. Setting the parameter to no restores strictly specified output, and the divergence is recorded on EXistOutputKeys.PRESERVE_CDATA. Measured on a branch before making the change: XQTS is unaffected -- 28,846 passed against a develop baseline of 28,846, with no test newly failing. That is possible only because the parameter is scoped to CDATA that was present in a parsed or stored document; defaulting it on without that scoping regressed Constr-ws-gencdata-1 and emitted CDATA sections into HTML5 script and style elements. Updates the four tests that asserted the old default on stored or parsed documents, and adds coverage for the explicit opt-out. cdataComputedDom keeps its escaped expectation, with a comment on why: its section is written in query source, so the parameter does not apply to it. Co-Authored-By: Claude Opus 5 <no...@an...> Commit: a484b7b7df04f6d40aa82629756cff60f7636ce7 https://github.com/eXist-db/exist/commit/a484b7b7df04f6d40aa82629756cff60f7636ce7 Author: Joe Wicentowski <jo...@gm...> Date: 2026-10-13 (Tue, 13 Oct 2026) Changed paths: M exist-core/src/test/java/org/exist/dom/memtree/DocumentBuilderReceiverCDataTest.java M exist-core/src/test/java/org/exist/util/serializer/PreserveCdataSerializationTest.java Log Message: ----------- [test] Migrate the CDATA tests to JUnit 5 DocumentBuilderReceiverCDataTest and PreserveCdataSerializationTest were written against JUnit 4, which exist-core no longer runs: it compiles, but the vintage engine is banned, so the tests would silently not run. Co-Authored-By: Claude Opus 5.5 (1M context) <no...@an...> Claude-Session: https://claude.ai/code/session_01TLXqAfAP6kHAcWM99TFrxa Commit: f69954f4b9b24de96519ee4078eb228ba9e3d491 https://github.com/eXist-db/exist/commit/f69954f4b9b24de96519ee4078eb228ba9e3d491 Author: Joe Wicentowski <jo...@gm...> Date: 2026-10-13 (Tue, 13 Oct 2026) Changed paths: M exist-core/src/main/java/org/exist/util/serializer/XMLWriter.java M exist-core/src/test/java/org/exist/util/serializer/PreserveCdataSerializationTest.java Log Message: ----------- [bugfix] Keep preserve-cdata from writing CDATA into HTML elements With exist:preserve-cdata on, fn:serialize with the html method wrote a CDATA section from a parsed or stored document as a literal <![CDATA[ inside <script> and <style>, which a browser reads as script or CSS text. HTML has no CDATA sections (Serialization 3.1 section 7.1). Apply the parameter only where shouldUseCdataSections() allows a section, as XHTMLWriter already does for cdata-section-elements: not in HTML elements under the html method, but in XML islands such as SVG, and under the xml and xhtml methods as before. Co-authored-by: duncdrum <d.p...@me...> Co-Authored-By: Claude Opus 5.5 (1M context) <no...@an...> Claude-Session: https://claude.ai/code/session_01TLXqAfAP6kHAcWM99TFrxa Commit: d51467d473ce45ed96252d4d0291b75cb2ac52c7 https://github.com/eXist-db/exist/commit/d51467d473ce45ed96252d4d0291b75cb2ac52c7 Author: duncdrum <d.p...@me...> Date: 2026-10-13 (Tue, 13 Oct 2026) Changed paths: M exist-core/src/main/java/org/exist/util/serializer/XHTMLWriter.java M exist-core/src/main/java/org/exist/util/serializer/XMLWriter.java A exist-core/src/test/java/org/exist/util/serializer/HtmlCdataWriterTest.java Log Message: ----------- [bugfix] Keep CDATA out of HTML elements when retrieving with the html output method Retrieving a stored document is not XDM serialization, so XMLWriter wrote every CDATA node as a CDATA section whatever the output method. Under the html method that put literal <![CDATA[ markers inside script and style elements, where a browser reads them as script or CSS text and the script breaks. Serialization 3.1 section 7.1 requires the html method to output <script><![CDATA[if (a < b) foo()]]></script> as <script>if (a < b) foo()</script>, and section 7.4.4 makes CDATA handling inapplicable to the html method except in XML islands. Add a supportsCdataSections() hook to XMLWriter, true by default, and override it in XHTMLWriter to return false for HTML elements under the html method. The non-XDM gate now consults it. XML islands such as SVG and MathML, and the xml and xhtml methods, keep their CDATA sections. This changes existing retrieval output: a stored CDATA section in a script element fetched with the html method is now returned as raw text. Closes https://github.com/eXist-db/exist/issues/6810 Co-Authored-By: Claude Sonnet 5.5 <no...@an...> Commit: 5fa2b05a653d7c433d71d0ec5f666be8f0ed5b2d https://github.com/eXist-db/exist/commit/5fa2b05a653d7c433d71d0ec5f666be8f0ed5b2d Author: duncdrum <d.p...@me...> Date: 2026-10-13 (Tue, 13 Oct 2026) Changed paths: A exist-core/src/test/java/org/exist/xmlrpc/GetDocumentHtmlCdataTest.java Log Message: ----------- [test] Cover html-method retrieval over XML-RPC getDocument with an html method option goes through the same non-XDM writer path as the unit test covers, but through the real XML-RPC servlet. Check that a stored CDATA section in a script element comes back as raw text under the html method, and still as a CDATA section when no method, xml or xhtml is requested. Without the writer change the html case returns <script><![CDATA[if (a < b) f()]]></script>. Refs https://github.com/eXist-db/exist/issues/6810 Co-Authored-By: Claude Sonnet 5.5 <no...@an...> Commit: d9b854559dd3cd7c0094ab0c472aed8e2d17234d https://github.com/eXist-db/exist/commit/d9b854559dd3cd7c0094ab0c472aed8e2d17234d Author: Duncan Paterson <dun...@us...> Date: 2026-10-13 (Tue, 13 Oct 2026) Changed paths: A exist-core/src/main/java/org/exist/dom/memtree/CDataSectionBuffer.java M exist-core/src/main/java/org/exist/dom/memtree/DocumentBuilderReceiver.java M exist-core/src/main/java/org/exist/dom/memtree/SAXAdapter.java M exist-core/src/main/java/org/exist/storage/serializers/EXistOutputKeys.java M exist-core/src/main/java/org/exist/util/serializer/XHTMLWriter.java M exist-core/src/main/java/org/exist/util/serializer/XMLWriter.java M exist-core/src/main/java/org/exist/xquery/CDATAConstructor.java M exist-core/src/main/java/org/exist/xquery/util/SerializerUtils.java A exist-core/src/test/java/org/exist/dom/memtree/DocumentBuilderReceiverCDataTest.java M exist-core/src/test/java/org/exist/dom/persistent/PersistentDomTest.java A exist-core/src/test/java/org/exist/util/serializer/HtmlCdataWriterTest.java A exist-core/src/test/java/org/exist/util/serializer/PreserveCdataSerializationTest.java A exist-core/src/test/java/org/exist/xmlrpc/GetDocumentHtmlCdataTest.java M exist-core/src/test/java/org/exist/xquery/XPathQueryTest.java M exist-core/src/test/java/org/exist/xquery/XQueryTest.java Log Message: ----------- Merge pull request #6729 from joewiz/bugfix/2081-cdata-request-get-data Compare: https://github.com/eXist-db/exist/compare/957f4a0cbd20...d9b854559dd3 To unsubscribe from these emails, change your notification settings at https://github.com/eXist-db/exist/settings/notifications |