exist-open — Open discussion

Re: [Exist-open] xmldb:collection-available not reporting existence of collection named "Å"

[Henrik N. <hen...@gm...>]

In case others might find this useful, I did find the solution:

http://localhost:8080/exist/rest/db/apps/irhb/?_query=xmldb:collection-available(xmldb:encode(%22/db/apps/irhb/data/originators/Å%22))
<http://localhost:8080/exist/rest/db/apps/irhb/?_query=xmldb:collection-available(xmldb:encode(%22/db/apps/irhb/data/originators/%C3%85%22))>

Henrik


On Tue, 4 Nov 2025 at 23:04, Henrik Nielsen <hen...@gm...> wrote:

>
> http://localhost:8080/exist/rest/db/apps/irhb/?_query=xmldb:collection-available(%22/db/apps/irhb/data/originators/A%22)
>
> *returns*
>
> <exist:result xmlns:exist="http://exist.sourceforge.net/NS/exist"
> exist:hits="1" exist:start="1" exist:count="1" exist:compilation-time="1"
> exist:execution-time="1">
> <exist:value exist:type="xs:boolean">*true*</exist:value>
> </exist:result>[,]
>
> *but*
>
>
> http://localhost:8080/exist/rest/db/apps/irhb/?_query=xmldb:collection-available(%22/db/apps/irhb/data/originators/Å%22)
> <http://localhost:8080/exist/rest/db/apps/irhb/?_query=xmldb:collection-available(%22/db/apps/irhb/data/originators/%C3%85%22)>
>
> *returns *
>
> <exist:result xmlns:exist="http://exist.sourceforge.net/NS/exist"
> exist:hits="1" exist:start="1" exist:count="1" exist:compilation-time="0"
> exist:execution-time="0">
> <exist:value exist:type="xs:boolean">*false*</exist:value>
> </exist:result>[.]
>
> I did create the collection "Å" and can confirm its existence as well as
> access documents in it via eXide and oXygen.
>
> Wrapping the parameter of xmldb:collection-available in util:unescape-uri(
> , "UTF-8"), xmldb:decode() or xmldb:decode-uri() doesn't help. Replacing
> "Å" in the URL with "%C3%85" also doesn't do the trick. How can I make xmldb:collection-available
> recognize the collection and others with name outside the a-z, A-Z ranges?
>
> Any help would be much appreciated,
>
> Henrik
>
>
>
>

[Exist-open] xmldb:collection-available not reporting existence of collection named "Å"

[Henrik N. <hen...@gm...>]

http://localhost:8080/exist/rest/db/apps/irhb/?_query=xmldb:collection-available(%22/db/apps/irhb/data/originators/A%22)

*returns*

<exist:result xmlns:exist="http://exist.sourceforge.net/NS/exist" exist:hits
="1" exist:start="1" exist:count="1" exist:compilation-time="1"
exist:execution-time="1">
<exist:value exist:type="xs:boolean">*true*</exist:value>
</exist:result>[,]

*but*

http://localhost:8080/exist/rest/db/apps/irhb/?_query=xmldb:collection-available(%22/db/apps/irhb/data/originators/Å%22)
<http://localhost:8080/exist/rest/db/apps/irhb/?_query=xmldb:collection-available(%22/db/apps/irhb/data/originators/%C3%85%22)>

*returns *

<exist:result xmlns:exist="http://exist.sourceforge.net/NS/exist" exist:hits
="1" exist:start="1" exist:count="1" exist:compilation-time="0"
exist:execution-time="0">
<exist:value exist:type="xs:boolean">*false*</exist:value>
</exist:result>[.]

I did create the collection "Å" and can confirm its existence as well as
access documents in it via eXide and oXygen.

Wrapping the parameter of xmldb:collection-available in util:unescape-uri(
, "UTF-8"), xmldb:decode() or xmldb:decode-uri() doesn't help. Replacing
"Å" in the URL with "%C3%85" also doesn't do the trick. How can I make
xmldb:collection-available
recognize the collection and others with name outside the a-z, A-Z ranges?

Any help would be much appreciated,

Henrik

Re: [Exist-open] Index field - Computed field function access to the same collection?

[Jean-Paul R. <re...@gm...>]

Dear Lars,

There are two possible issues at work.

The first is subtle and I only discovered it over time: computed fields in
eXist-db use the Lucene index, which in turn requires a text() node to be
available on the element containing an attribute you may wish to index. A
sample example demonstrates this:

<item corresp="foo"/>  -> cannot compute a field from the value in @corresp
<item corresp="foo">bar</item> -> can compute a field from the value
in @corresp

This may or may not affect your indexing situation.

Compounded with this is a second issue that you indicate. An index being
performed on a given document (say "X") cannot access the rest of the
contents held in the same immediate collection that contains "X". Thus
computed fields fail to populate when referring to documents in the same
immediate collection. My own work-around is to create separate collections
for documents which may be needed for indexing. For example, if I have a
corpus of TEI letters in collection /data/corpus, I will keep separate xml
files containing, for example, my ListPerson or List Place in
/data/listPerson and /data/listPlace respectively. This makes them fully
available for use in computed fields. They cannot be collections within
collections. Note as well, that if one triggers an index on all of /data/
with a collection.xconf in /data/ one would face the same problem.
Therefore each collection needs its own collection.xconf.

If your situation is like mine the first time, it means rebuilding the
collections, and refactoring and rewiring some code (and now I keep all my
XQuery paths in "global-like" variables so I only touch them in one place
if a change is needed.)

I hope this helps.

Best,

JPR



On Fri, Oct 31, 2025 at 12:21 PM Lars Scheideler <sch...@sa...>
wrote:

> Hello,
>
> I want to index a collection and need a field that contains specific
> computed values from the same collection.
> I have various TEI XML files with different biblStruct types. Some of
> these files refer to other biblStructs via ref elements. I need the IDs of
> the biblStructs that point to the current, indexed biblStruct.
> However, when I reindex the collection, the fields almost exclusively
> contain “empty” values.
>
> Here the code:
>
> <collection xmlns="http://exist-db.org/collection-config/1.0"
> <http://exist-db.org/collection-config/1.0>>
>     <index xmlns:tei="http://www.tei-c.org/ns/1.0"
> <http://www.tei-c.org/ns/1.0> xmlns:xs="http://www.w3.org/2001/XMLSchema"
> <http://www.w3.org/2001/XMLSchema>>
>         <lucene>
>             <module uri="http://application.de/xquery/facet-utils"
> <http://application.de/xquery/facet-utils> prefix="fu" at=
> "xmldb:exist:///db/apps/application/modules/index-facet-utils.xqm"/>
>             <text qname="tei:biblStruct">
>                 <field name="workgroupid" expression=
> "fu:get-workgroup-id(.)"/>
>             </text>
>         </lucene>
>     </index>
> </collection>
>
>
> declare function fu:get-workgroup-id($biblStruct as element()) as
> xs:string* {
>     let $corresp as xs:string := $biblStruct/@corresp/string()
>     let $workgroupsIds as xs:string* :=
> collection("/db/projects/application/data/bibls")//tei:biblStruct[@type
> eq 'dataset'][.//tei:ref[@type eq 'relDocRef']/@target/string() =
> $corresp]/@xml:id/string()
>     let $workgroupsIdsJoined := string-join($workgroupsIds,';')
>     return
>         if($workgroupsIdsJoined ne "") then $workgroupsIdsJoined else
> "empty"
> };
>
> I assume that I do not have access to the same collection that is actually
> indexed or to specific attributes/elements (this case is not
> described/addressed/mentioned in either the documentation or the book), but
> the function takes far too long for live productive use.
>
> Thanks for the help.
>
> Best regards
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

[Exist-open] Index field - Computed field function access to the same collection?

[Lars S. <sch...@sa...>]

Hello,

I want to index a collection and need a field that contains specific 
computed values from the same collection.
I have various TEI XML files with different biblStruct types. Some of 
these files refer to other biblStructs via ref elements. I need the IDs 
of the biblStructs that point to the current, indexed biblStruct.
However, when I reindex the collection, the fields almost exclusively 
contain “empty” values.

Here the code:

<collection xmlns="http://exist-db.org/collection-config/1.0">
     <index xmlns:tei="http://www.tei-c.org/ns/1.0" 
xmlns:xs="http://www.w3.org/2001/XMLSchema">
         <lucene>
             <module uri="http://application.de/xquery/facet-utils" 
prefix="fu" 
at="xmldb:exist:///db/apps/application/modules/index-facet-utils.xqm"/>
             <text qname="tei:biblStruct">
                 <field name="workgroupid" 
expression="fu:get-workgroup-id(.)"/>
             </text>
         </lucene>
     </index>
</collection>


declare function fu:get-workgroup-id($biblStruct as element()) as 
xs:string* {
     let $corresp as xs:string := $biblStruct/@corresp/string()
     let $workgroupsIds as xs:string* := 
collection("/db/projects/application/data/bibls")//tei:biblStruct[@type 
eq 'dataset'][.//tei:ref[@type eq 'relDocRef']/@target/string() = 
$corresp]/@xml:id/string()
     let $workgroupsIdsJoined := string-join($workgroupsIds,';')
     return
         if($workgroupsIdsJoined ne "") then $workgroupsIdsJoined else 
"empty"
};

I assume that I do not have access to the same collection that is 
actually indexed or to specific attributes/elements (this case is not 
described/addressed/mentioned in either the documentation or the book), 
but the function takes far too long for live productive use.

Thanks for the help.

Best regards

Re: [Exist-open] Urgent assistance with locking behaviour of existdb - or how to change the architecture of a scheduled ETL service

[Dannes W. <di...@ex...>]

Hi Willem

> On 18 Oct 2025, at 12:54, Willem van der Westhuizen <wi...@kw...> wrote:
> 
> The etl engine is an existdb module, that is called by an xquery. One hypothesis is that each time that the xquery runs, it loads an instance of the module into memory, and thats is where the locking happens. No way to tell if that is indeed the case, and what the internal limiations are.
> 
> The one interesting observation is that even in older version 3.3.3 if the xquery is pending up due to the remote service being called is stalled, then the number of instances in exists always stop responding at around 160 plus or minus, then the whole exist instance locks up. The number of database brokers, and memory etc. was never a limit.
> 
> In the later versions of exist the deadlock may occur at much lover levels. To give an example, we have rewritten most of the critical code that locked up in javascript, and there was no issue in the downstream services. 
> 
> The explanation offered of recursive calling of modules might be a clue, because that is certainly the case in our code. We would prefer to keep the ETL in exist though, hence trying to find a solution.
> 
Is it about the couch base driver I created a long time ago?
I never checked the code for newer eXist-db versions..

With kind regards

Dannes

Re: [Exist-open] Urgent assistance with locking behaviour of existdb - or how to change the architecture of a scheduled ETL service

[Willem v. d. W. <wi...@kw...>]

Hi Dannes,

No, it more the locking behaviour of later versions of exist where 
running many instances of the same xquery as a rest service that makes 
rest services external to existdb. Even though there are no updates, 
running multiple instances do seem to place some form of read lock on 
the xquery that locks the instance up.

Regards

Willem

On 2025/10/26 21:34, Dannes Wessels wrote:
> Hi Willem
>
>> On 18 Oct 2025, at 12:54, Willem van der Westhuizen 
>> <wi...@kw...> wrote:
>>
>> The etl engine is an existdb module, that is called by an xquery. One 
>> hypothesis is that each time that the xquery runs, it loads an 
>> instance of the module into memory, and thats is where the locking 
>> happens. No way to tell if that is indeed the case, and what the 
>> internal limiations are.
>>
>> The one interesting observation is that even in older version 3.3.3 
>> if the xquery is pending up due to the remote service being called is 
>> stalled, then the number of instances in exists always stop 
>> responding at around 160 plus or minus, then the whole exist instance 
>> locks up. The number of database brokers, and memory etc. was never a 
>> limit.
>>
>> In the later versions of exist the deadlock may occur at much lover 
>> levels. To give an example, we have rewritten most of the critical 
>> code that locked up in javascript, and there was no issue in the 
>> downstream services.
>>
>> The explanation offered of recursive calling of modules might be a 
>> clue, because that is certainly the case in our code. We would prefer 
>> to keep the ETL in exist though, hence trying to find a solution.
>>
> Is it about the couch base driver I created a long time ago?
> I never checked the code for newer eXist-db versions..
>
> With kind regards
>
> Dannes
>
>

Re: [Exist-open] exist multiple cores

[Craig B. <cra...@ma...>]

> On Oct 16, 2025, at 10:33 AM, Eduard Drenth <ed...@FR...> wrote:
> 
> Have been trying to upload a 386M document into exist 6.4.0 in order to test query performance compared to querying 70000 separate documents. 
> 
> Question before I continue trying different settings/jvm's/etc.:
> 
> Is querying one large document expected to be significantly faster compared to querying many separate small documents?


The answer to that very likely depends on how many nodes you have, what kind of indexes you have, and what kind of queries you're running.  In my experience, accessing nodes later in a large document using the structural index is much slower than accessing nodes earlier in the same document.  I demonstrated that for queries using the preceding and following axes here:

https://github.com/eXist-db/exist/issues/2129
wildcard following and preceding axes are slow · Issue #2129 · eXist-db/exist
github.com



________________________________________
Craig A. Berry

"... getting out of a sonnet is much more
 difficult than getting in."
                 Brad Leithauser

Re: [Exist-open] Urgent assistance with locking behaviour of existdb - or how to change the architecture of a scheduled ETL service

[Willem v. d. W. <wi...@kw...>]

Hi,

The etl engine is an existdb module, that is called by an xquery. One 
hypothesis is that each time that the xquery runs, it loads an instance 
of the module into memory, and thats is where the locking happens. No 
way to tell if that is indeed the case, and what the internal limiations 
are.

The one interesting observation is that even in older version 3.3.3 if 
the xquery is pending up due to the remote service being called is 
stalled, then the number of instances in exists always stop responding 
at around 160 plus or minus, then the whole exist instance locks up. The 
number of database brokers, and memory etc. was never a limit.

In the later versions of exist the deadlock may occur at much lover 
levels. To give an example, we have rewritten most of the critical code 
that locked up in javascript, and there was no issue in the downstream 
services.

The explanation offered of recursive calling of modules might be a clue, 
because that is certainly the case in our code. We would prefer to keep 
the ETL in exist though, hence trying to find a solution.

Regards

Willem

On 2025/10/18 11:38, Claudius Teodorescu wrote:
> Hi!
>
> Would it help to have the ETL engine as an eXist-db module?
>
>
> Claudius Teodorescu
>
> On Sat, 18 Oct 2025 at 11:29, Willem van der Westhuizen 
> <wi...@kw...> wrote:
>
>     Maybe there is a different question to ask. We are using exist and
>     xquery extensively with an etl engine that we wrote to read data from
>     couchdb and place into MySQL for reporting purposes. We are currently
>     using a shell script to launch 20 instances of the the etl with
>     different parameters to pick up a batch and process it. Would
>     there be a
>     better way in exist to write the scheduler to make those calls,
>     and then
>     to wait for each instance to be complete before calling the next,
>     that
>     would not involve calling the same rest service muliple times from
>     curl?
>
>     Regards
>
>     Willem
>
>     On 2025/10/15 11:48, Willem van der Westhuizen wrote:
>     > Definitely not updating. Just running,
>     >
>     > On 2025/10/15 10:30, Juri Leino wrote:
>     >> Hi Willem!
>     >>
>     >> The behaviour you are describing is usually a sign for an
>     attempt to
>     >> replace/update a running XQuery.  If this is the case the easiest
>     >> solution is to make sure that the query you want to replace /
>     change
>     >> is not being run at the moment.
>     >>
>     >> I agree that we should investigate if we even have to lock the
>     XQuery
>     >> in the first place. But that is a separate discussion.
>     >>
>     >> Regards,
>     >> Juri
>     >>
>     >> On 12.10.25 13:01, Willem van der Westhuizen wrote:
>     >>> Hi
>     >>>
>     >>> We use existdb extensively to run xqueries on our external
>     >>> databases, couchdb and others. We dont use the existdb
>     datastore for
>     >>> our data.
>     >>>
>     >>> We used to run on an old 3.3.3 version of existdb, and recently
>     >>> migrated to 6.2. Since then we have found that existdb locks the
>     >>> xquery that is being executed, causing deadlocks that eventually
>     >>> locks up the server. Is there an easy way to tell existdb to
>     revert
>     >>> to the older locking behaviour that does not lock up the xquery
>     >>> being run?
>     >>>
>     >>>
>     >>> Regards
>     >>>
>     >>
>     >>
>     >> _______________________________________________
>     >> Exist-open mailing list
>     >> Exi...@li...
>     >> https://lists.sourceforge.net/lists/listinfo/exist-open
>     >
>     >
>     > _______________________________________________
>     > Exist-open mailing list
>     > Exi...@li...
>     > https://lists.sourceforge.net/lists/listinfo/exist-open
>
>
>     _______________________________________________
>     Exist-open mailing list
>     Exi...@li...
>     https://lists.sourceforge.net/lists/listinfo/exist-open
>
>
>
> -- 
> Cu stimă,
> Claudius Teodorescu

Re: [Exist-open] Urgent assistance with locking behaviour of existdb - or how to change the architecture of a scheduled ETL service

[Claudius T. <cla...@gm...>]

Hi!

Would it help to have the ETL engine as an eXist-db module?


Claudius Teodorescu

On Sat, 18 Oct 2025 at 11:29, Willem van der Westhuizen <wi...@kw...>
wrote:

> Maybe there is a different question to ask. We are using exist and
> xquery extensively with an etl engine that we wrote to read data from
> couchdb and place into MySQL for reporting purposes. We are currently
> using a shell script to launch 20 instances of the the etl with
> different parameters to pick up a batch and process it. Would there be a
> better way in exist to write the scheduler to make those calls, and then
> to wait for each instance to be complete before calling the next, that
> would not involve calling the same rest service muliple times from curl?
>
> Regards
>
> Willem
>
> On 2025/10/15 11:48, Willem van der Westhuizen wrote:
> > Definitely not updating. Just running,
> >
> > On 2025/10/15 10:30, Juri Leino wrote:
> >> Hi Willem!
> >>
> >> The behaviour you are describing is usually a sign for an attempt to
> >> replace/update a running XQuery.  If this is the case the easiest
> >> solution is to make sure that the query you want to replace / change
> >> is not being run at the moment.
> >>
> >> I agree that we should investigate if we even have to lock the XQuery
> >> in the first place. But that is a separate discussion.
> >>
> >> Regards,
> >> Juri
> >>
> >> On 12.10.25 13:01, Willem van der Westhuizen wrote:
> >>> Hi
> >>>
> >>> We use existdb extensively to run xqueries on our external
> >>> databases, couchdb and others. We dont use the existdb datastore for
> >>> our data.
> >>>
> >>> We used to run on an old 3.3.3 version of existdb, and recently
> >>> migrated to 6.2. Since then we have found that existdb locks the
> >>> xquery that is being executed, causing deadlocks that eventually
> >>> locks up the server. Is there an easy way to tell existdb to revert
> >>> to the older locking behaviour that does not lock up the xquery
> >>> being run?
> >>>
> >>>
> >>> Regards
> >>>
> >>
> >>
> >> _______________________________________________
> >> Exist-open mailing list
> >> Exi...@li...
> >> https://lists.sourceforge.net/lists/listinfo/exist-open
> >
> >
> > _______________________________________________
> > Exist-open mailing list
> > Exi...@li...
> > https://lists.sourceforge.net/lists/listinfo/exist-open
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>


-- 
Cu stimă,
Claudius Teodorescu

Re: [Exist-open] exist multiple cores

[Eduard D. <ed...@fr...>]

Dear all,

Luckily I found settings that boost performance for our situation. I share them so they can be of use.

(My question about one large versus many small remains, many small will I think make collection() a lot faster.)

The most important is caching collection (in our case only changing with a new docker build):

declare variable $teidictjson:data :=
     if ($teidictjson:collcache) then
        if (fn:not(fn:empty(cache:get("teidictjson","coll")))) then
            cache:get("teidictjson","coll")
        else (let $r := cache:put("teidictjson","coll",collection($config:data-root))
                return cache:get("teidictjson","coll")
              )
     else collection($config:data-root);

docker-compose.yml, I suspect less memory would do as well:

   deploy:
      replicas: 1
      resources:
        limits:
          memory: "8GB"
      restart_policy:
        condition: any
    environment:
      # NOTE This overrides the maintained JAVA_TOOL_OPTIONS in the Dockerfile from the exist-db community
      # This is the only way to override settings, for example ARG CACHE_MEM=1024 in your Dockerfile won't work, neither will setting cache in conf.xml
      - JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 -Dsun.jnu.encoding=UTF-8 -Djava.awt.headless=true -Dorg.exist.db-connection.cacheSize=2048M -Dorg.exist.db-connection.pool.max=20 -Dlog4j.configurationFile=/exist/etc/log4j2.xml -Dexist.home=/exist -Dexist.configurationFile=/exist/etc/conf.xml -Djetty.home=/exist -Dexist.jetty.config=/exist/etc/jetty/standard.enabled-jetty-configs -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+ExitOnOutOfMemoryError -XX:MaxRAMPercentage=75.0

Regards, Eduard


________________________________
From: Eduard Drenth <ed...@fr...>
Sent: Thursday, October 16, 2025 5:33 PM
To: Lars Windauer <lar...@ex...>; Exist-open <Exi...@li...>
Subject: Re: [Exist-open] exist multiple cores

Have been trying to upload a 386M document into exist 6.4.0 in order to test query performance compared to querying 70000 separate documents.

Question before I continue trying different settings/jvm's/etc.:


  *
Is querying one large document expected to be significantly faster compared to querying many separate small documents?

Regards, Eduard
________________________________
From: Eduard Drenth <ed...@fr...>
Sent: Thursday, October 16, 2025 6:46 AM
To: Lars Windauer <lar...@ex...>; Exist-open <Exi...@li...>
Subject: Re: [Exist-open] exist multiple cores

Thanks, I understand it's hard to tell yes.

Could try this https://github.com/duncdrum/distroless-exist/blob/main/Dockerfile for java 21 with exist 6.4. But until now I rely on what is in docker hub instead of building images myself.

Could queries be faster when I merge ± 70.000 seperate TEI documents into one teiCorpus document?

Regards, Eduard
________________________________
From: Lars Windauer <lar...@ex...>
Sent: Tuesday, October 14, 2025 12:28 PM
To: Eduard Drenth <ed...@fr...>; Exist-open <Exi...@li...>
Subject: Re: [Exist-open] exist multiple cores

It’s really weird why the app should be slow on a system with more ghz. There must be some kind of other limitation in the new environment.

With regard to Java 21: Where did you read that eXist-db 6.x would be bound to max Java 17? I know at least two people running eXist-db 6 with Java 21. I’m aware of some discussion about some edge cases that could cause trouble with Java 21 but afaik it’s working fine at least for those two people. Did you ever try it yourself? Nevertheless, if the performance was fine in the old environment but is not in the new one, the Java version will very likely not be the problem because you did not run Java 21 in the old system.

Sorry I can’t help more but without actually looking in the new and old environment and running a few tests it’s kind of impossible to say, what the issue could be.

Fingers crossed you can sort it out!

Best,

Lars



> On 8. Oct 2025, at 17:34, Eduard Drenth <ed...@fr...> wrote:
>
> Thanks for the info, good to know about one xquery being bound to one core.
>
> In the mean time we turned hyperthreading off, that helped, but not enough yet.
>
> It is weird, the new cpu is 3.8ghz compared to the old 2.4ghz.
>
> The queries seem cpu bound as top shows cpu percentage above 200%.
>
> Increasing memory does not help, IO does not seem to be a problem.
>
> It is doable for now, but I am happy that new hardware is being ordered, hope that helps.
>
> Unfortunately I cannot benefit from the many performance improvements in java either, because exist 6 is bound to java 17 max.
>
> Regards,
> Eduard Drenth, Software Architekt
>
> ed...@fr...
>
> Doelestrjitte 8
> 8911 DX  Ljouwert
> (058) 213 14 14
> +31 62 094 34 28
> https://www.fryske-akademy.nl/
> https://frysker.nl/
> https://frisian.eu/<Outlook-d4b4dmlh.svg>
> gpg: https://keyserver.ubuntu.com/pks/lookup?search=eduard+drenth&fingerprint=on&op=index
> From: Lars Windauer <lar...@ex...>
> Sent: Tuesday, October 7, 2025 10:26 PM
> To: Exist-open <Exi...@li...>
> Subject: Re: [Exist-open] exist multiple cores
>  Dear Eduard,
>
> eXist-db can benefit from multiple cores without the need to configure anything. Having said this, please be aware that a single XQuery (dbbroker) can always run on only one core! So a single XQuery can’t utilise 20 cores but on a 20 core system you can have 20 parallel queries that each use one core.
>
> Can you provide us with information about the old and new environment? Gigahertz of your CPUs, how much RAM does the system have and how much of this is assigned to eXist-db. What about the hard drives, is eXist-db running from a SSD with the same filesystem as in the old environment?
>
> Best,
>
> Lars
>
>
> > On 6. Oct 2025, at 16:42, Eduard Drenth <ed...@fr...> wrote:
> >
> > Dear all,
> >
> > We are in the process of migrating exist from our own hardware to a provider.
> >
> > Performance drops significantly, the main cause is that exist seems to be using one core only.
> >
> > Can exist benefit from multiple cores? And if yes, how is that configured?
> >
> > Regards,
> >
> > Eduard Drenth, Software Architekt
> >
> > ed...@fr...
> > _______________________________________________
> > Exist-open mailing list
> > Exi...@li...
> > https://lists.sourceforge.net/lists/listinfo/exist-open
>
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] exist multiple cores

[Eduard D. <ed...@fr...>]

Have been trying to upload a 386M document into exist 6.4.0 in order to test query performance compared to querying 70000 separate documents.

Question before I continue trying different settings/jvm's/etc.:


  *
Is querying one large document expected to be significantly faster compared to querying many separate small documents?

Regards, Eduard
________________________________
From: Eduard Drenth <ed...@fr...>
Sent: Thursday, October 16, 2025 6:46 AM
To: Lars Windauer <lar...@ex...>; Exist-open <Exi...@li...>
Subject: Re: [Exist-open] exist multiple cores

Thanks, I understand it's hard to tell yes.

Could try this https://github.com/duncdrum/distroless-exist/blob/main/Dockerfile for java 21 with exist 6.4. But until now I rely on what is in docker hub instead of building images myself.

Could queries be faster when I merge ± 70.000 seperate TEI documents into one teiCorpus document?

Regards, Eduard
________________________________
From: Lars Windauer <lar...@ex...>
Sent: Tuesday, October 14, 2025 12:28 PM
To: Eduard Drenth <ed...@fr...>; Exist-open <Exi...@li...>
Subject: Re: [Exist-open] exist multiple cores

It’s really weird why the app should be slow on a system with more ghz. There must be some kind of other limitation in the new environment.

With regard to Java 21: Where did you read that eXist-db 6.x would be bound to max Java 17? I know at least two people running eXist-db 6 with Java 21. I’m aware of some discussion about some edge cases that could cause trouble with Java 21 but afaik it’s working fine at least for those two people. Did you ever try it yourself? Nevertheless, if the performance was fine in the old environment but is not in the new one, the Java version will very likely not be the problem because you did not run Java 21 in the old system.

Sorry I can’t help more but without actually looking in the new and old environment and running a few tests it’s kind of impossible to say, what the issue could be.

Fingers crossed you can sort it out!

Best,

Lars



> On 8. Oct 2025, at 17:34, Eduard Drenth <ed...@fr...> wrote:
>
> Thanks for the info, good to know about one xquery being bound to one core.
>
> In the mean time we turned hyperthreading off, that helped, but not enough yet.
>
> It is weird, the new cpu is 3.8ghz compared to the old 2.4ghz.
>
> The queries seem cpu bound as top shows cpu percentage above 200%.
>
> Increasing memory does not help, IO does not seem to be a problem.
>
> It is doable for now, but I am happy that new hardware is being ordered, hope that helps.
>
> Unfortunately I cannot benefit from the many performance improvements in java either, because exist 6 is bound to java 17 max.
>
> Regards,
> Eduard Drenth, Software Architekt
>
> ed...@fr...
>
> Doelestrjitte 8
> 8911 DX  Ljouwert
> (058) 213 14 14
> +31 62 094 34 28
> https://www.fryske-akademy.nl/
> https://frysker.nl/
> https://frisian.eu/<Outlook-d4b4dmlh.svg>
> gpg: https://keyserver.ubuntu.com/pks/lookup?search=eduard+drenth&fingerprint=on&op=index
> From: Lars Windauer <lar...@ex...>
> Sent: Tuesday, October 7, 2025 10:26 PM
> To: Exist-open <Exi...@li...>
> Subject: Re: [Exist-open] exist multiple cores
>  Dear Eduard,
>
> eXist-db can benefit from multiple cores without the need to configure anything. Having said this, please be aware that a single XQuery (dbbroker) can always run on only one core! So a single XQuery can’t utilise 20 cores but on a 20 core system you can have 20 parallel queries that each use one core.
>
> Can you provide us with information about the old and new environment? Gigahertz of your CPUs, how much RAM does the system have and how much of this is assigned to eXist-db. What about the hard drives, is eXist-db running from a SSD with the same filesystem as in the old environment?
>
> Best,
>
> Lars
>
>
> > On 6. Oct 2025, at 16:42, Eduard Drenth <ed...@fr...> wrote:
> >
> > Dear all,
> >
> > We are in the process of migrating exist from our own hardware to a provider.
> >
> > Performance drops significantly, the main cause is that exist seems to be using one core only.
> >
> > Can exist benefit from multiple cores? And if yes, how is that configured?
> >
> > Regards,
> >
> > Eduard Drenth, Software Architekt
> >
> > ed...@fr...
> > _______________________________________________
> > Exist-open mailing list
> > Exi...@li...
> > https://lists.sourceforge.net/lists/listinfo/exist-open
>
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] exist multiple cores

[Eduard D. <ed...@fr...>]

Thanks, I understand it's hard to tell yes.

Could try this https://github.com/duncdrum/distroless-exist/blob/main/Dockerfile for java 21 with exist 6.4. But until now I rely on what is in docker hub instead of building images myself.

Could queries be faster when I merge ± 70.000 seperate TEI documents into one teiCorpus document?

Regards, Eduard
________________________________
From: Lars Windauer <lar...@ex...>
Sent: Tuesday, October 14, 2025 12:28 PM
To: Eduard Drenth <ed...@fr...>; Exist-open <Exi...@li...>
Subject: Re: [Exist-open] exist multiple cores

It’s really weird why the app should be slow on a system with more ghz. There must be some kind of other limitation in the new environment.

With regard to Java 21: Where did you read that eXist-db 6.x would be bound to max Java 17? I know at least two people running eXist-db 6 with Java 21. I’m aware of some discussion about some edge cases that could cause trouble with Java 21 but afaik it’s working fine at least for those two people. Did you ever try it yourself? Nevertheless, if the performance was fine in the old environment but is not in the new one, the Java version will very likely not be the problem because you did not run Java 21 in the old system.

Sorry I can’t help more but without actually looking in the new and old environment and running a few tests it’s kind of impossible to say, what the issue could be.

Fingers crossed you can sort it out!

Best,

Lars



> On 8. Oct 2025, at 17:34, Eduard Drenth <ed...@fr...> wrote:
>
> Thanks for the info, good to know about one xquery being bound to one core.
>
> In the mean time we turned hyperthreading off, that helped, but not enough yet.
>
> It is weird, the new cpu is 3.8ghz compared to the old 2.4ghz.
>
> The queries seem cpu bound as top shows cpu percentage above 200%.
>
> Increasing memory does not help, IO does not seem to be a problem.
>
> It is doable for now, but I am happy that new hardware is being ordered, hope that helps.
>
> Unfortunately I cannot benefit from the many performance improvements in java either, because exist 6 is bound to java 17 max.
>
> Regards,
> Eduard Drenth, Software Architekt
>
> ed...@fr...
>
> Doelestrjitte 8
> 8911 DX  Ljouwert
> (058) 213 14 14
> +31 62 094 34 28
> https://www.fryske-akademy.nl/
> https://frysker.nl/
> https://frisian.eu/<Outlook-d4b4dmlh.svg>
> gpg: https://keyserver.ubuntu.com/pks/lookup?search=eduard+drenth&fingerprint=on&op=index
> From: Lars Windauer <lar...@ex...>
> Sent: Tuesday, October 7, 2025 10:26 PM
> To: Exist-open <Exi...@li...>
> Subject: Re: [Exist-open] exist multiple cores
>  Dear Eduard,
>
> eXist-db can benefit from multiple cores without the need to configure anything. Having said this, please be aware that a single XQuery (dbbroker) can always run on only one core! So a single XQuery can’t utilise 20 cores but on a 20 core system you can have 20 parallel queries that each use one core.
>
> Can you provide us with information about the old and new environment? Gigahertz of your CPUs, how much RAM does the system have and how much of this is assigned to eXist-db. What about the hard drives, is eXist-db running from a SSD with the same filesystem as in the old environment?
>
> Best,
>
> Lars
>
>
> > On 6. Oct 2025, at 16:42, Eduard Drenth <ed...@fr...> wrote:
> >
> > Dear all,
> >
> > We are in the process of migrating exist from our own hardware to a provider.
> >
> > Performance drops significantly, the main cause is that exist seems to be using one core only.
> >
> > Can exist benefit from multiple cores? And if yes, how is that configured?
> >
> > Regards,
> >
> > Eduard Drenth, Software Architekt
> >
> > ed...@fr...
> > _______________________________________________
> > Exist-open mailing list
> > Exi...@li...
> > https://lists.sourceforge.net/lists/listinfo/exist-open
>
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] Urgent assistance with locking behaviour of existdb

[Willem v. d. W. <wi...@kw...>]

that the one in image that I shared? or how should I find it?

Willem

On 2025/10/15 13:55, Adam Retter wrote:
>
>     In our case we are not updating when the locking occurs.
>
>
> If all of your operations are read-only then it should not be possible 
> for a deadlock to occur.
> Would it be possible to see a dump of the lock table when that happens 
> Willem?
>
> -- 
> Adam Retter
>
> eXist Core Developer
> { United Kingdom }
> ad...@ex...

Re: [Exist-open] Urgent assistance with locking behaviour of existdb / deadlocking

[Pieter L. <pie...@be...>]

I don't know how to get that dump, would that be with jstack {pid} > 
C:\jstack.txt? I am not too keen on breaking my instance as I have a lot 
to do with it. Also, exist completely freezes so I do not know whether 
it would be possible to extract anything.

-p

On 10/15/2025 1:55 PM, Adam Retter wrote:
> Would it be possible to see a dump of the lock table when that happens 
> Pieter?
>
> On Wed, 15 Oct 2025 at 12:52, Pieter Lamers via Exist-open 
> <exi...@li...> wrote:
>
>     Hi Adam,
>
>     It is not trivial to create a reproduceable case for the
>     deadlocking problem but I am seeing deadlocks all the time when I
>     do not take the necessary precautions when updating. I think it
>     has to do with module imports. I think the deadlock occurs when
>     the update of a library module is attempted which is also holding
>     a read-lock because of an active process. Or the other way around:
>     you make a request which involves a library which is being
>     updated. At least one of these scenarios and maybe both can break
>     my setup.
>
>     This does not seem to be Willem's problem tho.
>
>     Best,
>     Pieter
>
>     On 10/14/2025 9:59 AM, Adam Retter wrote:
>>
>>     Hi Willem,
>>
>>     In eXist-db 5.x.x and 6.x.x, the XQuery (if stored in the
>>     database) is treated as any other document, and so it is locked
>>     for the duration of operations that happen upon it. Unfortunately
>>     this means that whilst a query is executing the document holding
>>     that query will keep a READ lock on the document.
>>
>>     A document may have many either a single write lock, or one or
>>     more read locks. This means that the query can only be updated by
>>     one person/client at a time, but that multiple clients can run
>>     the query in parallel.
>>
>>     I am not sure how a deadlock could occur with this. Can you
>>     provide some further information about your issue, and how to
>>     reproduce it please?
>>
>>     Thanks, Adam.
>>
>>     On Tue, 14 Oct 2025 at 08:53, Willem van der Westhuizen
>>     <wi...@kw...> wrote:
>>
>>         Hi
>>
>>         We use existdb extensively to run xqueries on our external
>>         databases,
>>         couchdb and others. We dont use the existdb datastore for our
>>         data.
>>
>>         We used to run on an old 3.3.3 version of existdb, and
>>         recently migrated
>>         to 6.2. Since then we have found that existdb locks the
>>         xquery that is
>>         being executed, causing deadlocks that eventually locks up
>>         the server.
>>         Is there an easy way to tell existdb to revert to the older
>>         locking
>>         behaviour that does not lock up the xquery being run?
>>
>>
>>         Regards
>>
>>         -- 
>>         Willem van der Westhuizen
>>         +27 82 9200718
>>
>>
>>
>>         _______________________________________________
>>         Exist-open mailing list
>>         Exi...@li...
>>         https://lists.sourceforge.net/lists/listinfo/exist-open
>>
>>
>>
>>     -- 
>>     Adam Retter
>>
>>     eXist Core Developer
>>     { United Kingdom }
>>     ad...@ex...
>>
>>
>>     _______________________________________________
>>     Exist-open mailing list
>>     Exi...@li...
>>     https://lists.sourceforge.net/lists/listinfo/exist-open
>
>     -- 
>     Pieter Lamers
>     John Benjamins Publishing Company
>     Postal Address: P.O. Box 36224, 1020 ME AMSTERDAM, The Netherlands
>     Visiting Address: Klaprozenweg 75G, 1033 NN AMSTERDAM, The Netherlands
>     Warehouse: Kelvinstraat 11-13, 1446 TK PURMEREND, The Netherlands
>     tel: +31 20 630 4747
>     web:www.benjamins.com <http://www.benjamins.com>
>
>
>     _______________________________________________
>     Exist-open mailing list
>     Exi...@li...
>     https://lists.sourceforge.net/lists/listinfo/exist-open
>
>
>
> -- 
> Adam Retter
>
> eXist Core Developer
> { United Kingdom }
> ad...@ex...

-- 
Pieter Lamers
John Benjamins Publishing Company
Postal Address: P.O. Box 36224, 1020 ME AMSTERDAM, The Netherlands
Visiting Address: Klaprozenweg 75G, 1033 NN AMSTERDAM, The Netherlands
Warehouse: Kelvinstraat 11-13, 1446 TK PURMEREND, The Netherlands
tel: +31 20 630 4747
web:www.benjamins.com

Re: [Exist-open] Urgent assistance with locking behaviour of existdb / deadlocking

[Adam R. <ad...@ex...>]

Would it be possible to see a dump of the lock table when that happens
Pieter?

On Wed, 15 Oct 2025 at 12:52, Pieter Lamers via Exist-open <
exi...@li...> wrote:

> Hi Adam,
>
> It is not trivial to create a reproduceable case for the deadlocking
> problem but I am seeing deadlocks all the time when I do not take the
> necessary precautions when updating. I think it has to do with module
> imports. I think the deadlock occurs when the update of a library module is
> attempted which is also holding a read-lock because of an active process.
> Or the other way around: you make a request which involves a library which
> is being updated. At least one of these scenarios and maybe both can break
> my setup.
>
> This does not seem to be Willem's problem tho.
>
> Best,
> Pieter
>
> On 10/14/2025 9:59 AM, Adam Retter wrote:
>
>
> Hi Willem,
>
> In eXist-db 5.x.x and 6.x.x, the XQuery (if stored in the database) is
> treated as any other document, and so it is locked for the duration of
> operations that happen upon it. Unfortunately this means that whilst a
> query is executing the document holding that query will keep a READ lock on
> the document.
>
> A document may have many either a single write lock, or one or more read
> locks. This means that the query can only be updated by one person/client
> at a time, but that multiple clients can run the query in parallel.
>
> I am not sure how a deadlock could occur with this. Can you provide some
> further information about your issue, and how to reproduce it please?
>
> Thanks, Adam.
>
> On Tue, 14 Oct 2025 at 08:53, Willem van der Westhuizen <wi...@kw...>
> wrote:
>
>> Hi
>>
>> We use existdb extensively to run xqueries on our external databases,
>> couchdb and others. We dont use the existdb datastore for our data.
>>
>> We used to run on an old 3.3.3 version of existdb, and recently migrated
>> to 6.2. Since then we have found that existdb locks the xquery that is
>> being executed, causing deadlocks that eventually locks up the server.
>> Is there an easy way to tell existdb to revert to the older locking
>> behaviour that does not lock up the xquery being run?
>>
>>
>> Regards
>>
>> --
>> Willem van der Westhuizen
>> +27 82 9200718
>>
>>
>>
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>
>
>
> --
> Adam Retter
>
> eXist Core Developer
> { United Kingdom }
> ad...@ex...
>
>
> _______________________________________________
> Exist-open mailing lis...@li...://lists.sourceforge.net/lists/listinfo/exist-open
>
>
> --
> Pieter Lamers
> John Benjamins Publishing Company
> Postal Address: P.O. Box 36224, 1020 ME AMSTERDAM, The Netherlands
> Visiting Address: Klaprozenweg 75G, 1033 NN AMSTERDAM, The Netherlands
> Warehouse: Kelvinstraat 11-13, 1446 TK PURMEREND, The Netherlands
> tel: +31 20 630 4747
> web: www.benjamins.com
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>


-- 
Adam Retter

eXist Core Developer
{ United Kingdom }
ad...@ex...

Re: [Exist-open] Urgent assistance with locking behaviour of existdb

[Adam R. <ad...@ex...>]

> My personal experience is that
> lock ups occur when I try to update XQuery resources while they are in
> use. Therefore I do not update the applications while they are running
> production.
>

Theoretically that should not be possible as the XQuery whislt it is
executing will hold a READ lock. Whilst the READ lock is held, any requests
to update the XQuery will wait until the READ lock is released, each
pending update operation will then be executed one after another, with each
one taking the WRITE lock.

If there is a deadlock, there must be more resources involved than just the
XQuery.

Would it be possible to see a dump of the lock table when that happens
Pieter?

-- 
Adam Retter

eXist Core Developer
{ United Kingdom }
ad...@ex...

Re: [Exist-open] Urgent assistance with locking behaviour of existdb

[Adam R. <ad...@ex...>]

>
> In our case we are not updating when the locking occurs.
>

If all of your operations are read-only then it should not be possible for
a deadlock to occur.
Would it be possible to see a dump of the lock table when that happens
Willem?


-- 
Adam Retter

eXist Core Developer
{ United Kingdom }
ad...@ex...

Re: [Exist-open] Urgent assistance with locking behaviour of existdb - or how to change the architecture of a scheduled ETL service

[Willem v. d. W. <wi...@kw...>]

Maybe there is a different question to ask. We are using exist and 
xquery extensively with an etl engine that we wrote to read data from 
couchdb and place into MySQL for reporting purposes. We are currently 
using a shell script to launch 20 instances of the the etl with 
different parameters to pick up a batch and process it. Would there be a 
better way in exist to write the scheduler to make those calls, and then 
to wait for each instance to be complete before calling the next, that 
would not involve calling the same rest service muliple times from curl?

Regards

Willem

On 2025/10/15 11:48, Willem van der Westhuizen wrote:
> Definitely not updating. Just running,
>
> On 2025/10/15 10:30, Juri Leino wrote:
>> Hi Willem!
>>
>> The behaviour you are describing is usually a sign for an attempt to 
>> replace/update a running XQuery.  If this is the case the easiest 
>> solution is to make sure that the query you want to replace / change 
>> is not being run at the moment.
>>
>> I agree that we should investigate if we even have to lock the XQuery 
>> in the first place. But that is a separate discussion.
>>
>> Regards,
>> Juri
>>
>> On 12.10.25 13:01, Willem van der Westhuizen wrote:
>>> Hi
>>>
>>> We use existdb extensively to run xqueries on our external 
>>> databases, couchdb and others. We dont use the existdb datastore for 
>>> our data.
>>>
>>> We used to run on an old 3.3.3 version of existdb, and recently 
>>> migrated to 6.2. Since then we have found that existdb locks the 
>>> xquery that is being executed, causing deadlocks that eventually 
>>> locks up the server. Is there an easy way to tell existdb to revert 
>>> to the older locking behaviour that does not lock up the xquery 
>>> being run?
>>>
>>>
>>> Regards
>>>
>>
>>
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] Urgent assistance with locking behaviour of existdb

[Willem v. d. W. <wi...@kw...>]

Definitely not updating. Just running,

On 2025/10/15 10:30, Juri Leino wrote:
> Hi Willem!
>
> The behaviour you are describing is usually a sign for an attempt to 
> replace/update a running XQuery.  If this is the case the easiest 
> solution is to make sure that the query you want to replace / change 
> is not being run at the moment.
>
> I agree that we should investigate if we even have to lock the XQuery 
> in the first place. But that is a separate discussion.
>
> Regards,
> Juri
>
> On 12.10.25 13:01, Willem van der Westhuizen wrote:
>> Hi
>>
>> We use existdb extensively to run xqueries on our external databases, 
>> couchdb and others. We dont use the existdb datastore for our data.
>>
>> We used to run on an old 3.3.3 version of existdb, and recently 
>> migrated to 6.2. Since then we have found that existdb locks the 
>> xquery that is being executed, causing deadlocks that eventually 
>> locks up the server. Is there an easy way to tell existdb to revert 
>> to the older locking behaviour that does not lock up the xquery being 
>> run?
>>
>>
>> Regards
>>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] Urgent assistance with locking behaviour of existdb / deadlocking

[Pieter L. <pie...@be...>]

Hi Adam,

It is not trivial to create a reproduceable case for the deadlocking 
problem but I am seeing deadlocks all the time when I do not take the 
necessary precautions when updating. I think it has to do with module 
imports. I think the deadlock occurs when the update of a library module 
is attempted which is also holding a read-lock because of an active 
process. Or the other way around: you make a request which involves a 
library which is being updated. At least one of these scenarios and 
maybe both can break my setup.

This does not seem to be Willem's problem tho.

Best,
Pieter

On 10/14/2025 9:59 AM, Adam Retter wrote:
>
> Hi Willem,
>
> In eXist-db 5.x.x and 6.x.x, the XQuery (if stored in the database) is 
> treated as any other document, and so it is locked for the duration of 
> operations that happen upon it. Unfortunately this means that whilst a 
> query is executing the document holding that query will keep a READ 
> lock on the document.
>
> A document may have many either a single write lock, or one or more 
> read locks. This means that the query can only be updated by one 
> person/client at a time, but that multiple clients can run the query 
> in parallel.
>
> I am not sure how a deadlock could occur with this. Can you provide 
> some further information about your issue, and how to reproduce it please?
>
> Thanks, Adam.
>
> On Tue, 14 Oct 2025 at 08:53, Willem van der Westhuizen 
> <wi...@kw...> wrote:
>
>     Hi
>
>     We use existdb extensively to run xqueries on our external databases,
>     couchdb and others. We dont use the existdb datastore for our data.
>
>     We used to run on an old 3.3.3 version of existdb, and recently
>     migrated
>     to 6.2. Since then we have found that existdb locks the xquery
>     that is
>     being executed, causing deadlocks that eventually locks up the
>     server.
>     Is there an easy way to tell existdb to revert to the older locking
>     behaviour that does not lock up the xquery being run?
>
>
>     Regards
>
>     -- 
>     Willem van der Westhuizen
>     +27 82 9200718
>
>
>
>     _______________________________________________
>     Exist-open mailing list
>     Exi...@li...
>     https://lists.sourceforge.net/lists/listinfo/exist-open
>
>
>
> -- 
> Adam Retter
>
> eXist Core Developer
> { United Kingdom }
> ad...@ex...
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

-- 
Pieter Lamers
John Benjamins Publishing Company
Postal Address: P.O. Box 36224, 1020 ME AMSTERDAM, The Netherlands
Visiting Address: Klaprozenweg 75G, 1033 NN AMSTERDAM, The Netherlands
Warehouse: Kelvinstraat 11-13, 1446 TK PURMEREND, The Netherlands
tel: +31 20 630 4747
web:www.benjamins.com

Re: [Exist-open] Urgent assistance with locking behaviour of existdb

[Willem v. d. W. <wi...@kw...>]

One further issue,

If the calls are not too rapid, then it does not happen. But many 
different calls to the same xquery will increase the likelihood of a lockup.

Willem

On 2025/10/15 10:30, Juri Leino wrote:
> Hi Willem!
>
> The behaviour you are describing is usually a sign for an attempt to 
> replace/update a running XQuery.  If this is the case the easiest 
> solution is to make sure that the query you want to replace / change 
> is not being run at the moment.
>
> I agree that we should investigate if we even have to lock the XQuery 
> in the first place. But that is a separate discussion.
>
> Regards,
> Juri
>
> On 12.10.25 13:01, Willem van der Westhuizen wrote:
>> Hi
>>
>> We use existdb extensively to run xqueries on our external databases, 
>> couchdb and others. We dont use the existdb datastore for our data.
>>
>> We used to run on an old 3.3.3 version of existdb, and recently 
>> migrated to 6.2. Since then we have found that existdb locks the 
>> xquery that is being executed, causing deadlocks that eventually 
>> locks up the server. Is there an easy way to tell existdb to revert 
>> to the older locking behaviour that does not lock up the xquery being 
>> run?
>>
>>
>> Regards
>>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] Urgent assistance with locking behaviour of existdb

[Pieter L. <pie...@be...>]

Hi,

Might it be that there are any modules that include each other causing 
circular dependencies? I believe eXist has become a little pickier about 
this since version 4 and we've ironed them all out of our code base.

Best,
Pieter

On 10/15/2025 12:24 PM, Willem van der Westhuizen wrote:
>
> Hi,
>
> In our case we are not updating when the locking occurs. We simply 
> call the rest services, and then they call other rest services outside 
> of exist. We use local curl statements to call the rest services, and 
> that causes the lockup. And we can see that the waiting for locks are 
> growing. It seems that if the rest services are called from the 
> outside, and they themselves call rest services, the later versions 
> lock up. And when the server reached about 160 processed it always 
> crashes.    I have attached the screen shots of the monex. If you look 
> at the logs, its frozen, nothing running.
>
>
>
> On 2025/10/14 11:23, Pieter Lamers via Exist-open wrote:
>> Hi Willem,
>>
>> I would suggest using the latest release or build rather than 6.2.0 
>> and checking if the same behavior occurs. My personal experience is 
>> that lock ups occur when I try to update XQuery resources while they 
>> are in use. Therefore I do not update the applications while they are 
>> running production.
>>
>> Best,
>> Pieter
>>
>> On 10/12/2025 1:01 PM, Willem van der Westhuizen wrote:
>>> Hi
>>>
>>> We use existdb extensively to run xqueries on our external 
>>> databases, couchdb and others. We dont use the existdb datastore for 
>>> our data.
>>>
>>> We used to run on an old 3.3.3 version of existdb, and recently 
>>> migrated to 6.2. Since then we have found that existdb locks the 
>>> xquery that is being executed, causing deadlocks that eventually 
>>> locks up the server. Is there an easy way to tell existdb to revert 
>>> to the older locking behaviour that does not lock up the xquery 
>>> being run?
>>>
>>>
>>> Regards
>>>
>>

-- 
Pieter Lamers
John Benjamins Publishing Company
Postal Address: P.O. Box 36224, 1020 ME AMSTERDAM, The Netherlands
Visiting Address: Klaprozenweg 75G, 1033 NN AMSTERDAM, The Netherlands
Warehouse: Kelvinstraat 11-13, 1446 TK PURMEREND, The Netherlands
tel: +31 20 630 4747
web:www.benjamins.com

Re: [Exist-open] Urgent assistance with locking behaviour of existdb

[Willem v. d. W. <wi...@kw...>]

Hi,

In our case we are not updating when the locking occurs. We simply call 
the rest services, and then they call other rest services outside of 
exist. We use local curl statements to call the rest services, and that 
causes the lockup. And we can see that the waiting for locks are 
growing. It seems that if the rest services are called from the outside, 
and they themselves call rest services, the later versions lock up. And 
when the server reached about 160 processed it always crashes.    I have 
attached the screen shots of the monex. If you look at the logs, its 
frozen, nothing running.



On 2025/10/14 11:23, Pieter Lamers via Exist-open wrote:
> Hi Willem,
>
> I would suggest using the latest release or build rather than 6.2.0 
> and checking if the same behavior occurs. My personal experience is 
> that lock ups occur when I try to update XQuery resources while they 
> are in use. Therefore I do not update the applications while they are 
> running production.
>
> Best,
> Pieter
>
> On 10/12/2025 1:01 PM, Willem van der Westhuizen wrote:
>> Hi
>>
>> We use existdb extensively to run xqueries on our external databases, 
>> couchdb and others. We dont use the existdb datastore for our data.
>>
>> We used to run on an old 3.3.3 version of existdb, and recently 
>> migrated to 6.2. Since then we have found that existdb locks the 
>> xquery that is being executed, causing deadlocks that eventually 
>> locks up the server. Is there an easy way to tell existdb to revert 
>> to the older locking behaviour that does not lock up the xquery being 
>> run?
>>
>>
>> Regards
>>
>

Re: [Exist-open] Urgent assistance with locking behaviour of existdb

[Juri L. <ju...@ex...>]

Hi Willem!

The behaviour you are describing is usually a sign for an attempt to 
replace/update a running XQuery.  If this is the case the easiest 
solution is to make sure that the query you want to replace / change is 
not being run at the moment.

I agree that we should investigate if we even have to lock the XQuery in 
the first place. But that is a separate discussion.

Regards,
Juri

On 12.10.25 13:01, Willem van der Westhuizen wrote:
> Hi
>
> We use existdb extensively to run xqueries on our external databases, 
> couchdb and others. We dont use the existdb datastore for our data.
>
> We used to run on an old 3.3.3 version of existdb, and recently 
> migrated to 6.2. Since then we have found that existdb locks the 
> xquery that is being executed, causing deadlocks that eventually locks 
> up the server. Is there an easy way to tell existdb to revert to the 
> older locking behaviour that does not lock up the xquery being run?
>
>
> Regards
>

Re: [Exist-open] exist multiple cores

[Lars W. <lar...@ex...>]

It’s really weird why the app should be slow on a system with more ghz. There must be some kind of other limitation in the new environment.

With regard to Java 21: Where did you read that eXist-db 6.x would be bound to max Java 17? I know at least two people running eXist-db 6 with Java 21. I’m aware of some discussion about some edge cases that could cause trouble with Java 21 but afaik it’s working fine at least for those two people. Did you ever try it yourself? Nevertheless, if the performance was fine in the old environment but is not in the new one, the Java version will very likely not be the problem because you did not run Java 21 in the old system. 

Sorry I can’t help more but without actually looking in the new and old environment and running a few tests it’s kind of impossible to say, what the issue could be. 

Fingers crossed you can sort it out! 

Best, 

Lars



> On 8. Oct 2025, at 17:34, Eduard Drenth <ed...@fr...> wrote:
> 
> Thanks for the info, good to know about one xquery being bound to one core.
> 
> In the mean time we turned hyperthreading off, that helped, but not enough yet.
> 
> It is weird, the new cpu is 3.8ghz compared to the old 2.4ghz.
> 
> The queries seem cpu bound as top shows cpu percentage above 200%.
> 
> Increasing memory does not help, IO does not seem to be a problem.
> 
> It is doable for now, but I am happy that new hardware is being ordered, hope that helps.
> 
> Unfortunately I cannot benefit from the many performance improvements in java either, because exist 6 is bound to java 17 max.
> 
> Regards,
> Eduard Drenth, Software Architekt
> 
> ed...@fr...
> 
> Doelestrjitte 8
> 8911 DX  Ljouwert
> (058) 213 14 14
> +31 62 094 34 28
> https://www.fryske-akademy.nl/
> https://frysker.nl/
> https://frisian.eu/<Outlook-d4b4dmlh.svg>
> gpg: https://keyserver.ubuntu.com/pks/lookup?search=eduard+drenth&fingerprint=on&op=index
> From: Lars Windauer <lar...@ex...>
> Sent: Tuesday, October 7, 2025 10:26 PM
> To: Exist-open <Exi...@li...>
> Subject: Re: [Exist-open] exist multiple cores
>  Dear Eduard, 
> 
> eXist-db can benefit from multiple cores without the need to configure anything. Having said this, please be aware that a single XQuery (dbbroker) can always run on only one core! So a single XQuery can’t utilise 20 cores but on a 20 core system you can have 20 parallel queries that each use one core. 
> 
> Can you provide us with information about the old and new environment? Gigahertz of your CPUs, how much RAM does the system have and how much of this is assigned to eXist-db. What about the hard drives, is eXist-db running from a SSD with the same filesystem as in the old environment? 
> 
> Best, 
> 
> Lars
> 
> 
> > On 6. Oct 2025, at 16:42, Eduard Drenth <ed...@fr...> wrote:
> > 
> > Dear all,
> > 
> > We are in the process of migrating exist from our own hardware to a provider.
> > 
> > Performance drops significantly, the main cause is that exist seems to be using one core only.
> > 
> > Can exist benefit from multiple cores? And if yes, how is that configured?
> > 
> > Regards,
> > 
> > Eduard Drenth, Software Architekt
> > 
> > ed...@fr...
> > _______________________________________________
> > Exist-open mailing list
> > Exi...@li...
> > https://lists.sourceforge.net/lists/listinfo/exist-open
> 
> 
> 
> 
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open