Re: [Exist-open] exist multiple cores

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Dear all,

Luckily I found settings that boost performance for our situation. I share them so they can be of use.

(My question about one large versus many small remains, many small will I think make collection() a lot faster.)

The most important is caching collection (in our case only changing with a new docker build):

declare variable $teidictjson:data :=
     if ($teidictjson:collcache) then
        if (fn:not(fn:empty(cache:get("teidictjson","coll")))) then
            cache:get("teidictjson","coll")
        else (let $r := cache:put("teidictjson","coll",collection($config:data-root))
                return cache:get("teidictjson","coll")
              )
     else collection($config:data-root);

docker-compose.yml, I suspect less memory would do as well:

   deploy:
      replicas: 1
      resources:
        limits:
          memory: "8GB"
      restart_policy:
        condition: any
    environment:
      # NOTE This overrides the maintained JAVA_TOOL_OPTIONS in the Dockerfile from the exist-db community
      # This is the only way to override settings, for example ARG CACHE_MEM=1024 in your Dockerfile won't work, neither will setting cache in conf.xml
      - JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 -Dsun.jnu.encoding=UTF-8 -Djava.awt.headless=true -Dorg.exist.db-connection.cacheSize=2048M -Dorg.exist.db-connection.pool.max=20 -Dlog4j.configurationFile=/exist/etc/log4j2.xml -Dexist.home=/exist -Dexist.configurationFile=/exist/etc/conf.xml -Djetty.home=/exist -Dexist.jetty.config=/exist/etc/jetty/standard.enabled-jetty-configs -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+ExitOnOutOfMemoryError -XX:MaxRAMPercentage=75.0

Regards, Eduard

________________________________
From: Eduard Drenth <ed...@fr...>
Sent: Thursday, October 16, 2025 5:33 PM
To: Lars Windauer <lar...@ex...>; Exist-open <Exi...@li...>
Subject: Re: [Exist-open] exist multiple cores

Have been trying to upload a 386M document into exist 6.4.0 in order to test query performance compared to querying 70000 separate documents.

Question before I continue trying different settings/jvm's/etc.:

  *
Is querying one large document expected to be significantly faster compared to querying many separate small documents?

Regards, Eduard
________________________________
From: Eduard Drenth <ed...@fr...>
Sent: Thursday, October 16, 2025 6:46 AM
To: Lars Windauer <lar...@ex...>; Exist-open <Exi...@li...>
Subject: Re: [Exist-open] exist multiple cores

Thanks, I understand it's hard to tell yes.

Could try this https://github.com/duncdrum/distroless-exist/blob/main/Dockerfile for java 21 with exist 6.4. But until now I rely on what is in docker hub instead of building images myself.

Could queries be faster when I merge ± 70.000 seperate TEI documents into one teiCorpus document?

Regards, Eduard
________________________________
From: Lars Windauer <lar...@ex...>
Sent: Tuesday, October 14, 2025 12:28 PM
To: Eduard Drenth <ed...@fr...>; Exist-open <Exi...@li...>
Subject: Re: [Exist-open] exist multiple cores

It’s really weird why the app should be slow on a system with more ghz. There must be some kind of other limitation in the new environment.

With regard to Java 21: Where did you read that eXist-db 6.x would be bound to max Java 17? I know at least two people running eXist-db 6 with Java 21. I’m aware of some discussion about some edge cases that could cause trouble with Java 21 but afaik it’s working fine at least for those two people. Did you ever try it yourself? Nevertheless, if the performance was fine in the old environment but is not in the new one, the Java version will very likely not be the problem because you did not run Java 21 in the old system.

Sorry I can’t help more but without actually looking in the new and old environment and running a few tests it’s kind of impossible to say, what the issue could be.

Fingers crossed you can sort it out!

Best,

Lars

> On 8. Oct 2025, at 17:34, Eduard Drenth <ed...@fr...> wrote:
>
> Thanks for the info, good to know about one xquery being bound to one core.
>
> In the mean time we turned hyperthreading off, that helped, but not enough yet.
>
> It is weird, the new cpu is 3.8ghz compared to the old 2.4ghz.
>
> The queries seem cpu bound as top shows cpu percentage above 200%.
>
> Increasing memory does not help, IO does not seem to be a problem.
>
> It is doable for now, but I am happy that new hardware is being ordered, hope that helps.
>
> Unfortunately I cannot benefit from the many performance improvements in java either, because exist 6 is bound to java 17 max.
>
> Regards,
> Eduard Drenth, Software Architekt
>
> ed...@fr...
>
> Doelestrjitte 8
> 8911 DX  Ljouwert
> (058) 213 14 14
> +31 62 094 34 28
> https://www.fryske-akademy.nl/
> https://frysker.nl/
> https://frisian.eu/<Outlook-d4b4dmlh.svg>
> gpg: https://keyserver.ubuntu.com/pks/lookup?search=eduard+drenth&fingerprint=on&op=index
> From: Lars Windauer <lar...@ex...>
> Sent: Tuesday, October 7, 2025 10:26 PM
> To: Exist-open <Exi...@li...>
> Subject: Re: [Exist-open] exist multiple cores
>  Dear Eduard,
>
> eXist-db can benefit from multiple cores without the need to configure anything. Having said this, please be aware that a single XQuery (dbbroker) can always run on only one core! So a single XQuery can’t utilise 20 cores but on a 20 core system you can have 20 parallel queries that each use one core.
>
> Can you provide us with information about the old and new environment? Gigahertz of your CPUs, how much RAM does the system have and how much of this is assigned to eXist-db. What about the hard drives, is eXist-db running from a SSD with the same filesystem as in the old environment?
>
> Best,
>
> Lars
>
>
> > On 6. Oct 2025, at 16:42, Eduard Drenth <ed...@fr...> wrote:
> >
> > Dear all,
> >
> > We are in the process of migrating exist from our own hardware to a provider.
> >
> > Performance drops significantly, the main cause is that exist seems to be using one core only.
> >
> > Can exist benefit from multiple cores? And if yes, how is that configured?
> >
> > Regards,
> >
> > Eduard Drenth, Software Architekt
> >
> > ed...@fr...
> > _______________________________________________
> > Exist-open mailing list
> > Exi...@li...
> > https://lists.sourceforge.net/lists/listinfo/exist-open
>
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] exist multiple cores

eXist-db is a feature rich Open Source native XML database

Re: [Exist-open] exist multiple cores