You can subscribe to this list here.
2001 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(31) |
Nov
(25) |
Dec
(33) |
---|---|---|---|---|---|---|---|---|---|---|---|---|
2002 |
Jan
(48) |
Feb
(62) |
Mar
(22) |
Apr
(29) |
May
(9) |
Jun
(45) |
Jul
(28) |
Aug
(41) |
Sep
(60) |
Oct
(96) |
Nov
(99) |
Dec
(70) |
2003 |
Jan
(98) |
Feb
(159) |
Mar
(164) |
Apr
(150) |
May
(143) |
Jun
(97) |
Jul
(184) |
Aug
(143) |
Sep
(207) |
Oct
(126) |
Nov
(159) |
Dec
(165) |
2004 |
Jan
(131) |
Feb
(229) |
Mar
(220) |
Apr
(212) |
May
(320) |
Jun
(223) |
Jul
(191) |
Aug
(390) |
Sep
(261) |
Oct
(229) |
Nov
(215) |
Dec
(184) |
2005 |
Jan
(221) |
Feb
(312) |
Mar
(336) |
Apr
(273) |
May
(359) |
Jun
(277) |
Jul
(303) |
Aug
(321) |
Sep
(256) |
Oct
(415) |
Nov
(428) |
Dec
(508) |
2006 |
Jan
(585) |
Feb
(419) |
Mar
(496) |
Apr
(296) |
May
(403) |
Jun
(404) |
Jul
(553) |
Aug
(296) |
Sep
(252) |
Oct
(416) |
Nov
(414) |
Dec
(245) |
2007 |
Jan
(354) |
Feb
(422) |
Mar
(389) |
Apr
(298) |
May
(397) |
Jun
(318) |
Jul
(315) |
Aug
(339) |
Sep
(253) |
Oct
(317) |
Nov
(350) |
Dec
(264) |
2008 |
Jan
(353) |
Feb
(313) |
Mar
(433) |
Apr
(383) |
May
(343) |
Jun
(355) |
Jul
(321) |
Aug
(338) |
Sep
(242) |
Oct
(206) |
Nov
(199) |
Dec
(279) |
2009 |
Jan
(327) |
Feb
(221) |
Mar
(280) |
Apr
(278) |
May
(237) |
Jun
(345) |
Jul
(322) |
Aug
(324) |
Sep
(676) |
Oct
(586) |
Nov
(735) |
Dec
(329) |
2010 |
Jan
(619) |
Feb
(424) |
Mar
(529) |
Apr
(241) |
May
(312) |
Jun
(554) |
Jul
(698) |
Aug
(576) |
Sep
(408) |
Oct
(268) |
Nov
(391) |
Dec
(426) |
2011 |
Jan
(629) |
Feb
(512) |
Mar
(465) |
Apr
(467) |
May
(475) |
Jun
(403) |
Jul
(426) |
Aug
(542) |
Sep
(418) |
Oct
(620) |
Nov
(614) |
Dec
(358) |
2012 |
Jan
(357) |
Feb
(466) |
Mar
(344) |
Apr
(215) |
May
(408) |
Jun
(375) |
Jul
(241) |
Aug
(260) |
Sep
(401) |
Oct
(461) |
Nov
(498) |
Dec
(294) |
2013 |
Jan
(453) |
Feb
(447) |
Mar
(434) |
Apr
(326) |
May
(295) |
Jun
(471) |
Jul
(463) |
Aug
(278) |
Sep
(525) |
Oct
(343) |
Nov
(389) |
Dec
(405) |
2014 |
Jan
(564) |
Feb
(324) |
Mar
(319) |
Apr
(319) |
May
(384) |
Jun
(259) |
Jul
(210) |
Aug
(219) |
Sep
(315) |
Oct
(478) |
Nov
(207) |
Dec
(316) |
2015 |
Jan
(222) |
Feb
(234) |
Mar
(201) |
Apr
(145) |
May
(367) |
Jun
(318) |
Jul
(195) |
Aug
(210) |
Sep
(234) |
Oct
(248) |
Nov
(217) |
Dec
(189) |
2016 |
Jan
(219) |
Feb
(177) |
Mar
(110) |
Apr
(91) |
May
(159) |
Jun
(124) |
Jul
(192) |
Aug
(119) |
Sep
(125) |
Oct
(64) |
Nov
(80) |
Dec
(68) |
2017 |
Jan
(156) |
Feb
(312) |
Mar
(386) |
Apr
(217) |
May
(89) |
Jun
(115) |
Jul
(79) |
Aug
(122) |
Sep
(100) |
Oct
(99) |
Nov
(129) |
Dec
(77) |
2018 |
Jan
(106) |
Feb
(78) |
Mar
(160) |
Apr
(73) |
May
(110) |
Jun
(160) |
Jul
(93) |
Aug
(92) |
Sep
(75) |
Oct
(147) |
Nov
(114) |
Dec
(97) |
2019 |
Jan
(141) |
Feb
(78) |
Mar
(158) |
Apr
(60) |
May
(123) |
Jun
(54) |
Jul
(44) |
Aug
(147) |
Sep
(117) |
Oct
(54) |
Nov
(74) |
Dec
(96) |
2020 |
Jan
(113) |
Feb
(125) |
Mar
(142) |
Apr
(57) |
May
(71) |
Jun
(99) |
Jul
(58) |
Aug
(81) |
Sep
(49) |
Oct
(50) |
Nov
(63) |
Dec
(37) |
2021 |
Jan
(37) |
Feb
(45) |
Mar
(39) |
Apr
(18) |
May
(14) |
Jun
(9) |
Jul
(44) |
Aug
(23) |
Sep
(13) |
Oct
(31) |
Nov
(13) |
Dec
(33) |
2022 |
Jan
(17) |
Feb
(8) |
Mar
(32) |
Apr
(7) |
May
(17) |
Jun
(7) |
Jul
(36) |
Aug
(29) |
Sep
(9) |
Oct
(20) |
Nov
(10) |
Dec
(1) |
2023 |
Jan
(30) |
Feb
(37) |
Mar
(23) |
Apr
(1) |
May
(14) |
Jun
(5) |
Jul
(3) |
Aug
(6) |
Sep
(5) |
Oct
(48) |
Nov
(4) |
Dec
(29) |
2024 |
Jan
(1) |
Feb
|
Mar
(21) |
Apr
(6) |
May
(16) |
Jun
(41) |
Jul
(11) |
Aug
(17) |
Sep
(16) |
Oct
(11) |
Nov
(3) |
Dec
(9) |
2025 |
Jan
(7) |
Feb
(7) |
Mar
(6) |
Apr
(6) |
May
(30) |
Jun
(8) |
Jul
(10) |
Aug
(4) |
Sep
(10) |
Oct
(28) |
Nov
|
Dec
|
From: Craig B. <cra...@ma...> - 2025-10-18 12:10:45
|
> On Oct 16, 2025, at 10:33 AM, Eduard Drenth <ed...@FR...> wrote: > > Have been trying to upload a 386M document into exist 6.4.0 in order to test query performance compared to querying 70000 separate documents. > > Question before I continue trying different settings/jvm's/etc.: > > Is querying one large document expected to be significantly faster compared to querying many separate small documents? The answer to that very likely depends on how many nodes you have, what kind of indexes you have, and what kind of queries you're running. In my experience, accessing nodes later in a large document using the structural index is much slower than accessing nodes earlier in the same document. I demonstrated that for queries using the preceding and following axes here: https://github.com/eXist-db/exist/issues/2129 wildcard following and preceding axes are slow · Issue #2129 · eXist-db/exist github.com ________________________________________ Craig A. Berry "... getting out of a sonnet is much more difficult than getting in." Brad Leithauser |
From: Willem v. d. W. <wi...@kw...> - 2025-10-18 10:56:35
|
Hi, The etl engine is an existdb module, that is called by an xquery. One hypothesis is that each time that the xquery runs, it loads an instance of the module into memory, and thats is where the locking happens. No way to tell if that is indeed the case, and what the internal limiations are. The one interesting observation is that even in older version 3.3.3 if the xquery is pending up due to the remote service being called is stalled, then the number of instances in exists always stop responding at around 160 plus or minus, then the whole exist instance locks up. The number of database brokers, and memory etc. was never a limit. In the later versions of exist the deadlock may occur at much lover levels. To give an example, we have rewritten most of the critical code that locked up in javascript, and there was no issue in the downstream services. The explanation offered of recursive calling of modules might be a clue, because that is certainly the case in our code. We would prefer to keep the ETL in exist though, hence trying to find a solution. Regards Willem On 2025/10/18 11:38, Claudius Teodorescu wrote: > Hi! > > Would it help to have the ETL engine as an eXist-db module? > > > Claudius Teodorescu > > On Sat, 18 Oct 2025 at 11:29, Willem van der Westhuizen > <wi...@kw...> wrote: > > Maybe there is a different question to ask. We are using exist and > xquery extensively with an etl engine that we wrote to read data from > couchdb and place into MySQL for reporting purposes. We are currently > using a shell script to launch 20 instances of the the etl with > different parameters to pick up a batch and process it. Would > there be a > better way in exist to write the scheduler to make those calls, > and then > to wait for each instance to be complete before calling the next, > that > would not involve calling the same rest service muliple times from > curl? > > Regards > > Willem > > On 2025/10/15 11:48, Willem van der Westhuizen wrote: > > Definitely not updating. Just running, > > > > On 2025/10/15 10:30, Juri Leino wrote: > >> Hi Willem! > >> > >> The behaviour you are describing is usually a sign for an > attempt to > >> replace/update a running XQuery. If this is the case the easiest > >> solution is to make sure that the query you want to replace / > change > >> is not being run at the moment. > >> > >> I agree that we should investigate if we even have to lock the > XQuery > >> in the first place. But that is a separate discussion. > >> > >> Regards, > >> Juri > >> > >> On 12.10.25 13:01, Willem van der Westhuizen wrote: > >>> Hi > >>> > >>> We use existdb extensively to run xqueries on our external > >>> databases, couchdb and others. We dont use the existdb > datastore for > >>> our data. > >>> > >>> We used to run on an old 3.3.3 version of existdb, and recently > >>> migrated to 6.2. Since then we have found that existdb locks the > >>> xquery that is being executed, causing deadlocks that eventually > >>> locks up the server. Is there an easy way to tell existdb to > revert > >>> to the older locking behaviour that does not lock up the xquery > >>> being run? > >>> > >>> > >>> Regards > >>> > >> > >> > >> _______________________________________________ > >> Exist-open mailing list > >> Exi...@li... > >> https://lists.sourceforge.net/lists/listinfo/exist-open > > > > > > _______________________________________________ > > Exist-open mailing list > > Exi...@li... > > https://lists.sourceforge.net/lists/listinfo/exist-open > > > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open > > > > -- > Cu stimă, > Claudius Teodorescu |
From: Claudius T. <cla...@gm...> - 2025-10-18 09:38:25
|
Hi! Would it help to have the ETL engine as an eXist-db module? Claudius Teodorescu On Sat, 18 Oct 2025 at 11:29, Willem van der Westhuizen <wi...@kw...> wrote: > Maybe there is a different question to ask. We are using exist and > xquery extensively with an etl engine that we wrote to read data from > couchdb and place into MySQL for reporting purposes. We are currently > using a shell script to launch 20 instances of the the etl with > different parameters to pick up a batch and process it. Would there be a > better way in exist to write the scheduler to make those calls, and then > to wait for each instance to be complete before calling the next, that > would not involve calling the same rest service muliple times from curl? > > Regards > > Willem > > On 2025/10/15 11:48, Willem van der Westhuizen wrote: > > Definitely not updating. Just running, > > > > On 2025/10/15 10:30, Juri Leino wrote: > >> Hi Willem! > >> > >> The behaviour you are describing is usually a sign for an attempt to > >> replace/update a running XQuery. If this is the case the easiest > >> solution is to make sure that the query you want to replace / change > >> is not being run at the moment. > >> > >> I agree that we should investigate if we even have to lock the XQuery > >> in the first place. But that is a separate discussion. > >> > >> Regards, > >> Juri > >> > >> On 12.10.25 13:01, Willem van der Westhuizen wrote: > >>> Hi > >>> > >>> We use existdb extensively to run xqueries on our external > >>> databases, couchdb and others. We dont use the existdb datastore for > >>> our data. > >>> > >>> We used to run on an old 3.3.3 version of existdb, and recently > >>> migrated to 6.2. Since then we have found that existdb locks the > >>> xquery that is being executed, causing deadlocks that eventually > >>> locks up the server. Is there an easy way to tell existdb to revert > >>> to the older locking behaviour that does not lock up the xquery > >>> being run? > >>> > >>> > >>> Regards > >>> > >> > >> > >> _______________________________________________ > >> Exist-open mailing list > >> Exi...@li... > >> https://lists.sourceforge.net/lists/listinfo/exist-open > > > > > > _______________________________________________ > > Exist-open mailing list > > Exi...@li... > > https://lists.sourceforge.net/lists/listinfo/exist-open > > > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open > -- Cu stimă, Claudius Teodorescu |
From: Eduard D. <ed...@fr...> - 2025-10-17 07:10:42
|
Dear all, Luckily I found settings that boost performance for our situation. I share them so they can be of use. (My question about one large versus many small remains, many small will I think make collection() a lot faster.) The most important is caching collection (in our case only changing with a new docker build): declare variable $teidictjson:data := if ($teidictjson:collcache) then if (fn:not(fn:empty(cache:get("teidictjson","coll")))) then cache:get("teidictjson","coll") else (let $r := cache:put("teidictjson","coll",collection($config:data-root)) return cache:get("teidictjson","coll") ) else collection($config:data-root); docker-compose.yml, I suspect less memory would do as well: deploy: replicas: 1 resources: limits: memory: "8GB" restart_policy: condition: any environment: # NOTE This overrides the maintained JAVA_TOOL_OPTIONS in the Dockerfile from the exist-db community # This is the only way to override settings, for example ARG CACHE_MEM=1024 in your Dockerfile won't work, neither will setting cache in conf.xml - JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 -Dsun.jnu.encoding=UTF-8 -Djava.awt.headless=true -Dorg.exist.db-connection.cacheSize=2048M -Dorg.exist.db-connection.pool.max=20 -Dlog4j.configurationFile=/exist/etc/log4j2.xml -Dexist.home=/exist -Dexist.configurationFile=/exist/etc/conf.xml -Djetty.home=/exist -Dexist.jetty.config=/exist/etc/jetty/standard.enabled-jetty-configs -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+ExitOnOutOfMemoryError -XX:MaxRAMPercentage=75.0 Regards, Eduard ________________________________ From: Eduard Drenth <ed...@fr...> Sent: Thursday, October 16, 2025 5:33 PM To: Lars Windauer <lar...@ex...>; Exist-open <Exi...@li...> Subject: Re: [Exist-open] exist multiple cores Have been trying to upload a 386M document into exist 6.4.0 in order to test query performance compared to querying 70000 separate documents. Question before I continue trying different settings/jvm's/etc.: * Is querying one large document expected to be significantly faster compared to querying many separate small documents? Regards, Eduard ________________________________ From: Eduard Drenth <ed...@fr...> Sent: Thursday, October 16, 2025 6:46 AM To: Lars Windauer <lar...@ex...>; Exist-open <Exi...@li...> Subject: Re: [Exist-open] exist multiple cores Thanks, I understand it's hard to tell yes. Could try this https://github.com/duncdrum/distroless-exist/blob/main/Dockerfile for java 21 with exist 6.4. But until now I rely on what is in docker hub instead of building images myself. Could queries be faster when I merge ± 70.000 seperate TEI documents into one teiCorpus document? Regards, Eduard ________________________________ From: Lars Windauer <lar...@ex...> Sent: Tuesday, October 14, 2025 12:28 PM To: Eduard Drenth <ed...@fr...>; Exist-open <Exi...@li...> Subject: Re: [Exist-open] exist multiple cores It’s really weird why the app should be slow on a system with more ghz. There must be some kind of other limitation in the new environment. With regard to Java 21: Where did you read that eXist-db 6.x would be bound to max Java 17? I know at least two people running eXist-db 6 with Java 21. I’m aware of some discussion about some edge cases that could cause trouble with Java 21 but afaik it’s working fine at least for those two people. Did you ever try it yourself? Nevertheless, if the performance was fine in the old environment but is not in the new one, the Java version will very likely not be the problem because you did not run Java 21 in the old system. Sorry I can’t help more but without actually looking in the new and old environment and running a few tests it’s kind of impossible to say, what the issue could be. Fingers crossed you can sort it out! Best, Lars > On 8. Oct 2025, at 17:34, Eduard Drenth <ed...@fr...> wrote: > > Thanks for the info, good to know about one xquery being bound to one core. > > In the mean time we turned hyperthreading off, that helped, but not enough yet. > > It is weird, the new cpu is 3.8ghz compared to the old 2.4ghz. > > The queries seem cpu bound as top shows cpu percentage above 200%. > > Increasing memory does not help, IO does not seem to be a problem. > > It is doable for now, but I am happy that new hardware is being ordered, hope that helps. > > Unfortunately I cannot benefit from the many performance improvements in java either, because exist 6 is bound to java 17 max. > > Regards, > Eduard Drenth, Software Architekt > > ed...@fr... > > Doelestrjitte 8 > 8911 DX Ljouwert > (058) 213 14 14 > +31 62 094 34 28 > https://www.fryske-akademy.nl/ > https://frysker.nl/ > https://frisian.eu/<Outlook-d4b4dmlh.svg> > gpg: https://keyserver.ubuntu.com/pks/lookup?search=eduard+drenth&fingerprint=on&op=index > From: Lars Windauer <lar...@ex...> > Sent: Tuesday, October 7, 2025 10:26 PM > To: Exist-open <Exi...@li...> > Subject: Re: [Exist-open] exist multiple cores > Dear Eduard, > > eXist-db can benefit from multiple cores without the need to configure anything. Having said this, please be aware that a single XQuery (dbbroker) can always run on only one core! So a single XQuery can’t utilise 20 cores but on a 20 core system you can have 20 parallel queries that each use one core. > > Can you provide us with information about the old and new environment? Gigahertz of your CPUs, how much RAM does the system have and how much of this is assigned to eXist-db. What about the hard drives, is eXist-db running from a SSD with the same filesystem as in the old environment? > > Best, > > Lars > > > > On 6. Oct 2025, at 16:42, Eduard Drenth <ed...@fr...> wrote: > > > > Dear all, > > > > We are in the process of migrating exist from our own hardware to a provider. > > > > Performance drops significantly, the main cause is that exist seems to be using one core only. > > > > Can exist benefit from multiple cores? And if yes, how is that configured? > > > > Regards, > > > > Eduard Drenth, Software Architekt > > > > ed...@fr... > > _______________________________________________ > > Exist-open mailing list > > Exi...@li... > > https://lists.sourceforge.net/lists/listinfo/exist-open > > > > > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open |
From: Eduard D. <ed...@fr...> - 2025-10-16 15:33:58
|
Have been trying to upload a 386M document into exist 6.4.0 in order to test query performance compared to querying 70000 separate documents. Question before I continue trying different settings/jvm's/etc.: * Is querying one large document expected to be significantly faster compared to querying many separate small documents? Regards, Eduard ________________________________ From: Eduard Drenth <ed...@fr...> Sent: Thursday, October 16, 2025 6:46 AM To: Lars Windauer <lar...@ex...>; Exist-open <Exi...@li...> Subject: Re: [Exist-open] exist multiple cores Thanks, I understand it's hard to tell yes. Could try this https://github.com/duncdrum/distroless-exist/blob/main/Dockerfile for java 21 with exist 6.4. But until now I rely on what is in docker hub instead of building images myself. Could queries be faster when I merge ± 70.000 seperate TEI documents into one teiCorpus document? Regards, Eduard ________________________________ From: Lars Windauer <lar...@ex...> Sent: Tuesday, October 14, 2025 12:28 PM To: Eduard Drenth <ed...@fr...>; Exist-open <Exi...@li...> Subject: Re: [Exist-open] exist multiple cores It’s really weird why the app should be slow on a system with more ghz. There must be some kind of other limitation in the new environment. With regard to Java 21: Where did you read that eXist-db 6.x would be bound to max Java 17? I know at least two people running eXist-db 6 with Java 21. I’m aware of some discussion about some edge cases that could cause trouble with Java 21 but afaik it’s working fine at least for those two people. Did you ever try it yourself? Nevertheless, if the performance was fine in the old environment but is not in the new one, the Java version will very likely not be the problem because you did not run Java 21 in the old system. Sorry I can’t help more but without actually looking in the new and old environment and running a few tests it’s kind of impossible to say, what the issue could be. Fingers crossed you can sort it out! Best, Lars > On 8. Oct 2025, at 17:34, Eduard Drenth <ed...@fr...> wrote: > > Thanks for the info, good to know about one xquery being bound to one core. > > In the mean time we turned hyperthreading off, that helped, but not enough yet. > > It is weird, the new cpu is 3.8ghz compared to the old 2.4ghz. > > The queries seem cpu bound as top shows cpu percentage above 200%. > > Increasing memory does not help, IO does not seem to be a problem. > > It is doable for now, but I am happy that new hardware is being ordered, hope that helps. > > Unfortunately I cannot benefit from the many performance improvements in java either, because exist 6 is bound to java 17 max. > > Regards, > Eduard Drenth, Software Architekt > > ed...@fr... > > Doelestrjitte 8 > 8911 DX Ljouwert > (058) 213 14 14 > +31 62 094 34 28 > https://www.fryske-akademy.nl/ > https://frysker.nl/ > https://frisian.eu/<Outlook-d4b4dmlh.svg> > gpg: https://keyserver.ubuntu.com/pks/lookup?search=eduard+drenth&fingerprint=on&op=index > From: Lars Windauer <lar...@ex...> > Sent: Tuesday, October 7, 2025 10:26 PM > To: Exist-open <Exi...@li...> > Subject: Re: [Exist-open] exist multiple cores > Dear Eduard, > > eXist-db can benefit from multiple cores without the need to configure anything. Having said this, please be aware that a single XQuery (dbbroker) can always run on only one core! So a single XQuery can’t utilise 20 cores but on a 20 core system you can have 20 parallel queries that each use one core. > > Can you provide us with information about the old and new environment? Gigahertz of your CPUs, how much RAM does the system have and how much of this is assigned to eXist-db. What about the hard drives, is eXist-db running from a SSD with the same filesystem as in the old environment? > > Best, > > Lars > > > > On 6. Oct 2025, at 16:42, Eduard Drenth <ed...@fr...> wrote: > > > > Dear all, > > > > We are in the process of migrating exist from our own hardware to a provider. > > > > Performance drops significantly, the main cause is that exist seems to be using one core only. > > > > Can exist benefit from multiple cores? And if yes, how is that configured? > > > > Regards, > > > > Eduard Drenth, Software Architekt > > > > ed...@fr... > > _______________________________________________ > > Exist-open mailing list > > Exi...@li... > > https://lists.sourceforge.net/lists/listinfo/exist-open > > > > > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open |
From: Eduard D. <ed...@fr...> - 2025-10-16 08:20:18
|
Thanks, I understand it's hard to tell yes. Could try this https://github.com/duncdrum/distroless-exist/blob/main/Dockerfile for java 21 with exist 6.4. But until now I rely on what is in docker hub instead of building images myself. Could queries be faster when I merge ± 70.000 seperate TEI documents into one teiCorpus document? Regards, Eduard ________________________________ From: Lars Windauer <lar...@ex...> Sent: Tuesday, October 14, 2025 12:28 PM To: Eduard Drenth <ed...@fr...>; Exist-open <Exi...@li...> Subject: Re: [Exist-open] exist multiple cores It’s really weird why the app should be slow on a system with more ghz. There must be some kind of other limitation in the new environment. With regard to Java 21: Where did you read that eXist-db 6.x would be bound to max Java 17? I know at least two people running eXist-db 6 with Java 21. I’m aware of some discussion about some edge cases that could cause trouble with Java 21 but afaik it’s working fine at least for those two people. Did you ever try it yourself? Nevertheless, if the performance was fine in the old environment but is not in the new one, the Java version will very likely not be the problem because you did not run Java 21 in the old system. Sorry I can’t help more but without actually looking in the new and old environment and running a few tests it’s kind of impossible to say, what the issue could be. Fingers crossed you can sort it out! Best, Lars > On 8. Oct 2025, at 17:34, Eduard Drenth <ed...@fr...> wrote: > > Thanks for the info, good to know about one xquery being bound to one core. > > In the mean time we turned hyperthreading off, that helped, but not enough yet. > > It is weird, the new cpu is 3.8ghz compared to the old 2.4ghz. > > The queries seem cpu bound as top shows cpu percentage above 200%. > > Increasing memory does not help, IO does not seem to be a problem. > > It is doable for now, but I am happy that new hardware is being ordered, hope that helps. > > Unfortunately I cannot benefit from the many performance improvements in java either, because exist 6 is bound to java 17 max. > > Regards, > Eduard Drenth, Software Architekt > > ed...@fr... > > Doelestrjitte 8 > 8911 DX Ljouwert > (058) 213 14 14 > +31 62 094 34 28 > https://www.fryske-akademy.nl/ > https://frysker.nl/ > https://frisian.eu/<Outlook-d4b4dmlh.svg> > gpg: https://keyserver.ubuntu.com/pks/lookup?search=eduard+drenth&fingerprint=on&op=index > From: Lars Windauer <lar...@ex...> > Sent: Tuesday, October 7, 2025 10:26 PM > To: Exist-open <Exi...@li...> > Subject: Re: [Exist-open] exist multiple cores > Dear Eduard, > > eXist-db can benefit from multiple cores without the need to configure anything. Having said this, please be aware that a single XQuery (dbbroker) can always run on only one core! So a single XQuery can’t utilise 20 cores but on a 20 core system you can have 20 parallel queries that each use one core. > > Can you provide us with information about the old and new environment? Gigahertz of your CPUs, how much RAM does the system have and how much of this is assigned to eXist-db. What about the hard drives, is eXist-db running from a SSD with the same filesystem as in the old environment? > > Best, > > Lars > > > > On 6. Oct 2025, at 16:42, Eduard Drenth <ed...@fr...> wrote: > > > > Dear all, > > > > We are in the process of migrating exist from our own hardware to a provider. > > > > Performance drops significantly, the main cause is that exist seems to be using one core only. > > > > Can exist benefit from multiple cores? And if yes, how is that configured? > > > > Regards, > > > > Eduard Drenth, Software Architekt > > > > ed...@fr... > > _______________________________________________ > > Exist-open mailing list > > Exi...@li... > > https://lists.sourceforge.net/lists/listinfo/exist-open > > > > > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open |
From: Willem v. d. W. <wi...@kw...> - 2025-10-15 12:25:13
|
that the one in image that I shared? or how should I find it? Willem On 2025/10/15 13:55, Adam Retter wrote: > > In our case we are not updating when the locking occurs. > > > If all of your operations are read-only then it should not be possible > for a deadlock to occur. > Would it be possible to see a dump of the lock table when that happens > Willem? > > -- > Adam Retter > > eXist Core Developer > { United Kingdom } > ad...@ex... |
From: Pieter L. <pie...@be...> - 2025-10-15 12:09:14
|
I don't know how to get that dump, would that be with jstack {pid} > C:\jstack.txt? I am not too keen on breaking my instance as I have a lot to do with it. Also, exist completely freezes so I do not know whether it would be possible to extract anything. -p On 10/15/2025 1:55 PM, Adam Retter wrote: > Would it be possible to see a dump of the lock table when that happens > Pieter? > > On Wed, 15 Oct 2025 at 12:52, Pieter Lamers via Exist-open > <exi...@li...> wrote: > > Hi Adam, > > It is not trivial to create a reproduceable case for the > deadlocking problem but I am seeing deadlocks all the time when I > do not take the necessary precautions when updating. I think it > has to do with module imports. I think the deadlock occurs when > the update of a library module is attempted which is also holding > a read-lock because of an active process. Or the other way around: > you make a request which involves a library which is being > updated. At least one of these scenarios and maybe both can break > my setup. > > This does not seem to be Willem's problem tho. > > Best, > Pieter > > On 10/14/2025 9:59 AM, Adam Retter wrote: >> >> Hi Willem, >> >> In eXist-db 5.x.x and 6.x.x, the XQuery (if stored in the >> database) is treated as any other document, and so it is locked >> for the duration of operations that happen upon it. Unfortunately >> this means that whilst a query is executing the document holding >> that query will keep a READ lock on the document. >> >> A document may have many either a single write lock, or one or >> more read locks. This means that the query can only be updated by >> one person/client at a time, but that multiple clients can run >> the query in parallel. >> >> I am not sure how a deadlock could occur with this. Can you >> provide some further information about your issue, and how to >> reproduce it please? >> >> Thanks, Adam. >> >> On Tue, 14 Oct 2025 at 08:53, Willem van der Westhuizen >> <wi...@kw...> wrote: >> >> Hi >> >> We use existdb extensively to run xqueries on our external >> databases, >> couchdb and others. We dont use the existdb datastore for our >> data. >> >> We used to run on an old 3.3.3 version of existdb, and >> recently migrated >> to 6.2. Since then we have found that existdb locks the >> xquery that is >> being executed, causing deadlocks that eventually locks up >> the server. >> Is there an easy way to tell existdb to revert to the older >> locking >> behaviour that does not lock up the xquery being run? >> >> >> Regards >> >> -- >> Willem van der Westhuizen >> +27 82 9200718 >> >> >> >> _______________________________________________ >> Exist-open mailing list >> Exi...@li... >> https://lists.sourceforge.net/lists/listinfo/exist-open >> >> >> >> -- >> Adam Retter >> >> eXist Core Developer >> { United Kingdom } >> ad...@ex... >> >> >> _______________________________________________ >> Exist-open mailing list >> Exi...@li... >> https://lists.sourceforge.net/lists/listinfo/exist-open > > -- > Pieter Lamers > John Benjamins Publishing Company > Postal Address: P.O. Box 36224, 1020 ME AMSTERDAM, The Netherlands > Visiting Address: Klaprozenweg 75G, 1033 NN AMSTERDAM, The Netherlands > Warehouse: Kelvinstraat 11-13, 1446 TK PURMEREND, The Netherlands > tel: +31 20 630 4747 > web:www.benjamins.com <http://www.benjamins.com> > > > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open > > > > -- > Adam Retter > > eXist Core Developer > { United Kingdom } > ad...@ex... -- Pieter Lamers John Benjamins Publishing Company Postal Address: P.O. Box 36224, 1020 ME AMSTERDAM, The Netherlands Visiting Address: Klaprozenweg 75G, 1033 NN AMSTERDAM, The Netherlands Warehouse: Kelvinstraat 11-13, 1446 TK PURMEREND, The Netherlands tel: +31 20 630 4747 web:www.benjamins.com |
From: Adam R. <ad...@ex...> - 2025-10-15 12:02:55
|
Would it be possible to see a dump of the lock table when that happens Pieter? On Wed, 15 Oct 2025 at 12:52, Pieter Lamers via Exist-open < exi...@li...> wrote: > Hi Adam, > > It is not trivial to create a reproduceable case for the deadlocking > problem but I am seeing deadlocks all the time when I do not take the > necessary precautions when updating. I think it has to do with module > imports. I think the deadlock occurs when the update of a library module is > attempted which is also holding a read-lock because of an active process. > Or the other way around: you make a request which involves a library which > is being updated. At least one of these scenarios and maybe both can break > my setup. > > This does not seem to be Willem's problem tho. > > Best, > Pieter > > On 10/14/2025 9:59 AM, Adam Retter wrote: > > > Hi Willem, > > In eXist-db 5.x.x and 6.x.x, the XQuery (if stored in the database) is > treated as any other document, and so it is locked for the duration of > operations that happen upon it. Unfortunately this means that whilst a > query is executing the document holding that query will keep a READ lock on > the document. > > A document may have many either a single write lock, or one or more read > locks. This means that the query can only be updated by one person/client > at a time, but that multiple clients can run the query in parallel. > > I am not sure how a deadlock could occur with this. Can you provide some > further information about your issue, and how to reproduce it please? > > Thanks, Adam. > > On Tue, 14 Oct 2025 at 08:53, Willem van der Westhuizen <wi...@kw...> > wrote: > >> Hi >> >> We use existdb extensively to run xqueries on our external databases, >> couchdb and others. We dont use the existdb datastore for our data. >> >> We used to run on an old 3.3.3 version of existdb, and recently migrated >> to 6.2. Since then we have found that existdb locks the xquery that is >> being executed, causing deadlocks that eventually locks up the server. >> Is there an easy way to tell existdb to revert to the older locking >> behaviour that does not lock up the xquery being run? >> >> >> Regards >> >> -- >> Willem van der Westhuizen >> +27 82 9200718 >> >> >> >> _______________________________________________ >> Exist-open mailing list >> Exi...@li... >> https://lists.sourceforge.net/lists/listinfo/exist-open >> > > > -- > Adam Retter > > eXist Core Developer > { United Kingdom } > ad...@ex... > > > _______________________________________________ > Exist-open mailing lis...@li...://lists.sourceforge.net/lists/listinfo/exist-open > > > -- > Pieter Lamers > John Benjamins Publishing Company > Postal Address: P.O. Box 36224, 1020 ME AMSTERDAM, The Netherlands > Visiting Address: Klaprozenweg 75G, 1033 NN AMSTERDAM, The Netherlands > Warehouse: Kelvinstraat 11-13, 1446 TK PURMEREND, The Netherlands > tel: +31 20 630 4747 > web: www.benjamins.com > > > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open > -- Adam Retter eXist Core Developer { United Kingdom } ad...@ex... |
From: Adam R. <ad...@ex...> - 2025-10-15 12:02:32
|
> My personal experience is that > lock ups occur when I try to update XQuery resources while they are in > use. Therefore I do not update the applications while they are running > production. > Theoretically that should not be possible as the XQuery whislt it is executing will hold a READ lock. Whilst the READ lock is held, any requests to update the XQuery will wait until the READ lock is released, each pending update operation will then be executed one after another, with each one taking the WRITE lock. If there is a deadlock, there must be more resources involved than just the XQuery. Would it be possible to see a dump of the lock table when that happens Pieter? -- Adam Retter eXist Core Developer { United Kingdom } ad...@ex... |
From: Adam R. <ad...@ex...> - 2025-10-15 12:02:32
|
> > In our case we are not updating when the locking occurs. > If all of your operations are read-only then it should not be possible for a deadlock to occur. Would it be possible to see a dump of the lock table when that happens Willem? -- Adam Retter eXist Core Developer { United Kingdom } ad...@ex... |
From: Willem v. d. W. <wi...@kw...> - 2025-10-15 11:21:14
|
Maybe there is a different question to ask. We are using exist and xquery extensively with an etl engine that we wrote to read data from couchdb and place into MySQL for reporting purposes. We are currently using a shell script to launch 20 instances of the the etl with different parameters to pick up a batch and process it. Would there be a better way in exist to write the scheduler to make those calls, and then to wait for each instance to be complete before calling the next, that would not involve calling the same rest service muliple times from curl? Regards Willem On 2025/10/15 11:48, Willem van der Westhuizen wrote: > Definitely not updating. Just running, > > On 2025/10/15 10:30, Juri Leino wrote: >> Hi Willem! >> >> The behaviour you are describing is usually a sign for an attempt to >> replace/update a running XQuery. If this is the case the easiest >> solution is to make sure that the query you want to replace / change >> is not being run at the moment. >> >> I agree that we should investigate if we even have to lock the XQuery >> in the first place. But that is a separate discussion. >> >> Regards, >> Juri >> >> On 12.10.25 13:01, Willem van der Westhuizen wrote: >>> Hi >>> >>> We use existdb extensively to run xqueries on our external >>> databases, couchdb and others. We dont use the existdb datastore for >>> our data. >>> >>> We used to run on an old 3.3.3 version of existdb, and recently >>> migrated to 6.2. Since then we have found that existdb locks the >>> xquery that is being executed, causing deadlocks that eventually >>> locks up the server. Is there an easy way to tell existdb to revert >>> to the older locking behaviour that does not lock up the xquery >>> being run? >>> >>> >>> Regards >>> >> >> >> _______________________________________________ >> Exist-open mailing list >> Exi...@li... >> https://lists.sourceforge.net/lists/listinfo/exist-open > > > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open |
From: Willem v. d. W. <wi...@kw...> - 2025-10-15 10:48:41
|
Definitely not updating. Just running, On 2025/10/15 10:30, Juri Leino wrote: > Hi Willem! > > The behaviour you are describing is usually a sign for an attempt to > replace/update a running XQuery. If this is the case the easiest > solution is to make sure that the query you want to replace / change > is not being run at the moment. > > I agree that we should investigate if we even have to lock the XQuery > in the first place. But that is a separate discussion. > > Regards, > Juri > > On 12.10.25 13:01, Willem van der Westhuizen wrote: >> Hi >> >> We use existdb extensively to run xqueries on our external databases, >> couchdb and others. We dont use the existdb datastore for our data. >> >> We used to run on an old 3.3.3 version of existdb, and recently >> migrated to 6.2. Since then we have found that existdb locks the >> xquery that is being executed, causing deadlocks that eventually >> locks up the server. Is there an easy way to tell existdb to revert >> to the older locking behaviour that does not lock up the xquery being >> run? >> >> >> Regards >> > > > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open |
From: Pieter L. <pie...@be...> - 2025-10-15 10:40:38
|
Hi Adam, It is not trivial to create a reproduceable case for the deadlocking problem but I am seeing deadlocks all the time when I do not take the necessary precautions when updating. I think it has to do with module imports. I think the deadlock occurs when the update of a library module is attempted which is also holding a read-lock because of an active process. Or the other way around: you make a request which involves a library which is being updated. At least one of these scenarios and maybe both can break my setup. This does not seem to be Willem's problem tho. Best, Pieter On 10/14/2025 9:59 AM, Adam Retter wrote: > > Hi Willem, > > In eXist-db 5.x.x and 6.x.x, the XQuery (if stored in the database) is > treated as any other document, and so it is locked for the duration of > operations that happen upon it. Unfortunately this means that whilst a > query is executing the document holding that query will keep a READ > lock on the document. > > A document may have many either a single write lock, or one or more > read locks. This means that the query can only be updated by one > person/client at a time, but that multiple clients can run the query > in parallel. > > I am not sure how a deadlock could occur with this. Can you provide > some further information about your issue, and how to reproduce it please? > > Thanks, Adam. > > On Tue, 14 Oct 2025 at 08:53, Willem van der Westhuizen > <wi...@kw...> wrote: > > Hi > > We use existdb extensively to run xqueries on our external databases, > couchdb and others. We dont use the existdb datastore for our data. > > We used to run on an old 3.3.3 version of existdb, and recently > migrated > to 6.2. Since then we have found that existdb locks the xquery > that is > being executed, causing deadlocks that eventually locks up the > server. > Is there an easy way to tell existdb to revert to the older locking > behaviour that does not lock up the xquery being run? > > > Regards > > -- > Willem van der Westhuizen > +27 82 9200718 > > > > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open > > > > -- > Adam Retter > > eXist Core Developer > { United Kingdom } > ad...@ex... > > > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open -- Pieter Lamers John Benjamins Publishing Company Postal Address: P.O. Box 36224, 1020 ME AMSTERDAM, The Netherlands Visiting Address: Klaprozenweg 75G, 1033 NN AMSTERDAM, The Netherlands Warehouse: Kelvinstraat 11-13, 1446 TK PURMEREND, The Netherlands tel: +31 20 630 4747 web:www.benjamins.com |
From: Willem v. d. W. <wi...@kw...> - 2025-10-15 10:35:03
|
One further issue, If the calls are not too rapid, then it does not happen. But many different calls to the same xquery will increase the likelihood of a lockup. Willem On 2025/10/15 10:30, Juri Leino wrote: > Hi Willem! > > The behaviour you are describing is usually a sign for an attempt to > replace/update a running XQuery. If this is the case the easiest > solution is to make sure that the query you want to replace / change > is not being run at the moment. > > I agree that we should investigate if we even have to lock the XQuery > in the first place. But that is a separate discussion. > > Regards, > Juri > > On 12.10.25 13:01, Willem van der Westhuizen wrote: >> Hi >> >> We use existdb extensively to run xqueries on our external databases, >> couchdb and others. We dont use the existdb datastore for our data. >> >> We used to run on an old 3.3.3 version of existdb, and recently >> migrated to 6.2. Since then we have found that existdb locks the >> xquery that is being executed, causing deadlocks that eventually >> locks up the server. Is there an easy way to tell existdb to revert >> to the older locking behaviour that does not lock up the xquery being >> run? >> >> >> Regards >> > > > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open |
From: Pieter L. <pie...@be...> - 2025-10-15 10:34:47
|
Hi, Might it be that there are any modules that include each other causing circular dependencies? I believe eXist has become a little pickier about this since version 4 and we've ironed them all out of our code base. Best, Pieter On 10/15/2025 12:24 PM, Willem van der Westhuizen wrote: > > Hi, > > In our case we are not updating when the locking occurs. We simply > call the rest services, and then they call other rest services outside > of exist. We use local curl statements to call the rest services, and > that causes the lockup. And we can see that the waiting for locks are > growing. It seems that if the rest services are called from the > outside, and they themselves call rest services, the later versions > lock up. And when the server reached about 160 processed it always > crashes. I have attached the screen shots of the monex. If you look > at the logs, its frozen, nothing running. > > > > On 2025/10/14 11:23, Pieter Lamers via Exist-open wrote: >> Hi Willem, >> >> I would suggest using the latest release or build rather than 6.2.0 >> and checking if the same behavior occurs. My personal experience is >> that lock ups occur when I try to update XQuery resources while they >> are in use. Therefore I do not update the applications while they are >> running production. >> >> Best, >> Pieter >> >> On 10/12/2025 1:01 PM, Willem van der Westhuizen wrote: >>> Hi >>> >>> We use existdb extensively to run xqueries on our external >>> databases, couchdb and others. We dont use the existdb datastore for >>> our data. >>> >>> We used to run on an old 3.3.3 version of existdb, and recently >>> migrated to 6.2. Since then we have found that existdb locks the >>> xquery that is being executed, causing deadlocks that eventually >>> locks up the server. Is there an easy way to tell existdb to revert >>> to the older locking behaviour that does not lock up the xquery >>> being run? >>> >>> >>> Regards >>> >> -- Pieter Lamers John Benjamins Publishing Company Postal Address: P.O. Box 36224, 1020 ME AMSTERDAM, The Netherlands Visiting Address: Klaprozenweg 75G, 1033 NN AMSTERDAM, The Netherlands Warehouse: Kelvinstraat 11-13, 1446 TK PURMEREND, The Netherlands tel: +31 20 630 4747 web:www.benjamins.com |
From: Willem v. d. W. <wi...@kw...> - 2025-10-15 10:25:17
|
Hi, In our case we are not updating when the locking occurs. We simply call the rest services, and then they call other rest services outside of exist. We use local curl statements to call the rest services, and that causes the lockup. And we can see that the waiting for locks are growing. It seems that if the rest services are called from the outside, and they themselves call rest services, the later versions lock up. And when the server reached about 160 processed it always crashes. I have attached the screen shots of the monex. If you look at the logs, its frozen, nothing running. On 2025/10/14 11:23, Pieter Lamers via Exist-open wrote: > Hi Willem, > > I would suggest using the latest release or build rather than 6.2.0 > and checking if the same behavior occurs. My personal experience is > that lock ups occur when I try to update XQuery resources while they > are in use. Therefore I do not update the applications while they are > running production. > > Best, > Pieter > > On 10/12/2025 1:01 PM, Willem van der Westhuizen wrote: >> Hi >> >> We use existdb extensively to run xqueries on our external databases, >> couchdb and others. We dont use the existdb datastore for our data. >> >> We used to run on an old 3.3.3 version of existdb, and recently >> migrated to 6.2. Since then we have found that existdb locks the >> xquery that is being executed, causing deadlocks that eventually >> locks up the server. Is there an easy way to tell existdb to revert >> to the older locking behaviour that does not lock up the xquery being >> run? >> >> >> Regards >> > |
From: Juri L. <ju...@ex...> - 2025-10-15 08:30:53
|
Hi Willem! The behaviour you are describing is usually a sign for an attempt to replace/update a running XQuery. If this is the case the easiest solution is to make sure that the query you want to replace / change is not being run at the moment. I agree that we should investigate if we even have to lock the XQuery in the first place. But that is a separate discussion. Regards, Juri On 12.10.25 13:01, Willem van der Westhuizen wrote: > Hi > > We use existdb extensively to run xqueries on our external databases, > couchdb and others. We dont use the existdb datastore for our data. > > We used to run on an old 3.3.3 version of existdb, and recently > migrated to 6.2. Since then we have found that existdb locks the > xquery that is being executed, causing deadlocks that eventually locks > up the server. Is there an easy way to tell existdb to revert to the > older locking behaviour that does not lock up the xquery being run? > > > Regards > |
From: Lars W. <lar...@ex...> - 2025-10-14 10:29:21
|
It’s really weird why the app should be slow on a system with more ghz. There must be some kind of other limitation in the new environment. With regard to Java 21: Where did you read that eXist-db 6.x would be bound to max Java 17? I know at least two people running eXist-db 6 with Java 21. I’m aware of some discussion about some edge cases that could cause trouble with Java 21 but afaik it’s working fine at least for those two people. Did you ever try it yourself? Nevertheless, if the performance was fine in the old environment but is not in the new one, the Java version will very likely not be the problem because you did not run Java 21 in the old system. Sorry I can’t help more but without actually looking in the new and old environment and running a few tests it’s kind of impossible to say, what the issue could be. Fingers crossed you can sort it out! Best, Lars > On 8. Oct 2025, at 17:34, Eduard Drenth <ed...@fr...> wrote: > > Thanks for the info, good to know about one xquery being bound to one core. > > In the mean time we turned hyperthreading off, that helped, but not enough yet. > > It is weird, the new cpu is 3.8ghz compared to the old 2.4ghz. > > The queries seem cpu bound as top shows cpu percentage above 200%. > > Increasing memory does not help, IO does not seem to be a problem. > > It is doable for now, but I am happy that new hardware is being ordered, hope that helps. > > Unfortunately I cannot benefit from the many performance improvements in java either, because exist 6 is bound to java 17 max. > > Regards, > Eduard Drenth, Software Architekt > > ed...@fr... > > Doelestrjitte 8 > 8911 DX Ljouwert > (058) 213 14 14 > +31 62 094 34 28 > https://www.fryske-akademy.nl/ > https://frysker.nl/ > https://frisian.eu/<Outlook-d4b4dmlh.svg> > gpg: https://keyserver.ubuntu.com/pks/lookup?search=eduard+drenth&fingerprint=on&op=index > From: Lars Windauer <lar...@ex...> > Sent: Tuesday, October 7, 2025 10:26 PM > To: Exist-open <Exi...@li...> > Subject: Re: [Exist-open] exist multiple cores > Dear Eduard, > > eXist-db can benefit from multiple cores without the need to configure anything. Having said this, please be aware that a single XQuery (dbbroker) can always run on only one core! So a single XQuery can’t utilise 20 cores but on a 20 core system you can have 20 parallel queries that each use one core. > > Can you provide us with information about the old and new environment? Gigahertz of your CPUs, how much RAM does the system have and how much of this is assigned to eXist-db. What about the hard drives, is eXist-db running from a SSD with the same filesystem as in the old environment? > > Best, > > Lars > > > > On 6. Oct 2025, at 16:42, Eduard Drenth <ed...@fr...> wrote: > > > > Dear all, > > > > We are in the process of migrating exist from our own hardware to a provider. > > > > Performance drops significantly, the main cause is that exist seems to be using one core only. > > > > Can exist benefit from multiple cores? And if yes, how is that configured? > > > > Regards, > > > > Eduard Drenth, Software Architekt > > > > ed...@fr... > > _______________________________________________ > > Exist-open mailing list > > Exi...@li... > > https://lists.sourceforge.net/lists/listinfo/exist-open > > > > > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open |
From: Pieter L. <pie...@be...> - 2025-10-14 09:38:42
|
Hi Willem, I would suggest using the latest release or build rather than 6.2.0 and checking if the same behavior occurs. My personal experience is that lock ups occur when I try to update XQuery resources while they are in use. Therefore I do not update the applications while they are running production. Best, Pieter On 10/12/2025 1:01 PM, Willem van der Westhuizen wrote: > Hi > > We use existdb extensively to run xqueries on our external databases, > couchdb and others. We dont use the existdb datastore for our data. > > We used to run on an old 3.3.3 version of existdb, and recently > migrated to 6.2. Since then we have found that existdb locks the > xquery that is being executed, causing deadlocks that eventually locks > up the server. Is there an easy way to tell existdb to revert to the > older locking behaviour that does not lock up the xquery being run? > > > Regards > -- Pieter Lamers John Benjamins Publishing Company Postal Address: P.O. Box 36224, 1020 ME AMSTERDAM, The Netherlands Visiting Address: Klaprozenweg 75G, 1033 NN AMSTERDAM, The Netherlands Warehouse: Kelvinstraat 11-13, 1446 TK PURMEREND, The Netherlands tel: +31 20 630 4747 web: www.benjamins.com |
From: Adam R. <ad...@ex...> - 2025-10-14 08:49:22
|
Thanks for the info, good to know about one xquery being bound to one core. > It's not as simple as that I am afraid. The shared resources within eXist-db are guarded by locks. At the dbx file level, eXist-db uses a mutex as a lock, and so can only ever have 1 thread, that thread can be either reading or writing, but not both. At the higher Document and Collection object level, each of those also has at least one ReadWrite lock, there may be a single writer thread, or multiple reader threads, but throughput here is limited by the underlying dbx level when needing to load or save those objects. If you are not performing store/load operations, then key to performance can be how you layout your data in documents and collections. Both use a hierarchical locking scheme along their URI path, so you want to avoid common parent collections where contention for locks is high. In the mean time we turned hyperthreading off, that helped, but not enough > yet. > Makes sense. For eXist-db you will want the most performance per-core possible. It is weird, the new cpu is 3.8ghz compared to the old 2.4ghz. > The queries seem cpu bound as top shows cpu percentage above 200%. > It would be interesting to know where that time is spent. Is it lock contention, or is it meaningful work? > Increasing memory does not help, IO does not seem to be a problem. > If you are CPU bound, changing those worn't help you. Unfortunately I cannot benefit from the many performance improvements in > java either, because exist 6 is bound to java 17 max. > eXist-db 6 actually requires Java 8. -- Adam Retter eXist Core Developer { United Kingdom } ad...@ex... |
From: Adam R. <ad...@ex...> - 2025-10-14 08:30:20
|
Hi Willem, In eXist-db 5.x.x and 6.x.x, the XQuery (if stored in the database) is treated as any other document, and so it is locked for the duration of operations that happen upon it. Unfortunately this means that whilst a query is executing the document holding that query will keep a READ lock on the document. A document may have many either a single write lock, or one or more read locks. This means that the query can only be updated by one person/client at a time, but that multiple clients can run the query in parallel. I am not sure how a deadlock could occur with this. Can you provide some further information about your issue, and how to reproduce it please? Thanks, Adam. On Tue, 14 Oct 2025 at 08:53, Willem van der Westhuizen <wi...@kw...> wrote: > Hi > > We use existdb extensively to run xqueries on our external databases, > couchdb and others. We dont use the existdb datastore for our data. > > We used to run on an old 3.3.3 version of existdb, and recently migrated > to 6.2. Since then we have found that existdb locks the xquery that is > being executed, causing deadlocks that eventually locks up the server. > Is there an easy way to tell existdb to revert to the older locking > behaviour that does not lock up the xquery being run? > > > Regards > > -- > Willem van der Westhuizen > +27 82 9200718 > > > > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open > -- Adam Retter eXist Core Developer { United Kingdom } ad...@ex... |
From: Alasdair D. <ala...@gm...> - 2025-10-13 21:09:09
|
Hi Dannes, You’re in luck, grab yourself a bowl of popcorn and enjoy the movie. I save using the eXide Save button. You will see the screencast shows the unusual behaviour. See: https://share.icloud.com/photos/001aTqWO4WgYunQQaVKSTHJdA Thanks in advance, Alasdair Sent from my iPhone > On 14 Oct 2025, at 6:01 am, Dannes Wessels <di...@ex...> wrote: > > Hi, > >> On 29 Sep 2025, at 03:56, Alasdair Dougall <ala...@gm...> wrote: >> >> The issue is on saving and closing the file, it expand the href to: > > How do you do the save? > > Cheers > > Dannes > |
From: Dannes W. <di...@ex...> - 2025-10-13 21:04:17
|
Hi, > On 29 Sep 2025, at 03:56, Alasdair Dougall <ala...@gm...> wrote: > > The issue is on saving and closing the file, it expand the href to: How do you do the save? Cheers Dannes |
From: Willem v. d. W. <wi...@kw...> - 2025-10-12 12:05:32
|
Hi We use existdb extensively to run xqueries on our external databases, couchdb and others. We dont use the existdb datastore for our data. We used to run on an old 3.3.3 version of existdb, and recently migrated to 6.2. Since then we have found that existdb locks the xquery that is being executed, causing deadlocks that eventually locks up the server. Is there an easy way to tell existdb to revert to the older locking behaviour that does not lock up the xquery being run? Regards -- Willem van der Westhuizen +27 82 9200718 |