Menu
▾
▴
Re: [Exist-open] Urgent assistance with locking behaviour of existdb - or how to change the architecture of a scheduled ETL service
|
From: Willem v. d. W. <wi...@kw...> - 2025-10-18 10:56:35
|
Hi, The etl engine is an existdb module, that is called by an xquery. One hypothesis is that each time that the xquery runs, it loads an instance of the module into memory, and thats is where the locking happens. No way to tell if that is indeed the case, and what the internal limiations are. The one interesting observation is that even in older version 3.3.3 if the xquery is pending up due to the remote service being called is stalled, then the number of instances in exists always stop responding at around 160 plus or minus, then the whole exist instance locks up. The number of database brokers, and memory etc. was never a limit. In the later versions of exist the deadlock may occur at much lover levels. To give an example, we have rewritten most of the critical code that locked up in javascript, and there was no issue in the downstream services. The explanation offered of recursive calling of modules might be a clue, because that is certainly the case in our code. We would prefer to keep the ETL in exist though, hence trying to find a solution. Regards Willem On 2025/10/18 11:38, Claudius Teodorescu wrote: > Hi! > > Would it help to have the ETL engine as an eXist-db module? > > > Claudius Teodorescu > > On Sat, 18 Oct 2025 at 11:29, Willem van der Westhuizen > <wi...@kw...> wrote: > > Maybe there is a different question to ask. We are using exist and > xquery extensively with an etl engine that we wrote to read data from > couchdb and place into MySQL for reporting purposes. We are currently > using a shell script to launch 20 instances of the the etl with > different parameters to pick up a batch and process it. Would > there be a > better way in exist to write the scheduler to make those calls, > and then > to wait for each instance to be complete before calling the next, > that > would not involve calling the same rest service muliple times from > curl? > > Regards > > Willem > > On 2025/10/15 11:48, Willem van der Westhuizen wrote: > > Definitely not updating. Just running, > > > > On 2025/10/15 10:30, Juri Leino wrote: > >> Hi Willem! > >> > >> The behaviour you are describing is usually a sign for an > attempt to > >> replace/update a running XQuery. If this is the case the easiest > >> solution is to make sure that the query you want to replace / > change > >> is not being run at the moment. > >> > >> I agree that we should investigate if we even have to lock the > XQuery > >> in the first place. But that is a separate discussion. > >> > >> Regards, > >> Juri > >> > >> On 12.10.25 13:01, Willem van der Westhuizen wrote: > >>> Hi > >>> > >>> We use existdb extensively to run xqueries on our external > >>> databases, couchdb and others. We dont use the existdb > datastore for > >>> our data. > >>> > >>> We used to run on an old 3.3.3 version of existdb, and recently > >>> migrated to 6.2. Since then we have found that existdb locks the > >>> xquery that is being executed, causing deadlocks that eventually > >>> locks up the server. Is there an easy way to tell existdb to > revert > >>> to the older locking behaviour that does not lock up the xquery > >>> being run? > >>> > >>> > >>> Regards > >>> > >> > >> > >> _______________________________________________ > >> Exist-open mailing list > >> Exi...@li... > >> https://lists.sourceforge.net/lists/listinfo/exist-open > > > > > > _______________________________________________ > > Exist-open mailing list > > Exi...@li... > > https://lists.sourceforge.net/lists/listinfo/exist-open > > > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open > > > > -- > Cu stimă, > Claudius Teodorescu |
