Re: [Exist-open] Urgent assistance with locking behaviour of existdb - or how to change the architecture of a scheduled ETL service

[Willem v. d. W. <wi...@kw...>]

Hi Dannes,

No, it more the locking behaviour of later versions of exist where 
running many instances of the same xquery as a rest service that makes 
rest services external to existdb. Even though there are no updates, 
running multiple instances do seem to place some form of read lock on 
the xquery that locks the instance up.

Regards

Willem

On 2025/10/26 21:34, Dannes Wessels wrote:
> Hi Willem
>
>> On 18 Oct 2025, at 12:54, Willem van der Westhuizen 
>> <wi...@kw...> wrote:
>>
>> The etl engine is an existdb module, that is called by an xquery. One 
>> hypothesis is that each time that the xquery runs, it loads an 
>> instance of the module into memory, and thats is where the locking 
>> happens. No way to tell if that is indeed the case, and what the 
>> internal limiations are.
>>
>> The one interesting observation is that even in older version 3.3.3 
>> if the xquery is pending up due to the remote service being called is 
>> stalled, then the number of instances in exists always stop 
>> responding at around 160 plus or minus, then the whole exist instance 
>> locks up. The number of database brokers, and memory etc. was never a 
>> limit.
>>
>> In the later versions of exist the deadlock may occur at much lover 
>> levels. To give an example, we have rewritten most of the critical 
>> code that locked up in javascript, and there was no issue in the 
>> downstream services.
>>
>> The explanation offered of recursive calling of modules might be a 
>> clue, because that is certainly the case in our code. We would prefer 
>> to keep the ETL in exist though, hence trying to find a solution.
>>
> Is it about the couch base driver I created a long time ago?
> I never checked the code for newer eXist-db versions..
>
> With kind regards
>
> Dannes
>
>