Re: [Exist-open] Urgent assistance with locking behaviour of existdb - or how to change the architecture of a scheduled ETL service

[Dannes W. <di...@ex...>]

Hi Willem

> On 18 Oct 2025, at 12:54, Willem van der Westhuizen <wi...@kw...> wrote:
> 
> The etl engine is an existdb module, that is called by an xquery. One hypothesis is that each time that the xquery runs, it loads an instance of the module into memory, and thats is where the locking happens. No way to tell if that is indeed the case, and what the internal limiations are.
> 
> The one interesting observation is that even in older version 3.3.3 if the xquery is pending up due to the remote service being called is stalled, then the number of instances in exists always stop responding at around 160 plus or minus, then the whole exist instance locks up. The number of database brokers, and memory etc. was never a limit.
> 
> In the later versions of exist the deadlock may occur at much lover levels. To give an example, we have rewritten most of the critical code that locked up in javascript, and there was no issue in the downstream services. 
> 
> The explanation offered of recursive calling of modules might be a clue, because that is certainly the case in our code. We would prefer to keep the ETL in exist though, hence trying to find a solution.
> 
Is it about the couch base driver I created a long time ago?
I never checked the code for newer eXist-db versions..

With kind regards

Dannes