Re: [Exist-open] 504 error in relatively small query

[Lizzi, V. <Vin...@ta...>]

Hi Ralph,

In addition to what Chris wrote about storing the results in the database, it may be helpful to create a separate collection for storing the query results to a avoid a write lock contention on the collection that holds the documents that are being read.  I tend to use a /db/temp collection for this purpose.

Another technique for long-running queries is to schedule the query to run as a background task with the Scheduler module, and then use Monex to see when the query finishes.

scheduler:schedule-xquery-periodic-job('/db/queries/name.xq', 0, 'name', (), 0, 0)

It can also be helpful to use util:log() to record messages about the progress or completion of the query in exist.log.

If you are getting errors about timeout or exceeding the maximum number of nodes, you might need to increase the timeout setting or output size limit setting, which can be done by declaring these options at the start of a query:

declare option exist:timeout "900000";
declare option exist:output-size-limit "1000000";

Most of this information is in the online documentation at exist-db.org but it can take a while to find and piece together what you need, so this forum is a helpful place to ask questions.

Cheers,
Vincent

_____________________________________________
Vincent M. Lizzi
Head of Information Standards | Taylor & Francis Group
vin...@ta...<mailto:vin...@ta...>

From: Ralph Corrigan <ra...@ra...>
Sent: Monday, March 13, 2023 11:37 AM
To: Chris Wallace <kit...@gm...>
Cc: ol...@ex...; Exi...@li...
Subject: Re: [Exist-open] 504 error in relatively small query


Thanks Chris, also useful to know.



On 13/03/2023 15:13, Chris Wallace wrote:
Doesn't the query keep running anyway?  I often have long jobs which timeout in the browser but run to completion (unless I terminate them in the Monitor).  When this is anticipated , I make sure the job writes output to a database file rather than (only) to return HTML

Chris Wallace

On Mon, Mar 13, 2023 at 3:00 PM Ralph Corrigan <ra...@ra...<mailto:ra...@ra...>> wrote:

Thanks Olaf, yes we are. I'll take a look at that...



Ralph



On 13/03/2023 14:35, ol...@ex...<mailto:ol...@ex...> wrote:
This returns absolutely fine up to ~1200 items in the subsequence $ref but
by the time we get closer to 2000 (and more) this returns a 504 gateway
time-out error in roughly 60s.

Are you running eXist behind a proxy like nginx?  I've had 504s created
from nginx frontend proxies.  In that case, you need to increase timeouts
in the nginx config.

Olaf


_______________________________________________
Exist-open mailing list
Exi...@li...<mailto:Exi...@li...>
https://lists.sourceforge.net/lists/listinfo/exist-open<https://lists.sourceforge.net/lists/listinfo/exist-open>
_______________________________________________
Exist-open mailing list
Exi...@li...<mailto:Exi...@li...>
https://lists.sourceforge.net/lists/listinfo/exist-open<https://lists.sourceforge.net/lists/listinfo/exist-open>


Information Classification: General