[Exist-open] 502 Proxy Error / HTTP timeout

[mountainbiker <mai...@gm...>]

Background:
Friday evening I wrote an XQuery that calls an external web service with an
id to retrieve an XML data packet.  That is, one call using doc('web service
with id') per id.  The XQuery works as desired with a few calls to the web
service.  But as I increase the number, I "timeout".  In short, I will need
to call the web service 5,000 times, and (if not timed out) I would estimate
the XQuery to run for 30 minutes.  The purpose of this XQuery is to build a
XML file (i.e., "lookup table") once a week.

Problem:
After a short period of time, the browser receives a 502 Proxy Error:  proxy
server received an invalid response from the upstream server.  (Yes, I am
running eXist behind an Apache proxy.)  Note:  even after I see this browser
response, I can see the XQuery still running and logging for many additional
minutes (but not to completion).

Ideas:
In Apache I was thinking of setting the proxy timeout as well as the
proxy-initial-not-pooled variable.  About the latter variable, Apache
states, "If this variable is set no pooled connection will be reused if the
client connection is an initial connection. This avoids the "proxy: error
reading status line from remote server" error message caused by the race
condition that the backend server closed the pooled connection after the
connection check by the proxy and before data sent by the proxy reached the
backend."

Additional, I was thinking of using eXist's httpclient:get() instead of
doc(), so I can set "persistent" to true().

Questions:
1.  Has anyone encountered such a problem and resolved it?
2.  Do you see any issues with my ideas?
3.  Can I run the XQuery via the "command line" without having to kick it
off via the browser (and, thus, avoid the timeout altogether)?

Thank you in advance for any suggestions.
-- 
View this message in context: http://exist.2174344.n4.nabble.com/502-Proxy-Error-HTTP-timeout-tp2261695p2261695.html
Sent from the exist-open mailing list archive at Nabble.com.