[Exist-open] Query taking too long: "Empty reply from server" and "TransformerException"

[Ralf J. <ral...@gm...>]

Hi list,

I'm using an XQuery script to import data into the database: It is sent a huge 
XML document via POST and then stores all it's children as individual 
documents in a collection on the server. It also does some processing (like 
creating a sort index). The request is issued via curl like this:

curl --header 'content-type: text/xml' --data "@$file" -X POST 
http://localhost:8088/db/services/import.xql

However, one of those files that is imported is >40MiB in size (containing >9k 
children), so importing takes quite long. After about 5 to 6 minutes, curl 
reports "curl: (52) Empty reply from server". Looking at the server log files, 
the processing is going on for some more minutes, and at the end, the log 
looks like this:

2010-12-11 17:33:51,565 [qtp1634151355-25] DEBUG (Accounting.java [stats]:172) 
- hits: 62858; misses: 141; thrashing: 1; thrashing period: 30000 
2010-12-11 17:33:51,566 [qtp1634151355-25] DEBUG (DefaultCacheManager.java 
[requestMem]:224) - Growing cache sort.dbx (a 
org.exist.storage.cache.BTreeCache) from 64 to 96 
2010-12-11 17:33:51,654 [qtp1634151355-25] DEBUG (XQuery.java [execute]:242) - 
Execution took 488.905 ms 
2010-12-11 17:33:51,655 [qtp1634151355-25] DEBUG (HTTPUtils.java 
[addLastModifiedHeader]:61) - mostRecentDocumentTime: 0 
2010-12-11 17:33:51,657 [qtp1634151355-25] WARN  (RESTServer.java 
[writeResults]:1931) - javax.xml.transform.TransformerException: 
org.eclipse.jetty.io.EofException

Is there some way to either make putting documents in a collection (or, in 
many cases, replacing existing ones) faster or to increase that timeout? This 
operation is not time-critical, but of course it'd be nice if it was quicker. 
But it is quite important that I know that everything works well, which is 
hard to tell if curl exits unexpectedly.
Even worse, above call is done in a loop, so if I do not cancel on error so 
that the remaining documents are left unimported, they are imported while the 
other import is still going on, making things even slower (so I had to do some 
grepping to get above log, the output is mixed with another thread importing 
another file),

Kind regards,
Ralf Jung