[Exist-open] too many open files when getting query result

[Christophe T. <ct...@ta...>]

Hi

We are having big performance and scalability issues with eXist right 
now. One of the blocking problems is we keep getting XMLDBException 
("too many open files") when accessing xquery results in a stress test / 
benchmark. We have millions of very small documents and just a few (15) 
concurrent users.

OS is Ubuntu 9.10 "Karmic", eXist is 1.4 deployed as a WAR in JBoss 
4.2.2, but still accessed through XML-RPC. The same occurs with a 
separate eXist 1.4 (started through server.sh).

Complete stack below.

"org.exist.xmldb.AbstractRemoteResource.getRemoteContentIntoLocalFile" 
really stroke me, so I looked into the source code of org.exist.xmldb 
and was chocked to find out the implementation of Resource.getContent 
actually creates files (eXistRPC*.xml and eXistARR*.xml). Apparently the 
files are closed on finalized(), so on a Linux box with a default limit 
of 1024 file descriptors per process but a lot of memory to spare, we 
were bound to hit the wall before the finalization kicked in.

1) Considering our Web browsing & searching capabilities, we execute 
many different queries, so we must be creating thousands of *.xml files 
on the server. I guess there are very good reasons for creating those 
files. What are they?

2) If we can't escape creating one or several files for each query, any 
idea on how to alleviate our problem ("ulimit" is not an answer, thank 
you!)? I was thinking of calling setContent(null) on each Resource to 
close the file early.

Thanks in advance

-Christophe Toum
PM MDM & DW
Talend


org.xmldb.api.base.XMLDBException: Failed to invoke method 
retrieveFirstChunk in class org.exist.xmlrpc.RpcConnection: 
java.io.FileNotFoundException: 
/home/pascal/dev/talend/installations/TMDMEE-Server-All-r43528-V4.0.2NB/jboss-4.2.2.GA/server/default/work/jboss.web/localhost/exist/cocoon-files/cache-dir/eXistRPCC6569818493073070477.xml 
(Too many open files)
         at 
org.exist.xmldb.AbstractRemoteResource.getRemoteContentIntoLocalFile(AbstractRemoteResource.java:332)
         at 
org.exist.xmldb.AbstractRemoteResource.getExtendedContentInternal(AbstractRemoteResource.java:449)
         at 
org.exist.xmldb.RemoteXMLResource.getExtendedContent(RemoteXMLResource.java:431)
         at 
org.exist.xmldb.AbstractRemoteResource.getContent(AbstractRemoteResource.java:89)
         at 
org.exist.xmldb.RemoteXMLResource.getContent(RemoteXMLResource.java:116)
         at 
com.amalto.xmldb.XmldbSLWrapper.runQuery(XmldbSLWrapper.java:829)
         at com.amalto.core.ejb.XmlServerSLWrapperBean.runQuery(Unknown 
Source)
         at sun.reflect.GeneratedMethodAccessor228.invoke(Unknown Source)
         at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
         at java.lang.reflect.Method.invoke(Method.java:597)
         at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
         at 
org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)
         at 
org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
         at 
org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)
         at 
org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
         at 
org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
         at 
org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
         at 
org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
         at 
org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)
         at 
org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
         at 
org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
         at 
org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)
         at org.jboss.ejb.Container.invoke(Container.java:960)
         at 
org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:430)
         at 
org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:103)
         at $Proxy224.runQuery(Unknown Source)
         at 
com.amalto.core.delegator.impl.DefaultItemCtrlDelegator.viewSearch(Unknown 
Source)
         at com.amalto.core.ejb.ItemCtrl2Bean.viewSearch(Unknown Source)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
         at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
         at java.lang.reflect.Method.invoke(Method.java:597)
         at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
         at 
org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)
         at 
org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
         at 
org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)
         at 
org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
         at 
org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
         at 
org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
         at 
org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
         at 
org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)
         at 
org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
         at 
org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
         at 
org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)
         at org.jboss.ejb.Container.invoke(Container.java:960)
         at 
org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:430)
         at 
org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:103)
         at $Proxy233.viewSearch(Unknown Source)
         at 
com.amalto.webapp.core.util.IXtentisRMIPort.viewSearch(Unknown Source)
         at 
com.amalto.webapp.v3.itemsbrowser.servlet.ItemsRemotePaging.doPost(Unknown 
Source)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
         at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
         at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
         at 
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilterjava:96)
         at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
         at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
         at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValvejava:230)
         at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValvejava:175)
         at 
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
         at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:524)
         at 
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
         at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
         at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
         at 
org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
         at 
org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:420)
         at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
         at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
         at 
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
         at 
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
         at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
         at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.xmlrpc.XmlRpcException: Failed to invoke method 
retrieveFirstChunk in class org.exist.xmlrpc.RpcConnection: 
java.io.FileNotFoundException: 
/home/pascal/dev/talend/installations/TMDMEE-Server-All-r43528-V4.0.2NB/jboss-4.2.2.GA/server/default/work/jboss.web/localhost/exist/cocoon-files/cache-dir/eXistRPCC6569818493073070477.xml 
(Too many open files)
         at 
org.apache.xmlrpc.client.XmlRpcStreamTransport.readResponse(XmlRpcStreamTransport.java:197)
         at 
org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:156)
         at 
org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:115)
         at 
org.apache.xmlrpc.client.XmlRpcSunHttpTransport.sendRequest(XmlRpcSunHttpTransport.java:69)
         at 
org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56)
         at 
org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167)
         at 
org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:158)
         at 
org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:147)
         at 
org.exist.xmldb.AbstractRemoteResource.getRemoteContentIntoLocalFile(AbstractRemoteResource.java:268)
         ... 71 more


This message and any attachment (the "message") is intended solely for the addressees and is confidential. If you receive this message by mistake, please delete it and notify the sender immediately. Any use not in accordance with its purpose, any out-spread or disclosure, either as a whole or partially, is prohibited except with formal approval. Internet cannot guarantee the integrity of this message, therefore Talend will not be liable for the message if modified.