Menu
▾
▴
[Exist-open] Uploading a large file (21GB) causes "java.lang.OutOfMemoryError: Java heap space" error
|
From: Arash J. <ara...@gm...> - 2013-10-03 12:25:34
|
Hi, I have installed the latest version (2.1) and can use the Java Admin Client to upload relatively small files (<1GB) with no problem, however, uploading a 21GB xml file I get the following exception thrown 2013-10-02 20:34:52,487 [eXistThread-58] WARN (NativeTextEngine.java [storeAttribute]:209) - Token length exceeded 2048: 160,nbsp,38,amp,162,... 2013-10-02 20:36:44,065 [eXistThread-58] WARN (NativeTextEngine.java [storeAttribute]:209) - Token length exceeded 2048: wepdwukltg4mdg2nja2n... 2013-10-02 20:56:48,833 [eXistThread-58] WARN (NativeTextEngine.java [storeAttribute]:209) - Token length exceeded 2048: http://view.ncbi.nlm... 2013-10-02 20:56:48,834 [eXistThread-58] WARN (NativeTextEngine.java [storeAttribute]:209) - Token length exceeded 2048: http://view.ncbi.nlm... 2013-10-02 20:57:04,736 [eXistThread-58] WARN (NativeTextEngine.java [storeAttribute]:209) - Token length exceeded 2048: dqo8zz4ncgk8zz4ncgkj... 2013-10-02 21:07:41,020 [eXistThread-58] WARN (NativeTextEngine.java [storeAttribute]:209) - Token length exceeded 2048: 13763208823213770506... 2013-10-02 21:12:37,301 [eXistThread-58] WARN (NativeTextEngine.java [storeAttribute]:209) - Token length exceeded 2048: wepdwujmty1odq0njqxd... 2013-10-02 21:17:12,017 [eXistThread-58] WARN (NativeTextEngine.java [storeAttribute]:209) - Token length exceeded 2048: users/jgc/gwt-versio... 2013-10-02 21:19:11,786 [eXistThread-58] WARN (BFile.java [append]:217) - Key length exceeds page size! Skipping key ... 2013-10-02 21:20:02,752 [eXistThread-58] WARN (NativeTextEngine.java [storeAttribute]:209) - Token length exceeded 2048: d4334241363334433346... 2013-10-02 21:20:02,753 [eXistThread-58] WARN (NativeTextEngine.java [storeAttribute]:209) - Token length exceeded 2048: d4334241363334433346... 2013-10-02 21:27:24,126 [eXistThread-58] WARN (BFile.java [append]:217) - Key length exceeds page size! Skipping key ... 2013-10-02 21:32:51,738 [eXistThread-58] WARN (NativeTextEngine.java [storeAttribute]:209) - Token length exceeded 2048: 288,217,503,426,497,... 2013-10-02 21:48:50,070 [eXistThread-58] WARN (NativeTextEngine.java [storeAttribute]:209) - Token length exceeded 2048: ffd8ffe000104a464946... 2013-10-02 21:53:55,648 [eXistThread-58] WARN (NativeTextEngine.java [storeAttribute]:209) - Token length exceeded 2048: 37107287533902102798... 2013-10-02 21:53:55,649 [eXistThread-58] WARN (NativeTextEngine.java [storeAttribute]:209) - Token length exceeded 2048: 37107287533902102798... 2013-10-02 21:53:55,649 [eXistThread-58] WARN (NativeTextEngine.java [storeAttribute]:209) - Token length exceeded 2048: 37107287533902102798... 2013-10-02 22:04:01,852 [eXistThread-58] WARN (NativeTextEngine.java [storeAttribute]:209) - Token length exceeded 2048: wepdwulltewmtk1mtq1m... 2013-10-02 22:14:58,124 [eXistThread-58] WARN (NativeTextEngine.java [storeAttribute]:209) - Token length exceeded 2048: n/9dgufvvvvvvvvvvvvv... 2013-10-02 22:15:02,949 [eXistThread-58] WARN (NativeTextEngine.java [storeAttribute]:209) - Token length exceeded 2048: u0041-\u005a\u0061-\... 2013-10-02 22:15:02,979 [eXistThread-58] WARN (NativeTextEngine.java [storeAttribute]:209) - Token length exceeded 2048: 0041-005a0061-007a00... 2013-10-02 22:15:05,010 [eXistThread-58] WARN (NativeTextEngine.java [storeAttribute]:209) - Token length exceeded 2048: pd94bwwgdmvyc2lvbj0i... 2013-10-02 22:28:10,273 [eXistThread-58] WARN (NativeTextEngine.java [storeAttribute]:209) - Token length exceeded 2048: 25461638709540284156... 2013-10-02 22:34:21,215 [eXistThread-58] WARN (NativeTextEngine.java [storeAttribute]:209) - Token length exceeded 2048: 7.7495976,12.1007857... 2013-10-02 22:36:36,100 [eXistThread-58] INFO (NativeBroker.java [sync]:3669) - Memory: 10,353,856K total; 11,832,896K max; 5,207,632K free 2013-10-02 22:36:36,100 [eXistThread-58] INFO (BTree.java [printStatistics]:2387) - dom.dbx INDEX Buffers occupation : 100% (144 out of 144) Cache efficiency : 100% 2013-10-02 22:36:36,101 [eXistThread-58] INFO (DOMFile.java [printStatistics]:1369) - dom.dbx DATA Buffers occupation : 100% (256 out of 256) Cache efficiency : 100% 2013-10-02 22:36:36,101 [eXistThread-58] INFO (BTree.java [printStatistics]:2387) - collections.dbx INDEX Buffers occupation : 16% (10 out of 64) Cache efficiency : 99% 2013-10-02 22:36:36,101 [eXistThread-58] INFO (BFile.java [printStatistics]:434) - collections.dbx DATA Buffers occupation : 31% (20 out of 64) Cache efficiency : 99% 2013-10-02 22:36:36,101 [eXistThread-58] INFO (BTree.java [printStatistics]:2387) - values.dbx INDEX Buffers occupation : 0% (0 out of 64) Cache efficiency : N/A 2013-10-02 22:36:36,101 [eXistThread-58] INFO (BFile.java [printStatistics]:434) - values.dbx DATA Buffers occupation : 0% (0 out of 64) Cache efficiency : N/A 2013-10-02 22:36:36,814 [eXistThread-58] DEBUG (RpcConnection.java [handleException]:124) - Java heap space java.lang.OutOfMemoryError: Java heap space at org.exist.storage.OccurrenceList.ensureCapacity(OccurrenceList.java:85) at org.exist.storage.OccurrenceList.add(OccurrenceList.java:39) at org.exist.storage.NativeTextEngine$InvertedIndex.addAttribute(NativeTextEngine.java:939) at org.exist.storage.NativeTextEngine.storeAttribute(NativeTextEngine.java:223) at org.exist.fulltext.FTIndexWorker$FTStreamListener.attribute(FTIndexWorker.java:268) at org.exist.indexing.AbstractStreamListener.attribute(AbstractStreamListener.java:61) at org.exist.storage.structural.NativeStructuralIndexWorker$NativeStructuralStreamListener.attribute(NativeStructuralIndexWorker.java:688) at org.exist.Indexer.startElement(Indexer.java:676) at org.exist.collections.triggers.ContentHandlerWrapper.startElement(ContentHandlerWrapper.java:89) at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source) at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source) at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at org.exist.collections.Collection$2.run(Collection.java:1226) at org.exist.collections.Collection.storeXMLInternal(Collection.java:1379) at org.exist.collections.Collection.store(Collection.java:1206) at org.exist.xmlrpc.RpcConnection.parseLocal(RpcConnection.java:2469) at org.exist.xmlrpc.RpcConnection.parseLocalExt(RpcConnection.java:2370) at org.exist.xmlrpc.RpcConnection.parseLocalExt(RpcConnection.java:5447) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.xmlrpc.server.ReflectiveXmlRpcHandler.invoke(ReflectiveXmlRpcHandler.java:115) at org.apache.xmlrpc.server.ReflectiveXmlRpcHandler.execute(ReflectiveXmlRpcHandler.java:106) at org.apache.xmlrpc.server.XmlRpcServerWorker.execute(XmlRpcServerWorker.java:46) 2013-10-02 22:36:36,918 [eXistThread-58] INFO (NativeBroker.java [sync]:3669) - Memory: 10,353,856K total; 11,832,896K max; 5,175,307K free 2013-10-02 22:36:36,919 [eXistThread-58] INFO (BTree.java [printStatistics]:2387) - dom.dbx INDEX Buffers occupation : 100% (144 out of 144) Cache efficiency : 100% 2013-10-02 22:36:36,919 [eXistThread-58] INFO (DOMFile.java [printStatistics]:1369) - dom.dbx DATA Buffers occupation : 100% (256 out of 256) Cache efficiency : 100% 2013-10-02 22:36:36,919 [eXistThread-58] INFO (BTree.java [printStatistics]:2387) - collections.dbx INDEX Buffers occupation : 16% (10 out of 64) Cache efficiency : 99% 2013-10-02 22:36:36,919 [eXistThread-58] INFO (BFile.java [printStatistics]:434) - collections.dbx DATA Buffers occupation : 31% (20 out of 64) Cache efficiency : 99% 2013-10-02 22:36:36,919 [eXistThread-58] INFO (BTree.java [printStatistics]:2387) - values.dbx INDEX Buffers occupation : 0% (0 out of 64) Cache efficiency : N/A 2013-10-02 22:36:36,919 [eXistThread-58] INFO (BFile.java [printStatistics]:434) - values.dbx DATA Buffers occupation : 0% (0 out of 64) Cache efficiency : N/A 64 bit Ubuntu + 16GB RAM eXists-settings.sh . . . set_client_java_options() { if [ -z "${CLIENT_JAVA_OPTIONS}" ]; then CLIENT_JAVA_OPTIONS="-Xms128m -Xmx13000m -Dfile.encoding=UTF-8"; fi OS=`uname` if [ "${OS}" == "Darwin" ]; then CLIENT_NAME="Client" JAVA_OPTIONS="${CLIENT_JAVA_OPTIONS} -Djava.endorsed.dirs=${JAVA_ENDORSED_DIRS} -Xdock:icon=${EXIST_HOME}/icon.png -Xdock:name=${CLIENT_NAME}"; else JAVA_OPTIONS="${CLIENT_JAVA_OPTIONS} -Djava.endorsed.dirs=${JAVA_ENDORSED_DIRS}"; fi } set_java_options() { if [ -z "${JAVA_OPTIONS}" ]; then JAVA_OPTIONS="-Xms128m -Xmx13000m -Dfile.encoding=UTF-8"; fi JAVA_OPTIONS="${JAVA_OPTIONS} -Djava.endorsed.dirs=${JAVA_ENDORSED_DIRS}"; } conf.xml . . . <db-connection cacheSize="293" checkMaxCacheSize="true" collectionCache="6207" database="native" files="/home/arash/eXist-db/webapp/WEB-INF/data" pageSize="4096" nodesBuffer="1000" cacheShrinkThreshold="10000" doc-ids="default" minDiskSpace="128M"> |
