Re: [Exist-open] eXist production server is unable to start up!

[Wolfgang M. <wol...@ex...>]

Hi, 

the exception indicates an error in collections.dbx, which stores the collection hierarchy. I guess the btree could not be fully synced to disk when the db was stopped due to the disk space issues.

I fear you need to backup/restore (though this may not take much longer than a reindex). Use the export tool to generate a low-level backup. It will probably complain about inconsistencies in the collection tree, but hopefully it will be able to extract all data. If not, try with the "direct access" option. Make sure to check the created backup zip, in particular the "lost and found" folder. After creating the backup, remove everything from the data directory and restore.

Wolfgang 

Am Montag, 24. September 2012 um 12:15 schrieb Schouten, Sander:

> Hi,
> 
> We are using eXist 1.4.2 on Linux.
> 
> After the server was run out of diskspace while making a backup, we removed some old exports.
> After starting the eXist db we are getting this error:
> 
> 2012-09-24 09:44:54,738 [WrapperListener_start_runner] INFO (XmlLibraryChecker.java [check]:135) - Looking for a valid Parser...
> Checking for Xerces, found version Xerces-J 2.9.1
> OK!
> 
> 2012-09-24 09:44:54,743 [WrapperListener_start_runner] INFO (XmlLibraryChecker.java [check]:144) - Looking for a valid Transformer...
> Checking for Saxon, found version 9.2.1.5
> OK!
> 
> 2012-09-24 09:44:54,744 [WrapperListener_start_runner] INFO (XmlLibraryChecker.java [check]:153) - Looking for a valid Resolver...
> Checking for Resolver, found version XmlResolver 1.2
> OK!
> 
> 2012-09-24 09:44:54,749 [WrapperListener_start_runner] INFO (XmlLibraryChecker.java [check]:160) - Using parser org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser
> 2012-09-24 09:44:54,799 [WrapperListener_start_runner] INFO (XmlLibraryChecker.java [check]:161) - Using transformer net.sf.saxon.IdentityTransformer
> 2012-09-24 09:44:54,866 [WrapperListener_start_runner] INFO (JettyStart.java [run]:90) - Configuring eXist from /data/existdb-1.4.2/tools/wrapper/bin/../../../conf.xml
> 2012-09-24 09:44:54,866 [WrapperListener_start_runner] INFO (JettyStart.java [run]:91) -
> 2012-09-24 09:44:54,866 [WrapperListener_start_runner] INFO (JettyStart.java [run]:92) - Running with Java 1.6.0_26 [Sun Microsystems Inc. (Java HotSpot(TM) 64-Bit Server VM) in /usr/java/jdk1.6.0_26/jre]
> 2012-09-24 09:44:54,867 [WrapperListener_start_runner] INFO (JettyStart.java [run]:97) -
> 2012-09-24 09:44:54,867 [WrapperListener_start_runner] INFO (JettyStart.java [run]:101) - [eXist Version : 1.4.2]
> 2012-09-24 09:44:54,867 [WrapperListener_start_runner] INFO (JettyStart.java [run]:103) - [eXist Build : 20120416]
> 2012-09-24 09:44:54,868 [WrapperListener_start_runner] INFO (JettyStart.java [run]:105) - [eXist Home : ../../..]
> 2012-09-24 09:44:54,868 [WrapperListener_start_runner] INFO (JettyStart.java [run]:107) - [SVN Revision : 16251]
> 2012-09-24 09:44:54,869 [WrapperListener_start_runner] INFO (JettyStart.java [run]:115) - [Operating System : Linux 2.6.32-131.4.1.el6.x86_64 amd64]
> 2012-09-24 09:44:54,869 [WrapperListener_start_runner] INFO (JettyStart.java [run]:118) - [jetty.home : ../../../tools/jetty]
> 2012-09-24 09:44:54,869 [WrapperListener_start_runner] INFO (JettyStart.java [run]:120) - [log4j.configuration : file:../conf/log4j.xml]
> 2012-09-24 09:44:54,870 [WrapperListener_start_runner] INFO (Configuration.java [<init>]:266) - Reading configuration from file ../../../conf.xml
> 2012-09-24 09:44:55,221 [WrapperListener_start_runner] INFO (eXistURLStreamHandlerFactory.java [init]:53) - Succesfully registered eXistURLStreamHandlerFactory.
> 2012-09-24 09:44:55,224 [WrapperListener_start_runner] INFO (Configuration.java [configureValidation]:1233) - Add catalog uri file:/data/existdb-1.4.2/webapp//WEB-INF/catalog.xml
> 2012-09-24 09:44:55,225 [WrapperListener_start_runner] INFO (GrammarPool.java [<init>]:53) - Initializing GrammarPool.
> 2012-09-24 09:44:55,232 [WrapperListener_start_runner] INFO (BrokerPool.java [<init>]:589) - database instance 'exist' will have between 1 and 20 brokers
> 2012-09-24 09:44:55,232 [WrapperListener_start_runner] INFO (BrokerPool.java [<init>]:596) - database instance 'exist' will be synchronized every 120,000 ms
> 2012-09-24 09:44:55,232 [WrapperListener_start_runner] INFO (BrokerPool.java [<init>]:603) - database instance 'exist' will wait 120,000 ms during shutdown
> 2012-09-24 09:44:55,232 [WrapperListener_start_runner] INFO (BrokerPool.java [<init>]:609) - database instance 'exist' is enabled for transactions : true
> 2012-09-24 09:44:55,344 [WrapperListener_start_runner] INFO (FileLock.java [message]:276) - File lock last access timestamp: Sep 24, 2012 /data/existdb-1.4.2/webapp/WEB-INF/data/dbx_dir.lck
> 2012-09-24 09:44:55,345 [WrapperListener_start_runner] INFO (FileLock.java [message]:276) - Found a stale lockfile. Trying to remove it: /data/existdb-1.4.2/webapp/WEB-INF/data/dbx_dir.lck
> 2012-09-24 09:44:55,345 [WrapperListener_start_runner] INFO (FileLock.java [release]:175) - Deleting lock file: /data/existdb-1.4.2/webapp/WEB-INF/data/dbx_dir.lck
> 2012-09-24 09:44:55,371 [WrapperListener_start_runner] INFO (DefaultCacheManager.java [<init>]:141) - Cache settings: 921,600k; totalPages: 230,400; maxCacheSize: 207,360
> 2012-09-24 09:44:55,393 [WrapperListener_start_runner] INFO (XQueryPool.java [<init>]:112) - QueryPool: size = 128; maxStackSize = 5; timeout = 120,000; timeoutCheckInterval = 30,000
> 2012-09-24 09:44:55,414 [WrapperListener_start_runner] INFO (FileLock.java [message]:276) - File lock last access timestamp: Sep 24, 2012 /data/existdb-1.4.2/webapp/WEB-INF/data/journal.lck
> 2012-09-24 09:44:55,414 [WrapperListener_start_runner] INFO (FileLock.java [message]:276) - Found a stale lockfile. Trying to remove it: /data/existdb-1.4.2/webapp/WEB-INF/data/journal.lck
> 2012-09-24 09:44:55,415 [WrapperListener_start_runner] INFO (FileLock.java [release]:175) - Deleting lock file: /data/existdb-1.4.2/webapp/WEB-INF/data/journal.lck
> 2012-09-24 09:44:55,475 [WrapperListener_start_runner] INFO (IndexManager.java [<init>]:89) - Registered index org.exist.indexing.ngram.NGramIndex as ngram-index
> 2012-09-24 09:44:55,578 [WrapperListener_start_runner] INFO (IndexManager.java [<init>]:89) - Registered index org.exist.indexing.lucene.LuceneIndex as lucene-index
> 2012-09-24 09:44:55,710 [WrapperListener_start_runner] INFO (RecoveryManager.java [recover]:109) - Scanning journal...
> 2012-09-24 09:44:55,721 [WrapperListener_start_runner] ERROR (BrokerPool.java [configure]:242) - Unable to initialialize database instance 'exist': -20984
> java.lang.ArrayIndexOutOfBoundsException: -20984
> at org.exist.util.ByteConversion.byteToShort(ByteConversion.java:60)
> at org.exist.storage.btree.BTree$BTreeNode.read(BTree.java:1024)
> at org.exist.storage.btree.BTree$BTreeNode.access$1400(BTree.java:736)
> at org.exist.storage.btree.BTree.getBTreeNode(BTree.java:422)
> at org.exist.storage.btree.BTree.access$2500(BTree.java:110)
> at org.exist.storage.btree.BTree$BTreeNode.getChildNode(BTree.java:1127)
> at org.exist.storage.btree.BTree$BTreeNode.findValue(BTree.java:1456)
> at org.exist.storage.btree.BTree$BTreeNode.findValue(BTree.java:1466)
> at org.exist.storage.btree.BTree$BTreeNode.access$200(BTree.java:736)
> at org.exist.storage.btree.BTree.findValue(BTree.java:291)
> at org.exist.storage.index.BFile.getAsStream(BFile.java:460)
> at org.exist.storage.NativeBroker.openCollection(NativeBroker.java:690)
> at org.exist.storage.NativeBroker.openCollection(NativeBroker.java:666)
> at org.exist.storage.NativeBroker.getCollection(NativeBroker.java:662)
> at org.exist.storage.BrokerPool.initialize(BrokerPool.java:803)
> at org.exist.storage.BrokerPool.<init>(BrokerPool.java:661)
> at org.exist.storage.BrokerPool.configure(BrokerPool.java:220)
> at org.exist.storage.BrokerPool.configure(BrokerPool.java:192)
> at org.exist.JettyStart.run(JettyStart.java:136)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.exist.wrapper.Main.start(Unknown Source)
> at org.tanukisoftware.wrapper.WrapperManager$12.run(WrapperManager.java:2788)
> 
> We have restarted the server, but no success.
> We have followed the following procedure:
> 1. Stop the running eXist database instance.
> 2. Change into directory EXIST_HOME/webapp/WEB-INF/data or another
> directory you specified as data directory in the configuration
> (conf.xml).
> 3. Delete all .log, .lck, .lock, and .dbx files *except*
> collections.dbx,
> dom.dbx, and symbols.dbx. Be sure to look in the lucene directory for
> any .lock files, and delete them too.
> 4. Start eXist again. Assuming that cleared the block, you're all set
> - just trigger a reindex to rebuild your indexes.
> Also no success.
> 
> Please what else can we do?
> 
> Best regards,
> 
> Sander Schouten | Logica, now part of CGI | Experienced software engineer
> Prof. W.H. Keesomlaan 14, 1183 DJ Amstelveen | Nederland
> T: +31 (0) 88 5646929 | M: 00 31 6 83 178 819 
> san...@lo... (mailto:san...@lo...) | www.cgi.com (http://www.cgi.com/) | www.logica.com (http://www.logica.com/) 
> 
> 
> Think green - keep it on the screen. This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.