[Exist-open] 2.2-:develop-15c5a27-20141116 suddenly keeps shutting down database

[Brad C. <bk...@mu...>]

Hi,

We've been running eXist 2.2-:develop-15c5a27-20141116 for 2 couple of 
years on Ubuntu 14.04 on Java 1.7.0_121 64Bit without problems.

This weekend I edited conf.xml and adjusted the consistency-check 
settings to change when full and incrementals are done.

I restarted eXist and went home thinking all was well. but now I find 
this morning that it's not actually working at all.

It seems to be putting the database into readonly mode but I cannot find 
out why.

after restart, I try editing a file via webdav and see this:

2017-04-03 08:01:43,878 [eXistThread-35] FATAL (BrokerPool.java 
[isReadOnly]:1334) - Partition containing DATA_DIR: 
/opt/eXist-2.2/webapp/WEB-INF/data/dbx_dir.lck is running out of disk 
space. Switching eXist-db to read only to prevent data loss!
2017-04-03 08:01:43,879 [eXistThread-35] INFO (BrokerPool.java 
[setReadOnly]:1343) - Switching to read-only mode!!!
2017-04-03 08:01:43,879 [eXistThread-35] ERROR (ExistCollection.java 
[createFile]:447) - org.exist.security.PermissionDeniedException: 
Database is read-only
2017-04-03 08:01:43,880 [eXistThread-35] WARN (TransactionManager.java 
[execute]:216) - Failed to write abort record to journal: Database is 
shut down.
2017-04-03 08:01:43,881 [eXistThread-35] WARN (Journal.java 
[flushToLog]:290) - Failed to trigger checkpoint!

But we have plenty of space

/dev/mapper/exist1--vg-opt          32G   13G   18G  43% /opt

and plentyof inodes

/dev/mapper/exist1--vg-opt        2097152 283992 1813160 14% /opt


I temporarily changed conf.xml minDiskSpace to -1

Now I get a different error

2017-04-03 08:15:31,246 [eXistThread-35] INFO  (HttpManager.java 
[process]:213) - PROPFIND :: 
http://localhost:8680/exist/webdav/db/orgs/sfiusa/test.xml - 
http://localhost:8680/exist/webdav/db/orgs/sfiusa/test.xml
2017-04-03 08:15:31,248 [eXistThread-35] INFO (MiltonResource.java 
[authorise]:350) - PROPFIND /db/orgs/sfiusa/test.xml (write=false)
2017-04-03 08:15:31,250 [eXistThread-35] INFO (MiltonResource.java 
[authorise]:350) - PROPFIND /db/orgs/sfiusa/test.xml (write=false)
2017-04-03 08:15:31,252 [eXistThread-35] INFO (HttpManager.java 
[process]:213) - UNLOCK :: 
http://localhost:8680/exist/webdav/db/orgs/sfiusa/test.xml - 
http://localhost:8680/exist/webdav/db/orgs/sfiusa/test.xml
2017-04-03 08:15:31,253 [eXistThread-35] INFO (MiltonResource.java 
[authorise]:350) - UNLOCK /db/orgs/sfiusa/test.xml (write=true)
2017-04-03 08:15:31,253 [eXistThread-35] WARN (TransactionManager.java 
[execute]:164) - Failed to create transaction. Error writing to log file.

I have rebooted the server and it's not coming up with fsck errors, so I 
am pretty confused why it's failing to write to disk.

The java process runs as root, most files are owned by a regular user.. 
I don't recall a 'run as' userid setting anywhere.



It does seem to be actually writing to disk, given the dates the data 
files were touched.

-rw-rw-r--  1 bkc  bkc  3771809792 Apr  3 08:17 dom.dbx
drwxrwxr-x  3 bkc  bkc       12288 Apr  3 08:17 range/
drwxrwxr-x  3 bkc  bkc        4096 Apr  3 08:17 lucene/
-rw-rw-r--  1 bkc  bkc    43859968 Apr  3 08:17 collections.dbx
-rw-rw-r--  1 bkc  bkc  4906831872 Apr  3 08:15 structure.dbx
-rw-rw-r--  1 bkc  bkc       36401 Apr  3 08:15 symbols.dbx
drwxrwxr-x  8 bkc  bkc        4096 Apr  3 08:15 ./
-rw-r--r--  1 root root        113 Apr  3 08:15 000000073f.log
drwxrwxr-x  2 bkc  bkc      172032 Mar 31 16:18 fs.journal/
drwxr-xr-x  3 bkc  bkc        4096 Jan  5 20:16 export/
-rw-r--r--  1 bkc  bkc           0 Jan 13  2016 counters
-rw-r--r--  1 bkc  bkc         140 Jan 30  2015 jmxservlet.token
drwxrwxr-x 15 bkc  bkc        4096 Jan 29  2015 expathrepo/
-rw-rw-r--  1 bkc  bkc       12288 Jan 29  2015 values.dbx
-rw-rw-r--  1 bkc  bkc      647168 Jan 29  2015 ngram.dbx
-rw-rw-r--  1 bkc  bkc        1085 Jan 29  2015 restxq.registry
drwxrwxr-x  3 bkc  bkc        4096 Jan 29  2015 fs/
-rw-rw-r--  1 bkc  bkc        8192 Jan 29  2015 sort.dbx
drwxrwxr-x  7 bkc  bkc        4096 Jan 29  2015 ../
-rw-rw-r--  1 bkc  bkc           0 Jul  1  2014 .DO_NOT_DELETE

Does anyone have any suggestions?

Thanks



-- 
Brad Clements,                bk...@mu...    (315)268-1000
Jabber/XMPP: bkc...@gm...