[Exist-open] Index corruption under heavy load

[Willem v. d. W. <wav...@gm...>]

Hi,

We run eXist Version   1.4.2dev Build:     20111124 on a linux CENTOS boxes.

Two instances of the same application, that have been experiencing heavy 
load. One particular resource is used as a listing of scheduled entries, 
which is queried each minute from rest based service called from a cron 
job. This would run a query to find any tasks that are due, and triggers 
a task based on the entry. The same file is updated on a regular basis.

The structure of the data that is queried looks as follows:
<Project Name="172049">
<KwantuObject Id="172049-CWP" Domain="132140" ApplicationId="CWP">
<Job Seq="1" name="spMonthlyProgress" Period="true">
<CreatedDateTime>2012-04-01</CreatedDateTime>
<StartedDateTime>2012-04-23+02:00</StartedDateTime>
<ExecutionTime>2012-04-23+02:00</ExecutionTime>
</Job>
</KwantuObject>
</Project>

It works without problem, but we have noticed lately that under heavy 
load, the file gets corrupted. It is the only file in the application 
that is shared by all projects(tasks) on the system, resulting in reads 
and updates happening all the time.

Currently, the project would insert the scheduled <Job ..../>, and when 
it runs this job is removed from this file and inserted into a completed 
jobs file.

Our problem is that recently, that file tends to get corrupted, and 
queries on it starts failing. None of the rest of the application or 
database is affected. When we then backup the collection in which the 
file resides, rename the file, and restore the collection from the 
backup it works again fine.

Is the answer to this that we should rather have a separate xml file for 
each project. This will result in about 15,000 files in the collection. 
Would that improve stability and performance? Or would it be best not to 
remove the <Job..../> nodes, and just add a flag that identifies what 
has been completed?

We cannot really replicate the actual corruption, but is there some 
information that we can give to help identify why this would happen?

Willem