Menu
▾
▴
Re: [Exist-open] eXist database collection corrupt / inconsistent
|
From: Withanage, D. <wit...@as...> - 2012-07-28 09:06:01
|
Hi Casey,
* How many documents/collections does it run over?
APPRX. 13000 (altogether) and about 20 pdf files for content extraction.
* Is the code Java or XQuery?
I ran a xquery script which sets acl right to this collection.
* Did you see a relationship between the files which this code was run on and the effected files?
Running the script and accesing this collection using a active directory account is the only thing, we did.
* Are there any other operations which access a large amount of files?
Other than backup setting (3 times a day).
Each 10 minutes i send a http request to check if the server is available throuth an external tool.
Regards,
Dulip
---------------------------------------
Dulip Withanage
Senior Software developer.
Karl Jaspers Center
University of Heidelberg
Germany
________________________________________
From: Casey Jordan [cas...@jo...]
Sent: Friday, July 27, 2012 5:12 PM
To: Jens Østergaard Petersen
Cc: Withanage, Dulip; exi...@li...
Subject: Re: [Exist-open] eXist database collection corrupt / inconsistent
Ok, we have some idea's as to why this might be happening. Can you please describe the process you run on your files in more detail?
For instance:
* How many documents/collections does it run over?
* Is the code Java or XQuery?
* Did you see a relationship between the files which this code was run on and the effected files?
* Are there any other operations which access a large amount of files?
If you can share the code or parts of the code that would be very helpful.
Cheers,
Casey
On Thu, Jul 26, 2012 at 5:41 PM, Jens Østergaard Petersen <oe...@gm...<mailto:oe...@gm...>> wrote:
Hi Casey,
I have looked a little into a recent backup (Dulip and I work at the same place).
I know of no crashes with the present installation, but we have a __lost_and_found__ in the backup with a lot a of empty __contents__.xml, as well as 72 empty records that come from (have same names as files from) various collections (most from collection B mentioned below). All our files are named with UUIDs, therefore we can track them even if they don't stay put.
The 40 empty (zero bytes) records that were placed wrongly in collection A all came from collection B. (Collection B has more than 10,000 records, if that is relevant.) There are no other collections with empty records in their backup.
We noticed the 40 empty records when trying to import collection A - empty records are of course not well-formed, so the Java client protested ("premature end of file", I think it was). We wanted to import the latest backup since we were not able to access the collection using the admin client or oXygen. I assume that this was because it contained empty records. The collection responded well to searches, but we did not test all 900+ records, so probably the 40 missing ones were - missing.
All files with empty counterparts had ACEs - except one, which was in a collection the parent of which had an ACE, if that is relevant.
It is not like 40 extra (empty) records were aded to collection A. 40 records were missing from collection A, as I can see from comparing backups. The missing records do not have the same names as the empty records - as I mentioned above, the files are named with UUIDs - and we have not run out of them yet!
None of the 40 records, missing or extra, were listed in the __contents__.xml file in the backup of collection A. It contained only <resource> entries for the full extant files.
So 40 records disappeared from collection A and 40 records were copied from collection B into A, but emptied of contents. Does this make sense? Why the "neatness" of 40/40?
Cheers - all the same,
Jens
On Jul 26, 2012, at 7:55 PM, Casey Jordan wrote:
Dulip,
Can you describe the type of operations which were occurring around the time you noticed this?
Log output, output from consistency check or errors you get when trying to start the database would help.
Likely one of the core team will need to give you further instruction, however we have had similar issues lately so it would be useful to correlate.
Thanks,
Casey
On Thu, Jul 26, 2012 at 1:52 PM, Withanage, Dulip <wit...@as...<mailto:wit...@as...>> wrote:
Hi Casey,
I have
java -Xms2128m -Xmx5002m
Revision: 16852
I have one more thing that perhaps will be helpful to find the cause.
Some of the files were moved to a different collection, leaving a zero byte file. I can assure that I did not try to manually move them.
In the newer location , the dfiles are present.
I can make you some files available , if you like to have a thorough look.
Best,
Dulip
---------------------------------------
Dulip Withanage
Senior Software developer.
Karl Jaspers Center
University of Heidelberg
Germany
________________________________________
From: Casey Jordan [cas...@jo...<mailto:cas...@jo...>]
Sent: Thursday, July 26, 2012 5:53 PM
To: Withanage, Dulip
Cc: exi...@li...<mailto:exi...@li...>
Subject: Re: [Exist-open] eXist database collection corrupt / inconsistent
Can you please provide eXist version, as well as what memory or configuration settings you are using that are different from the default.
Thanks,
Casey
On Thu, Jul 26, 2012 at 5:20 AM, Dulip Withanage <wit...@as...<mailto:wit...@as...><mailto:wit...@as...<mailto:wit...@as...>>> wrote:
Hi all ,
My eXist database has a collection, which is not accessible with java
client , web interface or external programs like oxygen.
Although I can access its files using, xquery commands.
This happened to me a long time ago and I got the impression that the
JVM did not have enough memory for this to occur. Although now it has
been assured that the JVM is set enough, this has happened again. I
see nothing in the logs.
Can somebody help give me tip, where and how to debug this problem?
Dulip
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Exist-open mailing list
Exi...@li...<mailto:Exi...@li...><mailto:Exi...@li...<mailto:Exi...@li...>>
https://lists.sourceforge.net/lists/listinfo/exist-open
--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399<tel:%28585%29%20348%207399>
easydita.com<http://easydita.com/><http://easydita.com<http://easydita.com/>>
This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law. If you are not the intended recipient,
please be advised that any disclosure copying, distribution, or use of
the information contained herein is prohibited. If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.
--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399<tel:%28585%29%20348%207399>
easydita.com<http://easydita.com/>
This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law. If you are not the intended recipient,
please be advised that any disclosure copying, distribution, or use of
the information contained herein is prohibited. If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/_______________________________________________
Exist-open mailing list
Exi...@li...<mailto:Exi...@li...>
https://lists.sourceforge.net/lists/listinfo/exist-open
--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com<http://easydita.com>
This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law. If you are not the intended recipient,
please be advised that any disclosure copying, distribution, or use of
the information contained herein is prohibited. If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.
|