Re: [Exist-open] eXist-db repair and reindex process taking a very long time

[Pieter L. <pie...@be...>]

Hi Vincent,

When I want to skip the re-indexing I stop, remove journal.lck and *.log 
from the exist-db/data folder and restart.
If the database was taken down during an update then this will lead to 
corruption. If no write was taking place, then usually (to my 
experience) this is harmless.
Please note that my exist-db/data folder is not the exist resource data 
folder (which we put somewhere else) but the journaling folder. There 
are usually only two files in that one.

good luck! If I end up needing to re-index on our 20GB data set it 
usually takes an hour and a half. But I might be using lighter indexes 
than you, so not sure if this is comparable.

Best,
Pieter


On 10/09/2024 21:10, Lizzi, Vincent wrote:
>
> Hello eXist-db community,
>
> By comparing the folders data\lucene and data\range to a previous 
> backup, going by the number of files and total size of these folders, 
> it looks like the reindexing process is about 50% done. The reindexing 
> process has been running since August 13 and is using over 100 Gb of 
> memory.
>
> Is there any way to start eXist-db and allow it to go through its 
> recovery process but stop it from reindexing database files?  I’m 
> wondering if that could be a way to get the database operational 
> again, and then I could manually run xmldb:reindex().
>
> Thanks,
>
> Vincent
>
> _____________________________________________
>
> *Vincent M. Lizzi*
>
> Head of Information Standards | Taylor & Francis Group
>
> vin...@ta... 
> <mailto:vin...@ta...>
>
>
> Information Classification: General
>
> *From:*Lizzi, Vincent
> *Sent:* Tuesday, September 3, 2024 11:36 AM
> *To:* Exist-open <exi...@li...>
> *Subject:* eXist-db repair and reindex process taking a very long time
>
> Hello eXist-db community,
>
> I’ve been monitoring an eXist-db database that is going through its 
> automated recovery process, and am wondering if there is any way to 
> get more information about its progress and how soon the process will 
> finish.
>
> This eXist-db has full text indexing and range indexes configured on 
> several large collections. The EC2 server on which eXist-db is hosted 
> had an outage. When eXist-db was restarted its automatic repair 
> process began. That was about 3 weeks ago. Through Windows Resource 
> Monitor I can see that the eXist-db process is reading from dob.dbx 
> and structure.dbx and writing to structure.dbx and writing to files in 
> the “lucene” and “range” folders, and the process is using about 6% of 
> CPU consistently, and memory usage has increased gradually to over 100 
> GB. The last line in exist.log is still:
>
> 2024-08-13 17:02:59,710 [main] INFO (NativeBroker.java [repair]:3692) 
> - Reindexing database files ...
>
> Is there any way to find out more what the process is doing, estimate 
> when it will finish, or release any bottlenecks, without interrupting 
> the process?
>
> Thanks,
>
> Vincent
>
> ______________________________________________
>
> *Vincent M. Lizzi*
>
> Head of Information Standards | Taylor & Francis Group
>
> 530 Walnut St., Suite 850, Philadelphia, PA 19106
>
> E-Mail: vin...@ta... 
> <mailto:vin...@ta...>
>
> Web: www.tandfonline.com <http://www.tandfonline.com>
>
> Taylor & Francis is a trading name of Informa UK Limited,
>
> registered in England under no. 1072954
>
> "Everything should be made as simple as possible, but not simpler."
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

-- 
Pieter Lamers
John Benjamins Publishing Company
Postal Address: P.O. Box 36224, 1020 ME AMSTERDAM, The Netherlands
Visiting Address: Klaprozenweg 75G, 1033 NN AMSTERDAM, The Netherlands
Warehouse: Kelvinstraat 11-13, 1446 TK PURMEREND, The Netherlands
tel: +31 20 630 4747
web:www.benjamins.com