You can subscribe to this list here.
2001 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(31) |
Nov
(25) |
Dec
(33) |
---|---|---|---|---|---|---|---|---|---|---|---|---|
2002 |
Jan
(48) |
Feb
(62) |
Mar
(22) |
Apr
(29) |
May
(9) |
Jun
(45) |
Jul
(28) |
Aug
(41) |
Sep
(60) |
Oct
(96) |
Nov
(99) |
Dec
(70) |
2003 |
Jan
(98) |
Feb
(159) |
Mar
(164) |
Apr
(150) |
May
(143) |
Jun
(97) |
Jul
(184) |
Aug
(143) |
Sep
(207) |
Oct
(126) |
Nov
(159) |
Dec
(165) |
2004 |
Jan
(131) |
Feb
(229) |
Mar
(220) |
Apr
(212) |
May
(320) |
Jun
(223) |
Jul
(191) |
Aug
(390) |
Sep
(261) |
Oct
(229) |
Nov
(215) |
Dec
(184) |
2005 |
Jan
(221) |
Feb
(312) |
Mar
(336) |
Apr
(273) |
May
(359) |
Jun
(277) |
Jul
(303) |
Aug
(321) |
Sep
(256) |
Oct
(415) |
Nov
(428) |
Dec
(508) |
2006 |
Jan
(585) |
Feb
(419) |
Mar
(496) |
Apr
(296) |
May
(403) |
Jun
(404) |
Jul
(553) |
Aug
(296) |
Sep
(252) |
Oct
(416) |
Nov
(414) |
Dec
(245) |
2007 |
Jan
(354) |
Feb
(422) |
Mar
(389) |
Apr
(298) |
May
(397) |
Jun
(318) |
Jul
(315) |
Aug
(339) |
Sep
(253) |
Oct
(317) |
Nov
(350) |
Dec
(264) |
2008 |
Jan
(353) |
Feb
(313) |
Mar
(433) |
Apr
(383) |
May
(343) |
Jun
(355) |
Jul
(321) |
Aug
(338) |
Sep
(242) |
Oct
(206) |
Nov
(199) |
Dec
(279) |
2009 |
Jan
(327) |
Feb
(221) |
Mar
(280) |
Apr
(278) |
May
(237) |
Jun
(345) |
Jul
(322) |
Aug
(324) |
Sep
(676) |
Oct
(586) |
Nov
(735) |
Dec
(329) |
2010 |
Jan
(619) |
Feb
(424) |
Mar
(529) |
Apr
(241) |
May
(312) |
Jun
(554) |
Jul
(698) |
Aug
(576) |
Sep
(408) |
Oct
(268) |
Nov
(391) |
Dec
(426) |
2011 |
Jan
(629) |
Feb
(512) |
Mar
(465) |
Apr
(467) |
May
(475) |
Jun
(403) |
Jul
(426) |
Aug
(542) |
Sep
(418) |
Oct
(620) |
Nov
(614) |
Dec
(358) |
2012 |
Jan
(357) |
Feb
(466) |
Mar
(344) |
Apr
(215) |
May
(408) |
Jun
(375) |
Jul
(241) |
Aug
(260) |
Sep
(401) |
Oct
(461) |
Nov
(498) |
Dec
(294) |
2013 |
Jan
(453) |
Feb
(447) |
Mar
(434) |
Apr
(326) |
May
(295) |
Jun
(471) |
Jul
(463) |
Aug
(278) |
Sep
(525) |
Oct
(343) |
Nov
(389) |
Dec
(405) |
2014 |
Jan
(564) |
Feb
(324) |
Mar
(319) |
Apr
(319) |
May
(384) |
Jun
(259) |
Jul
(210) |
Aug
(219) |
Sep
(315) |
Oct
(478) |
Nov
(207) |
Dec
(316) |
2015 |
Jan
(222) |
Feb
(234) |
Mar
(201) |
Apr
(145) |
May
(367) |
Jun
(318) |
Jul
(195) |
Aug
(210) |
Sep
(234) |
Oct
(248) |
Nov
(217) |
Dec
(189) |
2016 |
Jan
(219) |
Feb
(177) |
Mar
(110) |
Apr
(91) |
May
(159) |
Jun
(124) |
Jul
(192) |
Aug
(119) |
Sep
(125) |
Oct
(64) |
Nov
(80) |
Dec
(68) |
2017 |
Jan
(156) |
Feb
(312) |
Mar
(386) |
Apr
(217) |
May
(89) |
Jun
(115) |
Jul
(79) |
Aug
(122) |
Sep
(100) |
Oct
(99) |
Nov
(129) |
Dec
(77) |
2018 |
Jan
(106) |
Feb
(78) |
Mar
(160) |
Apr
(73) |
May
(110) |
Jun
(160) |
Jul
(93) |
Aug
(92) |
Sep
(75) |
Oct
(147) |
Nov
(114) |
Dec
(97) |
2019 |
Jan
(141) |
Feb
(78) |
Mar
(158) |
Apr
(60) |
May
(123) |
Jun
(54) |
Jul
(44) |
Aug
(147) |
Sep
(117) |
Oct
(54) |
Nov
(74) |
Dec
(96) |
2020 |
Jan
(113) |
Feb
(125) |
Mar
(142) |
Apr
(57) |
May
(71) |
Jun
(99) |
Jul
(58) |
Aug
(81) |
Sep
(49) |
Oct
(50) |
Nov
(63) |
Dec
(37) |
2021 |
Jan
(37) |
Feb
(45) |
Mar
(39) |
Apr
(18) |
May
(14) |
Jun
(9) |
Jul
(44) |
Aug
(23) |
Sep
(13) |
Oct
(31) |
Nov
(13) |
Dec
(33) |
2022 |
Jan
(17) |
Feb
(8) |
Mar
(32) |
Apr
(7) |
May
(17) |
Jun
(7) |
Jul
(36) |
Aug
(29) |
Sep
(9) |
Oct
(20) |
Nov
(10) |
Dec
(1) |
2023 |
Jan
(30) |
Feb
(37) |
Mar
(23) |
Apr
(1) |
May
(14) |
Jun
(5) |
Jul
(3) |
Aug
(6) |
Sep
(5) |
Oct
(48) |
Nov
(4) |
Dec
(29) |
2024 |
Jan
(1) |
Feb
|
Mar
(21) |
Apr
(6) |
May
(16) |
Jun
(41) |
Jul
(11) |
Aug
(17) |
Sep
(16) |
Oct
(11) |
Nov
(3) |
Dec
(9) |
2025 |
Jan
(7) |
Feb
(7) |
Mar
(6) |
Apr
(6) |
May
(30) |
Jun
(8) |
Jul
(10) |
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
From: Michael W. <wes...@ja...> - 2025-07-25 00:36:20
|
Hi Tony, Okay, the "Failed to open database" message concerns me. under /usr/local/src/exist3-data do you have the following files? -rw-rw-r--. 1 exist exist 1451 Jul 25 08:57 00000000d6.log -rw-rw-r--. 1 exist exist 13991936 Jul 24 22:40 collections.dbx -rw-r--r--. 1 exist exist 1515 Jul 24 22:38 counters -rw-rw-r--. 1 exist exist 16 Jul 25 08:57 dbx_dir.lck -rw-rw-r--. 1 exist exist 6785642496 Jul 24 22:38 dom.dbx drwxrwxr-x. 15 exist exist 4096 Sep 27 2020 expathrepo drwxrwxr-x. 3 exist exist 132 May 20 12:14 export drwxrwxr-x. 3 exist exist 16 Apr 3 2018 fs drwxrwxr-x. 2 exist exist 6 Jul 21 20:47 fs.journal -rw-rw-r--. 1 exist exist 140 Apr 16 2018 jmxservlet.token -rw-rw-r--. 1 exist exist 16 Jul 25 08:57 journal.lck drwxrwxr-x. 2 exist exist 103 Jul 23 2022 lucene -rw-rw-r--. 1 exist exist 4636672 May 22 21:46 ngram.dbx drwxrwxr-x. 2 exist exist 8192 Jul 25 08:44 range -rw-rw-r--. 1 exist exist 120 Apr 9 2018 README -rw-rw-r--. 1 exist exist 257 Apr 9 2018 RECOVERY -rw-------. 1 exist exist 11 Mar 8 10:20 restxq.registry drwxrwxr-x. 2 exist exist 42 Jul 23 2022 sanity -rw-rw-r--. 1 exist exist 8192 Jul 23 2022 sort.dbx -rw-rw-r--. 1 exist exist 6046371840 Jul 24 22:40 structure.dbx -rw-rw-r--. 1 exist exist 63342 Jul 21 20:35 symbols.dbx -rw-rw-r--. 1 exist exist 46583808 Jul 24 22:40 values.dbx The dom.dbx file should be very large. That's where all of your data is at. If this file is broken, then the data isn't recoverable. The RECOVERY file states: RECOVERY ======== If the index got corrupted, this is how to recover: - stop database - remove all ".log" transaction log files - remove all ".lck" lock files - remove all ".dbx" files *EXCEPT dom.dbx, collections.dbx and symbols.dbx* - start database I really don't like doing it without a backup. So first, make a tarball of /usr/local/src/exist3-data as a backup of the raw data then start the recovery process above. And once the database is up and running again, run: cd /usr/local/src/exist3 bin/backup.sh --dir ~/tmp/2025-07-25-backup --backup /db And this is my script modified for your setup that I run daily via crontab: #!/bin/sh unset JAVA_OPTS export JAVA_HOME=/usr export EXIST_HOME=/usr/local/src/exist3 export TARGET=/tmp export DATE=`date +"%Y-%m-%d"` export PATH=$JAVA_HOME/bin:/usr/local/bin:/usr/bin:/bin cd $EXIST_HOME bin/backup.sh --backup /db --dir $TARGET/full-$DATE -u "admin" -p "password" I thought it was supposed to get the username and password from the $EXIST_HOME/backup.properties file, but when I tested it by hand just now without the -u/-p parameters it failed. Not sure why. Anyway, daily backups going forward would be a very good idea. Take care. 2025年7月24日(木) 23:21 Tony Graham <tg...@an...>: > Michael, > > Thanks for following through with this. > > On 23/07/2025 09:00, Michael Westbay wrote: > > Okay. Let's first find EXIST_HOME by finding where the conf.xml file is. > > > > find /usr/local -name conf.xml > > It found multiple, but the ones of interest are: > > /usr/local/src/exist3/installer/conf.xml > /usr/local/src/exist3/conf.xml > > > If /usr/local is EXIST_HOME, then run.sh will be in /usr/local/bin along > > with all of the other eXist executables. > > The only things in '/usr/local/bin' are 'tracd' and 'trac-admin' for use > with Trac. (Still the best issue tracking/wiki system.) > > > This conf.xml file will then specify where the data directory is: > > '/usr/local/src/exist3/conf.xml' has: > > <db-connection cacheSize="256M" checkMaxCacheSize="true" > collectionCache="64M" database="native" > files="/usr/local/src/exist3-data" pageSize="4096" > nodesBuffer="1000" cacheShrinkThreshold="10000" > doc-ids="default" minDiskSpace="128M"> > > > You could also search for run.sh the same way: > > > > find /usr/local -name run.sh > > As before, I don't have a 'run.sh' (unless you count 'run.sh' for > Antenna House Formatter). > > Running the Java installer as a non-root user has never given me a > 'run.sh'. > > I downloaded the 'run.sh' [1] for that eXist-db version from GitHub, > edited it to use the same Java alternative as used for the database, ran > it, and got: > > ---- > cd "/usr/local/src/exist3/" ; ./bin/run.sh org.exist.backup.ExportMain > -x -d /mnt/tmp > ERROR: Failed to open database: > org.exist.storage.BrokerPoolServiceException: org.exist.EXistException: > java.io.IOException: Collection /db/system/security/exist/accounts > cannot be created. > ---- > > If this can be solved, the problem then becomes that we don't know how > many other collections will have similar problems, so I'm not hopeful. > > Regards, > > > Tony Graham. > -- > Senior Architect > XML Division > Antenna House, Inc. > ---- > Skerries, Ireland > tg...@an... > > [1] > > https://raw.githubusercontent.com/eXist-db/exist/refs/tags/eXist-3.0.RC2/bin/run.sh > > > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open > -- Michael Westbay Writer/System Administrator http://www.japanesebaseball.com/ |
From: Tony G. <tg...@an...> - 2025-07-23 12:02:42
|
Michael, Thanks for following through with this. On 23/07/2025 09:00, Michael Westbay wrote: > Okay. Let's first find EXIST_HOME by finding where the conf.xml file is. > > find /usr/local -name conf.xml It found multiple, but the ones of interest are: /usr/local/src/exist3/installer/conf.xml /usr/local/src/exist3/conf.xml > If /usr/local is EXIST_HOME, then run.sh will be in /usr/local/bin along > with all of the other eXist executables. The only things in '/usr/local/bin' are 'tracd' and 'trac-admin' for use with Trac. (Still the best issue tracking/wiki system.) > This conf.xml file will then specify where the data directory is: '/usr/local/src/exist3/conf.xml' has: <db-connection cacheSize="256M" checkMaxCacheSize="true" collectionCache="64M" database="native" files="/usr/local/src/exist3-data" pageSize="4096" nodesBuffer="1000" cacheShrinkThreshold="10000" doc-ids="default" minDiskSpace="128M"> > You could also search for run.sh the same way: > > find /usr/local -name run.sh As before, I don't have a 'run.sh' (unless you count 'run.sh' for Antenna House Formatter). Running the Java installer as a non-root user has never given me a 'run.sh'. I downloaded the 'run.sh' [1] for that eXist-db version from GitHub, edited it to use the same Java alternative as used for the database, ran it, and got: ---- cd "/usr/local/src/exist3/" ; ./bin/run.sh org.exist.backup.ExportMain -x -d /mnt/tmp ERROR: Failed to open database: org.exist.storage.BrokerPoolServiceException: org.exist.EXistException: java.io.IOException: Collection /db/system/security/exist/accounts cannot be created. ---- If this can be solved, the problem then becomes that we don't know how many other collections will have similar problems, so I'm not hopeful. Regards, Tony Graham. -- Senior Architect XML Division Antenna House, Inc. ---- Skerries, Ireland tg...@an... [1] https://raw.githubusercontent.com/eXist-db/exist/refs/tags/eXist-3.0.RC2/bin/run.sh |
From: Michael W. <wes...@ja...> - 2025-07-23 08:01:23
|
Hi Tony, Okay. Let's first find EXIST_HOME by finding where the conf.xml file is. find /usr/local -name conf.xml If /usr/local is EXIST_HOME, then run.sh will be in /usr/local/bin along with all of the other eXist executables. This conf.xml file will then specify where the data directory is: <db-connection cacheSize="256M" checkMaxCacheSize="true" collectionCache="64M" database="native" files="*webapp/WEB-INF/data*" pageSize="4096" nodesBuffer="1000" cacheShrinkThreshold="10000" doc-ids="default" minDiskSpace="128M"> [Emphasis added] You could also search for run.sh the same way: find /usr/local -name run.sh Once we know where the executables are, let's go from there. Take care. > On Linux and installed from the installer jar. I don't have 'run.sh', -- Michael Westbay Writer/System Administrator http://www.japanesebaseball.com/ |
From: Tony G. <tg...@an...> - 2025-07-22 14:23:56
|
Michael, On 22/07/2025 12:25, Michael Westbay wrote: > Thanks, but still no luck. I don't have a 'run.sh'. [1] > > > That's odd. I have eXist 3.x and the EXIST_HOME/bin directory has the > following: > > backup.bat batch.d client.sh eXist_32x32.ico run.bat server.bat > shutdown.bat startup.bat > backup.sh client.bat deprecated functions.d *run.sh* server.sh > shutdown.sh startup.sh On Linux and installed from the installer jar. I don't have 'run.sh', nor do I have any '.bat' files. Similarly with exist6 installed from installer jar. > These shell scripts make sure that all of the CLASS_PATH stuff gets > setup properly for running the utilities like > `org.exist.backup.ExportMain`. I don't doubt it, but I don't have it. The next thing to try will be to use a different installer to see if that comes with 'run.sh'. ... > I got multiple messages of the form: > > INFO: None of content dirs exist: > '/usr/local/src/exist3-data/expathrepo/doc-0.4.7/content' and > '/usr/local/src/exist3-data/expathrepo/doc-0.4.7/doc' > > Are you trying to execute from the build directory? Did you not run the > installer to place it all in a > user's directory? Are you running it from a container and pointing to a > data directory on the host > machine? I have '/usr/local/src/exist3-data/expathrepo/doc-0.4.7/' but not the subdirectories. That directory was last modified in 2022, so the absent directories are not a by-product of my current problem after all. > then I got: > > 17:09:17.550 [main] ERROR org.exist.storage.DBBroker - no data found at > tid 25; page 60995 > java.io.IOException: no data found at tid 25; page 60995 > at > org.exist.storage.index.BFile.getAsStream(BFile.java:492) ~[exist.jar:?] > at > org.exist.storage.index.BFile.getAsStream(BFile.java:462) ~[exist.jar:?] > at > org.exist.storage.NativeBroker.openCollection(NativeBroker.java:928) > [exist.jar:?] > ... > 17:09:17.554 [main] ERROR org.exist.storage.DBBroker - Collection > '/db/system/security/exist/accounts' found in subCollections set but is > missing from collections.dbx! > 17:09:17.569 [main] ERROR org.exist.storage.BrokerPools - Unable to > initialize database instance 'exist': > org.exist.storage.BrokerPoolServiceException: org.exist.EXistException: > java.io.IOException: Collection /db/system/security/exist/accounts > cannot be created. > > > Before doing anything -- like bringing the eXist server back up, you > want to make an emergency > backup. Be sure to make the target a local directory that exists and > that you have write permission > for. I use "~/backup/date-reason" (where "date" is the date that it shut > down and "reason" is "locked" > or "out-of-memory"). The "backup" directory exists in my exist user's > home directory. I don't have an exist user. The stack trace is from running 'org.exist.backup.ExportMain', not from trying to bring the server back up. > /usr/local/* is not usually writable by ordinary users. I am the only user, so both ordinary and extraordinary (and still 'sudo' to do things as 'root'). Putting things in '/usr/local/src' is a habit that I picked up over thirty years ago with multiple people working on projects on a SunOS machine. Still seems wrong to put projects under my home directory even when I am the only user. > Let's start by locating where the EXIST_HOME/bin directory is. I installed in '/usr/local/src/exist3', with data in '/usr/local/src/ exist3-data'. Regards, Tony Graham. -- Senior Architect XML Division Antenna House, Inc. ---- Skerries, Ireland tg...@an... |
From: Michael W. <wes...@ja...> - 2025-07-22 11:26:29
|
Hi Tony, Thanks, but still no luck. I don't have a 'run.sh'. [1] That's odd. I have eXist 3.x and the EXIST_HOME/bin directory has the following: backup.bat batch.d client.sh eXist_32x32.ico run.bat server.bat shutdown.bat startup.bat backup.sh client.bat deprecated functions.d *run.sh* server.sh shutdown.sh startup.sh These shell scripts make sure that all of the CLASS_PATH stuff gets setup properly for running the utilities like `org.exist.backup.ExportMain`. If you're on Windows, you would use `run.bat`. > After I made a > script that no longer bombed out from missing dependencies, I ran > 'org.exist.backup.ExportMain'. > > I got multiple messages of the form: > > INFO: None of content dirs exist: > '/usr/local/src/exist3-data/expathrepo/doc-0.4.7/content' and > '/usr/local/src/exist3-data/expathrepo/doc-0.4.7/doc' > Are you trying to execute from the build directory? Did you not run the installer to place it all in a user's directory? Are you running it from a container and pointing to a data directory on the host machine? > then I got: > > 17:09:17.550 [main] ERROR org.exist.storage.DBBroker - no data found at > tid 25; page 60995 > java.io.IOException: no data found at tid 25; page 60995 > at org.exist.storage.index.BFile.getAsStream(BFile.java:492) > ~[exist.jar:?] > at org.exist.storage.index.BFile.getAsStream(BFile.java:462) > ~[exist.jar:?] > at > org.exist.storage.NativeBroker.openCollection(NativeBroker.java:928) > [exist.jar:?] > ... > 17:09:17.554 [main] ERROR org.exist.storage.DBBroker - Collection > '/db/system/security/exist/accounts' found in subCollections set but is > missing from collections.dbx! > 17:09:17.569 [main] ERROR org.exist.storage.BrokerPools - Unable to > initialize database instance 'exist': > org.exist.storage.BrokerPoolServiceException: org.exist.EXistException: > java.io.IOException: Collection /db/system/security/exist/accounts > cannot be created. Before doing anything -- like bringing the eXist server back up, you want to make an emergency backup. Be sure to make the target a local directory that exists and that you have write permission for. I use "~/backup/date-reason" (where "date" is the date that it shut down and "reason" is "locked" or "out-of-memory"). The "backup" directory exists in my exist user's home directory. /usr/local/* is not usually writable by ordinary users. Let's start by locating where the EXIST_HOME/bin directory is. -- Michael Westbay Writer/System Administrator http://www.japanesebaseball.com/ |
From: Tony G. <tg...@an...> - 2025-07-20 22:56:40
|
On 20/07/2025 06:25, Michael Westbay wrote: > At this point, I do a full emergency backup and restore. > > From the EXIST_HOME directory, run: > > bin/run.sh org.exist.backup.ExportMain -x -d path/to/backup/file Thanks, but still no luck. I don't have a 'run.sh'. [1] After I made a script that no longer bombed out from missing dependencies, I ran 'org.exist.backup.ExportMain'. I got multiple messages of the form: INFO: None of content dirs exist: '/usr/local/src/exist3-data/expathrepo/doc-0.4.7/content' and '/usr/local/src/exist3-data/expathrepo/doc-0.4.7/doc' then I got: 17:09:17.550 [main] ERROR org.exist.storage.DBBroker - no data found at tid 25; page 60995 java.io.IOException: no data found at tid 25; page 60995 at org.exist.storage.index.BFile.getAsStream(BFile.java:492) ~[exist.jar:?] at org.exist.storage.index.BFile.getAsStream(BFile.java:462) ~[exist.jar:?] at org.exist.storage.NativeBroker.openCollection(NativeBroker.java:928) [exist.jar:?] ... 17:09:17.554 [main] ERROR org.exist.storage.DBBroker - Collection '/db/system/security/exist/accounts' found in subCollections set but is missing from collections.dbx! 17:09:17.569 [main] ERROR org.exist.storage.BrokerPools - Unable to initialize database instance 'exist': org.exist.storage.BrokerPoolServiceException: org.exist.EXistException: java.io.IOException: Collection /db/system/security/exist/accounts cannot be created. Regards, Tony Graham. -- Senior Architect XML Division Antenna House, Inc. ---- Skerries, Ireland tg...@an... [1] The only reference to it that I found was an email from you to this list in 2021. |
From: Michael W. <wes...@ja...> - 2025-07-20 05:31:26
|
Hi Tony, Oh, and if you rely on any counters, once the new "data" directory has been created, be sure to copy the old "counters" file into it. It's a pain to fix counter IDs after they've been assigned a second time. Take care. 2025年7月20日(日) 14:25 Michael Westbay <wes...@ja...>: > Hi Tony, > > At this point, I do a full emergency backup and restore. > > From the EXIST_HOME directory, run: > > bin/run.sh org.exist.backup.ExportMain -x -d path/to/backup/file > > The target filename is usually the date and what went wrong, like > "2025-07-20-out-of-space." > > After making the backup, mv the EXIST_HOME/webapp/WEB-INF/data directory > elsewhere, then start up eXist. > > Next, extract the emergency backup and cd to db and run: > > ${EXIST_HOME}/bin/backup.sh -r `pwd`/system/security/__contents__.xml -u > admin -P "new_password" > > Then run: > > ${EXIST_HOME}/bin/backup.sh -r `pwd`/__contents__.xml -u admin -p > "new_password" > > The first restore will restore the security collection and set the new > password for admin. > > The second restore will restore the rest of your database using the new > password for admin. > > Finally, I have two scripts that I always run after a restore to restore > the repos that I use and restore permissions to certain collections. > > > xquery version "3.0"; > > <fix-repo>{( > repo:remove("http://exist-db.org/apps/some_app"), > repo:install-and-deploy("http://exist-db.org/apps/shared", "1.2.3", " > http://exist-db.org/exist/apps/public-repo/some_app"), > ... > )}</fix-repo> > > Hope this helps. > > Take care. > > > > 2025年7月20日(日) 11:51 Tony Graham <tg...@an...>: > >> I have an eXist-3 (yes, eXist-3) that won't respond and won't shutdown >> (short of killing the process). Starting it devolves into multiple >> stack traces with the message in the subject. Is there any hope for it? >> >> I got here after killing eXist after a server backup filled the >> filesystem and eXist became unresponsive. (The backup was because a >> Jenkins job running an Ant task had created some ghost collections that >> showed only in the Java client and that the 'admin' user couldn't >> delete, and I was going to tweak then reload the backup.) >> >> Regards, >> >> >> Tony Graham. >> -- >> Senior Architect >> XML Division >> Antenna House, Inc. >> ---- >> Skerries, Ireland >> tg...@an... >> >> >> >> _______________________________________________ >> Exist-open mailing list >> Exi...@li... >> https://lists.sourceforge.net/lists/listinfo/exist-open >> > > > -- > Michael Westbay > Writer/System Administrator > http://www.japanesebaseball.com/ > -- Michael Westbay Writer/System Administrator http://www.japanesebaseball.com/ |
From: Michael W. <wes...@ja...> - 2025-07-20 05:25:46
|
Hi Tony, At this point, I do a full emergency backup and restore. >From the EXIST_HOME directory, run: bin/run.sh org.exist.backup.ExportMain -x -d path/to/backup/file The target filename is usually the date and what went wrong, like "2025-07-20-out-of-space." After making the backup, mv the EXIST_HOME/webapp/WEB-INF/data directory elsewhere, then start up eXist. Next, extract the emergency backup and cd to db and run: ${EXIST_HOME}/bin/backup.sh -r `pwd`/system/security/__contents__.xml -u admin -P "new_password" Then run: ${EXIST_HOME}/bin/backup.sh -r `pwd`/__contents__.xml -u admin -p "new_password" The first restore will restore the security collection and set the new password for admin. The second restore will restore the rest of your database using the new password for admin. Finally, I have two scripts that I always run after a restore to restore the repos that I use and restore permissions to certain collections. xquery version "3.0"; <fix-repo>{( repo:remove("http://exist-db.org/apps/some_app"), repo:install-and-deploy("http://exist-db.org/apps/shared", "1.2.3", " http://exist-db.org/exist/apps/public-repo/some_app"), ... )}</fix-repo> Hope this helps. Take care. 2025年7月20日(日) 11:51 Tony Graham <tg...@an...>: > I have an eXist-3 (yes, eXist-3) that won't respond and won't shutdown > (short of killing the process). Starting it devolves into multiple > stack traces with the message in the subject. Is there any hope for it? > > I got here after killing eXist after a server backup filled the > filesystem and eXist became unresponsive. (The backup was because a > Jenkins job running an Ant task had created some ghost collections that > showed only in the Java client and that the 'admin' user couldn't > delete, and I was going to tweak then reload the backup.) > > Regards, > > > Tony Graham. > -- > Senior Architect > XML Division > Antenna House, Inc. > ---- > Skerries, Ireland > tg...@an... > > > > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open > -- Michael Westbay Writer/System Administrator http://www.japanesebaseball.com/ |
From: Tony G. <tg...@an...> - 2025-07-18 11:04:23
|
I have an eXist-3 (yes, eXist-3) that won't respond and won't shutdown (short of killing the process). Starting it devolves into multiple stack traces with the message in the subject. Is there any hope for it? I got here after killing eXist after a server backup filled the filesystem and eXist became unresponsive. (The backup was because a Jenkins job running an Ant task had created some ghost collections that showed only in the Java client and that the 'admin' user couldn't delete, and I was going to tweak then reload the backup.) Regards, Tony Graham. -- Senior Architect XML Division Antenna House, Inc. ---- Skerries, Ireland tg...@an... |
From: XML S. S. <he...@xm...> - 2025-07-16 11:03:43
|
Hello XML Summer School Community, We’re delighted to share that this year marks the 25th Anniversary of the XML Summer School! This is a special milestone for us, and we’re planning a few extra touches to make it a memorable one. Whether you’re a long-time friend of the Summer School or have been curious about joining, we’d love for you to be part of it. The 2025 Summer School will take place at St Edmund Hall (Teddy Hall), Oxford, from Sunday 14th to Friday 19th September. The schedule and registration information can be viewed on our website: https://xmlsummerschool.org <https://xmlsummerschool.org/> As a little thank you, we’re offering 10% off registration if you sign up before the end of July; just use the promo code JULY at checkout. The week will include a mix of tutorials, talks, and discussions covering XML, but also related technologies like web standards, data modeling, accessible publishing, semantics, and more. There’s also a full line-up of social activities every evening; past favourites include punting on the river, guided tours of Oxford, pub nights, museum visits, and a formal dinner in one of Oxford’s historic college halls. If you happen to arrive early, Oxford Open Doors takes place over the weekend (13–14 September), giving you free access to many of the city’s usually hidden historic sites: https://www.oxfordpreservation.org.uk/oxford-open-doors <https://www.oxfordpreservation.org.uk/oxford-open-doors> Please feel free to share this with colleagues, friends, or anyone who might enjoy a week of tech, history, and summer fun in Oxford. Hope to see you there, The XML Summer School Team |
From: Joe W. <jo...@gm...> - 2025-06-12 17:10:53
|
Hi Seena, I'm not certain that the errors you shared indicate data corruption. Seeing the full stack trace of the error in your initial post in this thread would help. The primary cause of database corruption is improper shutdown. This is typically fixed automatically when next starting eXist, since it detects an unclean shutdown by the presence of lock files and triggers a reindex. If this built-in process is not successful, you can perform the emergency recovery technique of creating a backup and restoring from the backup. This is documented here: https://exist-db.org/exist/apps/doc/backup#emergency-export-tool Proactively performing backups - via a scheduled job, for example - allows you to expedite this process by restoring from a recent backup, rather than needing to perform the backup upon a possibly corrupted database. Best, Joe On Thu, Jun 12, 2025 at 5:29 AM Sanil, Seena via Exist-open < exi...@li...> wrote: > Thanks Michael W. > > > > Currently we are reindexing the entire collection. For future, we would > like to know if there is a way to avoid data/collection being corrupted ? > If it gets corrupted, what is the best way to fix it? > > > > Thanks > > Seena > > > > *From: *Michael Westbay <wes...@ja...> > *Date: *Tuesday, June 10, 2025 at 7:52 PM > *To: *Sanil, Seena <ss...@bl...> > *Cc: *eXist DB ML <exi...@li...>, Mulvaney, Michael < > MMu...@bl...>, Lubey, Brian < > bl...@bl...> > *Subject: *Re: [Exist-open] existdb errors on xQueries > > *CAUTION:* *This email originated from outside of the organization. Do > not click links or open attachments unless you recognize the sender and > know the content is safe. Please report suspicious emails to: > phi...@bl... <phi...@bl...>* > > > > Hi Seena, > > > > The return type error is an easy one to fix. The function declaration > expects to always return 1 and only 1 configuration. But it is getting none. > > > > If returning no configuration is an option, redefine the function as: > > > > ```xquery > > declare function config:get-configuration() as element(configuration)? { > > ... > > }; > > ``` > > > > If it should always return a configuration, then you need to determine why > it is not finding the configuration. Since you aren't passing a parameter > into the function, it doesn't appear to be something on a per-user basis -- > so there should always be the global configuration. > > > > Hope this helps point you in the right direction. > > > > Take care. > > > > > > > > > > > > 2025年6月10日(火) 22:08 Sanil, Seena via Exist-open < > exi...@li...>: > > Also errors like this “ > > exerr:*ERROR* The actual return type does not match the sequence type > declared in the function's signature: config:get-configuration() as > element(configuration). Expected cardinality: exactly one, got 0. [at line > 42, column 42, source: /db/apps/eXide/controller.xq]\nIn > function:\n\tconfig:get-configuration() as element(configuration)” > > > > *From: *Sanil, Seena <ss...@bl...> > *Date: *Monday, June 9, 2025 at 7:44 PM > *To: *eXist DB ML <exi...@li...> > *Cc: *Lubey, Brian <bl...@bl...>, Mulvaney, Michael < > MMu...@bl...> > *Subject: *existdb errors on xQueries > > Hello, > > We are getting some errors while running xQuery’s all at once against > our eXist database with 1.3 million files from different clients like > browser, oxygen desktop, Postman. > > > > The errors are “Error while processing /exist/xmlrpc: An unknown error > occurred: org.apache.xmlrpc.XmlRpcException: I/O error while processing > request”. . After that, the existdb instance hosted in the the cloud is > not coming up. Is there a reason for this behavior? How do we avoid this in > the future? > > > > The xQuery we used are : > > _query=//cite.bna.reference&_howmany=1000 , on a collection > > > > Thanks > > Seena Sanil > > > > > > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open > > > > -- > > Michael Westbay > Writer/System Administrator > http://www.japanesebaseball.com/ > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open > |
From: Sanil, S. <ss...@bl...> - 2025-06-11 12:53:55
|
Thanks Michael W. Currently we are reindexing the entire collection. For future, we would like to know if there is a way to avoid data/collection being corrupted ? If it gets corrupted, what is the best way to fix it? Thanks Seena From: Michael Westbay <wes...@ja...> Date: Tuesday, June 10, 2025 at 7:52 PM To: Sanil, Seena <ss...@bl...> Cc: eXist DB ML <exi...@li...>, Mulvaney, Michael <MMu...@bl...>, Lubey, Brian <bl...@bl...> Subject: Re: [Exist-open] existdb errors on xQueries CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report suspicious emails to: phi...@bl...<mailto:phi...@bl...> Hi Seena, The return type error is an easy one to fix. The function declaration expects to always return 1 and only 1 configuration. But it is getting none. If returning no configuration is an option, redefine the function as: ```xquery declare function config:get-configuration() as element(configuration)? { ... }; ``` If it should always return a configuration, then you need to determine why it is not finding the configuration. Since you aren't passing a parameter into the function, it doesn't appear to be something on a per-user basis -- so there should always be the global configuration. Hope this helps point you in the right direction. Take care. 2025年6月10日(火) 22:08 Sanil, Seena via Exist-open <exi...@li...<mailto:exi...@li...>>: Also errors like this “ exerr:ERROR The actual return type does not match the sequence type declared in the function's signature: config:get-configuration() as element(configuration). Expected cardinality: exactly one, got 0. [at line 42, column 42, source: /db/apps/eXide/controller.xq]\nIn function:\n\tconfig:get-configuration() as element(configuration)” From: Sanil, Seena <ss...@bl...<mailto:ss...@bl...>> Date: Monday, June 9, 2025 at 7:44 PM To: eXist DB ML <exi...@li...<mailto:exi...@li...>> Cc: Lubey, Brian <bl...@bl...<mailto:bl...@bl...>>, Mulvaney, Michael <MMu...@bl...<mailto:MMu...@bl...>> Subject: existdb errors on xQueries Hello, We are getting some errors while running xQuery’s all at once against our eXist database with 1.3 million files from different clients like browser, oxygen desktop, Postman. The errors are “Error while processing /exist/xmlrpc: An unknown error occurred: org.apache.xmlrpc.XmlRpcException: I/O error while processing request”. . After that, the existdb instance hosted in the the cloud is not coming up. Is there a reason for this behavior? How do we avoid this in the future? The xQuery we used are : _query=//cite.bna.reference&_howmany=1000 , on a collection Thanks Seena Sanil _______________________________________________ Exist-open mailing list Exi...@li...<mailto:Exi...@li...> https://lists.sourceforge.net/lists/listinfo/exist-open -- Michael Westbay Writer/System Administrator http://www.japanesebaseball.com/ |
From: Michael W. <wes...@ja...> - 2025-06-10 23:52:15
|
Hi Seena, The return type error is an easy one to fix. The function declaration expects to always return 1 and only 1 configuration. But it is getting none. If returning no configuration is an option, redefine the function as: ```xquery declare function config:get-configuration() as element(configuration)? { ... }; ``` If it should always return a configuration, then you need to determine why it is not finding the configuration. Since you aren't passing a parameter into the function, it doesn't appear to be something on a per-user basis -- so there should always be the global configuration. Hope this helps point you in the right direction. Take care. 2025年6月10日(火) 22:08 Sanil, Seena via Exist-open < exi...@li...>: > Also errors like this “ > > exerr:*ERROR* The actual return type does not match the sequence type > declared in the function's signature: config:get-configuration() as > element(configuration). Expected cardinality: exactly one, got 0. [at line > 42, column 42, source: /db/apps/eXide/controller.xq]\nIn > function:\n\tconfig:get-configuration() as element(configuration)” > > > > *From: *Sanil, Seena <ss...@bl...> > *Date: *Monday, June 9, 2025 at 7:44 PM > *To: *eXist DB ML <exi...@li...> > *Cc: *Lubey, Brian <bl...@bl...>, Mulvaney, Michael < > MMu...@bl...> > *Subject: *existdb errors on xQueries > > Hello, > > We are getting some errors while running xQuery’s all at once against > our eXist database with 1.3 million files from different clients like > browser, oxygen desktop, Postman. > > > > The errors are “Error while processing /exist/xmlrpc: An unknown error > occurred: org.apache.xmlrpc.XmlRpcException: I/O error while processing > request”. . After that, the existdb instance hosted in the the cloud is > not coming up. Is there a reason for this behavior? How do we avoid this in > the future? > > > > The xQuery we used are : > > _query=//cite.bna.reference&_howmany=1000 , on a collection > > > > Thanks > > Seena Sanil > > > > > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open > -- Michael Westbay Writer/System Administrator http://www.japanesebaseball.com/ |
From: Sanil, S. <ss...@bl...> - 2025-06-10 16:33:32
|
We are on the latest 6.4.0, jdk 17.0 and Linux/X86_64 based ECS container hosted in AWS Thanks. From: Joe Wicentowski <jo...@gm...> Date: Tuesday, June 10, 2025 at 11:10 AM To: Sanil, Seena <ss...@bl...> Cc: eXist DB ML <exi...@li...>, Mulvaney, Michael <MMu...@bl...>, Lubey, Brian <bl...@bl...> Subject: Re: [Exist-open] existdb errors on xQueries CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report suspicious emails to: phi...@bl...<mailto:phi...@bl...> Hi Seena, To help others here who may be able provide suggestions or answers, let me quote from Juri's reply to your previous question [1]: It would help a lot to know the version of the eXist-db instance you are working with... This information, along with operating system and java version is generally helpful when posting questions here. Could you please provide this contextual information to give us a better perspective on your situation? Thanks, Joe [1] https://sourceforge.net/p/exist/mailman/exist-open/thread/70cabacc-a5a7-4fec-92d0-584151999ad0%40existsolutions.com/#msg59185883 On Tue, Jun 10, 2025 at 9:09 AM Sanil, Seena via Exist-open <exi...@li...<mailto:exi...@li...>> wrote: Also errors like this “ exerr:ERROR The actual return type does not match the sequence type declared in the function's signature: config:get-configuration() as element(configuration). Expected cardinality: exactly one, got 0. [at line 42, column 42, source: /db/apps/eXide/controller.xq]\nIn function:\n\tconfig:get-configuration() as element(configuration)” From: Sanil, Seena <ss...@bl...<mailto:ss...@bl...>> Date: Monday, June 9, 2025 at 7:44 PM To: eXist DB ML <exi...@li...<mailto:exi...@li...>> Cc: Lubey, Brian <bl...@bl...<mailto:bl...@bl...>>, Mulvaney, Michael <MMu...@bl...<mailto:MMu...@bl...>> Subject: existdb errors on xQueries Hello, We are getting some errors while running xQuery’s all at once against our eXist database with 1.3 million files from different clients like browser, oxygen desktop, Postman. The errors are “Error while processing /exist/xmlrpc: An unknown error occurred: org.apache.xmlrpc.XmlRpcException: I/O error while processing request”. . After that, the existdb instance hosted in the the cloud is not coming up. Is there a reason for this behavior? How do we avoid this in the future? The xQuery we used are : _query=//cite.bna.reference&_howmany=1000 , on a collection Thanks Seena Sanil _______________________________________________ Exist-open mailing list Exi...@li...<mailto:Exi...@li...> https://lists.sourceforge.net/lists/listinfo/exist-open |
From: Joe W. <jo...@gm...> - 2025-06-10 15:10:58
|
Hi Seena, To help others here who may be able provide suggestions or answers, let me quote from Juri's reply to your previous question [1]: It would help a lot to know the version of the eXist-db instance you are > working with... This information, along with operating system and java > version is generally helpful when posting questions here. Could you please provide this contextual information to give us a better perspective on your situation? Thanks, Joe [1] https://sourceforge.net/p/exist/mailman/exist-open/thread/70cabacc-a5a7-4fec-92d0-584151999ad0%40existsolutions.com/#msg59185883 On Tue, Jun 10, 2025 at 9:09 AM Sanil, Seena via Exist-open < exi...@li...> wrote: > Also errors like this “ > > exerr:*ERROR* The actual return type does not match the sequence type > declared in the function's signature: config:get-configuration() as > element(configuration). Expected cardinality: exactly one, got 0. [at line > 42, column 42, source: /db/apps/eXide/controller.xq]\nIn > function:\n\tconfig:get-configuration() as element(configuration)” > > > > *From: *Sanil, Seena <ss...@bl...> > *Date: *Monday, June 9, 2025 at 7:44 PM > *To: *eXist DB ML <exi...@li...> > *Cc: *Lubey, Brian <bl...@bl...>, Mulvaney, Michael < > MMu...@bl...> > *Subject: *existdb errors on xQueries > > Hello, > > We are getting some errors while running xQuery’s all at once against > our eXist database with 1.3 million files from different clients like > browser, oxygen desktop, Postman. > > > > The errors are “Error while processing /exist/xmlrpc: An unknown error > occurred: org.apache.xmlrpc.XmlRpcException: I/O error while processing > request”. . After that, the existdb instance hosted in the the cloud is > not coming up. Is there a reason for this behavior? How do we avoid this in > the future? > > > > The xQuery we used are : > > _query=//cite.bna.reference&_howmany=1000 , on a collection > > > > Thanks > > Seena Sanil > > > > > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open > |
From: Sanil, S. <ss...@bl...> - 2025-06-10 03:20:00
|
Also errors like this “ exerr:ERROR The actual return type does not match the sequence type declared in the function's signature: config:get-configuration() as element(configuration). Expected cardinality: exactly one, got 0. [at line 42, column 42, source: /db/apps/eXide/controller.xq]\nIn function:\n\tconfig:get-configuration() as element(configuration)” From: Sanil, Seena <ss...@bl...> Date: Monday, June 9, 2025 at 7:44 PM To: eXist DB ML <exi...@li...> Cc: Lubey, Brian <bl...@bl...>, Mulvaney, Michael <MMu...@bl...> Subject: existdb errors on xQueries Hello, We are getting some errors while running xQuery’s all at once against our eXist database with 1.3 million files from different clients like browser, oxygen desktop, Postman. The errors are “Error while processing /exist/xmlrpc: An unknown error occurred: org.apache.xmlrpc.XmlRpcException: I/O error while processing request”. . After that, the existdb instance hosted in the the cloud is not coming up. Is there a reason for this behavior? How do we avoid this in the future? The xQuery we used are : _query=//cite.bna.reference&_howmany=1000 , on a collection Thanks Seena Sanil |
From: Sanil, S. <ss...@bl...> - 2025-06-10 02:18:15
|
Hello, We are getting some errors while running xQuery’s all at once against our eXist database with 1.3 million files from different clients like browser, oxygen desktop, Postman. The errors are “Error while processing /exist/xmlrpc: An unknown error occurred: org.apache.xmlrpc.XmlRpcException: I/O error while processing request”. . After that, the existdb instance hosted in the the cloud is not coming up. Is there a reason for this behavior? How do we avoid this in the future? The xQuery we used are : _query=//cite.bna.reference&_howmany=1000 , on a collection Thanks Seena Sanil |
From: Adam R. <ad...@ex...> - 2025-06-03 00:24:16
|
> > 2. Has anybody looked at reading and writing that object to prometheus? > Yes we have connected this to Promethus in the past and exposed it via Grafana. -- Adam Retter eXist Core Developer in Exile { United Kingdom } ad...@ex... |
From: Dannes W. <di...@ex...> - 2025-05-29 20:19:01
|
Hi Willem, Long time no see! > On 22 May 2025, at 15:50, Willem van der Westhuizen <wi...@kw...> wrote: > > 1. Is it possible to access the $jmx object from outside existdb as in the snippet below. Any pointers for us to look > > <instance name="localhost" > url="http://localhost:8080/exist" > token=“3268b570-xxxxxxx" cron="0 * * * * ?"> > <poll cron="0/30 * * * * ?" store="yes"> > <alert name="More than 30 threads waiting for locks to be released" > condition="count($jmx//LockManager/WaitingThreads/row) > 30”/ Initially I did not recognize the snippet :-) I’d recommend to use the JMX endpoint on http://<hostname<:8080/exist/status?token=<yourToken> for the full XML file, the output can be trimmed with some parameters like http://<hostname>:8080/exist/status?c=instances&c=processes&c=locking&c=memory&c=caches&c=system&c=operatingsystem&token=<token> See https://exist-db.org/apps/doc/jmx for more details. HTH With kind regards Dannes |
From: Alberto S. <has...@gm...> - 2025-05-23 08:19:49
|
Thank you, Adam. That makes sense. On Thu, May 22, 2025 at 8:39 PM Adam Retter <ad...@ex...> wrote: > If you need to search all the documents each time, then it is most > performant to have them in a single collection to avoid locking overhead > between collections. If you only need to search a subset each time, and you > can divide them by collection and just search one (or a few collections) > this could be faster, but you would need to benchmark based on your own > corpus and search requirements. > > Adam Retter > > eXist Core Developer > { United Kingdom } > ad...@ex... > > On Mon, 19 May 2025, 10:23 Alberto Simões, <has...@gm...> wrote: > >> Hello >> >> Are there differences in terms of performance between having a large >> collection (150k docs) with or without a folder structure? >> >> I want to treat them as a single collection, but I don't know if it helps >> to have sub-collections to organise them, or if that is irrelevant to eXist. >> >> I appreciate any help you can provide. >> Alberto >> >> -- >> Alberto Simões >> _______________________________________________ >> Exist-open mailing list >> Exi...@li... >> https://lists.sourceforge.net/lists/listinfo/exist-open >> > -- Alberto Simões |
From: Adam R. <ad...@ex...> - 2025-05-22 19:39:29
|
If you need to search all the documents each time, then it is most performant to have them in a single collection to avoid locking overhead between collections. If you only need to search a subset each time, and you can divide them by collection and just search one (or a few collections) this could be faster, but you would need to benchmark based on your own corpus and search requirements. Adam Retter eXist Core Developer { United Kingdom } ad...@ex... On Mon, 19 May 2025, 10:23 Alberto Simões, <has...@gm...> wrote: > Hello > > Are there differences in terms of performance between having a large > collection (150k docs) with or without a folder structure? > > I want to treat them as a single collection, but I don't know if it helps > to have sub-collections to organise them, or if that is irrelevant to eXist. > > I appreciate any help you can provide. > Alberto > > -- > Alberto Simões > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open > |
From: Willem v. d. W. <wi...@kw...> - 2025-05-22 14:14:00
|
Hi everyone, We want to implement some performance observability tools for exist. Two questions: 1. Is it possible to access the $jmx object from outside existdb as in the snippet below. Any pointers for us to look <instance name="localhost" url="http://localhost:8080/exist" token="3268b570-392e-56ea-9550-117012413e15" cron="0 * * * * ?"> <poll cron="0/30 * * * * ?" store="yes"> <alert name="More than 30 threads waiting for locks to be released" condition="count($jmx//LockManager/WaitingThreads/row) > 30"/> <alert name="More than 40 brokers active" condition="$jmx//Database/ActiveBrokers > 10"/> <alert name="Process CPU load > 1.0" condition="$jmx//UnixOperatingSystem/ProcessCpuLoad > 0.5"/> </poll> </instance> 2. Has anybody looked at reading and writing that object to prometheus? Willem |
From: Jo C. <Jo....@ha...> - 2025-05-21 09:51:03
|
Hi all, If you do a lot of element lookups against a highly structured set of collections (for example, .../0/0, .../0/1 for storing files under uuids), you end up having to put a read lock on every subcollection in turn (256+16+1=273 collections in the uuid case) for each search. Best regards, -- Jo On Tue, May 20, 2025 at 9:17 AM Alberto Simões <has...@gm...> wrote: > Hello, Michael > > Thanks for sharing your use case. > Indeed, it might get useful > Thanks > > On Mon, May 19, 2025 at 2:23 PM Michael Westbay < > wes...@ja...> wrote: > >> Hi Alberto, >> >> For me, splitting them makes them more manageable when I am going through >> a given collection with a WebDAV editor. >> >> For example, I have a database of baseball players. The XML file for a >> given player is in the format: "surname-givenname.xml." I sort them >> under the persons collection as: >> >> [image: image.png] >> >> Each first letter is divided into two or three letter sub-collections. I >> try to keep each to around 100 names each, but as the database grows, some >> have grown as large as 300 names. That usually means that I want to divide >> it up some more. (The _ collection is for names in Kanji -- Japanese >> characters.) >> >> The reason I break them up is because WebDAV is really slow when there >> are a lot of files in a single collection. If I only processed the XML >> files, it wouldn't be an issue. But I often go in and manually edit files, >> so the hierarchy helps. >> >> A quick count of the number of players I have: >> >> xquery version "3.0"; >> >> let $start-time := current-dateTime() >> let $players := collection('/db/uni/persons')/*:person >> let $count := count($players) >> let $end-time := current-dateTime() >> >> return <result start-time="{$start-time}" end-time="{$end-time}" >> count="{$count}"/> >> >> <result start-time="2025-05-19T22:20:24.288+09:00" >> end-time="2025-05-19T22:20:24.288+09:00" count="43434"></result> >> >> Looks like it's pretty much instantaneous to get 43,434 players. In >> reality, it took a couple of seconds to display the result. >> >> >> 2025年5月19日(月) 20:12 Alberto Simões <has...@gm...>: >> >>> Hello, Michael >>> >>> I cannot split them so that I can specify different collection names. >>> In that case, splitting does not bring any additional value? >>> >>> Thanks >>> >>> On Mon, May 19, 2025 at 10:25 AM Michael Westbay < >>> wes...@ja...> wrote: >>> >>>> Hi Alberto, >>>> >>>> collection("/db/records")/record will match all <record>...</record> >>>> documents under /db/records and sub-folders (sub-collections?). >>>> >>>> If you can organize them by date (year sub-folders), including that in >>>> the collection parameter will mean less records to search. And all >>>> sub-folders under that collection will still be included in the XPath >>>> search. >>>> >>>> >>>> >>>> 2025年5月19日(月) 17:23 Alberto Simões <has...@gm...>: >>>> >>>>> Hello >>>>> >>>>> Are there differences in terms of performance between having a large >>>>> collection (150k docs) with or without a folder structure? >>>>> >>>>> I want to treat them as a single collection, but I don't know if it >>>>> helps to have sub-collections to organise them, or if that is irrelevant to >>>>> eXist. >>>>> >>>>> I appreciate any help you can provide. >>>>> Alberto >>>>> >>>>> -- >>>>> Alberto Simões >>>>> _______________________________________________ >>>>> Exist-open mailing list >>>>> Exi...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/exist-open >>>>> >>>> >>>> >>>> -- >>>> Michael Westbay >>>> Writer/System Administrator >>>> http://www.japanesebaseball.com/ >>>> >>> >>> >>> -- >>> Alberto Simões >>> >> >> >> -- >> Michael Westbay >> Writer/System Administrator >> http://www.japanesebaseball.com/ >> > > > -- > Alberto Simões > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open > |
From: Michael W. <wes...@ja...> - 2025-05-20 13:03:25
|
Ah, that's more realistic: <result start-time="2025-05-20T22:00:38.828+09:00" end-time="2025-05-20T22:00:41.07+09:00" count="43434"></result> Just over 2 seconds. Thank you, Joe, for the reminder. Take care. 2025年5月20日(火) 21:58 Joe Wicentowski <jo...@gm...>: > Hi Michael, > > For calculating execution time, use util:system-dateTime() instead of > fn:current-dateTime(). > > From the function documentation for util:system-dateTime(): > > > Contrary to fn:current-dateTime, this function is not stable, i.e. the > returned xs:dateTime will change during the evaluation time of a query and > can be used to measure time differences. > > See: > > > https://exist-db.org/exist/apps/fundocs/index.html?q=util:system-dateTime > > See also this good explanation from Dannes: > > https://sourceforge.net/p/exist/mailman/message/32809375/ > > Joe > > On Tue, May 20, 2025 at 4:54 AM Michael Westbay < > wes...@ja...> wrote: > >> Hi Alberto, >> >> For me, splitting them makes them more manageable when I am going through >> a given collection with a WebDAV editor. >> >> For example, I have a database of baseball players. The XML file for a >> given player is in the format: "surname-givenname.xml." I sort them >> under the persons collection as: >> >> [image: image.png] >> >> Each first letter is divided into two or three letter sub-collections. I >> try to keep each to around 100 names each, but as the database grows, some >> have grown as large as 300 names. That usually means that I want to divide >> it up some more. (The _ collection is for names in Kanji -- Japanese >> characters.) >> >> The reason I break them up is because WebDAV is really slow when there >> are a lot of files in a single collection. If I only processed the XML >> files, it wouldn't be an issue. But I often go in and manually edit files, >> so the hierarchy helps. >> >> A quick count of the number of players I have: >> >> xquery version "3.0"; >> >> let $start-time := current-dateTime() >> let $players := collection('/db/uni/persons')/*:person >> let $count := count($players) >> let $end-time := current-dateTime() >> >> return <result start-time="{$start-time}" end-time="{$end-time}" >> count="{$count}"/> >> >> <result start-time="2025-05-19T22:20:24.288+09:00" >> end-time="2025-05-19T22:20:24.288+09:00" count="43434"></result> >> >> Looks like it's pretty much instantaneous to get 43,434 players. In >> reality, it took a couple of seconds to display the result. >> >> >> 2025年5月19日(月) 20:12 Alberto Simões <has...@gm...>: >> >>> Hello, Michael >>> >>> I cannot split them so that I can specify different collection names. >>> In that case, splitting does not bring any additional value? >>> >>> Thanks >>> >>> On Mon, May 19, 2025 at 10:25 AM Michael Westbay < >>> wes...@ja...> wrote: >>> >>>> Hi Alberto, >>>> >>>> collection("/db/records")/record will match all <record>...</record> >>>> documents under /db/records and sub-folders (sub-collections?). >>>> >>>> If you can organize them by date (year sub-folders), including that in >>>> the collection parameter will mean less records to search. And all >>>> sub-folders under that collection will still be included in the XPath >>>> search. >>>> >>>> >>>> >>>> 2025年5月19日(月) 17:23 Alberto Simões <has...@gm...>: >>>> >>>>> Hello >>>>> >>>>> Are there differences in terms of performance between having a large >>>>> collection (150k docs) with or without a folder structure? >>>>> >>>>> I want to treat them as a single collection, but I don't know if it >>>>> helps to have sub-collections to organise them, or if that is irrelevant to >>>>> eXist. >>>>> >>>>> I appreciate any help you can provide. >>>>> Alberto >>>>> >>>>> -- >>>>> Alberto Simões >>>>> _______________________________________________ >>>>> Exist-open mailing list >>>>> Exi...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/exist-open >>>>> >>>> >>>> >>>> -- >>>> Michael Westbay >>>> Writer/System Administrator >>>> http://www.japanesebaseball.com/ >>>> >>> >>> >>> -- >>> Alberto Simões >>> >> >> >> -- >> Michael Westbay >> Writer/System Administrator >> http://www.japanesebaseball.com/ >> _______________________________________________ >> Exist-open mailing list >> Exi...@li... >> https://lists.sourceforge.net/lists/listinfo/exist-open >> > -- Michael Westbay Writer/System Administrator http://www.japanesebaseball.com/ |
From: Michael W. <wes...@ja...> - 2025-05-20 13:00:38
|
Hi Joe, Thank you. It did seem odd that they were both the same instant. And I seem to recall this coming up many years ago. 2025年5月20日(火) 21:58 Joe Wicentowski <jo...@gm...>: > Hi Michael, > > For calculating execution time, use util:system-dateTime() instead of > fn:current-dateTime(). > > From the function documentation for util:system-dateTime(): > > > Contrary to fn:current-dateTime, this function is not stable, i.e. the > returned xs:dateTime will change during the evaluation time of a query and > can be used to measure time differences. > > See: > > > https://exist-db.org/exist/apps/fundocs/index.html?q=util:system-dateTime > > See also this good explanation from Dannes: > > https://sourceforge.net/p/exist/mailman/message/32809375/ > > Joe > > On Tue, May 20, 2025 at 4:54 AM Michael Westbay < > wes...@ja...> wrote: > >> Hi Alberto, >> >> For me, splitting them makes them more manageable when I am going through >> a given collection with a WebDAV editor. >> >> For example, I have a database of baseball players. The XML file for a >> given player is in the format: "surname-givenname.xml." I sort them >> under the persons collection as: >> >> [image: image.png] >> >> Each first letter is divided into two or three letter sub-collections. I >> try to keep each to around 100 names each, but as the database grows, some >> have grown as large as 300 names. That usually means that I want to divide >> it up some more. (The _ collection is for names in Kanji -- Japanese >> characters.) >> >> The reason I break them up is because WebDAV is really slow when there >> are a lot of files in a single collection. If I only processed the XML >> files, it wouldn't be an issue. But I often go in and manually edit files, >> so the hierarchy helps. >> >> A quick count of the number of players I have: >> >> xquery version "3.0"; >> >> let $start-time := current-dateTime() >> let $players := collection('/db/uni/persons')/*:person >> let $count := count($players) >> let $end-time := current-dateTime() >> >> return <result start-time="{$start-time}" end-time="{$end-time}" >> count="{$count}"/> >> >> <result start-time="2025-05-19T22:20:24.288+09:00" >> end-time="2025-05-19T22:20:24.288+09:00" count="43434"></result> >> >> Looks like it's pretty much instantaneous to get 43,434 players. In >> reality, it took a couple of seconds to display the result. >> >> >> 2025年5月19日(月) 20:12 Alberto Simões <has...@gm...>: >> >>> Hello, Michael >>> >>> I cannot split them so that I can specify different collection names. >>> In that case, splitting does not bring any additional value? >>> >>> Thanks >>> >>> On Mon, May 19, 2025 at 10:25 AM Michael Westbay < >>> wes...@ja...> wrote: >>> >>>> Hi Alberto, >>>> >>>> collection("/db/records")/record will match all <record>...</record> >>>> documents under /db/records and sub-folders (sub-collections?). >>>> >>>> If you can organize them by date (year sub-folders), including that in >>>> the collection parameter will mean less records to search. And all >>>> sub-folders under that collection will still be included in the XPath >>>> search. >>>> >>>> >>>> >>>> 2025年5月19日(月) 17:23 Alberto Simões <has...@gm...>: >>>> >>>>> Hello >>>>> >>>>> Are there differences in terms of performance between having a large >>>>> collection (150k docs) with or without a folder structure? >>>>> >>>>> I want to treat them as a single collection, but I don't know if it >>>>> helps to have sub-collections to organise them, or if that is irrelevant to >>>>> eXist. >>>>> >>>>> I appreciate any help you can provide. >>>>> Alberto >>>>> >>>>> -- >>>>> Alberto Simões >>>>> _______________________________________________ >>>>> Exist-open mailing list >>>>> Exi...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/exist-open >>>>> >>>> >>>> >>>> -- >>>> Michael Westbay >>>> Writer/System Administrator >>>> http://www.japanesebaseball.com/ >>>> >>> >>> >>> -- >>> Alberto Simões >>> >> >> >> -- >> Michael Westbay >> Writer/System Administrator >> http://www.japanesebaseball.com/ >> _______________________________________________ >> Exist-open mailing list >> Exi...@li... >> https://lists.sourceforge.net/lists/listinfo/exist-open >> > -- Michael Westbay Writer/System Administrator http://www.japanesebaseball.com/ |