exist-open — Open discussion

Re: [Exist-open] "Collection /db/system/security/exist/accounts cannot be created."

[Michael W. <wes...@ja...>]

Hi Tony,

Okay, the "Failed to open database" message concerns me.

under /usr/local/src/exist3-data do you have the following files?

-rw-rw-r--.  1 exist exist       1451 Jul 25 08:57 00000000d6.log
-rw-rw-r--.  1 exist exist   13991936 Jul 24 22:40 collections.dbx
-rw-r--r--.  1 exist exist       1515 Jul 24 22:38 counters
-rw-rw-r--.  1 exist exist         16 Jul 25 08:57 dbx_dir.lck
-rw-rw-r--.  1 exist exist 6785642496 Jul 24 22:38 dom.dbx
drwxrwxr-x. 15 exist exist       4096 Sep 27  2020 expathrepo
drwxrwxr-x.  3 exist exist        132 May 20 12:14 export
drwxrwxr-x.  3 exist exist         16 Apr  3  2018 fs
drwxrwxr-x.  2 exist exist          6 Jul 21 20:47 fs.journal
-rw-rw-r--.  1 exist exist        140 Apr 16  2018 jmxservlet.token
-rw-rw-r--.  1 exist exist         16 Jul 25 08:57 journal.lck
drwxrwxr-x.  2 exist exist        103 Jul 23  2022 lucene
-rw-rw-r--.  1 exist exist    4636672 May 22 21:46 ngram.dbx
drwxrwxr-x.  2 exist exist       8192 Jul 25 08:44 range
-rw-rw-r--.  1 exist exist        120 Apr  9  2018 README
-rw-rw-r--.  1 exist exist        257 Apr  9  2018 RECOVERY
-rw-------.  1 exist exist         11 Mar  8 10:20 restxq.registry
drwxrwxr-x.  2 exist exist         42 Jul 23  2022 sanity
-rw-rw-r--.  1 exist exist       8192 Jul 23  2022 sort.dbx
-rw-rw-r--.  1 exist exist 6046371840 Jul 24 22:40 structure.dbx
-rw-rw-r--.  1 exist exist      63342 Jul 21 20:35 symbols.dbx
-rw-rw-r--.  1 exist exist   46583808 Jul 24 22:40 values.dbx

The dom.dbx file should be very large. That's where all of your data is at.
If this file is
broken, then the data isn't recoverable.

The RECOVERY file states:

RECOVERY
========
If the index got corrupted, this is how to recover:

- stop database
- remove all ".log" transaction log files
- remove all ".lck" lock files
- remove all ".dbx" files *EXCEPT dom.dbx, collections.dbx and symbols.dbx*
- start database

I really don't like doing it without a backup. So first, make a tarball
of /usr/local/src/exist3-data
as a backup of the raw data then start the recovery process above.

And once the database is up and running again, run:

cd /usr/local/src/exist3
bin/backup.sh --dir ~/tmp/2025-07-25-backup --backup /db

And this is my script modified for your setup that I run daily via crontab:

#!/bin/sh
unset JAVA_OPTS
export JAVA_HOME=/usr
export EXIST_HOME=/usr/local/src/exist3
export TARGET=/tmp
export DATE=`date +"%Y-%m-%d"`
export PATH=$JAVA_HOME/bin:/usr/local/bin:/usr/bin:/bin
cd $EXIST_HOME
bin/backup.sh --backup /db --dir $TARGET/full-$DATE -u "admin" -p "password"

I thought it was supposed to get the username and password from the
$EXIST_HOME/backup.properties
file, but when I tested it by hand just now without the -u/-p parameters it
failed. Not sure why.

Anyway, daily backups going forward would be a very good idea.

Take care.

2025年7月24日(木) 23:21 Tony Graham <tg...@an...>:

> Michael,
>
> Thanks for following through with this.
>
> On 23/07/2025 09:00, Michael Westbay wrote:
> > Okay. Let's first find EXIST_HOME by finding where the conf.xml file is.
> >
> > find /usr/local -name conf.xml
>
> It found multiple, but the ones of interest are:
>
> /usr/local/src/exist3/installer/conf.xml
> /usr/local/src/exist3/conf.xml
>
> > If /usr/local is EXIST_HOME, then run.sh will be in /usr/local/bin along
> > with all of the other eXist executables.
>
> The only things in '/usr/local/bin' are 'tracd' and 'trac-admin' for use
> with Trac. (Still the best issue tracking/wiki system.)
>
> > This conf.xml file will then specify where the data directory is:
>
> '/usr/local/src/exist3/conf.xml' has:
>
>      <db-connection cacheSize="256M" checkMaxCacheSize="true"
> collectionCache="64M" database="native"
>          files="/usr/local/src/exist3-data" pageSize="4096"
> nodesBuffer="1000" cacheShrinkThreshold="10000"
>          doc-ids="default" minDiskSpace="128M">
>
> > You could also search for run.sh the same way:
> >
> > find /usr/local -name run.sh
>
> As before, I don't have a 'run.sh' (unless you count 'run.sh' for
> Antenna House Formatter).
>
> Running the Java installer as a non-root user has never given me a
> 'run.sh'.
>
> I downloaded the 'run.sh' [1] for that eXist-db version from GitHub,
> edited it to use the same Java alternative as used for the database, ran
> it, and got:
>
> ----
> cd "/usr/local/src/exist3/" ; ./bin/run.sh org.exist.backup.ExportMain
> -x -d /mnt/tmp
> ERROR: Failed to open database:
> org.exist.storage.BrokerPoolServiceException: org.exist.EXistException:
> java.io.IOException: Collection /db/system/security/exist/accounts
> cannot be created.
> ----
>
> If this can be solved, the problem then becomes that we don't know how
> many other collections will have similar problems, so I'm not hopeful.
>
> Regards,
>
>
> Tony Graham.
> --
> Senior Architect
> XML Division
> Antenna House, Inc.
> ----
> Skerries, Ireland
> tg...@an...
>
> [1]
>
> https://raw.githubusercontent.com/eXist-db/exist/refs/tags/eXist-3.0.RC2/bin/run.sh
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>


-- 
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/

Re: [Exist-open] "Collection /db/system/security/exist/accounts cannot be created."

[Tony G. <tg...@an...>]

Michael,

Thanks for following through with this.

On 23/07/2025 09:00, Michael Westbay wrote:
> Okay. Let's first find EXIST_HOME by finding where the conf.xml file is.
> 
> find /usr/local -name conf.xml

It found multiple, but the ones of interest are:

/usr/local/src/exist3/installer/conf.xml
/usr/local/src/exist3/conf.xml

> If /usr/local is EXIST_HOME, then run.sh will be in /usr/local/bin along 
> with all of the other eXist executables.

The only things in '/usr/local/bin' are 'tracd' and 'trac-admin' for use
with Trac. (Still the best issue tracking/wiki system.)

> This conf.xml file will then specify where the data directory is:

'/usr/local/src/exist3/conf.xml' has:

     <db-connection cacheSize="256M" checkMaxCacheSize="true" 
collectionCache="64M" database="native"
         files="/usr/local/src/exist3-data" pageSize="4096" 
nodesBuffer="1000" cacheShrinkThreshold="10000"
         doc-ids="default" minDiskSpace="128M">

> You could also search for run.sh the same way:
> 
> find /usr/local -name run.sh

As before, I don't have a 'run.sh' (unless you count 'run.sh' for
Antenna House Formatter).

Running the Java installer as a non-root user has never given me a 'run.sh'.

I downloaded the 'run.sh' [1] for that eXist-db version from GitHub,
edited it to use the same Java alternative as used for the database, ran
it, and got:

----
cd "/usr/local/src/exist3/" ; ./bin/run.sh org.exist.backup.ExportMain 
-x -d /mnt/tmp
ERROR: Failed to open database: 
org.exist.storage.BrokerPoolServiceException: org.exist.EXistException: 
java.io.IOException: Collection /db/system/security/exist/accounts 
cannot be created.
----

If this can be solved, the problem then becomes that we don't know how
many other collections will have similar problems, so I'm not hopeful.

Regards,


Tony Graham.
-- 
Senior Architect
XML Division
Antenna House, Inc.
----
Skerries, Ireland
tg...@an...

[1] 
https://raw.githubusercontent.com/eXist-db/exist/refs/tags/eXist-3.0.RC2/bin/run.sh

Re: [Exist-open] "Collection /db/system/security/exist/accounts cannot be created."

[Michael W. <wes...@ja...>]

Hi Tony,

Okay. Let's first find EXIST_HOME by finding where the conf.xml file is.

find /usr/local -name conf.xml

If /usr/local is EXIST_HOME, then run.sh will be in /usr/local/bin along
with all of the other eXist executables.

This conf.xml file will then specify where the data directory is:

    <db-connection cacheSize="256M" checkMaxCacheSize="true"
collectionCache="64M" database="native"
        files="*webapp/WEB-INF/data*" pageSize="4096" nodesBuffer="1000"
cacheShrinkThreshold="10000"
        doc-ids="default" minDiskSpace="128M">

[Emphasis added]

You could also search for run.sh the same way:

find /usr/local -name run.sh

Once we know where the executables are, let's go from there.

Take care.


> On Linux and installed from the installer jar.  I don't have 'run.sh',



-- 
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/

Re: [Exist-open] "Collection /db/system/security/exist/accounts cannot be created."

[Tony G. <tg...@an...>]

Michael,

On 22/07/2025 12:25, Michael Westbay wrote:
>     Thanks, but still no luck. I don't have a 'run.sh'. [1] 
> 
> 
> That's odd. I have eXist 3.x and the EXIST_HOME/bin directory has the 
> following:
> 
> backup.bat  batch.d     client.sh   eXist_32x32.ico  run.bat  server.bat 
>   shutdown.bat  startup.bat
> backup.sh   client.bat  deprecated  functions.d *run.sh*   server.sh   
> shutdown.sh   startup.sh

On Linux and installed from the installer jar.  I don't have 'run.sh',
nor do I have any '.bat' files.

Similarly with exist6 installed from installer jar.

> These shell scripts make sure that all of the CLASS_PATH stuff gets 
> setup properly for running the utilities like
> `org.exist.backup.ExportMain`.

I don't doubt it, but I don't have it.  The next thing to try will be to
use a different installer to see if that comes with 'run.sh'.

...
>     I got multiple messages of the form:
> 
>     INFO: None of content dirs exist:
>     '/usr/local/src/exist3-data/expathrepo/doc-0.4.7/content' and
>     '/usr/local/src/exist3-data/expathrepo/doc-0.4.7/doc'
> 
> Are you trying to execute from the build directory? Did you not run the 
> installer to place it all in a
> user's directory? Are you running it from a container and pointing to a 
> data directory on the host
> machine?

I have '/usr/local/src/exist3-data/expathrepo/doc-0.4.7/' but not the
subdirectories.  That directory was last modified in 2022, so the absent
directories are not a by-product of my current problem after all.

>     then I got:
> 
>     17:09:17.550 [main] ERROR org.exist.storage.DBBroker - no data found at
>     tid 25; page 60995
>     java.io.IOException: no data found at tid 25; page 60995
>              at
>     org.exist.storage.index.BFile.getAsStream(BFile.java:492) ~[exist.jar:?]
>              at
>     org.exist.storage.index.BFile.getAsStream(BFile.java:462) ~[exist.jar:?]
>              at
>     org.exist.storage.NativeBroker.openCollection(NativeBroker.java:928)
>     [exist.jar:?]
>     ...
>     17:09:17.554 [main] ERROR org.exist.storage.DBBroker - Collection
>     '/db/system/security/exist/accounts' found in subCollections set but is
>     missing from collections.dbx!
>     17:09:17.569 [main] ERROR org.exist.storage.BrokerPools - Unable to
>     initialize database instance 'exist':
>     org.exist.storage.BrokerPoolServiceException: org.exist.EXistException:
>     java.io.IOException: Collection /db/system/security/exist/accounts
>     cannot be created.
> 
> 
> Before doing anything -- like bringing the eXist server back up, you 
> want to make an emergency
> backup. Be sure to make the target a local directory that exists and 
> that you have write permission
> for. I use "~/backup/date-reason" (where "date" is the date that it shut 
> down and "reason" is "locked"
> or "out-of-memory"). The "backup" directory exists in my exist user's 
> home directory.

I don't have an exist user.  The stack trace is from running
'org.exist.backup.ExportMain', not from trying to bring the server back up.

> /usr/local/* is not usually writable by ordinary users.

I am the only user, so both ordinary and extraordinary (and still 'sudo'
to do things as 'root').

Putting things in '/usr/local/src' is a habit that I picked up over
thirty years ago with multiple people working on projects on a SunOS
machine.  Still seems wrong to put projects under my home directory even
when I am the only user.

> Let's start by locating where the EXIST_HOME/bin directory is.

I installed in '/usr/local/src/exist3', with data in '/usr/local/src/
exist3-data'.

Regards,


Tony Graham.
-- 
Senior Architect
XML Division
Antenna House, Inc.
----
Skerries, Ireland
tg...@an...

Re: [Exist-open] "Collection /db/system/security/exist/accounts cannot be created."

[Michael W. <wes...@ja...>]

Hi Tony,

Thanks, but still no luck. I don't have a 'run.sh'. [1]


That's odd. I have eXist 3.x and the EXIST_HOME/bin directory has the
following:

backup.bat  batch.d     client.sh   eXist_32x32.ico  run.bat  server.bat
 shutdown.bat  startup.bat
backup.sh   client.bat  deprecated  functions.d      *run.sh*   server.sh
shutdown.sh   startup.sh

These shell scripts make sure that all of the CLASS_PATH stuff gets setup
properly for running the
utilities like `org.exist.backup.ExportMain`.

If you're on Windows, you would use `run.bat`.


> After I made a
> script that no longer bombed out from missing dependencies, I ran
> 'org.exist.backup.ExportMain'.
>
> I got multiple messages of the form:
>
> INFO: None of content dirs exist:
> '/usr/local/src/exist3-data/expathrepo/doc-0.4.7/content' and
> '/usr/local/src/exist3-data/expathrepo/doc-0.4.7/doc'
>

Are you trying to execute from the build directory? Did you not run the
installer to place it all in a
user's directory? Are you running it from a container and pointing to a
data directory on the host
machine?


> then I got:
>
> 17:09:17.550 [main] ERROR org.exist.storage.DBBroker - no data found at
> tid 25; page 60995
> java.io.IOException: no data found at tid 25; page 60995
>         at org.exist.storage.index.BFile.getAsStream(BFile.java:492)
> ~[exist.jar:?]
>         at org.exist.storage.index.BFile.getAsStream(BFile.java:462)
> ~[exist.jar:?]
>         at
> org.exist.storage.NativeBroker.openCollection(NativeBroker.java:928)
> [exist.jar:?]
> ...
> 17:09:17.554 [main] ERROR org.exist.storage.DBBroker - Collection
> '/db/system/security/exist/accounts' found in subCollections set but is
> missing from collections.dbx!
> 17:09:17.569 [main] ERROR org.exist.storage.BrokerPools - Unable to
> initialize database instance 'exist':
> org.exist.storage.BrokerPoolServiceException: org.exist.EXistException:
> java.io.IOException: Collection /db/system/security/exist/accounts
> cannot be created.


Before doing anything -- like bringing the eXist server back up, you want
to make an emergency
backup. Be sure to make the target a local directory that exists and that
you have write permission
for. I use "~/backup/date-reason" (where "date" is the date that it shut
down and "reason" is "locked"
or "out-of-memory"). The "backup" directory exists in my exist user's home
directory.

/usr/local/* is not usually writable by ordinary users.

Let's start by locating where the EXIST_HOME/bin directory is.

-- 
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/

Re: [Exist-open] "Collection /db/system/security/exist/accounts cannot be created."

[Tony G. <tg...@an...>]

On 20/07/2025 06:25, Michael Westbay wrote:
> At this point, I do a full emergency backup and restore.
> 
>  From the EXIST_HOME directory, run:
> 
> bin/run.sh org.exist.backup.ExportMain -x -d path/to/backup/file

Thanks, but still no luck. I don't have a 'run.sh'. [1]  After I made a
script that no longer bombed out from missing dependencies, I ran
'org.exist.backup.ExportMain'.

I got multiple messages of the form:

INFO: None of content dirs exist: 
'/usr/local/src/exist3-data/expathrepo/doc-0.4.7/content' and 
'/usr/local/src/exist3-data/expathrepo/doc-0.4.7/doc'

then I got:

17:09:17.550 [main] ERROR org.exist.storage.DBBroker - no data found at 
tid 25; page 60995
java.io.IOException: no data found at tid 25; page 60995
	at org.exist.storage.index.BFile.getAsStream(BFile.java:492) ~[exist.jar:?]
	at org.exist.storage.index.BFile.getAsStream(BFile.java:462) ~[exist.jar:?]
	at org.exist.storage.NativeBroker.openCollection(NativeBroker.java:928) 
[exist.jar:?]
...
17:09:17.554 [main] ERROR org.exist.storage.DBBroker - Collection 
'/db/system/security/exist/accounts' found in subCollections set but is 
missing from collections.dbx!
17:09:17.569 [main] ERROR org.exist.storage.BrokerPools - Unable to 
initialize database instance 'exist': 
org.exist.storage.BrokerPoolServiceException: org.exist.EXistException: 
java.io.IOException: Collection /db/system/security/exist/accounts 
cannot be created.

Regards,


Tony Graham.
-- 
Senior Architect
XML Division
Antenna House, Inc.
----
Skerries, Ireland
tg...@an...

[1] The only reference to it that I found was an email from you to this 
list in 2021.

Re: [Exist-open] "Collection /db/system/security/exist/accounts cannot be created."

[Michael W. <wes...@ja...>]

Hi Tony,

Oh, and if you rely on any counters, once the new "data" directory has been
created, be sure to copy the old "counters" file into it. It's a pain to
fix counter IDs after they've been assigned a second time.

Take care.

2025年7月20日(日) 14:25 Michael Westbay <wes...@ja...>:

> Hi Tony,
>
> At this point, I do a full emergency backup and restore.
>
> From the EXIST_HOME directory, run:
>
> bin/run.sh org.exist.backup.ExportMain -x -d path/to/backup/file
>
> The target filename is usually the date and what went wrong, like
> "2025-07-20-out-of-space."
>
> After making the backup, mv the EXIST_HOME/webapp/WEB-INF/data directory
> elsewhere, then start up eXist.
>
> Next, extract the emergency backup and cd to db and run:
>
> ${EXIST_HOME}/bin/backup.sh -r `pwd`/system/security/__contents__.xml -u
> admin -P "new_password"
>
> Then run:
>
> ${EXIST_HOME}/bin/backup.sh -r `pwd`/__contents__.xml -u admin -p
> "new_password"
>
> The first restore will restore the security collection and set the new
> password for admin.
>
> The second restore will restore the rest of your database using the new
> password for admin.
>
> Finally, I have two scripts that I always run after a restore to restore
> the repos that I use and restore permissions to certain collections.
>
>
> xquery version "3.0";
>
> <fix-repo>{(
>   repo:remove("http://exist-db.org/apps/some_app"),
>   repo:install-and-deploy("http://exist-db.org/apps/shared", "1.2.3", "
> http://exist-db.org/exist/apps/public-repo/some_app"),
>   ...
> )}</fix-repo>
>
> Hope this helps.
>
> Take care.
>
>
>
> 2025年7月20日(日) 11:51 Tony Graham <tg...@an...>:
>
>> I have an eXist-3 (yes, eXist-3) that won't respond and won't shutdown
>> (short of killing the process).  Starting it devolves into multiple
>> stack traces with the message in the subject.  Is there any hope for it?
>>
>> I got here after killing eXist after a server backup filled the
>> filesystem and eXist became unresponsive.  (The backup was because a
>> Jenkins job running an Ant task had created some ghost collections that
>> showed only in the Java client and that the 'admin' user couldn't
>> delete, and I was going to tweak then reload the backup.)
>>
>> Regards,
>>
>>
>> Tony Graham.
>> --
>> Senior Architect
>> XML Division
>> Antenna House, Inc.
>> ----
>> Skerries, Ireland
>> tg...@an...
>>
>>
>>
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>
>
>
> --
> Michael Westbay
> Writer/System Administrator
> http://www.japanesebaseball.com/
>


-- 
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/

Re: [Exist-open] "Collection /db/system/security/exist/accounts cannot be created."

[Michael W. <wes...@ja...>]

Hi Tony,

At this point, I do a full emergency backup and restore.

>From the EXIST_HOME directory, run:

bin/run.sh org.exist.backup.ExportMain -x -d path/to/backup/file

The target filename is usually the date and what went wrong, like
"2025-07-20-out-of-space."

After making the backup, mv the EXIST_HOME/webapp/WEB-INF/data directory
elsewhere, then start up eXist.

Next, extract the emergency backup and cd to db and run:

${EXIST_HOME}/bin/backup.sh -r `pwd`/system/security/__contents__.xml -u
admin -P "new_password"

Then run:

${EXIST_HOME}/bin/backup.sh -r `pwd`/__contents__.xml -u admin -p
"new_password"

The first restore will restore the security collection and set the new
password for admin.

The second restore will restore the rest of your database using the new
password for admin.

Finally, I have two scripts that I always run after a restore to restore
the repos that I use and restore permissions to certain collections.


xquery version "3.0";

<fix-repo>{(
  repo:remove("http://exist-db.org/apps/some_app"),
  repo:install-and-deploy("http://exist-db.org/apps/shared", "1.2.3", "
http://exist-db.org/exist/apps/public-repo/some_app"),
  ...
)}</fix-repo>

Hope this helps.

Take care.



2025年7月20日(日) 11:51 Tony Graham <tg...@an...>:

> I have an eXist-3 (yes, eXist-3) that won't respond and won't shutdown
> (short of killing the process).  Starting it devolves into multiple
> stack traces with the message in the subject.  Is there any hope for it?
>
> I got here after killing eXist after a server backup filled the
> filesystem and eXist became unresponsive.  (The backup was because a
> Jenkins job running an Ant task had created some ghost collections that
> showed only in the Java client and that the 'admin' user couldn't
> delete, and I was going to tweak then reload the backup.)
>
> Regards,
>
>
> Tony Graham.
> --
> Senior Architect
> XML Division
> Antenna House, Inc.
> ----
> Skerries, Ireland
> tg...@an...
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>


-- 
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/

[Exist-open] "Collection /db/system/security/exist/accounts cannot be created."

[Tony G. <tg...@an...>]

I have an eXist-3 (yes, eXist-3) that won't respond and won't shutdown
(short of killing the process).  Starting it devolves into multiple
stack traces with the message in the subject.  Is there any hope for it?

I got here after killing eXist after a server backup filled the
filesystem and eXist became unresponsive.  (The backup was because a
Jenkins job running an Ant task had created some ghost collections that
showed only in the Java client and that the 'admin' user couldn't
delete, and I was going to tweak then reload the backup.)

Regards,


Tony Graham.
-- 
Senior Architect
XML Division
Antenna House, Inc.
----
Skerries, Ireland
tg...@an...

[Exist-open] [ANN] XML Summer School in Oxford: Register Now for Our 25th Year!

[XML S. S. <he...@xm...>]

Hello XML Summer School Community,

We’re delighted to share that this year marks the 25th Anniversary of the XML Summer School! This is a special milestone for us, and we’re planning a few extra touches to make it a memorable one. Whether you’re a long-time friend of the Summer School or have been curious about joining, we’d love for you to be part of it.

The 2025 Summer School will take place at St Edmund Hall (Teddy Hall), Oxford, from Sunday 14th to Friday 19th September. The schedule and registration information can be viewed on our website: https://xmlsummerschool.org <https://xmlsummerschool.org/>
As a little thank you, we’re offering 10% off registration if you sign up before the end of July; just use the promo code JULY at checkout.

The week will include a mix of tutorials, talks, and discussions covering XML, but also related technologies like web standards, data modeling, accessible publishing, semantics, and more. There’s also a full line-up of social activities every evening; past favourites include punting on the river, guided tours of Oxford, pub nights, museum visits, and a formal dinner in one of Oxford’s historic college halls.

If you happen to arrive early, Oxford Open Doors takes place over the weekend (13–14 September), giving you free access to many of the city’s usually hidden historic sites: https://www.oxfordpreservation.org.uk/oxford-open-doors <https://www.oxfordpreservation.org.uk/oxford-open-doors>
Please feel free to share this with colleagues, friends, or anyone who might enjoy a week of tech, history, and summer fun in Oxford.

Hope to see you there,
The XML Summer School Team

Re: [Exist-open] existdb errors on xQueries

[Joe W. <jo...@gm...>]

Hi Seena,

I'm not certain that the errors you shared indicate data corruption. Seeing
the full stack trace of the error in your initial post in this thread would
help.

The primary cause of database corruption is improper shutdown. This is
typically fixed automatically when next starting eXist, since it detects an
unclean shutdown by the presence of lock files and triggers a reindex. If
this built-in process is not successful, you can perform the emergency
recovery technique of creating a backup and restoring from the backup. This
is documented here:

  https://exist-db.org/exist/apps/doc/backup#emergency-export-tool

Proactively performing backups - via a scheduled job, for example - allows
you to expedite this process by restoring from a recent backup, rather than
needing to perform the backup upon a possibly corrupted database.

Best,
Joe

On Thu, Jun 12, 2025 at 5:29 AM Sanil, Seena via Exist-open <
exi...@li...> wrote:

> Thanks Michael W.
>
>
>
> Currently we are reindexing the entire collection.  For future, we would
> like to know if there is a way to avoid data/collection being corrupted ?
> If it gets corrupted, what is the best way to fix it?
>
>
>
> Thanks
>
> Seena
>
>
>
> *From: *Michael Westbay <wes...@ja...>
> *Date: *Tuesday, June 10, 2025 at 7:52 PM
> *To: *Sanil, Seena <ss...@bl...>
> *Cc: *eXist DB ML <exi...@li...>, Mulvaney, Michael <
> MMu...@bl...>, Lubey, Brian <
> bl...@bl...>
> *Subject: *Re: [Exist-open] existdb errors on xQueries
>
> *CAUTION:* *This email originated from outside of the organization. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe. Please report suspicious emails to:
> phi...@bl... <phi...@bl...>*
>
>
>
> Hi Seena,
>
>
>
> The return type error is an easy one to fix. The function declaration
> expects to always return 1 and only 1 configuration. But it is getting none.
>
>
>
> If returning no configuration is an option, redefine the function as:
>
>
>
> ```xquery
>
> declare function config:get-configuration() as element(configuration)? {
>
>   ...
>
> };
>
> ```
>
>
>
> If it should always return a configuration, then you need to determine why
> it is not finding the configuration. Since you aren't passing a parameter
> into the function, it doesn't appear to be something on a per-user basis --
> so there should always be the global configuration.
>
>
>
> Hope this helps point you in the right direction.
>
>
>
> Take care.
>
>
>
>
>
>
>
>
>
>
>
> 2025年6月10日(火) 22:08 Sanil, Seena via Exist-open <
> exi...@li...>:
>
> Also errors like this “
>
> exerr:*ERROR* The actual return type does not match the sequence type
> declared in the function's signature: config:get-configuration() as
> element(configuration). Expected cardinality: exactly one, got 0. [at line
> 42, column 42, source: /db/apps/eXide/controller.xq]\nIn
> function:\n\tconfig:get-configuration() as element(configuration)”
>
>
>
> *From: *Sanil, Seena <ss...@bl...>
> *Date: *Monday, June 9, 2025 at 7:44 PM
> *To: *eXist DB ML <exi...@li...>
> *Cc: *Lubey, Brian <bl...@bl...>, Mulvaney, Michael <
> MMu...@bl...>
> *Subject: *existdb errors on xQueries
>
> Hello,
>
> We are getting some errors while running xQuery’s  all at once against
> our eXist database with 1.3 million files from different clients like
> browser, oxygen desktop, Postman.
>
>
>
> The errors are “Error while processing /exist/xmlrpc: An unknown error
> occurred: org.apache.xmlrpc.XmlRpcException: I/O error while processing
> request”. . After that, the existdb instance hosted in the the cloud is
> not coming up. Is there a reason for this behavior? How do we avoid this in
> the future?
>
>
>
> The xQuery we used are :
>
> _query=//cite.bna.reference&_howmany=1000 , on a collection
>
>
>
> Thanks
>
> Seena Sanil
>
>
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>
>
>
> --
>
> Michael Westbay
> Writer/System Administrator
> http://www.japanesebaseball.com/
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

Re: [Exist-open] existdb errors on xQueries

[Sanil, S. <ss...@bl...>]

Thanks Michael W.

Currently we are reindexing the entire collection.  For future, we would like to know if there is a way to avoid data/collection being corrupted ? If it gets corrupted, what is the best way to fix it?

Thanks
Seena

From: Michael Westbay <wes...@ja...>
Date: Tuesday, June 10, 2025 at 7:52 PM
To: Sanil, Seena <ss...@bl...>
Cc: eXist DB ML <exi...@li...>, Mulvaney, Michael <MMu...@bl...>, Lubey, Brian <bl...@bl...>
Subject: Re: [Exist-open] existdb errors on xQueries
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report suspicious emails to: phi...@bl...<mailto:phi...@bl...>

Hi Seena,

The return type error is an easy one to fix. The function declaration expects to always return 1 and only 1 configuration. But it is getting none.

If returning no configuration is an option, redefine the function as:

```xquery
declare function config:get-configuration() as element(configuration)? {
  ...
};
```

If it should always return a configuration, then you need to determine why it is not finding the configuration. Since you aren't passing a parameter into the function, it doesn't appear to be something on a per-user basis -- so there should always be the global configuration.

Hope this helps point you in the right direction.

Take care.





2025年6月10日(火) 22:08 Sanil, Seena via Exist-open <exi...@li...<mailto:exi...@li...>>:
Also errors like this “
exerr:ERROR The actual return type does not match the sequence type declared in the function's signature: config:get-configuration() as element(configuration). Expected cardinality: exactly one, got 0. [at line 42, column 42, source: /db/apps/eXide/controller.xq]\nIn function:\n\tconfig:get-configuration() as element(configuration)”

From: Sanil, Seena <ss...@bl...<mailto:ss...@bl...>>
Date: Monday, June 9, 2025 at 7:44 PM
To: eXist DB ML <exi...@li...<mailto:exi...@li...>>
Cc: Lubey, Brian <bl...@bl...<mailto:bl...@bl...>>, Mulvaney, Michael <MMu...@bl...<mailto:MMu...@bl...>>
Subject: existdb errors on xQueries
Hello,
We are getting some errors while running xQuery’s  all at once against our eXist database with 1.3 million files from different clients like browser, oxygen desktop, Postman.

The errors are “Error while processing /exist/xmlrpc: An unknown error occurred: org.apache.xmlrpc.XmlRpcException: I/O error while processing request”. . After that, the existdb instance hosted in the the cloud is not coming up. Is there a reason for this behavior? How do we avoid this in the future?

The xQuery we used are :
_query=//cite.bna.reference&_howmany=1000 , on a collection

Thanks
Seena Sanil


_______________________________________________
Exist-open mailing list
Exi...@li...<mailto:Exi...@li...>
https://lists.sourceforge.net/lists/listinfo/exist-open


--
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/

Re: [Exist-open] existdb errors on xQueries

[Michael W. <wes...@ja...>]

Hi Seena,

The return type error is an easy one to fix. The function declaration
expects to always return 1 and only 1 configuration. But it is getting none.

If returning no configuration is an option, redefine the function as:

```xquery
declare function config:get-configuration() as element(configuration)? {
  ...
};
```

If it should always return a configuration, then you need to determine why
it is not finding the configuration. Since you aren't passing a parameter
into the function, it doesn't appear to be something on a per-user basis --
so there should always be the global configuration.

Hope this helps point you in the right direction.

Take care.





2025年6月10日(火) 22:08 Sanil, Seena via Exist-open <
exi...@li...>:

> Also errors like this “
>
> exerr:*ERROR* The actual return type does not match the sequence type
> declared in the function's signature: config:get-configuration() as
> element(configuration). Expected cardinality: exactly one, got 0. [at line
> 42, column 42, source: /db/apps/eXide/controller.xq]\nIn
> function:\n\tconfig:get-configuration() as element(configuration)”
>
>
>
> *From: *Sanil, Seena <ss...@bl...>
> *Date: *Monday, June 9, 2025 at 7:44 PM
> *To: *eXist DB ML <exi...@li...>
> *Cc: *Lubey, Brian <bl...@bl...>, Mulvaney, Michael <
> MMu...@bl...>
> *Subject: *existdb errors on xQueries
>
> Hello,
>
> We are getting some errors while running xQuery’s  all at once against
> our eXist database with 1.3 million files from different clients like
> browser, oxygen desktop, Postman.
>
>
>
> The errors are “Error while processing /exist/xmlrpc: An unknown error
> occurred: org.apache.xmlrpc.XmlRpcException: I/O error while processing
> request”. . After that, the existdb instance hosted in the the cloud is
> not coming up. Is there a reason for this behavior? How do we avoid this in
> the future?
>
>
>
> The xQuery we used are :
>
> _query=//cite.bna.reference&_howmany=1000 , on a collection
>
>
>
> Thanks
>
> Seena Sanil
>
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>


-- 
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/

Re: [Exist-open] existdb errors on xQueries

[Sanil, S. <ss...@bl...>]

We are on the latest 6.4.0, jdk 17.0 and Linux/X86_64 based ECS container hosted in AWS

Thanks.

From: Joe Wicentowski <jo...@gm...>
Date: Tuesday, June 10, 2025 at 11:10 AM
To: Sanil, Seena <ss...@bl...>
Cc: eXist DB ML <exi...@li...>, Mulvaney, Michael <MMu...@bl...>, Lubey, Brian <bl...@bl...>
Subject: Re: [Exist-open] existdb errors on xQueries
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report suspicious emails to: phi...@bl...<mailto:phi...@bl...>

Hi Seena,

To help others here who may be able provide suggestions or answers, let me quote from Juri's reply to your previous question [1]:

It would help a lot to know the version of the eXist-db instance you are working with... This information, along with operating system and java version is generally helpful when posting questions here.

Could you please provide this contextual information to give us a better perspective on your situation?

Thanks,
Joe

[1] https://sourceforge.net/p/exist/mailman/exist-open/thread/70cabacc-a5a7-4fec-92d0-584151999ad0%40existsolutions.com/#msg59185883

On Tue, Jun 10, 2025 at 9:09 AM Sanil, Seena via Exist-open <exi...@li...<mailto:exi...@li...>> wrote:
Also errors like this “
exerr:ERROR The actual return type does not match the sequence type declared in the function's signature: config:get-configuration() as element(configuration). Expected cardinality: exactly one, got 0. [at line 42, column 42, source: /db/apps/eXide/controller.xq]\nIn function:\n\tconfig:get-configuration() as element(configuration)”

From: Sanil, Seena <ss...@bl...<mailto:ss...@bl...>>
Date: Monday, June 9, 2025 at 7:44 PM
To: eXist DB ML <exi...@li...<mailto:exi...@li...>>
Cc: Lubey, Brian <bl...@bl...<mailto:bl...@bl...>>, Mulvaney, Michael <MMu...@bl...<mailto:MMu...@bl...>>
Subject: existdb errors on xQueries
Hello,
We are getting some errors while running xQuery’s  all at once against our eXist database with 1.3 million files from different clients like browser, oxygen desktop, Postman.

The errors are “Error while processing /exist/xmlrpc: An unknown error occurred: org.apache.xmlrpc.XmlRpcException: I/O error while processing request”. . After that, the existdb instance hosted in the the cloud is not coming up. Is there a reason for this behavior? How do we avoid this in the future?

The xQuery we used are :
_query=//cite.bna.reference&_howmany=1000 , on a collection

Thanks
Seena Sanil


_______________________________________________
Exist-open mailing list
Exi...@li...<mailto:Exi...@li...>
https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] existdb errors on xQueries

[Joe W. <jo...@gm...>]

Hi Seena,

To help others here who may be able provide suggestions or answers, let me
quote from Juri's reply to your previous question [1]:

It would help a lot to know the version of the eXist-db instance you are
> working with... This information, along with operating system and java
> version is generally helpful when posting questions here.


Could you please provide this contextual information to give us a better
perspective on your situation?

Thanks,
Joe

[1]
https://sourceforge.net/p/exist/mailman/exist-open/thread/70cabacc-a5a7-4fec-92d0-584151999ad0%40existsolutions.com/#msg59185883

On Tue, Jun 10, 2025 at 9:09 AM Sanil, Seena via Exist-open <
exi...@li...> wrote:

> Also errors like this “
>
> exerr:*ERROR* The actual return type does not match the sequence type
> declared in the function's signature: config:get-configuration() as
> element(configuration). Expected cardinality: exactly one, got 0. [at line
> 42, column 42, source: /db/apps/eXide/controller.xq]\nIn
> function:\n\tconfig:get-configuration() as element(configuration)”
>
>
>
> *From: *Sanil, Seena <ss...@bl...>
> *Date: *Monday, June 9, 2025 at 7:44 PM
> *To: *eXist DB ML <exi...@li...>
> *Cc: *Lubey, Brian <bl...@bl...>, Mulvaney, Michael <
> MMu...@bl...>
> *Subject: *existdb errors on xQueries
>
> Hello,
>
> We are getting some errors while running xQuery’s  all at once against
> our eXist database with 1.3 million files from different clients like
> browser, oxygen desktop, Postman.
>
>
>
> The errors are “Error while processing /exist/xmlrpc: An unknown error
> occurred: org.apache.xmlrpc.XmlRpcException: I/O error while processing
> request”. . After that, the existdb instance hosted in the the cloud is
> not coming up. Is there a reason for this behavior? How do we avoid this in
> the future?
>
>
>
> The xQuery we used are :
>
> _query=//cite.bna.reference&_howmany=1000 , on a collection
>
>
>
> Thanks
>
> Seena Sanil
>
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

Re: [Exist-open] existdb errors on xQueries

[Sanil, S. <ss...@bl...>]

Also errors like this “
exerr:ERROR The actual return type does not match the sequence type declared in the function's signature: config:get-configuration() as element(configuration). Expected cardinality: exactly one, got 0. [at line 42, column 42, source: /db/apps/eXide/controller.xq]\nIn function:\n\tconfig:get-configuration() as element(configuration)”

From: Sanil, Seena <ss...@bl...>
Date: Monday, June 9, 2025 at 7:44 PM
To: eXist DB ML <exi...@li...>
Cc: Lubey, Brian <bl...@bl...>, Mulvaney, Michael <MMu...@bl...>
Subject: existdb errors on xQueries
Hello,
We are getting some errors while running xQuery’s  all at once against our eXist database with 1.3 million files from different clients like browser, oxygen desktop, Postman.

The errors are “Error while processing /exist/xmlrpc: An unknown error occurred: org.apache.xmlrpc.XmlRpcException: I/O error while processing request”. . After that, the existdb instance hosted in the the cloud is not coming up. Is there a reason for this behavior? How do we avoid this in the future?

The xQuery we used are :
_query=//cite.bna.reference&_howmany=1000 , on a collection

Thanks
Seena Sanil

[Exist-open] existdb errors on xQueries

[Sanil, S. <ss...@bl...>]

Hello,
We are getting some errors while running xQuery’s  all at once against our eXist database with 1.3 million files from different clients like browser, oxygen desktop, Postman.

The errors are “Error while processing /exist/xmlrpc: An unknown error occurred: org.apache.xmlrpc.XmlRpcException: I/O error while processing request”. . After that, the existdb instance hosted in the the cloud is not coming up. Is there a reason for this behavior? How do we avoid this in the future?

The xQuery we used are :
_query=//cite.bna.reference&_howmany=1000 , on a collection

Thanks
Seena Sanil

Re: [Exist-open] Performance monitoring for exist

[Adam R. <ad...@ex...>]

>
> 2. Has anybody looked at reading and writing that object to prometheus?
>

Yes we have connected this to Promethus in the past and exposed it via
Grafana.


-- 
Adam Retter

eXist Core Developer in Exile
{ United Kingdom }
ad...@ex...

Re: [Exist-open] Performance monitoring for exist

[Dannes W. <di...@ex...>]

Hi Willem,

Long time no see!


> On 22 May 2025, at 15:50, Willem van der Westhuizen <wi...@kw...> wrote:
> 
> 1. Is it possible to access the $jmx object from outside existdb as in the snippet below. Any pointers for us to look
> 
> <instance name="localhost"
>     url="http://localhost:8080/exist"
>     token=“3268b570-xxxxxxx" cron="0 * * * * ?">
>     <poll cron="0/30 * * * * ?" store="yes">
>     <alert name="More than 30 threads waiting for locks to be released"
>         condition="count($jmx//LockManager/WaitingThreads/row) > 30”/

Initially I did not recognize the snippet :-)  I’d recommend to use the JMX endpoint on 

http://<hostname<:8080/exist/status?token=<yourToken> 

for the full XML file, the output can be trimmed with some parameters like

http://<hostname>:8080/exist/status?c=instances&c=processes&c=locking&c=memory&c=caches&c=system&c=operatingsystem&token=<token>

See https://exist-db.org/apps/doc/jmx for more details.

HTH

With kind regards

Dannes

Re: [Exist-open] Organizing large collection

[Alberto S. <has...@gm...>]

Thank you, Adam.
That makes sense.

On Thu, May 22, 2025 at 8:39 PM Adam Retter <ad...@ex...> wrote:

> If you need to search all the documents each time, then it is most
> performant to have them in a single collection to avoid locking overhead
> between collections. If you only need to search a subset each time, and you
> can divide them by collection and just search one (or a few collections)
> this could be faster, but you would need to benchmark based on your own
> corpus and search requirements.
>
> Adam Retter
>
> eXist Core Developer
> { United Kingdom }
> ad...@ex...
>
> On Mon, 19 May 2025, 10:23 Alberto Simões, <has...@gm...> wrote:
>
>> Hello
>>
>> Are there differences in terms of performance between having a large
>> collection (150k docs) with or without a folder structure?
>>
>> I want to treat them as a single collection, but I don't know if it helps
>> to have sub-collections to organise them, or if that is irrelevant to eXist.
>>
>> I appreciate any help you can provide.
>> Alberto
>>
>> --
>> Alberto Simões
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>
>

-- 
Alberto Simões

Re: [Exist-open] Organizing large collection

[Adam R. <ad...@ex...>]

If you need to search all the documents each time, then it is most
performant to have them in a single collection to avoid locking overhead
between collections. If you only need to search a subset each time, and you
can divide them by collection and just search one (or a few collections)
this could be faster, but you would need to benchmark based on your own
corpus and search requirements.

Adam Retter

eXist Core Developer
{ United Kingdom }
ad...@ex...

On Mon, 19 May 2025, 10:23 Alberto Simões, <has...@gm...> wrote:

> Hello
>
> Are there differences in terms of performance between having a large
> collection (150k docs) with or without a folder structure?
>
> I want to treat them as a single collection, but I don't know if it helps
> to have sub-collections to organise them, or if that is irrelevant to eXist.
>
> I appreciate any help you can provide.
> Alberto
>
> --
> Alberto Simões
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

[Exist-open] Performance monitoring for exist

[Willem v. d. W. <wi...@kw...>]

Hi everyone,

We want to implement some performance observability tools for exist. Two 
questions:

1. Is it possible to access the $jmx object from outside existdb as in 
the snippet below. Any pointers for us to look

<instance name="localhost"
     url="http://localhost:8080/exist"
     token="3268b570-392e-56ea-9550-117012413e15" cron="0 * * * * ?">
     <poll cron="0/30 * * * * ?" store="yes">
     <alert name="More than 30 threads waiting for locks to be released"
         condition="count($jmx//LockManager/WaitingThreads/row) > 30"/>
     <alert name="More than 40 brokers active"
         condition="$jmx//Database/ActiveBrokers &gt; 10"/>
     <alert name="Process CPU load &gt; 1.0"
         condition="$jmx//UnixOperatingSystem/ProcessCpuLoad &gt; 0.5"/>
     </poll>
</instance>

2. Has anybody looked at reading and writing that object to prometheus?

Willem

Re: [Exist-open] Organizing large collection

[Jo C. <Jo....@ha...>]

Hi all,

If you do a lot of element lookups against a highly structured set of
collections (for example, .../0/0, .../0/1 for storing files under uuids),
you end up having to put a read lock on every subcollection in turn
(256+16+1=273 collections in the uuid case) for each search.

Best regards, -- Jo

On Tue, May 20, 2025 at 9:17 AM Alberto Simões <has...@gm...> wrote:

> Hello, Michael
>
> Thanks for sharing your use case.
> Indeed, it might get useful
> Thanks
>
> On Mon, May 19, 2025 at 2:23 PM Michael Westbay <
> wes...@ja...> wrote:
>
>> Hi Alberto,
>>
>> For me, splitting them makes them more manageable when I am going through
>> a given collection with a WebDAV editor.
>>
>> For example, I have a database of baseball players. The XML file for a
>> given player is in the format: "surname-givenname.xml." I sort them
>> under the persons collection as:
>>
>> [image: image.png]
>>
>> Each first letter is divided into two or three letter sub-collections. I
>> try to keep each to around 100 names each, but as the database grows, some
>> have grown as large as 300 names. That usually means that I want to divide
>> it up some more. (The _ collection is for names in Kanji -- Japanese
>> characters.)
>>
>> The reason I break them up is because WebDAV is really slow when there
>> are a lot of files in a single collection. If I only processed the XML
>> files, it wouldn't be an issue. But I often go in and manually edit files,
>> so the hierarchy helps.
>>
>> A quick count of the number of players I have:
>>
>> xquery version "3.0";
>>
>> let $start-time := current-dateTime()
>> let $players := collection('/db/uni/persons')/*:person
>> let $count := count($players)
>> let $end-time := current-dateTime()
>>
>> return <result start-time="{$start-time}" end-time="{$end-time}"
>> count="{$count}"/>
>>
>> <result start-time="2025-05-19T22:20:24.288+09:00"
>> end-time="2025-05-19T22:20:24.288+09:00" count="43434"></result>
>>
>> Looks like it's pretty much instantaneous to get 43,434 players. In
>> reality, it took a couple of seconds to display the result.
>>
>>
>> 2025年5月19日(月) 20:12 Alberto Simões <has...@gm...>:
>>
>>> Hello, Michael
>>>
>>> I cannot split them so that I can specify different collection names.
>>> In that case, splitting does not bring any additional value?
>>>
>>> Thanks
>>>
>>> On Mon, May 19, 2025 at 10:25 AM Michael Westbay <
>>> wes...@ja...> wrote:
>>>
>>>> Hi Alberto,
>>>>
>>>> collection("/db/records")/record will match all <record>...</record>
>>>> documents under /db/records and sub-folders (sub-collections?).
>>>>
>>>> If you can organize them by date (year sub-folders), including that in
>>>> the collection parameter will mean less records to search. And all
>>>> sub-folders under that collection will still be included in the XPath
>>>> search.
>>>>
>>>>
>>>>
>>>> 2025年5月19日(月) 17:23 Alberto Simões <has...@gm...>:
>>>>
>>>>> Hello
>>>>>
>>>>> Are there differences in terms of performance between having a large
>>>>> collection (150k docs) with or without a folder structure?
>>>>>
>>>>> I want to treat them as a single collection, but I don't know if it
>>>>> helps to have sub-collections to organise them, or if that is irrelevant to
>>>>> eXist.
>>>>>
>>>>> I appreciate any help you can provide.
>>>>> Alberto
>>>>>
>>>>> --
>>>>> Alberto Simões
>>>>> _______________________________________________
>>>>> Exist-open mailing list
>>>>> Exi...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>>>>
>>>>
>>>>
>>>> --
>>>> Michael Westbay
>>>> Writer/System Administrator
>>>> http://www.japanesebaseball.com/
>>>>
>>>
>>>
>>> --
>>> Alberto Simões
>>>
>>
>>
>> --
>> Michael Westbay
>> Writer/System Administrator
>> http://www.japanesebaseball.com/
>>
>
>
> --
> Alberto Simões
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

Re: [Exist-open] Organizing large collection

[Michael W. <wes...@ja...>]

Ah, that's more realistic:

<result start-time="2025-05-20T22:00:38.828+09:00"
end-time="2025-05-20T22:00:41.07+09:00" count="43434"></result>

Just over 2 seconds.

Thank you, Joe, for the reminder.

Take care.

2025年5月20日(火) 21:58 Joe Wicentowski <jo...@gm...>:

> Hi Michael,
>
> For calculating execution time, use util:system-dateTime() instead of
> fn:current-dateTime().
>
> From the function documentation for util:system-dateTime():
>
> > Contrary to fn:current-dateTime, this function is not stable, i.e. the
> returned xs:dateTime will change during the evaluation time of a query and
> can be used to measure time differences.
>
> See:
>
>
> https://exist-db.org/exist/apps/fundocs/index.html?q=util:system-dateTime
>
> See also this good explanation from Dannes:
>
>   https://sourceforge.net/p/exist/mailman/message/32809375/
>
> Joe
>
> On Tue, May 20, 2025 at 4:54 AM Michael Westbay <
> wes...@ja...> wrote:
>
>> Hi Alberto,
>>
>> For me, splitting them makes them more manageable when I am going through
>> a given collection with a WebDAV editor.
>>
>> For example, I have a database of baseball players. The XML file for a
>> given player is in the format: "surname-givenname.xml." I sort them
>> under the persons collection as:
>>
>> [image: image.png]
>>
>> Each first letter is divided into two or three letter sub-collections. I
>> try to keep each to around 100 names each, but as the database grows, some
>> have grown as large as 300 names. That usually means that I want to divide
>> it up some more. (The _ collection is for names in Kanji -- Japanese
>> characters.)
>>
>> The reason I break them up is because WebDAV is really slow when there
>> are a lot of files in a single collection. If I only processed the XML
>> files, it wouldn't be an issue. But I often go in and manually edit files,
>> so the hierarchy helps.
>>
>> A quick count of the number of players I have:
>>
>> xquery version "3.0";
>>
>> let $start-time := current-dateTime()
>> let $players := collection('/db/uni/persons')/*:person
>> let $count := count($players)
>> let $end-time := current-dateTime()
>>
>> return <result start-time="{$start-time}" end-time="{$end-time}"
>> count="{$count}"/>
>>
>> <result start-time="2025-05-19T22:20:24.288+09:00"
>> end-time="2025-05-19T22:20:24.288+09:00" count="43434"></result>
>>
>> Looks like it's pretty much instantaneous to get 43,434 players. In
>> reality, it took a couple of seconds to display the result.
>>
>>
>> 2025年5月19日(月) 20:12 Alberto Simões <has...@gm...>:
>>
>>> Hello, Michael
>>>
>>> I cannot split them so that I can specify different collection names.
>>> In that case, splitting does not bring any additional value?
>>>
>>> Thanks
>>>
>>> On Mon, May 19, 2025 at 10:25 AM Michael Westbay <
>>> wes...@ja...> wrote:
>>>
>>>> Hi Alberto,
>>>>
>>>> collection("/db/records")/record will match all <record>...</record>
>>>> documents under /db/records and sub-folders (sub-collections?).
>>>>
>>>> If you can organize them by date (year sub-folders), including that in
>>>> the collection parameter will mean less records to search. And all
>>>> sub-folders under that collection will still be included in the XPath
>>>> search.
>>>>
>>>>
>>>>
>>>> 2025年5月19日(月) 17:23 Alberto Simões <has...@gm...>:
>>>>
>>>>> Hello
>>>>>
>>>>> Are there differences in terms of performance between having a large
>>>>> collection (150k docs) with or without a folder structure?
>>>>>
>>>>> I want to treat them as a single collection, but I don't know if it
>>>>> helps to have sub-collections to organise them, or if that is irrelevant to
>>>>> eXist.
>>>>>
>>>>> I appreciate any help you can provide.
>>>>> Alberto
>>>>>
>>>>> --
>>>>> Alberto Simões
>>>>> _______________________________________________
>>>>> Exist-open mailing list
>>>>> Exi...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>>>>
>>>>
>>>>
>>>> --
>>>> Michael Westbay
>>>> Writer/System Administrator
>>>> http://www.japanesebaseball.com/
>>>>
>>>
>>>
>>> --
>>> Alberto Simões
>>>
>>
>>
>> --
>> Michael Westbay
>> Writer/System Administrator
>> http://www.japanesebaseball.com/
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>
>

-- 
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/

Re: [Exist-open] Organizing large collection

[Michael W. <wes...@ja...>]

Hi Joe,

Thank you. It did seem odd that they were both the same instant. And I seem
to recall this coming up many years ago.

2025年5月20日(火) 21:58 Joe Wicentowski <jo...@gm...>:

> Hi Michael,
>
> For calculating execution time, use util:system-dateTime() instead of
> fn:current-dateTime().
>
> From the function documentation for util:system-dateTime():
>
> > Contrary to fn:current-dateTime, this function is not stable, i.e. the
> returned xs:dateTime will change during the evaluation time of a query and
> can be used to measure time differences.
>
> See:
>
>
> https://exist-db.org/exist/apps/fundocs/index.html?q=util:system-dateTime
>
> See also this good explanation from Dannes:
>
>   https://sourceforge.net/p/exist/mailman/message/32809375/
>
> Joe
>
> On Tue, May 20, 2025 at 4:54 AM Michael Westbay <
> wes...@ja...> wrote:
>
>> Hi Alberto,
>>
>> For me, splitting them makes them more manageable when I am going through
>> a given collection with a WebDAV editor.
>>
>> For example, I have a database of baseball players. The XML file for a
>> given player is in the format: "surname-givenname.xml." I sort them
>> under the persons collection as:
>>
>> [image: image.png]
>>
>> Each first letter is divided into two or three letter sub-collections. I
>> try to keep each to around 100 names each, but as the database grows, some
>> have grown as large as 300 names. That usually means that I want to divide
>> it up some more. (The _ collection is for names in Kanji -- Japanese
>> characters.)
>>
>> The reason I break them up is because WebDAV is really slow when there
>> are a lot of files in a single collection. If I only processed the XML
>> files, it wouldn't be an issue. But I often go in and manually edit files,
>> so the hierarchy helps.
>>
>> A quick count of the number of players I have:
>>
>> xquery version "3.0";
>>
>> let $start-time := current-dateTime()
>> let $players := collection('/db/uni/persons')/*:person
>> let $count := count($players)
>> let $end-time := current-dateTime()
>>
>> return <result start-time="{$start-time}" end-time="{$end-time}"
>> count="{$count}"/>
>>
>> <result start-time="2025-05-19T22:20:24.288+09:00"
>> end-time="2025-05-19T22:20:24.288+09:00" count="43434"></result>
>>
>> Looks like it's pretty much instantaneous to get 43,434 players. In
>> reality, it took a couple of seconds to display the result.
>>
>>
>> 2025年5月19日(月) 20:12 Alberto Simões <has...@gm...>:
>>
>>> Hello, Michael
>>>
>>> I cannot split them so that I can specify different collection names.
>>> In that case, splitting does not bring any additional value?
>>>
>>> Thanks
>>>
>>> On Mon, May 19, 2025 at 10:25 AM Michael Westbay <
>>> wes...@ja...> wrote:
>>>
>>>> Hi Alberto,
>>>>
>>>> collection("/db/records")/record will match all <record>...</record>
>>>> documents under /db/records and sub-folders (sub-collections?).
>>>>
>>>> If you can organize them by date (year sub-folders), including that in
>>>> the collection parameter will mean less records to search. And all
>>>> sub-folders under that collection will still be included in the XPath
>>>> search.
>>>>
>>>>
>>>>
>>>> 2025年5月19日(月) 17:23 Alberto Simões <has...@gm...>:
>>>>
>>>>> Hello
>>>>>
>>>>> Are there differences in terms of performance between having a large
>>>>> collection (150k docs) with or without a folder structure?
>>>>>
>>>>> I want to treat them as a single collection, but I don't know if it
>>>>> helps to have sub-collections to organise them, or if that is irrelevant to
>>>>> eXist.
>>>>>
>>>>> I appreciate any help you can provide.
>>>>> Alberto
>>>>>
>>>>> --
>>>>> Alberto Simões
>>>>> _______________________________________________
>>>>> Exist-open mailing list
>>>>> Exi...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>>>>
>>>>
>>>>
>>>> --
>>>> Michael Westbay
>>>> Writer/System Administrator
>>>> http://www.japanesebaseball.com/
>>>>
>>>
>>>
>>> --
>>> Alberto Simões
>>>
>>
>>
>> --
>> Michael Westbay
>> Writer/System Administrator
>> http://www.japanesebaseball.com/
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>
>

-- 
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/