exist-open — Open discussion

Re: [Exist-open] existdb errors on xQueries

[Joe W. <jo...@gm...>]

Hi Seena,

I'm not certain that the errors you shared indicate data corruption. Seeing
the full stack trace of the error in your initial post in this thread would
help.

The primary cause of database corruption is improper shutdown. This is
typically fixed automatically when next starting eXist, since it detects an
unclean shutdown by the presence of lock files and triggers a reindex. If
this built-in process is not successful, you can perform the emergency
recovery technique of creating a backup and restoring from the backup. This
is documented here:

  https://exist-db.org/exist/apps/doc/backup#emergency-export-tool

Proactively performing backups - via a scheduled job, for example - allows
you to expedite this process by restoring from a recent backup, rather than
needing to perform the backup upon a possibly corrupted database.

Best,
Joe

On Thu, Jun 12, 2025 at 5:29 AM Sanil, Seena via Exist-open <
exi...@li...> wrote:

> Thanks Michael W.
>
>
>
> Currently we are reindexing the entire collection.  For future, we would
> like to know if there is a way to avoid data/collection being corrupted ?
> If it gets corrupted, what is the best way to fix it?
>
>
>
> Thanks
>
> Seena
>
>
>
> *From: *Michael Westbay <wes...@ja...>
> *Date: *Tuesday, June 10, 2025 at 7:52 PM
> *To: *Sanil, Seena <ss...@bl...>
> *Cc: *eXist DB ML <exi...@li...>, Mulvaney, Michael <
> MMu...@bl...>, Lubey, Brian <
> bl...@bl...>
> *Subject: *Re: [Exist-open] existdb errors on xQueries
>
> *CAUTION:* *This email originated from outside of the organization. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe. Please report suspicious emails to:
> phi...@bl... <phi...@bl...>*
>
>
>
> Hi Seena,
>
>
>
> The return type error is an easy one to fix. The function declaration
> expects to always return 1 and only 1 configuration. But it is getting none.
>
>
>
> If returning no configuration is an option, redefine the function as:
>
>
>
> ```xquery
>
> declare function config:get-configuration() as element(configuration)? {
>
>   ...
>
> };
>
> ```
>
>
>
> If it should always return a configuration, then you need to determine why
> it is not finding the configuration. Since you aren't passing a parameter
> into the function, it doesn't appear to be something on a per-user basis --
> so there should always be the global configuration.
>
>
>
> Hope this helps point you in the right direction.
>
>
>
> Take care.
>
>
>
>
>
>
>
>
>
>
>
> 2025年6月10日(火) 22:08 Sanil, Seena via Exist-open <
> exi...@li...>:
>
> Also errors like this “
>
> exerr:*ERROR* The actual return type does not match the sequence type
> declared in the function's signature: config:get-configuration() as
> element(configuration). Expected cardinality: exactly one, got 0. [at line
> 42, column 42, source: /db/apps/eXide/controller.xq]\nIn
> function:\n\tconfig:get-configuration() as element(configuration)”
>
>
>
> *From: *Sanil, Seena <ss...@bl...>
> *Date: *Monday, June 9, 2025 at 7:44 PM
> *To: *eXist DB ML <exi...@li...>
> *Cc: *Lubey, Brian <bl...@bl...>, Mulvaney, Michael <
> MMu...@bl...>
> *Subject: *existdb errors on xQueries
>
> Hello,
>
> We are getting some errors while running xQuery’s  all at once against
> our eXist database with 1.3 million files from different clients like
> browser, oxygen desktop, Postman.
>
>
>
> The errors are “Error while processing /exist/xmlrpc: An unknown error
> occurred: org.apache.xmlrpc.XmlRpcException: I/O error while processing
> request”. . After that, the existdb instance hosted in the the cloud is
> not coming up. Is there a reason for this behavior? How do we avoid this in
> the future?
>
>
>
> The xQuery we used are :
>
> _query=//cite.bna.reference&_howmany=1000 , on a collection
>
>
>
> Thanks
>
> Seena Sanil
>
>
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>
>
>
> --
>
> Michael Westbay
> Writer/System Administrator
> http://www.japanesebaseball.com/
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

Re: [Exist-open] existdb errors on xQueries

[Sanil, S. <ss...@bl...>]

Thanks Michael W.

Currently we are reindexing the entire collection.  For future, we would like to know if there is a way to avoid data/collection being corrupted ? If it gets corrupted, what is the best way to fix it?

Thanks
Seena

From: Michael Westbay <wes...@ja...>
Date: Tuesday, June 10, 2025 at 7:52 PM
To: Sanil, Seena <ss...@bl...>
Cc: eXist DB ML <exi...@li...>, Mulvaney, Michael <MMu...@bl...>, Lubey, Brian <bl...@bl...>
Subject: Re: [Exist-open] existdb errors on xQueries
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report suspicious emails to: phi...@bl...<mailto:phi...@bl...>

Hi Seena,

The return type error is an easy one to fix. The function declaration expects to always return 1 and only 1 configuration. But it is getting none.

If returning no configuration is an option, redefine the function as:

```xquery
declare function config:get-configuration() as element(configuration)? {
  ...
};
```

If it should always return a configuration, then you need to determine why it is not finding the configuration. Since you aren't passing a parameter into the function, it doesn't appear to be something on a per-user basis -- so there should always be the global configuration.

Hope this helps point you in the right direction.

Take care.





2025年6月10日(火) 22:08 Sanil, Seena via Exist-open <exi...@li...<mailto:exi...@li...>>:
Also errors like this “
exerr:ERROR The actual return type does not match the sequence type declared in the function's signature: config:get-configuration() as element(configuration). Expected cardinality: exactly one, got 0. [at line 42, column 42, source: /db/apps/eXide/controller.xq]\nIn function:\n\tconfig:get-configuration() as element(configuration)”

From: Sanil, Seena <ss...@bl...<mailto:ss...@bl...>>
Date: Monday, June 9, 2025 at 7:44 PM
To: eXist DB ML <exi...@li...<mailto:exi...@li...>>
Cc: Lubey, Brian <bl...@bl...<mailto:bl...@bl...>>, Mulvaney, Michael <MMu...@bl...<mailto:MMu...@bl...>>
Subject: existdb errors on xQueries
Hello,
We are getting some errors while running xQuery’s  all at once against our eXist database with 1.3 million files from different clients like browser, oxygen desktop, Postman.

The errors are “Error while processing /exist/xmlrpc: An unknown error occurred: org.apache.xmlrpc.XmlRpcException: I/O error while processing request”. . After that, the existdb instance hosted in the the cloud is not coming up. Is there a reason for this behavior? How do we avoid this in the future?

The xQuery we used are :
_query=//cite.bna.reference&_howmany=1000 , on a collection

Thanks
Seena Sanil


_______________________________________________
Exist-open mailing list
Exi...@li...<mailto:Exi...@li...>
https://lists.sourceforge.net/lists/listinfo/exist-open


--
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/

Re: [Exist-open] existdb errors on xQueries

[Michael W. <wes...@ja...>]

Hi Seena,

The return type error is an easy one to fix. The function declaration
expects to always return 1 and only 1 configuration. But it is getting none.

If returning no configuration is an option, redefine the function as:

```xquery
declare function config:get-configuration() as element(configuration)? {
  ...
};
```

If it should always return a configuration, then you need to determine why
it is not finding the configuration. Since you aren't passing a parameter
into the function, it doesn't appear to be something on a per-user basis --
so there should always be the global configuration.

Hope this helps point you in the right direction.

Take care.





2025年6月10日(火) 22:08 Sanil, Seena via Exist-open <
exi...@li...>:

> Also errors like this “
>
> exerr:*ERROR* The actual return type does not match the sequence type
> declared in the function's signature: config:get-configuration() as
> element(configuration). Expected cardinality: exactly one, got 0. [at line
> 42, column 42, source: /db/apps/eXide/controller.xq]\nIn
> function:\n\tconfig:get-configuration() as element(configuration)”
>
>
>
> *From: *Sanil, Seena <ss...@bl...>
> *Date: *Monday, June 9, 2025 at 7:44 PM
> *To: *eXist DB ML <exi...@li...>
> *Cc: *Lubey, Brian <bl...@bl...>, Mulvaney, Michael <
> MMu...@bl...>
> *Subject: *existdb errors on xQueries
>
> Hello,
>
> We are getting some errors while running xQuery’s  all at once against
> our eXist database with 1.3 million files from different clients like
> browser, oxygen desktop, Postman.
>
>
>
> The errors are “Error while processing /exist/xmlrpc: An unknown error
> occurred: org.apache.xmlrpc.XmlRpcException: I/O error while processing
> request”. . After that, the existdb instance hosted in the the cloud is
> not coming up. Is there a reason for this behavior? How do we avoid this in
> the future?
>
>
>
> The xQuery we used are :
>
> _query=//cite.bna.reference&_howmany=1000 , on a collection
>
>
>
> Thanks
>
> Seena Sanil
>
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>


-- 
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/

Re: [Exist-open] existdb errors on xQueries

[Sanil, S. <ss...@bl...>]

We are on the latest 6.4.0, jdk 17.0 and Linux/X86_64 based ECS container hosted in AWS

Thanks.

From: Joe Wicentowski <jo...@gm...>
Date: Tuesday, June 10, 2025 at 11:10 AM
To: Sanil, Seena <ss...@bl...>
Cc: eXist DB ML <exi...@li...>, Mulvaney, Michael <MMu...@bl...>, Lubey, Brian <bl...@bl...>
Subject: Re: [Exist-open] existdb errors on xQueries
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report suspicious emails to: phi...@bl...<mailto:phi...@bl...>

Hi Seena,

To help others here who may be able provide suggestions or answers, let me quote from Juri's reply to your previous question [1]:

It would help a lot to know the version of the eXist-db instance you are working with... This information, along with operating system and java version is generally helpful when posting questions here.

Could you please provide this contextual information to give us a better perspective on your situation?

Thanks,
Joe

[1] https://sourceforge.net/p/exist/mailman/exist-open/thread/70cabacc-a5a7-4fec-92d0-584151999ad0%40existsolutions.com/#msg59185883

On Tue, Jun 10, 2025 at 9:09 AM Sanil, Seena via Exist-open <exi...@li...<mailto:exi...@li...>> wrote:
Also errors like this “
exerr:ERROR The actual return type does not match the sequence type declared in the function's signature: config:get-configuration() as element(configuration). Expected cardinality: exactly one, got 0. [at line 42, column 42, source: /db/apps/eXide/controller.xq]\nIn function:\n\tconfig:get-configuration() as element(configuration)”

From: Sanil, Seena <ss...@bl...<mailto:ss...@bl...>>
Date: Monday, June 9, 2025 at 7:44 PM
To: eXist DB ML <exi...@li...<mailto:exi...@li...>>
Cc: Lubey, Brian <bl...@bl...<mailto:bl...@bl...>>, Mulvaney, Michael <MMu...@bl...<mailto:MMu...@bl...>>
Subject: existdb errors on xQueries
Hello,
We are getting some errors while running xQuery’s  all at once against our eXist database with 1.3 million files from different clients like browser, oxygen desktop, Postman.

The errors are “Error while processing /exist/xmlrpc: An unknown error occurred: org.apache.xmlrpc.XmlRpcException: I/O error while processing request”. . After that, the existdb instance hosted in the the cloud is not coming up. Is there a reason for this behavior? How do we avoid this in the future?

The xQuery we used are :
_query=//cite.bna.reference&_howmany=1000 , on a collection

Thanks
Seena Sanil


_______________________________________________
Exist-open mailing list
Exi...@li...<mailto:Exi...@li...>
https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] existdb errors on xQueries

[Joe W. <jo...@gm...>]

Hi Seena,

To help others here who may be able provide suggestions or answers, let me
quote from Juri's reply to your previous question [1]:

It would help a lot to know the version of the eXist-db instance you are
> working with... This information, along with operating system and java
> version is generally helpful when posting questions here.


Could you please provide this contextual information to give us a better
perspective on your situation?

Thanks,
Joe

[1]
https://sourceforge.net/p/exist/mailman/exist-open/thread/70cabacc-a5a7-4fec-92d0-584151999ad0%40existsolutions.com/#msg59185883

On Tue, Jun 10, 2025 at 9:09 AM Sanil, Seena via Exist-open <
exi...@li...> wrote:

> Also errors like this “
>
> exerr:*ERROR* The actual return type does not match the sequence type
> declared in the function's signature: config:get-configuration() as
> element(configuration). Expected cardinality: exactly one, got 0. [at line
> 42, column 42, source: /db/apps/eXide/controller.xq]\nIn
> function:\n\tconfig:get-configuration() as element(configuration)”
>
>
>
> *From: *Sanil, Seena <ss...@bl...>
> *Date: *Monday, June 9, 2025 at 7:44 PM
> *To: *eXist DB ML <exi...@li...>
> *Cc: *Lubey, Brian <bl...@bl...>, Mulvaney, Michael <
> MMu...@bl...>
> *Subject: *existdb errors on xQueries
>
> Hello,
>
> We are getting some errors while running xQuery’s  all at once against
> our eXist database with 1.3 million files from different clients like
> browser, oxygen desktop, Postman.
>
>
>
> The errors are “Error while processing /exist/xmlrpc: An unknown error
> occurred: org.apache.xmlrpc.XmlRpcException: I/O error while processing
> request”. . After that, the existdb instance hosted in the the cloud is
> not coming up. Is there a reason for this behavior? How do we avoid this in
> the future?
>
>
>
> The xQuery we used are :
>
> _query=//cite.bna.reference&_howmany=1000 , on a collection
>
>
>
> Thanks
>
> Seena Sanil
>
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

Re: [Exist-open] existdb errors on xQueries

[Sanil, S. <ss...@bl...>]

Also errors like this “
exerr:ERROR The actual return type does not match the sequence type declared in the function's signature: config:get-configuration() as element(configuration). Expected cardinality: exactly one, got 0. [at line 42, column 42, source: /db/apps/eXide/controller.xq]\nIn function:\n\tconfig:get-configuration() as element(configuration)”

From: Sanil, Seena <ss...@bl...>
Date: Monday, June 9, 2025 at 7:44 PM
To: eXist DB ML <exi...@li...>
Cc: Lubey, Brian <bl...@bl...>, Mulvaney, Michael <MMu...@bl...>
Subject: existdb errors on xQueries
Hello,
We are getting some errors while running xQuery’s  all at once against our eXist database with 1.3 million files from different clients like browser, oxygen desktop, Postman.

The errors are “Error while processing /exist/xmlrpc: An unknown error occurred: org.apache.xmlrpc.XmlRpcException: I/O error while processing request”. . After that, the existdb instance hosted in the the cloud is not coming up. Is there a reason for this behavior? How do we avoid this in the future?

The xQuery we used are :
_query=//cite.bna.reference&_howmany=1000 , on a collection

Thanks
Seena Sanil

[Exist-open] existdb errors on xQueries

[Sanil, S. <ss...@bl...>]

Hello,
We are getting some errors while running xQuery’s  all at once against our eXist database with 1.3 million files from different clients like browser, oxygen desktop, Postman.

The errors are “Error while processing /exist/xmlrpc: An unknown error occurred: org.apache.xmlrpc.XmlRpcException: I/O error while processing request”. . After that, the existdb instance hosted in the the cloud is not coming up. Is there a reason for this behavior? How do we avoid this in the future?

The xQuery we used are :
_query=//cite.bna.reference&_howmany=1000 , on a collection

Thanks
Seena Sanil

Re: [Exist-open] Performance monitoring for exist

[Adam R. <ad...@ex...>]

>
> 2. Has anybody looked at reading and writing that object to prometheus?
>

Yes we have connected this to Promethus in the past and exposed it via
Grafana.


-- 
Adam Retter

eXist Core Developer in Exile
{ United Kingdom }
ad...@ex...

Re: [Exist-open] Performance monitoring for exist

[Dannes W. <di...@ex...>]

Hi Willem,

Long time no see!


> On 22 May 2025, at 15:50, Willem van der Westhuizen <wi...@kw...> wrote:
> 
> 1. Is it possible to access the $jmx object from outside existdb as in the snippet below. Any pointers for us to look
> 
> <instance name="localhost"
>     url="http://localhost:8080/exist"
>     token=“3268b570-xxxxxxx" cron="0 * * * * ?">
>     <poll cron="0/30 * * * * ?" store="yes">
>     <alert name="More than 30 threads waiting for locks to be released"
>         condition="count($jmx//LockManager/WaitingThreads/row) > 30”/

Initially I did not recognize the snippet :-)  I’d recommend to use the JMX endpoint on 

http://<hostname<:8080/exist/status?token=<yourToken> 

for the full XML file, the output can be trimmed with some parameters like

http://<hostname>:8080/exist/status?c=instances&c=processes&c=locking&c=memory&c=caches&c=system&c=operatingsystem&token=<token>

See https://exist-db.org/apps/doc/jmx for more details.

HTH

With kind regards

Dannes

Re: [Exist-open] Organizing large collection

[Alberto S. <has...@gm...>]

Thank you, Adam.
That makes sense.

On Thu, May 22, 2025 at 8:39 PM Adam Retter <ad...@ex...> wrote:

> If you need to search all the documents each time, then it is most
> performant to have them in a single collection to avoid locking overhead
> between collections. If you only need to search a subset each time, and you
> can divide them by collection and just search one (or a few collections)
> this could be faster, but you would need to benchmark based on your own
> corpus and search requirements.
>
> Adam Retter
>
> eXist Core Developer
> { United Kingdom }
> ad...@ex...
>
> On Mon, 19 May 2025, 10:23 Alberto Simões, <has...@gm...> wrote:
>
>> Hello
>>
>> Are there differences in terms of performance between having a large
>> collection (150k docs) with or without a folder structure?
>>
>> I want to treat them as a single collection, but I don't know if it helps
>> to have sub-collections to organise them, or if that is irrelevant to eXist.
>>
>> I appreciate any help you can provide.
>> Alberto
>>
>> --
>> Alberto Simões
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>
>

-- 
Alberto Simões

Re: [Exist-open] Organizing large collection

[Adam R. <ad...@ex...>]

If you need to search all the documents each time, then it is most
performant to have them in a single collection to avoid locking overhead
between collections. If you only need to search a subset each time, and you
can divide them by collection and just search one (or a few collections)
this could be faster, but you would need to benchmark based on your own
corpus and search requirements.

Adam Retter

eXist Core Developer
{ United Kingdom }
ad...@ex...

On Mon, 19 May 2025, 10:23 Alberto Simões, <has...@gm...> wrote:

> Hello
>
> Are there differences in terms of performance between having a large
> collection (150k docs) with or without a folder structure?
>
> I want to treat them as a single collection, but I don't know if it helps
> to have sub-collections to organise them, or if that is irrelevant to eXist.
>
> I appreciate any help you can provide.
> Alberto
>
> --
> Alberto Simões
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

[Exist-open] Performance monitoring for exist

[Willem v. d. W. <wi...@kw...>]

Hi everyone,

We want to implement some performance observability tools for exist. Two 
questions:

1. Is it possible to access the $jmx object from outside existdb as in 
the snippet below. Any pointers for us to look

<instance name="localhost"
     url="http://localhost:8080/exist"
     token="3268b570-392e-56ea-9550-117012413e15" cron="0 * * * * ?">
     <poll cron="0/30 * * * * ?" store="yes">
     <alert name="More than 30 threads waiting for locks to be released"
         condition="count($jmx//LockManager/WaitingThreads/row) > 30"/>
     <alert name="More than 40 brokers active"
         condition="$jmx//Database/ActiveBrokers &gt; 10"/>
     <alert name="Process CPU load &gt; 1.0"
         condition="$jmx//UnixOperatingSystem/ProcessCpuLoad &gt; 0.5"/>
     </poll>
</instance>

2. Has anybody looked at reading and writing that object to prometheus?

Willem

Re: [Exist-open] Organizing large collection

[Jo C. <Jo....@ha...>]

Hi all,

If you do a lot of element lookups against a highly structured set of
collections (for example, .../0/0, .../0/1 for storing files under uuids),
you end up having to put a read lock on every subcollection in turn
(256+16+1=273 collections in the uuid case) for each search.

Best regards, -- Jo

On Tue, May 20, 2025 at 9:17 AM Alberto Simões <has...@gm...> wrote:

> Hello, Michael
>
> Thanks for sharing your use case.
> Indeed, it might get useful
> Thanks
>
> On Mon, May 19, 2025 at 2:23 PM Michael Westbay <
> wes...@ja...> wrote:
>
>> Hi Alberto,
>>
>> For me, splitting them makes them more manageable when I am going through
>> a given collection with a WebDAV editor.
>>
>> For example, I have a database of baseball players. The XML file for a
>> given player is in the format: "surname-givenname.xml." I sort them
>> under the persons collection as:
>>
>> [image: image.png]
>>
>> Each first letter is divided into two or three letter sub-collections. I
>> try to keep each to around 100 names each, but as the database grows, some
>> have grown as large as 300 names. That usually means that I want to divide
>> it up some more. (The _ collection is for names in Kanji -- Japanese
>> characters.)
>>
>> The reason I break them up is because WebDAV is really slow when there
>> are a lot of files in a single collection. If I only processed the XML
>> files, it wouldn't be an issue. But I often go in and manually edit files,
>> so the hierarchy helps.
>>
>> A quick count of the number of players I have:
>>
>> xquery version "3.0";
>>
>> let $start-time := current-dateTime()
>> let $players := collection('/db/uni/persons')/*:person
>> let $count := count($players)
>> let $end-time := current-dateTime()
>>
>> return <result start-time="{$start-time}" end-time="{$end-time}"
>> count="{$count}"/>
>>
>> <result start-time="2025-05-19T22:20:24.288+09:00"
>> end-time="2025-05-19T22:20:24.288+09:00" count="43434"></result>
>>
>> Looks like it's pretty much instantaneous to get 43,434 players. In
>> reality, it took a couple of seconds to display the result.
>>
>>
>> 2025年5月19日(月) 20:12 Alberto Simões <has...@gm...>:
>>
>>> Hello, Michael
>>>
>>> I cannot split them so that I can specify different collection names.
>>> In that case, splitting does not bring any additional value?
>>>
>>> Thanks
>>>
>>> On Mon, May 19, 2025 at 10:25 AM Michael Westbay <
>>> wes...@ja...> wrote:
>>>
>>>> Hi Alberto,
>>>>
>>>> collection("/db/records")/record will match all <record>...</record>
>>>> documents under /db/records and sub-folders (sub-collections?).
>>>>
>>>> If you can organize them by date (year sub-folders), including that in
>>>> the collection parameter will mean less records to search. And all
>>>> sub-folders under that collection will still be included in the XPath
>>>> search.
>>>>
>>>>
>>>>
>>>> 2025年5月19日(月) 17:23 Alberto Simões <has...@gm...>:
>>>>
>>>>> Hello
>>>>>
>>>>> Are there differences in terms of performance between having a large
>>>>> collection (150k docs) with or without a folder structure?
>>>>>
>>>>> I want to treat them as a single collection, but I don't know if it
>>>>> helps to have sub-collections to organise them, or if that is irrelevant to
>>>>> eXist.
>>>>>
>>>>> I appreciate any help you can provide.
>>>>> Alberto
>>>>>
>>>>> --
>>>>> Alberto Simões
>>>>> _______________________________________________
>>>>> Exist-open mailing list
>>>>> Exi...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>>>>
>>>>
>>>>
>>>> --
>>>> Michael Westbay
>>>> Writer/System Administrator
>>>> http://www.japanesebaseball.com/
>>>>
>>>
>>>
>>> --
>>> Alberto Simões
>>>
>>
>>
>> --
>> Michael Westbay
>> Writer/System Administrator
>> http://www.japanesebaseball.com/
>>
>
>
> --
> Alberto Simões
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

Re: [Exist-open] Organizing large collection

[Michael W. <wes...@ja...>]

Ah, that's more realistic:

<result start-time="2025-05-20T22:00:38.828+09:00"
end-time="2025-05-20T22:00:41.07+09:00" count="43434"></result>

Just over 2 seconds.

Thank you, Joe, for the reminder.

Take care.

2025年5月20日(火) 21:58 Joe Wicentowski <jo...@gm...>:

> Hi Michael,
>
> For calculating execution time, use util:system-dateTime() instead of
> fn:current-dateTime().
>
> From the function documentation for util:system-dateTime():
>
> > Contrary to fn:current-dateTime, this function is not stable, i.e. the
> returned xs:dateTime will change during the evaluation time of a query and
> can be used to measure time differences.
>
> See:
>
>
> https://exist-db.org/exist/apps/fundocs/index.html?q=util:system-dateTime
>
> See also this good explanation from Dannes:
>
>   https://sourceforge.net/p/exist/mailman/message/32809375/
>
> Joe
>
> On Tue, May 20, 2025 at 4:54 AM Michael Westbay <
> wes...@ja...> wrote:
>
>> Hi Alberto,
>>
>> For me, splitting them makes them more manageable when I am going through
>> a given collection with a WebDAV editor.
>>
>> For example, I have a database of baseball players. The XML file for a
>> given player is in the format: "surname-givenname.xml." I sort them
>> under the persons collection as:
>>
>> [image: image.png]
>>
>> Each first letter is divided into two or three letter sub-collections. I
>> try to keep each to around 100 names each, but as the database grows, some
>> have grown as large as 300 names. That usually means that I want to divide
>> it up some more. (The _ collection is for names in Kanji -- Japanese
>> characters.)
>>
>> The reason I break them up is because WebDAV is really slow when there
>> are a lot of files in a single collection. If I only processed the XML
>> files, it wouldn't be an issue. But I often go in and manually edit files,
>> so the hierarchy helps.
>>
>> A quick count of the number of players I have:
>>
>> xquery version "3.0";
>>
>> let $start-time := current-dateTime()
>> let $players := collection('/db/uni/persons')/*:person
>> let $count := count($players)
>> let $end-time := current-dateTime()
>>
>> return <result start-time="{$start-time}" end-time="{$end-time}"
>> count="{$count}"/>
>>
>> <result start-time="2025-05-19T22:20:24.288+09:00"
>> end-time="2025-05-19T22:20:24.288+09:00" count="43434"></result>
>>
>> Looks like it's pretty much instantaneous to get 43,434 players. In
>> reality, it took a couple of seconds to display the result.
>>
>>
>> 2025年5月19日(月) 20:12 Alberto Simões <has...@gm...>:
>>
>>> Hello, Michael
>>>
>>> I cannot split them so that I can specify different collection names.
>>> In that case, splitting does not bring any additional value?
>>>
>>> Thanks
>>>
>>> On Mon, May 19, 2025 at 10:25 AM Michael Westbay <
>>> wes...@ja...> wrote:
>>>
>>>> Hi Alberto,
>>>>
>>>> collection("/db/records")/record will match all <record>...</record>
>>>> documents under /db/records and sub-folders (sub-collections?).
>>>>
>>>> If you can organize them by date (year sub-folders), including that in
>>>> the collection parameter will mean less records to search. And all
>>>> sub-folders under that collection will still be included in the XPath
>>>> search.
>>>>
>>>>
>>>>
>>>> 2025年5月19日(月) 17:23 Alberto Simões <has...@gm...>:
>>>>
>>>>> Hello
>>>>>
>>>>> Are there differences in terms of performance between having a large
>>>>> collection (150k docs) with or without a folder structure?
>>>>>
>>>>> I want to treat them as a single collection, but I don't know if it
>>>>> helps to have sub-collections to organise them, or if that is irrelevant to
>>>>> eXist.
>>>>>
>>>>> I appreciate any help you can provide.
>>>>> Alberto
>>>>>
>>>>> --
>>>>> Alberto Simões
>>>>> _______________________________________________
>>>>> Exist-open mailing list
>>>>> Exi...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>>>>
>>>>
>>>>
>>>> --
>>>> Michael Westbay
>>>> Writer/System Administrator
>>>> http://www.japanesebaseball.com/
>>>>
>>>
>>>
>>> --
>>> Alberto Simões
>>>
>>
>>
>> --
>> Michael Westbay
>> Writer/System Administrator
>> http://www.japanesebaseball.com/
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>
>

-- 
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/

Re: [Exist-open] Organizing large collection

[Michael W. <wes...@ja...>]

Hi Joe,

Thank you. It did seem odd that they were both the same instant. And I seem
to recall this coming up many years ago.

2025年5月20日(火) 21:58 Joe Wicentowski <jo...@gm...>:

> Hi Michael,
>
> For calculating execution time, use util:system-dateTime() instead of
> fn:current-dateTime().
>
> From the function documentation for util:system-dateTime():
>
> > Contrary to fn:current-dateTime, this function is not stable, i.e. the
> returned xs:dateTime will change during the evaluation time of a query and
> can be used to measure time differences.
>
> See:
>
>
> https://exist-db.org/exist/apps/fundocs/index.html?q=util:system-dateTime
>
> See also this good explanation from Dannes:
>
>   https://sourceforge.net/p/exist/mailman/message/32809375/
>
> Joe
>
> On Tue, May 20, 2025 at 4:54 AM Michael Westbay <
> wes...@ja...> wrote:
>
>> Hi Alberto,
>>
>> For me, splitting them makes them more manageable when I am going through
>> a given collection with a WebDAV editor.
>>
>> For example, I have a database of baseball players. The XML file for a
>> given player is in the format: "surname-givenname.xml." I sort them
>> under the persons collection as:
>>
>> [image: image.png]
>>
>> Each first letter is divided into two or three letter sub-collections. I
>> try to keep each to around 100 names each, but as the database grows, some
>> have grown as large as 300 names. That usually means that I want to divide
>> it up some more. (The _ collection is for names in Kanji -- Japanese
>> characters.)
>>
>> The reason I break them up is because WebDAV is really slow when there
>> are a lot of files in a single collection. If I only processed the XML
>> files, it wouldn't be an issue. But I often go in and manually edit files,
>> so the hierarchy helps.
>>
>> A quick count of the number of players I have:
>>
>> xquery version "3.0";
>>
>> let $start-time := current-dateTime()
>> let $players := collection('/db/uni/persons')/*:person
>> let $count := count($players)
>> let $end-time := current-dateTime()
>>
>> return <result start-time="{$start-time}" end-time="{$end-time}"
>> count="{$count}"/>
>>
>> <result start-time="2025-05-19T22:20:24.288+09:00"
>> end-time="2025-05-19T22:20:24.288+09:00" count="43434"></result>
>>
>> Looks like it's pretty much instantaneous to get 43,434 players. In
>> reality, it took a couple of seconds to display the result.
>>
>>
>> 2025年5月19日(月) 20:12 Alberto Simões <has...@gm...>:
>>
>>> Hello, Michael
>>>
>>> I cannot split them so that I can specify different collection names.
>>> In that case, splitting does not bring any additional value?
>>>
>>> Thanks
>>>
>>> On Mon, May 19, 2025 at 10:25 AM Michael Westbay <
>>> wes...@ja...> wrote:
>>>
>>>> Hi Alberto,
>>>>
>>>> collection("/db/records")/record will match all <record>...</record>
>>>> documents under /db/records and sub-folders (sub-collections?).
>>>>
>>>> If you can organize them by date (year sub-folders), including that in
>>>> the collection parameter will mean less records to search. And all
>>>> sub-folders under that collection will still be included in the XPath
>>>> search.
>>>>
>>>>
>>>>
>>>> 2025年5月19日(月) 17:23 Alberto Simões <has...@gm...>:
>>>>
>>>>> Hello
>>>>>
>>>>> Are there differences in terms of performance between having a large
>>>>> collection (150k docs) with or without a folder structure?
>>>>>
>>>>> I want to treat them as a single collection, but I don't know if it
>>>>> helps to have sub-collections to organise them, or if that is irrelevant to
>>>>> eXist.
>>>>>
>>>>> I appreciate any help you can provide.
>>>>> Alberto
>>>>>
>>>>> --
>>>>> Alberto Simões
>>>>> _______________________________________________
>>>>> Exist-open mailing list
>>>>> Exi...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>>>>
>>>>
>>>>
>>>> --
>>>> Michael Westbay
>>>> Writer/System Administrator
>>>> http://www.japanesebaseball.com/
>>>>
>>>
>>>
>>> --
>>> Alberto Simões
>>>
>>
>>
>> --
>> Michael Westbay
>> Writer/System Administrator
>> http://www.japanesebaseball.com/
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>
>

-- 
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/

Re: [Exist-open] Organizing large collection

[Joe W. <jo...@gm...>]

Hi Michael,

For calculating execution time, use util:system-dateTime() instead of
fn:current-dateTime().

>From the function documentation for util:system-dateTime():

> Contrary to fn:current-dateTime, this function is not stable, i.e. the
returned xs:dateTime will change during the evaluation time of a query and
can be used to measure time differences.

See:

  https://exist-db.org/exist/apps/fundocs/index.html?q=util:system-dateTime

See also this good explanation from Dannes:

  https://sourceforge.net/p/exist/mailman/message/32809375/

Joe

On Tue, May 20, 2025 at 4:54 AM Michael Westbay <
wes...@ja...> wrote:

> Hi Alberto,
>
> For me, splitting them makes them more manageable when I am going through
> a given collection with a WebDAV editor.
>
> For example, I have a database of baseball players. The XML file for a
> given player is in the format: "surname-givenname.xml." I sort them under
> the persons collection as:
>
> [image: image.png]
>
> Each first letter is divided into two or three letter sub-collections. I
> try to keep each to around 100 names each, but as the database grows, some
> have grown as large as 300 names. That usually means that I want to divide
> it up some more. (The _ collection is for names in Kanji -- Japanese
> characters.)
>
> The reason I break them up is because WebDAV is really slow when there are
> a lot of files in a single collection. If I only processed the XML files,
> it wouldn't be an issue. But I often go in and manually edit files, so the
> hierarchy helps.
>
> A quick count of the number of players I have:
>
> xquery version "3.0";
>
> let $start-time := current-dateTime()
> let $players := collection('/db/uni/persons')/*:person
> let $count := count($players)
> let $end-time := current-dateTime()
>
> return <result start-time="{$start-time}" end-time="{$end-time}"
> count="{$count}"/>
>
> <result start-time="2025-05-19T22:20:24.288+09:00"
> end-time="2025-05-19T22:20:24.288+09:00" count="43434"></result>
>
> Looks like it's pretty much instantaneous to get 43,434 players. In
> reality, it took a couple of seconds to display the result.
>
>
> 2025年5月19日(月) 20:12 Alberto Simões <has...@gm...>:
>
>> Hello, Michael
>>
>> I cannot split them so that I can specify different collection names.
>> In that case, splitting does not bring any additional value?
>>
>> Thanks
>>
>> On Mon, May 19, 2025 at 10:25 AM Michael Westbay <
>> wes...@ja...> wrote:
>>
>>> Hi Alberto,
>>>
>>> collection("/db/records")/record will match all <record>...</record>
>>> documents under /db/records and sub-folders (sub-collections?).
>>>
>>> If you can organize them by date (year sub-folders), including that in
>>> the collection parameter will mean less records to search. And all
>>> sub-folders under that collection will still be included in the XPath
>>> search.
>>>
>>>
>>>
>>> 2025年5月19日(月) 17:23 Alberto Simões <has...@gm...>:
>>>
>>>> Hello
>>>>
>>>> Are there differences in terms of performance between having a large
>>>> collection (150k docs) with or without a folder structure?
>>>>
>>>> I want to treat them as a single collection, but I don't know if it
>>>> helps to have sub-collections to organise them, or if that is irrelevant to
>>>> eXist.
>>>>
>>>> I appreciate any help you can provide.
>>>> Alberto
>>>>
>>>> --
>>>> Alberto Simões
>>>> _______________________________________________
>>>> Exist-open mailing list
>>>> Exi...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>>>
>>>
>>>
>>> --
>>> Michael Westbay
>>> Writer/System Administrator
>>> http://www.japanesebaseball.com/
>>>
>>
>>
>> --
>> Alberto Simões
>>
>
>
> --
> Michael Westbay
> Writer/System Administrator
> http://www.japanesebaseball.com/
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

Re: [Exist-open] Organizing large collection

[Alberto S. <has...@gm...>]

Hello, Michael

Thanks for sharing your use case.
Indeed, it might get useful
Thanks

On Mon, May 19, 2025 at 2:23 PM Michael Westbay <
wes...@ja...> wrote:

> Hi Alberto,
>
> For me, splitting them makes them more manageable when I am going through
> a given collection with a WebDAV editor.
>
> For example, I have a database of baseball players. The XML file for a
> given player is in the format: "surname-givenname.xml." I sort them under
> the persons collection as:
>
> [image: image.png]
>
> Each first letter is divided into two or three letter sub-collections. I
> try to keep each to around 100 names each, but as the database grows, some
> have grown as large as 300 names. That usually means that I want to divide
> it up some more. (The _ collection is for names in Kanji -- Japanese
> characters.)
>
> The reason I break them up is because WebDAV is really slow when there are
> a lot of files in a single collection. If I only processed the XML files,
> it wouldn't be an issue. But I often go in and manually edit files, so the
> hierarchy helps.
>
> A quick count of the number of players I have:
>
> xquery version "3.0";
>
> let $start-time := current-dateTime()
> let $players := collection('/db/uni/persons')/*:person
> let $count := count($players)
> let $end-time := current-dateTime()
>
> return <result start-time="{$start-time}" end-time="{$end-time}"
> count="{$count}"/>
>
> <result start-time="2025-05-19T22:20:24.288+09:00"
> end-time="2025-05-19T22:20:24.288+09:00" count="43434"></result>
>
> Looks like it's pretty much instantaneous to get 43,434 players. In
> reality, it took a couple of seconds to display the result.
>
>
> 2025年5月19日(月) 20:12 Alberto Simões <has...@gm...>:
>
>> Hello, Michael
>>
>> I cannot split them so that I can specify different collection names.
>> In that case, splitting does not bring any additional value?
>>
>> Thanks
>>
>> On Mon, May 19, 2025 at 10:25 AM Michael Westbay <
>> wes...@ja...> wrote:
>>
>>> Hi Alberto,
>>>
>>> collection("/db/records")/record will match all <record>...</record>
>>> documents under /db/records and sub-folders (sub-collections?).
>>>
>>> If you can organize them by date (year sub-folders), including that in
>>> the collection parameter will mean less records to search. And all
>>> sub-folders under that collection will still be included in the XPath
>>> search.
>>>
>>>
>>>
>>> 2025年5月19日(月) 17:23 Alberto Simões <has...@gm...>:
>>>
>>>> Hello
>>>>
>>>> Are there differences in terms of performance between having a large
>>>> collection (150k docs) with or without a folder structure?
>>>>
>>>> I want to treat them as a single collection, but I don't know if it
>>>> helps to have sub-collections to organise them, or if that is irrelevant to
>>>> eXist.
>>>>
>>>> I appreciate any help you can provide.
>>>> Alberto
>>>>
>>>> --
>>>> Alberto Simões
>>>> _______________________________________________
>>>> Exist-open mailing list
>>>> Exi...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>>>
>>>
>>>
>>> --
>>> Michael Westbay
>>> Writer/System Administrator
>>> http://www.japanesebaseball.com/
>>>
>>
>>
>> --
>> Alberto Simões
>>
>
>
> --
> Michael Westbay
> Writer/System Administrator
> http://www.japanesebaseball.com/
>


-- 
Alberto Simões

Re: [Exist-open] Organizing large collection

[Duncan P. <du...@ex...>]

Dear Alberto,

Internally it doesn’t matter to exist. However, practically I find it convenient to have more manageable chunks in sub-collections so that paging for requests become easier, or browsing the collection in your ide of choice. Here it very much makes a difference in that 150k files will degrade performance of any ide should one accidentally open the folder. 

Greetings
Duncan

Sent from my iPad

> On 19. May 2025, at 14:17, exi...@li... wrote:
> 
> Send Exist-open mailing list submissions to
>    exi...@li...
> 
> To subscribe or unsubscribe via the World Wide Web, visit
>    https://lists.sourceforge.net/lists/listinfo/exist-open
> or, via email, send a message with subject or body 'help' to
>    exi...@li...
> 
> You can reach the person managing the list at
>    exi...@li...
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Exist-open digest..."
> 
> 
> Today's Topics:
> 
>   1. Re: Not able to remove a collection (ai...@un...)
>   2. Organizing large collection (Alberto Sim?es)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Sun, 18 May 2025 10:07:08 +0200
> From: ai...@un...
> To: exi...@li...
> Subject: Re: [Exist-open] Not able to remove a collection
> Message-ID:
>    <202...@we...>
> Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes
> 
> that's probably the old bug with utf8 names (issue #681).
> The remedy is to avoid this: create/copy the resources with a  
> compliant name and delete the old one in backup.
> 
> regards
> Peter
> 
> Quoting "Sanil, Seena via Exist-open" <exi...@li...>:
> 
>> Hi,
>> When we try to remove a collection either through the UI or through  
>> calling the rest API we get this error. Is there any other way to  
>> remove the collection?
>> 
>> Please advise.
>> Thanks.
> 
> 
> 
> 
> 
> 
> ------------------------------
> 
> Message: 2
> Date: Sun, 18 May 2025 13:52:00 +0100
> From: Alberto Sim?es <has...@gm...>
> To: eXist DB ML <exi...@li...>
> Subject: [Exist-open] Organizing large collection
> Message-ID:
>    <CAA...@ma...>
> Content-Type: text/plain; charset="utf-8"
> 
> Hello
> 
> Are there differences in terms of performance between having a large
> collection (150k docs) with or without a folder structure?
> 
> I want to treat them as a single collection, but I don't know if it helps
> to have sub-collections to organise them, or if that is irrelevant to eXist.
> 
> I appreciate any help you can provide.
> Alberto
> 
> --
> Alberto Sim?es
> -------------- next part --------------
> An HTML attachment was scrubbed...
> 
> ------------------------------
> 
> 
> 
> ------------------------------
> 
> Subject: Digest Footer
> 
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
> 
> 
> ------------------------------
> 
> End of Exist-open Digest, Vol 228, Issue 8
> ******************************************

Re: [Exist-open] Organizing large collection

[Olaf S. <ol...@ex...>]

Alberto,

> Are there differences in terms of performance between having a large
> collection (150k docs) with or without a folder structure?
>
> I want to treat them as a single collection, but I don't know if it helps
> to have sub-collections to organise them, or if that is irrelevant to eXist.

Currently, we do not recommend to use a single collection of that size,
because that would probably impact performance.

We're already looking into improving this issue.

Regards,
Olaf

Re: [Exist-open] Organizing large collection

[Michael W. <wes...@ja...>]

Hi Alberto,

For me, splitting them makes them more manageable when I am going through a
given collection with a WebDAV editor.

For example, I have a database of baseball players. The XML file for a
given player is in the format: "surname-givenname.xml." I sort them under
the persons collection as:

[image: image.png]

Each first letter is divided into two or three letter sub-collections. I
try to keep each to around 100 names each, but as the database grows, some
have grown as large as 300 names. That usually means that I want to divide
it up some more. (The _ collection is for names in Kanji -- Japanese
characters.)

The reason I break them up is because WebDAV is really slow when there are
a lot of files in a single collection. If I only processed the XML files,
it wouldn't be an issue. But I often go in and manually edit files, so the
hierarchy helps.

A quick count of the number of players I have:

xquery version "3.0";

let $start-time := current-dateTime()
let $players := collection('/db/uni/persons')/*:person
let $count := count($players)
let $end-time := current-dateTime()

return <result start-time="{$start-time}" end-time="{$end-time}"
count="{$count}"/>

<result start-time="2025-05-19T22:20:24.288+09:00"
end-time="2025-05-19T22:20:24.288+09:00" count="43434"></result>

Looks like it's pretty much instantaneous to get 43,434 players. In
reality, it took a couple of seconds to display the result.


2025年5月19日(月) 20:12 Alberto Simões <has...@gm...>:

> Hello, Michael
>
> I cannot split them so that I can specify different collection names.
> In that case, splitting does not bring any additional value?
>
> Thanks
>
> On Mon, May 19, 2025 at 10:25 AM Michael Westbay <
> wes...@ja...> wrote:
>
>> Hi Alberto,
>>
>> collection("/db/records")/record will match all <record>...</record>
>> documents under /db/records and sub-folders (sub-collections?).
>>
>> If you can organize them by date (year sub-folders), including that in
>> the collection parameter will mean less records to search. And all
>> sub-folders under that collection will still be included in the XPath
>> search.
>>
>>
>>
>> 2025年5月19日(月) 17:23 Alberto Simões <has...@gm...>:
>>
>>> Hello
>>>
>>> Are there differences in terms of performance between having a large
>>> collection (150k docs) with or without a folder structure?
>>>
>>> I want to treat them as a single collection, but I don't know if it
>>> helps to have sub-collections to organise them, or if that is irrelevant to
>>> eXist.
>>>
>>> I appreciate any help you can provide.
>>> Alberto
>>>
>>> --
>>> Alberto Simões
>>> _______________________________________________
>>> Exist-open mailing list
>>> Exi...@li...
>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>>
>>
>>
>> --
>> Michael Westbay
>> Writer/System Administrator
>> http://www.japanesebaseball.com/
>>
>
>
> --
> Alberto Simões
>


-- 
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/

Re: [Exist-open] [ANN} Release of eXist-db 6.4.0

[Craig B. <cra...@ma...>]

> On May 14, 2025, at 10:51 AM, Joe Wicentowski <jo...@GM...> wrote:
> We are happy to announce that version 6.4.0 of eXist-db was released last week [2]. 

Thanks to Joe and all the others who contributed to this release.

> - DMG and installers are now signed by eXist Solutions GmbH and

For me on Sequoia, this did not get it through Gatekeeper -- I had to go into Privacy and Security in settings and explicitly allow an exception to open the app.  For future reference, it seems to be very picky about anything at all changing in the bundle _after_ the bundle has been signed and notarized, and there are a number of files that were added or changed:

% codesign --verify --deep --verbose /Applications/eXist-db.app
/Applications/eXist-db.app: a sealed resource is missing or invalid
file added: /Applications/eXist-db.app/Contents/Resources/etc/conf.xml.orig.20250519060515
file added: /Applications/eXist-db.app/Contents/Resources/etc/launcher.properties
file added: /Applications/eXist-db.app/Contents/Resources/logs/urlrewrite.log
file added: /Applications/eXist-db.app/Contents/Resources/logs/expath-repo.log
file added: /Applications/eXist-db.app/Contents/Resources/logs/statistics.log
file added: /Applications/eXist-db.app/Contents/Resources/logs/scheduler.log
file added: /Applications/eXist-db.app/Contents/Resources/logs/profile.log
file added: /Applications/eXist-db.app/Contents/Resources/logs/xmldb.log
file added: /Applications/eXist-db.app/Contents/Resources/logs/xmlrpc.log
file added: /Applications/eXist-db.app/Contents/Resources/logs/2025_05_19.request.log
file added: /Applications/eXist-db.app/Contents/Resources/logs/exist.log
file added: /Applications/eXist-db.app/Contents/Resources/logs/betterform.log
file added: /Applications/eXist-db.app/Contents/Resources/logs/launcher.log
file added: /Applications/eXist-db.app/Contents/Resources/logs/backup.log
file added: /Applications/eXist-db.app/Contents/Resources/logs/ensure-locking.log
file added: /Applications/eXist-db.app/Contents/Resources/logs/restxq.log
file added: /Applications/eXist-db.app/Contents/Resources/logs/ehcache.log
file added: /Applications/eXist-db.app/Contents/Resources/logs/locks.log
file modified: /Applications/eXist-db.app/Contents/Resources/etc/conf.xml


________________________________________
Craig A. Berry

"... getting out of a sonnet is much more
 difficult than getting in."
                 Brad Leithauser

Re: [Exist-open] Organizing large collection

[Alberto S. <has...@gm...>]

Hello, Michael

I cannot split them so that I can specify different collection names.
In that case, splitting does not bring any additional value?

Thanks

On Mon, May 19, 2025 at 10:25 AM Michael Westbay <
wes...@ja...> wrote:

> Hi Alberto,
>
> collection("/db/records")/record will match all <record>...</record>
> documents under /db/records and sub-folders (sub-collections?).
>
> If you can organize them by date (year sub-folders), including that in the
> collection parameter will mean less records to search. And all sub-folders
> under that collection will still be included in the XPath search.
>
>
>
> 2025年5月19日(月) 17:23 Alberto Simões <has...@gm...>:
>
>> Hello
>>
>> Are there differences in terms of performance between having a large
>> collection (150k docs) with or without a folder structure?
>>
>> I want to treat them as a single collection, but I don't know if it helps
>> to have sub-collections to organise them, or if that is irrelevant to eXist.
>>
>> I appreciate any help you can provide.
>> Alberto
>>
>> --
>> Alberto Simões
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>
>
>
> --
> Michael Westbay
> Writer/System Administrator
> http://www.japanesebaseball.com/
>


-- 
Alberto Simões

Re: [Exist-open] Organizing large collection

[Michael W. <wes...@ja...>]

Hi Alberto,

collection("/db/records")/record will match all <record>...</record>
documents under /db/records and sub-folders (sub-collections?).

If you can organize them by date (year sub-folders), including that in the
collection parameter will mean less records to search. And all sub-folders
under that collection will still be included in the XPath search.



2025年5月19日(月) 17:23 Alberto Simões <has...@gm...>:

> Hello
>
> Are there differences in terms of performance between having a large
> collection (150k docs) with or without a folder structure?
>
> I want to treat them as a single collection, but I don't know if it helps
> to have sub-collections to organise them, or if that is irrelevant to eXist.
>
> I appreciate any help you can provide.
> Alberto
>
> --
> Alberto Simões
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>


-- 
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/

Re: [Exist-open] Not able to remove a collection

[Juri L. <ju...@ex...>]

Hello Seena Sanil!

It would help a lot to know the version of the eXist-db instance you are 
working with. To my knowledge a similar issue was reported and fixed 
some time ago. This information, along with operating system and java 
version is generally helpful when posting questions here.

However, theses are additional methods to remove a collection in an 
eXist-db instance that come to my mind:

- evaluating an XQuery calling xmldb:remove in eXide or other means
https://exist-db.org/exist/apps/fundocs/index.html?q=xmldb%3Aremove&action=search&type=name

- using the XML-RPC API eXist-db offers which has a remove collection 
endpoint
   one available command line client is xst 
https://www.npmjs.com/package/@existdb/xst

- the java admin client should also offer a similar functionality

If none of the above work you could

- save all other resources and subcollections next to the one you cannot 
remove

- then remove the parent collection (regsum-ref in your case)


Hope this helps,

Juri Leino


On 16.05.25 15:23, Sanil, Seena via Exist-open wrote:
>
> Hi,
>
> When we try to remove a collection either through the UI or through 
> calling the rest API we get this error. Is there any other way to 
> remove the collection?
>
> Please advise.
>
> Thanks.
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

[Exist-open] Organizing large collection

[Alberto S. <has...@gm...>]

Hello

Are there differences in terms of performance between having a large
collection (150k docs) with or without a folder structure?

I want to treat them as a single collection, but I don't know if it helps
to have sub-collections to organise them, or if that is irrelevant to eXist.

I appreciate any help you can provide.
Alberto

-- 
Alberto Simões