Re: [Exist-open] Organizing large collection

[Michael W. <wes...@ja...>]

Hi Joe,

Thank you. It did seem odd that they were both the same instant. And I seem
to recall this coming up many years ago.

2025年5月20日(火) 21:58 Joe Wicentowski <jo...@gm...>:

> Hi Michael,
>
> For calculating execution time, use util:system-dateTime() instead of
> fn:current-dateTime().
>
> From the function documentation for util:system-dateTime():
>
> > Contrary to fn:current-dateTime, this function is not stable, i.e. the
> returned xs:dateTime will change during the evaluation time of a query and
> can be used to measure time differences.
>
> See:
>
>
> https://exist-db.org/exist/apps/fundocs/index.html?q=util:system-dateTime
>
> See also this good explanation from Dannes:
>
>   https://sourceforge.net/p/exist/mailman/message/32809375/
>
> Joe
>
> On Tue, May 20, 2025 at 4:54 AM Michael Westbay <
> wes...@ja...> wrote:
>
>> Hi Alberto,
>>
>> For me, splitting them makes them more manageable when I am going through
>> a given collection with a WebDAV editor.
>>
>> For example, I have a database of baseball players. The XML file for a
>> given player is in the format: "surname-givenname.xml." I sort them
>> under the persons collection as:
>>
>> [image: image.png]
>>
>> Each first letter is divided into two or three letter sub-collections. I
>> try to keep each to around 100 names each, but as the database grows, some
>> have grown as large as 300 names. That usually means that I want to divide
>> it up some more. (The _ collection is for names in Kanji -- Japanese
>> characters.)
>>
>> The reason I break them up is because WebDAV is really slow when there
>> are a lot of files in a single collection. If I only processed the XML
>> files, it wouldn't be an issue. But I often go in and manually edit files,
>> so the hierarchy helps.
>>
>> A quick count of the number of players I have:
>>
>> xquery version "3.0";
>>
>> let $start-time := current-dateTime()
>> let $players := collection('/db/uni/persons')/*:person
>> let $count := count($players)
>> let $end-time := current-dateTime()
>>
>> return <result start-time="{$start-time}" end-time="{$end-time}"
>> count="{$count}"/>
>>
>> <result start-time="2025-05-19T22:20:24.288+09:00"
>> end-time="2025-05-19T22:20:24.288+09:00" count="43434"></result>
>>
>> Looks like it's pretty much instantaneous to get 43,434 players. In
>> reality, it took a couple of seconds to display the result.
>>
>>
>> 2025年5月19日(月) 20:12 Alberto Simões <has...@gm...>:
>>
>>> Hello, Michael
>>>
>>> I cannot split them so that I can specify different collection names.
>>> In that case, splitting does not bring any additional value?
>>>
>>> Thanks
>>>
>>> On Mon, May 19, 2025 at 10:25 AM Michael Westbay <
>>> wes...@ja...> wrote:
>>>
>>>> Hi Alberto,
>>>>
>>>> collection("/db/records")/record will match all <record>...</record>
>>>> documents under /db/records and sub-folders (sub-collections?).
>>>>
>>>> If you can organize them by date (year sub-folders), including that in
>>>> the collection parameter will mean less records to search. And all
>>>> sub-folders under that collection will still be included in the XPath
>>>> search.
>>>>
>>>>
>>>>
>>>> 2025年5月19日(月) 17:23 Alberto Simões <has...@gm...>:
>>>>
>>>>> Hello
>>>>>
>>>>> Are there differences in terms of performance between having a large
>>>>> collection (150k docs) with or without a folder structure?
>>>>>
>>>>> I want to treat them as a single collection, but I don't know if it
>>>>> helps to have sub-collections to organise them, or if that is irrelevant to
>>>>> eXist.
>>>>>
>>>>> I appreciate any help you can provide.
>>>>> Alberto
>>>>>
>>>>> --
>>>>> Alberto Simões
>>>>> _______________________________________________
>>>>> Exist-open mailing list
>>>>> Exi...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>>>>
>>>>
>>>>
>>>> --
>>>> Michael Westbay
>>>> Writer/System Administrator
>>>> http://www.japanesebaseball.com/
>>>>
>>>
>>>
>>> --
>>> Alberto Simões
>>>
>>
>>
>> --
>> Michael Westbay
>> Writer/System Administrator
>> http://www.japanesebaseball.com/
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>
>

-- 
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/