Re: [Exist-open] Single resource vs xcollection of multipleresources, performance question

[Thomas W. <tho...@gm...>]

Adam and Wolfgang,

I think I need to add some more details to clarify my question.

I have a system where for each client I have a collection with a couple of 
XML files with client's data and one small (<1kB) XML file,  for every image 
this client uploads (images are saved in different collection). A typical 
case is to have 50-500 images, in rear cases we may have more then 2500 
images. My initial question was: If I combine all these small files in a 
single file instead of loading all of them with xcollection, will be there 
any performance benefit.

I realized, the number of the files is going to grow very quickly and some 
other questions came into my mind.

The client's data has the following structure:
There is a collection for each country and for each client there is a 
sub-collection as described above and a summary file in the country 
collection. The number of clients is expected to grow up to 200,000 where 
some of the countries may have 50,000+ clients.
In this context, my new questions are:

1. We know that when we have thousands of files in a single collection 
adding/deletion of files is slow because of the update of the collection 
indexes. Is it equally slow when insert/update/delete sub-nodes in any of 
the files in this collection?

2. If all thousands of summary files are combined into a single file, do we 
still have a slow insert/update/delete for sub-nodes. That is limiting - the 
total number of nodes in a collection or switching the indexes from one file 
to another file when update the indexes?

3. If we have a collection with thousands of sub-collections, is it going to 
be slow to add/remove a new sub-collection?

Kind regards,
Thomas


"Adam Retter" <ad...@ex...> wrote in message 
news:357...@ma......
>I think (I may be wrong) that the only performance issue around
> collections is when you wish to remove a single document from a
> collection that has many thousand documents - at that point document
> removal is slow.
>
> You should try and break up your documents into collections in a
> logical manner, collections can allow much more efficient searching by
> using them as a mechanism to reduce the input set to a query.
>
> 2009/10/26 Thomas White <tho...@gm...>:
>> I have a number of small XML resources located in a collection. While the
>> single resources are created and updated individually, the data from this
>> collection is always used as a whole - I use xcollection to load all 
>> parts.
>> Will be there any performance benefit if I combine all small pieces in a
>> single resource, compared to loading the data with xcollection command? 
>> What
>> about the way resources are cached in the internal eXist buffers - a 
>> single
>> file in one case, compared with 50 small files in the other.
>>
>> Thomas
>>
>>
>
> -- 
> Adam Retter
>
> eXist Developer
> { United Kingdom }
> ad...@ex...
> irc://irc.freenode.net/existdb