[Exist-open] Struggles with indexing

[Grant V. <gra...@xc...>]

Hi,
(I'm sorry if this is a double post)

I am new to eXist-db. However, I have been working with XML itself almost
since its inception in 1997. My past database experience has been with
Oracle XDB.

My current project requires an XML database for storing a small set of very
large documents. For this project, based on input I have received, I've
been looking at eXist-db as a better fit.

I have been struggling to get the indexing mechanism withing eXist to work.
What I have discovered in my efforts to get this to work is concerning me
greatly. The mistakes I am seeing are something I would often see a dozen
years ago with novice XML developers new to XML namespaces. I would not
expect to see them in something as mature and established as eXist-db would
seem to be.

My reference point is this document:

http://exist-db.org/exist/apps/doc/indexing.xml?q=indexing&field=all&id=D2.2.6#D2.2.6

I'll just start enumerating my issues:

1) The samples in the document don't validate in eXide. The reason is that
the the @type attribute is defined as an xs:QName in the schema. Therefore,
the values "xs:string" and "xs:integer" have to have an associated
namespace declaration for the xs prefix. This declaration is missing from
the collection.xconf samples

2) I modified the examples by defining the xs namespace prefix as xmlns:xs="
http://www.w3.org/2001/XMLSchema". This fixed the validation error.
However, as I customarily use the prefix "xsd" for XML Schema, I thought I
would change the example to match my convention. This is when I discovered
that the indexing config is hardwired to the xs prefix instead of actually
using the namespace URI. This means that xmlns:xsd="
http://www.w3.org/2001/XMLSchema" (not the "xsd" prefix) won't work whole
xmlns:xs="nonsense" will work, even with a bogus namespace URI. This is
quite concerning as it is a novice error in XML.

3) Even when I worked around the schema namespace issues, I still could not
get indexing to work with the documented examples. In tinkering around, I
further discovered that the samples set the default namespace to "
http://exist-db.org/collection-config/1.0". This seemed to hide the
unqualified element names in all the qname attributes. I'm assuming they're
being associated with the
http://exist-db.org/collection-config/1.0namespace instead of not
having a namespace. To remedy this, I changed the
default namespace to instead use the cfg prefix (xmlns:cfg="
http://exist-db.org/collection-config/1.0") and was able to get the
indexing to work. It was surprising to find that the documented examples
weren't usable.

4) As I use namespaces in my documents, I next tried to introduce
namespaces to the indexing. As per the instructions, I declared my
namespace on the <index> (now <cfg:index>) element. I tried to use my
namespace as the default namespace - <cfg:index xmlns="
http://xml.house.gov/schemas/uslm/1.0">. When I tried this, the indexing
did not work.

5) Instead, I introduced a "uslm" prefix in my namespace declarations in
the collection.xconf file and now indexing element names worked okay. But
attribute names, defined as in "<cfg:create qname="@value"
type="xs:integer"/>" did not work okay.

Aside: The eXist-db documentation refers to "xmlns" attributes. They're not
attributes - this is erroneous. They are declarations. They just happen to
have an appearance similar to attributes.

6) My end goal is to be able to index attributes in my "uslm" instance
documents. Three attributes in particular are important to me - @id,
@identifier, and @value. However, nothing I have tried has been able to
index these attributes. I tried using the "uslm" prefix - it didn't work. I
tried adding a secondary declaration of the "uslm" namespace, as the
default namespace - it didn't work. At this point, I'm guessing it is
impossible to index attributes if your instance document is defined with
namespaces. Is there something I haven't tried?

This is my test data:

<items xmlns="http://www.xcential.com/test">
    <item n="1">
        <name>Low Bookcase</name>
        <price>199.99</price>
    </item>
    <item n="2">
        <name>Tall Bookcase</name>
        <price>199.99</price>
    </item>
    <item n="3">
        <name>Desk</name>
        <price>299.99</price>
    </item>
    <item n="4">
        <name>Bed</name>
        <price>399.99</price>
    </item>
</items>

This is my collection.xconf file:

<cfg:collection xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:cfg="
http://exist-db.org/collection-config/1.0">
    <cfg:index xmlns:test="http://www.xcential.com/test">
        <cfg:create qname="test:name" type="xs:string"/>
        <cfg:create qname="test:price" type="xs:decimal"/>
        <cfg:create qname="@n" type="xs:integer"/>
    </cfg:index>
</cfg:collection>

And this is my test:

xquery version "3.0";
declare default element namespace "http://www.xcential.com/test";
declare function local:term-callback($term as xs:string, $data as
xs:integer+)
as element() {
   <entry xmlns="result">
     <term>{$term}</term>
     <frequency>{$data[1]}</frequency>
     <documents>{$data[2]}</documents>
     <position>{$data[3]}</position>
   </entry>
};
util:index-keys(collection('/db/test')//@n, "",
util:function(xs:QName("local:term-callback"), 2), 1000)
(: util:index-keys(collection('/db/test')//name, "",
util:function(xs:QName("local:term-callback"), 2), 1000) :)

Any help will be really appreciated. I've spent far too much time on this
already.

Regards,
   Grant Vergottini
____________________________________________________________________
Grant Vergottini
Xcential Group, LLC.
email: gra...@xc...
phone: 858.361.6738