exist-open — Open discussion

Re: [Exist-open] exist-db launcher icons

[John R. <jr...@gm...>]

EXist-db 5.2.0 on Windows 10 Pro 64 bit

For example, if I click on the icon labelled "xforms-tutorial" it opens to
the second screen shot. But let's say for example, I want to change that
file to another one:
http://localhost:8080/exist/apps/xforms-tutorial/FileLinkedFromTheLauncher.xqy
so that it opens to the xquery.
How do I change the uri associated with the icon in the Launcher tray? Is
there a file in eXist-db that I modify?




[image: image.png]

http://localhost:8080/exist/apps/xforms-tutorial/index.html

[image: image.png]

[image: image.png]


On Fri, Oct 30, 2020 at 8:40 AM Joe Wicentowski <jo...@gm...> wrote:

> Hi John,
>
> I'm not sure I'll have the answer, but let's establish a context so others
> might be able to help: Which version of eXist and which OS are you using?
> And could you also please clarify (perhaps with screenshots) what you mean
> by "the icon from the launcher", "the app icon", and "the file"?
>
> Joe
>
> On Fri, Oct 30, 2020 at 11:34 AM John Reed <jr...@gm...> wrote:
>
>> Hello, How can I change the uri of the file of my app at the launcher app
>> so that when I click on the icon from the launcher the app icon opens the
>> file? Thanks John Reed
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>
>

Re: [Exist-open] some EXPath packages are duplicated in public repo

[Claudius T. <cla...@gm...>]

Hi, Joe,

sure, we can work together for this, thanks.

Claudius

On Fri, 30 Oct 2020 at 17:06, Joe Wicentowski <jo...@gm...> wrote:

> Hi Joern,
>
> I agree, this state is not ideal.
>
> A duplicate entry will occur whenever a package's namespace URI changes.
> All of these packages changed from http://expath.org/lib/_ to
> http://expath.org/ns/_.
>
> The change in package namespace also roughly corresponds with a new
> minimum version requirement - from 2.2+ to some higher version. The earlier
> variant is still offered in the public-repo, because it is compatible with
> older versions of eXist than the newer variant. (This version issue is
> clearer in the public-repo than in the package manager, since version
> dependencies are displayed). The duplicates still appear in package manager
> because the older variants do not state a maximum version - so they appear
> to be compatible with all versions of eXist since 2.2.
>
> A partial solution would be for us to issue a new release of each older
> variant *with a maximum version declared*, corresponding to the minimum
> version for the newer variant. That way, the older variant would be
> filtered out of package manager views, since the package manager includes
> its version in its query to the public repo. We would have to pull any
> versions of the older variant that lack this minimum version declaration
> from the public-repo. We could also repair the package metadata for
> datatypes v0.2.0, which appears to be missing the description and abbrev
> fields.
>
> Claudius, do you think we could work together on preparing these new
> versions? I could check them before uploading and remove the earlier
> variants at the public repo.
>
> Joe
>
> p.s. I sent this response earlier this morning, but it appears to have
> bounced back to me because the screenshot I included of the public-repo was
> too large. Sorry if anyone receives a duplicate.
>
> On Fri, Oct 30, 2020 at 8:49 AM Joern Turner <joe...@gm...>
> wrote:
>
>> Goto packagemanager and click 'available' tab.
>>
>> Some EXPath packages occur as duplicates in the list (see attached
>> screenshot)
>>
>> Seems that by changing the underlying name is the problem. For
>> packagemanager this is now (correctly) assumed to be a new lib. IMO the
>> name should be guaranteed to be persistent as it constitutes the identity
>> of the lib. Or - on the other hand you have strong evidence to change it
>> anyway you should at least
>>
>>
>> also change the title.
>> [image: Screenshot_2020-10-30 existdb-dashboard.png]
>> was not sure where to file that issue.
>>
>> Joern
>>
>>
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>


-- 
http://kuberam.ro

Re: [Exist-open] exist-db launcher icons

[Joe W. <jo...@gm...>]

Hi John,

I'm not sure I'll have the answer, but let's establish a context so others
might be able to help: Which version of eXist and which OS are you using?
And could you also please clarify (perhaps with screenshots) what you mean
by "the icon from the launcher", "the app icon", and "the file"?

Joe

On Fri, Oct 30, 2020 at 11:34 AM John Reed <jr...@gm...> wrote:

> Hello, How can I change the uri of the file of my app at the launcher app
> so that when I click on the icon from the launcher the app icon opens the
> file? Thanks John Reed
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

[Exist-open] exist-db launcher icons

[John R. <jr...@gm...>]

Hello, How can I change the uri of the file of my app at the launcher app
so that when I click on the icon from the launcher the app icon opens the
file? Thanks John Reed

Re: [Exist-open] some EXPath packages are duplicated in public repo

[Joe W. <jo...@gm...>]

Hi Joern,

I agree, this state is not ideal.

A duplicate entry will occur whenever a package's namespace URI changes.
All of these packages changed from http://expath.org/lib/_ to
http://expath.org/ns/_.

The change in package namespace also roughly corresponds with a new minimum
version requirement - from 2.2+ to some higher version. The earlier variant
is still offered in the public-repo, because it is compatible with older
versions of eXist than the newer variant. (This version issue is clearer in
the public-repo than in the package manager, since version dependencies are
displayed). The duplicates still appear in package manager because the
older variants do not state a maximum version - so they appear to be
compatible with all versions of eXist since 2.2.

A partial solution would be for us to issue a new release of each older
variant *with a maximum version declared*, corresponding to the minimum
version for the newer variant. That way, the older variant would be
filtered out of package manager views, since the package manager includes
its version in its query to the public repo. We would have to pull any
versions of the older variant that lack this minimum version declaration
from the public-repo. We could also repair the package metadata for
datatypes v0.2.0, which appears to be missing the description and abbrev
fields.

Claudius, do you think we could work together on preparing these new
versions? I could check them before uploading and remove the earlier
variants at the public repo.

Joe

p.s. I sent this response earlier this morning, but it appears to have
bounced back to me because the screenshot I included of the public-repo was
too large. Sorry if anyone receives a duplicate.

On Fri, Oct 30, 2020 at 8:49 AM Joern Turner <joe...@gm...> wrote:

> Goto packagemanager and click 'available' tab.
>
> Some EXPath packages occur as duplicates in the list (see attached
> screenshot)
>
> Seems that by changing the underlying name is the problem. For
> packagemanager this is now (correctly) assumed to be a new lib. IMO the
> name should be guaranteed to be persistent as it constitutes the identity
> of the lib. Or - on the other hand you have strong evidence to change it
> anyway you should at least
>
>
> also change the title.
> [image: Screenshot_2020-10-30 existdb-dashboard.png]
> was not sure where to file that issue.
>
> Joern
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

Re: [Exist-open] some EXPath packages are duplicated in public repo

[Michael W. <wes...@ja...>]

But do some of those only work with eXist 1.x, 2.x, 3.x, 4.x, or 5.x? There
are still a number of legacy systems in use that may need the older
versions.


2020年10月30日(金) 21:49 Joern Turner <joe...@gm...>:

> Goto packagemanager and click 'available' tab.
>
> Some EXPath packages occur as duplicates in the list (see attached
> screenshot)
>
> Seems that by changing the underlying name is the problem. For
> packagemanager this is now (correctly) assumed to be a new lib. IMO the
> name should be guaranteed to be persistent as it constitutes the identity
> of the lib. Or - on the other hand you have strong evidence to change it
> anyway you should at least
>
>
> also change the title.
> [image: Screenshot_2020-10-30 existdb-dashboard.png]
> was not sure where to file that issue.
>
> Joern
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>


-- 
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/

[Exist-open] some EXPath packages are duplicated in public repo

[Joern T. <joe...@gm...>]

Goto packagemanager and click 'available' tab.

Some EXPath packages occur as duplicates in the list (see attached
screenshot)

Seems that by changing the underlying name is the problem. For
packagemanager this is now (correctly) assumed to be a new lib. IMO the
name should be guaranteed to be persistent as it constitutes the identity
of the lib. Or - on the other hand you have strong evidence to change it
anyway you should at least


also change the title.
[image: Screenshot_2020-10-30 existdb-dashboard.png]
was not sure where to file that issue.

Joern

Re: [Exist-open] de-compression in version 3.0 RC2

[Michael W. <wes...@ja...>]

Hi Chris,

[...] I'd prefer to just get the unzipped XML without storing it.
>

Unzipping ODF and other files in eXist 1.x, I find that everything works
best storing the XML file to a collection that has had indexes created for
the specific XML schema to be used. If you want to delete the original XML
file afterwards, that's fine. But while processing it, having it in the
database and indexed allows for many more uses of the data than the one you
may originally have.

-- 
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/

[Exist-open] de-compression in version 3.0 RC2

[Chris W. <kit...@gm...>]

I'm trying to access some Covid data where the API
https://coronavirus.data.gov.uk/ returns zipped XML so I have to unzip the
body of the reply sent in Base64

Does anyone have an example of a simple use of the compression module that
would work in 3.0 - I see the module as shown in the online documentation
has changed significantly.  I'd prefer to just get the unzipped XML without
storing it.

Chris

[Exist-open] Problems calling CollateX via http from exist-db

[Christian W. <cwi...@gm...>]

Dear eXist users,

Here is what I am trying to communicate with collate 
(https://collatex.net) via the http REST interface:

1) Generating a JSON representation of a part of my texts that I want to 
send over to collate

   - this is achieved by constructing an appropriate XML temporary tree 
and then serializing it with the serializer options like this:

serialize(local:collate-request($sid),
  <output:serialization-parameters>
  <output:method>json</output:method>
  <output:media-type>application/json</output:media-type>
</output:serialization-parameters>)

- If I save the result to a JSON file and then use curl to talk to 
collate like this:

curl http://127.0.0.1:7369/collate -H "Content-Type: application/json" 
-X POST --data-binary @97f34abc-1652-4888-8a9b-efe3db12d4dd.json > ret.json

I do get the expected result.

2) However, if I use the EXPATH httpclient (which works fine in other 
applications on this exist-db instance) using

import module namespace http="http://expath.org/ns/http-client";

and then

http:send-request(
<http:request http-version="1.1" href="http://127.0.0.1:7369/collate" 
method="post">
   <http:header name="Content-type" value="application/json"/>
   <http:body media-type="application/json" 
method="text">{$ret}</http:body></http:request>
)

3) Running the above from the oXygen interface will give me the error 
message "[B can not be cast to java.lang.String

4) Running the same from eXide gives me a response,

<hc:response xmlns:hc="http://expath.org/ns/http-client" status="200" 
message="OK" spent-millis="50"><hc:header 
name="access-control-allow-origin" value="*"/><hc:header 
name="access-control-allow-methods" value="GET, POST, HEAD, 
OPTIONS"/><hc:header name="access-control-allow-headers" 
value="Content-Type, Accept, X-Requested-With"/><hc:header 
name="access-control-max-age" value="86400"/><hc:header 
name="access-control-allow-credentials" value="true"/><hc:header 
name="content-type" value="application/json"/><hc:header name="date" 
value="Tue, 20 Oct 2020 06:02:34 GMT"/><hc:header 
name="transfer-encoding" value="chunked"/><hc:body 
media-type="application/json"/></hc:response>

I would expect the result to be the second item in the response 
sequence, but there does not seem to be a second one, eXide announces 
only 1 item as result.

Most likely there is some obvious error in the above, but I am unable to 
figure it out.  Does anybody has an idea what that might be?

All the best,

Christian

Re: [Exist-open] Difference in query-handling and treatment of Booleans eXist 44 vs 52?

[Eduard D. <ed...@fr...>]

And than the dummy question, did you do profiling in monex and looked
at the index and function tab?

-----Original Message-----
From: Espen S. Ore <esp...@gm...>
To: Joe Wicentowski <jo...@gm...>
Cc: exi...@li... <exi...@li...>
Subject: Re: [Exist-open] Difference in query-handling and treatment of
Booleans eXist 44 vs 52?
Date: Thu, 15 Oct 2020 10:52:59 +0200



  
  
    Hi Joe,

    

    Since the computer serving the 4.4 version si being turned off now,
    it is difficult for me to make a comparison. But some experiments
    hint that the problem is not in the union-based queries but in a
    later treatment and sorting of the results- And this leads me back
    to wondering how the sorting and BTrees work in 5.2 - if this is
    different from 4.4. In any case I will retire by the end of the
year
    and leave this unsolved problem for the people taking over :-)

    

    Espen

    

    Espen Ore

    University of Oslo,

    Norway

    

    Den 14.10.2020 16:57, skrev Joe
      Wicentowski:

    

    
>       
>       Hi Espen,
>         
> 
>         
>         I'm not aware of changes in eXist 5.x which would result in
>           a decrease in performance of the union ("|") operator.
> Could
>           you provide a minimal, self-contained test that
> demonstrates
>           the decrease in performance?
>         
> 
>         
>         Joe
>       
>       
> 
>       
>         On Wed, Oct 14, 2020 at 5:13
>           AM Espen S. Ore <esp...@gm...> wrote:
> 
>         
>         
> > Greetings,
> > 
> >           
> > 
> >           I am still trying to figure out why our searches take som
> > much
> >           longer 
> > 
> >           time with eXist 5.2 than with 4.4. And maybe it is
> > because our
> >           way of 
> > 
> >           doing queries with the or ("|") operator has become
> >           innefficient in 5.2 
> > 
> >           while it worked well in 4.4? This is an example of one of
> > the
> >           searches 
> > 
> >           from a rather big xQuery script:
> > 
> >           
> > 
> >           ...
> > 
> >           
> > 
> >           let $scopebrev :=
> > 
> >                (
> > 
> > doc('/his_1.3/ibsen/brev/B1844-
> > 1871ht.xml')|doc('/his_1.3/ibsen/brev/B1871-
> > 1879ht.xml')|doc('/his_1.3/ibsen/brev/B1880-
> > 1889ht.xml')|doc('/his_1.3/ibsen/brev/B1890-1905ht.xml')
> > 
> >                )
> > 
> >           ...
> > 
> >           
> > 
> >           let $search-string :=
> > 
> >             if ($frase eq "NEI") then
> > 
> > concat('$scopebrev//tei:text/descendant::tei:docTitle[ft:query(.,
> > "', 
> > 
> >           $q, '" , 
> > 
> > <options><default-operator>and</default-operator><leading-
> > wildcard>no</leading-
> > wildcard></options>)]|$scopebrev//tei:text/descendant::tei:titlePar
> > t[ft:query(.,
> >           
> > 
> >           "', $q, '" , 
> > 
> > <options><default-operator>and</default-operator><leading-
> > wildcard>no</leading-
> > wildcard></options>)]|$scopebrev//tei:text/descendant::tei:p[ft:que
> > ry(.,
> >           
> > 
> >           "', $q, '" , 
> > 
> > <options><default-operator>and</default-operator><leading-
> > wildcard>no</leading-
> > wildcard></options>)]|$scopebrev//tei:text/descendant::tei:head[ft:
> > query(.,
> >           
> > 
> >           "', $q, '" , 
> > 
> > <options><default-operator>and</default-operator><leading-
> > wildcard>no</leading-
> > wildcard></options>)]|$scopebrev//tei:text/descendant::tei:salute[f
> > t:query(.,
> >           
> > 
> >           "', $q, '" , 
> > 
> > <options><default-operator>and</default-operator><leading-
> > wildcard>no</leading-
> > wildcard></options>)]|$scopebrev//tei:text/descendant::tei:signed[f
> > t:query(.,
> >           
> > 
> >           "', $q, '" , 
> > 
> > <options><default-operator>and</default-operator><leading-
> > wildcard>no</leading-
> > wildcard></options>)]|$scopebrev//tei:text/descendant::his:hisStage
> > [ft:query(.,
> >           
> > 
> >           "', $q, '" , 
> > 
> > <options><default-operator>and</default-operator><leading-
> > wildcard>no</leading-
> > wildcard></options>)]|$scopebrev//tei:text/descendant::tei:dateline
> > [ft:query(.,
> >           
> > 
> >           "', $q, '" , 
> > 
> > <options><default-operator>and</default-operator><leading-
> > wildcard>no</leading-
> > wildcard></options>)]|$scopebrev//tei:text/descendant::tei:l[ft:que
> > ry(.,
> >           
> > 
> >           "', $q, '" , 
> > 
> > <options><default-operator>and</default-operator><leading-
> > wildcard>no</leading-
> > wildcard></options>)]|$scopebrev//tei:text/descendant::tei:cell[ft:
> > query(.,
> >           
> > 
> >           "', $q, '" , 
> > 
> > <options><default-operator>and</default-operator><leading-
> > wildcard>no</leading-
> > wildcard></options>)]|$scopebrev//tei:text/descendant::tei:item[ft:
> > query(.,
> >           
> > 
> >           "', $q, '" , 
> > 
> > <options><default-operator>and</default-operator><leading-
> > wildcard>no</leading-
> > wildcard></options>)]|$scopebrev//tei:text/descendant::his:hisDel[f
> > t:query(.,
> >           
> > 
> >           "', $q, '" , 
> > 
> > <options><default-operator>and</default-operator><leading-
> > wildcard>no</leading-
> > wildcard></options>)]|$scopebrev//tei:text/descendant::his:hisSpOpe
> > ner[ft:query(.,
> >           
> > 
> >           "', $q, '" , 
> > 
> > <options><default-operator>and</default-operator><leading-
> > wildcard>no</leading-wildcard></options>)]')
> > 
> >           
> > 
> >           ...
> > 
> >           
> > 
> >           let $hitsbrev := util:eval($search-string)
> > 
> >           
> > 
> >           ...
> > 
> >           
> > 
> >           Is there something in how this is handled in 5.2 that
> > would
> >           make it more 
> > 
> >           effficient if I organize this kind of search in another
> > way?
> >           And in that 
> > 
> >           case, which way?
> > 
> >           
> > 
> >           Best regards,
> > 
> >           
> > 
> >           Espen Ore
> > 
> >           University of Oslo, Norway
> > 
> >           
> > 
> >           -- 
> > 
> >           Espen S. Ore
> > 
> >           Mobile: (+47) 91390748 / (+30) 695 555 0615
> > 
> >           Ithaca, Greece: (+30) 267 40 31613
> > 
> >           
> > 
> >           
> > 
> >           
> > 
> >           _______________________________________________
> > 
> >           Exist-open mailing list
> > 
> >           Exi...@li...
> > 
> >           https://lists.sourceforge.net/lists/listinfo/exist-open
> > 
> >         
> 
>       
>     

    

    -- Espen S. OreMobile: (+47) 91390748 / (+30) 695 555 0615Ithaca,
Greece: (+30) 267 40 31613
  

_______________________________________________Exist-open mailing 
lis...@li...
https://lists.sourceforge.net/lists/listinfo/exist-open

-- 
Eduard Drenth, Software Architekt

ed...@fr...

Doelestrjitte 8
8911 DX  Ljouwert
+31 58 234 30 47
+31 62 094 34 28 (privé)

skype: eduarddrenth
https://github.com/eduarddrenth
frisian.eu
gpg: https://pgp.surfnet.nl/pks/lookup?search=eduarddrenth


Op freed bin ik thús/wurkje ik minder

Re: [Exist-open] Difference in query-handling and treatment of Booleans eXist 44 vs 52?

[Espen S. O. <esp...@gm...>]

Hi Joe,

Since the computer serving the 4.4 version si being turned off now, it 
is difficult for me to make a comparison. But some experiments hint that 
the problem is not in the union-based queries but in a later treatment 
and sorting of the results- And this leads me back to wondering how the 
sorting and BTrees work in 5.2 - if this is different from 4.4. In any 
case I will retire by the end of the year and leave this unsolved 
problem for the people taking over :-)

Espen

Espen Ore
University of Oslo,
Norway

Den 14.10.2020 16:57, skrev Joe Wicentowski:
> Hi Espen,
>
> I'm not aware of changes in eXist 5.x which would result in a decrease 
> in performance of the union ("|") operator. Could you provide a 
> minimal, self-contained test that demonstrates the decrease in 
> performance?
>
> Joe
>
> On Wed, Oct 14, 2020 at 5:13 AM Espen S. Ore <esp...@gm... 
> <mailto:esp...@gm...>> wrote:
>
>     Greetings,
>
>     I am still trying to figure out why our searches take som much longer
>     time with eXist 5.2 than with 4.4. And maybe it is because our way of
>     doing queries with the or ("|") operator has become innefficient
>     in 5.2
>     while it worked well in 4.4? This is an example of one of the
>     searches
>     from a rather big xQuery script:
>
>     ...
>
>     let $scopebrev :=
>          (
>     doc('/his_1.3/ibsen/brev/B1844-1871ht.xml')|doc('/his_1.3/ibsen/brev/B1871-1879ht.xml')|doc('/his_1.3/ibsen/brev/B1880-1889ht.xml')|doc('/his_1.3/ibsen/brev/B1890-1905ht.xml')
>          )
>     ...
>
>     let $search-string :=
>       if ($frase eq "NEI") then
>     concat('$scopebrev//tei:text/descendant::tei:docTitle[ft:query(., "',
>     $q, '" ,
>     <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:titlePart[ft:query(.,
>
>     "', $q, '" ,
>     <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:p[ft:query(.,
>
>     "', $q, '" ,
>     <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:head[ft:query(.,
>
>     "', $q, '" ,
>     <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:salute[ft:query(.,
>
>     "', $q, '" ,
>     <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:signed[ft:query(.,
>
>     "', $q, '" ,
>     <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::his:hisStage[ft:query(.,
>
>     "', $q, '" ,
>     <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:dateline[ft:query(.,
>
>     "', $q, '" ,
>     <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:l[ft:query(.,
>
>     "', $q, '" ,
>     <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:cell[ft:query(.,
>
>     "', $q, '" ,
>     <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:item[ft:query(.,
>
>     "', $q, '" ,
>     <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::his:hisDel[ft:query(.,
>
>     "', $q, '" ,
>     <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::his:hisSpOpener[ft:query(.,
>
>     "', $q, '" ,
>     <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]')
>
>     ...
>
>     let $hitsbrev := util:eval($search-string)
>
>     ...
>
>     Is there something in how this is handled in 5.2 that would make
>     it more
>     effficient if I organize this kind of search in another way? And
>     in that
>     case, which way?
>
>     Best regards,
>
>     Espen Ore
>     University of Oslo, Norway
>
>     -- 
>     Espen S. Ore
>     Mobile: (+47) 91390748 / (+30) 695 555 0615
>     Ithaca, Greece: (+30) 267 40 31613
>
>
>
>     _______________________________________________
>     Exist-open mailing list
>     Exi...@li...
>     <mailto:Exi...@li...>
>     https://lists.sourceforge.net/lists/listinfo/exist-open
>

-- 
Espen S. Ore
Mobile: (+47) 91390748 / (+30) 695 555 0615
Ithaca, Greece: (+30) 267 40 31613

Re: [Exist-open] Difference in query-handling and treatment of Booleans eXist 44 vs 52?

[Joe W. <jo...@gm...>]

Hi Espen,

I'm not aware of changes in eXist 5.x which would result in a decrease in
performance of the union ("|") operator. Could you provide a minimal,
self-contained test that demonstrates the decrease in performance?

Joe

On Wed, Oct 14, 2020 at 5:13 AM Espen S. Ore <esp...@gm...> wrote:

> Greetings,
>
> I am still trying to figure out why our searches take som much longer
> time with eXist 5.2 than with 4.4. And maybe it is because our way of
> doing queries with the or ("|") operator has become innefficient in 5.2
> while it worked well in 4.4? This is an example of one of the searches
> from a rather big xQuery script:
>
> ...
>
> let $scopebrev :=
>      (
>
> doc('/his_1.3/ibsen/brev/B1844-1871ht.xml')|doc('/his_1.3/ibsen/brev/B1871-1879ht.xml')|doc('/his_1.3/ibsen/brev/B1880-1889ht.xml')|doc('/his_1.3/ibsen/brev/B1890-1905ht.xml')
>      )
> ...
>
> let $search-string :=
>   if ($frase eq "NEI") then
> concat('$scopebrev//tei:text/descendant::tei:docTitle[ft:query(., "',
> $q, '" ,
> <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:titlePart[ft:query(.,
>
> "', $q, '" ,
> <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:p[ft:query(.,
>
> "', $q, '" ,
> <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:head[ft:query(.,
>
> "', $q, '" ,
> <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:salute[ft:query(.,
>
> "', $q, '" ,
> <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:signed[ft:query(.,
>
> "', $q, '" ,
> <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::his:hisStage[ft:query(.,
>
> "', $q, '" ,
> <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:dateline[ft:query(.,
>
> "', $q, '" ,
> <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:l[ft:query(.,
>
> "', $q, '" ,
> <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:cell[ft:query(.,
>
> "', $q, '" ,
> <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:item[ft:query(.,
>
> "', $q, '" ,
> <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::his:hisDel[ft:query(.,
>
> "', $q, '" ,
> <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::his:hisSpOpener[ft:query(.,
>
> "', $q, '" ,
>
> <options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]')
>
> ...
>
> let $hitsbrev := util:eval($search-string)
>
> ...
>
> Is there something in how this is handled in 5.2 that would make it more
> effficient if I organize this kind of search in another way? And in that
> case, which way?
>
> Best regards,
>
> Espen Ore
> University of Oslo, Norway
>
> --
> Espen S. Ore
> Mobile: (+47) 91390748 / (+30) 695 555 0615
> Ithaca, Greece: (+30) 267 40 31613
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

[Exist-open] Difference in query-handling and treatment of Booleans eXist 44 vs 52?

[Espen S. O. <esp...@gm...>]

Greetings,

I am still trying to figure out why our searches take som much longer 
time with eXist 5.2 than with 4.4. And maybe it is because our way of 
doing queries with the or ("|") operator has become innefficient in 5.2 
while it worked well in 4.4? This is an example of one of the searches 
from a rather big xQuery script:

...

let $scopebrev :=
     (
doc('/his_1.3/ibsen/brev/B1844-1871ht.xml')|doc('/his_1.3/ibsen/brev/B1871-1879ht.xml')|doc('/his_1.3/ibsen/brev/B1880-1889ht.xml')|doc('/his_1.3/ibsen/brev/B1890-1905ht.xml')
     )
...

let $search-string :=
  if ($frase eq "NEI") then
concat('$scopebrev//tei:text/descendant::tei:docTitle[ft:query(., "', 
$q, '" , 
<options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:titlePart[ft:query(., 
"', $q, '" , 
<options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:p[ft:query(., 
"', $q, '" , 
<options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:head[ft:query(., 
"', $q, '" , 
<options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:salute[ft:query(., 
"', $q, '" , 
<options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:signed[ft:query(., 
"', $q, '" , 
<options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::his:hisStage[ft:query(., 
"', $q, '" , 
<options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:dateline[ft:query(., 
"', $q, '" , 
<options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:l[ft:query(., 
"', $q, '" , 
<options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:cell[ft:query(., 
"', $q, '" , 
<options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::tei:item[ft:query(., 
"', $q, '" , 
<options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::his:hisDel[ft:query(., 
"', $q, '" , 
<options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]|$scopebrev//tei:text/descendant::his:hisSpOpener[ft:query(., 
"', $q, '" , 
<options><default-operator>and</default-operator><leading-wildcard>no</leading-wildcard></options>)]')

...

let $hitsbrev := util:eval($search-string)

...

Is there something in how this is handled in 5.2 that would make it more 
effficient if I organize this kind of search in another way? And in that 
case, which way?

Best regards,

Espen Ore
University of Oslo, Norway

-- 
Espen S. Ore
Mobile: (+47) 91390748 / (+30) 695 555 0615
Ithaca, Greece: (+30) 267 40 31613

Re: [Exist-open] Memory problems in eXist 5.2?

[Espen S. O. <esp...@gm...>]

An update:

The problem was indeed not a memory leak or something like that. The 
problem seems to be that the xQuery (xql) scripts run extremely much 
slower on the new machine (RHEL8) with eXist 52 than on the old RHEL6 
with eXist 4.4 (and before that 3.x). So with Jetty/eXist running as a 
separate server, but linked to with a ProxyPass/ProxyPassReerse from our 
main server, the main server got a timeout when the limit was set to 60 
seconds as it was on the old machine. I have now set the timeout to 600 
seconds - which is a little more than I like, but at least it doesn't 
crash the connection for timeout reasons.

The system setup is

eXist 5.2.0,
eXist build 20200123133609
Linux 4.18.0-193.6.3.el8_2.x86_64 amd64
Java 1.8.0_265
Total physical memory: 8189665280

For me it seems that the Btrees is where things are working and where 
things may get stuck.

I don't really understand what these values indicate or whether it is 
anything meaningful at all for the xql-speed:

dom.dbx(BTree) Size 729/Used 729
structure.dbx (BTree) Size 5530/Used 5530
collections.dbx (BTree) Size 96/Used 94
values.dbx (BTree) Size 216/Used 169

Espen Ore,
University of Oslo, Norway

-- 
Espen S. Ore
Mobile: (+47) 91390748 / (+30) 695 555 0615
Ithaca, Greece: (+30) 267 40 31613

Re: [Exist-open] Indexed nodes occasionally not accessible?

[Jo C. <Jo....@ha...>]

Hi Joe,

The relevant part of the index is:
<collection xmlns="http://exist-db.org/collection-config/1.0">
    <index xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:object="
http://example.com/object">
        <range>
            <create qname="object:identifier" type="xs:string"/>
        </range>
    </index>
</collection>

(which I think means "new range").  "$object/sec:accessControlList" relies
just on the structural index (no indexes on sec:principal or sec:mode).

The move operation is basically

declare function local:move-previous-version($resource-name)  {
    let $root := "/db/.../root"
    let $current-uri := $root || "/current"
    let $historic-uri :=  $root || "/historic"
    return xmldb:move($current-uri, $historic-uri, $resource-name)
};

(safe in the knowledge that $resource-name is unique across the board).

Best regards, -- Jo


On Mon, Oct 12, 2020 at 3:04 PM Joe Wicentowski <jo...@gm...> wrote:

> Hi Jo,
>
> Strange, indeed. Can you tell us about the index you’ve configured on the
> data collection - old range or new range? What function do you use to move
> the data?
>
> Joe
>
> On Sun, Oct 11, 2020 at 5:59 PM Jo Calder <Jo....@ha...>
> wrote:
>
>> Hi all,
>>
>> I have a large number of resources (tens of thousands) stored with the
>> following structure (or rather, with a structure homomorphic to the
>> following):
>>
>> <wrapper:object
>>         xmlns:wrapper="http://example.com/object-wrapper"
>>         xmlns:object="http://example.com/object">
>>
>> <object:identifier>a5ca528a-ae50-41f5-aef8-2ad4d0d2c350</object:identifier>
>>         <!-- interesting object data here -->
>>      <sec:accessControlList xmlns:sec="http://example.com/security">
>>                <sec:permission>
>>
>>  <sec:principal>76a29694-ecaf-4c0f-a429-df92d00f968f</principal>
>>                    <sec:mode>rwx</mode>
>>                 <sec:permission>
>>      </sec:accessControlList>
>> </wrapper:object>
>>
>> One of these resources, once created, is never updated, but may
>> (occasionally) be moved into a different collection.
>>
>> Most of the time, a function such as the following works exactly as one
>> would expect:
>>
>> declare function local:user-can-read($object-identifier,
>> $user-identifier) {
>>     let $object :=
>> collection($module-specified-uri)/wrapper:object[object:
>> object:identifier
>> identifier=$object-identifier]
>>     return if (empty($object))
>>               then error("object not found")
>>               else if ($object/sec:accessControlList[sec:principal =
>> $user-identifier][contains(sec:mode, "r")])
>>                       then true()
>>                       else false()
>> };
>>
>> However, occasionally and in circumstances I can't immediately identify
>> (sometimes, it seems with my server idle overnight), the above function
>> will return false() not because either of the predicates is not
>> satisfiable, but because the xpath
>>
>> $object/sec:accessControlList
>>
>> evaluates to an empty sequence.  Selecting the same object node and using
>> an operation such as
>>
>> count($object/*[local-name()='accessControlList'])
>>
>> indicates that the relevant nodes are present.  This condition can be
>> fixed by a reindex of the relevant collection.  i.e. before indexing:
>>
>> local:user-can-read( "a5ca528a-ae50-41f5-aef8-2ad4d0d2c350"  ,
>> "76a29694-ecaf-4c0f-a429-df92d00f968f")
>>
>> returns false() and after reindexing it returns true().  Returning to the
>> function definition above (and noting that values of "object:identifier"
>> are indexed in the relevant collection.xconf), I have also seen this
>> symptom in respect of the predicate
>> "[object:identifier=$object-identifier]", with an error being raised
>> incorrectly, again resolved via a reindex.  I have wondered whether the
>> change of namespace might be a factor here.   (The change in namespace is
>> given by the schemas I want to use and is not up for redesign.)  From what
>> I have seen, the loss of the indexes can be partial.  (i.e. a function may
>> correctly return true() or false() for a given $object-identifier, but
>> incorrectly error or return false() on a different $object-identifier).
>>
>> Has anyone else encountered behaviour like this?  If so, any hints as to
>> what operations might cause a problem with indexing or how to investigate
>> further?   Are there any tests that can be performed to check on index
>> health, short of rewriting to circumvent (in effect)  the indexes I want to
>> use?
>>
>> Install details are:
>>
>> eXist Version: 5.0.0
>> eXist Build: 20190902160649
>> Operating System: Linux 3.10.0-1062.12.1.el7.x86_64 amd64
>> Java Version: 1.8.0_242
>>
>> The OS is RHEL 7.7, running as a VM, with 3 G of Java heap space (and
>> with 600 MB available but unused).   I can see some mentions of indexes in
>> the release notes for later versions, but nothing that seems to correspond
>> to the symptoms above.
>>
>> In case this is relevant, here's the Monex report of cache usage:
>>
>> dom.dbx (BTree)
>> Size: 8295 / Used: 8295 / Fails: 415239 / Hits: 128523772
>> structure.dbx (BTree)
>> Size: 12442 / Used: 12442 / Fails: 1819533 / Hits: -1618620458
>> collections.dbx (BTree)
>> Size: 486 / Used: 486 / Fails: 3775 / Hits: 2994660
>> values.dbx (BTree)
>> Size: 64 / Used: 6 / Fails: 6 / Hits: 31122
>>
>> The first three appear to be maxed out, but this doesn't seem to affect
>> functionality.
>>
>> As above, any pointers gratefully received.
>>
>> Regards, -- Jo
>>
>>
>>
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>
> --
> Sent from my iPhone
>


-- 
==================
Garden Cottage
Pitnacree
PH9 0LW
07771 620433

Re: [Exist-open] proxy settings

[Adam R. <ad...@ex...>]

Hi Immanuel,

It depends where your errors come from.


Looking at your log, it looks like you already have eXist installed. So
rebuilding from source will not help you.

Your best bet is to make sure you have all of the Java environment
variables set correctly, eXist-db should use those.


On Fri, 9 Oct 2020, 13:23 Immanuel Normann, <imm...@gm...>
wrote:

> Hi,
> I am obliged to run existdb behind a firewall. It seems that http requests
> from existdb to the outside don't work:
>  2020-10-09 12:44:13,569 [qtp1747862060-44] ERROR (Deploy.java
> [installAndDeploy]:214) - connect timed out
>
> Is there an option to start existdb with appropriate proxy settings?
> Or does it make use of any such environment variables like:
> HTTP_PROXY=http://myproxy-server.com:4711
>
> Or (as last resort) would it help to build existdb with proxy settings
> included in maven like:
> <settings>
> ...
>   <proxies>
>     <proxy>
>       <id>myproxy</id>
>       <active>true</active>
>       <protocol>http</protocol>
>       <host>myproxy-server.com</host>
>       <port>4711</port>
>     </proxy>
>     ...
>   </proxies>
> ...
> </settings>
>
> Has anybody a positive experience with that?
>
> Best
> Immanuel
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

Re: [Exist-open] Indexed nodes occasionally not accessible?

[Joe W. <jo...@gm...>]

Hi Jo,

Strange, indeed. Can you tell us about the index you’ve configured on the
data collection - old range or new range? What function do you use to move
the data?

Joe

On Sun, Oct 11, 2020 at 5:59 PM Jo Calder <Jo....@ha...> wrote:

> Hi all,
>
> I have a large number of resources (tens of thousands) stored with the
> following structure (or rather, with a structure homomorphic to the
> following):
>
> <wrapper:object
>         xmlns:wrapper="http://example.com/object-wrapper"
>         xmlns:object="http://example.com/object">
>
> <object:identifier>a5ca528a-ae50-41f5-aef8-2ad4d0d2c350</object:identifier>
>         <!-- interesting object data here -->
>      <sec:accessControlList xmlns:sec="http://example.com/security">
>                <sec:permission>
>
>  <sec:principal>76a29694-ecaf-4c0f-a429-df92d00f968f</principal>
>                    <sec:mode>rwx</mode>
>                 <sec:permission>
>      </sec:accessControlList>
> </wrapper:object>
>
> One of these resources, once created, is never updated, but may
> (occasionally) be moved into a different collection.
>
> Most of the time, a function such as the following works exactly as one
> would expect:
>
> declare function local:user-can-read($object-identifier, $user-identifier)
> {
>     let $object := collection($module-specified-uri)/wrapper:object[object:
> object:identifier
> identifier=$object-identifier]
>     return if (empty($object))
>               then error("object not found")
>               else if ($object/sec:accessControlList[sec:principal =
> $user-identifier][contains(sec:mode, "r")])
>                       then true()
>                       else false()
> };
>
> However, occasionally and in circumstances I can't immediately identify
> (sometimes, it seems with my server idle overnight), the above function
> will return false() not because either of the predicates is not
> satisfiable, but because the xpath
>
> $object/sec:accessControlList
>
> evaluates to an empty sequence.  Selecting the same object node and using
> an operation such as
>
> count($object/*[local-name()='accessControlList'])
>
> indicates that the relevant nodes are present.  This condition can be
> fixed by a reindex of the relevant collection.  i.e. before indexing:
>
> local:user-can-read( "a5ca528a-ae50-41f5-aef8-2ad4d0d2c350"  ,
> "76a29694-ecaf-4c0f-a429-df92d00f968f")
>
> returns false() and after reindexing it returns true().  Returning to the
> function definition above (and noting that values of "object:identifier"
> are indexed in the relevant collection.xconf), I have also seen this
> symptom in respect of the predicate
> "[object:identifier=$object-identifier]", with an error being raised
> incorrectly, again resolved via a reindex.  I have wondered whether the
> change of namespace might be a factor here.   (The change in namespace is
> given by the schemas I want to use and is not up for redesign.)  From what
> I have seen, the loss of the indexes can be partial.  (i.e. a function may
> correctly return true() or false() for a given $object-identifier, but
> incorrectly error or return false() on a different $object-identifier).
>
> Has anyone else encountered behaviour like this?  If so, any hints as to
> what operations might cause a problem with indexing or how to investigate
> further?   Are there any tests that can be performed to check on index
> health, short of rewriting to circumvent (in effect)  the indexes I want to
> use?
>
> Install details are:
>
> eXist Version: 5.0.0
> eXist Build: 20190902160649
> Operating System: Linux 3.10.0-1062.12.1.el7.x86_64 amd64
> Java Version: 1.8.0_242
>
> The OS is RHEL 7.7, running as a VM, with 3 G of Java heap space (and with
> 600 MB available but unused).   I can see some mentions of indexes in the
> release notes for later versions, but nothing that seems to correspond to
> the symptoms above.
>
> In case this is relevant, here's the Monex report of cache usage:
>
> dom.dbx (BTree)
> Size: 8295 / Used: 8295 / Fails: 415239 / Hits: 128523772
> structure.dbx (BTree)
> Size: 12442 / Used: 12442 / Fails: 1819533 / Hits: -1618620458
> collections.dbx (BTree)
> Size: 486 / Used: 486 / Fails: 3775 / Hits: 2994660
> values.dbx (BTree)
> Size: 64 / Used: 6 / Fails: 6 / Hits: 31122
>
> The first three appear to be maxed out, but this doesn't seem to affect
> functionality.
>
> As above, any pointers gratefully received.
>
> Regards, -- Jo
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>
-- 
Sent from my iPhone

Re: [Exist-open] Memory problems in eXist 5.2?

[Espen S. O. <esp...@gm...>]

I have not managed to find anything in the logs, but I will try Monex.

Espen Ore
University of Oslo

Den 09.10.2020 17:21, skrev Craig Berry:
>
>> On Oct 7, 2020, at 11:19 PM, Espen Ore <esp...@gm...> wrote:
>>
>> Thanks!
>>
>> Acrually I have no proof that the problem is memory although that is the most easy thing to think about.
> My advice would be to stop thinking and start analyzing.  Are there error messages in exist.log or other logs?  When you observe the query running under Monex (available on the Dashboard) does eXist actually run out of memory? Can you get the query to return more rows by forcing garbage collection while it is running? If so, then yes, it may very well be eXist memory consumption.
>
> Have you looked at system-level memory exhaustion problems?  See: <https://upcloud.com/community/tutorials/troubleshoot-linux-memory-issues/>
>
>> The only other one I can think of is some kind of time-out threshold.
> Maybe.  I've seen lots of queries return N results but fail at N + 1 because of a cardinality error or other problem with the data.
>
> There are many other things that can go wrong. Most of them would leave a trace in some log file.
>
>> I am running eXist under jetty as a free standing server, RHEL8. I have checked the startup.sh and somehow thought that the max memory I could allocate was 2048m. I will try to change this to 3g and see if it makes any difference. But this may as you say not be the problem. Do you have any advice on the settings for a longer time before time out?
> Only what I can find by putting "exist-db query timeout" into one of these fancy things called an internet search engine :-).  Since I've already done that I'll just mention that "query-timeout" in conf.xml may be relevant.  See <https://www.exist-db.org/exist/apps/doc/configuration>.
>
>> Best regards,
>> Espen Ore
>> University of Oslo, Norway
>>
>> Espen S. Ore
>> (+47) 913 907 48 / (+30) 695 555 0615
>>
>>> 7. okt. 2020 kl. 21:59 skrev Craig Berry <cra...@ma...>:
>>>
>>> 
>>>
>>>> On Oct 7, 2020, at 6:22 AM, Espen S. Ore <esp...@gm...> wrote:
>>>>
>>>> Good afternoon,
>>>>
>>>> We have moved a set of xml-files and xqueries from eXist 4.4 to eXist 5.2. And we have some problems with what seems to be memory overrun.
>>> What is the specific evidence that tells you it's run out of memory?
>>>
>>>> Is this a known problem? If I set a limit on the number of found nodes I want to treat/sort and set this limit to 250, things work. For 300 they do not.
>>> What OS are you on, how much memory do you have, and how much is allocated to eXist?  The main difference I can think of between 4.x and 5.x is that you can no longer use Jetty settings to control eXist memory allocation.  On Linux, the only method I've found that works is to edit the Java command line in bin/startup.sh so you would have, for example, "-Xms8g -Xmx8g" to allocate 8 gigabytes to eXist.  On macOS, it asks you during installation and records the answers in a properties file in the application bundle.  Don't know how it works on Windows.
>>>
>>>> Our queries are set as 'xquery version "3.0";' - should this be changed?
>>> I think this would only have an effect if you use features that first became available in the XQuery 3.1 standard.
>>>
>>> ________________________________________
>>> Craig A. Berry
>>>
>>> "... getting out of a sonnet is much more
>>> difficult than getting in."
>>>                 Brad Leithauser
>>>
> ________________________________________
> Craig A. Berry
>
> "... getting out of a sonnet is much more
>   difficult than getting in."
>                   Brad Leithauser
>

-- 
Espen S. Ore
Mobile: (+47) 91390748 / (+30) 695 555 0615
Ithaca, Greece: (+30) 267 40 31613

[Exist-open] Indexed nodes occasionally not accessible?

[Jo C. <Jo....@ha...>]

Hi all,

I have a large number of resources (tens of thousands) stored with the
following structure (or rather, with a structure homomorphic to the
following):

<wrapper:object
        xmlns:wrapper="http://example.com/object-wrapper"
        xmlns:object="http://example.com/object">

<object:identifier>a5ca528a-ae50-41f5-aef8-2ad4d0d2c350</object:identifier>
        <!-- interesting object data here -->
     <sec:accessControlList xmlns:sec="http://example.com/security">
               <sec:permission>

 <sec:principal>76a29694-ecaf-4c0f-a429-df92d00f968f</principal>
                   <sec:mode>rwx</mode>
                <sec:permission>
     </sec:accessControlList>
</wrapper:object>

One of these resources, once created, is never updated, but may
(occasionally) be moved into a different collection.

Most of the time, a function such as the following works exactly as one
would expect:

declare function local:user-can-read($object-identifier, $user-identifier) {
    let $object :=
collection($module-specified-uri)/wrapper:object[object:identifier=$object-identifier]
    return if (empty($object))
              then error("object not found")
              else if ($object/sec:accessControlList[sec:principal =
$user-identifier][contains(sec:mode, "r")])
                      then true()
                      else false()
};

However, occasionally and in circumstances I can't immediately identify
(sometimes, it seems with my server idle overnight), the above function
will return false() not because either of the predicates is not
satisfiable, but because the xpath

$object/sec:accessControlList

evaluates to an empty sequence.  Selecting the same object node and using
an operation such as

count($object/*[local-name()='accessControlList'])

indicates that the relevant nodes are present.  This condition can be fixed
by a reindex of the relevant collection.  i.e. before indexing:

local:user-can-read( "a5ca528a-ae50-41f5-aef8-2ad4d0d2c350"  ,
"76a29694-ecaf-4c0f-a429-df92d00f968f")

returns false() and after reindexing it returns true().  Returning to the
function definition above (and noting that values of "object:identifier"
are indexed in the relevant collection.xconf), I have also seen this
symptom in respect of the predicate
"[object:identifier=$object-identifier]", with an error being raised
incorrectly, again resolved via a reindex.  I have wondered whether the
change of namespace might be a factor here.   (The change in namespace is
given by the schemas I want to use and is not up for redesign.)  From what
I have seen, the loss of the indexes can be partial.  (i.e. a function may
correctly return true() or false() for a given $object-identifier, but
incorrectly error or return false() on a different $object-identifier).

Has anyone else encountered behaviour like this?  If so, any hints as to
what operations might cause a problem with indexing or how to investigate
further?   Are there any tests that can be performed to check on index
health, short of rewriting to circumvent (in effect)  the indexes I want to
use?

Install details are:

eXist Version: 5.0.0
eXist Build: 20190902160649
Operating System: Linux 3.10.0-1062.12.1.el7.x86_64 amd64
Java Version: 1.8.0_242

The OS is RHEL 7.7, running as a VM, with 3 G of Java heap space (and with
600 MB available but unused).   I can see some mentions of indexes in the
release notes for later versions, but nothing that seems to correspond to
the symptoms above.

In case this is relevant, here's the Monex report of cache usage:

dom.dbx (BTree)
Size: 8295 / Used: 8295 / Fails: 415239 / Hits: 128523772
structure.dbx (BTree)
Size: 12442 / Used: 12442 / Fails: 1819533 / Hits: -1618620458
collections.dbx (BTree)
Size: 486 / Used: 486 / Fails: 3775 / Hits: 2994660
values.dbx (BTree)
Size: 64 / Used: 6 / Fails: 6 / Hits: 31122

The first three appear to be maxed out, but this doesn't seem to affect
functionality.

As above, any pointers gratefully received.

Regards, -- Jo

Re: [Exist-open] Memory problems in eXist 5.2?

[Dannes W. <di...@ex...>]

What java version?

> On 9 Oct 2020, at 17:22, Craig Berry via Exist-open <Exi...@li...> wrote:
> 
> 
> 
>> On Oct 7, 2020, at 11:19 PM, Espen Ore <esp...@gm...> wrote:
>> 
>> Thanks!
>> 
>> Acrually I have no proof that the problem is memory although that is the most easy thing to think about.
> 
> My advice would be to stop thinking and start analyzing.  Are there error messages in exist.log or other logs?  When you observe the query running under Monex (available on the Dashboard) does eXist actually run out of memory? Can you get the query to return more rows by forcing garbage collection while it is running? If so, then yes, it may very well be eXist memory consumption.
> 
> Have you looked at system-level memory exhaustion problems?  See: <https://upcloud.com/community/tutorials/troubleshoot-linux-memory-issues/>
> 
>> The only other one I can think of is some kind of time-out threshold.
> 
> Maybe.  I've seen lots of queries return N results but fail at N + 1 because of a cardinality error or other problem with the data.
> 
> There are many other things that can go wrong. Most of them would leave a trace in some log file.
> 
>> I am running eXist under jetty as a free standing server, RHEL8. I have checked the startup.sh and somehow thought that the max memory I could allocate was 2048m. I will try to change this to 3g and see if it makes any difference. But this may as you say not be the problem. Do you have any advice on the settings for a longer time before time out?
> 
> Only what I can find by putting "exist-db query timeout" into one of these fancy things called an internet search engine :-).  Since I've already done that I'll just mention that "query-timeout" in conf.xml may be relevant.  See <https://www.exist-db.org/exist/apps/doc/configuration>.
> 
>> 
>> Best regards,
>> Espen Ore
>> University of Oslo, Norway
>> 
>> Espen S. Ore
>> (+47) 913 907 48 / (+30) 695 555 0615
>> 
>>>> 7. okt. 2020 kl. 21:59 skrev Craig Berry <cra...@ma...>:
>>> 
>>> 
>>> 
>>>> On Oct 7, 2020, at 6:22 AM, Espen S. Ore <esp...@gm...> wrote:
>>>> 
>>>> Good afternoon,
>>>> 
>>>> We have moved a set of xml-files and xqueries from eXist 4.4 to eXist 5.2. And we have some problems with what seems to be memory overrun. 
>>> 
>>> What is the specific evidence that tells you it's run out of memory?
>>> 
>>>> Is this a known problem? If I set a limit on the number of found nodes I want to treat/sort and set this limit to 250, things work. For 300 they do not.
>>> 
>>> What OS are you on, how much memory do you have, and how much is allocated to eXist?  The main difference I can think of between 4.x and 5.x is that you can no longer use Jetty settings to control eXist memory allocation.  On Linux, the only method I've found that works is to edit the Java command line in bin/startup.sh so you would have, for example, "-Xms8g -Xmx8g" to allocate 8 gigabytes to eXist.  On macOS, it asks you during installation and records the answers in a properties file in the application bundle.  Don't know how it works on Windows.
>>> 
>>>> Our queries are set as 'xquery version "3.0";' - should this be changed?
>>> 
>>> I think this would only have an effect if you use features that first became available in the XQuery 3.1 standard.
>>> 
>>> ________________________________________
>>> Craig A. Berry
>>> 
>>> "... getting out of a sonnet is much more
>>> difficult than getting in."
>>>               Brad Leithauser
>>> 
> 
> ________________________________________
> Craig A. Berry
> 
> "... getting out of a sonnet is much more
> difficult than getting in."
>                 Brad Leithauser
> 
> 
> 
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] Memory problems in eXist 5.2?

[Craig B. <cra...@ma...>]

> On Oct 7, 2020, at 11:19 PM, Espen Ore <esp...@gm...> wrote:
> 
> Thanks!
> 
> Acrually I have no proof that the problem is memory although that is the most easy thing to think about.

My advice would be to stop thinking and start analyzing.  Are there error messages in exist.log or other logs?  When you observe the query running under Monex (available on the Dashboard) does eXist actually run out of memory? Can you get the query to return more rows by forcing garbage collection while it is running? If so, then yes, it may very well be eXist memory consumption.

Have you looked at system-level memory exhaustion problems?  See: <https://upcloud.com/community/tutorials/troubleshoot-linux-memory-issues/>

> The only other one I can think of is some kind of time-out threshold.

Maybe.  I've seen lots of queries return N results but fail at N + 1 because of a cardinality error or other problem with the data.

There are many other things that can go wrong. Most of them would leave a trace in some log file.

> I am running eXist under jetty as a free standing server, RHEL8. I have checked the startup.sh and somehow thought that the max memory I could allocate was 2048m. I will try to change this to 3g and see if it makes any difference. But this may as you say not be the problem. Do you have any advice on the settings for a longer time before time out?

Only what I can find by putting "exist-db query timeout" into one of these fancy things called an internet search engine :-).  Since I've already done that I'll just mention that "query-timeout" in conf.xml may be relevant.  See <https://www.exist-db.org/exist/apps/doc/configuration>.

> 
> Best regards,
> Espen Ore
> University of Oslo, Norway
> 
> Espen S. Ore
> (+47) 913 907 48 / (+30) 695 555 0615
> 
>> 7. okt. 2020 kl. 21:59 skrev Craig Berry <cra...@ma...>:
>> 
>> 
>> 
>>> On Oct 7, 2020, at 6:22 AM, Espen S. Ore <esp...@gm...> wrote:
>>> 
>>> Good afternoon,
>>> 
>>> We have moved a set of xml-files and xqueries from eXist 4.4 to eXist 5.2. And we have some problems with what seems to be memory overrun. 
>> 
>> What is the specific evidence that tells you it's run out of memory?
>> 
>>> Is this a known problem? If I set a limit on the number of found nodes I want to treat/sort and set this limit to 250, things work. For 300 they do not.
>> 
>> What OS are you on, how much memory do you have, and how much is allocated to eXist?  The main difference I can think of between 4.x and 5.x is that you can no longer use Jetty settings to control eXist memory allocation.  On Linux, the only method I've found that works is to edit the Java command line in bin/startup.sh so you would have, for example, "-Xms8g -Xmx8g" to allocate 8 gigabytes to eXist.  On macOS, it asks you during installation and records the answers in a properties file in the application bundle.  Don't know how it works on Windows.
>> 
>>> Our queries are set as 'xquery version "3.0";' - should this be changed?
>> 
>> I think this would only have an effect if you use features that first became available in the XQuery 3.1 standard.
>> 
>> ________________________________________
>> Craig A. Berry
>> 
>> "... getting out of a sonnet is much more
>> difficult than getting in."
>>                Brad Leithauser
>> 

________________________________________
Craig A. Berry

"... getting out of a sonnet is much more
 difficult than getting in."
                 Brad Leithauser

[Exist-open] proxy settings

[Immanuel N. <imm...@gm...>]

Hi,
I am obliged to run existdb behind a firewall. It seems that http requests
from existdb to the outside don't work:
 2020-10-09 12:44:13,569 [qtp1747862060-44] ERROR (Deploy.java
[installAndDeploy]:214) - connect timed out

Is there an option to start existdb with appropriate proxy settings?
Or does it make use of any such environment variables like:
HTTP_PROXY=http://myproxy-server.com:4711

Or (as last resort) would it help to build existdb with proxy settings
included in maven like:
<settings>
...
  <proxies>
    <proxy>
      <id>myproxy</id>
      <active>true</active>
      <protocol>http</protocol>
      <host>myproxy-server.com</host>
      <port>4711</port>
    </proxy>
    ...
  </proxies>
...
</settings>

Has anybody a positive experience with that?

Best
Immanuel

Re: [Exist-open] Outputting JSON from XQuery

[Alasdair D. <ala...@gm...>]

Hi Micheal,

Thanks for the reply.

Yes, I figured that would be the issue.  Back to JavaScript files and all the fun that I tails.

Alasdair

Sent from my iPhone

> On 9 Oct 2020, at 1:18 pm, Michael Westbay <wes...@ja...> wrote:
> 
> 
> Alasdair,
> 
> You have double quotes within the attribute. The serializer properly escapes them to "&#34;". That is all working as expected.
> 
> It appears that you want the serializer to maintain the single quotes to contain the attribute value and not escape the double quotes within the value. That will require fundamental changes to the serializer. And depending on how you implement it, expect ALL attributes to function that way.
> 
> I assume you want to do this for some JavaScript framework to use these special attributes to transform the output rather than use CSS. If so, the framework should know to interpret the &#34; as " when it gets the attribute value. That the way XML and HTML attributes work.
> 
> I've tried to make eXist XHTML output fit into JavaScript frameworks in the past. For those projects, I've come around to changing my approach to using a normal HTML file that the framework wants and fetching the data only from eXist rather than generate the XHTML within eXist (which I would prefer to do). Some JavaScript frameworks just don't play well with XHTML.
> 
> Hope this helps.
> 
> 
> 2020年10月9日(金) 11:25 Alasdair Dougall <ala...@gm...>:
>> Hi All,
>> 
>> Not the JSON serialisation, as such, but a tricky problem to do with setting a non standard div tag.
>> 
>> Currently trying to output the following:
>> 
>> <div class="row" data-masonry='{"percentPosition": true }'>
>>  .
>>  .
>> </div>
>> 
>> Notice the unusual construction. 
>> 
>> I have tried:
>> 
>> let $masonryDataTag := '{"percentPosition" : true }'
>> return
>>     <div data-masonary='{$masonryDataTag}'>
>>     </div>
>> 
>> Which returns:
>> 
>>     <div data-masonary="{&#34;percnetPosition&#34; : true }"/>
>> 
>> Has anyone got any ideas?
>> 
>> Alasdair
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
> 
> 
> -- 
> Michael Westbay
> Writer/System Administrator
> http://www.japanesebaseball.com/

Re: [Exist-open] Outputting JSON from XQuery

[Michael W. <wes...@ja...>]

Alasdair,

You have double quotes within the attribute. The serializer properly
escapes them to """. That is all working as expected.

It appears that you want the serializer to maintain the single quotes to
contain the attribute value and not escape the double quotes within the
value. That will require fundamental changes to the serializer. And
depending on how you implement it, expect ALL attributes to function that
way.

I assume you want to do this for some JavaScript framework to use these
special attributes to transform the output rather than use CSS. If so, the
framework should know to interpret the " as " when it gets the
attribute value. That the way XML and HTML attributes work.

I've tried to make eXist XHTML output fit into JavaScript frameworks in the
past. For those projects, I've come around to changing my approach to using
a normal HTML file that the framework wants and fetching the data only from
eXist rather than generate the XHTML within eXist (which I would prefer to
do). Some JavaScript frameworks just don't play well with XHTML.

Hope this helps.


2020年10月9日(金) 11:25 Alasdair Dougall <ala...@gm...>:

> Hi All,
>
> Not the JSON serialisation, as such, but a tricky problem to do with
> setting a non standard div tag.
>
> Currently trying to output the following:
>
> <div class="row" data-masonry='{"percentPosition": true }'>
>  .
>  .
> </div>
>
> Notice the unusual construction.
>
> I have tried:
>
> let $masonryDataTag := '{"percentPosition" : true }'
> return
>     <div data-masonary='{$masonryDataTag}'>
>     </div>
>
> Which returns:
>
>     <div data-masonary="{&#34;percnetPosition&#34; : true }"/>
>
> Has anyone got any ideas?
>
> Alasdair
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>


-- 
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/