exist-open — Open discussion

Re: [Exist-open] eXist-db does not see /db/apps

[Florian S. <ml-...@fl...>]

Nick,

i'm not sure how eXist handles users and permissions internally. But
maybe your current admin user still has no r/w/x access to all the
resources of your db. Did you check the permissions using the Java
Admin client? If the "missing" collections are r/w/x only to your
former admin user, you may have to modify access rights, setting your
current admin user as owner (or alternatively, grant r/w/x to the dba
group).

Yours, Florian

Am 05.12.20 um 20:20 schrieb Nick Sincaglia:
> Florian,
>     I think you might be right. When my system crashed, my admin login
> stopped working. I had a backup admin login that I was able to use to
> login and I recreated my original admin user.
>     But still, where do I from here?
> 
> On 12/5/20 2:27 AM, Florian Schmitt wrote:
>> Maybe missing permissions? As which user did you try to reindex / backup?
>>
>> HTH
>> Florian
>>

Re: [Exist-open] eXist-db does not see /db/apps

[Nick S. <nsi...@nu...>]

Florian,
     I think you might be right. When my system crashed, my admin login 
stopped working. I had a backup admin login that I was able to use to 
login and I recreated my original admin user.
     But still, where do I from here?

On 12/5/20 2:27 AM, Florian Schmitt wrote:
> Maybe missing permissions? As which user did you try to reindex / backup?
>
> HTH
> Florian
>
> Am 04.12.20 um 19:57 schrieb Nick Sincaglia:
>> I am running eXist-db v4.5.0. Recently, I discovered the system was
>> not responsive. I shut it down and started it back up after a little
>> effort. I am not sure what exactly happened that caused it stop being
>> responsive.
>>
>> Once I was able to get it up and running again, I noticed that a
>> number of my scripts were failing. I then tried to re-index the system.
>> let $x := xmldb:reindex('/db/')
>> return $x
>>
>> This returns very quickly with true().
>> However, if I try to reindex '/db/apps' it returns false()
>>
>> My conclusion is that eXist-db does not able to see /db/apps. I then
>> took a backup of the system. When I look at the backup log, a lot of
>> subfolders in apps did not get backed up.
>>
>> I am not sure what to do from here. I have another backup but it is a
>> bit older. If there is any way to recover the current system, I would
>> prefer that.
>>
>> Does anyone have any suggestions on what might be the next steps I
>> should try?
>>
>> Nick
>>
>> eXist Version: 	4.5.0
>> eXist Build: 	201901142136
>> Operating System: 	Linux 4.15.0-74-generic amd64
>> Java Version: 	11.0.9.1
>> Default Encoding: 	UTF8
>> Instance ID: 	exist
>> System CPU Load: 	0.00429185
>> Process CPU Load: 	0.00854701
>> Free Physical Memory: 	232964096
>> Total Physical Memory: 	8164757504
>>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

-- 
Nick Sincaglia
President/Founder
NueMeta, LLC
Digital Media & Technology
Phone: +1-630-303-7035
nsi...@nu...
http://www.nuemeta.com
Skype: nsincaglia

Re: [Exist-open] eXist-db does not see /db/apps

[Florian S. <ml-...@fl...>]

Maybe missing permissions? As which user did you try to reindex / backup?

HTH
Florian

Am 04.12.20 um 19:57 schrieb Nick Sincaglia:
> I am running eXist-db v4.5.0. Recently, I discovered the system was
> not responsive. I shut it down and started it back up after a little
> effort. I am not sure what exactly happened that caused it stop being
> responsive.
> 
> Once I was able to get it up and running again, I noticed that a
> number of my scripts were failing. I then tried to re-index the system.
> let $x := xmldb:reindex('/db/')
> return $x
> 
> This returns very quickly with true().
> However, if I try to reindex '/db/apps' it returns false()
> 
> My conclusion is that eXist-db does not able to see /db/apps. I then
> took a backup of the system. When I look at the backup log, a lot of
> subfolders in apps did not get backed up.
> 
> I am not sure what to do from here. I have another backup but it is a
> bit older. If there is any way to recover the current system, I would
> prefer that.
> 
> Does anyone have any suggestions on what might be the next steps I
> should try?
> 
> Nick
> 
> eXist Version: 	4.5.0
> eXist Build: 	201901142136
> Operating System: 	Linux 4.15.0-74-generic amd64
> Java Version: 	11.0.9.1
> Default Encoding: 	UTF8
> Instance ID: 	exist
> System CPU Load: 	0.00429185
> Process CPU Load: 	0.00854701
> Free Physical Memory: 	232964096
> Total Physical Memory: 	8164757504
> 

Re: [Exist-open] Filenames with awkward characters in eXist-db?

[Hungerburg 13 <pc...@my...>]

Might be of interest to the person, that builds the mac binaries:

 From personal messages of David's, eXist-db on mac does not use the 
-Dfile.encoding=UTF-8 nor the -Dsun.jnu.encoding=UTF-8 java flags, and 
it also has no locale set in the environment (it runs in the C locale 
then?), when click starting from Finder.

Not a mac expert here, perhaps those should be specified in some 
.properties file? Perhaps then, also non-technical users can have 
awkward chars on the mac?

Peter

[Exist-open] eXist-db does not see /db/apps

[Nick S. <nsi...@nu...>]

I am running eXist-db v4.5.0. Recently, I discovered the system was not 
responsive. I shut it down and started it back up after a little effort. 
I am not sure what exactly happened that caused it stop being responsive.

Once I was able to get it up and running again, I noticed that a number 
of my scripts were failing. I then tried to re-index the system.
let $x := xmldb:reindex('/db/')
return $x

This returns very quickly with true().
However, if I try to reindex '/db/apps' it returns false()

My conclusion is that eXist-db does not able to see /db/apps. I then 
took a backup of the system. When I look at the backup log, a lot of 
subfolders in apps did not get backed up.

I am not sure what to do from here. I have another backup but it is a 
bit older. If there is any way to recover the current system, I would 
prefer that.

Does anyone have any suggestions on what might be the next steps I 
should try?

Nick

eXist Version: 	4.5.0
eXist Build: 	201901142136
Operating System: 	Linux 4.15.0-74-generic amd64
Java Version: 	11.0.9.1
Default Encoding: 	UTF8
Instance ID: 	exist
System CPU Load: 	0.00429185
Process CPU Load: 	0.00854701
Free Physical Memory: 	232964096
Total Physical Memory: 	8164757504

-- 
Nick Sincaglia
President/Founder
NueMeta, LLC
Digital Media & Technology
Phone: +1-630-303-7035
nsi...@nu...
http://www.nuemeta.com
Skype: nsincaglia

Re: [Exist-open] ngram index with mixed content

[Christian W. <cwi...@gm...>]

Dear Joe,

Thank you for your help.  The bug was between my ears, eXist-db works as 
expected :-)

All the best,

Christian

On 03/12/2020 04.01, Joe Wicentowski wrote:
> Hi Christian,
>
> Could you please try:
>
>   //tei:p[ngram:contains(., "CD.EF")]
>
> Instead of:
>
>   //ngram:contains(., "CD.EF")
>
> Joe
>
> On Wed, Dec 2, 2020 at 4:41 AM Christian Wittern <cwi...@gm... 
> <mailto:cwi...@gm...>> wrote:
>
>     Dear eXist users,
>
>     I am trying to improve search recall for my application, which
>     contains
>     premodern Chinese text.  My documents are set up to have <tei:seg>
>     elements for every phrase, which are in turn contained by tei:p
>     elements. At the moment, I am simple defining a ngram index on
>     tei:seg,
>     but that of course limits the matches to the contents of one tei:seg
>     element.  To overcome this limitation, I am defining a ngram index on
>     tei:p as well, in the expectation that the ngrams will be
>     constructed by
>     concating the tei:seg elements that make up a paragraph.  So for
>     example:
>
>     <tei:p><tei:seg>ABCD.</tei:seg><tei:seg>EFGH</tei:seg></tei:p>
>
>     With such a text, I would expect to be able to search for "CD.EF" and
>     find one match. However there is no match for
>
>     //ngram:contains(., "CD.EF"), also not with
>     //ngram:wildcard-contains(.,
>     "CD.EF")
>
>     The reason for this assumption is that the documentation for the
>     ngram
>     module says:
>
>     "Note: a ngram match on mixed content may span multiple nodes. "
>     (this
>     is in the documentation for the ngram:filter-matches function).
>
>     Since there are no parameters when setting up an ngram index, I would
>     expect that elements with mixed content like the tei:p element
>     would be
>     able to find a term across tei:seg elements.
>
>     Is this a bug or am I missing something?  Any help appreciated,
>
>     Christian
>
>
>
>
>     _______________________________________________
>     Exist-open mailing list
>     Exi...@li...
>     <mailto:Exi...@li...>
>     https://lists.sourceforge.net/lists/listinfo/exist-open
>     <https://lists.sourceforge.net/lists/listinfo/exist-open>
>

Re: [Exist-open] ngram index with mixed content

[Joe W. <jo...@gm...>]

Hi Christian,

Could you please try:

  //tei:p[ngram:contains(., "CD.EF")]

Instead of:

  //ngram:contains(., "CD.EF")

Joe

On Wed, Dec 2, 2020 at 4:41 AM Christian Wittern <cwi...@gm...> wrote:

> Dear eXist users,
>
> I am trying to improve search recall for my application, which contains
> premodern Chinese text.  My documents are set up to have <tei:seg>
> elements for every phrase, which are in turn contained by tei:p
> elements. At the moment, I am simple defining a ngram index on tei:seg,
> but that of course limits the matches to the contents of one tei:seg
> element.  To overcome this limitation, I am defining a ngram index on
> tei:p as well, in the expectation that the ngrams will be constructed by
> concating the tei:seg elements that make up a paragraph.  So for example:
>
> <tei:p><tei:seg>ABCD.</tei:seg><tei:seg>EFGH</tei:seg></tei:p>
>
> With such a text, I would expect to be able to search for "CD.EF" and
> find one match. However there is no match for
>
> //ngram:contains(., "CD.EF"), also not with //ngram:wildcard-contains(.,
> "CD.EF")
>
> The reason for this assumption is that the documentation for the ngram
> module says:
>
> "Note: a ngram match on mixed content may span multiple nodes. " (this
> is in the documentation for the ngram:filter-matches function).
>
> Since there are no parameters when setting up an ngram index, I would
> expect that elements with mixed content like the tei:p element would be
> able to find a term across tei:seg elements.
>
> Is this a bug or am I missing something?  Any help appreciated,
>
> Christian
>
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

[Exist-open] ngram index with mixed content

[Christian W. <cwi...@gm...>]

Dear eXist users,

I am trying to improve search recall for my application, which contains 
premodern Chinese text.  My documents are set up to have <tei:seg> 
elements for every phrase, which are in turn contained by tei:p 
elements. At the moment, I am simple defining a ngram index on tei:seg, 
but that of course limits the matches to the contents of one tei:seg 
element.  To overcome this limitation, I am defining a ngram index on 
tei:p as well, in the expectation that the ngrams will be constructed by 
concating the tei:seg elements that make up a paragraph.  So for example:

<tei:p><tei:seg>ABCD.</tei:seg><tei:seg>EFGH</tei:seg></tei:p>

With such a text, I would expect to be able to search for "CD.EF" and 
find one match. However there is no match for

//ngram:contains(., "CD.EF"), also not with //ngram:wildcard-contains(., 
"CD.EF")

The reason for this assumption is that the documentation for the ngram 
module says:

"Note: a ngram match on mixed content may span multiple nodes. " (this 
is in the documentation for the ngram:filter-matches function).

Since there are no parameters when setting up an ngram index, I would 
expect that elements with mixed content like the tei:p element would be 
able to find a term across tei:seg elements.

Is this a bug or am I missing something?  Any help appreciated,

Christian

Re: [Exist-open] Programmatically call the view.xql module

[Alasdair D. <ala...@gm...>]

Hi Dannes,

Thanks for the reply.

An update for you and anyone else interested.

I have dropped the idea, due to the tricky nature of getting things to work
as a trigger.  As I have already done the development work and it generates
the appropriate webpages, I am resigned to running a xquery script that
locates all recipe documents created after a given date and writes out the
CURL command to create the web page and saves it as a text file.

So, I don't need to reach into the structured filestore of eXist, and
simply can run an 'all documents' command if ever I need to
generate updated web pages.  Also, if the 'page' file is not found, Nginx
proxypass will simply call the eXist-db web server to get the dynamic page.

Alasdair


On Sat, Nov 28, 2020 at 11:11 PM Dannes Wessels <da...@ex...> wrote:

> Hi,
>
> On 25 Nov 2020, at 23:48 , Alasdair Dougall <ala...@gm...>
> wrote:
>
> As I have moved to Nginx, I am now using the try_files option before going
> to eXist to render the page. It is very quick to return the page.  So, if a
> new recipe is added that has not had the static part generated, eXists
> takes care of it without a single issue.
>
>
> “try_files” … does this mean that you refer directly to the files visible
> in webapp/WEB-INF/data/ ?
> I guess that is attractive but also tricky. In eXist-db 5.x this will not
> work anymore as we found this direct access too risky (and we needed to
> have additional features like deduplication)
>
> regards
> Dannes
>
>
>
> eXist-db Native XML Database
> http://www.exist-db.org
>
>

[Exist-open] Warning about eXist Crypto module

[Joe W. <jo...@gm...>]

Hi all,

Following up on today's Community Call, users of the eXist Crypto Module
should be warned about problems in the current release of the
package, 5.3.0:

  https://github.com/eXist-db/crypto-exist-java-lib/issues/33

The problems and next steps are described in the issue.

Joe

[Exist-open] Future of eXist's Crypto library

[Joe W. <jo...@gm...>]

Hi all,

Following up on last week’s discussion about the future of the EXPath
Crypto spec and lib in the eXist-db Community Call, I posted an issue to
the EXPath community group, to invite ideas. Please feel free to chime in:
https://github.com/expath/expath-cg/issues/132.

Joe

Re: [Exist-open] Problem accessing /apps/eXide/modules/search.xql.

[Joe W. <jo...@gm...>]

Sorry, I meant search.xql, not search.cal. (Darn autocorrect.)

On Sat, Nov 28, 2020 at 9:20 AM Joe Wicentowski <jo...@gm...> wrote:

> Hi Nick,
>
> Having an associated error message may help diagnose the problem. I’d
> suggest checking both exist.log and the Developer Tools > Network >
> search.cal > Response.
>
> Joe
>
> On Sat, Nov 28, 2020 at 2:25 AM Nick Sincaglia <nsi...@nu...>
> wrote:
>
>> Hi Joe,
>>     I appreciate the effort in trying to recreate the issue. Your
>> experience is what I would expect would happen. The question I was asking
>> was, is there any way to fix a "Find in files" search that returns an
>> exception? Is there any information I can provide that would help someone
>> with an educated eye understand why this search is not working on my
>> system? Is there anything I can do to fix this issue or should I just
>> install a fresh eXIst-db and migrate my data over to that install?
>>
>>
>> Nick
>>
>>
>> On 11/27/20 3:07 PM, Joe Wicentowski wrote:
>>
>> Hi Nick,
>>
>> I just downloaded eXist 4.7.1 and performed a clean installation. I went
>> to the package manager and applied all available updates: eXide (2.4.9),
>> Shared Resources (0.9.1), FunctX (1.0.1), and Monex (1.0.1). Then I went to
>> eXide, selected Edit > Find in Files, copied your selections - of "Type:
>> XQuery" and "In: All", entered a keyword of my choosing, "concat", and hit
>> the Search button. The search results came back without error. Attached is
>> a screenshot showing the results, with the Developer Tools open showing the
>> response.
>>
>> Thus, I'm afraid I'm unable to reproduce the error you describe.
>>
>> Best,
>> Joe
>>
>> On Wed, Nov 25, 2020 at 2:23 PM Nick Sincaglia <nsi...@nu...>
>> wrote:
>>
>>> From what I can tell, all search terms are throwing the error.
>>>
>>> On 11/25/20 11:02 AM, Joe Wicentowski wrote:
>>>
>>> Does any search term produce the error, or only some? Ideally, we’d have
>>> a term that will produce the error on a stock system.
>>>
>>> On Wed, Nov 25, 2020 at 11:54 AM Nick Sincaglia <nsi...@nu...>
>>> wrote:
>>>
>>>> I am using eXist-db version = eXist Version:     4.7.1
>>>>
>>>> The steps to produce it are:
>>>>
>>>> Once in eXide, I navigate to the "Edit" menu. Click the pull down menu.
>>>> Select "Find in file". Enter my search term in the "Search" bar in the
>>>> pop-up window. Select "XQuery" in the 'Type' pull down. Click the "All"
>>>> radio button. Then click the "Search" button in the lower left hand corner
>>>> of the the Pop-up window.
>>>>
>>>> Nick
>>>>
>>>> On 11/25/20 10:46 AM, Joe Wicentowski wrote:
>>>>
>>>> Hi Nick,
>>>>
>>>> Which version of eXist and eXide?
>>>>
>>>> Can you provide steps to reproduce the error on a stock installation of
>>>> eXist?
>>>>
>>>> Joe
>>>>
>>>> On Wed, Nov 25, 2020 at 11:30 AM Nick Sincaglia <nsi...@nu...>
>>>> wrote:
>>>>
>>>>> Occasionally, I come across this issue and I was wondering if anyone
>>>>> might have a suggestion on how to address it.
>>>>>
>>>>> When I am using eXide, I try to use the "Find in files" pull down and
>>>>> I
>>>>> get the below error. I tried re-indexing my database. That does not
>>>>> work. I found /apps/eXide/modules/search.xql and added a white space
>>>>> to
>>>>> the bottom of the file and re-saved it. I have this helps sometimes
>>>>> when
>>>>> eXist-db can't locate a file.
>>>>>
>>>>> Any suggestions on how to resolve this issue?
>>>>>
>>>>>
>>>>> HTTP ERROR 500
>>>>>
>>>>> Problem accessing /apps/eXide/modules/search.xql. Reason:
>>>>>
>>>>>      Server Error
>>>>>
>>>>> Caused by:
>>>>>
>>>>> javax.servlet.ServletException: javax.servlet.ServletException: An
>>>>> error
>>>>> occurred while processing request to /apps/eXide/modules/search.xql:
>>>>> An
>>>>> error occurred: null
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>>>>      at org.eclipse.jetty.server.Server.handle(Server.java:502)
>>>>>      at
>>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
>>>>>      at
>>>>> org.eclipse.jetty.io
>>>>> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
>>>>>      at org.eclipse.jetty.io
>>>>> .FillInterest.fillable(FillInterest.java:103)
>>>>>      at org.eclipse.jetty.io
>>>>> .ChannelEndPoint$2.run(ChannelEndPoint.java:118)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
>>>>>      at java.lang.Thread.run(Thread.java:748)
>>>>> Caused by: javax.servlet.ServletException: An error occurred while
>>>>> processing request to /apps/eXide/modules/search.xql: An error
>>>>> occurred:
>>>>> null
>>>>>      at
>>>>>
>>>>> org.exist.http.urlrewrite.XQueryURLRewrite.service(XQueryURLRewrite.java:370)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>>>>>      at
>>>>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:867)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
>>>>>      at
>>>>>
>>>>> de.betterform.agent.web.filter.XFormsFilter.doFilter(XFormsFilter.java:171)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
>>>>>      at
>>>>>
>>>>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
>>>>>      ... 16 more
>>>>>
>>>>> --
>>>>> Nick Sincaglia
>>>>> President/Founder
>>>>> NueMeta, LLC
>>>>> Digital Media & Technology
>>>>> Phone: +1-630-303-7035
>>>>> nsi...@nu...
>>>>> http://www.nuemeta.com
>>>>> Skype: nsincaglia
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Exist-open mailing list
>>>>> Exi...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>>>>
>>>>
>>>> --
>>>> Nick Sincaglia
>>>> President/Founder
>>>> NueMeta, LLC
>>>> Digital Media & Technology
>>>> Phone: +1-...@nu... http://www.nuemeta.com
>>>> Skype: nsincaglia
>>>>
>>>> --
>>> Sent from my iPhone
>>>
>>>
>>> --
>>> Nick Sincaglia
>>> President/Founder
>>> NueMeta, LLC
>>> Digital Media & Technology
>>> Phone: +1-...@nu... http://www.nuemeta.com
>>> Skype: nsincaglia
>>>
>>>
>> --
>> Nick Sincaglia
>> President/Founder
>> NueMeta, LLC
>> Digital Media & Technology
>> Phone: +1-...@nu... http://www.nuemeta.com
>> Skype: nsincaglia
>>
>> --
> Sent from my iPhone
>

Re: [Exist-open] Problem accessing /apps/eXide/modules/search.xql.

[Joe W. <jo...@gm...>]

Hi Nick,

Having an associated error message may help diagnose the problem. I’d
suggest checking both exist.log and the Developer Tools > Network >
search.cal > Response.

Joe

On Sat, Nov 28, 2020 at 2:25 AM Nick Sincaglia <nsi...@nu...>
wrote:

> Hi Joe,
>     I appreciate the effort in trying to recreate the issue. Your
> experience is what I would expect would happen. The question I was asking
> was, is there any way to fix a "Find in files" search that returns an
> exception? Is there any information I can provide that would help someone
> with an educated eye understand why this search is not working on my
> system? Is there anything I can do to fix this issue or should I just
> install a fresh eXIst-db and migrate my data over to that install?
>
>
> Nick
>
>
> On 11/27/20 3:07 PM, Joe Wicentowski wrote:
>
> Hi Nick,
>
> I just downloaded eXist 4.7.1 and performed a clean installation. I went
> to the package manager and applied all available updates: eXide (2.4.9),
> Shared Resources (0.9.1), FunctX (1.0.1), and Monex (1.0.1). Then I went to
> eXide, selected Edit > Find in Files, copied your selections - of "Type:
> XQuery" and "In: All", entered a keyword of my choosing, "concat", and hit
> the Search button. The search results came back without error. Attached is
> a screenshot showing the results, with the Developer Tools open showing the
> response.
>
> Thus, I'm afraid I'm unable to reproduce the error you describe.
>
> Best,
> Joe
>
> On Wed, Nov 25, 2020 at 2:23 PM Nick Sincaglia <nsi...@nu...>
> wrote:
>
>> From what I can tell, all search terms are throwing the error.
>>
>> On 11/25/20 11:02 AM, Joe Wicentowski wrote:
>>
>> Does any search term produce the error, or only some? Ideally, we’d have
>> a term that will produce the error on a stock system.
>>
>> On Wed, Nov 25, 2020 at 11:54 AM Nick Sincaglia <nsi...@nu...>
>> wrote:
>>
>>> I am using eXist-db version = eXist Version:     4.7.1
>>>
>>> The steps to produce it are:
>>>
>>> Once in eXide, I navigate to the "Edit" menu. Click the pull down menu.
>>> Select "Find in file". Enter my search term in the "Search" bar in the
>>> pop-up window. Select "XQuery" in the 'Type' pull down. Click the "All"
>>> radio button. Then click the "Search" button in the lower left hand corner
>>> of the the Pop-up window.
>>>
>>> Nick
>>>
>>> On 11/25/20 10:46 AM, Joe Wicentowski wrote:
>>>
>>> Hi Nick,
>>>
>>> Which version of eXist and eXide?
>>>
>>> Can you provide steps to reproduce the error on a stock installation of
>>> eXist?
>>>
>>> Joe
>>>
>>> On Wed, Nov 25, 2020 at 11:30 AM Nick Sincaglia <nsi...@nu...>
>>> wrote:
>>>
>>>> Occasionally, I come across this issue and I was wondering if anyone
>>>> might have a suggestion on how to address it.
>>>>
>>>> When I am using eXide, I try to use the "Find in files" pull down and I
>>>> get the below error. I tried re-indexing my database. That does not
>>>> work. I found /apps/eXide/modules/search.xql and added a white space to
>>>> the bottom of the file and re-saved it. I have this helps sometimes
>>>> when
>>>> eXist-db can't locate a file.
>>>>
>>>> Any suggestions on how to resolve this issue?
>>>>
>>>>
>>>> HTTP ERROR 500
>>>>
>>>> Problem accessing /apps/eXide/modules/search.xql. Reason:
>>>>
>>>>      Server Error
>>>>
>>>> Caused by:
>>>>
>>>> javax.servlet.ServletException: javax.servlet.ServletException: An
>>>> error
>>>> occurred while processing request to /apps/eXide/modules/search.xql: An
>>>> error occurred: null
>>>>      at
>>>>
>>>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
>>>>      at
>>>>
>>>> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
>>>>      at
>>>>
>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>>>      at org.eclipse.jetty.server.Server.handle(Server.java:502)
>>>>      at
>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
>>>>      at
>>>>
>>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
>>>>      at
>>>> org.eclipse.jetty.io
>>>> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
>>>>      at org.eclipse.jetty.io
>>>> .FillInterest.fillable(FillInterest.java:103)
>>>>      at org.eclipse.jetty.io
>>>> .ChannelEndPoint$2.run(ChannelEndPoint.java:118)
>>>>      at
>>>>
>>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
>>>>      at
>>>>
>>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
>>>>      at
>>>>
>>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>>>>      at
>>>>
>>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
>>>>      at
>>>>
>>>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
>>>>      at
>>>>
>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
>>>>      at
>>>>
>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
>>>>      at java.lang.Thread.run(Thread.java:748)
>>>> Caused by: javax.servlet.ServletException: An error occurred while
>>>> processing request to /apps/eXide/modules/search.xql: An error
>>>> occurred:
>>>> null
>>>>      at
>>>>
>>>> org.exist.http.urlrewrite.XQueryURLRewrite.service(XQueryURLRewrite.java:370)
>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>>>>      at
>>>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:867)
>>>>      at
>>>>
>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
>>>>      at
>>>>
>>>> de.betterform.agent.web.filter.XFormsFilter.doFilter(XFormsFilter.java:171)
>>>>      at
>>>>
>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>>>>      at
>>>>
>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>>>>      at
>>>>
>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>>>>      at
>>>>
>>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
>>>>      at
>>>>
>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>>>      at
>>>>
>>>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>>>>      at
>>>>
>>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
>>>>      at
>>>>
>>>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
>>>>      at
>>>>
>>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
>>>>      at
>>>>
>>>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
>>>>      at
>>>>
>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
>>>>      at
>>>>
>>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
>>>>      at
>>>>
>>>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
>>>>      at
>>>>
>>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
>>>>      at
>>>>
>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
>>>>      at
>>>>
>>>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
>>>>      at
>>>>
>>>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
>>>>      ... 16 more
>>>>
>>>> --
>>>> Nick Sincaglia
>>>> President/Founder
>>>> NueMeta, LLC
>>>> Digital Media & Technology
>>>> Phone: +1-630-303-7035
>>>> nsi...@nu...
>>>> http://www.nuemeta.com
>>>> Skype: nsincaglia
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Exist-open mailing list
>>>> Exi...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>>>
>>>
>>> --
>>> Nick Sincaglia
>>> President/Founder
>>> NueMeta, LLC
>>> Digital Media & Technology
>>> Phone: +1-...@nu... http://www.nuemeta.com
>>> Skype: nsincaglia
>>>
>>> --
>> Sent from my iPhone
>>
>>
>> --
>> Nick Sincaglia
>> President/Founder
>> NueMeta, LLC
>> Digital Media & Technology
>> Phone: +1-...@nu... http://www.nuemeta.com
>> Skype: nsincaglia
>>
>>
> --
> Nick Sincaglia
> President/Founder
> NueMeta, LLC
> Digital Media & Technology
> Phone: +1-...@nu... http://www.nuemeta.com
> Skype: nsincaglia
>
> --
Sent from my iPhone

Re: [Exist-open] Programmatically call the view.xql module

[Dannes W. <da...@ex...>]

Hi,

> On 25 Nov 2020, at 23:48 , Alasdair Dougall <ala...@gm...> wrote:
> 
> As I have moved to Nginx, I am now using the try_files option before going to eXist to render the page. It is very quick to return the page.  So, if a new recipe is added that has not had the static part generated, eXists takes care of it without a single issue.

“try_files” … does this mean that you refer directly to the files visible in webapp/WEB-INF/data/ ?
I guess that is attractive but also tricky. In eXist-db 5.x this will not work anymore as we found this direct access too risky (and we needed to have additional features like deduplication)

regards
Dannes



eXist-db Native XML Database
http://www.exist-db.org

Re: [Exist-open] Problem accessing /apps/eXide/modules/search.xql.

[Nick S. <nsi...@nu...>]

Hi Joe,
     I appreciate the effort in trying to recreate the issue. Your 
experience is what I would expect would happen. The question I was 
asking was, is there any way to fix a "Find in files" search that 
returns an exception? Is there any information I can provide that would 
help someone with an educated eye understand why this search is not 
working on my system? Is there anything I can do to fix this issue or 
should I just install a fresh eXIst-db and migrate my data over to that 
install?

Nick


On 11/27/20 3:07 PM, Joe Wicentowski wrote:
> Hi Nick,
>
> I just downloaded eXist 4.7.1 and performed a clean installation. I 
> went to the package manager and applied all available updates: eXide 
> (2.4.9), Shared Resources (0.9.1), FunctX (1.0.1), and Monex (1.0.1). 
> Then I went to eXide, selected Edit > Find in Files, copied your 
> selections - of "Type: XQuery" and "In: All", entered a keyword of my 
> choosing, "concat", and hit the Search button. The search results came 
> back without error. Attached is a screenshot showing the results, with 
> the Developer Tools open showing the response.
>
> Thus, I'm afraid I'm unable to reproduce the error you describe.
>
> Best,
> Joe
>
> On Wed, Nov 25, 2020 at 2:23 PM Nick Sincaglia <nsi...@nu... 
> <mailto:nsi...@nu...>> wrote:
>
>     From what I can tell, all search terms are throwing the error.
>
>     On 11/25/20 11:02 AM, Joe Wicentowski wrote:
>>     Does any search term produce the error, or only some? Ideally,
>>     we’d have a term that will produce the error on a stock system.
>>
>>     On Wed, Nov 25, 2020 at 11:54 AM Nick Sincaglia
>>     <nsi...@nu... <mailto:nsi...@nu...>> wrote:
>>
>>         I am using eXist-db version = eXist Version:     4.7.1
>>
>>         The steps to produce it are:
>>
>>         Once in eXide, I navigate to the "Edit" menu. Click the pull
>>         down menu. Select "Find in file". Enter my search term in the
>>         "Search" bar in the pop-up window. Select "XQuery" in the
>>         'Type' pull down. Click the "All" radio button. Then click
>>         the "Search" button in the lower left hand corner of the the
>>         Pop-up window.
>>
>>         Nick
>>
>>         On 11/25/20 10:46 AM, Joe Wicentowski wrote:
>>>         Hi Nick,
>>>
>>>         Which version of eXist and eXide?
>>>
>>>         Can you provide steps to reproduce the error on a stock
>>>         installation of eXist?
>>>
>>>         Joe
>>>
>>>         On Wed, Nov 25, 2020 at 11:30 AM Nick Sincaglia
>>>         <nsi...@nu... <mailto:nsi...@nu...>> wrote:
>>>
>>>             Occasionally, I come across this issue and I was
>>>             wondering if anyone
>>>             might have a suggestion on how to address it.
>>>
>>>             When I am using eXide, I try to use the "Find in files"
>>>             pull down and I
>>>             get the below error. I tried re-indexing my database.
>>>             That does not
>>>             work. I found /apps/eXide/modules/search.xql and added a
>>>             white space to
>>>             the bottom of the file and re-saved it. I have this
>>>             helps sometimes when
>>>             eXist-db can't locate a file.
>>>
>>>             Any suggestions on how to resolve this issue?
>>>
>>>
>>>             HTTP ERROR 500
>>>
>>>             Problem accessing /apps/eXide/modules/search.xql. Reason:
>>>
>>>                  Server Error
>>>
>>>             Caused by:
>>>
>>>             javax.servlet.ServletException:
>>>             javax.servlet.ServletException: An error
>>>             occurred while processing request to
>>>             /apps/eXide/modules/search.xql: An
>>>             error occurred: null
>>>                  at
>>>             org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
>>>                  at
>>>             org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
>>>                  at
>>>             org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>>                  at
>>>             org.eclipse.jetty.server.Server.handle(Server.java:502)
>>>                  at
>>>             org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
>>>                  at
>>>             org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
>>>                  at
>>>             org.eclipse.jetty.io
>>>             <http://org.eclipse.jetty.io>.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
>>>                  at org.eclipse.jetty.io
>>>             <http://org.eclipse.jetty.io>.FillInterest.fillable(FillInterest.java:103)
>>>                  at org.eclipse.jetty.io
>>>             <http://org.eclipse.jetty.io>.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
>>>                  at
>>>             org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
>>>                  at
>>>             org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
>>>                  at
>>>             org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>>>                  at
>>>             org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
>>>                  at
>>>             org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
>>>                  at
>>>             org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
>>>                  at
>>>             org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
>>>                  at java.lang.Thread.run(Thread.java:748)
>>>             Caused by: javax.servlet.ServletException: An error
>>>             occurred while
>>>             processing request to /apps/eXide/modules/search.xql: An
>>>             error occurred:
>>>             null
>>>                  at
>>>             org.exist.http.urlrewrite.XQueryURLRewrite.service(XQueryURLRewrite.java:370)
>>>                  at
>>>             javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>>>                  at
>>>             org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:867)
>>>                  at
>>>             org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
>>>                  at
>>>             de.betterform.agent.web.filter.XFormsFilter.doFilter(XFormsFilter.java:171)
>>>                  at
>>>             org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>>>                  at
>>>             org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>>>                  at
>>>             org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>>>                  at
>>>             org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
>>>                  at
>>>             org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>>                  at
>>>             org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>>>                  at
>>>             org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
>>>                  at
>>>             org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
>>>                  at
>>>             org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
>>>                  at
>>>             org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
>>>                  at
>>>             org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
>>>                  at
>>>             org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
>>>                  at
>>>             org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
>>>                  at
>>>             org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
>>>                  at
>>>             org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
>>>                  at
>>>             org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
>>>                  at
>>>             org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
>>>                  ... 16 more
>>>
>>>             -- 
>>>             Nick Sincaglia
>>>             President/Founder
>>>             NueMeta, LLC
>>>             Digital Media & Technology
>>>             Phone: +1-630-303-7035
>>>             nsi...@nu... <mailto:nsi...@nu...>
>>>             http://www.nuemeta.com <http://www.nuemeta.com>
>>>             Skype: nsincaglia
>>>
>>>
>>>
>>>             _______________________________________________
>>>             Exist-open mailing list
>>>             Exi...@li...
>>>             <mailto:Exi...@li...>
>>>             https://lists.sourceforge.net/lists/listinfo/exist-open
>>>             <https://lists.sourceforge.net/lists/listinfo/exist-open>
>>>
>>
>>         -- 
>>         Nick Sincaglia
>>         President/Founder
>>         NueMeta, LLC
>>         Digital Media & Technology
>>         Phone: +1-630-303-7035
>>         nsi...@nu...  <mailto:nsi...@nu...>  
>>         http://www.nuemeta.com  <http://www.nuemeta.com>
>>         Skype: nsincaglia
>>
>>     -- 
>>     Sent from my iPhone
>
>     -- 
>     Nick Sincaglia
>     President/Founder
>     NueMeta, LLC
>     Digital Media & Technology
>     Phone: +1-630-303-7035
>     nsi...@nu...  <mailto:nsi...@nu...>  
>     http://www.nuemeta.com  <http://www.nuemeta.com>
>     Skype: nsincaglia
>

-- 
Nick Sincaglia
President/Founder
NueMeta, LLC
Digital Media & Technology
Phone: +1-630-303-7035
nsi...@nu...
http://www.nuemeta.com
Skype: nsincaglia

Re: [Exist-open] Problem accessing /apps/eXide/modules/search.xql.

[Joe W. <jo...@gm...>]

Hi Nick,

I just downloaded eXist 4.7.1 and performed a clean installation. I went to
the package manager and applied all available updates: eXide (2.4.9),
Shared Resources (0.9.1), FunctX (1.0.1), and Monex (1.0.1). Then I went to
eXide, selected Edit > Find in Files, copied your selections - of "Type:
XQuery" and "In: All", entered a keyword of my choosing, "concat", and hit
the Search button. The search results came back without error. Attached is
a screenshot showing the results, with the Developer Tools open showing the
response.

Thus, I'm afraid I'm unable to reproduce the error you describe.

Best,
Joe

On Wed, Nov 25, 2020 at 2:23 PM Nick Sincaglia <nsi...@nu...>
wrote:

> From what I can tell, all search terms are throwing the error.
>
> On 11/25/20 11:02 AM, Joe Wicentowski wrote:
>
> Does any search term produce the error, or only some? Ideally, we’d have a
> term that will produce the error on a stock system.
>
> On Wed, Nov 25, 2020 at 11:54 AM Nick Sincaglia <nsi...@nu...>
> wrote:
>
>> I am using eXist-db version = eXist Version:     4.7.1
>>
>> The steps to produce it are:
>>
>> Once in eXide, I navigate to the "Edit" menu. Click the pull down menu.
>> Select "Find in file". Enter my search term in the "Search" bar in the
>> pop-up window. Select "XQuery" in the 'Type' pull down. Click the "All"
>> radio button. Then click the "Search" button in the lower left hand corner
>> of the the Pop-up window.
>>
>> Nick
>>
>> On 11/25/20 10:46 AM, Joe Wicentowski wrote:
>>
>> Hi Nick,
>>
>> Which version of eXist and eXide?
>>
>> Can you provide steps to reproduce the error on a stock installation of
>> eXist?
>>
>> Joe
>>
>> On Wed, Nov 25, 2020 at 11:30 AM Nick Sincaglia <nsi...@nu...>
>> wrote:
>>
>>> Occasionally, I come across this issue and I was wondering if anyone
>>> might have a suggestion on how to address it.
>>>
>>> When I am using eXide, I try to use the "Find in files" pull down and I
>>> get the below error. I tried re-indexing my database. That does not
>>> work. I found /apps/eXide/modules/search.xql and added a white space to
>>> the bottom of the file and re-saved it. I have this helps sometimes when
>>> eXist-db can't locate a file.
>>>
>>> Any suggestions on how to resolve this issue?
>>>
>>>
>>> HTTP ERROR 500
>>>
>>> Problem accessing /apps/eXide/modules/search.xql. Reason:
>>>
>>>      Server Error
>>>
>>> Caused by:
>>>
>>> javax.servlet.ServletException: javax.servlet.ServletException: An error
>>> occurred while processing request to /apps/eXide/modules/search.xql: An
>>> error occurred: null
>>>      at
>>>
>>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
>>>      at
>>>
>>> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
>>>      at
>>>
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>>      at org.eclipse.jetty.server.Server.handle(Server.java:502)
>>>      at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
>>>      at
>>>
>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
>>>      at
>>> org.eclipse.jetty.io
>>> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
>>>      at org.eclipse.jetty.io
>>> .FillInterest.fillable(FillInterest.java:103)
>>>      at org.eclipse.jetty.io
>>> .ChannelEndPoint$2.run(ChannelEndPoint.java:118)
>>>      at
>>>
>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
>>>      at
>>>
>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
>>>      at
>>>
>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>>>      at
>>>
>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
>>>      at
>>>
>>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
>>>      at
>>>
>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
>>>      at
>>>
>>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
>>>      at java.lang.Thread.run(Thread.java:748)
>>> Caused by: javax.servlet.ServletException: An error occurred while
>>> processing request to /apps/eXide/modules/search.xql: An error occurred:
>>> null
>>>      at
>>>
>>> org.exist.http.urlrewrite.XQueryURLRewrite.service(XQueryURLRewrite.java:370)
>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>>>      at
>>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:867)
>>>      at
>>>
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
>>>      at
>>>
>>> de.betterform.agent.web.filter.XFormsFilter.doFilter(XFormsFilter.java:171)
>>>      at
>>>
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>>>      at
>>>
>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>>>      at
>>>
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>>>      at
>>>
>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
>>>      at
>>>
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>>      at
>>>
>>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>>>      at
>>>
>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
>>>      at
>>>
>>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
>>>      at
>>>
>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
>>>      at
>>>
>>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
>>>      at
>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
>>>      at
>>>
>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
>>>      at
>>>
>>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
>>>      at
>>>
>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
>>>      at
>>>
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
>>>      at
>>>
>>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
>>>      at
>>>
>>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
>>>      ... 16 more
>>>
>>> --
>>> Nick Sincaglia
>>> President/Founder
>>> NueMeta, LLC
>>> Digital Media & Technology
>>> Phone: +1-630-303-7035
>>> nsi...@nu...
>>> http://www.nuemeta.com
>>> Skype: nsincaglia
>>>
>>>
>>>
>>> _______________________________________________
>>> Exist-open mailing list
>>> Exi...@li...
>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>>
>>
>> --
>> Nick Sincaglia
>> President/Founder
>> NueMeta, LLC
>> Digital Media & Technology
>> Phone: +1-...@nu... http://www.nuemeta.com
>> Skype: nsincaglia
>>
>> --
> Sent from my iPhone
>
>
> --
> Nick Sincaglia
> President/Founder
> NueMeta, LLC
> Digital Media & Technology
> Phone: +1-...@nu... http://www.nuemeta.com
> Skype: nsincaglia
>
>

Re: [Exist-open] Filenames with awkward characters in eXist-db?

[Hungerburg 13 <pc...@my...>]

Am 26.11.20 um 09:34 schrieb Hungerburg 13:
> David, All, One more observation, that might be of interest:
> The /locale/ that the exist-db process runs with makes a difference

Attached updated sample java code, source and object, that prints some 
relevant information. When charset is UTF-8, string will echo "ё", when 
ASCII it will came out "?". The brackets will always escape. Might be 
helpful if run on diverse systems to get an overview of what is out 
there in the wild.


run1: java awkward
   Charset: UTF-8
run2: LANG=C java awkward
   Charset: US-ASCII
run4: LANG=C java -Dfile.encoding=UTF-8 awkward
   Charset: UTF-8
run4: LANG=C java -Dsun.jnu.encoding=UTF-8 awkward
   Charset: US-ASCII


Curiously, compiling in C-locale fails: LANG=C javac awkward.java
   error: unmappable character for encoding ASCII
How to force an exception like that during runtime?

Some more info on the topic 
https://dzone.com/articles/java-may-use-utf-8-as-its-default-charset - 
notice the mention of URLencoder() and also, that file paths are handled 
by an extra flag (run4 above).

Peter

Re: [Exist-open] Filenames with awkward characters in eXist-db?

[Michael W. <wes...@ja...>]

In the 1990s, I made it a habit to ONLY use 7-bit ASCII characters in file
names, avoiding anything outside of  [0-9A-Za-z_-]. The reason being that I
moved between MS-DOS/Windows (using Shift_JIS character encoding) and
FreeBSD (using euc-jp encoding) a great deal and needed everything to work
well from the command line in both.

The late 1990s saw the beginning of UTF-8 taking over as a unified
character set, and once I could set that as my main character set in
FreeBSD (and by extension OSX in the early 2000s) as well as in MySQL and
other databases, I started feeling better about allowing Japanese file
names into my systems. But I still avoided special characters that had
meaning to bash and other shells. Again, I wanted names that could be typed
on the command line without having to escape a bunch of characters. I
wouldn't (and still don't) even put spaces into file names.
!#$%&'"*+,;{}()[]/\ in file names are just asking for trouble. (The person
who decided to use \ as a directory separator for MS-DOS has caused more
damage to the PC industry than can be calculated.)

When I've needed to automate something based on a file name, I would use a
series like _xyz_ to denote it. I could use regular expressions to extract
the name from between underscore pairs (I use hyphens as word separators,
so underscores are ONLY used for variable substitutions).

The point is, the conventions you use for file names should (1) avoid
special characters that may get you in trouble with the command line and
(2) allow you the flexibility you need within the constraints of #1.

Unfortunately, that doesn't help with the Cyrillic ё problem. My eXist
procedures were all started around version 1.2, and most of what I do with
eXist is done by running REST bases XQueries. The above naming conventions
work fine internally. XML files are mainly modified via an XQuery process.
I may sometimes touch up a file directly with oXygen or Nova via WebDAV
(only occasionally using eXide). But nothing else touches the XML files, so
there are no problems. My processes were developed over the past decade,
which is probably why my processes have evolved this way. These were the
tools with the least amount of resistance at the time. You have so many
more choices in work flows now.

GUI operating systems have made it much easier to go ahead and use special
characters in file names. A file name that would have caused a corrupt hard
disk in the past is now taken care of in the background so users don't have
to worry about it. So long as you're staying within that system, you should
be okay. Try to integrate that externally and all bets are off. With the
main bridge between eXist and other systems being RESTful XQueries, I know
that I can keep everything consistent up to the output.

Consumers of that output, such as old versions of the Japanese version of
Excel can only handle Shift_JIS characters. I can output in Sift_JIS, but
my data has many names of people with characters outside of that character
set. So clients who insist on using old software get a ? in place of those
characters (吉國 becomes 吉?). It's a known limitation of the client side
software that is a problem. The only work around is for the customer to
upgrade to modern software.

Well, I hope this short history helps in some way.

Take care.

2020年11月27日(金) 1:50 David Birnbaum <dj...@gm...>:

> Dear exist-open (cc Christian, Peter),
>
> Thank you, Christian, for this information, which I had not known about
> previously. If eXist-db installs apps by first unzipping them onto the
> server filesystem in a expathrepo directory (giving the OS a chance to
> normalize the filenames), does this mean that whether those who install my
> app get a composed or decomposed representation of a composite character in
> a filename may depend on their OS? If that is the case, it may not matter
> where the confusion of the two representations happens, since I need it to
> be possible for users on MacOS and other OSs to install the app, and to be
> able to address the files by name over REST using the same HTTP URIs.
>
> In response to Peter's observation, although I can set the locale on my
> own machine, those who install my app may not know how to set a locale, or
> be (reasonably) unable or unwilling to change their setting. My locale is
> set to en_US.UTF-8, and response headers on a REST return show
> "application/xml; charset=UTF-8" as the content type, so I thought it
> should have been able to handle either the composed or decomposed
> representation of Cyrillic ё. "Should have been able" has been a leitmotif
> in this thread, though, and the additional observations from Christian and
> Peter seem to suggest that even should the eXist-db resource management
> interfaces be updated to handle filenames with non-ASCII characters
> robustly (as the WebDAV interface already seems to do), interacting with
> the files by filename using the REST interface may face challenges that
> originate outside eXist-db.
>
> Best,
>
> David
>
> On Wed, Nov 25, 2020 at 11:39 PM Christian Wittern <cwi...@gm...>
> wrote:
>
>> Dear David,
>>
>> Just chiming in on this very specific point:
>>
>> On 26/11/2020 03.05, David Birnbaum wrote:
>> > But for those who are curious: on my Mac, with the data directory left
>> > with the default value /Users/djb/Library/Application
>> > Support/org.exist, packages are unpacked
>> > under /Users/djb/Library/Application Support/org.exist/expathrepo/, so
>> > trying to install the xar file let me see whether ё was represented by
>> > a single composite character or by a base followed by a combining
>> > diacritic.
>>
>> The macOS file system will normalize characters to use decomposed forms
>> of characters that are encoded as pre-composed characters in Unicode as
>> a rule, so in this case the observed fact does not allow any conclusions
>> regarding the handling within eXist or its I/O pipelines.  For that
>> purpose, you would need to use a different OS.
>>
>> All the best,
>>
>> Christian
>>
>>
>>
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>


-- 
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/

Re: [Exist-open] Filenames with awkward characters in eXist-db?

[David B. <dj...@gm...>]

Dear exist-open (cc Christian, Peter),

Thank you, Christian, for this information, which I had not known about
previously. If eXist-db installs apps by first unzipping them onto the
server filesystem in a expathrepo directory (giving the OS a chance to
normalize the filenames), does this mean that whether those who install my
app get a composed or decomposed representation of a composite character in
a filename may depend on their OS? If that is the case, it may not matter
where the confusion of the two representations happens, since I need it to
be possible for users on MacOS and other OSs to install the app, and to be
able to address the files by name over REST using the same HTTP URIs.

In response to Peter's observation, although I can set the locale on my own
machine, those who install my app may not know how to set a locale, or be
(reasonably) unable or unwilling to change their setting. My locale is set
to en_US.UTF-8, and response headers on a REST return show
"application/xml; charset=UTF-8" as the content type, so I thought it
should have been able to handle either the composed or decomposed
representation of Cyrillic ё. "Should have been able" has been a leitmotif
in this thread, though, and the additional observations from Christian and
Peter seem to suggest that even should the eXist-db resource management
interfaces be updated to handle filenames with non-ASCII characters
robustly (as the WebDAV interface already seems to do), interacting with
the files by filename using the REST interface may face challenges that
originate outside eXist-db.

Best,

David

On Wed, Nov 25, 2020 at 11:39 PM Christian Wittern <cwi...@gm...>
wrote:

> Dear David,
>
> Just chiming in on this very specific point:
>
> On 26/11/2020 03.05, David Birnbaum wrote:
> > But for those who are curious: on my Mac, with the data directory left
> > with the default value /Users/djb/Library/Application
> > Support/org.exist, packages are unpacked
> > under /Users/djb/Library/Application Support/org.exist/expathrepo/, so
> > trying to install the xar file let me see whether ё was represented by
> > a single composite character or by a base followed by a combining
> > diacritic.
>
> The macOS file system will normalize characters to use decomposed forms
> of characters that are encoded as pre-composed characters in Unicode as
> a rule, so in this case the observed fact does not allow any conclusions
> regarding the handling within eXist or its I/O pipelines.  For that
> purpose, you would need to use a different OS.
>
> All the best,
>
> Christian
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

Re: [Exist-open] Filenames with awkward characters in eXist-db?

[Hungerburg 13 <pc...@my...>]

David, All, One more observation, that might be of interest:

The /locale/ that the exist-db process runs with makes a difference

Here the java code of "awkward.java":

> import java.net.*;
> class awkward {
>   static String str = "tit[ё]tat";
>   public static void main(String[] args) throws Exception {
>     URI uri = new URI("http", "host", "/"+str, null);
>     System.out.println(uri.normalize().toString());
>     System.out.println(uri.normalize().getRawPath());
>   }
> }

Output changes when run in an 8-bit locale:

> $ javac awkward.java
> $
> $ java awkward
> http://host/tit%5Bё%5Dtat
> /tit%5Bё%5Dtat
> $
> $ LANG=C java awkward
> http://host/tit%5B?%5Dtat
> /tit%5B?%5Dtat

There is now the ominous question mark that David sees! I guess it just 
says, that the character cannot be represented in the chosen encoding. 
No exception thrown, impossible to say, where the substitution occurs.

Peter

Re: [Exist-open] Filenames with awkward characters in eXist-db?

[Christian W. <cwi...@gm...>]

Dear David,

Just chiming in on this very specific point:

On 26/11/2020 03.05, David Birnbaum wrote:
> But for those who are curious: on my Mac, with the data directory left 
> with the default value /Users/djb/Library/Application 
> Support/org.exist, packages are unpacked 
> under /Users/djb/Library/Application Support/org.exist/expathrepo/, so 
> trying to install the xar file let me see whether ё was represented by 
> a single composite character or by a base followed by a combining 
> diacritic. 

The macOS file system will normalize characters to use decomposed forms 
of characters that are encoded as pre-composed characters in Unicode as 
a rule, so in this case the observed fact does not allow any conclusions 
regarding the handling within eXist or its I/O pipelines.  For that 
purpose, you would need to use a different OS.

All the best,

Christian

Re: [Exist-open] Programmatically call the view.xql module

[Alasdair D. <ala...@gm...>]

Hi Michael,

The recipes are in XML format, and the XSLT is the means to convert it into the required format. It works well and is easy to maintain.

As I have moved to Nginx, I am now using the try_files option before going to eXist to render the page. It is very quick to return the page.  So, if a new recipe is added that has not had the static part generated, eXists takes care of it without a single issue.

Alasdair 



Sent from my iPhone

> On 25 Nov 2020, at 10:00 pm, Michael Westbay <wes...@ja...> wrote:
> 
> 
> Hi Alisdair,
> 
> I would do everything in XQuery. No need for XSLT. If all of the recipes are in a uniform HTML structure, such as the recipe wrapped in a <div id="recipe">...</div>, that makes it very easy to grab with an XPath. xmldb:store(...) then puts it into your eXist database. Depending on how well the recipe pages are structured, you can extract ingredients and other metadata for storing in metadata of your own definition. Pretty much anything that can be done with XSLT can be done with XQuery/XPath.
> 
> Take care.
> 
> 2020年11月25日(水) 16:26 Jean-Paul Rehr <re...@gm...>:
>> Dear Alisdair, is there a reason not to use eXist's triggers?
>> 
>> https://exist-db.org/exist/apps/doc/triggers
>> 
>> ie. store the exterior source in a specific collection -> triggers XSLT transformation and stores new result (it seems to me any templates you are using can be rewritten into the transformation Xquery itself and avoid going through view at all)?
>> 
>> This might allow finer control of each step, particularly when things go sideways?
>> 
>> Best,
>> JPR
>> 
>> 
>> 
>>> On Wed, Nov 25, 2020 at 7:28 AM Alasdair Dougall <ala...@gm...> wrote:
>>> Hi All,
>>> 
>>> My wife's recipe website has recipes that are mainly static with some dynamic sections.
>>> 
>>> In effect, I can crawl the site to get the pages, but I would rather trigger a xquery on saving a document within given directories to generate html files and to store them in the appropriate directory.  So, is it possible to run a query that would produce similar output to:
>>> 
>>> <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
>>>     <forward url="{$exist:controller}/xquery/{$pagetype}.xql">
>>>         <set-attribute name="indexkey" value="{$exist:resource}"/>
>>>          <set-attribute name="xslt.input" value="model"/>
>>>          <set-attribute name="xslt.stylesheet" value="{concat($exist:root,$exist:controller,"/templates/",$pagetype,".xslt")}"/>
>>>          <set-attribute name="title" value="{request:get-attribute('title')}"/>
>>>          <set-attribute name="keywords" value="{request:get-attribute('keywords')}"/>
>>>      </forward>
>>>   <view>
>>>     <forward servlet="XSLTServlet">
>>>        <set-attribute name="xslt.uri" value="{request:get-uri()}"/>
>>>        <clear-attribute name="xslt.input"/>
>>>     </forward>
>>>     <forward url="{$exist:controller}/modules/view.xql"/>
>>>   </view>
>>>   <error-handler>
>>>   .....
>>>   </error-handler>
>>> </dispatch>
>>> 
>>> So the sequence of calls are:
>>> 
>>> 1.  Call pagetype.xql as in recipe.xql
>>> 2.  Transform withXSLTServlet using a forward to pagetype.xslt, i.e. recipe.xslt
>>> 3.  Call view to expand templating entities
>>> 
>>> Thanks in advance,
>>> 
>>> Alasdair
>>> 
>>> _______________________________________________
>>> Exist-open mailing list
>>> Exi...@li...
>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
> 
> 
> -- 
> Michael Westbay
> Writer/System Administrator
> http://www.japanesebaseball.com/

Re: [Exist-open] Programmatically call the view.xql module

[Alasdair D. <ala...@gm...>]

Hi Jean-Paul,

Yes, I had heard there may be some issues around triggers. I currently generate the files using curl and a xquery to generate the calls to run manually.

Guess more experimentation required.

Alasdair 

Sent from my iPhone

> On 25 Nov 2020, at 5:25 pm, Jean-Paul Rehr <re...@gm...> wrote:
> 
> 
> Dear Alisdair, is there a reason not to use eXist's triggers?
> 
> https://exist-db.org/exist/apps/doc/triggers
> 
> ie. store the exterior source in a specific collection -> triggers XSLT transformation and stores new result (it seems to me any templates you are using can be rewritten into the transformation Xquery itself and avoid going through view at all)?
> 
> This might allow finer control of each step, particularly when things go sideways?
> 
> Best,
> JPR
> 
> 
> 
>> On Wed, Nov 25, 2020 at 7:28 AM Alasdair Dougall <ala...@gm...> wrote:
>> Hi All,
>> 
>> My wife's recipe website has recipes that are mainly static with some dynamic sections.
>> 
>> In effect, I can crawl the site to get the pages, but I would rather trigger a xquery on saving a document within given directories to generate html files and to store them in the appropriate directory.  So, is it possible to run a query that would produce similar output to:
>> 
>> <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
>>     <forward url="{$exist:controller}/xquery/{$pagetype}.xql">
>>         <set-attribute name="indexkey" value="{$exist:resource}"/>
>>          <set-attribute name="xslt.input" value="model"/>
>>          <set-attribute name="xslt.stylesheet" value="{concat($exist:root,$exist:controller,"/templates/",$pagetype,".xslt")}"/>
>>          <set-attribute name="title" value="{request:get-attribute('title')}"/>
>>          <set-attribute name="keywords" value="{request:get-attribute('keywords')}"/>
>>      </forward>
>>   <view>
>>     <forward servlet="XSLTServlet">
>>        <set-attribute name="xslt.uri" value="{request:get-uri()}"/>
>>        <clear-attribute name="xslt.input"/>
>>     </forward>
>>     <forward url="{$exist:controller}/modules/view.xql"/>
>>   </view>
>>   <error-handler>
>>   .....
>>   </error-handler>
>> </dispatch>
>> 
>> So the sequence of calls are:
>> 
>> 1.  Call pagetype.xql as in recipe.xql
>> 2.  Transform withXSLTServlet using a forward to pagetype.xslt, i.e. recipe.xslt
>> 3.  Call view to expand templating entities
>> 
>> Thanks in advance,
>> 
>> Alasdair
>> 
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] Filenames with awkward characters in eXist-db?

[David B. <dj...@gm...>]

Dear exist-open (cc Joe),

Thank you for pointing out this lingering issue. I've updated it with a
comment that summarizes and links to the recent exist-open discussion, as a
way of making it easily accessible from the issue, since the mailing-list
discussion provides new and updated information about some of the details.

Best,

David

On Wed, Nov 25, 2020 at 4:03 PM Joe Wicentowski <jo...@gm...> wrote:

> Hi David and all,
>
> I meant to chime in earlier and link to a discussion in the issue tracker
> on this perennial topic:
>
>   https://github.com/eXist-db/exist/issues/1824#issuecomment-382231963
>
> I second your call for a comprehensive review of how characters in
> resource & collection names are handled in eXist and its various
> interfaces. In my experience, eXist's WebDAV interface does the best job
> storing and retrieving special characters in resource & collection names -
> and thanks to Dannes for his hard work in this area. I've tried at various
> times to fix problems in eXide (e.g.,
> https://github.com/eXist-db/eXide/issues/185), but decided this was
> futile until the issue in the eXist core was addressed.
>
> Joe
>
> On Wed, Nov 25, 2020 at 1:06 PM David Birnbaum <dj...@gm...> wrote:
>
>> Dear exist-open (cc Peter, Eduard, Michael),
>>
>> By way of context, for those just joining the thread, I've been
>> installing my files with unfriendly filenames (names that include percent
>> signs and square brackets, which would not be valid URI components unless
>> they were precent-escaped first) through the <oXygen/> WebDAV interface,
>> and today I tried installing a xar file of the app with the package
>> manager. The installation broke with the following error message (shown in
>> the package manager interface): "Invalid URI: Malformed escape pair at
>> index 7: verb-1a%[x]%-forms.xml". This is, indeed, an invalid URI, but it
>> isn't an invalid filename, and since package manager apparently uses URIs
>> for installation, I would have expected it to percent-escape the percent
>> signs and square brackets as part of the installation routine, instead of
>> assuming that the filename can be repurposed as URI without modification.
>>
>> I tried installing the xar file in order to follow up on Peter's
>> suggestion, below, that I examine how ё is represented in a filename on the
>> file system after installation, and it revealed a relevant issue earlier in
>> my pipeline that I had overlooked. This paragraph is a digression in the
>> context of my exploration of eXist-db file management, so readers should
>> feel free to skip to the next paragraph. But for those who are curious: on
>> my Mac, with the data directory left with the default
>> value /Users/djb/Library/Application Support/org.exist, packages are
>> unpacked under /Users/djb/Library/Application
>> Support/org.exist/expathrepo/, so trying to install the xar file let me see
>> whether ё was represented by a single composite character or by a base
>> followed by a combining diacritic. Although my original data, which I used
>> earlier in my development pipeline to create the output filenames, has the
>> single composite character, the filename that gets unpacked during
>> installation has the two-character base+diacritic representation. I don't
>> think this discovery fully explains the REST issue that I reported, since I
>> had tried accessing the file with both the one-character and the
>> two-character representations in REST and received an error in both cases.
>> But this experiment did lead to the discovery that the two-character
>> representation in the filename seems to be created early in the pipeline
>> (that is, it gets into the xar file because it was in the xml file when I
>> added it to the eXist-db app), so that I read data with the one-character
>> representation at the beginning of my project, long before eXist-db comes
>> into the picture; use it to construct a filename; and wind up with a
>> filename that has the two-character representation, which then gets passed
>> along to my eXist-db app. That isn't an eXist-db issue, obviously, and I
>> would want to track it down and deal with it in order to run a more
>> controlled and orderly test of how the eXist-db REST interface deals with
>> the file.
>>
>> Meanwhile, though, that the package manager makes the (apparently
>> undocumented, or am I reading carelessly?) assumption that filenames can be
>> used to compose URIs without percent-encoding them first is a deal-breaker.
>> Specifically, if package manager were not a requirement, I could grumble a
>> bit about the limitations of the Java admin client and eXide file manager
>> and then work around them by using the <oXygen/> WebDAV interface. But
>> that's a developer's perspective, and users of my app need to be able to
>> install it with package manager. This discovery about filename restrictions
>> within package manager, then, seems to take me back to the beginning, that
>> is, to the unwelcome conclusion that this is a losing battle, and I need to
>> give up on my original hope of using self-documenting filenames and instead
>> distort my filenames by avoiding the characters that pose challenges for
>> the eXist-db file management interfaces.
>>
>> I understand why those who developed the Java admin client, eXide file
>> manager, package manager, and perhaps also the REST interface might
>> reasonably not have anticipated that eXist-db users would have a need for
>> such unusual filenames, and might therefore either 1) not have been aware
>> of the issue (and accidentally not created tests for it) or 2) knew about
>> it but concluded that nobody would have this need, and that the problem was
>> therefore only theoretical (that is, they may have thought it reasonable to
>> decide not to fix it). Over the past couple of weeks I've learned a lot
>> that I didn't know previously about eXist-db file management, and alongside
>> this knowledge gained I think the takeaway, given that package-manager
>> functionality is a strict requirement, is that I should give up and change
>> my filenames. I would like to see better filename ~ URI round-tripping
>> within the various eXist-db file management interfaces, but I do not have
>> the programming skills to write that code myself, and I appreciate that my
>> filename needs 1) are an edge case that most users won't care about, and 2)
>> have a work-around in that I can change my filenames, which means that
>> fixing this behavior is, for understandable reasons, unlikely to be a high
>> priority. I am grateful for the help and advice that I've received on this
>> list during my explorations, and especially the generous contributions by
>> Peter, Eduard, and Michael.
>>
>> Sincerely,
>>
>> David
>>
>> On Tue, Nov 24, 2020 at 5:21 PM Hungerburg 13 <pc...@my...> wrote:
>>
>>> Am 24.11.20 um 22:41 schrieb David Birnbaum:
>>> > Dear exist-open (cc Peter, Eduard),
>>> >
>>> > Thank you, Peter, for the follow-up. I managed to upload a file called
>>> > "verb-1a%[x]%-forms.xml" into my eXist-db instance by using the
>>> > <oXygen/> interface, and when I tried:
>>> >
>>> >
>>> http://localhost:8080/exist/rest/db/apps/cz/xml/by-form/verb-1a%25%5Bx%5D%25-forms.xml
>>> >
>>> > in Chrome, Firefox, and curl it returned the resource.
>>>
>>> Cool, that would say, that webdav and rest interfaces are compatible, at
>>> least in regard to URI special characters within the ASCII range.
>>>
>>> > But the error I
>>> > reported (in both browsers and curl) with "verb-1a, ё.xml" is still
>>> > there. When I try to retrieve "verb-1a, ё.xml" with:q >
>>>
>>> http://localhost:8080/exist/rest/db/apps/cz/xml/by-form/verb-1a%2C%20%D1%91-forms.xml
>>>
>>> unicode letter ё is UTF-8 two bytes: 0xD1 0x91 - looks legit - When I
>>> modify the little java prog from the previous mail, it will be echoed as
>>> "ё", not escaped.
>>>
>>> I cant help out any further, perhaps to debug further, would be good to
>>> upload "ё.txt" file, and then look how it will show in the local
>>> filsystem on the server?
>>>
>>> Peter
>>>
>>> PS: Do not do any of this on a live system!
>>>
>>>
>>> _______________________________________________
>>> Exist-open mailing list
>>> Exi...@li...
>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>>
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>
>

Re: [Exist-open] Filenames with awkward characters in eXist-db?

[Joe W. <jo...@gm...>]

Hi David and all,

I meant to chime in earlier and link to a discussion in the issue tracker
on this perennial topic:

  https://github.com/eXist-db/exist/issues/1824#issuecomment-382231963

I second your call for a comprehensive review of how characters in resource
& collection names are handled in eXist and its various interfaces. In my
experience, eXist's WebDAV interface does the best job storing and
retrieving special characters in resource & collection names - and thanks
to Dannes for his hard work in this area. I've tried at various times to
fix problems in eXide (e.g., https://github.com/eXist-db/eXide/issues/185),
but decided this was futile until the issue in the eXist core was addressed.

Joe

On Wed, Nov 25, 2020 at 1:06 PM David Birnbaum <dj...@gm...> wrote:

> Dear exist-open (cc Peter, Eduard, Michael),
>
> By way of context, for those just joining the thread, I've been installing
> my files with unfriendly filenames (names that include percent signs and
> square brackets, which would not be valid URI components unless they were
> precent-escaped first) through the <oXygen/> WebDAV interface, and today I
> tried installing a xar file of the app with the package manager. The
> installation broke with the following error message (shown in the package
> manager interface): "Invalid URI: Malformed escape pair at index 7:
> verb-1a%[x]%-forms.xml". This is, indeed, an invalid URI, but it isn't an
> invalid filename, and since package manager apparently uses URIs for
> installation, I would have expected it to percent-escape the percent signs
> and square brackets as part of the installation routine, instead of
> assuming that the filename can be repurposed as URI without modification.
>
> I tried installing the xar file in order to follow up on Peter's
> suggestion, below, that I examine how ё is represented in a filename on the
> file system after installation, and it revealed a relevant issue earlier in
> my pipeline that I had overlooked. This paragraph is a digression in the
> context of my exploration of eXist-db file management, so readers should
> feel free to skip to the next paragraph. But for those who are curious: on
> my Mac, with the data directory left with the default
> value /Users/djb/Library/Application Support/org.exist, packages are
> unpacked under /Users/djb/Library/Application
> Support/org.exist/expathrepo/, so trying to install the xar file let me see
> whether ё was represented by a single composite character or by a base
> followed by a combining diacritic. Although my original data, which I used
> earlier in my development pipeline to create the output filenames, has the
> single composite character, the filename that gets unpacked during
> installation has the two-character base+diacritic representation. I don't
> think this discovery fully explains the REST issue that I reported, since I
> had tried accessing the file with both the one-character and the
> two-character representations in REST and received an error in both cases.
> But this experiment did lead to the discovery that the two-character
> representation in the filename seems to be created early in the pipeline
> (that is, it gets into the xar file because it was in the xml file when I
> added it to the eXist-db app), so that I read data with the one-character
> representation at the beginning of my project, long before eXist-db comes
> into the picture; use it to construct a filename; and wind up with a
> filename that has the two-character representation, which then gets passed
> along to my eXist-db app. That isn't an eXist-db issue, obviously, and I
> would want to track it down and deal with it in order to run a more
> controlled and orderly test of how the eXist-db REST interface deals with
> the file.
>
> Meanwhile, though, that the package manager makes the (apparently
> undocumented, or am I reading carelessly?) assumption that filenames can be
> used to compose URIs without percent-encoding them first is a deal-breaker.
> Specifically, if package manager were not a requirement, I could grumble a
> bit about the limitations of the Java admin client and eXide file manager
> and then work around them by using the <oXygen/> WebDAV interface. But
> that's a developer's perspective, and users of my app need to be able to
> install it with package manager. This discovery about filename restrictions
> within package manager, then, seems to take me back to the beginning, that
> is, to the unwelcome conclusion that this is a losing battle, and I need to
> give up on my original hope of using self-documenting filenames and instead
> distort my filenames by avoiding the characters that pose challenges for
> the eXist-db file management interfaces.
>
> I understand why those who developed the Java admin client, eXide file
> manager, package manager, and perhaps also the REST interface might
> reasonably not have anticipated that eXist-db users would have a need for
> such unusual filenames, and might therefore either 1) not have been aware
> of the issue (and accidentally not created tests for it) or 2) knew about
> it but concluded that nobody would have this need, and that the problem was
> therefore only theoretical (that is, they may have thought it reasonable to
> decide not to fix it). Over the past couple of weeks I've learned a lot
> that I didn't know previously about eXist-db file management, and alongside
> this knowledge gained I think the takeaway, given that package-manager
> functionality is a strict requirement, is that I should give up and change
> my filenames. I would like to see better filename ~ URI round-tripping
> within the various eXist-db file management interfaces, but I do not have
> the programming skills to write that code myself, and I appreciate that my
> filename needs 1) are an edge case that most users won't care about, and 2)
> have a work-around in that I can change my filenames, which means that
> fixing this behavior is, for understandable reasons, unlikely to be a high
> priority. I am grateful for the help and advice that I've received on this
> list during my explorations, and especially the generous contributions by
> Peter, Eduard, and Michael.
>
> Sincerely,
>
> David
>
> On Tue, Nov 24, 2020 at 5:21 PM Hungerburg 13 <pc...@my...> wrote:
>
>> Am 24.11.20 um 22:41 schrieb David Birnbaum:
>> > Dear exist-open (cc Peter, Eduard),
>> >
>> > Thank you, Peter, for the follow-up. I managed to upload a file called
>> > "verb-1a%[x]%-forms.xml" into my eXist-db instance by using the
>> > <oXygen/> interface, and when I tried:
>> >
>> >
>> http://localhost:8080/exist/rest/db/apps/cz/xml/by-form/verb-1a%25%5Bx%5D%25-forms.xml
>> >
>> > in Chrome, Firefox, and curl it returned the resource.
>>
>> Cool, that would say, that webdav and rest interfaces are compatible, at
>> least in regard to URI special characters within the ASCII range.
>>
>> > But the error I
>> > reported (in both browsers and curl) with "verb-1a, ё.xml" is still
>> > there. When I try to retrieve "verb-1a, ё.xml" with:q >
>>
>> http://localhost:8080/exist/rest/db/apps/cz/xml/by-form/verb-1a%2C%20%D1%91-forms.xml
>>
>> unicode letter ё is UTF-8 two bytes: 0xD1 0x91 - looks legit - When I
>> modify the little java prog from the previous mail, it will be echoed as
>> "ё", not escaped.
>>
>> I cant help out any further, perhaps to debug further, would be good to
>> upload "ё.txt" file, and then look how it will show in the local
>> filsystem on the server?
>>
>> Peter
>>
>> PS: Do not do any of this on a live system!
>>
>>
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>