exist-open — Open discussion

Re: [Exist-open] Lucene search issue with existed 2.1

[Joe W. <jo...@gm...>]

Hi Alem,

The customized stopwords feature I described was added in eXist 1.4.3, so
it should work in 2.1 as well. See
https://github.com/eXist-db/exist/commit/744d104e8bacb17da5d54cc0d17d484601346867
.

Give a close look at the @type attribute on the <param> element though in
the examples added in that commit:
https://github.com/eXist-db/exist/commit/744d104e8bacb17da5d54cc0d17d484601346867#diff-35d71d352dd07d7f78245dc1936346e5a2db7a1908b593eda31f4e4f427d1b8dR341-R364
.

Notice that it was <param type="java.util.Set">. In contrast, in the
current documentation I linked to, it's
<param type="org.apache.lucene.analysis.util.CharArraySet">. So perhaps for
2.1 you should use "java.util.Set."

Joe

On Fri, Mar 18, 2022 at 10:38 AM Areki, Alem <aa...@ri...> wrote:

> Thanks, Joe.
>
>
>
> It works with my local existdb version 4.1 but not version 2.1.
>
>
>
> thanks
>
>
>
> --
>
> Alem T. Areki
>
> Senior Web Developer – Web Services
>
> University of Richmond
>
> phone: 804-289-8899 | Fax: 804-289-8988
>
>
>
> Please send technical questions or requests to we...@ri... and
> not to individuals if at all possible.
>
>
>
> Thank you for your email!  Someone from Web Services will contact you
> regarding your request by the end of the next business day.
>
>
>
> To submit technical support requests or defect reports please use the
> SpiderTechNet System:
>
>
> https://spidertechnet.richmond.edu/TDClient/1955/Portal/Requests/ServiceCatalog?CategoryID=12767
>
>
>
> *For other Technology Questions and Issues visit: *
> *https://spidertechnet.richmond.edu/*
> <https://spidertechnet.richmond.edu/>
>
>
>
>
>
>
>
> *From: *Joe Wicentowski <jo...@gm...>
> *Date: *Friday, March 18, 2022 at 8:57 AM
> *To: *"Areki, Alem" <aa...@ri...>
> *Cc: *"exi...@li..." <exi...@li...
> >
> *Subject: *Re: [Exist-open] Lucene search issue with existed 2.1
>
>
>
> *External Email:* Use caution in opening links, attachments, and buying
> gift cards.
>
>
>
> Hi Alem,
>
>
>
> The words "to" and "the" are included in Lucene's default list of
> stopwords. They're dropped when indexing documents and ignored when
> querying them. To ensure these words are indexed, you have to either
> customize the list of stopwords or explicitly eliminate all stopwords,
> adding the following <param> element to the <analyzer> element in your
> .xconf file:
>
>
>
> <analyzer class="org.apache.lucene.analysis.standard.StandardAnalyzer">
>     <!-- Specify stop words - or remove them entirely -->
>     <param name="stopwords" type=
> "org.apache.lucene.analysis.util.CharArraySet">
>         <!--<value>the</value>-->
>     </param>
> </analyzer>
>
>
>
> For the list of words treated as stopwords by default in Lucene, see
> https://markmail.org/message/wxmtjzbskgrj2cug. Place any stopwords you
> want to keep in <value> elements. By omitting any <value> elements (as
> shown above), the you've removed all stopwords. For the current eXist docs
> on configuring Lucene's stopword facility, see
> https://exist-db.org/exist/apps/doc/lucene#conf.
>
>
>
> Joe
>
>
>
> On Thu, Mar 17, 2022 at 5:59 PM Areki, Alem <aa...@ri...> wrote:
>
> Hi all,
>
>
>
> I am having a problem Lucene searching with stop words in between the
> phrase keywords
>
>    - The title of the article to be searched: “*Where to watch: Catch the
>    Spiders in March Madness*”
>    - A searching phrase like “*March Madness*” works but phrases like “*Where
>    to Watch*” or “*Catch the Spiders*” don’t work.
>    - Searching the phrases doesn’t work has stopwords in between them.
>
>
>
> Does anyone have any clue why I am having this issue?
>
>
>
> Thanks
>
> Alem
>
> --
>
> Alem T. Areki
>
> Senior Web Developer – Web Services
>
> University of Richmond
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>
>

Re: [Exist-open] Lucene search issue with existed 2.1

[Areki, A. <aa...@ri...>]

Thanks, Joe.

It works with my local existdb version 4.1 but not version 2.1.

thanks

--
Alem T. Areki
Senior Web Developer – Web Services
University of Richmond
phone: 804-289-8899 | Fax: 804-289-8988

Please send technical questions or requests to we...@ri...<mailto:we...@ri...> and not to individuals if at all possible.

Thank you for your email!  Someone from Web Services will contact you regarding your request by the end of the next business day.

To submit technical support requests or defect reports please use the SpiderTechNet System:
https://spidertechnet.richmond.edu/TDClient/1955/Portal/Requests/ServiceCatalog?CategoryID=12767

For other Technology Questions and Issues visit: https://spidertechnet.richmond.edu/



From: Joe Wicentowski <jo...@gm...>
Date: Friday, March 18, 2022 at 8:57 AM
To: "Areki, Alem" <aa...@ri...>
Cc: "exi...@li..." <exi...@li...>
Subject: Re: [Exist-open] Lucene search issue with existed 2.1

External Email: Use caution in opening links, attachments, and buying gift cards.

Hi Alem,

The words "to" and "the" are included in Lucene's default list of stopwords. They're dropped when indexing documents and ignored when querying them. To ensure these words are indexed, you have to either customize the list of stopwords or explicitly eliminate all stopwords, adding the following <param> element to the <analyzer> element in your .xconf file:

<analyzer class="org.apache.lucene.analysis.standard.StandardAnalyzer">
    <!-- Specify stop words - or remove them entirely -->
    <param name="stopwords" type="org.apache.lucene.analysis.util.CharArraySet">
        <!--<value>the</value>-->
    </param>
</analyzer>

For the list of words treated as stopwords by default in Lucene, see https://markmail.org/message/wxmtjzbskgrj2cug. Place any stopwords you want to keep in <value> elements. By omitting any <value> elements (as shown above), the you've removed all stopwords. For the current eXist docs on configuring Lucene's stopword facility, see https://exist-db.org/exist/apps/doc/lucene#conf.

Joe

On Thu, Mar 17, 2022 at 5:59 PM Areki, Alem <aa...@ri...<mailto:aa...@ri...>> wrote:
Hi all,

I am having a problem Lucene searching with stop words in between the phrase keywords

  *   The title of the article to be searched: “Where to watch: Catch the Spiders in March Madness”
  *   A searching phrase like “March Madness” works but phrases like “Where to Watch” or “Catch the Spiders” don’t work.
  *   Searching the phrases doesn’t work has stopwords in between them.

Does anyone have any clue why I am having this issue?

Thanks
Alem
--
Alem T. Areki
Senior Web Developer – Web Services
University of Richmond

_______________________________________________
Exist-open mailing list
Exi...@li...<mailto:Exi...@li...>
https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] Lucene search issue with existed 2.1

[Juri L. <ju...@ex...>]

More recent versions of exist, I believe since 5.3.0, allow you to have
separate analyzers for indexing and querying.
Index without stop-words but query with them is a good combination. That
way non-phrase searches would not regard stop-words; as one usually wants,
but phrase searches can still find them as they are part of the index do
include them.
It is at least worth a try.

On 18. March 2022 at 13:57:25, Joe Wicentowski (jo...@gm...) wrote:

> Hi Alem,
>
> The words "to" and "the" are included in Lucene's default list of
> stopwords. They're dropped when indexing documents and ignored when
> querying them. To ensure these words are indexed, you have to either
> customize the list of stopwords or explicitly eliminate all stopwords,
> adding the following <param> element to the <analyzer> element in your
> .xconf file:
>
> <analyzer class="org.apache.lucene.analysis.standard.StandardAnalyzer">
>>     <!-- Specify stop words - or remove them entirely -->
>>     <param name="stopwords" type=
>> "org.apache.lucene.analysis.util.CharArraySet">
>>         <!--<value>the</value>-->
>>     </param>
>> </analyzer>
>
>
> For the list of words treated as stopwords by default in Lucene, see
> https://markmail.org/message/wxmtjzbskgrj2cug. Place any stopwords you
> want to keep in <value> elements. By omitting any <value> elements (as
> shown above), the you've removed all stopwords. For the current eXist docs
> on configuring Lucene's stopword facility, see
> https://exist-db.org/exist/apps/doc/lucene#conf.
>
> Joe
>
> On Thu, Mar 17, 2022 at 5:59 PM Areki, Alem <aa...@ri...> wrote:
>
>> Hi all,
>>
>>
>>
>> I am having a problem Lucene searching with stop words in between the
>> phrase keywords
>>
>>    - The title of the article to be searched: “*Where to watch: Catch
>>    the Spiders in March Madness*”
>>    - A searching phrase like “*March Madness*” works but phrases like “*Where
>>    to Watch*” or “*Catch the Spiders*” don’t work.
>>    - Searching the phrases doesn’t work has stopwords in between them.
>>
>>
>>
>> Does anyone have any clue why I am having this issue?
>>
>>
>>
>> Thanks
>>
>> Alem
>>
>> --
>>
>> Alem T. Areki
>>
>> Senior Web Developer – Web Services
>>
>> University of Richmond
>>
>>
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

Re: [Exist-open] Lucene search issue with existed 2.1

[Joe W. <jo...@gm...>]

Hi Alem,

The words "to" and "the" are included in Lucene's default list of
stopwords. They're dropped when indexing documents and ignored when
querying them. To ensure these words are indexed, you have to either
customize the list of stopwords or explicitly eliminate all stopwords,
adding the following <param> element to the <analyzer> element in your
.xconf file:

<analyzer class="org.apache.lucene.analysis.standard.StandardAnalyzer">
>     <!-- Specify stop words - or remove them entirely -->
>     <param name="stopwords" type=
> "org.apache.lucene.analysis.util.CharArraySet">
>         <!--<value>the</value>-->
>     </param>
> </analyzer>


For the list of words treated as stopwords by default in Lucene, see
https://markmail.org/message/wxmtjzbskgrj2cug. Place any stopwords you want
to keep in <value> elements. By omitting any <value> elements (as shown
above), the you've removed all stopwords. For the current eXist docs on
configuring Lucene's stopword facility, see
https://exist-db.org/exist/apps/doc/lucene#conf.

Joe

On Thu, Mar 17, 2022 at 5:59 PM Areki, Alem <aa...@ri...> wrote:

> Hi all,
>
>
>
> I am having a problem Lucene searching with stop words in between the
> phrase keywords
>
>    - The title of the article to be searched: “*Where to watch: Catch the
>    Spiders in March Madness*”
>    - A searching phrase like “*March Madness*” works but phrases like “*Where
>    to Watch*” or “*Catch the Spiders*” don’t work.
>    - Searching the phrases doesn’t work has stopwords in between them.
>
>
>
> Does anyone have any clue why I am having this issue?
>
>
>
> Thanks
>
> Alem
>
> --
>
> Alem T. Areki
>
> Senior Web Developer – Web Services
>
> University of Richmond
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

[Exist-open] Lucene search issue with existed 2.1

[Areki, A. <aa...@ri...>]

Hi all,

I am having a problem Lucene searching with stop words in between the phrase keywords

  *   The title of the article to be searched: “Where to watch: Catch the Spiders in March Madness”
  *   A searching phrase like “March Madness” works but phrases like “Where to Watch” or “Catch the Spiders” don’t work.
  *   Searching the phrases doesn’t work has stopwords in between them.

Does anyone have any clue why I am having this issue?

Thanks
Alem
--
Alem T. Areki
Senior Web Developer – Web Services
University of Richmond

Re: [Exist-open] Page rendering

[Kevin B. <kev...@xp...>]

Yes my bad. I thought (well because I am part owner of RenderX) that you were looking at rendering the page and not just scraping it.

 

…. When your favorite tool is a hammer, everything looks like a nail ….

 

Kevin

 

From: Roy Walter <gar...@ya...> 
Sent: Wednesday, March 16, 2022 12:15 PM
To: Kevin Brown <kev...@xp...>; wes...@ja...; Chris Wallace <kit...@gm...>
Cc: exist-open <exi...@li...>
Subject: Re: [Exist-open] Page rendering

 

Hi Chris,,

 

That's what I  meant. Just grab (scrape!) the page HTML in nodejs and pass it back to eXist for processing. You shouldn't have any cookie issues if you configure puppeteer the right way. See puppeteer-extra. Just let you existing scraper in eXist do its thing with what you get back from nodejs.

 

R.

 

 

 

On Wednesday, 16 March 2022, 18:39:45 GMT, Chris Wallace < <mailto:kit...@gm...> kit...@gm...> wrote: 

 

 

Thanks guys for the help.   I fancy leaving the page scaping in exist and simply returning the full html as a response 

from pupetteer - there may be issues with cookies I expect.

  

Kevin, a page scraper doesn't need CSS to be rendered, it's only looking at the HTML, not the screen image.  Puppeteer does render CSS to form an image anyway doesn't it -  I thought its  main purpose was to automate unit testing by generating screen shots, with possibly different screen/browser configurations?

 

Chris

 

On Wed, Mar 16, 2022 at 6:11 PM Kevin Brown < <mailto:kev...@xp...> kev...@xp...> wrote:

The problem with that is that the CSS is not resolved. Hence I pointed you the the @cloudformatter. You can examine the JS in those pages that actually resolves the CSS and yields full HTML.

 

Kevin

 

 

 

Sent from my Verizon, Samsung Galaxy smartphone

 

 

-------- Original message --------

From: Roy Walter < <mailto:gar...@ya...> gar...@ya...> 

Date: 3/16/22 4:38 AM (GMT-08:00) 

To:  <mailto:kev...@xp...> kev...@xp..., Chris Wallace < <mailto:kit...@gm...> kit...@gm...> 

Cc: exist-open < <mailto:exi...@li...> exi...@li...> 

Subject: Re: [Exist-open] Page rendering 

 

Hi Chris,

 

You can do that. (We do.) 

 

Configure a web server in nodejs and send a GET request from eXist with the URL as a parameter.Then do the scraping in puppeteer/Chromium. You can either process the returned html/xml payload directly or PUT the file to eXist from nodejs.

 

R.

 

 

On Tuesday, 15 March 2022, 23:12:38 GMT, Chris Wallace < <mailto:kit...@gm...> kit...@gm...> wrote: 

 

 

Thanks Roy - yes that's just the ticket .  Actually I was wondering if one could set up an HTTP service based on puppeteer which took a url, rendered the page and returned the full page HTML so that the current scraper would be unchanged.  Surprised there isn't a service already to do just that - perhaps there is.

 

Kevin, it's just a case of the common problem nowadays of scraping the Javascript-generated web we see these days.  XQuery is great for scraping HTML pages but Javascript-generated pages have to be rendered by a browser engine before we get the displayed HTML.  I wondered if anyone had wrapped a browser engine for use in exist-db.   Perhaps mentioning PDFs was confusing - I parse PDF documents using the Content Extraction Module which wraps Apache Tika ( does a good job although it does miss spaces sometimes which is a problem for text analysis) and this is unchanged.

 

Chris

 

On Tue, Mar 15, 2022 at 8:23 PM Roy Walter < <mailto:gar...@ya...> gar...@ya...> wrote:

Can't be done.

 

nodejs/puppeteer/Chromium is the way.

 

-- Roy

 

 

 

On Tuesday, 15 March 2022, 19:49:28 GMT, Chris Wallace < <mailto:kit...@gm...> kit...@gm...> wrote: 

 

 

I developed an interface to the Bristol City Planning portal which relies on page-scraping and PDF parsing:

 

 <https://bristoltrees.space/Planning/map> https://bristoltrees.space/Planning/map

 

The latest version of the software (by iDox) has made the change that so many sites have done, generating pages in JavaScript.  This would require first rendering each page and scraping the result.  Any thoughts on doing this in exist_db?

 

And no, there is no public API that I can find

 

Chris

_______________________________________________
Exist-open mailing list
 <mailto:Exi...@li...> Exi...@li...
 <https://lists.sourceforge.net/lists/listinfo/exist-open> https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] Page rendering

[Roy W. <gar...@ya...>]

 Hi Chris,,
That's what I  meant. Just grab (scrape!) the page HTML in nodejs and pass it back to eXist for processing. You shouldn't have any cookie issues if you configure puppeteer the right way. See puppeteer-extra. Just let you existing scraper in eXist do its thing with what you get back from nodejs.
R.
 

    On Wednesday, 16 March 2022, 18:39:45 GMT, Chris Wallace <kit...@gm...> wrote:  
 
 Thanks guys for the help.   I fancy leaving the page scaping in exist and simply returning the full html as a response from pupetteer - there may be issues with cookies I expect.  Kevin, a page scraper doesn't need CSS to be rendered, it's only looking at the HTML, not the screen image.  Puppeteer does render CSS to form an image anyway doesn't it -  I thought its  main purpose was to automate unit testing by generating screen shots, with possibly different screen/browser configurations?

Chris
On Wed, Mar 16, 2022 at 6:11 PM Kevin Brown <kev...@xp...> wrote:

The problem with that is that the CSS is not resolved. Hence I pointed you the the @cloudformatter. You can examine the JS in those pages that actually resolves the CSS and yields full HTML.
Kevin


Sent from my Verizon, Samsung Galaxy smartphone

-------- Original message --------From: Roy Walter <gar...@ya...> Date: 3/16/22 4:38 AM (GMT-08:00) To: kev...@xp..., Chris Wallace <kit...@gm...> Cc: exist-open <exi...@li...> Subject: Re: [Exist-open] Page rendering 
 Hi Chris,
You can do that. (We do.) 

Configure a web server in nodejs and send a GET request from eXist with the URL as a parameter.Then do the scraping in puppeteer/Chromium. You can either process the returned html/xml payload directly or PUT the file to eXist from nodejs.
R.

    On Tuesday, 15 March 2022, 23:12:38 GMT, Chris Wallace <kit...@gm...> wrote:  
 
 Thanks Roy - yes that's just the ticket .  Actually I was wondering if one could set up an HTTP service based on puppeteer which took a url, rendered the page and returned the full page HTML so that the current scraper would be unchanged.  Surprised there isn't a service already to do just that - perhaps there is.
Kevin, it's just a case of the common problem nowadays of scraping the Javascript-generated web we see these days.  XQuery is great for scraping HTML pages but Javascript-generated pages have to be rendered by a browser engine before we get the displayed HTML.  I wondered if anyone had wrapped a browser engine for use in exist-db.   Perhaps mentioning PDFs was confusing - I parse PDF documents using the Content Extraction Module which wraps Apache Tika ( does a good job although it does miss spaces sometimes which is a problem for text analysis) and this is unchanged.
Chris
On Tue, Mar 15, 2022 at 8:23 PM Roy Walter <gar...@ya...> wrote:

 Can't be done.
nodejs/puppeteer/Chromium is the way.
-- Roy



    On Tuesday, 15 March 2022, 19:49:28 GMT, Chris Wallace <kit...@gm...> wrote:  
 
 I developed an interface to the Bristol City Planning portal which relies on page-scraping and PDF parsing:
https://bristoltrees.space/Planning/map
The latest version of the software (by iDox) has made the change that so many sites have done, generating pages in JavaScript.  This would require first rendering each page and scraping the result.  Any thoughts on doing this in exist_db?
And no, there is no public API that I can find
Chris_______________________________________________
Exist-open mailing list
Exi...@li...
https://lists.sourceforge.net/lists/listinfo/exist-open
  
  
  

Re: [Exist-open] Page rendering

[Chris W. <kit...@gm...>]

Thanks guys for the help.   I fancy leaving the page scaping in exist and
simply returning the full html as a response
from pupetteer - there may be issues with cookies I expect.

Kevin, a page scraper doesn't need CSS to be rendered, it's only looking at
the HTML, not the screen image.  Puppeteer does render CSS to form an image
anyway doesn't it -  I thought its  main purpose was to automate unit
testing by generating screen shots, with possibly different screen/browser
configurations?

Chris

On Wed, Mar 16, 2022 at 6:11 PM Kevin Brown <kev...@xp...>
wrote:

> The problem with that is that the CSS is not resolved. Hence I pointed you
> the the @cloudformatter. You can examine the JS in those pages that
> actually resolves the CSS and yields full HTML.
>
> Kevin
>
>
>
> Sent from my Verizon, Samsung Galaxy smartphone
>
>
> -------- Original message --------
> From: Roy Walter <gar...@ya...>
> Date: 3/16/22 4:38 AM (GMT-08:00)
> To: kev...@xp..., Chris Wallace <kit...@gm...>
> Cc: exist-open <exi...@li...>
> Subject: Re: [Exist-open] Page rendering
>
> Hi Chris,
>
> You can do that. (We do.)
>
> Configure a web server in nodejs and send a GET request from eXist with
> the URL as a parameter.Then do the scraping in puppeteer/Chromium. You can
> either process the returned html/xml payload directly or PUT the file to
> eXist from nodejs.
>
> R.
>
>
> On Tuesday, 15 March 2022, 23:12:38 GMT, Chris Wallace <
> kit...@gm...> wrote:
>
>
> Thanks Roy - yes that's just the ticket .  Actually I was wondering if one
> could set up an HTTP service based on puppeteer which took a url, rendered
> the page and returned the full page HTML so that the current scraper would
> be unchanged.  Surprised there isn't a service already to do just that -
> perhaps there is.
>
> Kevin, it's just a case of the common problem nowadays of scraping the
> Javascript-generated web we see these days.  XQuery is great for scraping
> HTML pages but Javascript-generated pages have to be rendered by a browser
> engine before we get the displayed HTML.  I wondered if anyone had wrapped
> a browser engine for use in exist-db.   Perhaps mentioning PDFs was
> confusing - I parse PDF documents using the Content Extraction Module which
> wraps Apache Tika ( does a good job although it does miss spaces sometimes
> which is a problem for text analysis) and this is unchanged.
>
> Chris
>
> On Tue, Mar 15, 2022 at 8:23 PM Roy Walter <gar...@ya...>
> wrote:
>
> Can't be done.
>
> nodejs/puppeteer/Chromium is the way.
>
> -- Roy
>
>
>
> On Tuesday, 15 March 2022, 19:49:28 GMT, Chris Wallace <
> kit...@gm...> wrote:
>
>
> I developed an interface to the Bristol City Planning portal which relies
> on page-scraping and PDF parsing:
>
> https://bristoltrees.space/Planning/map
>
> The latest version of the software (by iDox) has made the change that so
> many sites have done, generating pages in JavaScript.  This would require
> first rendering each page and scraping the result.  Any thoughts on doing
> this in exist_db?
>
> And no, there is no public API that I can find
>
> Chris
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>
>

Re: [Exist-open] Page rendering

[Kevin B. <kev...@xp...>]

The problem with that is that the CSS is not resolved. Hence I pointed you the the @cloudformatter. You can examine the JS in those pages that actually resolves the CSS and yields full HTML.KevinSent from my Verizon, Samsung Galaxy smartphone
-------- Original message --------From: Roy Walter <gar...@ya...> Date: 3/16/22  4:38 AM  (GMT-08:00) To: kev...@xp..., Chris Wallace <kit...@gm...> Cc: exist-open <exi...@li...> Subject: Re: [Exist-open] Page rendering 
        Hi Chris,You can do that. (We do.) Configure a web server in nodejs and send a GET request from eXist with the URL as a parameter.Then do the scraping in puppeteer/Chromium. You can either process the returned html/xml payload directly or PUT the file to eXist from nodejs.R.
        
        
            
                
                
                    On Tuesday, 15 March 2022, 23:12:38 GMT, Chris Wallace <kit...@gm...> wrote:
                
                
                
                Thanks Roy - yes that's just the ticket .  Actually I was wondering if one could set up an HTTP service based on puppeteer which took a url, rendered the page and returned the full page HTML so that the current scraper would be unchanged.  Surprised there isn't a service already to do just that - perhaps there is.Kevin, it's just a case of the common problem nowadays of scraping the Javascript-generated web we see these days.  XQuery is great for scraping HTML pages but Javascript-generated pages have to be rendered by a browser engine before we get the displayed HTML.  I wondered if anyone had wrapped a browser engine for use in exist-db.   Perhaps mentioning PDFs was confusing - I parse PDF documents using the Content Extraction Module which wraps Apache Tika ( does a good job although it does miss spaces sometimes which is a problem for text analysis) and this is unchanged.ChrisOn Tue, Mar 15, 2022 at 8:23 PM Roy Walter <gar...@ya...> wrote:
        Can't be done.nodejs/puppeteer/Chromium is the way.-- Roy
        
        
            
                
                
                    On Tuesday, 15 March 2022, 19:49:28 GMT, Chris Wallace <kit...@gm...> wrote:
                
                
                
                I developed an interface to the Bristol City Planning portal which relies on page-scraping and PDF parsing:https://bristoltrees.space/Planning/mapThe latest version of the software (by iDox) has made the change that so many sites have done, generating pages in JavaScript.  This would require first rendering each page and scraping the result.  Any thoughts on doing this in exist_db?And no, there is no public API that I can findChris
_______________________________________________Exist-open mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/exist-open
            
        

            
        

Re: [Exist-open] Page rendering

[Roy W. <gar...@ya...>]

 Hi Chris,
You can do that. (We do.) 

Configure a web server in nodejs and send a GET request from eXist with the URL as a parameter.Then do the scraping in puppeteer/Chromium. You can either process the returned html/xml payload directly or PUT the file to eXist from nodejs.
R.

    On Tuesday, 15 March 2022, 23:12:38 GMT, Chris Wallace <kit...@gm...> wrote:  
 
 Thanks Roy - yes that's just the ticket .  Actually I was wondering if one could set up an HTTP service based on puppeteer which took a url, rendered the page and returned the full page HTML so that the current scraper would be unchanged.  Surprised there isn't a service already to do just that - perhaps there is.
Kevin, it's just a case of the common problem nowadays of scraping the Javascript-generated web we see these days.  XQuery is great for scraping HTML pages but Javascript-generated pages have to be rendered by a browser engine before we get the displayed HTML.  I wondered if anyone had wrapped a browser engine for use in exist-db.   Perhaps mentioning PDFs was confusing - I parse PDF documents using the Content Extraction Module which wraps Apache Tika ( does a good job although it does miss spaces sometimes which is a problem for text analysis) and this is unchanged.
Chris
On Tue, Mar 15, 2022 at 8:23 PM Roy Walter <gar...@ya...> wrote:

 Can't be done.
nodejs/puppeteer/Chromium is the way.
-- Roy



    On Tuesday, 15 March 2022, 19:49:28 GMT, Chris Wallace <kit...@gm...> wrote:  
 
 I developed an interface to the Bristol City Planning portal which relies on page-scraping and PDF parsing:
https://bristoltrees.space/Planning/map
The latest version of the software (by iDox) has made the change that so many sites have done, generating pages in JavaScript.  This would require first rendering each page and scraping the result.  Any thoughts on doing this in exist_db?
And no, there is no public API that I can find
Chris_______________________________________________
Exist-open mailing list
Exi...@li...
https://lists.sourceforge.net/lists/listinfo/exist-open
  
  

Re: [Exist-open] Page rendering

[Michael W. <wes...@ja...>]

I second this approach. Here is the relevant code needed to do it from the
eXist side:

...
declare variable $pages:GET-PUPPETEER-PAGE := 'get-puppet-page.js';
declare variable $pages:NODE-BIN := '/usr/local/bin/node';
declare variable $pages:NODE-OPTIONS := <options>
  <workingDir>/home/my-user/bin</workingDir>
  <environment>
    <env name="CHROME_DEVEL_SANDBOX"
value="/usr/local/sbin/chrome-devel-sandbox"/>
  </environment>
</options>;
declare variable $pages:STRIP-LINES :=
util:function(xs:QName('local:strip-lines'), 1);
declare function local:strip-lines($text as xs:string) as xs:string {
  replace(replace(replace($text, '<line>', ''), '</line>', '
'), '<line/>', '
')
};
...

declare function pages:puppet-page($url) as xs:string? {

  let $cmd := ($pages:NODE-BIN, $pages:GET-PUPPETEER-PAGE, $url)
  let $raw-page := process:execute($cmd, $pages:NODE-OPTIONS)
  return if (exists($raw-page))
    then util:call($pages:STRIP-LINES, $raw-page)
    else ()
};


The STRIP-LINES stuff is there because process:execute returns each line of
the result with <LINE>contents</LINE>, which needs to be stripped out
before XHTML-izing the rest. This was originally written for eXist 2.x, so
I didn't use pipes in the filter. If I were to rewrite it now, it would
probably be like:

declare function local:strip-lines($text as xs:string) as xs:string {
  $text
  => replace('<line>','')
  => replace('</line'>,'
')
};


Notice that the CR is within the replacement quotes in the second replace.
It helps a lot for readability of the results.

I also have a whole slew of other checks in my version, along with caching
of the results that are not included here.

Hope this helps get you started.

Take care.


2022年3月16日(水) 8:17 Roy Walter via Exist-open <
exi...@li...>:

> Can't be done.
>
> nodejs/puppeteer/Chromium is the way.
>
> -- Roy
>
>
>
> On Tuesday, 15 March 2022, 19:49:28 GMT, Chris Wallace <
> kit...@gm...> wrote:
>
>
> I developed an interface to the Bristol City Planning portal which relies
> on page-scraping and PDF parsing:
>
> https://bristoltrees.space/Planning/map
>
> The latest version of the software (by iDox) has made the change that so
> many sites have done, generating pages in JavaScript.  This would require
> first rendering each page and scraping the result.  Any thoughts on doing
> this in exist_db?
>
> And no, there is no public API that I can find
>
> Chris
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>


-- 
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/

Re: [Exist-open] Page rendering

[Kevin B. <kev...@xp...>]

What is it that you are looking for?

Rendering content to PDF?

Scraping content from PDF?
Both?

 

I think that you need to explain a little more to help us understand what you are looking for exactly … I for one am confused.

 

Kevin Brown
RenderX

 

From: Chris Wallace <kit...@gm...> 
Sent: Tuesday, March 15, 2022 12:49 PM
To: exist-open <Exi...@li...>
Subject: [Exist-open] Page rendering

 

I developed an interface to the Bristol City Planning portal which relies on page-scraping and PDF parsing:

 

https://bristoltrees.space/Planning/map

 

The latest version of the software (by iDox) has made the change that so many sites have done, generating pages in JavaScript.  This would require first rendering each page and scraping the result.  Any thoughts on doing this in exist_db?

 

And no, there is no public API that I can find

 

Chris

Re: [Exist-open] Page rendering

[Kevin B. <kev...@xp...>]

See my system @cloudformatter. But it is “in browser” … but it scrapes the HTML, converts that HTML to XSL FO and uses RenderX to format the page.

Post Javascript.

 

www.cloudformatter.com/css2pdf <http://www.cloudformatter.com/css2pdf> 

 

We have many installations of this sitting on top of eXist.

 

Kevin

 

From: Chris Wallace <kit...@gm...> 
Sent: Tuesday, March 15, 2022 4:14 PM
To: Roy Walter <gar...@ya...>; kev...@xp...
Cc: exist-open <exi...@li...>
Subject: Re: [Exist-open] Page rendering

 

Thanks Roy - yes that's just the ticket .  Actually I was wondering if one could set up an HTTP service based on puppeteer which took a url, rendered the page and returned the full page HTML so that the current scraper would be unchanged.  Surprised there isn't a service already to do just that - perhaps there is.

 

Kevin, it's just a case of the common problem nowadays of scraping the Javascript-generated web we see these days.  XQuery is great for scraping HTML pages but Javascript-generated pages have to be rendered by a browser engine before we get the displayed HTML.  I wondered if anyone had wrapped a browser engine for use in exist-db.   Perhaps mentioning PDFs was confusing - I parse PDF documents using the Content Extraction Module which wraps Apache Tika ( does a good job although it does miss spaces sometimes which is a problem for text analysis) and this is unchanged.

 

Chris

 

On Tue, Mar 15, 2022 at 8:23 PM Roy Walter <gar...@ya... <mailto:gar...@ya...> > wrote:

Can't be done.

 

nodejs/puppeteer/Chromium is the way.

 

-- Roy

 

 

 

On Tuesday, 15 March 2022, 19:49:28 GMT, Chris Wallace <kit...@gm... <mailto:kit...@gm...> > wrote: 

 

 

I developed an interface to the Bristol City Planning portal which relies on page-scraping and PDF parsing:

 

https://bristoltrees.space/Planning/map

 

The latest version of the software (by iDox) has made the change that so many sites have done, generating pages in JavaScript.  This would require first rendering each page and scraping the result.  Any thoughts on doing this in exist_db?

 

And no, there is no public API that I can find

 

Chris

_______________________________________________
Exist-open mailing list
Exi...@li... <mailto:Exi...@li...> 
https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] Page rendering

[Roy W. <gar...@ya...>]

 Can't be done.
nodejs/puppeteer/Chromium is the way.
-- Roy



    On Tuesday, 15 March 2022, 19:49:28 GMT, Chris Wallace <kit...@gm...> wrote:  
 
 I developed an interface to the Bristol City Planning portal which relies on page-scraping and PDF parsing:
https://bristoltrees.space/Planning/map
The latest version of the software (by iDox) has made the change that so many sites have done, generating pages in JavaScript.  This would require first rendering each page and scraping the result.  Any thoughts on doing this in exist_db?
And no, there is no public API that I can find
Chris_______________________________________________
Exist-open mailing list
Exi...@li...
https://lists.sourceforge.net/lists/listinfo/exist-open
  

Re: [Exist-open] Page rendering

[Chris W. <kit...@gm...>]

Thanks Roy - yes that's just the ticket .  Actually I was wondering if one
could set up an HTTP service based on puppeteer which took a url, rendered
the page and returned the full page HTML so that the current scraper would
be unchanged.  Surprised there isn't a service already to do just that -
perhaps there is.

Kevin, it's just a case of the common problem nowadays of scraping the
Javascript-generated web we see these days.  XQuery is great for scraping
HTML pages but Javascript-generated pages have to be rendered by a browser
engine before we get the displayed HTML.  I wondered if anyone had wrapped
a browser engine for use in exist-db.   Perhaps mentioning PDFs was
confusing - I parse PDF documents using the Content Extraction Module which
wraps Apache Tika ( does a good job although it does miss spaces sometimes
which is a problem for text analysis) and this is unchanged.

Chris

On Tue, Mar 15, 2022 at 8:23 PM Roy Walter <gar...@ya...>
wrote:

> Can't be done.
>
> nodejs/puppeteer/Chromium is the way.
>
> -- Roy
>
>
>
> On Tuesday, 15 March 2022, 19:49:28 GMT, Chris Wallace <
> kit...@gm...> wrote:
>
>
> I developed an interface to the Bristol City Planning portal which relies
> on page-scraping and PDF parsing:
>
> https://bristoltrees.space/Planning/map
>
> The latest version of the software (by iDox) has made the change that so
> many sites have done, generating pages in JavaScript.  This would require
> first rendering each page and scraping the result.  Any thoughts on doing
> this in exist_db?
>
> And no, there is no public API that I can find
>
> Chris
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

[Exist-open] Page rendering

[Chris W. <kit...@gm...>]

I developed an interface to the Bristol City Planning portal which relies
on page-scraping and PDF parsing:

https://bristoltrees.space/Planning/map

The latest version of the software (by iDox) has made the change that so
many sites have done, generating pages in JavaScript.  This would require
first rendering each page and scraping the result.  Any thoughts on doing
this in exist_db?

And no, there is no public API that I can find

Chris

Re: [Exist-open] ExistDB 4.10 and Java 11 problems

[Mansell, G. <Gar...@ri...>]

I have now finally been able to get Exist-DB 4.10 installed on a Windows Server with both Java 8 and Java 11 installed. I can start the application both manually and via Windows Services and can run the Client admin tool too. The only thing that I still can't get to work is the Windows Task Tray icon (start.jar) - but I can live without this as the command line tools suffice.

I am logged in as an Admin (not Local Admin) and run an Administrative Command Prompt to perform all tasks, this is what works for me:

>From an Admin Command prompt

set JAVA_HOME=C:\PROGRA~1\AMAZON~1\JDK18~1.0_3
set JAVA_RUN=C:\PROGRA~1\AMAZON~1\JDK18~1.0_3\bin\java
set EXIST_HOME=c:\eXist-db
set PATH=C:\Program Files\Amazon Corretto\jdk1.8.0_312\bin

Java -jar C:\Install\eXist-db-setup-4.10.0.jar

Start Exist-DB from the above Admin Command prompt for the first time (as it has to download/install stuff the first time it runs and takes a while):

        C:\eXist-db\bin\startup.bat

Check that you can access it at http://localhost:8080 and then kill it from the command prompt

>From the same Admin command prompt install the Windows Service:

        "C:\ProgramData\Microsoft\Windows\Start Menu\Programs\eXist-db XML Database\Install eXist-db as Service.lnk"

Start the Windows Service and check that you can access it at http://localhost:8080 and login as admin

Start the Java Client tool from the same Admin command prompt:

        c:\eXist-db\bin\client.bat


I note that the Windows Program Menu icon to start the Exist-DB Task Tray icon (start.jar) still does not start, even from the above configured Admin Command prompt, so if anyone has any suggestions as to how to get this to work I would be keen to hear them.


I hope the above helps people as there just was not a lot of info/support on how to get Exist-DB installed and working properly.

Regards

Gary

--------------------------------------------------------------------------------------------------------------------------------------------------------------
This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are
addressed. If you have received this e-mail in error please notify the sender immediately and delete this e-mail from your system.
Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those
of Ricardo (save for reports and other documentation formally approved and signed for release to the intended recipient). Only Directors
are authorised to enter into legally binding obligations on behalf of Ricardo. Ricardo may monitor outgoing and incoming e-mails and
other telecommunications systems. By replying to this e-mail you give consent to such monitoring. The recipient should check e-mail and
any attachments for the presence of viruses. Ricardo accepts no liability for any damage caused by any virus transmitted by this e-mail.
"Ricardo" means Ricardo plc and its subsidiary companies.
Ricardo plc is a public limited company registered in England with registered number 00222915.
The registered office of Ricardo plc is Shoreham Technical Centre, Shoreham-by Sea, West Sussex, BN43 5FG.
--------------------------------------------------------------------------------------------------------------------------------------------------------------

Re: [Exist-open] ExistDB 4.10 and Java 11 problems

[Mansell, G. <Gar...@ri...>]

So my startup issue is not related to Java 11 as I have just tried it on a Windows machine with only Oracle Java 8_202 (x64), and still I cannot start the Windows Task tray applet from the Exist-DB Programs menu.

I have done the install directly as Local Admin and the menu icon to start exist-db does not work. When I then try and run (the start.jar) from the command line, I can see this error in the console:


c:\eXist-db>java -jar start.jar
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.exist.start.Main.invokeMain(Main.java:140)
        at org.exist.start.Main.run(Main.java:453)
        at org.exist.start.Main.main(Main.java:64)
Caused by: java.lang.NoClassDefFoundError: org/apache/commons/text/lookup/StringLookupFactory
        at org.apache.commons.configuration2.interpol.DefaultLookups.<clinit>(DefaultLookups.java:67)
        at org.apache.commons.configuration2.interpol.ConfigurationInterpolator.<clinit>(ConfigurationInterpolator.java:111)
        at org.apache.commons.configuration2.AbstractConfiguration.installDefaultInterpolator(AbstractConfiguration.java:455)
        at org.apache.commons.configuration2.AbstractConfiguration.<init>(AbstractConfiguration.java:132)
        at org.apache.commons.configuration2.BaseConfiguration.<init>(BaseConfiguration.java:39)
        at org.apache.commons.configuration2.PropertiesConfiguration.<init>(PropertiesConfiguration.java:292)
        at org.exist.launcher.LauncherWrapper.getVMProperties(LauncherWrapper.java:145)
        at org.exist.launcher.LauncherWrapper.launch(LauncherWrapper.java:77)
        at org.exist.launcher.LauncherWrapper.main(LauncherWrapper.java:64)
        ... 7 more
Caused by: java.lang.ClassNotFoundException: org.apache.commons.text.lookup.StringLookupFactory
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        ... 16 more


If I run from the command line with Startup.bat - this works and I can also now Add and Start the Windows Service (this did not work before) - it is just the Windows Task Tray icon (start.jar) that does not work now.

Any suggestions - it would really be nice for this important functionality to work?

Thanks

Gary

--------------------------------------------------------------------------------------------------------------------------------------------------------------
This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are
addressed. If you have received this e-mail in error please notify the sender immediately and delete this e-mail from your system.
Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those
of Ricardo (save for reports and other documentation formally approved and signed for release to the intended recipient). Only Directors
are authorised to enter into legally binding obligations on behalf of Ricardo. Ricardo may monitor outgoing and incoming e-mails and
other telecommunications systems. By replying to this e-mail you give consent to such monitoring. The recipient should check e-mail and
any attachments for the presence of viruses. Ricardo accepts no liability for any damage caused by any virus transmitted by this e-mail.
"Ricardo" means Ricardo plc and its subsidiary companies.
Ricardo plc is a public limited company registered in England with registered number 00222915.
The registered office of Ricardo plc is Shoreham Technical Centre, Shoreham-by Sea, West Sussex, BN43 5FG.
--------------------------------------------------------------------------------------------------------------------------------------------------------------

Re: [Exist-open] ExistDB 4.10 and Java 11 problems

[Josef W. UL <jwe...@hi...>]

Hi Gary, Hi Sava
I have a similar problem, although on Linux and not on Windows. I 
described it already in my message to this list from 7. Mar. (details 
see there).

My problem is:

I try to replace my working eXist-db-4.7.1 installation with the 
recently released eXist-db-4.10.0.

My setup is running on Debian Linux 5.10.0-9-amd64 with Java 11.0.13 
[Debian (OpenJDK 64-Bit Server VM) ...]. eXist-db-4.7.1 is running since 
a long time quite nicely.

eXist-db-4.10.1 in the contrary displays the following error message 
when I try to load the Dashboard (or my app as well):


    HTTP ERROR 503

Problem accessing /exist/. Reason:

     Service Unavailable


      Caused by:

javax.servlet.ServletException: org.eclipse.jetty.servlet.ServletHolder$1: java.lang.SecurityException: class "javax.annotation.Nullable"'s signer information does not match signer information of other classes in the same package
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
... (more lines here)

eXist-db and jetty do start, according to the log. But when I try to 
load the web pages (dashboard etc.) as before, I get the above error 
message.


Hope to find a solution, Greetings


Josef

On 09.03.22 17:44, Mansell, Gary wrote:
> Hi Sava, thanks very much for your response.
>
> I have just tried installing again from scratch as Local Admin with the System default as Java 11 and get the same error in the wrapper log when it tries to start the Windows Service:
>
>
> WARNING|wrapper|Service eXist-db|22-03-09 16:39:07|YAJSW: yajsw-stable-12.12
> WARNING|wrapper|Service eXist-db|22-03-09 16:39:07|OS   : Windows Server 2019/10.0/amd64
> WARNING|wrapper|Service eXist-db|22-03-09 16:39:07|JVM  : Amazon.com Inc./11.0.13/C:\Program Files\Amazon Corretto\jdk11.0.13_8/64
> INFO|wrapper|Service eXist-db|22-03-09 16:39:08|start delay: 0
> INFO|wrapper|Service eXist-db|22-03-09 16:39:08|started process with pid 1640
> INFO|1640/0|Service eXist-db|22-03-09 16:39:09|[INFO] StandardFileSystemManager - Using "c:\eXist-db\tools\yajsw\tmp\vfs_cache" as temporary files store.
> INFO|1640/0|Service eXist-db|22-03-09 16:39:09|no java main method found -> aborting
> INFO|1640/0|Service eXist-db|22-03-09 16:39:09|java.lang.NoClassDefFoundError: org/exist/start/EXistClassLoader
> INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
> INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3166)
> INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at java.base/java.lang.Class.getMethodsRecursive(Class.java:3307)
> INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at java.base/java.lang.Class.getMethod0(Class.java:3293)
> INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at java.base/java.lang.Class.getMethod(Class.java:2106)
> INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at org.rzo.yajsw.app.WrapperManagerImpl.init(WrapperManagerImpl.java:336)
> INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at org.rzo.yajsw.app.WrapperManagerProxy.getWrapperManager(WrapperManagerProxy.java:53)
> INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at org.rzo.yajsw.app.AbstractWrapperJVMMain$1.run(AbstractWrapperJVMMain.java:64)
> INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at java.base/java.security.AccessController.doPrivileged(Native Method)
> INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at org.rzo.yajsw.app.AbstractWrapperJVMMain.preExecute(AbstractWrapperJVMMain.java:56)
> INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at org.rzo.yajsw.app.WrapperJVMMain.main(WrapperJVMMain.java:41)
> INFO|1640/0|Service eXist-db|22-03-09 16:39:09|Caused by: java.lang.ClassNotFoundException: org.exist.start.EXistClassLoader
> INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
> INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
> INFO|1640/0|Service eXist-db|22-03-09 16:39:09| ... 11 more
>
> I also notice that the Exist-DB App still does not start from the Programs Menu manually.
>
> Does anyone know what the problem might be?
>
> Should Exist-DB 4.10 work with Java 11?
>
> Any advice appreciated.
>
> Gary
>
> --------------------------------------------------------------------------------------------------------------------------------------------------------------
> This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are
> addressed. If you have received this e-mail in error please notify the sender immediately and delete this e-mail from your system.
> Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those
> of Ricardo (save for reports and other documentation formally approved and signed for release to the intended recipient). Only Directors
> are authorised to enter into legally binding obligations on behalf of Ricardo. Ricardo may monitor outgoing and incoming e-mails and
> other telecommunications systems. By replying to this e-mail you give consent to such monitoring. The recipient should check e-mail and
> any attachments for the presence of viruses. Ricardo accepts no liability for any damage caused by any virus transmitted by this e-mail.
> "Ricardo" means Ricardo plc and its subsidiary companies.
> Ricardo plc is a public limited company registered in England with registered number 00222915.
> The registered office of Ricardo plc is Shoreham Technical Centre, Shoreham-by Sea, West Sussex, BN43 5FG.
> --------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] ExistDB 4.10 and Java 11 problems

[<sju...@mr...>]

 

Hi Gary, 

ran into the similar issue ... for us, it had nothing to do with Java
version. 

My issue (exactly the same behavior as what you described) was the
actual Windows account. 

If I install everything using the native Administrator account, all is
good 

If I login with account that has admin credentials, I could never get it
going. 

Hope this helps 

Sava 

On 2022-03-09 08:19, Mansell, Gary wrote: 

> Further to this issue, I have just tried explicitly installing using Java 8 from an Admin Command prompt:
> 
> set JAVA_HOME=C:\PROGRA~1\AMAZON~1\JDK18~1.0_3
> C:\PROGRA~1\AMAZON~1\JDK18~1.0_3\bin\java.exe -jar c:\install\eXist-db-setup-4.10.0.jar
> 
> When I run the install service from the Start Programs Menu - this works, but I then get this error in the wrapper logs:
> 
> WARNING|wrapper|Service eXist-db|22-03-09 16:03:48|YAJSW: yajsw-stable-12.12
> WARNING|wrapper|Service eXist-db|22-03-09 16:03:48|OS   : Windows Server 2019/10.0/amd64
> WARNING|wrapper|Service eXist-db|22-03-09 16:03:48|JVM  : Amazon.com Inc./1.8.0_312/C:\Program Files\Amazon Corretto\jdk1.8.0_312\jre/64
> INFO|wrapper|Service eXist-db|22-03-09 16:03:48|start delay: 0
> INFO|wrapper|Service eXist-db|22-03-09 16:03:48|started process with pid 9612
> INFO|9612/0|Service eXist-db|22-03-09 16:03:49|[INFO] StandardFileSystemManager - Using "c:\eXist-db\tools\yajsw\tmp\vfs_cache" as temporary files store.
> INFO|9612/0|Service eXist-db|22-03-09 16:03:49|no java main method found -> aborting
> INFO|9612/0|Service eXist-db|22-03-09 16:03:49|java.lang.NoClassDefFoundError: org/exist/start/EXistClassLoader
> INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at java.lang.Class.getDeclaredMethods0(Native Method)
> INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at java.lang.Class.privateGetDeclaredMethods(Class.java:2729)
> INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at java.lang.Class.privateGetMethodRecursive(Class.java:3076)
> INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at java.lang.Class.getMethod0(Class.java:3046)
> INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at java.lang.Class.getMethod(Class.java:1812)
> INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at org.rzo.yajsw.app.WrapperManagerImpl.init(WrapperManagerImpl.java:336)
> INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at org.rzo.yajsw.app.WrapperManagerProxy.getWrapperManager(WrapperManagerProxy.java:53)
> INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at org.rzo.yajsw.app.AbstractWrapperJVMMain$1.run(AbstractWrapperJVMMain.java:64)
> INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at java.security.AccessController.doPrivileged(Native Method)
> INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at org.rzo.yajsw.app.AbstractWrapperJVMMain.preExecute(AbstractWrapperJVMMain.java:56)
> INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at org.rzo.yajsw.app.WrapperJVMMain.main(WrapperJVMMain.java:41)
> INFO|9612/0|Service eXist-db|22-03-09 16:03:49|Caused by: java.lang.ClassNotFoundException: org.exist.start.EXistClassLoader
> INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
> INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> INFO|9612/0|Service eXist-db|22-03-09 16:03:49| ... 11 more
> 
> So, I am still stuck trying to get it to run as a service - any ideas?
> 
> Thanks
> 
> Gary
> 
> --------------------------------------------------------------------------------------------------------------------------------------------------------------
> This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are
> addressed. If you have received this e-mail in error please notify the sender immediately and delete this e-mail from your system.
> Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those
> of Ricardo (save for reports and other documentation formally approved and signed for release to the intended recipient). Only Directors
> are authorised to enter into legally binding obligations on behalf of Ricardo. Ricardo may monitor outgoing and incoming e-mails and
> other telecommunications systems. By replying to this e-mail you give consent to such monitoring. The recipient should check e-mail and
> any attachments for the presence of viruses. Ricardo accepts no liability for any damage caused by any virus transmitted by this e-mail.
> "Ricardo" means Ricardo plc and its subsidiary companies.
> Ricardo plc is a public limited company registered in England with registered number 00222915.
> The registered office of Ricardo plc is Shoreham Technical Centre, Shoreham-by Sea, West Sussex, BN43 5FG.
> --------------------------------------------------------------------------------------------------------------------------------------------------------------
> 
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

  

Re: [Exist-open] ExistDB 4.10 and Java 11 problems

[Mansell, G. <Gar...@ri...>]

Hi Sava, thanks very much for your response.

I have just tried installing again from scratch as Local Admin with the System default as Java 11 and get the same error in the wrapper log when it tries to start the Windows Service:


WARNING|wrapper|Service eXist-db|22-03-09 16:39:07|YAJSW: yajsw-stable-12.12
WARNING|wrapper|Service eXist-db|22-03-09 16:39:07|OS   : Windows Server 2019/10.0/amd64
WARNING|wrapper|Service eXist-db|22-03-09 16:39:07|JVM  : Amazon.com Inc./11.0.13/C:\Program Files\Amazon Corretto\jdk11.0.13_8/64
INFO|wrapper|Service eXist-db|22-03-09 16:39:08|start delay: 0
INFO|wrapper|Service eXist-db|22-03-09 16:39:08|started process with pid 1640
INFO|1640/0|Service eXist-db|22-03-09 16:39:09|[INFO] StandardFileSystemManager - Using "c:\eXist-db\tools\yajsw\tmp\vfs_cache" as temporary files store.
INFO|1640/0|Service eXist-db|22-03-09 16:39:09|no java main method found -> aborting
INFO|1640/0|Service eXist-db|22-03-09 16:39:09|java.lang.NoClassDefFoundError: org/exist/start/EXistClassLoader
INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3166)
INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at java.base/java.lang.Class.getMethodsRecursive(Class.java:3307)
INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at java.base/java.lang.Class.getMethod0(Class.java:3293)
INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at java.base/java.lang.Class.getMethod(Class.java:2106)
INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at org.rzo.yajsw.app.WrapperManagerImpl.init(WrapperManagerImpl.java:336)
INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at org.rzo.yajsw.app.WrapperManagerProxy.getWrapperManager(WrapperManagerProxy.java:53)
INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at org.rzo.yajsw.app.AbstractWrapperJVMMain$1.run(AbstractWrapperJVMMain.java:64)
INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at java.base/java.security.AccessController.doPrivileged(Native Method)
INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at org.rzo.yajsw.app.AbstractWrapperJVMMain.preExecute(AbstractWrapperJVMMain.java:56)
INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at org.rzo.yajsw.app.WrapperJVMMain.main(WrapperJVMMain.java:41)
INFO|1640/0|Service eXist-db|22-03-09 16:39:09|Caused by: java.lang.ClassNotFoundException: org.exist.start.EXistClassLoader
INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
INFO|1640/0|Service eXist-db|22-03-09 16:39:09| at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
INFO|1640/0|Service eXist-db|22-03-09 16:39:09| ... 11 more

I also notice that the Exist-DB App still does not start from the Programs Menu manually.

Does anyone know what the problem might be?

Should Exist-DB 4.10 work with Java 11?

Any advice appreciated.

Gary

--------------------------------------------------------------------------------------------------------------------------------------------------------------
This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are
addressed. If you have received this e-mail in error please notify the sender immediately and delete this e-mail from your system.
Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those
of Ricardo (save for reports and other documentation formally approved and signed for release to the intended recipient). Only Directors
are authorised to enter into legally binding obligations on behalf of Ricardo. Ricardo may monitor outgoing and incoming e-mails and
other telecommunications systems. By replying to this e-mail you give consent to such monitoring. The recipient should check e-mail and
any attachments for the presence of viruses. Ricardo accepts no liability for any damage caused by any virus transmitted by this e-mail.
"Ricardo" means Ricardo plc and its subsidiary companies.
Ricardo plc is a public limited company registered in England with registered number 00222915.
The registered office of Ricardo plc is Shoreham Technical Centre, Shoreham-by Sea, West Sussex, BN43 5FG.
--------------------------------------------------------------------------------------------------------------------------------------------------------------

Re: [Exist-open] ExistDB 4.10 and Java 11 problems

[Mansell, G. <Gar...@ri...>]

Further to this issue, I have just tried explicitly installing using Java 8 from an Admin Command prompt:

   set JAVA_HOME=C:\PROGRA~1\AMAZON~1\JDK18~1.0_3
   C:\PROGRA~1\AMAZON~1\JDK18~1.0_3\bin\java.exe -jar c:\install\eXist-db-setup-4.10.0.jar


When I run the install service from the Start Programs Menu - this works, but I then get this error in the wrapper logs:

WARNING|wrapper|Service eXist-db|22-03-09 16:03:48|YAJSW: yajsw-stable-12.12
WARNING|wrapper|Service eXist-db|22-03-09 16:03:48|OS   : Windows Server 2019/10.0/amd64
WARNING|wrapper|Service eXist-db|22-03-09 16:03:48|JVM  : Amazon.com Inc./1.8.0_312/C:\Program Files\Amazon Corretto\jdk1.8.0_312\jre/64
INFO|wrapper|Service eXist-db|22-03-09 16:03:48|start delay: 0
INFO|wrapper|Service eXist-db|22-03-09 16:03:48|started process with pid 9612
INFO|9612/0|Service eXist-db|22-03-09 16:03:49|[INFO] StandardFileSystemManager - Using "c:\eXist-db\tools\yajsw\tmp\vfs_cache" as temporary files store.
INFO|9612/0|Service eXist-db|22-03-09 16:03:49|no java main method found -> aborting
INFO|9612/0|Service eXist-db|22-03-09 16:03:49|java.lang.NoClassDefFoundError: org/exist/start/EXistClassLoader
INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at java.lang.Class.getDeclaredMethods0(Native Method)
INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at java.lang.Class.privateGetDeclaredMethods(Class.java:2729)
INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at java.lang.Class.privateGetMethodRecursive(Class.java:3076)
INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at java.lang.Class.getMethod0(Class.java:3046)
INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at java.lang.Class.getMethod(Class.java:1812)
INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at org.rzo.yajsw.app.WrapperManagerImpl.init(WrapperManagerImpl.java:336)
INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at org.rzo.yajsw.app.WrapperManagerProxy.getWrapperManager(WrapperManagerProxy.java:53)
INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at org.rzo.yajsw.app.AbstractWrapperJVMMain$1.run(AbstractWrapperJVMMain.java:64)
INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at java.security.AccessController.doPrivileged(Native Method)
INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at org.rzo.yajsw.app.AbstractWrapperJVMMain.preExecute(AbstractWrapperJVMMain.java:56)
INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at org.rzo.yajsw.app.WrapperJVMMain.main(WrapperJVMMain.java:41)
INFO|9612/0|Service eXist-db|22-03-09 16:03:49|Caused by: java.lang.ClassNotFoundException: org.exist.start.EXistClassLoader
INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
INFO|9612/0|Service eXist-db|22-03-09 16:03:49| at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
INFO|9612/0|Service eXist-db|22-03-09 16:03:49| ... 11 more


So, I am still stuck trying to get it to run as a service - any ideas?

Thanks

Gary

--------------------------------------------------------------------------------------------------------------------------------------------------------------
This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are
addressed. If you have received this e-mail in error please notify the sender immediately and delete this e-mail from your system.
Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those
of Ricardo (save for reports and other documentation formally approved and signed for release to the intended recipient). Only Directors
are authorised to enter into legally binding obligations on behalf of Ricardo. Ricardo may monitor outgoing and incoming e-mails and
other telecommunications systems. By replying to this e-mail you give consent to such monitoring. The recipient should check e-mail and
any attachments for the presence of viruses. Ricardo accepts no liability for any damage caused by any virus transmitted by this e-mail.
"Ricardo" means Ricardo plc and its subsidiary companies.
Ricardo plc is a public limited company registered in England with registered number 00222915.
The registered office of Ricardo plc is Shoreham Technical Centre, Shoreham-by Sea, West Sussex, BN43 5FG.
--------------------------------------------------------------------------------------------------------------------------------------------------------------

[Exist-open] ExistDB 4.10 and Java 11 problems

[Mansell, G. <Gar...@ri...>]

Hi,

I have a server that has (Amazon Corretto) Java 8 and Java 11 installed - with Java 11 being the last installed and hence set first in the path and as JAVA_HOME. This has to be the case so that my other Java applications on the server run successfully.

I have installed Exist-DB 4.10, but when I try and run it from the Programs Menu (or run start.jar from a command prompt) the Exist-DB task tray icon does not appear.

I presume that this must be some problem with running Exist-DB 4.10 with Java 11 - is it perhaps not supported?

I have found that if I set JAVA_HOME=C:\PROGRA~1\AMAZON~1\JDK18~1.0_3 (note, it has to be the old 8.3 notation), that I can then run both startup.bat and client.bat successfully and ExistDB works OK.

My problem is that I cannot use the Programs menu Icon to get it configured as a Windows Service to start at boot - presumably due to it trying to run with Java 11 instead of Java 8 by default. It creates the Windows Service, which then tries to start Exist-DB but then stops (presumably as it is trying and failing to run it using Java 11).

So, how can I get the Windows Service installed but using JAVA_HOME=C:\PROGRA~1\AMAZON~1\JDK18~1.0_3 like I have done to get the App to run manually?

Or, am I missing something with Java 11?

Thanks in advance for any help.


Gary



--------------------------------------------------------------------------------------------------------------------------------------------------------------
This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are
addressed. If you have received this e-mail in error please notify the sender immediately and delete this e-mail from your system.
Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those
of Ricardo (save for reports and other documentation formally approved and signed for release to the intended recipient). Only Directors
are authorised to enter into legally binding obligations on behalf of Ricardo. Ricardo may monitor outgoing and incoming e-mails and
other telecommunications systems. By replying to this e-mail you give consent to such monitoring. The recipient should check e-mail and
any attachments for the presence of viruses. Ricardo accepts no liability for any damage caused by any virus transmitted by this e-mail.
"Ricardo" means Ricardo plc and its subsidiary companies.
Ricardo plc is a public limited company registered in England with registered number 00222915.
The registered office of Ricardo plc is Shoreham Technical Centre, Shoreham-by Sea, West Sussex, BN43 5FG.
--------------------------------------------------------------------------------------------------------------------------------------------------------------

[Exist-open] eXist-db-4.10.0 giving HTTP error 503

[Josef W. UL <jwe...@hi...>]

Hello eXist userlist

I try to replace my working eXist-db-4.7.1 installation with the 
recently released eXist-db-4.10.0.

My setup is running on Debian Linux 5.10.0-9-amd64 with Java 11.0.13 
[Debian (OpenJDK 64-Bit Server VM) ...]. eXist-db-4.7.1 is running since 
a long time quite nicely.

eXist-db-4.10.1 in the contrary displays the following error message 
when I try to load the Dashboard (or my app as well):


    HTTP ERROR 503

Problem accessing /exist/. Reason:

     Service Unavailable


      Caused by:

javax.servlet.ServletException: org.eclipse.jetty.servlet.ServletHolder$1: java.lang.SecurityException: class "javax.annotation.Nullable"'s signer information does not match signer information of other classes in the same package
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:502)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.eclipse.jetty.servlet.ServletHolder$1: java.lang.SecurityException: class "javax.annotation.Nullable"'s signer information does not match signer information of other classes in the same package
...

I attach the log files of the startup process of both my working 
eXist-db-4.7.1 and the failing eXist-db-4.10.0 in full to this mail.

I would be very gratefull if someone could give me a hint on how to 
proceed to avoid this problem.

Greetings

Josef Wetzel

[Exist-open] shutdown.sh

[Roy W. <gar...@ya...>]

Hi,
How to configure the shutdown scipt to target a server not on 8080?
Thanks,Roy