exist-open — Open discussion

Re: [Exist-open] Updating Java version and installing eXist 6

[Ted H. <meg...@gm...>]

I'm using ubuntu lunar lobster.

On Mon, Oct 16, 2023, 10:46 PM Claudius Teodorescu <
cla...@gm...> wrote:

> Hi,
>
> What operating system is installed?
>
> Claudius
>
> On Tue, 17 Oct 2023, 06:19 Ted Hickox, <meg...@gm...> wrote:
>
>> It's been a long time since I've worked with the eXist database.  But I
>> would like to work with it again.  Unfortunately I have a very old Java
>> version in my computer and I need some help in updating Java so I can
>> successfully run eXist.  Can anyone assist me?
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>
>

Re: [Exist-open] Updating Java version and installing eXist 6

[Claudius T. <cla...@gm...>]

Hi,

What operating system is installed?

Claudius

On Tue, 17 Oct 2023, 06:19 Ted Hickox, <meg...@gm...> wrote:

> It's been a long time since I've worked with the eXist database.  But I
> would like to work with it again.  Unfortunately I have a very old Java
> version in my computer and I need some help in updating Java so I can
> successfully run eXist.  Can anyone assist me?
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

[Exist-open] Updating Java version and installing eXist 6

[Ted H. <meg...@gm...>]

It's been a long time since I've worked with the eXist database.  But I
would like to work with it again.  Unfortunately I have a very old Java
version in my computer and I need some help in updating Java so I can
successfully run eXist.  Can anyone assist me?

Re: [Exist-open] Job scheduler

[Nicolas S. <sou...@gm...>]

Hi,

Thank you for your answer.

I finally found the solution, the issue didn't come from the xmldb:login or
SetUid/SetGid bits on the XQuery file's permissions but from the security
I've add to the application, removing the guest user's ability to execute a
function.

I've changed this and everything works perfectly now.

-- 
Best regards,

Nicolas

Re: [Exist-open] Job scheduler

[Jean-Paul R. <re...@gm...>]

Hi

Execution as "guest" is expected behaviour. Per
https://exist-db.org/exist/apps/doc/scheduler#job-author you need to switch
permissions or set the SetUid/SetGid bits on the XQuery file's permissions.

Authoring Jobs

The scheduler supports jobs that are authored in either XQuery or Java:
XQuery Jobs

Jobs coded in XQuery can be user type jobs only.

An XQuery job is a standard XQuery Main Module which is stored in the
database. You configure the scheduling of the job by providing the database
path to the XQuery, for example: /db/my-collection/my-job.xq.

XQuery job's are launched under the *guest* account. If you wish to perform
tasks as another user , either switch permissions by calling xmldb:login() from
within your job, or set the SetUid/SetGid bits on the XQuery file's
permissions (see Security <https://exist-db.org/exist/apps/doc/security> for
more information).
Cheers,
JPR


On Thu, Oct 12, 2023 at 1:52 PM Nicolas Souchon <
sou...@gm...> wrote:

> Dear All,
>
> I'm trying to schedule a job to run an XQuery function every day, but I'm
> encountering a problem that I don't know how to solve. According to the
> documentation, I've added the following lines to eXist-db/etc/conf.xml:
>
> <job type="user" xquery="/db/apps/myapp/myjob.xql" period="86400000"
> unschedule-on-exception="false">
> <parameter name="user" value="userName"/>
> <parameter name="password" value="userPassword"/>
> </job>
>
> With the XQuery attribute referring to my XQuery file (with a working
> XQuery function) and the parameters defining the username and password
> (which are also given in the XQuery file with xmldb:login()), but it
> doesn't work.
>
> It seems that there are problems with the user, as indicated by the
> scheduler.log file:
>
> 2023-10-12 13:31:45,491 [db.exist.scheduler.quartz-worker-3] INFO
>  (JobRunShell.java [run]:207) - Job eXist.User.XQuery:
> /db/apps/myapp/myjob.xql threw a JobExecutionException:
> org.quartz.JobExecutionException: UserXQueryJob Failed: Permission denied
> for the scheduling user: guest! Unscheduling UserXQueryJob.
> at org.exist.scheduler.UserXQueryJob.abort(UserXQueryJob.java:251)
> ~[exist-core-6.2.0.jar:6.2.0]
> at org.exist.scheduler.UserXQueryJob.abort(UserXQueryJob.java:247)
> ~[exist-core-6.2.0.jar:6.2.0]
> at org.exist.scheduler.UserXQueryJob.execute(UserXQueryJob.java:173)
> ~[exist-core-6.2.0.jar:6.2.0]
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> ~[quartz-2.3.2.jar:?]
> at
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> ~[quartz-2.3.2.jar:?]
>
> Can anyone tell me how to solve this? I don't understand why the scheduler
> is trying to run the function as Guest user.
>
> Best regards,
>
> Nicolas
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

[Exist-open] Job scheduler

[Nicolas S. <sou...@gm...>]

Dear All,

I'm trying to schedule a job to run an XQuery function every day, but I'm
encountering a problem that I don't know how to solve. According to the
documentation, I've added the following lines to eXist-db/etc/conf.xml:

<job type="user" xquery="/db/apps/myapp/myjob.xql" period="86400000"
unschedule-on-exception="false">
<parameter name="user" value="userName"/>
<parameter name="password" value="userPassword"/>
</job>

With the XQuery attribute referring to my XQuery file (with a working
XQuery function) and the parameters defining the username and password
(which are also given in the XQuery file with xmldb:login()), but it
doesn't work.

It seems that there are problems with the user, as indicated by the
scheduler.log file:

2023-10-12 13:31:45,491 [db.exist.scheduler.quartz-worker-3] INFO
 (JobRunShell.java [run]:207) - Job eXist.User.XQuery:
/db/apps/myapp/myjob.xql threw a JobExecutionException:
org.quartz.JobExecutionException: UserXQueryJob Failed: Permission denied
for the scheduling user: guest! Unscheduling UserXQueryJob.
at org.exist.scheduler.UserXQueryJob.abort(UserXQueryJob.java:251)
~[exist-core-6.2.0.jar:6.2.0]
at org.exist.scheduler.UserXQueryJob.abort(UserXQueryJob.java:247)
~[exist-core-6.2.0.jar:6.2.0]
at org.exist.scheduler.UserXQueryJob.execute(UserXQueryJob.java:173)
~[exist-core-6.2.0.jar:6.2.0]
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
~[quartz-2.3.2.jar:?]
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
~[quartz-2.3.2.jar:?]

Can anyone tell me how to solve this? I don't understand why the scheduler
is trying to run the function as Guest user.

Best regards,

Nicolas

Re: [Exist-open] [EXTERNAL] [BULK] send-email function

[Winona S. <wsa...@gm...>]

Interesting, thanks.
-Winona


On Wed, Sep 13, 2023 at 7:27 AM Harris, Bernard T. (GSFC-5870) <
ber...@na...> wrote:

> I recently tested it on exist 6.2 and RH 7 and it worked (recipients
> received the email).  I used
>
> mail:send-email($msg, (), ())
>
> I do not know if that difference is significant or about other
> configurations.
>
>
>
> Bernie
>
>
>
> *From: *Winona Salesky <wsa...@gm...>
> *Date: *Wednesday, September 13, 2023 at 7:17 AM
> *To: *exist-open <exi...@li...>
> *Subject: *[EXTERNAL] [BULK] [Exist-open] send-email function
>
>
>
> *CAUTION:* This email originated from outside of NASA.  Please take care
> when clicking links or opening attachments.  Use the "Report Message"
> button to report suspicious messages to the NASA SOC.
>
>
>
> Has the send-email() function changed or been depreciated? It no longer
> works on my 6.1.0 version of eXist. I am trying to use the local send mail
> server for the SMTP server. When I use it on a locally installed version, I
> get true() but no email is sent, when I try using it on my production
> version of my app (installed via docker on AWS) I get a 'connection
> refused' error.
>
>
>
> Sample xquery:
>
> xquery version "3.1";
>
> let $email :=
> <mail>
>     <from>wsa...@gm...</from>
>     <to>wsa...@gm...</to>
>     <subject>Test email client in eXist-db 2</subject>
>     <message>
>       <xhtml>
>            <html xmlns="http://www.w3.org/1999/xhtml">
>                <head>
>                  <title>Test email client in eXist-db</title>
>                </head>
>                <body>
>                  <p>Name: winona</p>
>                  <p>e-mail: winona</p>
>                  <p>Subject: test</p>
>               </body>
>            </html>
>       </xhtml>
>     </message>
>   </mail>
>
> return mail:send-email($email, '', '')
>
>
>
>
>
> Thanks for any insights.
>
> -Winona
>
>
>

[Exist-open] Templating library debugging

[Winona S. <wsa...@gm...>]

Hi All,
Since migrating to the new templating library (v1.1.0 in eXist-db v6.1.0)
the error reporting has changed. With the new error reporting I do not have
any way to trace back to the function that is causing the issue, it only
tells me the templates:load-source function is failing. I used to get the
name of my local function that was causing the issue, making it much easier
to debug, especially when migrating a large old project. Is there a trick
to debugging these sorts of errors, another way to track down the source of
the bug?

Thanks for the help,
-Winona

This is the error I get with the new templating library:
templates:NotFound No template function found for call
templates:load-source (Max arity of 20 has been exceeded in searching for
this template function. If needed, adjust $templates:MAX_ARITY in the
templates.xql module.) [at line 191, column 85, source:
/Users/wsalesky/Library/Application
Support/org.exist61/expathrepo/templating-1.1.0/content/templates.xqm] In
function: templates:call(item(), element(), map(*))
[145:37:/Users/wsalesky/Library/Application
Support/org.exist61/expathrepo/templating-1.1.0/content/templates.xqm]
templates:process(node()*, map(*))
[148:81:/Users/wsalesky/Library/Application
Support/org.exist61/expathrepo/templating-1.1.0/content/templates.xqm]
templates:process(node()*, map(*))
[148:81:/Users/wsalesky/Library/Application
Support/org.exist61/expathrepo/templating-1.1.0/content/templates.xqm]
templates:process(node()*, map(*))
[148:81:/Users/wsalesky/Library/Application
Support/org.exist61/expathrepo/templating-1.1.0/content/templates.xqm]
templates:process(node()*, map(*))
[148:81:/Users/wsalesky/Library/Application
Support/org.exist61/expathrepo/templating-1.1.0/content/templates.xqm]
templates:process(node()*, map(*))
[148:81:/Users/wsalesky/Library/Application
Support/org.exist61/expathrepo/templating-1.1.0/content/templates.xqm]
templates:process(node()*, map(*))
[435:17:/Users/wsalesky/Library/Application
Support/org.exist61/expathrepo/templating-1.1.0/content/templates.xqm]
templates:process-output(element(), map(*), item()*)
[230:9:/Users/wsalesky/Library/Application
Support/org.exist61/expathrepo/templating-1.1.0/content/templates.xqm]
templates:process-output(element(), map(*), item()*, element(function))
[211:9:/Users/wsalesky/Library/Application
Support/org.exist61/expathrepo/templating-1.1.0/content/templates.xqm]
templates:call-by-introspection(element(), map(*), map(*), function(*))
[189:28:/Users/wsalesky/Library/Application
Support/org.exist61/expathrepo/templating-1.1.0/content/templates.xqm]
templates:call(item(), element(), map(*))
[145:37:/Users/wsalesky/Library/Application
Support/org.exist61/expathrepo/templating-1.1.0/content/templates.xqm]
templates:process(node()*, map(*))
[133:51:/Users/wsalesky/Library/Application
Support/org.exist61/expathrepo/templating-1.1.0/content/templates.xqm]
templates:process(node()*, map(*))
[90:9:/Users/wsalesky/Library/Application
Support/org.exist61/expathrepo/templating-1.1.0/content/templates.xqm]
templates:apply(node()+, function(*), map(*)?, map(*)?)
[50:5:/Users/wsalesky/Library/Application
Support/org.exist61/expathrepo/templating-1.1.0/content/templates.xqm]

Re: [Exist-open] [EXTERNAL] [BULK] send-email function

[Harris, B. T. (GSFC-5870) <ber...@na...>]

I recently tested it on exist 6.2 and RH 7 and it worked (recipients received the email).  I used
mail:send-email($msg, (), ())
I do not know if that difference is significant or about other configurations.

Bernie

From: Winona Salesky <wsa...@gm...>
Date: Wednesday, September 13, 2023 at 7:17 AM
To: exist-open <exi...@li...>
Subject: [EXTERNAL] [BULK] [Exist-open] send-email function

CAUTION: This email originated from outside of NASA.  Please take care when clicking links or opening attachments.  Use the "Report Message" button to report suspicious messages to the NASA SOC.


Has the send-email() function changed or been depreciated? It no longer works on my 6.1.0 version of eXist. I am trying to use the local send mail server for the SMTP server. When I use it on a locally installed version, I get true() but no email is sent, when I try using it on my production version of my app (installed via docker on AWS) I get a 'connection refused' error.

Sample xquery:
xquery version "3.1";

let $email :=
<mail>
    <from>wsa...@gm...<mailto:wsa...@gm...></from>
    <to>wsa...@gm...<mailto:wsa...@gm...></to>
    <subject>Test email client in eXist-db 2</subject>
    <message>
      <xhtml>
           <html xmlns="http://www.w3.org/1999/xhtml">
               <head>
                 <title>Test email client in eXist-db</title>
               </head>
               <body>
                 <p>Name: winona</p>
                 <p>e-mail: winona</p>
                 <p>Subject: test</p>
              </body>
           </html>
      </xhtml>
    </message>
  </mail>

return mail:send-email($email, '', '')


Thanks for any insights.
-Winona

[Exist-open] send-email function

[Winona S. <wsa...@gm...>]

Has the send-email() function changed or been depreciated? It no longer
works on my 6.1.0 version of eXist. I am trying to use the local send mail
server for the SMTP server. When I use it on a locally installed version, I
get true() but no email is sent, when I try using it on my production
version of my app (installed via docker on AWS) I get a 'connection
refused' error.

Sample xquery:
xquery version "3.1";

let $email :=
<mail>
    <from>wsa...@gm...</from>
    <to>wsa...@gm...</to>
    <subject>Test email client in eXist-db 2</subject>
    <message>
      <xhtml>
           <html xmlns="http://www.w3.org/1999/xhtml">
               <head>
                 <title>Test email client in eXist-db</title>
               </head>
               <body>
                 <p>Name: winona</p>
                 <p>e-mail: winona</p>
                 <p>Subject: test</p>
              </body>
           </html>
      </xhtml>
    </message>
  </mail>

return mail:send-email($email, '', '')


Thanks for any insights.
-Winona

[Exist-open] Declarative Amsterdam 2023 - Register now

[Erik S. <er...@xa...>]

Declarative Amsterdam 2023 will be happening at CWI, Amsterdam Science Park on Thursday/Friday November 2 and 3, 2023.

*	Thursday morning is a tutorial session, with a hands-on Advanced XForms Tutorial. It is a “bring your own device” event and you will be expected to have pre-loaded some materials before attending.
*	Thursday afternoon and all day Friday is the symposium.

There are many great presentations, as you can see in the full program at https://declarative.amsterdam/program

The conference is a hybrid event, live at the Science Park in Amsterdam, and live-streamed.
Registration is open; early bird registration closes 1st October, so hurry along to https://declarative.amsterdam/registration. 

Best wishes,

The Declarative Amsterdam Conference Committee.

 

Re: [Exist-open] high memory usage when constructing a string

[Jo C. <Jo....@ha...>]

Hi all,

Maybe turn things around.

Create the report first as a stub, say "<report/>".

Iterate over the source from the remote database, inserting something into
the report like

<item doc-id="{$doc-id}" version-id="{$version-id"}/>

This will give you a doc with 5 million+ nodes, but should be within
capability.  Then iterate over the item elements, inserting a new
attribute indicating the result of your test against  that item.

If you find yourself running into limits that can't be fixed
environmentally, reduce the scope of your SQL query, and generate  reports
for disjoint subsets.

Cheers, -- Jo

On Fri, Aug 11, 2023 at 3:54 PM Lizzi, Vincent <
Vin...@ta...> wrote:

> Hi Alasdair,
>
>
>
> Thanks for the response. Here is what I’m trying to do.
>
>
>
> The eXist database holds XML documents that were loaded from an archive.
> The number of documents is approaching 5 million but might be higher due to
> multiple versions of documents. I want to verify that the holdings in eXist
> are complete and up to date, and report any differences. The SQL query
> provides a list containing pairs of a document identifier (unique string)
> and a version identifier (unique integer). I divided this task into steps
> with separate XQueries to make it more manageable. The first step is to
> retrieve this list from a SQL database. The second step is to compare the
> list with the eXist database (which has range indexes on the identifiers)
> and add a status column to the list. The third step is to produce a report.
> The last step will be to use the report to correct any differences.  I
> previously tried writing one query to do the first three steps, but it
> seemed to work better as separate smaller steps.
>
>
>
> Any suggestions would be welcome.
>
>
>
> Vincent
>
>
>
> _____________________________________________
>
> *Vincent M. Lizzi*
>
> Head of Information Standards | Taylor & Francis Group
>
> vin...@ta...
>
>
>
> Information Classification: General
>
> *From:* Alasdair Dougall <ala...@gm...>
> *Sent:* Thursday, August 10, 2023 5:34 PM
> *To:* Lizzi, Vincent <Vin...@ta...>
> *Cc:* Jean-Paul Rehr <re...@gm...>; Exist-open <
> exi...@li...>
> *Subject:* Re: [Exist-open] high memory usage when constructing a string
>
>
>
> Vincent,
>
>
>
> As Jean-Paul said in his reply, there are alternative ways to build large
> outputs.
>
>
>
> Maybe describe the outcome you are trying to achieve as opposed to the
> issue you are finding.  There are generally more than one way to do things
> in eXist.
>
>
>
> Alasdair
>
> Sent from my iPhone
>
>
>
> On 11 Aug 2023, at 1:42 am, Lizzi, Vincent <
> Vin...@ta...> wrote:
>
> 
>
> Hi JPR,
>
>
>
> That sounds like a good idea. Unfortunately, trying to use JSON for this
> amount of data also results in out of memory errors.
>
>
>
> array:join(
>
> (1 to 200000000) ! ['1']
>
> ) => serialize(map{'method': 'json'})
>
>
>
> I have gotten eXist to process actual data in tab separated text format
> when eXist is freshly started and has plenty of memory available, but even
> in this ideal condition memory usage is still unexpectedly high and
> sometimes produces out of memory errors.
>
>
>
> Thanks,
>
> Vincent
>
>
>
> _____________________________________________
>
> *Vincent M. Lizzi*
>
> Head of Information Standards | Taylor & Francis Group
>
> vin...@ta...
>
>
>
>
>
> Information Classification: General
>
> *From:* Jean-Paul Rehr <re...@gm...>
> *Sent:* Thursday, August 10, 2023 10:06 AM
> *To:* Exist-open <exi...@li...>
> *Subject:* Re: [Exist-open] high memory usage when constructing a string
>
>
>
> Dear Vincent,
>
>
>
> Have you tried outputting JSON structure data from the database? It will
> be easier to store and access later using Xquery via arrays and maps. You
> could even chunk it if eXist chokes on a large file - I've never handled
> one of that size.
>
>
>
> Cheers,
>
> JPR
>
>
>
> On Thu, Aug 10, 2023 at 3:24 PM Lizzi, Vincent <
> Vin...@ta...> wrote:
>
> Hello eXist folks,
>
>
>
> I’m observing that asking eXist to construct a string of about 200 Mb uses
> more than 4 Gb of memory.
>
>
>
> When running a query such as this that constructs a string of 200,000,000
> bytes:
>
>
>
> string-join(( (1 to 200000000) ! '1' ))
>
>
>
> Monex shows the memory usage climbs to over 4 Gb.
>
>
>
> This sometimes exhausts the available memory and results in a stack trace
> being printed exist.log, the top of which shows:
>
>
>
> 2023-08-09 23:12:40,553 [qtp353417634-34] ERROR (XQueryServlet.java
> [process]:549) - Java heap space
>
> java.lang.OutOfMemoryError: Java heap space
>
>                at
> org.exist.dom.memtree.DocumentImpl.addChars(DocumentImpl.java:273)
> ~[exist-core-6.2.0.jar:6.2.0]
>
>                at
> org.exist.dom.memtree.MemTreeBuilder.characters(MemTreeBuilder.java:382)
> ~[exist-core-6.2.0.jar:6.2.0]
>
>
>
> I would guess that constructing a document uses a lot more memory than
> constructing a simple string. Is there a way to signal eXist to use a
> simple string builder instead?
>
>
>
> The example above is a simplification of what I actually want to do, which
> is to retrieve data from a SQL database and store the results in eXist to
> use for later processing. The data contains nearly 5 million rows with only
> 2 columns, so tab separated text would seem to be the most efficient way to
> store this data in eXist. Here is the relevant portion of the query:
>
>
>
> let $data as xs:string :=
>
>     string-join(
>
>         sql:execute($cn, $sh, (), true())/sql:row/string-join(*/text(),
> '&#x9;')
>
>     , '&#xA;')
>
> return xmldb:store-as-binary('/db/temp', 'list.txt', $data)
>
>
>
> Is there a more efficient way to do this in eXist?
>
>
>
> Thanks,
>
> Vincent
>
>
>
> ______________________________________________
>
> *Vincent M. Lizzi*
>
> Head of Information Standards | Taylor & Francis Group
>
> 530 Walnut St., Suite 850, Philadelphia, PA 19106
>
> E-Mail: vin...@ta...
>
> Web: www.tandfonline.com
>
>
>
> Taylor & Francis is a trading name of Informa UK Limited,
>
> registered in England under no. 1072954
>
>
>
> "Everything should be made as simple as possible, but not simpler."
>
>
>
>
>
> Information Classification: General
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

Re: [Exist-open] high memory usage when constructing a string

[Lizzi, V. <Vin...@ta...>]

Hi Alasdair,

Thanks for the response. Here is what I’m trying to do.

The eXist database holds XML documents that were loaded from an archive. The number of documents is approaching 5 million but might be higher due to multiple versions of documents. I want to verify that the holdings in eXist are complete and up to date, and report any differences. The SQL query provides a list containing pairs of a document identifier (unique string) and a version identifier (unique integer). I divided this task into steps with separate XQueries to make it more manageable. The first step is to retrieve this list from a SQL database. The second step is to compare the list with the eXist database (which has range indexes on the identifiers) and add a status column to the list. The third step is to produce a report. The last step will be to use the report to correct any differences.  I previously tried writing one query to do the first three steps, but it seemed to work better as separate smaller steps.

Any suggestions would be welcome.

Vincent

_____________________________________________
Vincent M. Lizzi
Head of Information Standards | Taylor & Francis Group
vin...@ta...<mailto:vin...@ta...>



Information Classification: General
From: Alasdair Dougall <ala...@gm...>
Sent: Thursday, August 10, 2023 5:34 PM
To: Lizzi, Vincent <Vin...@ta...>
Cc: Jean-Paul Rehr <re...@gm...>; Exist-open <exi...@li...>
Subject: Re: [Exist-open] high memory usage when constructing a string

Vincent,

As Jean-Paul said in his reply, there are alternative ways to build large outputs.

Maybe describe the outcome you are trying to achieve as opposed to the issue you are finding.  There are generally more than one way to do things in eXist.

Alasdair
Sent from my iPhone


On 11 Aug 2023, at 1:42 am, Lizzi, Vincent <Vin...@ta...<mailto:Vin...@ta...>> wrote:

Hi JPR,

That sounds like a good idea. Unfortunately, trying to use JSON for this amount of data also results in out of memory errors.

array:join(
(1 to 200000000) ! ['1']
) => serialize(map{'method': 'json'})

I have gotten eXist to process actual data in tab separated text format when eXist is freshly started and has plenty of memory available, but even in this ideal condition memory usage is still unexpectedly high and sometimes produces out of memory errors.

Thanks,
Vincent

_____________________________________________
Vincent M. Lizzi
Head of Information Standards | Taylor & Francis Group
vin...@ta...<mailto:vin...@ta...>



Information Classification: General
From: Jean-Paul Rehr <re...@gm...<mailto:re...@gm...>>
Sent: Thursday, August 10, 2023 10:06 AM
To: Exist-open <exi...@li...<mailto:exi...@li...>>
Subject: Re: [Exist-open] high memory usage when constructing a string

Dear Vincent,

Have you tried outputting JSON structure data from the database? It will be easier to store and access later using Xquery via arrays and maps. You could even chunk it if eXist chokes on a large file - I've never handled one of that size.

Cheers,
JPR

On Thu, Aug 10, 2023 at 3:24 PM Lizzi, Vincent <Vin...@ta...<mailto:Vin...@ta...>> wrote:
Hello eXist folks,

I’m observing that asking eXist to construct a string of about 200 Mb uses more than 4 Gb of memory.

When running a query such as this that constructs a string of 200,000,000 bytes:

string-join(( (1 to 200000000) ! '1' ))

Monex shows the memory usage climbs to over 4 Gb.

This sometimes exhausts the available memory and results in a stack trace being printed exist.log, the top of which shows:

2023-08-09 23:12:40,553 [qtp353417634-34] ERROR (XQueryServlet.java [process]:549) - Java heap space
java.lang.OutOfMemoryError: Java heap space
               at org.exist.dom.memtree.DocumentImpl.addChars(DocumentImpl.java:273) ~[exist-core-6.2.0.jar:6.2.0]
               at org.exist.dom.memtree.MemTreeBuilder.characters(MemTreeBuilder.java:382) ~[exist-core-6.2.0.jar:6.2.0]

I would guess that constructing a document uses a lot more memory than constructing a simple string. Is there a way to signal eXist to use a simple string builder instead?

The example above is a simplification of what I actually want to do, which is to retrieve data from a SQL database and store the results in eXist to use for later processing. The data contains nearly 5 million rows with only 2 columns, so tab separated text would seem to be the most efficient way to store this data in eXist. Here is the relevant portion of the query:

let $data as xs:string :=
    string-join(
        sql:execute($cn, $sh, (), true())/sql:row/string-join(*/text(), '&#x9;')
    , '&#xA;')
return xmldb:store-as-binary('/db/temp', 'list.txt', $data)

Is there a more efficient way to do this in eXist?

Thanks,
Vincent

______________________________________________
Vincent M. Lizzi
Head of Information Standards | Taylor & Francis Group
530 Walnut St., Suite 850, Philadelphia, PA 19106
E-Mail: vin...@ta...<mailto:vin...@ta...>
Web: www.tandfonline.com<http://www.tandfonline.com>

Taylor & Francis is a trading name of Informa UK Limited,
registered in England under no. 1072954

"Everything should be made as simple as possible, but not simpler."



Information Classification: General
_______________________________________________
Exist-open mailing list
Exi...@li...<mailto:Exi...@li...>
https://lists.sourceforge.net/lists/listinfo/exist-open<https://lists.sourceforge.net/lists/listinfo/exist-open>
_______________________________________________
Exist-open mailing list
Exi...@li...<mailto:Exi...@li...>
https://lists.sourceforge.net/lists/listinfo/exist-open<https://lists.sourceforge.net/lists/listinfo/exist-open>

Re: [Exist-open] high memory usage when constructing a string

[Alasdair D. <ala...@gm...>]

<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div dir="ltr"><meta http-equiv="content-type" content="text/html; charset=utf-8">Vincent,<div><br></div><div>As Jean-Paul said in his reply, there are alternative ways to build large outputs.</div><div><br></div><div>Maybe describe the outcome you are trying to achieve as opposed to the issue you are finding. &nbsp;There are generally more than one way to do things in eXist.</div><div><br></div><div>Alasdair&nbsp;<br><br><div dir="ltr">Sent from my iPhone</div><div dir="ltr"><br><blockquote type="cite">On 11 Aug 2023, at 1:42 am, Lizzi, Vincent &lt;Vin...@ta...&gt; wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr">

<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style>@font-face { font-family: "Cambria Math"; }
@font-face { font-family: Calibri; }
@font-face { font-family: Verdana; }
@font-face { font-family: Rockwell; }
p.MsoNormal, li.MsoNormal, div.MsoNormal { margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; }
a:link, span.MsoHyperlink { color: blue; text-decoration: underline; }
span.EmailStyle20 { font-family: Calibri, sans-serif; color: windowtext; }
.MsoChpDefault { font-family: Calibri, sans-serif; }
@page WordSection1 { size: 8.5in 11in; margin: 1in; }
div.WordSection1 { page: WordSection1; }</style>


<div class="WordSection1">
<p class="MsoNormal">Hi JPR,<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">That sounds like a good idea. Unfortunately, trying to use JSON for this amount of data also results in out of memory errors.<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">array:join(<o:p></o:p></p>
<p class="MsoNormal">(1 to 200000000) ! ['1']<o:p></o:p></p>
<p class="MsoNormal">) =&gt; serialize(map{'method': 'json'})<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">I have gotten eXist to process actual data in tab separated text format when eXist is freshly started and has plenty of memory available, but even in this ideal condition memory usage is still unexpectedly high and sometimes produces out
 of memory errors. <o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal">Vincent<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:&quot;Arial&quot;,sans-serif">_____________________________________________<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:&quot;Arial&quot;,sans-serif">Vincent M. Lizzi<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:&quot;Arial&quot;,sans-serif">Head of Information Standards | Taylor &amp; Francis Group
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:&quot;Arial&quot;,sans-serif"><a href="mailto:vin...@ta..."><span style="color:#0563C1">vin...@ta...</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<br>
<p style="font-family:Rockwell;font-size:9pt;color:#0078D7;margin:15pt;font-style:normal;font-weight:normal;text-decoration:none;" align="Left">
Information Classification: General<br>
</p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Jean-Paul Rehr &lt;re...@gm...&gt; <br>
<b>Sent:</b> Thursday, August 10, 2023 10:06 AM<br>
<b>To:</b> Exist-open &lt;exi...@li...&gt;<br>
<b>Subject:</b> Re: [Exist-open] high memory usage when constructing a string<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<div>
<p class="MsoNormal">Dear Vincent,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal">Have you tried outputting JSON structure data&nbsp;from the&nbsp;database? It will be easier to store and access later using Xquery via arrays and maps. You could even chunk it if eXist chokes on a large file - I've never handled one of that size.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal">Cheers,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">JPR<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, Aug 10, 2023 at 3:24 PM Lizzi, Vincent &lt;<a href="mailto:Vin...@ta...">Vin...@ta...</a>&gt; wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hello eXist folks,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I’m observing that asking eXist to construct a string of about 200 Mb uses more than 4 Gb of memory.
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">When running a query such as this that constructs a string of 200,000,000 bytes:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">string-join(( (1 to 200000000) ! '1' ))<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Monex shows the memory usage climbs to over 4 Gb.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">This sometimes exhausts the available memory and results in a stack trace being printed exist.log, the top of which shows:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">2023-08-09 23:12:40,553 [qtp353417634-34] ERROR (XQueryServlet.java [process]:549) - Java heap space
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">java.lang.OutOfMemoryError: Java heap space<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.exist.dom.memtree.DocumentImpl.addChars(DocumentImpl.java:273) ~[exist-core-6.2.0.jar:6.2.0]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.exist.dom.memtree.MemTreeBuilder.characters(MemTreeBuilder.java:382) ~[exist-core-6.2.0.jar:6.2.0]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I would guess that constructing a document uses a lot more memory than constructing a simple string. Is there a way to signal eXist to use a simple string builder instead?<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The example above is a simplification of what I actually want to do, which is to retrieve data from a SQL database and store the results in eXist to use for later processing. The
 data contains nearly 5 million rows with only 2 columns, so tab separated text would seem to be the most efficient way to store this data in eXist. Here is the relevant portion of the query:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">let $data as xs:string :=<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;&nbsp;&nbsp; string-join(<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; sql:execute($cn, $sh, (), true())/sql:row/string-join(*/text(), '&amp;#x9;')<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;&nbsp;&nbsp; , '&amp;#xA;')<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">return xmldb:store-as-binary('/db/temp', 'list.txt', $data)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Is there a more efficient way to do this in eXist?<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Vincent<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:&quot;Arial&quot;,sans-serif">______________________________________________</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:9.0pt;font-family:&quot;Arial&quot;,sans-serif">Vincent M. Lizzi</span></b><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:&quot;Arial&quot;,sans-serif">Head of Information Standards | Taylor &amp; Francis Group
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:&quot;Verdana&quot;,sans-serif">530 Walnut St., Suite 850, Philadelphia, PA 19106</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:&quot;Verdana&quot;,sans-serif">E-Mail:
<a href="mailto:vin...@ta..." target="_blank">vin...@ta...</a></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:&quot;Verdana&quot;,sans-serif">Web:
<a href="http://www.tandfonline.com" target="_blank">
www.tandfonline.com</a></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:&quot;Verdana&quot;,sans-serif">&nbsp;</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:&quot;Verdana&quot;,sans-serif">Taylor &amp; Francis is a trading name of Informa UK Limited,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:&quot;Verdana&quot;,sans-serif">registered in England under no. 1072954</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:&quot;Arial&quot;,sans-serif">&nbsp;</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.0pt;font-family:&quot;Arial&quot;,sans-serif">"Everything should be made as simple as possible, but not simpler."</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p style="margin:15.0pt"><span style="font-size:9.0pt;font-family:&quot;Rockwell&quot;,serif;color:#0078D7">Information Classification: General<o:p></o:p></span></p>
</div>
<p class="MsoNormal">_______________________________________________<br>
Exist-open mailing list<br>
<a href="mailto:Exi...@li..." target="_blank">Exi...@li...</a><br>
<a href="https://lists.sourceforge.net/lists/listinfo/exist-open" target="_blank">https://lists.sourceforge.net/lists/listinfo/exist-open</a><o:p></o:p></p>
</div>
</blockquote>
</div>
</div>


<span>_______________________________________________</span><br><span>Exist-open mailing list</span><br><span>Exi...@li...</span><br><span>https://lists.sourceforge.net/lists/listinfo/exist-open</span><br></div></blockquote></div></div></body></html>

Re: [Exist-open] high memory usage when constructing a string

[Lizzi, V. <Vin...@ta...>]

Hi JPR,

That sounds like a good idea. Unfortunately, trying to use JSON for this amount of data also results in out of memory errors.

array:join(
(1 to 200000000) ! ['1']
) => serialize(map{'method': 'json'})

I have gotten eXist to process actual data in tab separated text format when eXist is freshly started and has plenty of memory available, but even in this ideal condition memory usage is still unexpectedly high and sometimes produces out of memory errors.

Thanks,
Vincent

_____________________________________________
Vincent M. Lizzi
Head of Information Standards | Taylor & Francis Group
vin...@ta...<mailto:vin...@ta...>



Information Classification: General
From: Jean-Paul Rehr <re...@gm...>
Sent: Thursday, August 10, 2023 10:06 AM
To: Exist-open <exi...@li...>
Subject: Re: [Exist-open] high memory usage when constructing a string

Dear Vincent,

Have you tried outputting JSON structure data from the database? It will be easier to store and access later using Xquery via arrays and maps. You could even chunk it if eXist chokes on a large file - I've never handled one of that size.

Cheers,
JPR

On Thu, Aug 10, 2023 at 3:24 PM Lizzi, Vincent <Vin...@ta...<mailto:Vin...@ta...>> wrote:
Hello eXist folks,

I’m observing that asking eXist to construct a string of about 200 Mb uses more than 4 Gb of memory.

When running a query such as this that constructs a string of 200,000,000 bytes:

string-join(( (1 to 200000000) ! '1' ))

Monex shows the memory usage climbs to over 4 Gb.

This sometimes exhausts the available memory and results in a stack trace being printed exist.log, the top of which shows:

2023-08-09 23:12:40,553 [qtp353417634-34] ERROR (XQueryServlet.java [process]:549) - Java heap space
java.lang.OutOfMemoryError: Java heap space
               at org.exist.dom.memtree.DocumentImpl.addChars(DocumentImpl.java:273) ~[exist-core-6.2.0.jar:6.2.0]
               at org.exist.dom.memtree.MemTreeBuilder.characters(MemTreeBuilder.java:382) ~[exist-core-6.2.0.jar:6.2.0]

I would guess that constructing a document uses a lot more memory than constructing a simple string. Is there a way to signal eXist to use a simple string builder instead?

The example above is a simplification of what I actually want to do, which is to retrieve data from a SQL database and store the results in eXist to use for later processing. The data contains nearly 5 million rows with only 2 columns, so tab separated text would seem to be the most efficient way to store this data in eXist. Here is the relevant portion of the query:

let $data as xs:string :=
    string-join(
        sql:execute($cn, $sh, (), true())/sql:row/string-join(*/text(), '&#x9;')
    , '&#xA;')
return xmldb:store-as-binary('/db/temp', 'list.txt', $data)

Is there a more efficient way to do this in eXist?

Thanks,
Vincent

______________________________________________
Vincent M. Lizzi
Head of Information Standards | Taylor & Francis Group
530 Walnut St., Suite 850, Philadelphia, PA 19106
E-Mail: vin...@ta...<mailto:vin...@ta...>
Web: www.tandfonline.com<http://www.tandfonline.com>

Taylor & Francis is a trading name of Informa UK Limited,
registered in England under no. 1072954

"Everything should be made as simple as possible, but not simpler."



Information Classification: General
_______________________________________________
Exist-open mailing list
Exi...@li...<mailto:Exi...@li...>
https://lists.sourceforge.net/lists/listinfo/exist-open<https://lists.sourceforge.net/lists/listinfo/exist-open>

Re: [Exist-open] high memory usage when constructing a string

[Jean-Paul R. <re...@gm...>]

Dear Vincent,

Have you tried outputting JSON structure data from the database? It will be
easier to store and access later using Xquery via arrays and maps. You
could even chunk it if eXist chokes on a large file - I've never handled
one of that size.

Cheers,
JPR

On Thu, Aug 10, 2023 at 3:24 PM Lizzi, Vincent <
Vin...@ta...> wrote:

> Hello eXist folks,
>
>
>
> I’m observing that asking eXist to construct a string of about 200 Mb uses
> more than 4 Gb of memory.
>
>
>
> When running a query such as this that constructs a string of 200,000,000
> bytes:
>
>
>
> string-join(( (1 to 200000000) ! '1' ))
>
>
>
> Monex shows the memory usage climbs to over 4 Gb.
>
>
>
> This sometimes exhausts the available memory and results in a stack trace
> being printed exist.log, the top of which shows:
>
>
>
> 2023-08-09 23:12:40,553 [qtp353417634-34] ERROR (XQueryServlet.java
> [process]:549) - Java heap space
>
> java.lang.OutOfMemoryError: Java heap space
>
>                at
> org.exist.dom.memtree.DocumentImpl.addChars(DocumentImpl.java:273)
> ~[exist-core-6.2.0.jar:6.2.0]
>
>                at
> org.exist.dom.memtree.MemTreeBuilder.characters(MemTreeBuilder.java:382)
> ~[exist-core-6.2.0.jar:6.2.0]
>
>
>
> I would guess that constructing a document uses a lot more memory than
> constructing a simple string. Is there a way to signal eXist to use a
> simple string builder instead?
>
>
>
> The example above is a simplification of what I actually want to do, which
> is to retrieve data from a SQL database and store the results in eXist to
> use for later processing. The data contains nearly 5 million rows with only
> 2 columns, so tab separated text would seem to be the most efficient way to
> store this data in eXist. Here is the relevant portion of the query:
>
>
>
> let $data as xs:string :=
>
>     string-join(
>
>         sql:execute($cn, $sh, (), true())/sql:row/string-join(*/text(),
> '&#x9;')
>
>     , '&#xA;')
>
> return xmldb:store-as-binary('/db/temp', 'list.txt', $data)
>
>
>
> Is there a more efficient way to do this in eXist?
>
>
>
> Thanks,
>
> Vincent
>
>
>
> ______________________________________________
>
> *Vincent M. Lizzi*
>
> Head of Information Standards | Taylor & Francis Group
>
> 530 Walnut St., Suite 850, Philadelphia, PA 19106
>
> E-Mail: vin...@ta...
>
> Web: www.tandfonline.com
>
>
>
> Taylor & Francis is a trading name of Informa UK Limited,
>
> registered in England under no. 1072954
>
>
>
> "Everything should be made as simple as possible, but not simpler."
>
>
>
> Information Classification: General
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

[Exist-open] high memory usage when constructing a string

[Lizzi, V. <Vin...@ta...>]

Hello eXist folks,

I'm observing that asking eXist to construct a string of about 200 Mb uses more than 4 Gb of memory.

When running a query such as this that constructs a string of 200,000,000 bytes:

string-join(( (1 to 200000000) ! '1' ))

Monex shows the memory usage climbs to over 4 Gb.

This sometimes exhausts the available memory and results in a stack trace being printed exist.log, the top of which shows:

2023-08-09 23:12:40,553 [qtp353417634-34] ERROR (XQueryServlet.java [process]:549) - Java heap space
java.lang.OutOfMemoryError: Java heap space
               at org.exist.dom.memtree.DocumentImpl.addChars(DocumentImpl.java:273) ~[exist-core-6.2.0.jar:6.2.0]
               at org.exist.dom.memtree.MemTreeBuilder.characters(MemTreeBuilder.java:382) ~[exist-core-6.2.0.jar:6.2.0]

I would guess that constructing a document uses a lot more memory than constructing a simple string. Is there a way to signal eXist to use a simple string builder instead?

The example above is a simplification of what I actually want to do, which is to retrieve data from a SQL database and store the results in eXist to use for later processing. The data contains nearly 5 million rows with only 2 columns, so tab separated text would seem to be the most efficient way to store this data in eXist. Here is the relevant portion of the query:

let $data as xs:string :=
    string-join(
        sql:execute($cn, $sh, (), true())/sql:row/string-join(*/text(), '	')
    , '
')
return xmldb:store-as-binary('/db/temp', 'list.txt', $data)

Is there a more efficient way to do this in eXist?

Thanks,
Vincent

______________________________________________
Vincent M. Lizzi
Head of Information Standards | Taylor & Francis Group
530 Walnut St., Suite 850, Philadelphia, PA 19106
E-Mail: vin...@ta...<mailto:vin...@ta...>
Web: www.tandfonline.com<http://www.tandfonline.com>

Taylor & Francis is a trading name of Informa UK Limited,
registered in England under no. 1072954

"Everything should be made as simple as possible, but not simpler."



Information Classification: General

Re: [Exist-open] LDAP authentication

[Craig B. <cra...@ma...>]

> On Jul 19, 2023, at 8:21 AM, Jo Calder <Jo....@ha...> wrote:
> 
> I am trying to get LDAP authentication to work against a Samba AD DC setup.  Here's an anonymized version of my config.xml
<snip>
> Does anyone have this or similar setup working?  
> 
> Install details are:
> eXist Version: 5.3.1  [but also relevant on later builds] eXist Build: 20211214004738 Operating System: Linux 3.10.0-1062.12.1.el7.x86_64 amd64 Java Version: 1.8.0_242
> 
> Any tips or hints are very welcome.  Best regards, -- Jo
> 

I've never tried LDAP with eXist, but if you haven't already done so be sure to verify LDAP is working independently of eXist and that the TLS situation is up to snuff (valid, unexpired cert, etc.).  Start with something like:

$ openssl s_client -showcerts samba-ad-dc.local.my-domain.com:636 <http://samba-ad-dc.local.my-domain.com:636/>

There are also various ldapsearch commands, and Samba apparently has logging options that might tell you what the exact authentication error is.

________________________________________
Craig A. Berry

"... getting out of a sonnet is much more
 difficult than getting in."
                 Brad Leithauser

[Exist-open] LDAP authentication

[Jo C. <Jo....@ha...>]

Hi all,

I am trying to get LDAP authentication to work against a Samba AD DC
setup.  Here's an anonymized version of my config.xml

<security-manager xmlns="http://exist-db.org/Configuration" version="2.1">

<authentication-entry-point>/authentication/login</authentication-entry-point>
    <realm id="LDAP" version="1.0" principals-are-case-insensitive="true">
  <context>
    <authentication>
      simple
    </authentication>
    <use-ssl>true</use-ssl>
    <url>ldaps://samba-ad-dc.local.my-domain.com</url>
    <domain>local.my-domain.com</domain>
    <search>
      <base>DC=local,DC=my-domain,DC=com</base>
      <default-username>te...@lo...</default-username>
      <default-password>test-password</default-password>
      <account>
        <search-filter-prefix>objectClass=user</search-filter-prefix>
        <search-attribute key="objectSid">objectSid</search-attribute>
        <search-attribute
key="primaryGroupID">primaryGroupID</search-attribute>
        <search-attribute key="name">sAMAccountName</search-attribute>
        <search-attribute key="dn">distinguishedName</search-attribute>
        <search-attribute key="memberOf">memberOf</search-attribute>
        <metadata-search-attribute key="http://axschema.org/namePerson/first
">givenName</metadata-search-attribute>
        <metadata-search-attribute key="http://axschema.org/contact/email
">mail</metadata-search-attribute>
        <metadata-search-attribute key="http://axschema.org/namePerson/last
">sn</metadata-search-attribute>
        <metadata-search-attribute key="http://axschema.org/namePerson
">name</metadata-search-attribute>
      </account>
      <group>
        <search-filter-prefix>objectClass=group</search-filter-prefix>
        <search-attribute key="member">member</search-attribute>
        <search-attribute
key="primaryGroupToken">primaryGroupToken</search-attribute>
        <search-attribute key="objectSid">objectSid</search-attribute>
        <search-attribute key="name">sAMAccountName</search-attribute>
        <search-attribute key="dn">distinguishedName</search-attribute>
        <whitelist>
          <principal>Domain Users</principal>
        </whitelist>
      </group>
    </search>
    <transformation>
      <add-group>group.users</add-group>
    </transformation>
  </context>
</realm>
<!--<events></events>-->
</security-manager>

based on http://demo.exist-db.org/exist/apps/doc/security and the example
here.

Logins as, e.g., me...@lo... fail with (what looks like) an
unknown exception:

org.exist.security.AuthenticationException:
samba-ad-dc.local.my-domain.com:636
        at
org.exist.security.realm.ldap.LDAPRealm.authenticate(LDAPRealm.java:152)
~[exist-security-ldap-5.3.1.jar:5.3.1]

Does anyone have this or similar setup working?

Install details are:
eXist Version: 5.3.1  [but also relevant on later builds]
eXist Build: 20211214004738
Operating System: Linux 3.10.0-1062.12.1.el7.x86_64 amd64
Java Version: 1.8.0_242

Any tips or hints are very welcome.  Best regards, -- Jo

[Exist-open] Reminder: Declarative Amsterdam Call for Presentations

[Erik S. <er...@xa...>]

Reminder:

The fifth edition of Declarative Amsterdam will take place on 2 and 3 November 2023 at the Science Park, Amsterdam. It will be a hybrid conference with the opportunity to attend live or online, for both attendees and presenters.

 

The first day will feature tutorials, combining presentations and hands-on sessions to give an introduction to specific topics. The second day will be a symposium, with shorter presentations. Speakers can discuss new ideas, frameworks, applications of declarative methods, and best practices.

 

Declarative techniques are a style of computing that expresses the purpose of computation without describing its control flow. It allows you to focus on the ‘what’, rather than the ‘how’.

 

Declarative Amsterdam will have presentations on past experiences, current trends and future perspectives in fields such as functional programming, declarative data modelling, databases, XML and related technologies, JSON, CSS, semantic web, data science, data visualization, grammars, parsing, and domain-specific languages.


Call for Presentations


We invite practitioners, software architects and engineers, academic researchers and others to submit a proposal for a tutorial or a presentation.

      

Tutorials can be between 1.5 and 2.5 hours, and preferably include hands-on sessions for participants. Presentations on the second day can be between 20 and 45 minutes. Speakers can present in person, or prepare a video, and be available online for the question and answer session afterwards.

 

Proposals should include a title, duration and summary (90 - 200 words), and may also include a full paper. Speakers have the option of submitting a full paper or slides, to be published on the Declarative Amsterdam website.

      

For papers and topics from previous years, see the website:  <https://declarative.amsterdam/> https://declarative.amsterdam/

 

Please submit proposals at   <https://declarative.amsterdam/cfp> https://declarative.amsterdam/cfp


Timeline


Submission deadline: 31 July

Acceptance: Beginning of September

Videos: Beginning of October

Conference: 2 and 3 November

 

All the best,

Declarative Amsterdam organizing committee



 

 

[Exist-open] Ranged Queries as XML in ft:query

[Gus R. <gu...@gm...>]

Hello,

I am trying to create a query for the lucene index using an xml fragment as
described here:
https://exist-db.org/exist/apps/doc/lucene

Let's say my query is something like this in the "normal" syntax:

for $hit in collection($config:data-index)//tei:TEI[ft:query(.,
textcontent:$guess AND timemin:[0000 TO 1600], $query_options)]

How would I write the query in XML? I have this for the first part:

<query>
  <wildcard field="textcontent">{$guess}</wildcard>
</query>

But I have no idea which element to use for the ranged query ( timemin:
[0000 TO 1600] ). The example in the documentation just says:

Ranged queries using TO are also supported. Suppose you have marked dates
and wish to return only results between 1600 and 1610.

let $query := "date:[1600 TO 1610]"
return ft:search($col, $query)//exist:match

It does not say how to do it using the XML definition of the query. Non of
the listed elements (wildcard, term, etc.) seems to work.

I would really appreciate some help on this. Thanks a lot!

Gustavo Fernández Riva
Wissenschaftlicher Mitarbeiter
Universität Heidelberg – Universitätsbibliothek

Plöck 107-109, D-69117 Heidelberg
Tel.: +49-(0)6221-54 2033
Mail: gus...@ub...
<gus...@un...>

Re: [Exist-open] Change ownership

[Nicolas S. <sou...@gm...>]

Dear Craig, Dear Joe,

Thank you very much for your help.

Craig's query actually didn't work when I first used it but I installed the
"Shared Resources: jquery, dojo, ace, templating, utilities" package and
everything works now.

-- 
Best regards,

Nicolas

Re: [Exist-open] Change ownership

[Joe W. <jo...@gm...>]

Hi Nicolas,

Craig's query will work if you have the shared-resources package installed,
but if not (it is no longer included in eXist releases), you will receive
an error:

> Cannot compile xquery: err:XQST0059 error found while loading module
dbutil: failed to load module: 'http://exist-db.org/xquery/dbutil' from: '
http://exist-db.org/xquery/dbutil', location: '
http://exist-db.org/xquery/dbutil'. Source not found.

The solution is either to install the shared-resources package via
Dashboard > Package Manager, or to use eXide's bundled version, by changing
the module import declaration to:

import module namespace dbutil="http://exist-db.org/xquery/dbutil" at
"/db/apps/eXide/modules/dbutils.xqm";

Joe

On Tue, Jun 6, 2023 at 3:42 PM Craig Berry via Exist-open <
exi...@li...> wrote:

>
>
> > On Jun 6, 2023, at 2:04 PM, Nicolas Souchon <
> sou...@gm...> wrote:
> >
> > Dear All,
> >
> > I'm new to eXist-db and can't figure out how to change ownership of a
> folder and all the folders and files it contains at once from eXide.
> >
> > I know how to change ownership of one folder or file at a time but how
> can I change ownership of a large number of folders/files at once? Knowing
> that changing the ownership of the folder does not change the ownership of
> the items it contains.
> >
> > To be more clear, my application is owned by user A but one of the
> folders composing it is owned by user B and user A cannot access the
> contents of this folder, so I would like to change the ownership of the
> folder and its content from user B to user A.
> >
> > Thank you in advance for your help.
>
> I had to do this the other day and couldn't find a straightforward way.  I
> ended up writing the following. Be sure to substitute your own root path
> and the desired owner, group and permissions.
>
> xquery version "3.1";
>
> import module namespace dbutil="http://exist-db.org/xquery/dbutil";
> import module namespace console="http://exist-db.org/xquery/console";
>
> declare function local:setperm($path as xs:string) {
>     let $x := sm:chown($path, 'owner')
>     let $x := sm:chgrp($path, 'group')
>     let $x := sm:chmod($path, 'rwxrwxr--')
>     let $x := console:log('Ownership and permissions set on ' || $path)
>     return $x
> };
>
> let $root := '/db/apps/somepath'
> return dbutil:scan(xs:anyURI($root), function($col, $path) {
>     let $x := local:setperm(($path, $col)[1])
>     return $x
> })
>
>
>
>
>
>
>
> ________________________________________
> Craig A. Berry
>
> "... getting out of a sonnet is much more
>  difficult than getting in."
>                  Brad Leithauser
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

Re: [Exist-open] Change ownership

[Craig B. <cra...@ma...>]

> On Jun 6, 2023, at 2:04 PM, Nicolas Souchon <sou...@gm...> wrote:
> 
> Dear All,
> 
> I'm new to eXist-db and can't figure out how to change ownership of a folder and all the folders and files it contains at once from eXide.
> 
> I know how to change ownership of one folder or file at a time but how can I change ownership of a large number of folders/files at once? Knowing that changing the ownership of the folder does not change the ownership of the items it contains. 
> 
> To be more clear, my application is owned by user A but one of the folders composing it is owned by user B and user A cannot access the contents of this folder, so I would like to change the ownership of the folder and its content from user B to user A.
> 
> Thank you in advance for your help.

I had to do this the other day and couldn't find a straightforward way.  I ended up writing the following. Be sure to substitute your own root path and the desired owner, group and permissions.

xquery version "3.1";

import module namespace dbutil="http://exist-db.org/xquery/dbutil";
import module namespace console="http://exist-db.org/xquery/console";

declare function local:setperm($path as xs:string) {
    let $x := sm:chown($path, 'owner')
    let $x := sm:chgrp($path, 'group')
    let $x := sm:chmod($path, 'rwxrwxr--')
    let $x := console:log('Ownership and permissions set on ' || $path)
    return $x
};

let $root := '/db/apps/somepath'
return dbutil:scan(xs:anyURI($root), function($col, $path) {
    let $x := local:setperm(($path, $col)[1])
    return $x
})







________________________________________
Craig A. Berry

"... getting out of a sonnet is much more
 difficult than getting in."
                 Brad Leithauser

[Exist-open] Change ownership

[Nicolas S. <sou...@gm...>]

Dear All,

I'm new to eXist-db and can't figure out how to change ownership of a
folder and all the folders and files it contains at once from eXide.

I know how to change ownership of one folder or file at a time but how can
I change ownership of a large number of folders/files at once? Knowing that
changing the ownership of the folder does not change the ownership of the
items it contains.

To be more clear, my application is owned by user A but one of the folders
composing it is owned by user B and user A cannot access the contents of
this folder, so I would like to change the ownership of the folder and its
content from user B to user A.

Thank you in advance for your help.

-- 
Best regards,

Nicolas