exist-open — Open discussion

[Exist-open] File upload leads to database crash

[Jannik F. <fr...@md...>]

Dear eXist-db community,

in our project we are using eXist-db for over two years now and since a
few weeks we are having a problem uploading a specific XML-file (via
eXide's upload interface).
As soon as the file is uploaded, the database crashes and the server has
to be restarted.
After the restart there remains an empty file in the database as a
result of the interrupted upload process.


The problem occurs only with this single file. It is a simple TEI-XML
document, 68KB small and not special in any way (and it didn't cause any
problems for the last 2 years).

Attached is a screenshot of the Monex report after uploading the file.


Has anyone had a similar problem or has any advice how to deal with it?

Many thanks in advance and best regards
Jannik Franz




Jannik Franz

Universität für Musik und darstellende Kunst Wien
Institut für Musikwissenschaft und Interpretationsforschung
Abteilung Wissenschaftszentrum Arnold Schönberg und die Wiener Schule
Schwarzenbergplatz 6/Zaunergasse 1-3
1030 Wien
E-Mail: fr...@md...

[Exist-open] storeResouce is too slow with remote interface

[Vladimir P. <pra...@gm...>]

eXistDB 6.2.0

Saving to a locally running database via the remote interface is several orders of magnitude slower compared to using an embedded database. I can't figure out where the problem lies, or how to store the data more efficiently.

I need to store about 100,000 records. When saving via the remote interface, the speed is around 100 records per second. However, when using the embedded database, the speed is about 9,000 records per second.

What am I doing wrong, or how can I improve this?

Below is a code example.



remote uri: xmldb:exist://localhost:8080/exist/xmlrpc
embeded uri: xmldb:exist://

In both cases the db is running on the same machine. db configuration file is attached.

private void createRandomTestData(String prefix, int records) throws Exception {
    final Random random = new Random();
    final HashMap<String, String> data = new HashMap<>();

    for (int i = 0; i < records; i++) {
        double latitude = MIN_LAT + (MAX_LAT - MIN_LAT) * random.nextDouble();
        double longitude = MIN_LON + (MAX_LON - MIN_LON) * random.nextDouble();

        String randomId = UUID.randomUUID().toString();
        String randomName = generateRandomString(random, 30);
        String randomValue = generateRandomString(random, 10);

        String resourceId = prefix + "/" + randomId;
        String xml = String.format(
                """
                        <entity>
                            <name>%s</name>
                            <value>%s</value>
                            <pos>%.8f %.8f</pos>
                        </entity>""",
                randomName, randomValue, latitude, longitude
        );
        data.put(resourceId, xml);
    }
    xmlDbService.saveEntity(data);
}

public void saveEntity(@NonNull HashMap<String, String> resources) throws Exception {
    var sortedResources = new TreeMap<>(resources);

    Collection col = null;
    String currentCollectionUri = null;
    long start = System.currentTimeMillis();

    int i = 0;
    int c = 0;
    try {
        for (var entry : sortedResources.entrySet()) {
            String resourceId = entry.getKey();
            String xmlData = entry.getValue();

            var param = splitResourceId(resourceId);
            String collectionUri = param[0];
            String resourceName = param[1];

            if (col == null || !collectionUri.equals(currentCollectionUri)) {
                if (col != null) {
                    col.close();
                }
                col = getOrCreateCollection(collectionUri);
                col.setProperty("indent", "no");
                currentCollectionUri = collectionUri;
            }

            i++;
            if (i % 1000 == 0) {
                long executionTime = System.currentTimeMillis() - start;
                log.info("Inserted: {}, rate {} / sec", i, c * 1000L / executionTime);
                start = System.currentTimeMillis();
                c = 0;
            }
            XMLResource res = (XMLResource) col.createResource(resourceName, XMLResource.RESOURCE_TYPE);
            res.setContent(xmlData);
            col.storeResource(res);
            c++;
            log.trace("Resource saved: {} in collection: {}", resourceName, currentCollectionUri);
        }
    } finally {
        if (col != null) {
            col.close();
        }
    }
}

<dependency>
    <groupId>org.exist-db</groupId>
    <artifactId>exist-core</artifactId>
    <version>6.2.0</version>
</dependency>

<dependency>
    <groupId>net.sf.xmldb-org</groupId>
    <artifactId>xmldb-api</artifactId>
    <version>1.7.0</version>
</dependency>
Thank you for your help and/or advice.

V.


Re: [Exist-open] eXist-db repair and reindex process taking a very long time

[Kampkaspar, D. <dar...@tu...>]

Hi Vincent,

when I’m sure that no write has taken place, I go along the same route as Pieter Lamers said in his post, i.e. deleting the log and all lock files. eXist then does not notice there has been an interruption and it should come up without re-index.

As I, too, have a lot of files and a large index definition, I usually try to avoid re-indexing. Meaning, if I know it’s going to re-index, I delete the whole thing and just do a re-ingest of all files from a recent DB dump. 1) you can go through your files collection by collection instead of doing everything at once (which gives you a much better overview of where in the process you actually are); and 2) the DB is not completely locked (which it is during the automatic re-index).
In my experience, the re-ingest is not significantly slower* than the automatic re-index and to me the benefit of knowing where you are in the process is worth more than a small speed bonus.

* While I have not timed it, it feels than a re-ingest is faster than the re-index. But I do not know nearly enough about eXist’s internal handling of re-indexing and indexing when ingesting to say whether there’s really a difference or other effects play into this.
Of course, when re-ingesting from a dump and that dump is situated on a different piece of hardware, a re-ingest would also benefit from the separation of the I/O tasks which might give some added boost (again, I have not really timed this in a clean environment).

--

That being said, almost 1 month of re-indexing and 100GB mem usage seem quite a lot. I do not dare say “excessive“ as I do not know your data and index definition, but my data’s usually done after about 3 to 4 days (no dedicated hardware and a max. of 6 GB RAM – a re-ingest is around 2–3 days, sometimes less).

How big are you data\lucene and data\range directories and how big are the dbx files? That might help compare these figures.

All best,
Dario


Am Dienstag, dem 10.09.2024 um 19:10 +0000 schrieb Lizzi, Vincent:
Hello eXist-db community,

By comparing the folders data\lucene and data\range to a previous backup, going by the number of files and total size of these folders, it looks like the reindexing process is about 50% done. The reindexing process has been running since August 13 and is using over 100 Gb of memory.

Is there any way to start eXist-db and allow it to go through its recovery process but stop it from reindexing database files?  I’m wondering if that could be a way to get the database operational again, and then I could manually run xmldb:reindex().

Thanks,
Vincent

_____________________________________________
Vincent M. Lizzi
Head of Information Standards | Taylor & Francis Group
vin...@ta...<mailto:vin...@ta...>



Information Classification: General
From: Lizzi, Vincent
Sent: Tuesday, September 3, 2024 11:36 AM
To: Exist-open <exi...@li...>
Subject: eXist-db repair and reindex process taking a very long time

Hello eXist-db community,

I’ve been monitoring an eXist-db database that is going through its automated recovery process, and am wondering if there is any way to get more information about its progress and how soon the process will finish.

This eXist-db has full text indexing and range indexes configured on several large collections. The EC2 server on which eXist-db is hosted had an outage. When eXist-db was restarted its automatic repair process began. That was about 3 weeks ago. Through Windows Resource Monitor I can see that the eXist-db process is reading from dob.dbx and structure.dbx and writing to structure.dbx and writing to files in the “lucene” and “range” folders, and the process is using about 6% of CPU consistently, and memory usage has increased gradually to over 100 GB. The last line in exist.log is still:

2024-08-13 17:02:59,710 [main] INFO  (NativeBroker.java [repair]:3692) - Reindexing database files ...

Is there any way to find out more what the process is doing, estimate when it will finish, or release any bottlenecks, without interrupting the process?

Thanks,
Vincent

______________________________________________
Vincent M. Lizzi
Head of Information Standards | Taylor & Francis Group
530 Walnut St., Suite 850, Philadelphia, PA 19106
E-Mail:vin...@ta...<mailto:vin...@ta...>
Web:www.tandfonline.com<http://www.tandfonline.com>

Taylor & Francis is a trading name of Informa UK Limited,
registered in England under no. 1072954

"Everything should be made as simple as possible, but not simpler."

_______________________________________________
Exist-open mailing list
Exi...@li...<mailto:Exi...@li...>
https://lists.sourceforge.net/lists/listinfo/exist-open


--

Dario Kampkaspar
Leitung ZEiD – Zentrum für digitale Editionen
Universitäts- und Landesbibliothek Darmstadt

Postadresse:
Magdalenenstr. 8
64289 Darmstadt

Besucheradresse:
Residenzschloss 1
64283 Darmstadt

+49 6151 16-76292
+49 151 29121599

Re: [Exist-open] eXist-db repair and reindex process taking a very long time

[Pieter L. <pie...@be...>]

Hi Vincent,

When I want to skip the re-indexing I stop, remove journal.lck and *.log 
from the exist-db/data folder and restart.
If the database was taken down during an update then this will lead to 
corruption. If no write was taking place, then usually (to my 
experience) this is harmless.
Please note that my exist-db/data folder is not the exist resource data 
folder (which we put somewhere else) but the journaling folder. There 
are usually only two files in that one.

good luck! If I end up needing to re-index on our 20GB data set it 
usually takes an hour and a half. But I might be using lighter indexes 
than you, so not sure if this is comparable.

Best,
Pieter


On 10/09/2024 21:10, Lizzi, Vincent wrote:
>
> Hello eXist-db community,
>
> By comparing the folders data\lucene and data\range to a previous 
> backup, going by the number of files and total size of these folders, 
> it looks like the reindexing process is about 50% done. The reindexing 
> process has been running since August 13 and is using over 100 Gb of 
> memory.
>
> Is there any way to start eXist-db and allow it to go through its 
> recovery process but stop it from reindexing database files?  I’m 
> wondering if that could be a way to get the database operational 
> again, and then I could manually run xmldb:reindex().
>
> Thanks,
>
> Vincent
>
> _____________________________________________
>
> *Vincent M. Lizzi*
>
> Head of Information Standards | Taylor & Francis Group
>
> vin...@ta... 
> <mailto:vin...@ta...>
>
>
> Information Classification: General
>
> *From:*Lizzi, Vincent
> *Sent:* Tuesday, September 3, 2024 11:36 AM
> *To:* Exist-open <exi...@li...>
> *Subject:* eXist-db repair and reindex process taking a very long time
>
> Hello eXist-db community,
>
> I’ve been monitoring an eXist-db database that is going through its 
> automated recovery process, and am wondering if there is any way to 
> get more information about its progress and how soon the process will 
> finish.
>
> This eXist-db has full text indexing and range indexes configured on 
> several large collections. The EC2 server on which eXist-db is hosted 
> had an outage. When eXist-db was restarted its automatic repair 
> process began. That was about 3 weeks ago. Through Windows Resource 
> Monitor I can see that the eXist-db process is reading from dob.dbx 
> and structure.dbx and writing to structure.dbx and writing to files in 
> the “lucene” and “range” folders, and the process is using about 6% of 
> CPU consistently, and memory usage has increased gradually to over 100 
> GB. The last line in exist.log is still:
>
> 2024-08-13 17:02:59,710 [main] INFO (NativeBroker.java [repair]:3692) 
> - Reindexing database files ...
>
> Is there any way to find out more what the process is doing, estimate 
> when it will finish, or release any bottlenecks, without interrupting 
> the process?
>
> Thanks,
>
> Vincent
>
> ______________________________________________
>
> *Vincent M. Lizzi*
>
> Head of Information Standards | Taylor & Francis Group
>
> 530 Walnut St., Suite 850, Philadelphia, PA 19106
>
> E-Mail: vin...@ta... 
> <mailto:vin...@ta...>
>
> Web: www.tandfonline.com <http://www.tandfonline.com>
>
> Taylor & Francis is a trading name of Informa UK Limited,
>
> registered in England under no. 1072954
>
> "Everything should be made as simple as possible, but not simpler."
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

-- 
Pieter Lamers
John Benjamins Publishing Company
Postal Address: P.O. Box 36224, 1020 ME AMSTERDAM, The Netherlands
Visiting Address: Klaprozenweg 75G, 1033 NN AMSTERDAM, The Netherlands
Warehouse: Kelvinstraat 11-13, 1446 TK PURMEREND, The Netherlands
tel: +31 20 630 4747
web:www.benjamins.com

Re: [Exist-open] eXist-db repair and reindex process taking a very long time

[Lizzi, V. <Vin...@ta...>]

Hello eXist-db community,

By comparing the folders data\lucene and data\range to a previous backup, going by the number of files and total size of these folders, it looks like the reindexing process is about 50% done. The reindexing process has been running since August 13 and is using over 100 Gb of memory.

Is there any way to start eXist-db and allow it to go through its recovery process but stop it from reindexing database files?  I'm wondering if that could be a way to get the database operational again, and then I could manually run xmldb:reindex().

Thanks,
Vincent

_____________________________________________
Vincent M. Lizzi
Head of Information Standards | Taylor & Francis Group
vin...@ta...<mailto:vin...@ta...>



Information Classification: General
From: Lizzi, Vincent
Sent: Tuesday, September 3, 2024 11:36 AM
To: Exist-open <exi...@li...>
Subject: eXist-db repair and reindex process taking a very long time

Hello eXist-db community,

I've been monitoring an eXist-db database that is going through its automated recovery process, and am wondering if there is any way to get more information about its progress and how soon the process will finish.

This eXist-db has full text indexing and range indexes configured on several large collections. The EC2 server on which eXist-db is hosted had an outage. When eXist-db was restarted its automatic repair process began. That was about 3 weeks ago. Through Windows Resource Monitor I can see that the eXist-db process is reading from dob.dbx and structure.dbx and writing to structure.dbx and writing to files in the "lucene" and "range" folders, and the process is using about 6% of CPU consistently, and memory usage has increased gradually to over 100 GB. The last line in exist.log is still:

2024-08-13 17:02:59,710 [main] INFO  (NativeBroker.java [repair]:3692) - Reindexing database files ...

Is there any way to find out more what the process is doing, estimate when it will finish, or release any bottlenecks, without interrupting the process?

Thanks,
Vincent

______________________________________________
Vincent M. Lizzi
Head of Information Standards | Taylor & Francis Group
530 Walnut St., Suite 850, Philadelphia, PA 19106
E-Mail: vin...@ta...<mailto:vin...@ta...>
Web: www.tandfonline.com<http://www.tandfonline.com>

Taylor & Francis is a trading name of Informa UK Limited,
registered in England under no. 1072954

"Everything should be made as simple as possible, but not simpler."

[Exist-open] Storing data retrieved javascript fetch command into global variable

[Ted H. <meg...@gm...>]

If I could use the await command, I would do so.  But when I use the await
command, I get the following error in the eXist database:  async functions
are only available in es8.  And I don't know how to set eXist so it will
use es8.  But I digress.

What I want to do is use one function to retrieve data in a .json file and
then use another function to store data from the JSON file into global
variables.  Then I will use those variables to set attributes of SVG
objects.  But I'm doing something wrong and I don't know what I'm doing
wrong.  Here is my code so far:

My HTML Code:

<html xmlns="http://www.w3.org/1999/xhtml" xmlns:svg="
http://www.w3.org/2000/svg" xmlns:xs="http://www.w3.org/1999/xhtml"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
http://www.w3.org/1999/xhtml SVG_Bezier_Curve_Webpage_XML_Schema.xsd">
    <head>
         <title>SVG_Diagonal_Line</title>
         <link rel="stylesheet" type="text/css" href="
http://localhost:8080/exist/rest/db/apps/HTML_Student/SVG_Diagonal_Line.css
"/>
         <script language="javascript" src="
http://localhost:8080/exist/rest/db/apps/HTML_Student/SVG_Diagonal_Line.js
"/>
    </head>
    <body onload="Store_Data()">
         <div id="Button_Box">
              <input type="text" id="Data_Text"/>
         </div>
         <div id="SVG_Position">
              <svg:svg id="My_SVG" height="500px" width="600px">
              <svg:line id="Diagonal_Line_1" x1="300px" x2="395px"
y1="200px" y2="300px"/>
              <svg:line id="Diagonal_Line_2" x1="500px" x2="405px"
y1="200px" y2="300px"/>
              <svg:rect id="Base_Rectangle"/>
              <svg:ellipse id="Germanium_Ellipse"/>
              <svg:ellipse id="Battery_1_Bottom" rx="20" ry="10" cx="250"
cy="350" fill="green" stroke="black" stroke-width="3"/>
              <svg:rect id="Battery_1__Body" x="230px" y="300px"
width="40px" height="50px" stroke="black" stroke-width="3" fill="green"/>
              <svg:ellipse id="Battery_1_Top" rx="20" ry="10" cx="250"
cy="300" stroke="black" stroke-width="3"/>
              <svg:rect id="Battery_2_Body" x="530px" y="300px"
width="40px" height="50px" stroke="black" stroke-width="3" fill="green"/>
              <svg:path id="Battery_2_Bottom" d="M 530,350 A 20,10 0,0,0
570,350" stroke="black" stroke-width="3" fill="green"/>
              <svg:ellipse id="Battery_2_Top" rx="20" ry="10" cx="550"
cy="300" stroke="black" stroke-width="3"/>
              </svg:svg>
         </div>
    </body>
</html>

My CSS Code:

#Diagonal_Line_1{
position:absolute;
stroke: #FFD700;
stroke-width: 3;
fill: none;}
#Diagonal_Line_2{
stroke: #FFD700;
stroke-width: 3;
fill: none;}
#Data_Text{
position: absolute;
top: 50px;
left: 300px;
height: 50px;
width: 300px;}
#My_Svg{
position: absolute;
top: 300px;
left: 200px;
height: 500px;
width: 500px;}
#Button_Box{
position: absolute;
left: 150px;}
#Base_Rectangle{
position: absolute;
height: 15px;
width: 100px;
fill: #b87333;
stroke: black;
stroke-width: 2;}
Germanium_Ellipse{
position: absolute;
fill: #d5d5d7;
stroke: blue;
stroke-width: 2;}

My Javascript Code:

var Retrieved_Data;
var Rectangle_X_Coordinate;
var URL;
var Rectangle_Object;
var Rectangle_Y_Coordinate;
var Germanium_Object;
var Germanium_X_Radius;
var Germanium_Y_Radius;
var Germanium_X_Center;
var Germanium_Y_Center;
var Set_Attributes;

function Retrieve_Data(){

        URL = '
http://localhost:8080/exist/rest/db/apps/HTML_Student/SVG_Diagonal_Line.json
';

         fetch(URL)
             .then ((res) => response.json());



}

function Store_Data() {



        Retrieved_Data = Retrieve_Data();


            document.getElementById("Data_Text").value =
data.Diagonal_Lines.Diagonal_Line_2.First_X1_Coordinate;
            document.getElementById("Data_Text").value =
data.Diagonal_Lines.Base_Rectangle.Rectangle_Y_Coordinate;
            Rectangle_Object = document.getElementById('Base_Rectangle');
            Germanium_Object = document.getElementById('Germanium_Ellipse');
            Rectangle_X_Coordinate =
JSON.stringify(Retrieved_Data.Rectangle_X_Coordinate);
            Rectangle_Y_Coordinate =
JSON.stringify(data.Diagonal_Lines.Base_Rectangle.Rectangle_Y_Coordinate);
            Germanium_X_Radius =
JSON.stringify(data.Diagonal_Lines.Germanium_Ellipse.Germanium_X_Radius);
            Germanium_Y_Radius =
JSON.stringify(data.Diagonal_Lines.Germanium_Ellipse.Germanium_Y_Radius);
            Germanium_X_Center =
JSON.stringify(data.Diagonal_Lines.Germanium_Ellipse.Germanium_X_Center);
            Germanium_Y_Center =
JSON.stringify(data.Diagonal_Lines.Germanium_Ellipse.Germanium_Y_Center);
            document.getElementById("Data_Text").value =
Rectangle_X_Coordinate;


    }

function Draw_Objects(){

            Draw_Objects(Rectangle_X_Coordinate, Rectangle_Y_Coordinate);
            document.getElementById("Data_Text").value =
Rectangle_X_Coordinate;

            Rectangle_Object.setAttribute('x', Rectangle_X_Coordinate);
            Rectangle_Object.setAttribute('y', Rectangle_Y_Coordinate);
            Germanium_Object.setAttribute('rx', Germanium_X_Radius);
            Germanium_Object.setAttribute('ry', Germanium_Y_Radius);
            Germanium_Object.setAttribute('cx', Germanium_X_Center);
            Germanium_Object.setAttribute('cy', Germanium_Y_Center);
            Germanium_Object.style = "fill: #d5d5d7; stroke: black;
stroke-width: 3;"

And my JSON Data:

{"Diagonal_Lines": {
     "Diagonal_Line_1":
          {"First_X1_Coordinate": "200px",
          "Second_X1_Coordinate": "295px",
          "First_Y1_Coordinate": "200px",
          "Second_Y1_Coordinate": "300px"},
       "Diagonal_Line_2":
           {"First_X1_Coordinate": "400px",
           "Second_X1_Coordinate": "305px",
           "First_Y1_Coordinate": "200px",
           "Second_Y1_Coordinate": "300px"},
        "Base_Rectangle":
           {"Rectangle_X_Coordinate": 350,
           "Rectangle_Y_Coordinate": 300},
   "Germanium_Ellipse":
       {"Germanium_X_Radius": 90,
        "Germanium_Y_Radius": 25,
        "Germanium_X_Center": 400,
        "Germanium_Y_Center": 340},
    "Battery_1_Bottom":
       {"Battery_1_X_Radius": 20,
        "Battery_1_Y_Radius": 10,
        "Battery_1_X_Center": 250,
        "Battery_1_Y_Center": 350},
    "Battery_1_Body":
       {"Battery_1_X": 230,
        "Battery_1_Y": 300},
    "Battery_2_Top":
       {"Battery_2_X_Radius": 20,
        "Battery_2_Y_Radius": 10,
        "Battery_2_X_Center": 250,
        "Battery_2_Y_Center": 350},
    "Battery_2_Body":
       {"Battery_2_X": 530,
        "Battery_2_Y": 300}}}

[Exist-open] Declarative Amsterdam 2024 - Register now

[Erik S. <er...@xa...>]

Declarative Amsterdam 2024 will be happening at CWI, Amsterdam Science Park on Thursday/Friday November 7 and 8, 2024.

*	On Thursday morning we start with tutorials on XForms, in-source testing and iXML. Make sure to bring your own device to participate optimally in the hands-on sessions.
*	The symposium will take place on Thursday afternoon and continue on Friday.

The program offers a blend of renowned speakers alongside lesser-known experts. The presentations cover a wide range of topics, including various techniques, tools, applications, and implementations. As in previous editions, we have an engaging and informative lineup for our attendees, as you can see for yourself at  <https://declarative.amsterdam/program> https://declarative.amsterdam/program

The conference is a hybrid event, live at the Science Park in Amsterdam, and live-streamed.
Registration is open; early bird registration closes October 1st, so hurry along to  <https://declarative.amsterdam/registration> https://declarative.amsterdam/registration. 

Best wishes,

The Declarative Amsterdam Conference Committee.

 

Re: [Exist-open] Fwd: Cross reference Index - why is so slow

[Lizzi, V. <Vin...@ta...>]

Hi Lars,

I’ve had a few observations from using eXist-db that might be of interest.


  *   Queries that take a long time to run and access documents in different collections can run be made to run in less time by placing the documents into a single collection.


  *   Simultaneously reading from and writing to the same collection can cause blocks.



  *   The tracing function in MoneX is useful to see what indexes are being used by a query.

It looks like your Person XML and Article XML are using different namespaces, so that would provide a way differentiate the two kinds of documents if they are in the same collection.

I hope this is helpful in some way.

Kind regards,
Vincent

_____________________________________________
Vincent M. Lizzi
Head of Information Standards | Taylor & Francis Group
vin...@ta...<mailto:vin...@ta...>



Information Classification: General
From: Lars Scheidelerl <sch...@sa...>
Sent: Tuesday, September 3, 2024 7:46 AM
To: Claudius Teodorescu <cla...@gm...>; exi...@li...
Subject: Re: [Exist-open] Fwd: Cross reference Index - why is so slow


Hey Claudius,

thank you for the insight.

But with the solution you presented, I ask myself why should I use eXistDb at all?
Then I could simply use an XML parser and read the fields directly from the files at file level or use another document-based DB that can do XML.
But I want to use the features that are in a complete system - I think the diversions with a custom search engine for combined fields is somehow good, but also thought around the corner.
Surely there must be a solution to the problem in eXistdb?

Best regards
Lars
Am 03.09.24 um 11:27 schrieb Claudius Teodorescu:
Dear Lars,

My name is Claudius Teodorescu, and I am currently working for the Academy of Mainz.

For the cases with publishing dictionaries (around 8, see https://clre.solirom.ro<https://clre.solirom.ro>), I have chosen the static website approach, along with static indexes, and a static search engine that is browser-based and is written in Rust and compiled to WebAssembly.

One can see some test data from the DWDS dictionary (around 250,000 entries, and four indexes) at https://claudius-teodorescu.gitlab.io/dwds-site/<https://claudius-teodorescu.gitlab.io/dwds-site/>.

Maybe this is of any help for you.

I think that I do not have to mention that the indexing takes seconds, as I have written the indexing engine in Rust language. :)

Best regards,
Claudius

On Tue, 3 Sept 2024 at 11:30, Lars Scheidelerl <sch...@sa...<mailto:sch...@sa...>> wrote:
Dear Boris,

thank you very much for the answer.

The person data and the article data are in different collections.
For every person we have one file, for every article we have one file.

We don't have a fixed order of when data is imported or written in,
as I understand it, what you are suggesting would make it necessary for
all the person or item data to
always be available before the index of the other collection, so that it
can be accessed in the index of the other.

Maybe we have to, like you pointed out, tests the module functions for
better performance.
But in the past however, after several different approaches that we have
already tried, we have realized that querying the data,
no matter how good it is in eXide, for example, is significantly slower
when it is used for the index.
Could it be that the structural index is not used when re-indexing?
Our assumption was that the data is iterated over differently or that
new blobs are written over the structure and content in addition to
indexing.
In other words, that it validates, saves and indexes the data at the
same time as re-indexing.

Other approaches include the use of cache or a helper file, where the
fields are composed beforehand and then indexed accordingly,
but this also takes a long time and unfortunately also blocks working on
the files.
So if we write a helper file with all fields in eXide, the whole process
takes about 90s, if we work with the fields as we would build the index,
i.e. with xml:id in the helper file, ~110s.
Re-Index 2-4 hours. Not really understandable.

We are now primarily trying to improve this using the xml:id/id()
function, but not much hope to improve the re-index on a production scale.
But if all data is re-indexed, i.e. the xml:id fields are not available,
it is in vain, too.

Would love to learn more about this topic and continue to share experiences.

Lars

Am 03.09.24 um 00:49 schrieb Boris Lehečka:
> Dear Lars,
>
>     I have similar issues with indexing dictionaries: my indexing
> procedure ask for data from the taxonomy (like expansions of
> abbreviations) and sometimes indexing a dictionary with about 36,000
> entries took a whole day.
>
>     I don't remember who (Juri Leino, I guess), pointed out to me that
> the index is saved only after the whole document is parsed. After
> moving each dictionary entry to a separate file, indexing took much
> less time (several hours).
>
>     However, this does not seem to be the cause of your problem.
>
>     In my opinion, your indexing code (in the module) is very
> complicated, sometimes it can be much simpler, for example without
> explicit conversion to string (like in tei:persName[string(@ref) eq
> $identifier ...]), or normalizing spaces in ip:getFullText (full-tex t
> search usually uses only the parts between spaces).
>
>     My suggestion is following: first, create an index for persons in
> a separate collection (with separate collection.xconf): compute fields
> with values you will query or want to return when you index the
> articles (in different collection). And second, use Lucene and
> fulltext search in your "index-persons" module to find data in the
> index from the first phase.
>
>   This is just an idea, not tested, I hope someone else is much more
> experienced in the magic of indexing.
>
>    Best,
>
>     Boris Lehečka
>
> Dne 02.09.2024 v 16:57 Lars Scheidelerl napsal(a):
>> Hey,
>>
>> we assume that we are not using the index in our project as intended.
>> Because when we try to build the index we have created, it takes a
>> very long time.
>>
>> We have two collections, one with a data stack of 687 in which data
>> is stored, and one with 400 xml where articles are are stored.
>>
>> For the personal article we want certain information from the
>> articles and vice versa.
>>
>> Person XML:
>>
>> <person xml:id="i0c9ab7e2-2e21-39ff-aea8-c56ad4702a7f" status="safe"
>> modified="2024-07-30T13:26:09.154+02:00">
>>             <name>Marcanton Zimara</name>
>>             <identifier
>> preferred="YES">https://d-nb.info/gnd/120156784<https://d-nb.info/gnd/120156784></identifier>
>>             <alternateName>Marcusantonius Zimara</alternateName>
>>             <alternateName>Marcus Anthonius Zimara</alternateName>
>>             <alternateName>Antonius Zimara</alternateName>
>>             <alternateName>M. Antonius Zimarra</alternateName>
>>             <alternateName>Marc Antoine Zimara</alternateName>
>>             <alternateName>M. Anto. Zimare</alternateName>
>>             <alternateName>Marco A. Zimara</alternateName>
>>             <alternateName>Marcus A. Zimara</alternateName>
>>             <alternateName>Marc Ant. Zimara</alternateName>
>>             <alternateName>Marcantonio Zimara</alternateName>
>>             <alternateName>Marcus Antonius Zimara</alternateName>
>>             <alternateName>Marcus Antonius Zimarra</alternateName>
>>             <alternateName>Marcianto Zimare</alternateName>
>>             <alternateName>Marco Antonio Zimarra</alternateName>
>>             <alternateName>Marco Antonio Zimare</alternateName>
>>             <birthDate>1460</birthDate>
>>             <deathDate>1532</deathDate>
>>             <description>JWO</description>
>>             <sortableName>Zimara, Marcanton </sortableName>
>>         </person>
>>
>> Articel XML:
>>
>> <?xml version="1.0" encoding="UTF-8"?>
>> <TEI xmlns="http://www.tei-c.org/ns/1.0<http://www.tei-c.org/ns/1.0>">
>>     <teiHeader>
>>         <fileDesc>
>>             <titleStmt>
>>                 <title>a nihilo nihil fit</title>
>>                 <author>
>>                     <persName
>> ref="/db/projects/jwo/data/lists/personenListe.xml#BS_d1e509"
>> xml:id="author_BS_d1e509">
>> <forename>Marcanton</forename>
>>                         <surname>Zimara</surname>
>>                     </persName>
>>                 </author>
>>             </titleStmt>
>>             <sourceDesc>
>>                 <p xml:id="p_sourceDesc_igw_tvr_pzb">born digital</p>
>>             </sourceDesc>
>>         </fileDesc>
>>     </teiHeader>
>>     <text xml:lang="de-DE" type="main">
>>         <body>
>>             <div1 xml:id="div1_d1e23_2">
>>                 <p xml:id="p_d1e27_1" n="1"> Lorem ipsum dolor sit
>> amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt
>> ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis
>> nostrud exercitation <persName xml:id="persName_sa123"
>> ref="https://d-nb.info/gnd/120156784<https://d-nb.info/gnd/120156784>"
>> rend="smallcaps">Zimara</persName> ullamco laboris nisi ut aliquip ex
>> ea commodo consequat. Duis aute irure dolor in reprehenderit in
>> voluptate velit esse cillum dolore eu <persName
>> xml:id="persName_s123"
>> ref="https://d-nb.info/gnd/120156784<https://d-nb.info/gnd/120156784>">Zimara</persName> fugiat nulla
>> pariatur. Excepteur sint occaecat cupidatat non proident, sunt in
>> culpa qui officia deserunt mollit anim id est laborum. </p>
>>             </div1>
>>         </body>
>>     </text>
>> </TEI>
>>
>> Collection.xconf:
>>
>> <collection xmlns="http://exist-db.org/collection-config/1.0<http://exist-db.org/collection-config/1.0>">
>>     <index xmlns:gndo="https://d-nb.info/standards/elementset/gnd#<https://d-nb.info/standards/elementset/gnd#>"
>> xmlns:owl="http://www.w3.org/2002/07/owl#<http://www.w3.org/2002/07/owl#>"
>> xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#<http://www.w3.org/1999/02/22-rdf-syntax-ns#>"
>> xmlns:xs="http://www.w3.org/2001/XMLSchema<http://www.w3.org/2001/XMLSchema>">
>>         <lucene>
>>             <module uri="http://place.sok.org/xquery/index-persons<http://place.sok.org/xquery/index-persons>"
>> prefix="ip"
>> at="xmldb:exist:///db/apps/sok-application/modules/index-persons.xqm"<xmldb:exist:///db/apps/sok-application/modules/index-persons.xqm>/>
>>             <analyzer
>> class="org.apache.lucene.analysis.standard.StandardAnalyzer"/>
>>             <analyzer
>> class="org.exist.indexing.lucene.analyzers.NoDiacriticsStandardAnalyzer"
>> id="nodiacritics"/>
>>             <text qname="adcache">
>>                 <field name="basicId"
>> expression="//person/@id/string()"/>
>>                 <field name="fullname"
>> expression="string(./basic/person/name)"/>
>>                 <field name="gndURI"
>> expression="string(./basic/person/identifier[@preferred eq 'YES'])"/>
>>                 <field name="gndID"
>> expression="substring-after(./basic/person/identifier[@preferred eq
>> 'YES']/string(), '/gnd/')"/>
>>                 <field name="status"
>> expression="./basic/person/@status/string()<mailto:./basic/person/@status/string()>"/>
>>                 <field name="articleID"
>> expression="ip:getArticleFromPersonCache(.)"<ip:getArticleFromPersonCache(.)>/>
>>                 <field name="articleRole"
>> expression="ip:getArticleRoleFromPersonCache(.)"<ip:getArticleRoleFromPersonCache(.)>/>
>>                 <field name="fulltext" expression="ip:getFullText(.)"<ip:getFullText(.)>/>
>>                </text>
>>         </lucene>
>>     </index>
>> </collection>
>>
>> Module Functions:
>>
>> module namespace ip                 =
>> "http://place.sok.org/xquery/index-persons<http://place.sok.org/xquery/index-persons>";
>>
>> declare namespace basic               =
>> "http://place.sok.org/xquery/basic<http://place.sok.org/xquery/basic>" ;
>>
>> declare namespace xs                = "http://www.w3.org/2001/XMLSchema<http://www.w3.org/2001/XMLSchema>";
>> declare namespace tei               = "http://www.tei-c.org/ns/1.0<http://www.tei-c.org/ns/1.0>";
>> declare namespace util              = "http://exist-db.org/xquery/util<http://exist-db.org/xquery/util>";
>>
>> declare function ip:getArticleFromPersonCache($adcache as element())
>> as xs:string* {
>>     let $parentCollectionPath as xs:anyURI? :=
>> ip:getParentCollection($adcache),
>>         $basicId as xs:string := $adcache/basic/person/@id/string(),
>>         $identifier as xs:string? :=
>> $adcache/basic/person/identifier[@preferred eq 'YES']/string(),
>>         $listId as xs:string? := collection(
>> $variables:jwo-lists-path )/tei:TEI//tei:person[
>> basic:basic-id-from-url(string(@sameAs)) eq $basicId]/@xml:id/string(),
>>         $foundInDocumentIds as xs:string* :=
>>             if ( matches($parentCollectionPath,'prepublish') )
>>             then
>>             (
>> collection($parentCollectionPath)/tei:TEI[./tei:teiHeader//tei:idno[1]/string()
>> ne
>> ''][matches(replace((normalize-space('||'||string-join(distinct-values(.//tei:persName[@ref]/@ref/string())
>> ! replace(.,'.*?#',''),
>> '||')||'||')||normalize-space('||'||string-join(distinct-values(.//tei:persName[@source]/@source/string())
>> ! replace(.,'.*?#',''),
>> '||')||'||')),'\|{4}',''),'\|{2}('||$basicId||'|'||$listId||'|'||$identifier||')\|{2}')]//tei:idno/string()
>>             )
>>             else
>>             (
>> collection($parentCollectionPath)/tei:TEI[./tei:teiHeader//tei:idno[1]/string()
>> ne
>> ''][matches(replace((normalize-space('||'||string-join(distinct-values(.//tei:persName[@ref][not(parent::editor)]/@ref/string())
>> ! replace(.,'.*?#',''),
>> '||')||'||')||normalize-space('||'||string-join(distinct-values(.//tei:persName[@source][not(parent::editor)]/@source/string())
>> ! replace(.,'.*?#',''),
>> '||')||'||')),'\|{4}',''),'\|{2}('||$basicId||'|'||$listId||'|'||$identifier||')\|{2}')]//tei:idno/string()
>>             )
>>     return
>>     (
>>         $foundInDocumentIds
>>     )
>> };
>>
>> declare
>> function ip:getAuthenticatedArticleCollection($collection-name as
>> xs:string) as item()* {
>>     if ($collection-name eq 'prepublish') then
>> xmldb:xcollection($variables:jwo-prepublish-path) else
>> xmldb:xcollection($variables:jwo-publish-path)
>> };
>>
>> declare function
>> ip:getPersNamesInCollectionFromCachedPerson($cached-person as
>> element(), $collection-name as xs:string) as element()* {
>>     let $basicId := $cached-person/basic/person/@id/string()
>>     let $identifier :=
>> $cached-person/basic/person/identifier[@preferred eq 'YES']/string()
>>     let $listId := collection( $variables:jwo-lists-path
>> )/tei:TEI/tei:text[1]/tei:body[1]/tei:listPerson[1]/tei:person[
>> basic:basic-id-from-url(string(@sameAs)) eq $basicId]/@xml:id/string()
>>     let $collection
>> := ip:getAuthenticatedArticleCollection($collection-name)
>>     return (
>>         $collection//tei:persName[
>>             string(@ref) eq $identifier
>>             or ip:getIdFromUri(string(@ref)) eq $listId
>>             or substring-after(string(@ref), '#') eq $basicId
>>             or substring-before(substring-after(string(@source),
>> 'persons/'), '.xml') eq $basicId]
>>     )
>> };
>>
>> declare function ip:getRoleFromPersName($persName as element(),
>> $collection-name as xs:string) as xs:string? {
>>     if ($persName/ancestor::*/local-name() = 'author')
>>     then ( 'author' )
>>     else if ($persName/ancestor::*/local-name() = 'editor')
>>     then (
>>         if ($collection-name eq 'prepublish') then ( 'editor' )
>>         (: Ignore editors in published case :)
>>         else ()
>>     )
>>     else ( 'annotated' )
>> };
>>
>> declare function ip:getArticleRoleFromPersonCache($cached-person as
>> element(), $collection-name as xs:string) as xs:string* {
>>     let $allPersNames :=
>>         if ($collection-name ne 'prepublish') then (
>> ip:getPersNamesInCollectionFromCachedPerson($cached-person,
>> $collection-name)[not(ancestor::*/local-name() = 'editor')]
>>         ) else (
>> ip:getPersNamesInCollectionFromCachedPerson($cached-person,
>> $collection-name)
>>         )
>>     return (
>>         for $articleGroup in $allPersNames
>>         let $articleID := $articleGroup/ancestor::tei:TEI//tei:idno[1]
>>         group by $articleID
>>         return (
>>             $articleID || '@@' || string-join(distinct-values(
>>                 for $persName in $articleGroup
>>                 let $role := ip:getRoleFromPersName($persName,
>> $collection-name)
>>                 order by $role
>>                 return $role
>>             ), ' ')
>>         )
>>     )
>> };
>>
>> declare function ip:getParentCollection($element as node()) as
>> xs:anyURI? {
>>     resolve-uri('../../', $element/base-uri())
>> };
>>
>> declare function ip:getIdFromUri($uri as xs:string) as xs:string {
>>     substring-after($uri, '#')
>> };
>>
>> declare function basic:basic-id-from-url($url as xs:string) as
>> xs:string? {
>>     substring-after(substring-before($url, '?dataset'),'persons/')
>> };
>>
>> declare function ip:getFullText($element) as xs:string {
>>     let $parentCollection as xs:anyURI? :=
>> ip:getParentCollection($element)
>>     return
>>     (
>>        normalize-space(string-join(
>>                     let $basicId as xs:string :=
>> $element/basic/person/@id/string(),
>>                         $identifier as xs:string* :=
>> $element/basic/person/identifier[@preferred eq 'YES']/string(),
>>                         $listId as item()* := collection(
>> $variables:lists-path
>> )/tei:TEI/tei:text[1]/tei:body[1]/tei:listPerson[1]/tei:person[
>> basic:basic-id-from-url(string(@sameAs)) eq $basicId]/@xml:id/string(),
>>                         $element-string as xs:string* :=
>> string($element),
>>                         $collections as item()* :=
>> collection($parentCollection)//tei:persName[string(@ref) eq
>> $identifier or ip:getIdFromUri(string(@ref)) eq $listId or
>> substring-after(string(@ref), '#') eq $basicId or
>> substring-before(substring-after(string(@source), 'persons/'),
>> '.xml') eq $basicId][1],
>>                         $element-cache-string as xs:string* :=
>> string-join(for $found-element in $collections where
>> count($found-element) > 0 return $found-element, ' ')
>>                     return
>>                     (
>>                         $element-string,$element-cache-string
>>                     ),' '))
>>     )
>> };
>>
>> Please help.
>>
>>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...<mailto:Exi...@li...>
> https://lists.sourceforge.net/lists/listinfo/exist-open<https://lists.sourceforge.net/lists/listinfo/exist-open>

--
Lars Scheideler
- wiss. technischer Mitarbeiter -
Althochdeutsches Wörterbuch & Digital Humanities

Sächsische Akademie der Wissenschaften zu Leipzig
Karl-Tauchnitz-Straße 1
04107 Leipzig

sch...@sa...<mailto:sch...@sa...>
www.saw-leipzig.de<http://www.saw-leipzig.de>



_______________________________________________
Exist-open mailing list
Exi...@li...<mailto:Exi...@li...>
https://lists.sourceforge.net/lists/listinfo/exist-open<https://lists.sourceforge.net/lists/listinfo/exist-open>


--
Cu stimă,
Claudius Teodorescu

--

Lars Scheideler

- wiss. technischer Mitarbeiter -

Althochdeutsches Wörterbuch & Digital Humanities



Sächsische Akademie der Wissenschaften zu Leipzig

Karl-Tauchnitz-Straße 1

04107 Leipzig



sch...@sa...<mailto:sch...@sa...>

www.saw-leipzig.de<http://www.saw-leipzig.de>

[Exist-open] eXist-db repair and reindex process taking a very long time

[Lizzi, V. <Vin...@ta...>]

Hello eXist-db community,

I've been monitoring an eXist-db database that is going through its automated recovery process, and am wondering if there is any way to get more information about its progress and how soon the process will finish.

This eXist-db has full text indexing and range indexes configured on several large collections. The EC2 server on which eXist-db is hosted had an outage. When eXist-db was restarted its automatic repair process began. That was about 3 weeks ago. Through Windows Resource Monitor I can see that the eXist-db process is reading from dob.dbx and structure.dbx and writing to structure.dbx and writing to files in the "lucene" and "range" folders, and the process is using about 6% of CPU consistently, and memory usage has increased gradually to over 100 GB. The last line in exist.log is still:

2024-08-13 17:02:59,710 [main] INFO  (NativeBroker.java [repair]:3692) - Reindexing database files ...

Is there any way to find out more what the process is doing, estimate when it will finish, or release any bottlenecks, without interrupting the process?

Thanks,
Vincent

______________________________________________
Vincent M. Lizzi
Head of Information Standards | Taylor & Francis Group
530 Walnut St., Suite 850, Philadelphia, PA 19106
E-Mail: vin...@ta...<mailto:vin...@ta...>
Web: www.tandfonline.com<http://www.tandfonline.com>

Taylor & Francis is a trading name of Informa UK Limited,
registered in England under no. 1072954

"Everything should be made as simple as possible, but not simpler."



Information Classification: General

Re: [Exist-open] Fwd: Cross reference Index - why is so slow

[Lars S. <sch...@sa...>]

Hey Claudius,

thank you for the insight.

But with the solution you presented, I ask myself why should I use 
eXistDb at all?
Then I could simply use an XML parser and read the fields directly from 
the files at file level or use another document-based DB that can do XML.
But I want to use the features that are in a complete system - I think 
the diversions with a custom search engine for combined fields is 
somehow good, but also thought around the corner.
Surely there must be a solution to the problem in eXistdb?

Best regards
Lars

Am 03.09.24 um 11:27 schrieb Claudius Teodorescu:
> Dear Lars,
>
> My name is Claudius Teodorescu, and I am currently working for the 
> Academy of Mainz.
>
> For the cases with publishing dictionaries (around 8, see 
> https://clre.solirom.ro), I have chosen the static website approach, 
> along with static indexes, and a static search engine that is 
> browser-based and is written in Rust and compiled to WebAssembly.
>
> One can see some test data from the DWDS dictionary (around 250,000 
> entries, and four indexes) at 
> https://claudius-teodorescu.gitlab.io/dwds-site/.
>
> Maybe this is of any help for you.
>
> I think that I do not have to mention that the indexing takes seconds, 
> as I have written the indexing engine in Rust language. :)
>
> Best regards,
> Claudius
>
> On Tue, 3 Sept 2024 at 11:30, Lars Scheidelerl 
> <sch...@sa...> wrote:
>
>     Dear Boris,
>
>     thank you very much for the answer.
>
>     The person data and the article data are in different collections.
>     For every person we have one file, for every article we have one file.
>
>     We don't have a fixed order of when data is imported or written in,
>     as I understand it, what you are suggesting would make it
>     necessary for
>     all the person or item data to
>     always be available before the index of the other collection, so
>     that it
>     can be accessed in the index of the other.
>
>     Maybe we have to, like you pointed out, tests the module functions
>     for
>     better performance.
>     But in the past however, after several different approaches that
>     we have
>     already tried, we have realized that querying the data,
>     no matter how good it is in eXide, for example, is significantly
>     slower
>     when it is used for the index.
>     Could it be that the structural index is not used when re-indexing?
>     Our assumption was that the data is iterated over differently or that
>     new blobs are written over the structure and content in addition to
>     indexing.
>     In other words, that it validates, saves and indexes the data at the
>     same time as re-indexing.
>
>     Other approaches include the use of cache or a helper file, where the
>     fields are composed beforehand and then indexed accordingly,
>     but this also takes a long time and unfortunately also blocks
>     working on
>     the files.
>     So if we write a helper file with all fields in eXide, the whole
>     process
>     takes about 90s, if we work with the fields as we would build the
>     index,
>     i.e. with xml:id in the helper file, ~110s.
>     Re-Index 2-4 hours. Not really understandable.
>
>     We are now primarily trying to improve this using the xml:id/id()
>     function, but not much hope to improve the re-index on a
>     production scale.
>     But if all data is re-indexed, i.e. the xml:id fields are not
>     available,
>     it is in vain, too.
>
>     Would love to learn more about this topic and continue to share
>     experiences.
>
>     Lars
>
>     Am 03.09.24 um 00:49 schrieb Boris Lehečka:
>     > Dear Lars,
>     >
>     >     I have similar issues with indexing dictionaries: my indexing
>     > procedure ask for data from the taxonomy (like expansions of
>     > abbreviations) and sometimes indexing a dictionary with about
>     36,000
>     > entries took a whole day.
>     >
>     >     I don't remember who (Juri Leino, I guess), pointed out to
>     me that
>     > the index is saved only after the whole document is parsed. After
>     > moving each dictionary entry to a separate file, indexing took much
>     > less time (several hours).
>     >
>     >     However, this does not seem to be the cause of your problem.
>     >
>     >     In my opinion, your indexing code (in the module) is very
>     > complicated, sometimes it can be much simpler, for example without
>     > explicit conversion to string (like in tei:persName[string(@ref) eq
>     > $identifier ...]), or normalizing spaces in ip:getFullText
>     (full-tex t
>     > search usually uses only the parts between spaces).
>     >
>     >     My suggestion is following: first, create an index for
>     persons in
>     > a separate collection (with separate collection.xconf): compute
>     fields
>     > with values you will query or want to return when you index the
>     > articles (in different collection). And second, use Lucene and
>     > fulltext search in your "index-persons" module to find data in the
>     > index from the first phase.
>     >
>     >   This is just an idea, not tested, I hope someone else is much
>     more
>     > experienced in the magic of indexing.
>     >
>     >    Best,
>     >
>     >     Boris Lehečka
>     >
>     > Dne 02.09.2024 v 16:57 Lars Scheidelerl napsal(a):
>     >> Hey,
>     >>
>     >> we assume that we are not using the index in our project as
>     intended.
>     >> Because when we try to build the index we have created, it takes a
>     >> very long time.
>     >>
>     >> We have two collections, one with a data stack of 687 in which
>     data
>     >> is stored, and one with 400 xml where articles are are stored.
>     >>
>     >> For the personal article we want certain information from the
>     >> articles and vice versa.
>     >>
>     >> Person XML:
>     >>
>     >> <person xml:id="i0c9ab7e2-2e21-39ff-aea8-c56ad4702a7f"
>     status="safe"
>     >> modified="2024-07-30T13:26:09.154+02:00">
>     >>             <name>Marcanton Zimara</name>
>     >>             <identifier
>     >> preferred="YES">https://d-nb.info/gnd/120156784</identifier>
>     >>             <alternateName>Marcusantonius Zimara</alternateName>
>     >>             <alternateName>Marcus Anthonius Zimara</alternateName>
>     >>             <alternateName>Antonius Zimara</alternateName>
>     >>             <alternateName>M. Antonius Zimarra</alternateName>
>     >>             <alternateName>Marc Antoine Zimara</alternateName>
>     >>             <alternateName>M. Anto. Zimare</alternateName>
>     >>             <alternateName>Marco A. Zimara</alternateName>
>     >>             <alternateName>Marcus A. Zimara</alternateName>
>     >>             <alternateName>Marc Ant. Zimara</alternateName>
>     >>             <alternateName>Marcantonio Zimara</alternateName>
>     >>             <alternateName>Marcus Antonius Zimara</alternateName>
>     >>             <alternateName>Marcus Antonius Zimarra</alternateName>
>     >>             <alternateName>Marcianto Zimare</alternateName>
>     >>             <alternateName>Marco Antonio Zimarra</alternateName>
>     >>             <alternateName>Marco Antonio Zimare</alternateName>
>     >>             <birthDate>1460</birthDate>
>     >>             <deathDate>1532</deathDate>
>     >> <description>JWO</description>
>     >>             <sortableName>Zimara, Marcanton </sortableName>
>     >>         </person>
>     >>
>     >> Articel XML:
>     >>
>     >> <?xml version="1.0" encoding="UTF-8"?>
>     >> <TEI xmlns="http://www.tei-c.org/ns/1.0">
>     >>     <teiHeader>
>     >>         <fileDesc>
>     >>             <titleStmt>
>     >>                 <title>a nihilo nihil fit</title>
>     >>                 <author>
>     >>                     <persName
>     >> ref="/db/projects/jwo/data/lists/personenListe.xml#BS_d1e509"
>     >> xml:id="author_BS_d1e509">
>     >> <forename>Marcanton</forename>
>     >> <surname>Zimara</surname>
>     >>                     </persName>
>     >>                 </author>
>     >>             </titleStmt>
>     >>             <sourceDesc>
>     >>                 <p xml:id="p_sourceDesc_igw_tvr_pzb">born
>     digital</p>
>     >>             </sourceDesc>
>     >>         </fileDesc>
>     >>     </teiHeader>
>     >>     <text xml:lang="de-DE" type="main">
>     >>         <body>
>     >>             <div1 xml:id="div1_d1e23_2">
>     >>                 <p xml:id="p_d1e27_1" n="1"> Lorem ipsum dolor sit
>     >> amet, consectetur adipiscing elit, sed do eiusmod tempor
>     incididunt
>     >> ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis
>     >> nostrud exercitation <persName xml:id="persName_sa123"
>     >> ref="https://d-nb.info/gnd/120156784"
>     >> rend="smallcaps">Zimara</persName> ullamco laboris nisi ut
>     aliquip ex
>     >> ea commodo consequat. Duis aute irure dolor in reprehenderit in
>     >> voluptate velit esse cillum dolore eu <persName
>     >> xml:id="persName_s123"
>     >> ref="https://d-nb.info/gnd/120156784">Zimara</persName> fugiat
>     nulla
>     >> pariatur. Excepteur sint occaecat cupidatat non proident, sunt in
>     >> culpa qui officia deserunt mollit anim id est laborum. </p>
>     >>             </div1>
>     >>         </body>
>     >>     </text>
>     >> </TEI>
>     >>
>     >> Collection.xconf:
>     >>
>     >> <collection xmlns="http://exist-db.org/collection-config/1.0">
>     >>     <index
>     xmlns:gndo="https://d-nb.info/standards/elementset/gnd#"
>     >> xmlns:owl="http://www.w3.org/2002/07/owl#"
>     >> xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
>     >> xmlns:xs="http://www.w3.org/2001/XMLSchema">
>     >>         <lucene>
>     >>             <module
>     uri="http://place.sok.org/xquery/index-persons"
>     >> prefix="ip"
>     >>
>     at="xmldb:exist:///db/apps/sok-application/modules/index-persons.xqm"/>
>     >>             <analyzer
>     >> class="org.apache.lucene.analysis.standard.StandardAnalyzer"/>
>     >>             <analyzer
>     >>
>     class="org.exist.indexing.lucene.analyzers.NoDiacriticsStandardAnalyzer"
>
>     >> id="nodiacritics"/>
>     >>             <text qname="adcache">
>     >>                 <field name="basicId"
>     >> expression="//person/@id/string()"/>
>     >>                 <field name="fullname"
>     >> expression="string(./basic/person/name)"/>
>     >>                 <field name="gndURI"
>     >> expression="string(./basic/person/identifier[@preferred eq
>     'YES'])"/>
>     >>                 <field name="gndID"
>     >>
>     expression="substring-after(./basic/person/identifier[@preferred eq
>     >> 'YES']/string(), '/gnd/')"/>
>     >>                 <field name="status"
>     >> expression="./basic/person/@status/string()"/>
>     >>                 <field name="articleID"
>     >> expression="ip:getArticleFromPersonCache(.)"/>
>     >>                 <field name="articleRole"
>     >> expression="ip:getArticleRoleFromPersonCache(.)"/>
>     >>                 <field name="fulltext"
>     expression="ip:getFullText(.)"/>
>     >>                </text>
>     >>         </lucene>
>     >>     </index>
>     >> </collection>
>     >>
>     >> Module Functions:
>     >>
>     >> module namespace ip                 =
>     >> "http://place.sok.org/xquery/index-persons";
>     >>
>     >> declare namespace basic               =
>     >> "http://place.sok.org/xquery/basic" ;
>     >>
>     >> declare namespace xs                =
>     "http://www.w3.org/2001/XMLSchema";
>     >> declare namespace tei               =
>     "http://www.tei-c.org/ns/1.0";
>     >> declare namespace util              =
>     "http://exist-db.org/xquery/util";
>     >>
>     >> declare function ip:getArticleFromPersonCache($adcache as
>     element())
>     >> as xs:string* {
>     >>     let $parentCollectionPath as xs:anyURI? :=
>     >> ip:getParentCollection($adcache),
>     >>         $basicId as xs:string :=
>     $adcache/basic/person/@id/string(),
>     >>         $identifier as xs:string? :=
>     >> $adcache/basic/person/identifier[@preferred eq 'YES']/string(),
>     >>         $listId as xs:string? := collection(
>     >> $variables:jwo-lists-path )/tei:TEI//tei:person[
>     >> basic:basic-id-from-url(string(@sameAs)) eq
>     $basicId]/@xml:id/string(),
>     >>         $foundInDocumentIds as xs:string* :=
>     >>             if ( matches($parentCollectionPath,'prepublish') )
>     >>             then
>     >>             (
>     >>
>     collection($parentCollectionPath)/tei:TEI[./tei:teiHeader//tei:idno[1]/string()
>
>     >> ne
>     >>
>     ''][matches(replace((normalize-space('||'||string-join(distinct-values(.//tei:persName[@ref]/@ref/string())
>
>     >> ! replace(.,'.*?#',''),
>     >>
>     '||')||'||')||normalize-space('||'||string-join(distinct-values(.//tei:persName[@source]/@source/string())
>
>     >> ! replace(.,'.*?#',''),
>     >>
>     '||')||'||')),'\|{4}',''),'\|{2}('||$basicId||'|'||$listId||'|'||$identifier||')\|{2}')]//tei:idno/string()
>     >>             )
>     >>             else
>     >>             (
>     >>
>     collection($parentCollectionPath)/tei:TEI[./tei:teiHeader//tei:idno[1]/string()
>
>     >> ne
>     >>
>     ''][matches(replace((normalize-space('||'||string-join(distinct-values(.//tei:persName[@ref][not(parent::editor)]/@ref/string())
>
>     >> ! replace(.,'.*?#',''),
>     >>
>     '||')||'||')||normalize-space('||'||string-join(distinct-values(.//tei:persName[@source][not(parent::editor)]/@source/string())
>
>     >> ! replace(.,'.*?#',''),
>     >>
>     '||')||'||')),'\|{4}',''),'\|{2}('||$basicId||'|'||$listId||'|'||$identifier||')\|{2}')]//tei:idno/string()
>     >>             )
>     >>     return
>     >>     (
>     >>         $foundInDocumentIds
>     >>     )
>     >> };
>     >>
>     >> declare
>     >> function ip:getAuthenticatedArticleCollection($collection-name as
>     >> xs:string) as item()* {
>     >>     if ($collection-name eq 'prepublish') then
>     >> xmldb:xcollection($variables:jwo-prepublish-path) else
>     >> xmldb:xcollection($variables:jwo-publish-path)
>     >> };
>     >>
>     >> declare function
>     >> ip:getPersNamesInCollectionFromCachedPerson($cached-person as
>     >> element(), $collection-name as xs:string) as element()* {
>     >>     let $basicId := $cached-person/basic/person/@id/string()
>     >>     let $identifier :=
>     >> $cached-person/basic/person/identifier[@preferred eq
>     'YES']/string()
>     >>     let $listId := collection( $variables:jwo-lists-path
>     >> )/tei:TEI/tei:text[1]/tei:body[1]/tei:listPerson[1]/tei:person[
>     >> basic:basic-id-from-url(string(@sameAs)) eq
>     $basicId]/@xml:id/string()
>     >>     let $collection
>     >> := ip:getAuthenticatedArticleCollection($collection-name)
>     >>     return (
>     >>         $collection//tei:persName[
>     >>             string(@ref) eq $identifier
>     >>             or ip:getIdFromUri(string(@ref)) eq $listId
>     >>             or substring-after(string(@ref), '#') eq $basicId
>     >>             or substring-before(substring-after(string(@source),
>     >> 'persons/'), '.xml') eq $basicId]
>     >>     )
>     >> };
>     >>
>     >> declare function ip:getRoleFromPersName($persName as element(),
>     >> $collection-name as xs:string) as xs:string? {
>     >>     if ($persName/ancestor::*/local-name() = 'author')
>     >>     then ( 'author' )
>     >>     else if ($persName/ancestor::*/local-name() = 'editor')
>     >>     then (
>     >>         if ($collection-name eq 'prepublish') then ( 'editor' )
>     >>         (: Ignore editors in published case :)
>     >>         else ()
>     >>     )
>     >>     else ( 'annotated' )
>     >> };
>     >>
>     >> declare function
>     ip:getArticleRoleFromPersonCache($cached-person as
>     >> element(), $collection-name as xs:string) as xs:string* {
>     >>     let $allPersNames :=
>     >>         if ($collection-name ne 'prepublish') then (
>     >> ip:getPersNamesInCollectionFromCachedPerson($cached-person,
>     >> $collection-name)[not(ancestor::*/local-name() = 'editor')]
>     >>         ) else (
>     >> ip:getPersNamesInCollectionFromCachedPerson($cached-person,
>     >> $collection-name)
>     >>         )
>     >>     return (
>     >>         for $articleGroup in $allPersNames
>     >>         let $articleID :=
>     $articleGroup/ancestor::tei:TEI//tei:idno[1]
>     >>         group by $articleID
>     >>         return (
>     >>             $articleID || '@@' || string-join(distinct-values(
>     >>                 for $persName in $articleGroup
>     >>                 let $role := ip:getRoleFromPersName($persName,
>     >> $collection-name)
>     >>                 order by $role
>     >>                 return $role
>     >>             ), ' ')
>     >>         )
>     >>     )
>     >> };
>     >>
>     >> declare function ip:getParentCollection($element as node()) as
>     >> xs:anyURI? {
>     >>     resolve-uri('../../', $element/base-uri())
>     >> };
>     >>
>     >> declare function ip:getIdFromUri($uri as xs:string) as xs:string {
>     >>     substring-after($uri, '#')
>     >> };
>     >>
>     >> declare function basic:basic-id-from-url($url as xs:string) as
>     >> xs:string? {
>     >>     substring-after(substring-before($url, '?dataset'),'persons/')
>     >> };
>     >>
>     >> declare function ip:getFullText($element) as xs:string {
>     >>     let $parentCollection as xs:anyURI? :=
>     >> ip:getParentCollection($element)
>     >>     return
>     >>     (
>     >>        normalize-space(string-join(
>     >>                     let $basicId as xs:string :=
>     >> $element/basic/person/@id/string(),
>     >>                         $identifier as xs:string* :=
>     >> $element/basic/person/identifier[@preferred eq 'YES']/string(),
>     >>                         $listId as item()* := collection(
>     >> $variables:lists-path
>     >> )/tei:TEI/tei:text[1]/tei:body[1]/tei:listPerson[1]/tei:person[
>     >> basic:basic-id-from-url(string(@sameAs)) eq
>     $basicId]/@xml:id/string(),
>     >>                         $element-string as xs:string* :=
>     >> string($element),
>     >>                         $collections as item()* :=
>     >> collection($parentCollection)//tei:persName[string(@ref) eq
>     >> $identifier or ip:getIdFromUri(string(@ref)) eq $listId or
>     >> substring-after(string(@ref), '#') eq $basicId or
>     >> substring-before(substring-after(string(@source), 'persons/'),
>     >> '.xml') eq $basicId][1],
>     >>                         $element-cache-string as xs:string* :=
>     >> string-join(for $found-element in $collections where
>     >> count($found-element) > 0 return $found-element, ' ')
>     >>                     return
>     >>                     (
>     >> $element-string,$element-cache-string
>     >>                     ),' '))
>     >>     )
>     >> };
>     >>
>     >> Please help.
>     >>
>     >>
>     >
>     >
>     > _______________________________________________
>     > Exist-open mailing list
>     > Exi...@li...
>     > https://lists.sourceforge.net/lists/listinfo/exist-open
>
>     -- 
>     Lars Scheideler
>     - wiss. technischer Mitarbeiter -
>     Althochdeutsches Wörterbuch & Digital Humanities
>
>     Sächsische Akademie der Wissenschaften zu Leipzig
>     Karl-Tauchnitz-Straße 1
>     04107 Leipzig
>
>     sch...@sa...
>     www.saw-leipzig.de <http://www.saw-leipzig.de>
>
>
>
>     _______________________________________________
>     Exist-open mailing list
>     Exi...@li...
>     https://lists.sourceforge.net/lists/listinfo/exist-open
>
>
>
> -- 
> Cu stimă,
> Claudius Teodorescu

-- 
Lars Scheideler
- wiss. technischer Mitarbeiter -
Althochdeutsches Wörterbuch & Digital Humanities

Sächsische Akademie der Wissenschaften zu Leipzig
Karl-Tauchnitz-Straße 1
04107 Leipzig

sch...@sa...
www.saw-leipzig.de

[Exist-open] Fwd: Cross reference Index - why is so slow

[Lars S. <sch...@sa...>]

Dear Boris,

thank you very much for the answer.

The person data and the article data are in different collections.
For every person we have one file, for every article we have one file.

We don't have a fixed order of when data is imported or written in,
as I understand it, what you are suggesting would make it necessary for 
all the person or item data to
always be available before the index of the other collection, so that it 
can be accessed in the index of the other.

Maybe we have to, like you pointed out, tests the module functions for 
better performance.
But in the past however, after several different approaches that we have 
already tried, we have realized that querying the data,
no matter how good it is in eXide, for example, is significantly slower 
when it is used for the index.
Could it be that the structural index is not used when re-indexing?
Our assumption was that the data is iterated over differently or that 
new blobs are written over the structure and content in addition to 
indexing.
In other words, that it validates, saves and indexes the data at the 
same time as re-indexing.

Other approaches include the use of cache or a helper file, where the 
fields are composed beforehand and then indexed accordingly,
but this also takes a long time and unfortunately also blocks working on 
the files.
So if we write a helper file with all fields in eXide, the whole process 
takes about 90s, if we work with the fields as we would build the index, 
i.e. with xml:id in the helper file, ~110s.
Re-Index 2-4 hours. Not really understandable.

We are now primarily trying to improve this using the xml:id/id() 
function, but not much hope to improve the re-index on a production scale.
But if all data is re-indexed, i.e. the xml:id fields are not available, 
it is in vain, too.

Would love to learn more about this topic and continue to share experiences.

Lars

Am 03.09.24 um 00:49 schrieb Boris Lehečka:
> Dear Lars,
>
>     I have similar issues with indexing dictionaries: my indexing 
> procedure ask for data from the taxonomy (like expansions of 
> abbreviations) and sometimes indexing a dictionary with about 36,000 
> entries took a whole day.
>
>     I don't remember who (Juri Leino, I guess), pointed out to me that 
> the index is saved only after the whole document is parsed. After 
> moving each dictionary entry to a separate file, indexing took much 
> less time (several hours).
>
>     However, this does not seem to be the cause of your problem.
>
>     In my opinion, your indexing code (in the module) is very 
> complicated, sometimes it can be much simpler, for example without 
> explicit conversion to string (like in tei:persName[string(@ref) eq 
> $identifier ...]), or normalizing spaces in ip:getFullText (full-tex t 
> search usually uses only the parts between spaces).
>
>     My suggestion is following: first, create an index for persons in 
> a separate collection (with separate collection.xconf): compute fields 
> with values you will query or want to return when you index the 
> articles (in different collection). And second, use Lucene and 
> fulltext search in your "index-persons" module to find data in the 
> index from the first phase.
>
>   This is just an idea, not tested, I hope someone else is much more 
> experienced in the magic of indexing.
>
>    Best,
>
>     Boris Lehečka
>
> Dne 02.09.2024 v 16:57 Lars Scheidelerl napsal(a):
>> Hey,
>>
>> we assume that we are not using the index in our project as intended.
>> Because when we try to build the index we have created, it takes a 
>> very long time.
>>
>> We have two collections, one with a data stack of 687 in which data 
>> is stored, and one with 400 xml where articles are are stored.
>>
>> For the personal article we want certain information from the 
>> articles and vice versa.
>>
>> Person XML:
>>
>> <person xml:id="i0c9ab7e2-2e21-39ff-aea8-c56ad4702a7f" status="safe" 
>> modified="2024-07-30T13:26:09.154+02:00">
>>             <name>Marcanton Zimara</name>
>>             <identifier 
>> preferred="YES">https://d-nb.info/gnd/120156784</identifier>
>>             <alternateName>Marcusantonius Zimara</alternateName>
>>             <alternateName>Marcus Anthonius Zimara</alternateName>
>>             <alternateName>Antonius Zimara</alternateName>
>>             <alternateName>M. Antonius Zimarra</alternateName>
>>             <alternateName>Marc Antoine Zimara</alternateName>
>>             <alternateName>M. Anto. Zimare</alternateName>
>>             <alternateName>Marco A. Zimara</alternateName>
>>             <alternateName>Marcus A. Zimara</alternateName>
>>             <alternateName>Marc Ant. Zimara</alternateName>
>>             <alternateName>Marcantonio Zimara</alternateName>
>>             <alternateName>Marcus Antonius Zimara</alternateName>
>>             <alternateName>Marcus Antonius Zimarra</alternateName>
>>             <alternateName>Marcianto Zimare</alternateName>
>>             <alternateName>Marco Antonio Zimarra</alternateName>
>>             <alternateName>Marco Antonio Zimare</alternateName>
>>             <birthDate>1460</birthDate>
>>             <deathDate>1532</deathDate>
>>             <description>JWO</description>
>>             <sortableName>Zimara, Marcanton </sortableName>
>>         </person>
>>
>> Articel XML:
>>
>> <?xml version="1.0" encoding="UTF-8"?>
>> <TEI xmlns="http://www.tei-c.org/ns/1.0">
>>     <teiHeader>
>>         <fileDesc>
>>             <titleStmt>
>>                 <title>a nihilo nihil fit</title>
>>                 <author>
>>                     <persName 
>> ref="/db/projects/jwo/data/lists/personenListe.xml#BS_d1e509" 
>> xml:id="author_BS_d1e509">
>> <forename>Marcanton</forename>
>>                         <surname>Zimara</surname>
>>                     </persName>
>>                 </author>
>>             </titleStmt>
>>             <sourceDesc>
>>                 <p xml:id="p_sourceDesc_igw_tvr_pzb">born digital</p>
>>             </sourceDesc>
>>         </fileDesc>
>>     </teiHeader>
>>     <text xml:lang="de-DE" type="main">
>>         <body>
>>             <div1 xml:id="div1_d1e23_2">
>>                 <p xml:id="p_d1e27_1" n="1"> Lorem ipsum dolor sit 
>> amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt 
>> ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis 
>> nostrud exercitation <persName xml:id="persName_sa123" 
>> ref="https://d-nb.info/gnd/120156784" 
>> rend="smallcaps">Zimara</persName> ullamco laboris nisi ut aliquip ex 
>> ea commodo consequat. Duis aute irure dolor in reprehenderit in 
>> voluptate velit esse cillum dolore eu <persName 
>> xml:id="persName_s123" 
>> ref="https://d-nb.info/gnd/120156784">Zimara</persName> fugiat nulla 
>> pariatur. Excepteur sint occaecat cupidatat non proident, sunt in 
>> culpa qui officia deserunt mollit anim id est laborum. </p>
>>             </div1>
>>         </body>
>>     </text>
>> </TEI>
>>
>> Collection.xconf:
>>
>> <collection xmlns="http://exist-db.org/collection-config/1.0">
>>     <index xmlns:gndo="https://d-nb.info/standards/elementset/gnd#" 
>> xmlns:owl="http://www.w3.org/2002/07/owl#" 
>> xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" 
>> xmlns:xs="http://www.w3.org/2001/XMLSchema">
>>         <lucene>
>>             <module uri="http://place.sok.org/xquery/index-persons" 
>> prefix="ip" 
>> at="xmldb:exist:///db/apps/sok-application/modules/index-persons.xqm"/>
>>             <analyzer 
>> class="org.apache.lucene.analysis.standard.StandardAnalyzer"/>
>>             <analyzer 
>> class="org.exist.indexing.lucene.analyzers.NoDiacriticsStandardAnalyzer" 
>> id="nodiacritics"/>
>>             <text qname="adcache">
>>                 <field name="basicId" 
>> expression="//person/@id/string()"/>
>>                 <field name="fullname" 
>> expression="string(./basic/person/name)"/>
>>                 <field name="gndURI" 
>> expression="string(./basic/person/identifier[@preferred eq 'YES'])"/>
>>                 <field name="gndID" 
>> expression="substring-after(./basic/person/identifier[@preferred eq 
>> 'YES']/string(), '/gnd/')"/>
>>                 <field name="status" 
>> expression="./basic/person/@status/string()"/>
>>                 <field name="articleID" 
>> expression="ip:getArticleFromPersonCache(.)"/>
>>                 <field name="articleRole" 
>> expression="ip:getArticleRoleFromPersonCache(.)"/>
>>                 <field name="fulltext" expression="ip:getFullText(.)"/>
>>                </text>
>>         </lucene>
>>     </index>
>> </collection>
>>
>> Module Functions:
>>
>> module namespace ip                 = 
>> "http://place.sok.org/xquery/index-persons";
>>
>> declare namespace basic               = 
>> "http://place.sok.org/xquery/basic" ;
>>
>> declare namespace xs                = "http://www.w3.org/2001/XMLSchema";
>> declare namespace tei               = "http://www.tei-c.org/ns/1.0";
>> declare namespace util              = "http://exist-db.org/xquery/util";
>>
>> declare function ip:getArticleFromPersonCache($adcache as element()) 
>> as xs:string* {
>>     let $parentCollectionPath as xs:anyURI? := 
>> ip:getParentCollection($adcache),
>>         $basicId as xs:string := $adcache/basic/person/@id/string(),
>>         $identifier as xs:string? := 
>> $adcache/basic/person/identifier[@preferred eq 'YES']/string(),
>>         $listId as xs:string? := collection( 
>> $variables:jwo-lists-path )/tei:TEI//tei:person[ 
>> basic:basic-id-from-url(string(@sameAs)) eq $basicId]/@xml:id/string(),
>>         $foundInDocumentIds as xs:string* :=
>>             if ( matches($parentCollectionPath,'prepublish') )
>>             then
>>             (
>> collection($parentCollectionPath)/tei:TEI[./tei:teiHeader//tei:idno[1]/string() 
>> ne 
>> ''][matches(replace((normalize-space('||'||string-join(distinct-values(.//tei:persName[@ref]/@ref/string()) 
>> ! replace(.,'.*?#',''), 
>> '||')||'||')||normalize-space('||'||string-join(distinct-values(.//tei:persName[@source]/@source/string()) 
>> ! replace(.,'.*?#',''), 
>> '||')||'||')),'\|{4}',''),'\|{2}('||$basicId||'|'||$listId||'|'||$identifier||')\|{2}')]//tei:idno/string()
>>             )
>>             else
>>             (
>> collection($parentCollectionPath)/tei:TEI[./tei:teiHeader//tei:idno[1]/string() 
>> ne 
>> ''][matches(replace((normalize-space('||'||string-join(distinct-values(.//tei:persName[@ref][not(parent::editor)]/@ref/string()) 
>> ! replace(.,'.*?#',''), 
>> '||')||'||')||normalize-space('||'||string-join(distinct-values(.//tei:persName[@source][not(parent::editor)]/@source/string()) 
>> ! replace(.,'.*?#',''), 
>> '||')||'||')),'\|{4}',''),'\|{2}('||$basicId||'|'||$listId||'|'||$identifier||')\|{2}')]//tei:idno/string()
>>             )
>>     return
>>     (
>>         $foundInDocumentIds
>>     )
>> };
>>
>> declare 
>> function ip:getAuthenticatedArticleCollection($collection-name as 
>> xs:string) as item()* {
>>     if ($collection-name eq 'prepublish') then 
>> xmldb:xcollection($variables:jwo-prepublish-path) else 
>> xmldb:xcollection($variables:jwo-publish-path)
>> };
>>
>> declare function 
>> ip:getPersNamesInCollectionFromCachedPerson($cached-person as 
>> element(), $collection-name as xs:string) as element()* {
>>     let $basicId := $cached-person/basic/person/@id/string()
>>     let $identifier := 
>> $cached-person/basic/person/identifier[@preferred eq 'YES']/string()
>>     let $listId := collection( $variables:jwo-lists-path 
>> )/tei:TEI/tei:text[1]/tei:body[1]/tei:listPerson[1]/tei:person[ 
>> basic:basic-id-from-url(string(@sameAs)) eq $basicId]/@xml:id/string()
>>     let $collection 
>> := ip:getAuthenticatedArticleCollection($collection-name)
>>     return (
>>         $collection//tei:persName[
>>             string(@ref) eq $identifier
>>             or ip:getIdFromUri(string(@ref)) eq $listId
>>             or substring-after(string(@ref), '#') eq $basicId
>>             or substring-before(substring-after(string(@source), 
>> 'persons/'), '.xml') eq $basicId]
>>     )
>> };
>>
>> declare function ip:getRoleFromPersName($persName as element(), 
>> $collection-name as xs:string) as xs:string? {
>>     if ($persName/ancestor::*/local-name() = 'author')
>>     then ( 'author' )
>>     else if ($persName/ancestor::*/local-name() = 'editor')
>>     then (
>>         if ($collection-name eq 'prepublish') then ( 'editor' )
>>         (: Ignore editors in published case :)
>>         else ()
>>     )
>>     else ( 'annotated' )
>> };
>>
>> declare function ip:getArticleRoleFromPersonCache($cached-person as 
>> element(), $collection-name as xs:string) as xs:string* {
>>     let $allPersNames :=
>>         if ($collection-name ne 'prepublish') then (
>> ip:getPersNamesInCollectionFromCachedPerson($cached-person, 
>> $collection-name)[not(ancestor::*/local-name() = 'editor')]
>>         ) else (
>> ip:getPersNamesInCollectionFromCachedPerson($cached-person, 
>> $collection-name)
>>         )
>>     return (
>>         for $articleGroup in $allPersNames
>>         let $articleID := $articleGroup/ancestor::tei:TEI//tei:idno[1]
>>         group by $articleID
>>         return (
>>             $articleID || '@@' || string-join(distinct-values(
>>                 for $persName in $articleGroup
>>                 let $role := ip:getRoleFromPersName($persName, 
>> $collection-name)
>>                 order by $role
>>                 return $role
>>             ), ' ')
>>         )
>>     )
>> };
>>
>> declare function ip:getParentCollection($element as node()) as 
>> xs:anyURI? {
>>     resolve-uri('../../', $element/base-uri())
>> };
>>
>> declare function ip:getIdFromUri($uri as xs:string) as xs:string {
>>     substring-after($uri, '#')
>> };
>>
>> declare function basic:basic-id-from-url($url as xs:string) as 
>> xs:string? {
>>     substring-after(substring-before($url, '?dataset'),'persons/')
>> };
>>
>> declare function ip:getFullText($element) as xs:string {
>>     let $parentCollection as xs:anyURI? := 
>> ip:getParentCollection($element)
>>     return
>>     (
>>        normalize-space(string-join(
>>                     let $basicId as xs:string := 
>> $element/basic/person/@id/string(),
>>                         $identifier as xs:string* := 
>> $element/basic/person/identifier[@preferred eq 'YES']/string(),
>>                         $listId as item()* := collection( 
>> $variables:lists-path 
>> )/tei:TEI/tei:text[1]/tei:body[1]/tei:listPerson[1]/tei:person[ 
>> basic:basic-id-from-url(string(@sameAs)) eq $basicId]/@xml:id/string(),
>>                         $element-string as xs:string* := 
>> string($element),
>>                         $collections as item()* := 
>> collection($parentCollection)//tei:persName[string(@ref) eq 
>> $identifier or ip:getIdFromUri(string(@ref)) eq $listId or 
>> substring-after(string(@ref), '#') eq $basicId or 
>> substring-before(substring-after(string(@source), 'persons/'), 
>> '.xml') eq $basicId][1],
>>                         $element-cache-string as xs:string* := 
>> string-join(for $found-element in $collections where 
>> count($found-element) > 0 return $found-element, ' ')
>>                     return
>>                     (
>>                         $element-string,$element-cache-string
>>                     ),' '))
>>     )
>> };
>>
>> Please help.
>>
>>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

-- 
Lars Scheideler
- wiss. technischer Mitarbeiter -
Althochdeutsches Wörterbuch & Digital Humanities

Sächsische Akademie der Wissenschaften zu Leipzig
Karl-Tauchnitz-Straße 1
04107 Leipzig

sch...@sa...
www.saw-leipzig.de

Re: [Exist-open] Cross reference Index - why is so slow

[Boris L. <bo...@da...>]

     Dear Lars,

     I have similar issues with indexing dictionaries: my indexing 
procedure ask for data from the taxonomy (like expansions of 
abbreviations) and sometimes indexing a dictionary with about 36,000 
entries took a whole day.

     I don't remember who (Juri Leino, I guess), pointed out to me that 
the index is saved only after the whole document is parsed. After moving 
each dictionary entry to a separate file, indexing took much less time 
(several hours).

     However, this does not seem to be the cause of your problem.

     In my opinion, your indexing code (in the module) is very 
complicated, sometimes it can be much simpler, for example without 
explicit conversion to string (like in tei:persName[string(@ref) eq 
$identifier ...]), or normalizing spaces in ip:getFullText (full-tex t 
search usually uses only the parts between spaces).

     My suggestion is following: first, create an index for persons in a 
separate collection (with separate collection.xconf): compute fields 
with values you will query or want to return when you index the articles 
(in different collection). And second, use Lucene and fulltext search in 
your "index-persons" module to find data in the index from the first phase.

   This is just an idea, not tested, I hope someone else is much more 
experienced in the magic of indexing.

    Best,

     Boris Lehečka

Dne 02.09.2024 v 16:57 Lars Scheidelerl napsal(a):
> Hey,
>
> we assume that we are not using the index in our project as intended.
> Because when we try to build the index we have created, it takes a 
> very long time.
>
> We have two collections, one with a data stack of 687 in which data is 
> stored, and one with 400 xml where articles are are stored.
>
> For the personal article we want certain information from the articles 
> and vice versa.
>
> Person XML:
>
> <person xml:id="i0c9ab7e2-2e21-39ff-aea8-c56ad4702a7f" status="safe" 
> modified="2024-07-30T13:26:09.154+02:00">
>             <name>Marcanton Zimara</name>
>             <identifier 
> preferred="YES">https://d-nb.info/gnd/120156784</identifier>
>             <alternateName>Marcusantonius Zimara</alternateName>
>             <alternateName>Marcus Anthonius Zimara</alternateName>
>             <alternateName>Antonius Zimara</alternateName>
>             <alternateName>M. Antonius Zimarra</alternateName>
>             <alternateName>Marc Antoine Zimara</alternateName>
>             <alternateName>M. Anto. Zimare</alternateName>
>             <alternateName>Marco A. Zimara</alternateName>
>             <alternateName>Marcus A. Zimara</alternateName>
>             <alternateName>Marc Ant. Zimara</alternateName>
>             <alternateName>Marcantonio Zimara</alternateName>
>             <alternateName>Marcus Antonius Zimara</alternateName>
>             <alternateName>Marcus Antonius Zimarra</alternateName>
>             <alternateName>Marcianto Zimare</alternateName>
>             <alternateName>Marco Antonio Zimarra</alternateName>
>             <alternateName>Marco Antonio Zimare</alternateName>
>             <birthDate>1460</birthDate>
>             <deathDate>1532</deathDate>
>             <description>JWO</description>
>             <sortableName>Zimara, Marcanton </sortableName>
>         </person>
>
> Articel XML:
>
> <?xml version="1.0" encoding="UTF-8"?>
> <TEI xmlns="http://www.tei-c.org/ns/1.0">
>     <teiHeader>
>         <fileDesc>
>             <titleStmt>
>                 <title>a nihilo nihil fit</title>
>                 <author>
>                     <persName 
> ref="/db/projects/jwo/data/lists/personenListe.xml#BS_d1e509" 
> xml:id="author_BS_d1e509">
>                         <forename>Marcanton</forename>
>                         <surname>Zimara</surname>
>                     </persName>
>                 </author>
>             </titleStmt>
>             <sourceDesc>
>                 <p xml:id="p_sourceDesc_igw_tvr_pzb">born digital</p>
>             </sourceDesc>
>         </fileDesc>
>     </teiHeader>
>     <text xml:lang="de-DE" type="main">
>         <body>
>             <div1 xml:id="div1_d1e23_2">
>                 <p xml:id="p_d1e27_1" n="1"> Lorem ipsum dolor sit 
> amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut 
> labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud 
> exercitation <persName xml:id="persName_sa123" 
> ref="https://d-nb.info/gnd/120156784" 
> rend="smallcaps">Zimara</persName> ullamco laboris nisi ut aliquip ex 
> ea commodo consequat. Duis aute irure dolor in reprehenderit in 
> voluptate velit esse cillum dolore eu <persName xml:id="persName_s123" 
> ref="https://d-nb.info/gnd/120156784">Zimara</persName> fugiat nulla 
> pariatur. Excepteur sint occaecat cupidatat non proident, sunt in 
> culpa qui officia deserunt mollit anim id est laborum. </p>
>             </div1>
>         </body>
>     </text>
> </TEI>
>
> Collection.xconf:
>
> <collection xmlns="http://exist-db.org/collection-config/1.0">
>     <index xmlns:gndo="https://d-nb.info/standards/elementset/gnd#" 
> xmlns:owl="http://www.w3.org/2002/07/owl#" 
> xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" 
> xmlns:xs="http://www.w3.org/2001/XMLSchema">
>         <lucene>
>             <module uri="http://place.sok.org/xquery/index-persons" 
> prefix="ip" 
> at="xmldb:exist:///db/apps/sok-application/modules/index-persons.xqm"/>
>             <analyzer 
> class="org.apache.lucene.analysis.standard.StandardAnalyzer"/>
>             <analyzer 
> class="org.exist.indexing.lucene.analyzers.NoDiacriticsStandardAnalyzer" 
> id="nodiacritics"/>
>             <text qname="adcache">
>                 <field name="basicId" 
> expression="//person/@id/string()"/>
>                 <field name="fullname" 
> expression="string(./basic/person/name)"/>
>                 <field name="gndURI" 
> expression="string(./basic/person/identifier[@preferred eq 'YES'])"/>
>                 <field name="gndID" 
> expression="substring-after(./basic/person/identifier[@preferred eq 
> 'YES']/string(), '/gnd/')"/>
>                 <field name="status" 
> expression="./basic/person/@status/string()"/>
>                 <field name="articleID" 
> expression="ip:getArticleFromPersonCache(.)"/>
>                 <field name="articleRole" 
> expression="ip:getArticleRoleFromPersonCache(.)"/>
>                 <field name="fulltext" expression="ip:getFullText(.)"/>
>                </text>
>         </lucene>
>     </index>
> </collection>
>
> Module Functions:
>
> module namespace ip                 = 
> "http://place.sok.org/xquery/index-persons";
>
> declare namespace basic               = 
> "http://place.sok.org/xquery/basic" ;
>
> declare namespace xs                = "http://www.w3.org/2001/XMLSchema";
> declare namespace tei               = "http://www.tei-c.org/ns/1.0";
> declare namespace util              = "http://exist-db.org/xquery/util";
>
> declare function ip:getArticleFromPersonCache($adcache as element()) 
> as xs:string* {
>     let $parentCollectionPath as xs:anyURI? := 
> ip:getParentCollection($adcache),
>         $basicId as xs:string := $adcache/basic/person/@id/string(),
>         $identifier as xs:string? := 
> $adcache/basic/person/identifier[@preferred eq 'YES']/string(),
>         $listId as xs:string? := collection( $variables:jwo-lists-path 
> )/tei:TEI//tei:person[ basic:basic-id-from-url(string(@sameAs)) eq 
> $basicId]/@xml:id/string(),
>         $foundInDocumentIds as xs:string* :=
>             if ( matches($parentCollectionPath,'prepublish') )
>             then
>             (
> collection($parentCollectionPath)/tei:TEI[./tei:teiHeader//tei:idno[1]/string() 
> ne 
> ''][matches(replace((normalize-space('||'||string-join(distinct-values(.//tei:persName[@ref]/@ref/string()) 
> ! replace(.,'.*?#',''), 
> '||')||'||')||normalize-space('||'||string-join(distinct-values(.//tei:persName[@source]/@source/string()) 
> ! replace(.,'.*?#',''), 
> '||')||'||')),'\|{4}',''),'\|{2}('||$basicId||'|'||$listId||'|'||$identifier||')\|{2}')]//tei:idno/string()
>             )
>             else
>             (
> collection($parentCollectionPath)/tei:TEI[./tei:teiHeader//tei:idno[1]/string() 
> ne 
> ''][matches(replace((normalize-space('||'||string-join(distinct-values(.//tei:persName[@ref][not(parent::editor)]/@ref/string()) 
> ! replace(.,'.*?#',''), 
> '||')||'||')||normalize-space('||'||string-join(distinct-values(.//tei:persName[@source][not(parent::editor)]/@source/string()) 
> ! replace(.,'.*?#',''), 
> '||')||'||')),'\|{4}',''),'\|{2}('||$basicId||'|'||$listId||'|'||$identifier||')\|{2}')]//tei:idno/string()
>             )
>     return
>     (
>         $foundInDocumentIds
>     )
> };
>
> declare function ip:getAuthenticatedArticleCollection($collection-name 
> as xs:string) as item()* {
>     if ($collection-name eq 'prepublish') then 
> xmldb:xcollection($variables:jwo-prepublish-path) else 
> xmldb:xcollection($variables:jwo-publish-path)
> };
>
> declare function 
> ip:getPersNamesInCollectionFromCachedPerson($cached-person as 
> element(), $collection-name as xs:string) as element()* {
>     let $basicId := $cached-person/basic/person/@id/string()
>     let $identifier := 
> $cached-person/basic/person/identifier[@preferred eq 'YES']/string()
>     let $listId := collection( $variables:jwo-lists-path 
> )/tei:TEI/tei:text[1]/tei:body[1]/tei:listPerson[1]/tei:person[ 
> basic:basic-id-from-url(string(@sameAs)) eq $basicId]/@xml:id/string()
>     let $collection 
> := ip:getAuthenticatedArticleCollection($collection-name)
>     return (
>         $collection//tei:persName[
>             string(@ref) eq $identifier
>             or ip:getIdFromUri(string(@ref)) eq $listId
>             or substring-after(string(@ref), '#') eq $basicId
>             or substring-before(substring-after(string(@source), 
> 'persons/'), '.xml') eq $basicId]
>     )
> };
>
> declare function ip:getRoleFromPersName($persName as element(), 
> $collection-name as xs:string) as xs:string? {
>     if ($persName/ancestor::*/local-name() = 'author')
>     then ( 'author' )
>     else if ($persName/ancestor::*/local-name() = 'editor')
>     then (
>         if ($collection-name eq 'prepublish') then ( 'editor' )
>         (: Ignore editors in published case :)
>         else ()
>     )
>     else ( 'annotated' )
> };
>
> declare function ip:getArticleRoleFromPersonCache($cached-person as 
> element(), $collection-name as xs:string) as xs:string* {
>     let $allPersNames :=
>         if ($collection-name ne 'prepublish') then (
> ip:getPersNamesInCollectionFromCachedPerson($cached-person, 
> $collection-name)[not(ancestor::*/local-name() = 'editor')]
>         ) else (
> ip:getPersNamesInCollectionFromCachedPerson($cached-person, 
> $collection-name)
>         )
>     return (
>         for $articleGroup in $allPersNames
>         let $articleID := $articleGroup/ancestor::tei:TEI//tei:idno[1]
>         group by $articleID
>         return (
>             $articleID || '@@' || string-join(distinct-values(
>                 for $persName in $articleGroup
>                 let $role := ip:getRoleFromPersName($persName, 
> $collection-name)
>                 order by $role
>                 return $role
>             ), ' ')
>         )
>     )
> };
>
> declare function ip:getParentCollection($element as node()) as 
> xs:anyURI? {
>     resolve-uri('../../', $element/base-uri())
> };
>
> declare function ip:getIdFromUri($uri as xs:string) as xs:string {
>     substring-after($uri, '#')
> };
>
> declare function basic:basic-id-from-url($url as xs:string) as 
> xs:string? {
>     substring-after(substring-before($url, '?dataset'),'persons/')
> };
>
> declare function ip:getFullText($element) as xs:string {
>     let $parentCollection as xs:anyURI? := 
> ip:getParentCollection($element)
>     return
>     (
>        normalize-space(string-join(
>                     let $basicId as xs:string := 
> $element/basic/person/@id/string(),
>                         $identifier as xs:string* := 
> $element/basic/person/identifier[@preferred eq 'YES']/string(),
>                         $listId as item()* := collection( 
> $variables:lists-path 
> )/tei:TEI/tei:text[1]/tei:body[1]/tei:listPerson[1]/tei:person[ 
> basic:basic-id-from-url(string(@sameAs)) eq $basicId]/@xml:id/string(),
>                         $element-string as xs:string* := 
> string($element),
>                         $collections as item()* := 
> collection($parentCollection)//tei:persName[string(@ref) eq 
> $identifier or ip:getIdFromUri(string(@ref)) eq $listId or 
> substring-after(string(@ref), '#') eq $basicId or 
> substring-before(substring-after(string(@source), 'persons/'), '.xml') 
> eq $basicId][1],
>                         $element-cache-string as xs:string* := 
> string-join(for $found-element in $collections where 
> count($found-element) > 0 return $found-element, ' ')
>                     return
>                     (
>                         $element-string,$element-cache-string
>                     ),' '))
>     )
> };
>
> Please help.
>
>

[Exist-open] Cross reference Index - why is so slow

[Lars S. <sch...@sa...>]

Hey,

we assume that we are not using the index in our project as intended.
Because when we try to build the index we have created, it takes a very 
long time.

We have two collections, one with a data stack of 687 in which data is 
stored, and one with 400 xml where articles are are stored.

For the personal article we want certain information from the articles 
and vice versa.

Person XML:

<person xml:id="i0c9ab7e2-2e21-39ff-aea8-c56ad4702a7f" status="safe" 
modified="2024-07-30T13:26:09.154+02:00">
             <name>Marcanton Zimara</name>
             <identifier 
preferred="YES">https://d-nb.info/gnd/120156784</identifier>
             <alternateName>Marcusantonius Zimara</alternateName>
             <alternateName>Marcus Anthonius Zimara</alternateName>
             <alternateName>Antonius Zimara</alternateName>
             <alternateName>M. Antonius Zimarra</alternateName>
             <alternateName>Marc Antoine Zimara</alternateName>
             <alternateName>M. Anto. Zimare</alternateName>
             <alternateName>Marco A. Zimara</alternateName>
             <alternateName>Marcus A. Zimara</alternateName>
             <alternateName>Marc Ant. Zimara</alternateName>
             <alternateName>Marcantonio Zimara</alternateName>
             <alternateName>Marcus Antonius Zimara</alternateName>
             <alternateName>Marcus Antonius Zimarra</alternateName>
             <alternateName>Marcianto Zimare</alternateName>
             <alternateName>Marco Antonio Zimarra</alternateName>
             <alternateName>Marco Antonio Zimare</alternateName>
             <birthDate>1460</birthDate>
             <deathDate>1532</deathDate>
             <description>JWO</description>
             <sortableName>Zimara, Marcanton </sortableName>
         </person>

Articel XML:

<?xml version="1.0" encoding="UTF-8"?>
<TEI xmlns="http://www.tei-c.org/ns/1.0">
     <teiHeader>
         <fileDesc>
             <titleStmt>
                 <title>a nihilo nihil fit</title>
                 <author>
                     <persName 
ref="/db/projects/jwo/data/lists/personenListe.xml#BS_d1e509" 
xml:id="author_BS_d1e509">
                         <forename>Marcanton</forename>
                         <surname>Zimara</surname>
                     </persName>
                 </author>
             </titleStmt>
             <sourceDesc>
                 <p xml:id="p_sourceDesc_igw_tvr_pzb">born digital</p>
             </sourceDesc>
         </fileDesc>
     </teiHeader>
     <text xml:lang="de-DE" type="main">
         <body>
             <div1 xml:id="div1_d1e23_2">
                 <p xml:id="p_d1e27_1" n="1"> Lorem ipsum dolor sit 
amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut 
labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud 
exercitation <persName xml:id="persName_sa123" 
ref="https://d-nb.info/gnd/120156784" rend="smallcaps">Zimara</persName> 
ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure 
dolor in reprehenderit in voluptate velit esse cillum dolore eu 
<persName xml:id="persName_s123" 
ref="https://d-nb.info/gnd/120156784">Zimara</persName> fugiat nulla 
pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa 
qui officia deserunt mollit anim id est laborum. </p>
             </div1>
         </body>
     </text>
</TEI>

Collection.xconf:

<collection xmlns="http://exist-db.org/collection-config/1.0">
     <index xmlns:gndo="https://d-nb.info/standards/elementset/gnd#" 
xmlns:owl="http://www.w3.org/2002/07/owl#" 
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" 
xmlns:xs="http://www.w3.org/2001/XMLSchema">
         <lucene>
             <module uri="http://place.sok.org/xquery/index-persons" 
prefix="ip" 
at="xmldb:exist:///db/apps/sok-application/modules/index-persons.xqm"/>
             <analyzer 
class="org.apache.lucene.analysis.standard.StandardAnalyzer"/>
             <analyzer 
class="org.exist.indexing.lucene.analyzers.NoDiacriticsStandardAnalyzer" 
id="nodiacritics"/>
             <text qname="adcache">
                 <field name="basicId" expression="//person/@id/string()"/>
                 <field name="fullname" 
expression="string(./basic/person/name)"/>
                 <field name="gndURI" 
expression="string(./basic/person/identifier[@preferred eq 'YES'])"/>
                 <field name="gndID" 
expression="substring-after(./basic/person/identifier[@preferred eq 
'YES']/string(), '/gnd/')"/>
                 <field name="status" 
expression="./basic/person/@status/string()"/>
                 <field name="articleID" 
expression="ip:getArticleFromPersonCache(.)"/>
                 <field name="articleRole" 
expression="ip:getArticleRoleFromPersonCache(.)"/>
                 <field name="fulltext" expression="ip:getFullText(.)"/>
                </text>
         </lucene>
     </index>
</collection>

Module Functions:

module namespace ip                 = 
"http://place.sok.org/xquery/index-persons";

declare namespace basic               = 
"http://place.sok.org/xquery/basic" ;

declare namespace xs                = "http://www.w3.org/2001/XMLSchema";
declare namespace tei               = "http://www.tei-c.org/ns/1.0";
declare namespace util              = "http://exist-db.org/xquery/util";

declare function ip:getArticleFromPersonCache($adcache as element()) as 
xs:string* {
     let $parentCollectionPath as xs:anyURI? := 
ip:getParentCollection($adcache),
         $basicId as xs:string := $adcache/basic/person/@id/string(),
         $identifier as xs:string? := 
$adcache/basic/person/identifier[@preferred eq 'YES']/string(),
         $listId as xs:string? := collection( $variables:jwo-lists-path 
)/tei:TEI//tei:person[ basic:basic-id-from-url(string(@sameAs)) eq 
$basicId]/@xml:id/string(),
         $foundInDocumentIds as xs:string* :=
             if ( matches($parentCollectionPath,'prepublish') )
             then
             (
collection($parentCollectionPath)/tei:TEI[./tei:teiHeader//tei:idno[1]/string() 
ne 
''][matches(replace((normalize-space('||'||string-join(distinct-values(.//tei:persName[@ref]/@ref/string()) 
! replace(.,'.*?#',''), 
'||')||'||')||normalize-space('||'||string-join(distinct-values(.//tei:persName[@source]/@source/string()) 
! replace(.,'.*?#',''), 
'||')||'||')),'\|{4}',''),'\|{2}('||$basicId||'|'||$listId||'|'||$identifier||')\|{2}')]//tei:idno/string()
             )
             else
             (
collection($parentCollectionPath)/tei:TEI[./tei:teiHeader//tei:idno[1]/string() 
ne 
''][matches(replace((normalize-space('||'||string-join(distinct-values(.//tei:persName[@ref][not(parent::editor)]/@ref/string()) 
! replace(.,'.*?#',''), 
'||')||'||')||normalize-space('||'||string-join(distinct-values(.//tei:persName[@source][not(parent::editor)]/@source/string()) 
! replace(.,'.*?#',''), 
'||')||'||')),'\|{4}',''),'\|{2}('||$basicId||'|'||$listId||'|'||$identifier||')\|{2}')]//tei:idno/string()
             )
     return
     (
         $foundInDocumentIds
     )
};

declare function ip:getAuthenticatedArticleCollection($collection-name 
as xs:string) as item()* {
     if ($collection-name eq 'prepublish') then 
xmldb:xcollection($variables:jwo-prepublish-path) else 
xmldb:xcollection($variables:jwo-publish-path)
};

declare function 
ip:getPersNamesInCollectionFromCachedPerson($cached-person as element(), 
$collection-name as xs:string) as element()* {
     let $basicId := $cached-person/basic/person/@id/string()
     let $identifier := 
$cached-person/basic/person/identifier[@preferred eq 'YES']/string()
     let $listId := collection( $variables:jwo-lists-path 
)/tei:TEI/tei:text[1]/tei:body[1]/tei:listPerson[1]/tei:person[ 
basic:basic-id-from-url(string(@sameAs)) eq $basicId]/@xml:id/string()
     let $collection 
:= ip:getAuthenticatedArticleCollection($collection-name)
     return (
         $collection//tei:persName[
             string(@ref) eq $identifier
             or ip:getIdFromUri(string(@ref)) eq $listId
             or substring-after(string(@ref), '#') eq $basicId
             or substring-before(substring-after(string(@source), 
'persons/'), '.xml') eq $basicId]
     )
};

declare function ip:getRoleFromPersName($persName as element(), 
$collection-name as xs:string) as xs:string? {
     if ($persName/ancestor::*/local-name() = 'author')
     then ( 'author' )
     else if ($persName/ancestor::*/local-name() = 'editor')
     then (
         if ($collection-name eq 'prepublish') then ( 'editor' )
         (: Ignore editors in published case :)
         else ()
     )
     else ( 'annotated' )
};

declare function ip:getArticleRoleFromPersonCache($cached-person as 
element(), $collection-name as xs:string) as xs:string* {
     let $allPersNames :=
         if ($collection-name ne 'prepublish') then (
ip:getPersNamesInCollectionFromCachedPerson($cached-person, 
$collection-name)[not(ancestor::*/local-name() = 'editor')]
         ) else (
ip:getPersNamesInCollectionFromCachedPerson($cached-person, 
$collection-name)
         )
     return (
         for $articleGroup in $allPersNames
         let $articleID := $articleGroup/ancestor::tei:TEI//tei:idno[1]
         group by $articleID
         return (
             $articleID || '@@' || string-join(distinct-values(
                 for $persName in $articleGroup
                 let $role := ip:getRoleFromPersName($persName, 
$collection-name)
                 order by $role
                 return $role
             ), ' ')
         )
     )
};

declare function ip:getParentCollection($element as node()) as xs:anyURI? {
     resolve-uri('../../', $element/base-uri())
};

declare function ip:getIdFromUri($uri as xs:string) as xs:string {
     substring-after($uri, '#')
};

declare function basic:basic-id-from-url($url as xs:string) as xs:string? {
     substring-after(substring-before($url, '?dataset'),'persons/')
};

declare function ip:getFullText($element) as xs:string {
     let $parentCollection as xs:anyURI? := ip:getParentCollection($element)
     return
     (
        normalize-space(string-join(
                     let $basicId as xs:string := 
$element/basic/person/@id/string(),
                         $identifier as xs:string* := 
$element/basic/person/identifier[@preferred eq 'YES']/string(),
                         $listId as item()* := collection( 
$variables:lists-path 
)/tei:TEI/tei:text[1]/tei:body[1]/tei:listPerson[1]/tei:person[ 
basic:basic-id-from-url(string(@sameAs)) eq $basicId]/@xml:id/string(),
                         $element-string as xs:string* := string($element),
                         $collections as item()* := 
collection($parentCollection)//tei:persName[string(@ref) eq $identifier 
or ip:getIdFromUri(string(@ref)) eq $listId or 
substring-after(string(@ref), '#') eq $basicId or 
substring-before(substring-after(string(@source), 'persons/'), '.xml') 
eq $basicId][1],
                         $element-cache-string as xs:string* := 
string-join(for $found-element in $collections where 
count($found-element) > 0 return $found-element, ' ')
                     return
                     (
                         $element-string,$element-cache-string
                     ),' '))
     )
};

Please help.


-- 
Lars Scheideler
- wiss. technischer Mitarbeiter -
Althochdeutsches Wörterbuch & Digital Humanities

Sächsische Akademie der Wissenschaften zu Leipzig
Karl-Tauchnitz-Straße 1
04107 Leipzig

sch...@sa...
www.saw-leipzig.de

[Exist-open] string-join and ancestor axis query very slow

[Liuzzo, P. <Pie...@bi...>]

Dear all,

I am facing a strange issue, and I wonder if anyone has any ideas about it.

I have been using a simple string-join on the result of an xpath on the ancestor axis without any issues and with very fast results, e.g.

let $ancs := $file/ancestor::tree:directory
return
    replace(string-join($ancs/@name, '%2F'), '\./', '’)

Now however this kind of queries take a lot of time. I had never seen these function appearing in profiling, so it must have been so fast to never make it up the list, whereas now it is reported with a major jump in time consumed (see below).

replace /db/apps/hertziana/modules/app.xql [885:21]     120     42.136
string-join     /db/apps/hertziana/modules/app.xql [885:29]     120     42.13
seo:Catalog     /db/apps/fotothek/modules/seo.xqm [-1:-1]       6       24.413

The “tree" document in question is rather large, but it always was and was not an issue at all.

Since I have not done any recent update to the application or the data or the indexes I am wondering what could have gone wrong and what I can do to recover the previous performance.

Thanks a lot for any ideas about this

all best
Pietro

Re: [Exist-open] Can't get eXIst to start

[steve h. <ste...@gm...>]

Hi all

Stet.  Used Jar installer and hacked startup.sh until I worked out what went wrong.  And yes, it was the classpath, but I’m still slightly in the dark about what I did differently ...

Steve

> On 20 Aug 2024, at 15:28, steve harris <Ste...@gm...> wrote:
> 
> Hi
> 
> Old returning eXist developer here and a little rusty ...
> 
> I have a M1 MacBook Pro running Sonama, eXist installed by JRE, java 11 JDK installed by Brew.  
> 
> Java - version works as expected, echo $JAVA_HOME contains a path with the openjdk bin folder.  
> 
> I keep getting ‘This application requires that Java 8 or later be installed …’ message.  I’ve seen previous versions of this call and I think I’m compliant with the fixes there.  I’ve also tried this with Oracle Java too with the same result.
> 
> Any ideas?  Or exact paths to stuff that you know works?
> 
> Best
> 
> Steve

[Exist-open] Can't get eXIst to start

[steve h. <ste...@gm...>]

Hi

Old returning eXist developer here and a little rusty ...

I have a M1 MacBook Pro running Sonama, eXist installed by JRE, java 11 JDK installed by Brew.  

Java - version works as expected, echo $JAVA_HOME contains a path with the openjdk bin folder.  

I keep getting ‘This application requires that Java 8 or later be installed …’ message.  I’ve seen previous versions of this call and I think I’m compliant with the fixes there.  I’ve also tried this with Oracle Java too with the same result.

Any ideas?  Or exact paths to stuff that you know works?

Best

Steve

Re: [Exist-open] Can you parse JSON with the eXist database?

[Ted H. <meg...@gm...>]

In case you were wondering how I fixed my project, I will give you the
details.

I borrowed some code from my SVG_Bezier_Curve Javascript page:

function Setup2() {

    var SVG_Data;
    var Retrieved_Data;
    var Bezier_Curve_1;
    var Bezier_Curve_2;
    var Counter;
    var Coordinate_Attribute;
    var Coordinate;
    var Style_Attribute;
    var Style_Details;
    var Bezier_Curve_Identification;

    SVG_Data = new XMLHttpRequest();
          SVG_Data.open("GET","
http://localhost:8080/exist/rest/db/apps/HTML_Student/SVG_Bezier_Curve.xq",
true);
          SVG_Data.onreadystatechange = function () {
              if (SVG_Data.readyState == 4) {
              Retrieved_Data = SVG_Data.responseText;
              Retrieved_Data = Retrieved_Data.split("/");
              Counter = 0;
              Bezier_Curve_1 = Retrieved_Data[Counter];
              Counter = Counter + 1;
              Bezier_Curve_2 = Retrieved_Data[Counter];}
              Bezier_Curve_1 = Bezier_Curve_1.split("*");
              Counter = 0;
              Coordinate_Attribute = Bezier_Curve_1[Counter];
              Counter = Counter + 1;
              Coordinate = Bezier_Curve_1[Counter];
              Counter = Counter + 1;
              Style_Attribute = Bezier_Curve_1[Counter];
              Counter = Counter + 1;
              Style_Details = Bezier_Curve_1[Counter];
              Bezier_Curve_Identification =
document.getElementById('Bezier_Curve_1');

Bezier_Curve_Identification.setAttribute(Coordinate_Attribute, Coordinate);
              Bezier_Curve_Identification.setAttribute(Style_Attribute,
Style_Details);
              Bezier_Curve_2 = Bezier_Curve_2.split("*");
              Counter = 0;
              Coordinate_Attribute = Bezier_Curve_2[Counter];
              Counter = Counter + 1;
              Coordinate = Bezier_Curve_2[Counter];
              Counter = Counter + 1;
              Style_Attribute = Bezier_Curve_2[Counter];
              Counter = Counter + 1;
              Style_Details = Bezier_Curve_2[Counter];
              Bezier_Curve_Identification =
document.getElementById('Bezier_Curve_2');

Bezier_Curve_Identification.setAttribute(Coordinate_Attribute, Coordinate);
              Bezier_Curve_Identification.setAttribute(Style_Attribute,
Style_Details);

              };

SVG_Data.send();}


I changed some variable names and a couple of other minor things and
developed this code:

function Setup() {
    var XMLHttp;
    var JSON_Data;
    var Retrieved_Data;



    XMLHttp = new XMLHttpRequest();
          XMLHttp.open("GET","
http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json", true);
          XMLHttp.onreadystatechange = function () {
            if (XMLHttp.readyState == 4) {
              Retrieved_Data = XMLHttp.responseText;
              document.getElementById("JSON_Text").value = Retrieved_Data}

          };
          XMLHttp.send();}

I also turned off a feature in the console because the CTRL F5 wasn't
working correctly.  When I finish my current project, I will publish a
video of my work on YouTube for everyone to watch.  I don't want anyone to
be as stuck as I was today.  It's hard when you don't know what you're
doing wrong and everything you try to do doesn't change anything.

On Mon, Aug 19, 2024 at 3:11 PM Ted Hickox <meg...@gm...> wrote:

> If I knew as much as all of you know, I wouldn't be asking you for
> assistance.  But I'm over 50 years old and I used to program on an NCR
> mainframe using COBOL.  So you might say I was at the foundation of
> computers.
>
> My first problem is that the CTRL F5 doesn't always refresh my webpage.
> So this presents a challenge.  And at the beginning of this interaction I
> showed everyone my HTML, CSS, Javascript, and JSON pages.  but I can only
> presume that the people who have participated in this conversation so far
> have not read my original post.  So once again I will post my code.,
>
> My HTML page:
>
> <html xmlns="http://www.w3.org/1999/xhtml">
> <head>
> <title>First JSON Experiment</title>
> <link rel="stylesheet" type="text/css" href="
> http://localhost:8080/exist/rest/db/apps/HTML_Student/JSON_Example.css"/>
> <script language="javascript" src="
> http://localhost:8080/exist/rest/db/apps/HTML_Student/JSON_Example_2.js"/>
> </head>
> <body onload="Setup()">
> <input type="text" id="JSON_Text">
> </input>
> </body>
> </html>
>
> My CSS page:
>
> #JSON_Text {
>     position: absolute;
>     top: 200px;
>     left: 300px;
>     height: 25px;
>     width: 300px;
> }
>
> My current Javascript page:
>
> function Setup() {
>     var XMLHttp;
>     var JSON_Data;
>
>     MLHttp = new XMLHttpRequest();
>     XMLHttp.open("GET", "
> http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json", true);
>     XMLHttp.send();
> }
>
> And my current JSON page:
>
> {"name":"John", "age":30, "city":"New York"}
>
> And my most recent error:
>
> [image: image.png]
>
> Up until now the console hasn't been displaying any errors.  So how could
> I correct something that I couldn't identify?
>
> On Mon, Aug 19, 2024 at 2:47 PM Florian Schmitt <
> mai...@fl...> wrote:
>
>> Ted, since nobody here knows how your complete code and db content looks
>> like, you'll have to fix it yourself. To do so, check how to use the Web
>> Console:
>>
>> For Firefox, see
>> https://firefox-source-docs.mozilla.org/devtools-user/web_console/index.html
>> For Edge, see
>> https://learn.microsoft.com/en-us/microsoft-edge/devtools-guide-chromium/console/
>> For Chrome, see
>> https://developer.chrome.com/docs/devtools/console?hl=en#javascript
>>
>> and test what happens if you enter the three commands from your js:
>>
>> XMLHttp = new XMLHttpRequest();
>> XMLHttp.open("GET", "
>> http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json",
>> true);
>> XMLHttp.send();
>>
>> in the Web Console.
>>
>>
>>

Re: [Exist-open] Can you parse JSON with the eXist database?

[Ted H. <meg...@gm...>]

Okay, I was stupid and forgot to add the send command.  But this still
didn't fix the problem.  Here is my adjusted code:

function Setup() {
    var XMLHttp;
    var JSON_Data;

    XMLHttp = new XMLHttpRequest();
    XMLHttp.open("GET", "
http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json", true);
        XMLHttp.onReadyStateChange = function(){

            document.getElementById("JSON_Text").value = "I'm right here.";

        };
XMLHttp.send();}

Unless I reposition my document.getElementById command, nothing will appear
in my textbox.

On Mon, Aug 19, 2024 at 1:23 PM Florian Schmitt <
ml-...@fl...> wrote:

> Ted, as far as I see, you never send() your request. The open() method
> specifies / initializes the request, but it doesn't send/execute it.
> See:
> https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/open
> https://www.w3schools.com/xml/ajax_xmlhttprequest_create.asp
>
> HTH
> Florian
>
>
> Am 19.08.24 um 19:52 schrieb Ted Hickox:
> > I keep hitting a stone wall.  If I write my code like this:
> >
> > function Setup() {
> >      var XMLHttp;
> >      var JSON_Data;
> >
> >      XMLHttp = new XMLHttpRequest();
> >      XMLHttp.open("GET",
> > "http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json
> > <http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json>",
> > true);
> >
> >
> >              document.getElementById("JSON_Text").value = "I'm right
> > here.";
> >
> > }
> > The words I'm right here appear in the textbox.
> >
> > But if I write my code like this:
> >
> > function Setup() {
> >      var XMLHttp;
> >      var JSON_Data;
> >
> >      XMLHttp = new XMLHttpRequest();
> >      XMLHttp.open("GET",
> > "http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json
> > <http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json>",
> > true);
> >      XMLHttp.onReadyStateChange = function(){
> >
> >              document.getElementById("JSON_Text").value = "I'm right
> > here.";
> >      };
> > }
> >
> > Nothing appears in my textbox.  And I don't understand why.  Can
> > someone please explain this to me?
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

Re: [Exist-open] Can you parse JSON with the eXist database?

[Florian S. <ml-...@fl...>]

Ted, as far as I see, you never send() your request. The open() method 
specifies / initializes the request, but it doesn't send/execute it.
See:
https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/open
https://www.w3schools.com/xml/ajax_xmlhttprequest_create.asp

HTH
Florian


Am 19.08.24 um 19:52 schrieb Ted Hickox:
> I keep hitting a stone wall.  If I write my code like this:
> 
> function Setup() {
>      var XMLHttp;
>      var JSON_Data;
> 
>      XMLHttp = new XMLHttpRequest();
>      XMLHttp.open("GET", 
> "http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json 
> <http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json>", 
> true);
> 
> 
>              document.getElementById("JSON_Text").value = "I'm right 
> here.";
> 
> }
> The words I'm right here appear in the textbox.
> 
> But if I write my code like this:
> 
> function Setup() {
>      var XMLHttp;
>      var JSON_Data;
> 
>      XMLHttp = new XMLHttpRequest();
>      XMLHttp.open("GET", 
> "http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json 
> <http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json>", 
> true);
>      XMLHttp.onReadyStateChange = function(){
> 
>              document.getElementById("JSON_Text").value = "I'm right 
> here.";
>      };
> }
> 
> Nothing appears in my textbox.  And I don't understand why.  Can 
> someone please explain this to me?

Re: [Exist-open] Can you parse JSON with the eXist database?

[Ted H. <meg...@gm...>]

I keep hitting a stone wall.  If I write my code like this:

function Setup() {
    var XMLHttp;
    var JSON_Data;

    XMLHttp = new XMLHttpRequest();
    XMLHttp.open("GET", "
http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json", true);


            document.getElementById("JSON_Text").value = "I'm right here.";

}
The words I'm right here appear in the textbox.

But if I write my code like this:

function Setup() {
    var XMLHttp;
    var JSON_Data;

    XMLHttp = new XMLHttpRequest();
    XMLHttp.open("GET", "
http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json", true);
    XMLHttp.onReadyStateChange = function(){

            document.getElementById("JSON_Text").value = "I'm right here.";
    };
}

Nothing appears in my textbox.  And I don't understand why.  Can someone
please explain this to me?

On Mon, Aug 19, 2024 at 12:16 PM Ted Hickox <meg...@gm...> wrote:

> One more update.  I was stupid.  I forgot to replace the variables.  Now I
> get undefined when I run the javascript code.
>
> function Setup() {
>     var XMLHttp;
>     var JSON_Data;
>
>     XMLHttp = new XMLHttpRequest();
>     XMLHttp.open("GET", "
> http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json", true);
>     XMLHttp.onreadystatechange = function () {
>         if(XMLHttp.readystate == 4) {
>             JSON_Data = XMLHttp.responseText;
>             JSON_Data = JSON.parse(JSON_Data);
>         }
>     };
>
>         document.getElementById("JSON_Text").value = JSON_Data;
>
>
> }
>
> On Mon, Aug 19, 2024 at 12:04 PM Ted Hickox <meg...@gm...> wrote:
>
>> I updated my code again and now nothing appears in the textbox.  This is
>> truly frustrating.
>>
>> javascript document:
>>
>> function Setup() {
>>
>>     XMLHttp = new XMLHttpRequest();
>>     XMLHttp.open("GET", "
>> http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json",
>> true);
>>     XMLHttp.onreadystatechange = function () {
>>         if(XMLHttp.readyState == 4){}
>>         document.getElementById("JSON_Text").value = "Taking another
>> step.";
>>
>>     };
>> }
>>
>> On Mon, Aug 19, 2024 at 11:33 AM Ted Hickox <meg...@gm...> wrote:
>>
>>> I fixed a couple of errors and got some new responses.  But I still
>>> didn't get the responses I needed.
>>>
>>> Here is my new Javascript document:
>>>
>>> function Setup() {
>>>
>>>     XMLHttp = new XMLHttpRequest();
>>>     XMLHttp.open("GET", "
>>> http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json",
>>> true);
>>>     XMLHttp.onreadystatechange = function () {
>>>          if(XMLHttp.readyState == 4){}{
>>>              JSON_Data = XMLHttp.responseText;
>>>              JSON_Data = JSON.parse(JSON_Data);
>>>              document.getElementById("JSON_Text").value = JSON_Data;
>>>          }
>>>
>>>     };
>>> }
>>>
>>> And here is my new people.json
>>>
>>> {"name":"John", "age":30, "city":"New York"}
>>>
>>> And here is the new response from the HTML textbox:
>>>
>>> [object Object]
>>>
>>> I'm still not sure what I'm doing wrong.
>>>
>>> On Mon, Aug 19, 2024 at 10:06 AM Florian Schmitt <
>>> ml-...@fl...> wrote:
>>>
>>>> Ted, if your resource is named People.json in eXist (with upper-case
>>>> "P"), you'll need to adapt your URL accordingly. Currently, the URL points
>>>> to people.json, not People.json.
>>>>
>>>> 19.08.2024 07:25:51 Ted Hickox <meg...@gm...>:
>>>>
>>>> My apology for giving you incorrect Javascript code.  I corrected my
>>>> code.  Here it is:
>>>>
>>>> function Setup() {
>>>>     var XMLHttp;
>>>>     var JSON_Data;
>>>>
>>>>
>>>>     XMLHttp = new XMLHttpRequest();
>>>>     XMLHttp.open("GET", "
>>>> http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json",
>>>> true);
>>>>     XMLHttp.onreadystatechange = function () {
>>>>          if(XMLHttp.readyState == 4){}{
>>>>              JSON_Data = XMLHttp.responseText;
>>>>              document.getElementById("JSON_Text").value = JSON_Data;
>>>>          }
>>>>
>>>>     };
>>>>
>>>> }
>>>>
>>>> When I run the code and check the console, it states that it can't find
>>>> people.json.  I don't know why my webpage can't find people.json  Can you
>>>> assist me?
>>>>
>>>> On Sun, Aug 18, 2024 at 11:42 AM Jean-Paul Rehr <re...@gm...>
>>>> wrote:
>>>>
>>>>> Ted, none of this has to do with eXist parsing JSON.
>>>>>
>>>>> The URL
>>>>> http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json is
>>>>> requesting a JSON document from eXist, that is, eXist is serving the
>>>>> document not parsing it.
>>>>>
>>>>> Incidentally, one can parse a json document in Xquery with the
>>>>> command:
>>>>>
>>>>> fn:json-doc("/db/apps/HTML_Student/people.json")
>>>>>
>>>>> JPR
>>>>>
>>>>> On Sun, Aug 18, 2024 at 6:26 PM Ted Hickox <meg...@gm...>
>>>>> wrote:
>>>>>
>>>>>> If it's possible to parse JSON with the eXist database, why is the
>>>>>> following code retrieving any data from my JSON document:
>>>>>>
>>>>>> JSON Example.xhtml
>>>>>>
>>>>>> <html xmlns="http://www.w3.org/1999/xhtml">
>>>>>> <head>
>>>>>> <title>First JSON Experiment</title>
>>>>>> <link rel="stylesheet" type="text/css" href="
>>>>>> http://localhost:8080/exist/rest/db/apps/HTML_Student/JSON_Example.css"/>
>>>>>>
>>>>>> <script language="javascript" src="
>>>>>> http://localhost:8080/exist/rest/db/apps/HTML_Student/JSON_Example.js"/>
>>>>>>
>>>>>> </head>
>>>>>> <body onload = "Setup()">
>>>>>> <input type = "text" id = "JSON_Text">
>>>>>> </input>
>>>>>> </body>
>>>>>> </html>
>>>>>>
>>>>>> JSON Example.css
>>>>>>
>>>>>> #JSON_Text {
>>>>>>     position: absolute;
>>>>>>     top: 200px;
>>>>>>     left: 300px;
>>>>>>     height: 25px;
>>>>>>     width: 300px;
>>>>>> }
>>>>>>
>>>>>> JSON Example.js
>>>>>>
>>>>>> function Setup() {
>>>>>>     var XMLHttp;
>>>>>>     var JSON_Data;
>>>>>>
>>>>>>     XMLHttp = new XMLHttpRequest();
>>>>>>
>>>>>>     XMLHttp.onreadystatechange = function () {
>>>>>>          if(XMLHttp.readyState == 4){
>>>>>>              JSON_Data = JSON.parse(XMLHttp.responseText);
>>>>>>          }
>>>>>>     };
>>>>>>
>>>>>>     XMLHttp.open("Get", "
>>>>>> http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json",
>>>>>> true);
>>>>>>     XMLHttp.send();
>>>>>>
>>>>>>
>>>>>>     document.getElementById("JSON_Text").value = JSON_Data;
>>>>>> }
>>>>>>
>>>>>> People.json
>>>>>>
>>>>>> {
>>>>>>      "People":{
>>>>>>           {
>>>>>>                "name": "Alex",
>>>>>>                "online": true
>>>>>>           },
>>>>>>           {
>>>>>>                "name": "Billy",
>>>>>>                "online": false
>>>>>>           }
>>>>>>
>>>>>>      }
>>>>>> }
>>>>>> _______________________________________________
>>>>>> Exist-open mailing list
>>>>>> Exi...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>>>>>
>>>>>

Re: [Exist-open] Can you parse JSON with the eXist database?

[Ted H. <meg...@gm...>]

One more update.  I was stupid.  I forgot to replace the variables.  Now I
get undefined when I run the javascript code.

function Setup() {
    var XMLHttp;
    var JSON_Data;

    XMLHttp = new XMLHttpRequest();
    XMLHttp.open("GET", "
http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json", true);
    XMLHttp.onreadystatechange = function () {
        if(XMLHttp.readystate == 4) {
            JSON_Data = XMLHttp.responseText;
            JSON_Data = JSON.parse(JSON_Data);
        }
    };

        document.getElementById("JSON_Text").value = JSON_Data;


}

On Mon, Aug 19, 2024 at 12:04 PM Ted Hickox <meg...@gm...> wrote:

> I updated my code again and now nothing appears in the textbox.  This is
> truly frustrating.
>
> javascript document:
>
> function Setup() {
>
>     XMLHttp = new XMLHttpRequest();
>     XMLHttp.open("GET", "
> http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json", true);
>     XMLHttp.onreadystatechange = function () {
>         if(XMLHttp.readyState == 4){}
>         document.getElementById("JSON_Text").value = "Taking another
> step.";
>
>     };
> }
>
> On Mon, Aug 19, 2024 at 11:33 AM Ted Hickox <meg...@gm...> wrote:
>
>> I fixed a couple of errors and got some new responses.  But I still
>> didn't get the responses I needed.
>>
>> Here is my new Javascript document:
>>
>> function Setup() {
>>
>>     XMLHttp = new XMLHttpRequest();
>>     XMLHttp.open("GET", "
>> http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json",
>> true);
>>     XMLHttp.onreadystatechange = function () {
>>          if(XMLHttp.readyState == 4){}{
>>              JSON_Data = XMLHttp.responseText;
>>              JSON_Data = JSON.parse(JSON_Data);
>>              document.getElementById("JSON_Text").value = JSON_Data;
>>          }
>>
>>     };
>> }
>>
>> And here is my new people.json
>>
>> {"name":"John", "age":30, "city":"New York"}
>>
>> And here is the new response from the HTML textbox:
>>
>> [object Object]
>>
>> I'm still not sure what I'm doing wrong.
>>
>> On Mon, Aug 19, 2024 at 10:06 AM Florian Schmitt <
>> ml-...@fl...> wrote:
>>
>>> Ted, if your resource is named People.json in eXist (with upper-case
>>> "P"), you'll need to adapt your URL accordingly. Currently, the URL points
>>> to people.json, not People.json.
>>>
>>> 19.08.2024 07:25:51 Ted Hickox <meg...@gm...>:
>>>
>>> My apology for giving you incorrect Javascript code.  I corrected my
>>> code.  Here it is:
>>>
>>> function Setup() {
>>>     var XMLHttp;
>>>     var JSON_Data;
>>>
>>>
>>>     XMLHttp = new XMLHttpRequest();
>>>     XMLHttp.open("GET", "
>>> http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json",
>>> true);
>>>     XMLHttp.onreadystatechange = function () {
>>>          if(XMLHttp.readyState == 4){}{
>>>              JSON_Data = XMLHttp.responseText;
>>>              document.getElementById("JSON_Text").value = JSON_Data;
>>>          }
>>>
>>>     };
>>>
>>> }
>>>
>>> When I run the code and check the console, it states that it can't find
>>> people.json.  I don't know why my webpage can't find people.json  Can you
>>> assist me?
>>>
>>> On Sun, Aug 18, 2024 at 11:42 AM Jean-Paul Rehr <re...@gm...>
>>> wrote:
>>>
>>>> Ted, none of this has to do with eXist parsing JSON.
>>>>
>>>> The URL
>>>> http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json is
>>>> requesting a JSON document from eXist, that is, eXist is serving the
>>>> document not parsing it.
>>>>
>>>> Incidentally, one can parse a json document in Xquery with the command:
>>>>
>>>> fn:json-doc("/db/apps/HTML_Student/people.json")
>>>>
>>>> JPR
>>>>
>>>> On Sun, Aug 18, 2024 at 6:26 PM Ted Hickox <meg...@gm...>
>>>> wrote:
>>>>
>>>>> If it's possible to parse JSON with the eXist database, why is the
>>>>> following code retrieving any data from my JSON document:
>>>>>
>>>>> JSON Example.xhtml
>>>>>
>>>>> <html xmlns="http://www.w3.org/1999/xhtml">
>>>>> <head>
>>>>> <title>First JSON Experiment</title>
>>>>> <link rel="stylesheet" type="text/css" href="
>>>>> http://localhost:8080/exist/rest/db/apps/HTML_Student/JSON_Example.css"/>
>>>>>
>>>>> <script language="javascript" src="
>>>>> http://localhost:8080/exist/rest/db/apps/HTML_Student/JSON_Example.js"/>
>>>>>
>>>>> </head>
>>>>> <body onload = "Setup()">
>>>>> <input type = "text" id = "JSON_Text">
>>>>> </input>
>>>>> </body>
>>>>> </html>
>>>>>
>>>>> JSON Example.css
>>>>>
>>>>> #JSON_Text {
>>>>>     position: absolute;
>>>>>     top: 200px;
>>>>>     left: 300px;
>>>>>     height: 25px;
>>>>>     width: 300px;
>>>>> }
>>>>>
>>>>> JSON Example.js
>>>>>
>>>>> function Setup() {
>>>>>     var XMLHttp;
>>>>>     var JSON_Data;
>>>>>
>>>>>     XMLHttp = new XMLHttpRequest();
>>>>>
>>>>>     XMLHttp.onreadystatechange = function () {
>>>>>          if(XMLHttp.readyState == 4){
>>>>>              JSON_Data = JSON.parse(XMLHttp.responseText);
>>>>>          }
>>>>>     };
>>>>>
>>>>>     XMLHttp.open("Get", "
>>>>> http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json",
>>>>> true);
>>>>>     XMLHttp.send();
>>>>>
>>>>>
>>>>>     document.getElementById("JSON_Text").value = JSON_Data;
>>>>> }
>>>>>
>>>>> People.json
>>>>>
>>>>> {
>>>>>      "People":{
>>>>>           {
>>>>>                "name": "Alex",
>>>>>                "online": true
>>>>>           },
>>>>>           {
>>>>>                "name": "Billy",
>>>>>                "online": false
>>>>>           }
>>>>>
>>>>>      }
>>>>> }
>>>>> _______________________________________________
>>>>> Exist-open mailing list
>>>>> Exi...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>>>>
>>>>

Re: [Exist-open] Can you parse JSON with the eXist database?

[Ted H. <meg...@gm...>]

I updated my code again and now nothing appears in the textbox.  This is
truly frustrating.

javascript document:

function Setup() {

    XMLHttp = new XMLHttpRequest();
    XMLHttp.open("GET", "
http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json", true);
    XMLHttp.onreadystatechange = function () {
        if(XMLHttp.readyState == 4){}
        document.getElementById("JSON_Text").value = "Taking another step.";

    };
}

On Mon, Aug 19, 2024 at 11:33 AM Ted Hickox <meg...@gm...> wrote:

> I fixed a couple of errors and got some new responses.  But I still didn't
> get the responses I needed.
>
> Here is my new Javascript document:
>
> function Setup() {
>
>     XMLHttp = new XMLHttpRequest();
>     XMLHttp.open("GET", "
> http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json", true);
>     XMLHttp.onreadystatechange = function () {
>          if(XMLHttp.readyState == 4){}{
>              JSON_Data = XMLHttp.responseText;
>              JSON_Data = JSON.parse(JSON_Data);
>              document.getElementById("JSON_Text").value = JSON_Data;
>          }
>
>     };
> }
>
> And here is my new people.json
>
> {"name":"John", "age":30, "city":"New York"}
>
> And here is the new response from the HTML textbox:
>
> [object Object]
>
> I'm still not sure what I'm doing wrong.
>
> On Mon, Aug 19, 2024 at 10:06 AM Florian Schmitt <
> ml-...@fl...> wrote:
>
>> Ted, if your resource is named People.json in eXist (with upper-case
>> "P"), you'll need to adapt your URL accordingly. Currently, the URL points
>> to people.json, not People.json.
>>
>> 19.08.2024 07:25:51 Ted Hickox <meg...@gm...>:
>>
>> My apology for giving you incorrect Javascript code.  I corrected my
>> code.  Here it is:
>>
>> function Setup() {
>>     var XMLHttp;
>>     var JSON_Data;
>>
>>
>>     XMLHttp = new XMLHttpRequest();
>>     XMLHttp.open("GET", "
>> http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json",
>> true);
>>     XMLHttp.onreadystatechange = function () {
>>          if(XMLHttp.readyState == 4){}{
>>              JSON_Data = XMLHttp.responseText;
>>              document.getElementById("JSON_Text").value = JSON_Data;
>>          }
>>
>>     };
>>
>> }
>>
>> When I run the code and check the console, it states that it can't find
>> people.json.  I don't know why my webpage can't find people.json  Can you
>> assist me?
>>
>> On Sun, Aug 18, 2024 at 11:42 AM Jean-Paul Rehr <re...@gm...>
>> wrote:
>>
>>> Ted, none of this has to do with eXist parsing JSON.
>>>
>>> The URL
>>> http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json is
>>> requesting a JSON document from eXist, that is, eXist is serving the
>>> document not parsing it.
>>>
>>> Incidentally, one can parse a json document in Xquery with the command:
>>>
>>> fn:json-doc("/db/apps/HTML_Student/people.json")
>>>
>>> JPR
>>>
>>> On Sun, Aug 18, 2024 at 6:26 PM Ted Hickox <meg...@gm...> wrote:
>>>
>>>> If it's possible to parse JSON with the eXist database, why is the
>>>> following code retrieving any data from my JSON document:
>>>>
>>>> JSON Example.xhtml
>>>>
>>>> <html xmlns="http://www.w3.org/1999/xhtml">
>>>> <head>
>>>> <title>First JSON Experiment</title>
>>>> <link rel="stylesheet" type="text/css" href="
>>>> http://localhost:8080/exist/rest/db/apps/HTML_Student/JSON_Example.css"/>
>>>>
>>>> <script language="javascript" src="
>>>> http://localhost:8080/exist/rest/db/apps/HTML_Student/JSON_Example.js"/>
>>>>
>>>> </head>
>>>> <body onload = "Setup()">
>>>> <input type = "text" id = "JSON_Text">
>>>> </input>
>>>> </body>
>>>> </html>
>>>>
>>>> JSON Example.css
>>>>
>>>> #JSON_Text {
>>>>     position: absolute;
>>>>     top: 200px;
>>>>     left: 300px;
>>>>     height: 25px;
>>>>     width: 300px;
>>>> }
>>>>
>>>> JSON Example.js
>>>>
>>>> function Setup() {
>>>>     var XMLHttp;
>>>>     var JSON_Data;
>>>>
>>>>     XMLHttp = new XMLHttpRequest();
>>>>
>>>>     XMLHttp.onreadystatechange = function () {
>>>>          if(XMLHttp.readyState == 4){
>>>>              JSON_Data = JSON.parse(XMLHttp.responseText);
>>>>          }
>>>>     };
>>>>
>>>>     XMLHttp.open("Get", "
>>>> http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json",
>>>> true);
>>>>     XMLHttp.send();
>>>>
>>>>
>>>>     document.getElementById("JSON_Text").value = JSON_Data;
>>>> }
>>>>
>>>> People.json
>>>>
>>>> {
>>>>      "People":{
>>>>           {
>>>>                "name": "Alex",
>>>>                "online": true
>>>>           },
>>>>           {
>>>>                "name": "Billy",
>>>>                "online": false
>>>>           }
>>>>
>>>>      }
>>>> }
>>>> _______________________________________________
>>>> Exist-open mailing list
>>>> Exi...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>>>
>>>

Re: [Exist-open] Can you parse JSON with the eXist database?

[Ted H. <meg...@gm...>]

I fixed a couple of errors and got some new responses.  But I still didn't
get the responses I needed.

Here is my new Javascript document:

function Setup() {

    XMLHttp = new XMLHttpRequest();
    XMLHttp.open("GET", "
http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json", true);
    XMLHttp.onreadystatechange = function () {
         if(XMLHttp.readyState == 4){}{
             JSON_Data = XMLHttp.responseText;
             JSON_Data = JSON.parse(JSON_Data);
             document.getElementById("JSON_Text").value = JSON_Data;
         }

    };
}

And here is my new people.json

{"name":"John", "age":30, "city":"New York"}

And here is the new response from the HTML textbox:

[object Object]

I'm still not sure what I'm doing wrong.

On Mon, Aug 19, 2024 at 10:06 AM Florian Schmitt <
ml-...@fl...> wrote:

> Ted, if your resource is named People.json in eXist (with upper-case "P"),
> you'll need to adapt your URL accordingly. Currently, the URL points to
> people.json, not People.json.
>
> 19.08.2024 07:25:51 Ted Hickox <meg...@gm...>:
>
> My apology for giving you incorrect Javascript code.  I corrected my
> code.  Here it is:
>
> function Setup() {
>     var XMLHttp;
>     var JSON_Data;
>
>
>     XMLHttp = new XMLHttpRequest();
>     XMLHttp.open("GET", "
> http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json",
> true);
>     XMLHttp.onreadystatechange = function () {
>          if(XMLHttp.readyState == 4){}{
>              JSON_Data = XMLHttp.responseText;
>              document.getElementById("JSON_Text").value = JSON_Data;
>          }
>
>     };
>
> }
>
> When I run the code and check the console, it states that it can't find
> people.json.  I don't know why my webpage can't find people.json  Can you
> assist me?
>
> On Sun, Aug 18, 2024 at 11:42 AM Jean-Paul Rehr <re...@gm...> wrote:
>
>> Ted, none of this has to do with eXist parsing JSON.
>>
>> The URL http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json
>> is requesting a JSON document from eXist, that is, eXist is serving the
>> document not parsing it.
>>
>> Incidentally, one can parse a json document in Xquery with the command:
>>
>> fn:json-doc("/db/apps/HTML_Student/people.json")
>>
>> JPR
>>
>> On Sun, Aug 18, 2024 at 6:26 PM Ted Hickox <meg...@gm...> wrote:
>>
>>> If it's possible to parse JSON with the eXist database, why is the
>>> following code retrieving any data from my JSON document:
>>>
>>> JSON Example.xhtml
>>>
>>> <html xmlns="http://www.w3.org/1999/xhtml">
>>> <head>
>>> <title>First JSON Experiment</title>
>>> <link rel="stylesheet" type="text/css" href="
>>> http://localhost:8080/exist/rest/db/apps/HTML_Student/JSON_Example.css"/>
>>>
>>> <script language="javascript" src="
>>> http://localhost:8080/exist/rest/db/apps/HTML_Student/JSON_Example.js"/>
>>>
>>> </head>
>>> <body onload = "Setup()">
>>> <input type = "text" id = "JSON_Text">
>>> </input>
>>> </body>
>>> </html>
>>>
>>> JSON Example.css
>>>
>>> #JSON_Text {
>>>     position: absolute;
>>>     top: 200px;
>>>     left: 300px;
>>>     height: 25px;
>>>     width: 300px;
>>> }
>>>
>>> JSON Example.js
>>>
>>> function Setup() {
>>>     var XMLHttp;
>>>     var JSON_Data;
>>>
>>>     XMLHttp = new XMLHttpRequest();
>>>
>>>     XMLHttp.onreadystatechange = function () {
>>>          if(XMLHttp.readyState == 4){
>>>              JSON_Data = JSON.parse(XMLHttp.responseText);
>>>          }
>>>     };
>>>
>>>     XMLHttp.open("Get", "
>>> http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json",
>>> true);
>>>     XMLHttp.send();
>>>
>>>
>>>     document.getElementById("JSON_Text").value = JSON_Data;
>>> }
>>>
>>> People.json
>>>
>>> {
>>>      "People":{
>>>           {
>>>                "name": "Alex",
>>>                "online": true
>>>           },
>>>           {
>>>                "name": "Billy",
>>>                "online": false
>>>           }
>>>
>>>      }
>>> }
>>> _______________________________________________
>>> Exist-open mailing list
>>> Exi...@li...
>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>>
>>

Re: [Exist-open] Can you parse JSON with the eXist database?

[Florian S. <ml-...@fl...>]

Ted, if your resource is named People.json in eXist (with upper-case "P"), you'll need to adapt your URL accordingly. Currently, the URL points to people.json, not People.json.

19.08.2024 07:25:51 Ted Hickox <meg...@gm...>:

> My apology for giving you incorrect Javascript code.  I corrected my code.  Here it is:
> 
> function Setup() {
>     var XMLHttp;
>     var JSON_Data;
>    
>    
>     XMLHttp = new XMLHttpRequest();
>     XMLHttp.open("GET", "http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json", true);
>     XMLHttp.onreadystatechange = function () {
>          if(XMLHttp.readyState == 4){}{
>              JSON_Data = XMLHttp.responseText;
>              document.getElementById("JSON_Text").value = JSON_Data;
>          }
>        
>     };
>    
> }    
> 
> When I run the code and check the console, it states that it can't find people.json.  I don't know why my webpage can't find people.json  Can you assist me?
> 
> On Sun, Aug 18, 2024 at 11:42 AM Jean-Paul Rehr <re...@gm...> wrote:
>> Ted, none of this has to do with eXist parsing JSON. 
>> 
>> The URL http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json is requesting a JSON document from eXist, that is, eXist is serving the document not parsing it.
>> 
>> Incidentally, one can parse a json document in Xquery with the command:
>> 
>> fn:json-doc("/db/apps/HTML_Student/people.json")
>> 
>> JPR
>> 
>> On Sun, Aug 18, 2024 at 6:26 PM Ted Hickox <meg...@gm...> wrote:
>>> If it's possible to parse JSON with the eXist database, why is the following code retrieving any data from my JSON document:
>>> 
>>> JSON Example.xhtml
>>> 
>>> <html xmlns="http://www.w3.org/1999/xhtml">
>>> <head>
>>> <title>First JSON Experiment</title>
>>> <link rel="stylesheet" type="text/css" href="http://localhost:8080/exist/rest/db/apps/HTML_Student/JSON_Example.css"/>
>>> <script language="javascript" src="http://localhost:8080/exist/rest/db/apps/HTML_Student/JSON_Example.js"/>
>>> </head>
>>> <body onload = "Setup()">
>>> <input type = "text" id = "JSON_Text">
>>> </input>
>>> </body>
>>> </html>
>>> 
>>> JSON Example.css
>>> 
>>> #JSON_Text {
>>>     position: absolute;
>>>     top: 200px;
>>>     left: 300px;
>>>     height: 25px;
>>>     width: 300px;
>>> }
>>> 
>>> JSON Example.js
>>> 
>>> function Setup() {
>>>     var XMLHttp;
>>>     var JSON_Data;
>>>    
>>>     XMLHttp = new XMLHttpRequest();
>>>    
>>>     XMLHttp.onreadystatechange = function () {
>>>          if(XMLHttp.readyState == 4){
>>>              JSON_Data = JSON.parse(XMLHttp.responseText);
>>>          }  
>>>     };
>>>    
>>>     XMLHttp.open("Get", "http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json", true);
>>>     XMLHttp.send();
>>>    
>>>    
>>>     document.getElementById("JSON_Text").value = JSON_Data;
>>> }
>>> 
>>> People.json
>>> 
>>> {
>>>      "People":{
>>>           {
>>>                "name": "Alex",
>>>                "online": true
>>>           },
>>>           {
>>>                "name": "Billy",
>>>                "online": false
>>>           }
>>>      
>>>      }
>>> }
>>> _______________________________________________
>>> Exist-open mailing list
>>> Exi...@li...
>>> https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] Can you parse JSON with the eXist database?

[Joern T. <joe...@gm...>]

Ted,

Make sure that your json is returned by your app by testing the  Get
request in the Browser.  Most likely your controller.xql doesn't return
that file and your need a rule for that.

hth,

Joern

Ted Hickox <meg...@gm...> schrieb am Mo., 19. Aug. 2024, 07:26:

> My apology for giving you incorrect Javascript code.  I corrected my
> code.  Here it is:
>
> function Setup() {
>     var XMLHttp;
>     var JSON_Data;
>
>
>     XMLHttp = new XMLHttpRequest();
>     XMLHttp.open("GET", "
> http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json", true);
>     XMLHttp.onreadystatechange = function () {
>          if(XMLHttp.readyState == 4){}{
>              JSON_Data = XMLHttp.responseText;
>              document.getElementById("JSON_Text").value = JSON_Data;
>          }
>
>     };
>
> }
>
> When I run the code and check the console, it states that it can't find
> people.json.  I don't know why my webpage can't find people.json  Can you
> assist me?
>
> On Sun, Aug 18, 2024 at 11:42 AM Jean-Paul Rehr <re...@gm...> wrote:
>
>> Ted, none of this has to do with eXist parsing JSON.
>>
>> The URL http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json
>> is requesting a JSON document from eXist, that is, eXist is serving the
>> document not parsing it.
>>
>> Incidentally, one can parse a json document in Xquery with the command:
>>
>> fn:json-doc("/db/apps/HTML_Student/people.json")
>>
>> JPR
>>
>> On Sun, Aug 18, 2024 at 6:26 PM Ted Hickox <meg...@gm...> wrote:
>>
>>> If it's possible to parse JSON with the eXist database, why is the
>>> following code retrieving any data from my JSON document:
>>>
>>> JSON Example.xhtml
>>>
>>> <html xmlns="http://www.w3.org/1999/xhtml">
>>> <head>
>>> <title>First JSON Experiment</title>
>>> <link rel="stylesheet" type="text/css" href="
>>> http://localhost:8080/exist/rest/db/apps/HTML_Student/JSON_Example.css
>>> "/>
>>> <script language="javascript" src="
>>> http://localhost:8080/exist/rest/db/apps/HTML_Student/JSON_Example.js"/>
>>> </head>
>>> <body onload = "Setup()">
>>> <input type = "text" id = "JSON_Text">
>>> </input>
>>> </body>
>>> </html>
>>>
>>> JSON Example.css
>>>
>>> #JSON_Text {
>>>     position: absolute;
>>>     top: 200px;
>>>     left: 300px;
>>>     height: 25px;
>>>     width: 300px;
>>> }
>>>
>>> JSON Example.js
>>>
>>> function Setup() {
>>>     var XMLHttp;
>>>     var JSON_Data;
>>>
>>>     XMLHttp = new XMLHttpRequest();
>>>
>>>     XMLHttp.onreadystatechange = function () {
>>>          if(XMLHttp.readyState == 4){
>>>              JSON_Data = JSON.parse(XMLHttp.responseText);
>>>          }
>>>     };
>>>
>>>     XMLHttp.open("Get", "
>>> http://localhost:8080/exist/rest/db/apps/HTML_Student/people.json",
>>> true);
>>>     XMLHttp.send();
>>>
>>>
>>>     document.getElementById("JSON_Text").value = JSON_Data;
>>> }
>>>
>>> People.json
>>>
>>> {
>>>      "People":{
>>>           {
>>>                "name": "Alex",
>>>                "online": true
>>>           },
>>>           {
>>>                "name": "Billy",
>>>                "online": false
>>>           }
>>>
>>>      }
>>> }
>>> _______________________________________________
>>> Exist-open mailing list
>>> Exi...@li...
>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>>
>> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>