[Exist-open] storeResouce is too slow with remote interface

[Vladimir P. <pra...@gm...>]

eXistDB 6.2.0

Saving to a locally running database via the remote interface is several orders of magnitude slower compared to using an embedded database. I can't figure out where the problem lies, or how to store the data more efficiently.

I need to store about 100,000 records. When saving via the remote interface, the speed is around 100 records per second. However, when using the embedded database, the speed is about 9,000 records per second.

What am I doing wrong, or how can I improve this?

Below is a code example.



remote uri: xmldb:exist://localhost:8080/exist/xmlrpc
embeded uri: xmldb:exist://

In both cases the db is running on the same machine. db configuration file is attached.

private void createRandomTestData(String prefix, int records) throws Exception {
    final Random random = new Random();
    final HashMap<String, String> data = new HashMap<>();

    for (int i = 0; i < records; i++) {
        double latitude = MIN_LAT + (MAX_LAT - MIN_LAT) * random.nextDouble();
        double longitude = MIN_LON + (MAX_LON - MIN_LON) * random.nextDouble();

        String randomId = UUID.randomUUID().toString();
        String randomName = generateRandomString(random, 30);
        String randomValue = generateRandomString(random, 10);

        String resourceId = prefix + "/" + randomId;
        String xml = String.format(
                """
                        <entity>
                            <name>%s</name>
                            <value>%s</value>
                            <pos>%.8f %.8f</pos>
                        </entity>""",
                randomName, randomValue, latitude, longitude
        );
        data.put(resourceId, xml);
    }
    xmlDbService.saveEntity(data);
}

public void saveEntity(@NonNull HashMap<String, String> resources) throws Exception {
    var sortedResources = new TreeMap<>(resources);

    Collection col = null;
    String currentCollectionUri = null;
    long start = System.currentTimeMillis();

    int i = 0;
    int c = 0;
    try {
        for (var entry : sortedResources.entrySet()) {
            String resourceId = entry.getKey();
            String xmlData = entry.getValue();

            var param = splitResourceId(resourceId);
            String collectionUri = param[0];
            String resourceName = param[1];

            if (col == null || !collectionUri.equals(currentCollectionUri)) {
                if (col != null) {
                    col.close();
                }
                col = getOrCreateCollection(collectionUri);
                col.setProperty("indent", "no");
                currentCollectionUri = collectionUri;
            }

            i++;
            if (i % 1000 == 0) {
                long executionTime = System.currentTimeMillis() - start;
                log.info("Inserted: {}, rate {} / sec", i, c * 1000L / executionTime);
                start = System.currentTimeMillis();
                c = 0;
            }
            XMLResource res = (XMLResource) col.createResource(resourceName, XMLResource.RESOURCE_TYPE);
            res.setContent(xmlData);
            col.storeResource(res);
            c++;
            log.trace("Resource saved: {} in collection: {}", resourceName, currentCollectionUri);
        }
    } finally {
        if (col != null) {
            col.close();
        }
    }
}

<dependency>
    <groupId>org.exist-db</groupId>
    <artifactId>exist-core</artifactId>
    <version>6.2.0</version>
</dependency>

<dependency>
    <groupId>net.sf.xmldb-org</groupId>
    <artifactId>xmldb-api</artifactId>
    <version>1.7.0</version>
</dependency>
Thank you for your help and/or advice.

V.