[Exist-open] XUpdate error and performance issue

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Hello,

I've been using the eXist database for more than a week now, and i
experience some weird issues with the XUpdate over SOAP implementation.

First of all, i use gSOAP v2.6 for my tests. The hardware is a PIV
2.8GHz, with 1GB RAM, running Linux RedHat 9.0.
I use the latest eXist snapshot (eXist-snapshot-20040528.jar), with JRE
version 1.4.2_03.

The XQuery implementation works fine. From the query.wsdl file (located
in src/org/exist/soap) were generated the right header file and C
functions, with gSOAP. I then coded a small XQuery client, to submit
queries and fetch results, which works very well.

I added to the database (with the Java Client) a small-size XML file,
with a couple of users in it :

-------------------------------------------------------------------------
<?xml version=3D"1.0" encoding=3D"UTF-8"?>
<directory xmlns:xsi=3D"http://www.w3.org/2001/XMLSchema-instance">
 <user xsi:type=3D"db:UserData">
  <firstname xsi:type=3D"xsd:string">Edward</firstname>
  <lastname xsi:type=3D"xsd:string">VanHelgen</lastname>
  <phonenumber xsi:type=3D"xsd:string">+33 (0) 012-5-3</phonenumber>
  <address xsi:type=3D"db:AddressType">
   <street xsi:type=3D"xsd:string">1, Downing Street</street>
   <city xsi:type=3D"xsd:string">Amsterdam</city>
   <zip xsi:type=3D"xsd:decimal">65000</zip>
   <country xsi:type=3D"xsd:string">Netherlands</country>
  </address>
 </user>
 <user>
  ...
 </user>
 <user>
  ...
 </user>
...
</directory>
-------------------------------------------------------------------------

As with the XQuery client, i coded an XUpdate Client from the admin.wsdl
and gSOAP. The two available updates are :
- Two users are added to the database (adduser.xml);
- These two users are removed (deluser.xml).
To parse and execute one of the file, i just launch my xupdate program,
with the name of the file as parameter.

The adduser.xml file content is:

-------------------------------------------------------------------------
<?xml version=3D"1.0" encoding=3D"UTF-8"?>
<xu:modifications version=3D"1.0"
		  xmlns:xsi=3D"http://www.w3.org/2001/XMLSchema-instance"
		  xmlns:xu=3D"http://www.xmldb.org/xupdate">
 <xu:insert-after
	select=3D"document('/db/soap/users.xml')
		//user[contains(firstname,'Edward')
		       and contains(lastname,'VanHelgen')]">
  <user xsi:type=3D"db:UserData">
   <firstname xsi:type=3D"xsd:string">Florian-1</firstname>
   <lastname xsi:type=3D"xsd:string">Ponroy</lastname>
   <phonenumber xsi:type=3D"xsd:string">+33 (0) 141303100</phonenumber>
   <address xsi:type=3D"db:AddressType">
    <street xsi:type=3D"xsd:string">Boulevard de Valmy</street>
    <city xsi:type=3D"xsd:string">Colombes</city>
    <zip xsi:type=3D"xsd:decimal">92000</zip>
    <country xsi:type=3D"xsd:string">France</country>
   </address>
  </user>
  <user xsi:type=3D"db:UserData">
   <firstname xsi:type=3D"xsd:string">Florian-2</firstname>
   <lastname xsi:type=3D"xsd:string">Ponroy</lastname>
   <phonenumber xsi:type=3D"xsd:string">+33 (0) 141303100</phonenumber>
   <address xsi:type=3D"db:AddressType">
    <street xsi:type=3D"xsd:string">Boulevard de Valmy</street>
    <city xsi:type=3D"xsd:string">Colombes</city>
    <zip xsi:type=3D"xsd:decimal">92000</zip>
    <country xsi:type=3D"xsd:string">France</country>
   </address>
  </user>
 </xu:insert-after>
</xu:modifications>
-------------------------------------------------------------------------

The deluser.xml file content is:

-------------------------------------------------------------------------
<?xml version=3D"1.0" encoding=3D"UTF-8"?>
<xu:modifications version=3D"1.0" xmlns:xu=3D"http://www.xmldb.org/xupdat=
e">
 <xu:remove
	select=3D"document('/db/soap/users.xml')
		//user[contains(lastname, 'Ponroy')]" />
</xu:modifications>
-------------------------------------------------------------------------

If i execute the two updates in a row, it works fine:

-------------------------------------------------------------------------
$ ./xupdate adduser.xml
Connection established. Session ID: 20973025.
XUpdate request successfuly loaded.
Session ID 20973025: Disconnected.

$ ./xupdate deluser.xml
Connection established. Session ID: 18740246.
XUpdate request successfuly loaded.
Session ID 18740246: Disconnected.
-------------------------------------------------------------------------

Then i wanted to stress the database a little bit, and execute an
endless loop, adding and deleting users:

-------------------------------------------------------------------------
while [ 1 ]; do
	./xupdate adduser.xml
	./xupdate deluser.xml
done
-------------------------------------------------------------------------

It has worked fine for 30 seconds, then threw a SOAP-FAULT:

-------------------------------------------------------------------------
SOAP FAULT: SOAP-ENV:Server.userException
"java.lang.NullPointerException"
-------------------------------------------------------------------------

From this point, the users.xml file is totally corrupted, there is no
way to recover it, even to delete it. I got to erase the collection it
is stored in.
Ok, i create a new collection, and put the users.xml file in it. Thing
is that XQuery results are abnormal:

-------------------------------------------------------------------------
$ ./xquery
Connection established. Session ID: 31935875.
Please enter your query: //user
Number of results to display: 1

Collection(s): /db/soap (users.xml [10 hits])

10, Downing Street

Session ID 31935875: Disconnected.
-------------------------------------------------------------------------

... whereas it should have printed the whole first user entry, as shown
below.
To make it work again, i have to restart the database, erase the
users.xml file, and add it to the collection again:

-------------------------------------------------------------------------
$ ./xquery
Connection established. Session ID: 8818963.
Please enter your query: //user
Number of results to display: 1

Collection(s): /db/soap (users.xml [10 hits])

<user xmlns:xsi=3D"http://www.w3.org/2001/XMLSchema-instance"
xsi:type=3D"db:UserData">
    <firstname xsi:type=3D"xsd:string">Edward</firstname>
    <lastname xsi:type=3D"xsd:string">VanHelgen</lastname>
    <phonenumber xsi:type=3D"xsd:string">+33 (0) 012-5-3</phonenumber>
    <address xsi:type=3D"db:AddressType">
        <street xsi:type=3D"xsd:string">1, Downing Street</street>
        <city xsi:type=3D"xsd:string">Amsterdam</city>
        <zip xsi:type=3D"xsd:decimal">65000</zip>
        <country xsi:type=3D"xsd:string">Netherlands</country>
    </address>
</user>

Session ID 8818963: Disconnected.
-------------------------------------------------------------------------

Another weird performance issue (unless it comes from my side). I added
about 1500 entries to the users file, then deleted them all with the
last simple deluser.xml XUpdate request.
The adding operation was pretty fast (about 6 seconds), but the deleting
operation took more than 37 minutes!
Here is the database DEBUG output:

-------------------------------------------------------------------------
07 Jun 2004 11:09:59,320 [SocketListener-4] DEBUG (NativeBroker.java
[reindex]:1079) - reindex took 3605ms.
07 Jun 2004 11:10:02,922 [SocketListener-4] DEBUG (NativeBroker.java
[reindex]:1079) - reindex took 3517ms.
07 Jun 2004 11:10:06,421 [SocketListener-4] DEBUG (NativeBroker.java
[reindex]:1079) - reindex took 3426ms.
07 Jun 2004 11:10:09,863 [SocketListener-4] DEBUG (NativeBroker.java
[reindex]:1079) - reindex took 3377ms.
07 Jun 2004 11:10:13,329 [SocketListener-4] DEBUG (NativeBroker.java
[reindex]:1079) - reindex took 3402ms.
07 Jun 2004 11:10:16,888 [SocketListener-4] DEBUG (NativeBroker.java
[reindex]:1079) - reindex took 3499ms.
...
07 Jun 2004 11:47:05,443 [SocketListener-4] DEBUG (NativeBroker.java
[reindex]:1079) - reindex took 17ms.
07 Jun 2004 11:47:05,462 [SocketListener-4] DEBUG (NativeBroker.java
[reindex]:1079) - reindex took 16ms.
07 Jun 2004 11:47:05,479 [SocketListener-4] DEBUG (NativeBroker.java
[reindex]:1079) - reindex took 14ms.
07 Jun 2004 11:47:05,494 [SocketListener-4] DEBUG (NativeBroker.java
[reindex]:1079) - reindex took 13ms.
07 Jun 2004 11:47:05,508 [SocketListener-4] DEBUG (NativeBroker.java
[reindex]:1079) - reindex took 12ms.
07 Jun 2004 11:47:05,509 [SocketListener-4] DEBUG (BrokerPool.java
[sync]:354) - syncing buffers to disk
-------------------------------------------------------------------------

Although the deleting operation seemed to have worked, the users.xml
file output is not correct, since there is only one user left, with
garbage at the end of the file, and the </directory> element which has
disappeared.

To conclude, my two main questions are:
- What about this =AB java.lang.NullPointerException =BB error when some
  users are inserted and deleted in a row, in an endless loop?
- Should it take so much time to perform an XUpdate delete request?

Thanks a lot for your time and your help,

Florian Ponroy.

[Exist-open] XUpdate error and performance issue

eXist-db is a feature rich Open Source native XML database

[Exist-open] XUpdate error and performance issue