exist-development Mailing List for eXist-db (Page 86)

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

> That will work too.  Kind of late for Wolfgang though.

No, that's fine.

Wolfgang

That will work too.  Kind of late for Wolfgang though.

-----Original Message-----
From: Adam Retter [mailto:ad...@ex...] 
Sent: Tuesday, August 31, 2010 2:27 PM
To: Jason Smith
Cc: Wolfgang Meier; Dmitriy Shabanov; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; loren
Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.

Wow steady on there! Thats a bit early! how about 3pm UTC?

On 31 August 2010 21:25, Jason Smith <js...@in...> wrote:
> I believe that is 8am my time.  I think I can swing that.
>
> -----Original Message-----
> From: Adam Retter [mailto:ad...@ex...]
> Sent: Tuesday, August 31, 2010 2:17 PM
> To: Jason Smith
> Cc: Wolfgang Meier; Dmitriy Shabanov; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; loren
> Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.
>
> How about 2pm UTC, if I have this correct (???) -
>
> 3pm my time (BST)
> 4pm Wolfgangs time (CEST)
> 9am your time???
>
>
> On 31 August 2010 20:57, Jason Smith <js...@in...> wrote:
>> I'm in the US, MST (Denver, CO).  Be kind when picking times, please.  :-)
>>
>> -----Original Message-----
>> From: Wolfgang Meier [mailto:wol...@ex...]
>> Sent: Tuesday, August 31, 2010 9:44 AM
>> To: Dmitriy Shabanov
>> Cc: Adam Retter; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; Jason Smith; loren
>> Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.
>>
>>>> Skype teleconference would be good :-)
>>>
>>> When? (ready to join)
>>
>> Ok, I think a teleconference would make it easier for me to explain my
>> roadmap for eXist and for Jason to describe his ideas in more detail.
>> I would have time for a talk tomorrow, but would prefer Thursday if
>> possible. Anytime between 10am and 9pm CEST.
>>
>> Wolfgang
>>
>
>
>
> --
> Adam Retter
>
> eXist Developer
> { United Kingdom }
> ad...@ex...
> irc://irc.freenode.net/existdb
>

-- 
Adam Retter

eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb

Wow steady on there! Thats a bit early! how about 3pm UTC?

On 31 August 2010 21:25, Jason Smith <js...@in...> wrote:
> I believe that is 8am my time.  I think I can swing that.
>
> -----Original Message-----
> From: Adam Retter [mailto:ad...@ex...]
> Sent: Tuesday, August 31, 2010 2:17 PM
> To: Jason Smith
> Cc: Wolfgang Meier; Dmitriy Shabanov; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; loren
> Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.
>
> How about 2pm UTC, if I have this correct (???) -
>
> 3pm my time (BST)
> 4pm Wolfgangs time (CEST)
> 9am your time???
>
>
> On 31 August 2010 20:57, Jason Smith <js...@in...> wrote:
>> I'm in the US, MST (Denver, CO).  Be kind when picking times, please.  :-)
>>
>> -----Original Message-----
>> From: Wolfgang Meier [mailto:wol...@ex...]
>> Sent: Tuesday, August 31, 2010 9:44 AM
>> To: Dmitriy Shabanov
>> Cc: Adam Retter; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; Jason Smith; loren
>> Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.
>>
>>>> Skype teleconference would be good :-)
>>>
>>> When? (ready to join)
>>
>> Ok, I think a teleconference would make it easier for me to explain my
>> roadmap for eXist and for Jason to describe his ideas in more detail.
>> I would have time for a talk tomorrow, but would prefer Thursday if
>> possible. Anytime between 10am and 9pm CEST.
>>
>> Wolfgang
>>
>
>
>
> --
> Adam Retter
>
> eXist Developer
> { United Kingdom }
> ad...@ex...
> irc://irc.freenode.net/existdb
>

-- 
Adam Retter

eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb

I believe that is 8am my time.  I think I can swing that.

-----Original Message-----
From: Adam Retter [mailto:ad...@ex...] 
Sent: Tuesday, August 31, 2010 2:17 PM
To: Jason Smith
Cc: Wolfgang Meier; Dmitriy Shabanov; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; loren
Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.

How about 2pm UTC, if I have this correct (???) -

3pm my time (BST)
4pm Wolfgangs time (CEST)
9am your time???

On 31 August 2010 20:57, Jason Smith <js...@in...> wrote:
> I'm in the US, MST (Denver, CO).  Be kind when picking times, please.  :-)
>
> -----Original Message-----
> From: Wolfgang Meier [mailto:wol...@ex...]
> Sent: Tuesday, August 31, 2010 9:44 AM
> To: Dmitriy Shabanov
> Cc: Adam Retter; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; Jason Smith; loren
> Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.
>
>>> Skype teleconference would be good :-)
>>
>> When? (ready to join)
>
> Ok, I think a teleconference would make it easier for me to explain my
> roadmap for eXist and for Jason to describe his ideas in more detail.
> I would have time for a talk tomorrow, but would prefer Thursday if
> possible. Anytime between 10am and 9pm CEST.
>
> Wolfgang
>

-- 
Adam Retter

eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb

How about 2pm UTC, if I have this correct (???) -

3pm my time (BST)
4pm Wolfgangs time (CEST)
9am your time???

On 31 August 2010 20:57, Jason Smith <js...@in...> wrote:
> I'm in the US, MST (Denver, CO).  Be kind when picking times, please.  :-)
>
> -----Original Message-----
> From: Wolfgang Meier [mailto:wol...@ex...]
> Sent: Tuesday, August 31, 2010 9:44 AM
> To: Dmitriy Shabanov
> Cc: Adam Retter; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; Jason Smith; loren
> Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.
>
>>> Skype teleconference would be good :-)
>>
>> When? (ready to join)
>
> Ok, I think a teleconference would make it easier for me to explain my
> roadmap for eXist and for Jason to describe his ideas in more detail.
> I would have time for a talk tomorrow, but would prefer Thursday if
> possible. Anytime between 10am and 9pm CEST.
>
> Wolfgang
>

-- 
Adam Retter

eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb

I'm in the US, MST (Denver, CO).  Be kind when picking times, please.  :-)

-----Original Message-----
From: Wolfgang Meier [mailto:wol...@ex...] 
Sent: Tuesday, August 31, 2010 9:44 AM
To: Dmitriy Shabanov
Cc: Adam Retter; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; Jason Smith; loren
Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.

>> Skype teleconference would be good :-)
>
> When? (ready to join)

Ok, I think a teleconference would make it easier for me to explain my
roadmap for eXist and for Jason to describe his ideas in more detail.
I would have time for a talk tomorrow, but would prefer Thursday if
possible. Anytime between 10am and 9pm CEST.

Wolfgang

On Tue, 2010-08-31 at 09:19 -0700, Jason Smith wrote:
> I guess I will have to install Skype.  :-)  Thursday is fine.  No hurries.
> 
> -----Original Message-----
> From: Adam Retter [mailto:ad...@ex...] 
> Sent: Tuesday, August 31, 2010 10:17 AM
> To: Wolfgang Meier
> Cc: Dmitriy Shabanov; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; Jason Smith; loren
> Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.
> 
> On 31 August 2010 16:43, Wolfgang Meier <wol...@ex...> wrote:
> >>> Skype teleconference would be good :-)
> >>
> >> When? (ready to join)
> >
> > Ok, I think a teleconference would make it easier for me to explain my
> > roadmap for eXist and for Jason to describe his ideas in more detail.
> > I would have time for a talk tomorrow, but would prefer Thursday if
> > possible. Anytime between 10am and 9pm CEST.
> 
> Thursday is fine for me also, but will have to be before 5pm BST.

Thursday, almost any time.

-- 
Cheers,

Dmitriy Shabanov

On 31 August 2010 16:43, Wolfgang Meier <wol...@ex...> wrote:
>>> Skype teleconference would be good :-)
>>
>> When? (ready to join)
>
> Ok, I think a teleconference would make it easier for me to explain my
> roadmap for eXist and for Jason to describe his ideas in more detail.
> I would have time for a talk tomorrow, but would prefer Thursday if
> possible. Anytime between 10am and 9pm CEST.

Thursday is fine for me also, but will have to be before 5pm BST.

> Wolfgang
>

-- 
Adam Retter

eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb

I guess I will have to install Skype.  :-)  Thursday is fine.  No hurries.

-----Original Message-----
From: Adam Retter [mailto:ad...@ex...] 
Sent: Tuesday, August 31, 2010 10:17 AM
To: Wolfgang Meier
Cc: Dmitriy Shabanov; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; Jason Smith; loren
Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.

On 31 August 2010 16:43, Wolfgang Meier <wol...@ex...> wrote:
>>> Skype teleconference would be good :-)
>>
>> When? (ready to join)
>
> Ok, I think a teleconference would make it easier for me to explain my
> roadmap for eXist and for Jason to describe his ideas in more detail.
> I would have time for a talk tomorrow, but would prefer Thursday if
> possible. Anytime between 10am and 9pm CEST.

Thursday is fine for me also, but will have to be before 5pm BST.

> Wolfgang
>

-- 
Adam Retter

eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb

I've noticed that when I try to display the Function Documentation web page, when you do a search or browse for a
specific set of functions in a namespace (eg. Util), it takes forever to display the page.  And it seems to lock up my
whole machine, not just Firefox when it's rendering the page.

It never used to do this....

Any ideas what is causing this?  I'm running the latest Firefox (3.6.8) on a 64-bit Ubuntu, fast quad processor box with
gobs of memory.....it should fly like the wind...but instead my machine will become non-responsive for 20-30 seconds.
Running an SVN build from late July (since the security stuff broke everything, and I haven't had a chance to test the
latest stuff of a few days ago).

Anyone have any idea what might be causing this massive slowdown?

Thanks.

-- 
Andrzej Taramina
Chaeron Corporation: Enterprise System Solutions
http://www.chaeron.com

>> Skype teleconference would be good :-)
>
> When? (ready to join)

Ok, I think a teleconference would make it easier for me to explain my
roadmap for eXist and for Jason to describe his ideas in more detail.
I would have time for a talk tomorrow, but would prefer Thursday if
possible. Anytime between 10am and 9pm CEST.

Wolfgang

On Tue, 2010-08-31 at 13:42 +0100, Adam Retter wrote:
> > As I said before, I welcome any exploration in this area. As James
> > just suggested, we may want to have a skype telecon on this to
> discuss
> > the possibilities and dangers.
> 
> Skype teleconference would be good :-)

When? (ready to join)

-- 
Cheers,

Dmitriy Shabanov

> Otherwise, eXist just avoids locking multiple collections at once
> wherever possible as it is known to be an expensive operation and
> limits concurrency.

We have discussed several times replacing eXist-db's current
collection mechanism with a virtualised implementation where
Collections are just another number in the system. This was discussed
for the purposes of performance when large collections are involved.

Would this simplify the overal problem domain? If so, perhaps this
work should be undertaken before a redesign of the locking system?

>> The second replacement I've come up with allows two read queries to run
>> simultaneously, even when they target the same collection, and when multiple collections
>> are used simultaneously.
>
> As I said before, I welcome any exploration in this area. As James
> just suggested, we may want to have a skype telecon on this to discuss
> the possibilities and dangers.

Skype teleconference would be good :-)

-- 
Adam Retter

eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb

> Again, I would not expect a final solution to be implemented the way I have done it.  I just want to be able to show some possibilities.

I am really excited to see such enthusiasm and knowledge of the
eXist-db internals and suggestions for improvements that could be
made.

> I am not expecting this conversation to take place quickly.  This is one of those "long-haul" things - perhaps over years, and several major releases.  I'm not in a hurry to do the wrong thing quickly!!!
>

Lets not loose this though, I think that any performance and
scalability enhancements are great.

And actually there is a flip side here that no one has mentioned:
Whilst everyone has said that you should have optimised queries and
appropriate indexes, which is invariably true for production systems,
what about new users and developers who come to eXist-db?
As a new user of eXist-db you dont necessarily understand about
optimising your queries or even how to create the correct indexes. Not
all eXist-db users are software developers! It would be great if these
new users saw great performance from the start, even if they havent
set up indexes and their queries are doing full scans of the dbx files
:-)

> -----Original Message-----
> From: Wolfgang Meier [mailto:wol...@ex...]
> Sent: Monday, August 30, 2010 10:38 AM
> To: Jason Smith
> Cc: Dmitriy Shabanov; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour
> Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.
>
>> I understand.  The solution we have in place right now is similar to the solution you
>> mentioned, but we put it in place a while ago.  Augmenting the locking with a singleton
>> lock does, indeed, work.
>
> Internally, eXist does need the singleton lock in rare cases only,
> mainly when reading or storing the collection configuration document
> for a collection, or when locking documents for an XQuery update
> expression.
>
> Otherwise, eXist just avoids locking multiple collections at once
> wherever possible as it is known to be an expensive operation and
> limits concurrency.
>
>> The second replacement I've come up with allows two read queries to run
>> simultaneously, even when they target the same collection, and when multiple collections
>> are used simultaneously.
>
> As I said before, I welcome any exploration in this area. As James
> just suggested, we may want to have a skype telecon on this to discuss
> the possibilities and dangers.
>
> Finally, just as a note to other users who code against the internal
> API: Dannes' WebDAV reimplementation shows some clean examples of how
> to use internals:
>
> http://exist.svn.sourceforge.net/viewvc/exist/branches/dizzzz/trunk-webdav-upgrade/extensions/webdav/src/org/exist/webdav/
>
> Wolfgang
> ------------------------------------------------------------------------------
> This SF.net Dev2Dev email is sponsored by:
>
> Show off your parallel programming skills.
> Enter the Intel(R) Threading Challenge 2010.
> http://p.sf.net/sfu/intel-thread-sfd
> _______________________________________________
> Exist-development mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-development
>

-- 
Adam Retter

eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb

> I do think to code separate methods with security checks on low level,
> so it going to be only one place check. That will make all interface
> behavior same.

Perhaps we should use a layered approach through composition. Standard
is without and then the additional layer has the security checks all
in one place.

> --
> Cheers,
>
> Dmitriy Shabanov
>
> ------------------------------------------------------------------------------
> Sell apps to millions through the Intel(R) Atom(Tm) Developer Program
> Be part of this innovative community and reach millions of netbook users
> worldwide. Take advantage of special opportunities to increase revenue and
> speed time-to-market. Join now, and jumpstart your future.
> http://p.sf.net/sfu/intel-atom-d2d
> _______________________________________________
> Exist-commits mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-commits
>
>

-- 
Adam Retter

eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb

Wolfgang, if you are interested, I could merge my code into a branch of eXist (not intended for human consumption), so that we could talk using concrete examples. In particular, about improving the locking mechanism so that it actually works (mine is more of a proof of concept, not a final solution).  It looks to me that a number of the concurrency limitations in eXist originate from having to work around the limitations of the current locking mechanism.

Again, I would not expect a final solution to be implemented the way I have done it.  I just want to be able to show some possibilities.

I am not expecting this conversation to take place quickly.  This is one of those "long-haul" things - perhaps over years, and several major releases.  I'm not in a hurry to do the wrong thing quickly!!!

-----Original Message-----
From: Wolfgang Meier [mailto:wol...@ex...] 
Sent: Monday, August 30, 2010 10:38 AM
To: Jason Smith
Cc: Dmitriy Shabanov; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour
Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.

> I understand.  The solution we have in place right now is similar to the solution you
> mentioned, but we put it in place a while ago.  Augmenting the locking with a singleton
> lock does, indeed, work.

Internally, eXist does need the singleton lock in rare cases only,
mainly when reading or storing the collection configuration document
for a collection, or when locking documents for an XQuery update
expression.

Otherwise, eXist just avoids locking multiple collections at once
wherever possible as it is known to be an expensive operation and
limits concurrency.

> The second replacement I've come up with allows two read queries to run
> simultaneously, even when they target the same collection, and when multiple collections
> are used simultaneously.

As I said before, I welcome any exploration in this area. As James
just suggested, we may want to have a skype telecon on this to discuss
the possibilities and dangers.

Finally, just as a note to other users who code against the internal
API: Dannes' WebDAV reimplementation shows some clean examples of how
to use internals:

http://exist.svn.sourceforge.net/viewvc/exist/branches/dizzzz/trunk-webdav-upgrade/extensions/webdav/src/org/exist/webdav/

Wolfgang

> I understand.  The solution we have in place right now is similar to the solution you
> mentioned, but we put it in place a while ago.  Augmenting the locking with a singleton
> lock does, indeed, work.

Internally, eXist does need the singleton lock in rare cases only,
mainly when reading or storing the collection configuration document
for a collection, or when locking documents for an XQuery update
expression.

Otherwise, eXist just avoids locking multiple collections at once
wherever possible as it is known to be an expensive operation and
limits concurrency.

> The second replacement I've come up with allows two read queries to run
> simultaneously, even when they target the same collection, and when multiple collections
> are used simultaneously.

As I said before, I welcome any exploration in this area. As James
just suggested, we may want to have a skype telecon on this to discuss
the possibilities and dangers.

Finally, just as a note to other users who code against the internal
API: Dannes' WebDAV reimplementation shows some clean examples of how
to use internals:

http://exist.svn.sourceforge.net/viewvc/exist/branches/dizzzz/trunk-webdav-upgrade/extensions/webdav/src/org/exist/webdav/

Wolfgang

On 30 August 2010 18:19, Jason Smith <js...@in...> wrote:
> I understand.  The solution we have in place right now is similar to the solution you mentioned, but we put it in place a while ago.  Augmenting the locking with a singleton lock does, indeed, work.
>
> That was my first replacement of the locking mechanism.
>
> The second replacement I've come up with allows two read queries to run simultaneously, even when they target the same collection, and when multiple collections are used simultaneously.  And it enforces atomic read and write operations, something the current locking does not appear to do as well (I am still researching this one).
>
> It isn't as concurrent as I would like (due to dom.dbx locking), and we still need to put it through its paces to test for stability.

Guys,

I have followed the thread and (as u would expect) agree mostly with
Wolfgangs thoughts and defo welcome any and all contributions but we
have to be careful to revisit all original assumptions. I am not
saying its a bad goal to try and make non index-assisted,
non-optimized access to the DOM more performant, but in a database
indexes are so important I think we sometimes ignore corner cases,
though I would be interested to see what knock on impact to general
performance with this modification we have to be careful with respect
to stability of the codebase.

Can I suggest a irc chat or skype call ... email sucks for resolving
deeper levels of tech detail.

James Fuller

> Anyway, as I tried to explain a few weeks back, the locking is not
> fail-safe. You can produce deadlocks if you do not follow certain
> conventions (which are hard to know since they are not documented). In
> particular, acquiring a lock across multiple collections can cause a
> deadlock.

> In this case, eXist-internal code acquires a lock on the global
> collection cache, which is a singleton, before acquiring the lock on
> more than one collection. This is safe, though it puts the db into
> single-task mode (but it happens only in one or two special cases
> anyway). If I remember well, I sent you a junit test to demonstrate
> this. Did you check your code if it does try to work across multiple
> collections? I think it does. If so, please try my fix.

I understand.  The solution we have in place right now is similar to the solution you mentioned, but we put it in place a while ago.  Augmenting the locking with a singleton lock does, indeed, work.

That was my first replacement of the locking mechanism.

The second replacement I've come up with allows two read queries to run simultaneously, even when they target the same collection, and when multiple collections are used simultaneously.  And it enforces atomic read and write operations, something the current locking does not appear to do as well (I am still researching this one).  

It isn't as concurrent as I would like (due to dom.dbx locking), and we still need to put it through its paces to test for stability.

-----Original Message-----
From: Wolfgang Meier [mailto:wol...@ex...] 
Sent: Sunday, August 29, 2010 3:34 AM
To: Dmitriy Shabanov
Cc: Jason Smith; eXist development
Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.

> Would 'normal' lock mechanism be suitable here? Or any restrictions that
> do not allow to use it?

I'm not yet sure of the consequences. I do believe without further
exploration that we could switch to a multi-read/exclusive write lock
mechanism in some places, though this would require some changes to
the cache management (which could - in return - result in new locks
being introduced ;-). The goal, let me repeat, would be to speed up
non index-assisted, non-optimized access to the DOM. Index-assisted
access itself is pretty fast and does allow for good concurrency.

But we have to be very careful here since the architecture is complex:
you have to consider transactional integrity, journalling, caching and
other aspects. If we change anything, we have to proceed carefully and
in very small steps. Stability is always my top priority.

Wolfgang

On 30 August 2010 18:03, Joe Wicentowski <jo...@gm...> wrote:
>>> Should the 2nd one be deleted, now that compression is required for
>>> the repo module?
>>
>> feel free to do so, I tend to leave any new 'core' modules in
>> commented out area in conf.xml until we package up next release ...
>> mostly as a reminder for myself to check on things whilst doing
>> release packaging then I remove.
>
> There's no particular urgency on my part - I'd rather leave it there
> if it's a useful reminder to you later on!

thx, I am still getting used to the idea of so many new people using
eXist and delving into the internals ... for a very long time we
roamed the 'corridors' of the eXist codebase alone and I for one have
plenty of habits that probably need to be reviewed ... very happy for
you and others to make any and all suggestions ;)

cheers, J

If you are referring to org.exist.storage.lock.ReentrantReadWriteLock, which serves as both the collection locking mechanism and the one used by "dom.dbx", "collections.dbx", etc., the problem is that this lock is a mutex.  The name is Reentrant...  However, the implementation uses a mutex over reads and writes.

The ideal would be to allow multiple readers and a single writer to any resource at any time.  The standard locking mechanism, when used with "dom.dbx", allows only one reader or writer at any time.  For long, unoptimized read queries, this results in a choke point on dom.dbx that looks to me like it slows down even optimized queries.

I hope I answered the right question...  :-)

-----Original Message-----
From: Dmitriy Shabanov [mailto:sha...@gm...] 
Sent: Sunday, August 29, 2010 12:56 AM
To: Wolfgang Meier
Cc: Jason Smith; eXist development
Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.

On Sat, 2010-08-28 at 19:06 +0200, Wolfgang Meier wrote:
> > Instead, eXist has artificially limited access to dom.dbx to a single thread (at a time).
> 
> The assumption is that - during a query - dom.dbx is only read at
> serialization time and only to read out a sequence of pages to display
> the final query result to the user.
> 
> It's a complex interplay between cache manager, transaction log and
> other components. I agree there could be ways to allow concurrent read
> access to dom.dbx at the same time, but we would need to carefully
> discuss the implications.

Would 'normal' lock mechanism be suitable here? Or any restrictions that
do not allow to use it?

-- 
Cheers,

Dmitriy Shabanov

> * Deadlocks using this locking mechanism are easy to create.
> * The read locks are mutexes.
> * The locks don't recognize collection hierarchy, so you aren't locking what you think you are.
> * There is a separate mechanism for locking resources (to work, it needs to be a combined mechanism).

Wait a second, I think we have to go back to the start of our
discussion months back. Parts of your application are written against
eXist's internal API, not against a public API like XML:DB, REST or
the XQuery interface. The internal API was never really meant to be
used in end-user applications (I recognize the fact that more and more
users work with it, so we'll need to document it and clean it up).

Anyway, as I tried to explain a few weeks back, the locking is not
fail-safe. You can produce deadlocks if you do not follow certain
conventions (which are hard to know since they are not documented). In
particular, acquiring a lock across multiple collections can cause a
deadlock.

In this case, eXist-internal code acquires a lock on the global
collection cache, which is a singleton, before acquiring the lock on
more than one collection. This is safe, though it puts the db into
single-task mode (but it happens only in one or two special cases
anyway). If I remember well, I sent you a junit test to demonstrate
this. Did you check your code if it does try to work across multiple
collections? I think it does. If so, please try my fix.

All public APIs are tested in environments with many concurrent users.
I'm not aware of any major deadlock situations, except for those
caused by WebDAV locking (the current WebDAV implementation will be
replaced soon). Using the internal API is risky. I do recognize it has
to be improved to allow developers to write code against it more
easily.

I welcome any attempt to improve the locking code to become easier and
more fail-safe. However, you have to be fair and give my suggestions a
try before stating that everything is flawed, while you are
programming against an API which has not been intended for general
use.

Wolfgang

>> Should the 2nd one be deleted, now that compression is required for
>> the repo module?
>
> feel free to do so, I tend to leave any new 'core' modules in
> commented out area in conf.xml until we package up next release ...
> mostly as a reminder for myself to check on things whilst doing
> release packaging then I remove.

There's no particular urgency on my part - I'd rather leave it there
if it's a useful reminder to you later on!

Joe

Okay, I'll try to keep this short.  :-)  I have a tendency to go long on emails...

So in my company, we are using eXist in a way that tends to query against multiple collections simultaneously.  In doing so, we ran into deadlocks in org.exist.storage.lock.ReentrantReadWrite lock.  I looked deeply into that locking mechanism and realized it is deeply flawed.
* The deadlock detection mechanism does not work, and has no chance of working.  It can't be made to work at all.
* Deadlocks using this locking mechanism are easy to create.
* The read locks are mutexes.
* The locks don't recognize collection hierarchy, so you aren't locking what you think you are.
* There is a separate mechanism for locking resources (to work, it needs to be a combined mechanism).

So I have now rewritten org.exist.storage.lock.ReentrantReadWrite, twice. :-) Once to prove that I could completely serialize all access to eXist safely (it worked), and the second time to allow concurrent reads.  This also works, but concurrent performance is limited.  That is, a short query will not be blocked by a long query, which is a very positive result.

The second rewrite lets me selectively either have mutex-read (legacy) or concurrent-read locking, depending on the thread.  It is, well, complicated.  But it's doable.

This isn't something we could put directly into eXist.  Well, we could, but it would likely slow down some operations - you can't take advantage of the concurrent-read locking without deadlock detection code at every entry point - where Sessions are created.  And the legacy locking mode is a global mutex, to prevent deadlocks.  Hey, it works.

We'd have to talk about the implications.  My lock can deadlock!!!  eXist code would have to be modified at multiple points to handle this.  However, I have worked out a reasonable scheme for recovering from deadlocks.  It would require some recoding to take advantage of, and if you don't do the recoding, it defaults to original behavior, which is mutex.

If you guys have time, I can go deeper into the theory (maintaining legacy compatibility makes it kind of heady stuff).  And show you some code.

I have failed, once again, to keep it short. :-/

-----Original Message-----
From: Wolfgang Meier [mailto:wol...@ex...] 
Sent: Sunday, August 29, 2010 3:34 AM
To: Dmitriy Shabanov
Cc: Jason Smith; eXist development
Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.

> Would 'normal' lock mechanism be suitable here? Or any restrictions that
> do not allow to use it?

I'm not yet sure of the consequences. I do believe without further
exploration that we could switch to a multi-read/exclusive write lock
mechanism in some places, though this would require some changes to
the cache management (which could - in return - result in new locks
being introduced ;-). The goal, let me repeat, would be to speed up
non index-assisted, non-optimized access to the DOM. Index-assisted
access itself is pretty fast and does allow for good concurrency.

But we have to be very careful here since the architecture is complex:
you have to consider transactional integrity, journalling, caching and
other aspects. If we change anything, we have to proceed carefully and
in very small steps. Stability is always my top priority.

Wolfgang

On 30 Aug 2010, at 12:55 , Joern Turner wrote:

>> we have been working on trunk this weekend: the cocoon stuff has changed a lot, it is now only available as cocoon-block. This means that we got rid of a significant load of jar files (7 meg or so), reducing potential conflicts with the jar set required for Betterform.
> Thanks for the message - we only had one conflict regarding ehcache.
> Does that mean that this won't be on the classpath any more?

it will never be on the 'exist-db core' classpath, even when the 'cocoon-as-block' extension is enabled.

> And further does this mean that Cocoon will stay (just for interest)?

only as a block (see building.xml#cocoon), that is, exist code is injected into the cocoon source tree. So it will not run as part of exist-db anymore.

Kind regards

Dannes

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

2009	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct (22)	Nov (85)	Dec (20)
2010	Jan (47)	Feb (127)	Mar (268)	Apr (78)	May (47)	Jun (38)	Jul (131)	Aug (221)	Sep (187)	Oct (54)	Nov (111)	Dec (84)
2011	Jan (152)	Feb (106)	Mar (94)	Apr (90)	May (53)	Jun (20)	Jul (24)	Aug (37)	Sep (32)	Oct (70)	Nov (22)	Dec (15)
2012	Jan (33)	Feb (110)	Mar (24)	Apr (1)	May (11)	Jun (8)	Jul (12)	Aug (37)	Sep (39)	Oct (81)	Nov (38)	Dec (50)
2013	Jan (23)	Feb (53)	Mar (23)	Apr (5)	May (19)	Jun (16)	Jul (16)	Aug (9)	Sep (21)	Oct (1)	Nov (2)	Dec (8)
2014	Jan (16)	Feb (6)	Mar (27)	Apr (1)	May (10)	Jun (1)	Jul (4)	Aug (10)	Sep (19)	Oct (22)	Nov (4)	Dec (6)
2015	Jan (3)	Feb (6)	Mar (9)	Apr	May (11)	Jun (23)	Jul (14)	Aug (10)	Sep (10)	Oct (9)	Nov (18)	Dec (4)
2016	Jan (5)	Feb (5)	Mar	Apr (2)	May (15)	Jun (2)	Jul (8)	Aug (2)	Sep (6)	Oct	Nov	Dec
2017	Jan (2)	Feb (12)	Mar (22)	Apr (6)	May	Jun	Jul (1)	Aug (1)	Sep (5)	Oct (2)	Nov	Dec
2018	Jan (2)	Feb	Mar	Apr	May	Jun (5)	Jul (3)	Aug	Sep (7)	Oct (19)	Nov	Dec
2021	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct (1)	Nov	Dec
2022	Jan	Feb	Mar	Apr	May	Jun (3)	Jul	Aug	Sep	Oct	Nov	Dec

exist-development Mailing List for eXist-db (Page 86)

eXist-db is a feature rich Open Source native XML database

exist-development — eXist Developer's List