exist-development — eXist Developer's List
You can subscribe to this list here.
| 2009 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(22) |
Nov
(85) |
Dec
(20) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2010 |
Jan
(47) |
Feb
(127) |
Mar
(268) |
Apr
(78) |
May
(47) |
Jun
(38) |
Jul
(131) |
Aug
(221) |
Sep
(187) |
Oct
(54) |
Nov
(111) |
Dec
(84) |
| 2011 |
Jan
(152) |
Feb
(106) |
Mar
(94) |
Apr
(90) |
May
(53) |
Jun
(20) |
Jul
(24) |
Aug
(37) |
Sep
(32) |
Oct
(70) |
Nov
(22) |
Dec
(15) |
| 2012 |
Jan
(33) |
Feb
(110) |
Mar
(24) |
Apr
(1) |
May
(11) |
Jun
(8) |
Jul
(12) |
Aug
(37) |
Sep
(39) |
Oct
(81) |
Nov
(38) |
Dec
(50) |
| 2013 |
Jan
(23) |
Feb
(53) |
Mar
(23) |
Apr
(5) |
May
(19) |
Jun
(16) |
Jul
(16) |
Aug
(9) |
Sep
(21) |
Oct
(1) |
Nov
(2) |
Dec
(8) |
| 2014 |
Jan
(16) |
Feb
(6) |
Mar
(27) |
Apr
(1) |
May
(10) |
Jun
(1) |
Jul
(4) |
Aug
(10) |
Sep
(19) |
Oct
(22) |
Nov
(4) |
Dec
(6) |
| 2015 |
Jan
(3) |
Feb
(6) |
Mar
(9) |
Apr
|
May
(11) |
Jun
(23) |
Jul
(14) |
Aug
(10) |
Sep
(10) |
Oct
(9) |
Nov
(18) |
Dec
(4) |
| 2016 |
Jan
(5) |
Feb
(5) |
Mar
|
Apr
(2) |
May
(15) |
Jun
(2) |
Jul
(8) |
Aug
(2) |
Sep
(6) |
Oct
|
Nov
|
Dec
|
| 2017 |
Jan
(2) |
Feb
(12) |
Mar
(22) |
Apr
(6) |
May
|
Jun
|
Jul
(1) |
Aug
(1) |
Sep
(5) |
Oct
(2) |
Nov
|
Dec
|
| 2018 |
Jan
(2) |
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| S | M | T | W | T | F | S |
|---|---|---|---|---|---|---|
|
1
|
2
(8) |
3
(3) |
4
|
5
|
6
(11) |
7
(9) |
|
8
(3) |
9
(19) |
10
(2) |
11
(14) |
12
(25) |
13
(5) |
14
|
|
15
(8) |
16
(11) |
17
(3) |
18
(12) |
19
(9) |
20
|
21
(3) |
|
22
(7) |
23
(4) |
24
(3) |
25
(3) |
26
(2) |
27
(4) |
28
(3) |
|
29
(9) |
30
(16) |
31
(25) |
|
|
|
|
|
From: Jason Smith <jsmith@in...> - 2010-08-31 22:55:20
|
Quick OS Survey: What operating systems is everyone using? For purposes of finding some common white-boarding app. I'm good with Windows and/or Linux. Is anyone planning to attend Linux-only? -----Original Message----- From: Wolfgang Meier [mailto:wolfgang@...] Sent: Tuesday, August 31, 2010 4:04 PM To: Jason Smith Cc: Adam Retter; Paul Ryan; eXist development; Michael J. Pelikan; Todd Gochenour Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries. > For example, if you have a single dom.dbx file, can you write to that from multiple threads at the same time safely? If you can't, then it doesn't make much sense to write-lock at the collection level, right? The bigger picture is much more complex. You have to talk about transactions, recovery, caching and more. You do not write to dom.dbx directly. You write to the page cache. And there's more than just dom.dbx. Writing the actual data just takes a small part of the overall indexing time. More time is spent with indexing, maintaining the transaction log etc. From my point of view, the next step in any redesign effort should be to remove the collection locks entirely. We have discussed this before. It will greatly simplify the locking and transaction log. My roadmap roughly looks like this: 1) remove collection dependency from core indexes: 1a) structural index, DONE 1b) range index, IN PROGESS 1c) remove document metadata from collection store and keep it separately. a collection is just a sequence of (arbitrary) document ids. a document can be linked to more than one collection. 2) drop all collection locks, except for the case where the collection metadata itself is modified Those steps have to be completed before we address other things. We need to simplify the architecture first, then try to do further redesigns. Any help will be welcome. As an added value, 1a and b will improve update/write performance in general. > Still, it seems like it would be nice to write to one document while querying against another document in the same collection... Normally, eXist will acquire a lock on the collection, acquire one on the document, release the collection lock, continue parsing the document. In some cases (node updates), the transaction handling has forced us to keep the lock on the collection longer than desired. But this can be changed (see above). > And this all has to be done in a system that currently does not support true transactional rollback (I think). eXist maintains a transaction log and does redo/undo on recovery. The only limitation is that the transaction log is incomplete, i.e. it does not cover any secondary indexes. Still transactional integrity has to be preserved and puts further requirements on the locking (this is the reason why collection locks are often not released early). Well, I will stop writing emails now and better explain everything on Thursday. Wolfgang |
|
From: Jason Smith <jsmith@in...> - 2010-08-31 22:37:46
|
No, write some more on this. I have heard you talk about removing the collection locks, but I am not sure how you plan to replace their purpose. That would be good to understand before Thursday. If you remove collection locks, does that mean you are planning to lock at the resource level? Wouldn't that potentially lead to an awful lot of locks being taken? Am I misreading the idea? -----Original Message----- From: Wolfgang Meier [mailto:wolfgang@...] Sent: Tuesday, August 31, 2010 4:04 PM To: Jason Smith Cc: Adam Retter; Paul Ryan; eXist development; Michael J. Pelikan; Todd Gochenour Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries. > For example, if you have a single dom.dbx file, can you write to that from multiple threads at the same time safely? If you can't, then it doesn't make much sense to write-lock at the collection level, right? The bigger picture is much more complex. You have to talk about transactions, recovery, caching and more. You do not write to dom.dbx directly. You write to the page cache. And there's more than just dom.dbx. Writing the actual data just takes a small part of the overall indexing time. More time is spent with indexing, maintaining the transaction log etc. From my point of view, the next step in any redesign effort should be to remove the collection locks entirely. We have discussed this before. It will greatly simplify the locking and transaction log. My roadmap roughly looks like this: 1) remove collection dependency from core indexes: 1a) structural index, DONE 1b) range index, IN PROGESS 1c) remove document metadata from collection store and keep it separately. a collection is just a sequence of (arbitrary) document ids. a document can be linked to more than one collection. 2) drop all collection locks, except for the case where the collection metadata itself is modified Those steps have to be completed before we address other things. We need to simplify the architecture first, then try to do further redesigns. Any help will be welcome. As an added value, 1a and b will improve update/write performance in general. > Still, it seems like it would be nice to write to one document while querying against another document in the same collection... Normally, eXist will acquire a lock on the collection, acquire one on the document, release the collection lock, continue parsing the document. In some cases (node updates), the transaction handling has forced us to keep the lock on the collection longer than desired. But this can be changed (see above). > And this all has to be done in a system that currently does not support true transactional rollback (I think). eXist maintains a transaction log and does redo/undo on recovery. The only limitation is that the transaction log is incomplete, i.e. it does not cover any secondary indexes. Still transactional integrity has to be preserved and puts further requirements on the locking (this is the reason why collection locks are often not released early). Well, I will stop writing emails now and better explain everything on Thursday. Wolfgang |
|
From: Wolfgang Meier <wolfgang@ex...> - 2010-08-31 22:04:28
|
> For example, if you have a single dom.dbx file, can you write to that from multiple threads at the same time safely? If you can't, then it doesn't make much sense to write-lock at the collection level, right? The bigger picture is much more complex. You have to talk about transactions, recovery, caching and more. You do not write to dom.dbx directly. You write to the page cache. And there's more than just dom.dbx. Writing the actual data just takes a small part of the overall indexing time. More time is spent with indexing, maintaining the transaction log etc. >From my point of view, the next step in any redesign effort should be to remove the collection locks entirely. We have discussed this before. It will greatly simplify the locking and transaction log. My roadmap roughly looks like this: 1) remove collection dependency from core indexes: 1a) structural index, DONE 1b) range index, IN PROGESS 1c) remove document metadata from collection store and keep it separately. a collection is just a sequence of (arbitrary) document ids. a document can be linked to more than one collection. 2) drop all collection locks, except for the case where the collection metadata itself is modified Those steps have to be completed before we address other things. We need to simplify the architecture first, then try to do further redesigns. Any help will be welcome. As an added value, 1a and b will improve update/write performance in general. > Still, it seems like it would be nice to write to one document while querying against another document in the same collection... Normally, eXist will acquire a lock on the collection, acquire one on the document, release the collection lock, continue parsing the document. In some cases (node updates), the transaction handling has forced us to keep the lock on the collection longer than desired. But this can be changed (see above). > And this all has to be done in a system that currently does not support true transactional rollback (I think). eXist maintains a transaction log and does redo/undo on recovery. The only limitation is that the transaction log is incomplete, i.e. it does not cover any secondary indexes. Still transactional integrity has to be preserved and puts further requirements on the locking (this is the reason why collection locks are often not released early). Well, I will stop writing emails now and better explain everything on Thursday. Wolfgang |
|
From: Jason Smith <jsmith@in...> - 2010-08-31 21:30:55
|
There is also dimdim. I've used that one before. Preferences? It would be nice to have a whiteboard app. From: Loren Cahlander [mailto:loren.cahlander@...] Sent: Tuesday, August 31, 2010 2:49 PM To: Dannes Wessels Cc: Loren Cahlander; Jason Smith; Wolfgang Meier; eXist development; Michael J. Pelikan; Todd Gochenour; Paul Ryan Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries. How about Google Docs? http://docs.google.com On Aug 31, 2010, at 03:41 PM, Dannes Wessels wrote: We should probably use http://www.doodle.com/ for these kind of international meetings :-) On 31 Aug 2010, at 22:36 , Jason Smith wrote: Kind of late for Wolfgang though. Kind regards Dannes -- eXist-db Native XML Database - http://exist-db.org<http://exist-db.org/>; Join us on linked-in: http://www.linkedin.com/groups?gid=35624 |
|
From: Jason Smith <jsmith@in...> - 2010-08-31 21:23:32
|
I think we need to talk about how much granularity is actually valuable. For example, if you have a single dom.dbx file, can you write to that from multiple threads at the same time safely? If you can't, then it doesn't make much sense to write-lock at the collection level, right?
Also, if you have a maximally granular locking mechanism, allowing locks on collections and resources, both deep locks and shallow locks, with multiple readers and a single writer allowed on each, the deadlock detection gets really complex. Performance on deadlock detection can blow up.
Still, it seems like it would be nice to write to one document while querying against another document in the same collection...
How the collections are implemented underneath isn't that important to the locking mechanism, other than this affects how much concurrency that you can actually take advantage of.
I think, though, that you **need to have the locking mechanism in place pretty early on.** If you design a wonderfully concurrent back end, but you control access though a global mutex, well, what have you got? :-)
Plus, in any non-trivial locking mechanism, there will be deadlocks and deadlock detection-and-recovery. And the software, as a whole, has to use the standards for detection-and-recovery if you want to be able to take advantage of the more concurrent locking.
And this all has to be done in a system that currently does not support true transactional rollback (I think). Which means there are some additional rules when it comes to write locking...
Too much information. I'll write something up for Thursday, and hopefully all this stuff will become more clear. This is not an easy topic for anyone, including myself!
-----Original Message-----
From: Adam Retter [mailto:adam@...]
Sent: Tuesday, August 31, 2010 6:42 AM
To: Wolfgang Meier
Cc: Jason Smith; Paul Ryan; eXist development; Michael J. Pelikan; Todd Gochenour
Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.
> Otherwise, eXist just avoids locking multiple collections at once
> wherever possible as it is known to be an expensive operation and
> limits concurrency.
We have discussed several times replacing eXist-db's current
collection mechanism with a virtualised implementation where
Collections are just another number in the system. This was discussed
for the purposes of performance when large collections are involved.
Would this simplify the overal problem domain? If so, perhaps this
work should be undertaken before a redesign of the locking system?
>> The second replacement I've come up with allows two read queries to run
>> simultaneously, even when they target the same collection, and when multiple collections
>> are used simultaneously.
>
> As I said before, I welcome any exploration in this area. As James
> just suggested, we may want to have a skype telecon on this to discuss
> the possibilities and dangers.
Skype teleconference would be good :-)
--
Adam Retter
eXist Developer
{ United Kingdom }
adam@...
irc://irc.freenode.net/existdb
|
|
From: Adam Retter <adam@ex...> - 2010-08-31 21:03:41
|
Anything that can be opened by either OpenOffice or a Web Browser
should be fine - that includes the de-facto standard MS office
formats...
On 31 August 2010 21:36, Jason Smith <jsmith@...> wrote:
> I'd like to prepare some materials. What is your preferred format? MS Office, Open Office, Maven XDoc... ? I will try to avoid PowerPoint.
>
> -----Original Message-----
> From: Wolfgang Meier [mailto:wolfgang@...]
> Sent: Tuesday, August 31, 2010 2:32 PM
> To: Jason Smith
> Cc: Adam Retter; Dmitriy Shabanov; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; loren
> Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.
>
>> That will work too. Kind of late for Wolfgang though.
>
> No, that's fine.
>
> Wolfgang
>
--
Adam Retter
eXist Developer
{ United Kingdom }
adam@...
irc://irc.freenode.net/existdb
|
|
From: Jason Smith <jsmith@in...> - 2010-08-31 20:55:30
|
The reason I have been sort of a pain in the tuckus about this is not to get a quick fix, but to get this on the roadmap. :-) I definitely don't want to lose it. -----Original Message----- From: Adam Retter [mailto:adam@...] Sent: Tuesday, August 31, 2010 6:47 AM To: Jason Smith Cc: Wolfgang Meier; Paul Ryan; eXist development; Michael J. Pelikan; Todd Gochenour Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries. > Again, I would not expect a final solution to be implemented the way I have done it. I just want to be able to show some possibilities. I am really excited to see such enthusiasm and knowledge of the eXist-db internals and suggestions for improvements that could be made. > I am not expecting this conversation to take place quickly. This is one of those "long-haul" things - perhaps over years, and several major releases. I'm not in a hurry to do the wrong thing quickly!!! > Lets not loose this though, I think that any performance and scalability enhancements are great. And actually there is a flip side here that no one has mentioned: Whilst everyone has said that you should have optimised queries and appropriate indexes, which is invariably true for production systems, what about new users and developers who come to eXist-db? As a new user of eXist-db you dont necessarily understand about optimising your queries or even how to create the correct indexes. Not all eXist-db users are software developers! It would be great if these new users saw great performance from the start, even if they havent set up indexes and their queries are doing full scans of the dbx files :-) |
|
From: Dannes Wessels <dannes@ex...> - 2010-08-31 20:41:36
|
We should probably use http://www.doodle.com/ for these kind of international meetings :-) On 31 Aug 2010, at 22:36 , Jason Smith wrote: >> Kind of late for Wolfgang though. Kind regards Dannes -- eXist-db Native XML Database - http://exist-db.org Join us on linked-in: http://www.linkedin.com/groups?gid=35624 |
|
From: Jason Smith <jsmith@in...> - 2010-08-31 20:37:20
|
I'd like to prepare some materials. What is your preferred format? MS Office, Open Office, Maven XDoc... ? I will try to avoid PowerPoint. -----Original Message----- From: Wolfgang Meier [mailto:wolfgang@...] Sent: Tuesday, August 31, 2010 2:32 PM To: Jason Smith Cc: Adam Retter; Dmitriy Shabanov; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; loren Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries. > That will work too. Kind of late for Wolfgang though. No, that's fine. Wolfgang |
|
From: Wolfgang Meier <wolfgang@ex...> - 2010-08-31 20:32:30
|
> That will work too. Kind of late for Wolfgang though. No, that's fine. Wolfgang |
|
From: Jason Smith <jsmith@in...> - 2010-08-31 20:29:58
|
That will work too. Kind of late for Wolfgang though.
-----Original Message-----
From: Adam Retter [mailto:adam@...]
Sent: Tuesday, August 31, 2010 2:27 PM
To: Jason Smith
Cc: Wolfgang Meier; Dmitriy Shabanov; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; loren
Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.
Wow steady on there! Thats a bit early! how about 3pm UTC?
On 31 August 2010 21:25, Jason Smith <jsmith@...> wrote:
> I believe that is 8am my time. I think I can swing that.
>
> -----Original Message-----
> From: Adam Retter [mailto:adam@...]
> Sent: Tuesday, August 31, 2010 2:17 PM
> To: Jason Smith
> Cc: Wolfgang Meier; Dmitriy Shabanov; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; loren
> Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.
>
> How about 2pm UTC, if I have this correct (???) -
>
> 3pm my time (BST)
> 4pm Wolfgangs time (CEST)
> 9am your time???
>
>
> On 31 August 2010 20:57, Jason Smith <jsmith@...> wrote:
>> I'm in the US, MST (Denver, CO). Be kind when picking times, please. :-)
>>
>> -----Original Message-----
>> From: Wolfgang Meier [mailto:wolfgang@...]
>> Sent: Tuesday, August 31, 2010 9:44 AM
>> To: Dmitriy Shabanov
>> Cc: Adam Retter; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; Jason Smith; loren
>> Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.
>>
>>>> Skype teleconference would be good :-)
>>>
>>> When? (ready to join)
>>
>> Ok, I think a teleconference would make it easier for me to explain my
>> roadmap for eXist and for Jason to describe his ideas in more detail.
>> I would have time for a talk tomorrow, but would prefer Thursday if
>> possible. Anytime between 10am and 9pm CEST.
>>
>> Wolfgang
>>
>
>
>
> --
> Adam Retter
>
> eXist Developer
> { United Kingdom }
> adam@...
> irc://irc.freenode.net/existdb
>
--
Adam Retter
eXist Developer
{ United Kingdom }
adam@...
irc://irc.freenode.net/existdb
|
|
From: Adam Retter <adam@ex...> - 2010-08-31 20:27:05
|
Wow steady on there! Thats a bit early! how about 3pm UTC?
On 31 August 2010 21:25, Jason Smith <jsmith@...> wrote:
> I believe that is 8am my time. I think I can swing that.
>
> -----Original Message-----
> From: Adam Retter [mailto:adam@...]
> Sent: Tuesday, August 31, 2010 2:17 PM
> To: Jason Smith
> Cc: Wolfgang Meier; Dmitriy Shabanov; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; loren
> Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.
>
> How about 2pm UTC, if I have this correct (???) -
>
> 3pm my time (BST)
> 4pm Wolfgangs time (CEST)
> 9am your time???
>
>
> On 31 August 2010 20:57, Jason Smith <jsmith@...> wrote:
>> I'm in the US, MST (Denver, CO). Be kind when picking times, please. :-)
>>
>> -----Original Message-----
>> From: Wolfgang Meier [mailto:wolfgang@...]
>> Sent: Tuesday, August 31, 2010 9:44 AM
>> To: Dmitriy Shabanov
>> Cc: Adam Retter; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; Jason Smith; loren
>> Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.
>>
>>>> Skype teleconference would be good :-)
>>>
>>> When? (ready to join)
>>
>> Ok, I think a teleconference would make it easier for me to explain my
>> roadmap for eXist and for Jason to describe his ideas in more detail.
>> I would have time for a talk tomorrow, but would prefer Thursday if
>> possible. Anytime between 10am and 9pm CEST.
>>
>> Wolfgang
>>
>
>
>
> --
> Adam Retter
>
> eXist Developer
> { United Kingdom }
> adam@...
> irc://irc.freenode.net/existdb
>
--
Adam Retter
eXist Developer
{ United Kingdom }
adam@...
irc://irc.freenode.net/existdb
|
|
From: Jason Smith <jsmith@in...> - 2010-08-31 20:25:24
|
I believe that is 8am my time. I think I can swing that.
-----Original Message-----
From: Adam Retter [mailto:adam@...]
Sent: Tuesday, August 31, 2010 2:17 PM
To: Jason Smith
Cc: Wolfgang Meier; Dmitriy Shabanov; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; loren
Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.
How about 2pm UTC, if I have this correct (???) -
3pm my time (BST)
4pm Wolfgangs time (CEST)
9am your time???
On 31 August 2010 20:57, Jason Smith <jsmith@...> wrote:
> I'm in the US, MST (Denver, CO). Be kind when picking times, please. :-)
>
> -----Original Message-----
> From: Wolfgang Meier [mailto:wolfgang@...]
> Sent: Tuesday, August 31, 2010 9:44 AM
> To: Dmitriy Shabanov
> Cc: Adam Retter; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; Jason Smith; loren
> Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.
>
>>> Skype teleconference would be good :-)
>>
>> When? (ready to join)
>
> Ok, I think a teleconference would make it easier for me to explain my
> roadmap for eXist and for Jason to describe his ideas in more detail.
> I would have time for a talk tomorrow, but would prefer Thursday if
> possible. Anytime between 10am and 9pm CEST.
>
> Wolfgang
>
--
Adam Retter
eXist Developer
{ United Kingdom }
adam@...
irc://irc.freenode.net/existdb
|
|
From: Adam Retter <adam@ex...> - 2010-08-31 20:17:05
|
How about 2pm UTC, if I have this correct (???) -
3pm my time (BST)
4pm Wolfgangs time (CEST)
9am your time???
On 31 August 2010 20:57, Jason Smith <jsmith@...> wrote:
> I'm in the US, MST (Denver, CO). Be kind when picking times, please. :-)
>
> -----Original Message-----
> From: Wolfgang Meier [mailto:wolfgang@...]
> Sent: Tuesday, August 31, 2010 9:44 AM
> To: Dmitriy Shabanov
> Cc: Adam Retter; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; Jason Smith; loren
> Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.
>
>>> Skype teleconference would be good :-)
>>
>> When? (ready to join)
>
> Ok, I think a teleconference would make it easier for me to explain my
> roadmap for eXist and for Jason to describe his ideas in more detail.
> I would have time for a talk tomorrow, but would prefer Thursday if
> possible. Anytime between 10am and 9pm CEST.
>
> Wolfgang
>
--
Adam Retter
eXist Developer
{ United Kingdom }
adam@...
irc://irc.freenode.net/existdb
|
|
From: Jason Smith <jsmith@in...> - 2010-08-31 19:57:45
|
I'm in the US, MST (Denver, CO). Be kind when picking times, please. :-) -----Original Message----- From: Wolfgang Meier [mailto:wolfgang@...] Sent: Tuesday, August 31, 2010 9:44 AM To: Dmitriy Shabanov Cc: Adam Retter; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; Jason Smith; loren Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries. >> Skype teleconference would be good :-) > > When? (ready to join) Ok, I think a teleconference would make it easier for me to explain my roadmap for eXist and for Jason to describe his ideas in more detail. I would have time for a talk tomorrow, but would prefer Thursday if possible. Anytime between 10am and 9pm CEST. Wolfgang |
|
From: Dmitriy Shabanov <shabanovd@gm...> - 2010-08-31 18:50:01
|
On Tue, 2010-08-31 at 09:19 -0700, Jason Smith wrote: > I guess I will have to install Skype. :-) Thursday is fine. No hurries. > > -----Original Message----- > From: Adam Retter [mailto:adam@...] > Sent: Tuesday, August 31, 2010 10:17 AM > To: Wolfgang Meier > Cc: Dmitriy Shabanov; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; Jason Smith; loren > Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries. > > On 31 August 2010 16:43, Wolfgang Meier <wolfgang@...> wrote: > >>> Skype teleconference would be good :-) > >> > >> When? (ready to join) > > > > Ok, I think a teleconference would make it easier for me to explain my > > roadmap for eXist and for Jason to describe his ideas in more detail. > > I would have time for a talk tomorrow, but would prefer Thursday if > > possible. Anytime between 10am and 9pm CEST. > > Thursday is fine for me also, but will have to be before 5pm BST. Thursday, almost any time. -- Cheers, Dmitriy Shabanov |
|
From: Adam Retter <adam@ex...> - 2010-08-31 16:24:36
|
On 31 August 2010 16:43, Wolfgang Meier <wolfgang@...> wrote:
>>> Skype teleconference would be good :-)
>>
>> When? (ready to join)
>
> Ok, I think a teleconference would make it easier for me to explain my
> roadmap for eXist and for Jason to describe his ideas in more detail.
> I would have time for a talk tomorrow, but would prefer Thursday if
> possible. Anytime between 10am and 9pm CEST.
Thursday is fine for me also, but will have to be before 5pm BST.
> Wolfgang
>
--
Adam Retter
eXist Developer
{ United Kingdom }
adam@...
irc://irc.freenode.net/existdb
|
|
From: Jason Smith <jsmith@in...> - 2010-08-31 16:19:44
|
I guess I will have to install Skype. :-) Thursday is fine. No hurries.
-----Original Message-----
From: Adam Retter [mailto:adam@...]
Sent: Tuesday, August 31, 2010 10:17 AM
To: Wolfgang Meier
Cc: Dmitriy Shabanov; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour; Jason Smith; loren
Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries.
On 31 August 2010 16:43, Wolfgang Meier <wolfgang@...> wrote:
>>> Skype teleconference would be good :-)
>>
>> When? (ready to join)
>
> Ok, I think a teleconference would make it easier for me to explain my
> roadmap for eXist and for Jason to describe his ideas in more detail.
> I would have time for a talk tomorrow, but would prefer Thursday if
> possible. Anytime between 10am and 9pm CEST.
Thursday is fine for me also, but will have to be before 5pm BST.
> Wolfgang
>
--
Adam Retter
eXist Developer
{ United Kingdom }
adam@...
irc://irc.freenode.net/existdb
|
|
From: Andrzej Jan Taramina <andrzej@ch...> - 2010-08-31 16:10:15
|
I've noticed that when I try to display the Function Documentation web page, when you do a search or browse for a specific set of functions in a namespace (eg. Util), it takes forever to display the page. And it seems to lock up my whole machine, not just Firefox when it's rendering the page. It never used to do this.... Any ideas what is causing this? I'm running the latest Firefox (3.6.8) on a 64-bit Ubuntu, fast quad processor box with gobs of memory.....it should fly like the wind...but instead my machine will become non-responsive for 20-30 seconds. Running an SVN build from late July (since the security stuff broke everything, and I haven't had a chance to test the latest stuff of a few days ago). Anyone have any idea what might be causing this massive slowdown? Thanks. -- Andrzej Taramina Chaeron Corporation: Enterprise System Solutions http://www.chaeron.com |
|
From: Wolfgang Meier <wolfgang@ex...> - 2010-08-31 15:50:44
|
>> Skype teleconference would be good :-) > > When? (ready to join) Ok, I think a teleconference would make it easier for me to explain my roadmap for eXist and for Jason to describe his ideas in more detail. I would have time for a talk tomorrow, but would prefer Thursday if possible. Anytime between 10am and 9pm CEST. Wolfgang |
|
From: Dmitriy Shabanov <shabanovd@gm...> - 2010-08-31 15:39:31
|
On Tue, 2010-08-31 at 13:42 +0100, Adam Retter wrote: > > As I said before, I welcome any exploration in this area. As James > > just suggested, we may want to have a skype telecon on this to > discuss > > the possibilities and dangers. > > Skype teleconference would be good :-) When? (ready to join) -- Cheers, Dmitriy Shabanov |
|
From: Adam Retter <adam@ex...> - 2010-08-31 13:39:09
|
> Otherwise, eXist just avoids locking multiple collections at once
> wherever possible as it is known to be an expensive operation and
> limits concurrency.
We have discussed several times replacing eXist-db's current
collection mechanism with a virtualised implementation where
Collections are just another number in the system. This was discussed
for the purposes of performance when large collections are involved.
Would this simplify the overal problem domain? If so, perhaps this
work should be undertaken before a redesign of the locking system?
>> The second replacement I've come up with allows two read queries to run
>> simultaneously, even when they target the same collection, and when multiple collections
>> are used simultaneously.
>
> As I said before, I welcome any exploration in this area. As James
> just suggested, we may want to have a skype telecon on this to discuss
> the possibilities and dangers.
Skype teleconference would be good :-)
--
Adam Retter
eXist Developer
{ United Kingdom }
adam@...
irc://irc.freenode.net/existdb
|
|
From: Adam Retter <adam@ex...> - 2010-08-31 12:47:23
|
> Again, I would not expect a final solution to be implemented the way I have done it. I just want to be able to show some possibilities. I am really excited to see such enthusiasm and knowledge of the eXist-db internals and suggestions for improvements that could be made. > I am not expecting this conversation to take place quickly. This is one of those "long-haul" things - perhaps over years, and several major releases. I'm not in a hurry to do the wrong thing quickly!!! > Lets not loose this though, I think that any performance and scalability enhancements are great. And actually there is a flip side here that no one has mentioned: Whilst everyone has said that you should have optimised queries and appropriate indexes, which is invariably true for production systems, what about new users and developers who come to eXist-db? As a new user of eXist-db you dont necessarily understand about optimising your queries or even how to create the correct indexes. Not all eXist-db users are software developers! It would be great if these new users saw great performance from the start, even if they havent set up indexes and their queries are doing full scans of the dbx files :-) > -----Original Message----- > From: Wolfgang Meier [mailto:wolfgang@...] > Sent: Monday, August 30, 2010 10:38 AM > To: Jason Smith > Cc: Dmitriy Shabanov; eXist development; Paul Ryan; Michael J. Pelikan; Todd Gochenour > Subject: Re: [Exist-development] [Exist-open] Performance of concurrent read queries. > >> I understand. The solution we have in place right now is similar to the solution you >> mentioned, but we put it in place a while ago. Augmenting the locking with a singleton >> lock does, indeed, work. > > Internally, eXist does need the singleton lock in rare cases only, > mainly when reading or storing the collection configuration document > for a collection, or when locking documents for an XQuery update > expression. > > Otherwise, eXist just avoids locking multiple collections at once > wherever possible as it is known to be an expensive operation and > limits concurrency. > >> The second replacement I've come up with allows two read queries to run >> simultaneously, even when they target the same collection, and when multiple collections >> are used simultaneously. > > As I said before, I welcome any exploration in this area. As James > just suggested, we may want to have a skype telecon on this to discuss > the possibilities and dangers. > > Finally, just as a note to other users who code against the internal > API: Dannes' WebDAV reimplementation shows some clean examples of how > to use internals: > > http://exist.svn.sourceforge.net/viewvc/exist/branches/dizzzz/trunk-webdav-upgrade/extensions/webdav/src/org/exist/webdav/ > > Wolfgang > ------------------------------------------------------------------------------ > This SF.net Dev2Dev email is sponsored by: > > Show off your parallel programming skills. > Enter the Intel(R) Threading Challenge 2010. > http://p.sf.net/sfu/intel-thread-sfd > _______________________________________________ > Exist-development mailing list > Exist-development@... > https://lists.sourceforge.net/lists/listinfo/exist-development > -- Adam Retter eXist Developer { United Kingdom } adam@... irc://irc.freenode.net/existdb |
|
From: Adam Retter <adam@ex...> - 2010-08-31 12:13:38
|
> I do think to code separate methods with security checks on low level, > so it going to be only one place check. That will make all interface > behavior same. Perhaps we should use a layered approach through composition. Standard is without and then the additional layer has the security checks all in one place. > -- > Cheers, > > Dmitriy Shabanov > > ------------------------------------------------------------------------------ > Sell apps to millions through the Intel(R) Atom(Tm) Developer Program > Be part of this innovative community and reach millions of netbook users > worldwide. Take advantage of special opportunities to increase revenue and > speed time-to-market. Join now, and jumpstart your future. > http://p.sf.net/sfu/intel-atom-d2d > _______________________________________________ > Exist-commits mailing list > Exist-commits@... > https://lists.sourceforge.net/lists/listinfo/exist-commits > > -- Adam Retter eXist Developer { United Kingdom } adam@... irc://irc.freenode.net/existdb |
1 message has been excluded from this view by a project administrator.