You can subscribe to this list here.
2009 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(22) |
Nov
(85) |
Dec
(20) |
---|---|---|---|---|---|---|---|---|---|---|---|---|
2010 |
Jan
(47) |
Feb
(127) |
Mar
(268) |
Apr
(78) |
May
(47) |
Jun
(38) |
Jul
(131) |
Aug
(221) |
Sep
(187) |
Oct
(54) |
Nov
(111) |
Dec
(84) |
2011 |
Jan
(152) |
Feb
(106) |
Mar
(94) |
Apr
(90) |
May
(53) |
Jun
(20) |
Jul
(24) |
Aug
(37) |
Sep
(32) |
Oct
(70) |
Nov
(22) |
Dec
(15) |
2012 |
Jan
(33) |
Feb
(110) |
Mar
(24) |
Apr
(1) |
May
(11) |
Jun
(8) |
Jul
(12) |
Aug
(37) |
Sep
(39) |
Oct
(81) |
Nov
(38) |
Dec
(50) |
2013 |
Jan
(23) |
Feb
(53) |
Mar
(23) |
Apr
(5) |
May
(19) |
Jun
(16) |
Jul
(16) |
Aug
(9) |
Sep
(21) |
Oct
(1) |
Nov
(2) |
Dec
(8) |
2014 |
Jan
(16) |
Feb
(6) |
Mar
(27) |
Apr
(1) |
May
(10) |
Jun
(1) |
Jul
(4) |
Aug
(10) |
Sep
(19) |
Oct
(22) |
Nov
(4) |
Dec
(6) |
2015 |
Jan
(3) |
Feb
(6) |
Mar
(9) |
Apr
|
May
(11) |
Jun
(23) |
Jul
(14) |
Aug
(10) |
Sep
(10) |
Oct
(9) |
Nov
(18) |
Dec
(4) |
2016 |
Jan
(5) |
Feb
(5) |
Mar
|
Apr
(2) |
May
(15) |
Jun
(2) |
Jul
(8) |
Aug
(2) |
Sep
(6) |
Oct
|
Nov
|
Dec
|
2017 |
Jan
(2) |
Feb
(12) |
Mar
(22) |
Apr
(6) |
May
|
Jun
|
Jul
(1) |
Aug
(1) |
Sep
(5) |
Oct
(2) |
Nov
|
Dec
|
2018 |
Jan
(2) |
Feb
|
Mar
|
Apr
|
May
|
Jun
(5) |
Jul
(3) |
Aug
|
Sep
(7) |
Oct
(19) |
Nov
|
Dec
|
2021 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(1) |
Nov
|
Dec
|
2022 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
(3) |
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
From: James F. <jam...@ex...> - 2010-03-31 21:21:13
|
done some further analysis on this and have come to a few 'forks in the road' * Adam Retters vision of /sql/system/sql/{database}/{schemaName}/{tableName} is great ... I would amend this too something simpler like /sql/{database}/{schemaName}/{tableName}/ and this can be the concept of a mounted database from anywhere in collection hierarchy. * this also means we get REST and friends entrypoints * additionally, we may want an alternate configuration e.g. if we consider introducing the concept of a new protocol e.g. rdbms:exist:////sql/{database}/{schemaName}/{tableName}/ which leaves us a bit more wiggle room instead of a mount point in the database e.g. we probably want to make sure people know when an rdbms is being queried in some situations. * from an xpath, xslt and xquery point of view using this protocol should mean nothing special is needed (other then initial setup) to use data from rdbms * I agree with Adam Retters approach of lazy eval though I would like to investigate using an xpath parser approach that generates sql statements ... probably could use a lot of Adam's existing sql extension for the underneath bits * as for registering and configuration I think if its a 'mount' point in the database approach then we can use collection.xconf which could have a snazzy xquery app behind it for CRUD entries. Otherwise the rdbms:exist protocol would be configured 'elsewhere' ok, now reality .... my plate is full (of xproc) and I probably should not get distracted ... so anyone who wants to pick up and run with this tidbit feel free, I have decided to step away from this for the time being. J |
From: Adam R. <ad...@ex...> - 2010-03-31 20:43:30
|
Andrezj, A chap on the mailing list has quite some experience of scaling eXist into the hundreds of gigabytes range, perhaps if you email him he could share some of his experiences with you as well. José María Fernández González. jmfernandez <at> cnb.uam.es On 29 March 2010 15:59, Andrzej Jan Taramina <an...@ch...> wrote: > Looking to get some guidance on how big you can scale an eXist database. > > Right now, our instances are about 15-25K documents where each document is in the 25K-2M range, probably averaging > around 150-200K. This results in a dom.dbx = 3.5G, structure.dbx = 1.8G, collections.dbx = 4.2M and values.dbx = 155M, > which is not all that large compared to some relational databases. > > What if we scale up 10x to nearly quarter of a million documents? The file sizes still shouldn't be all that big for > modern hardware, but will the performance scale linearly or close to it, assuming a powerful enough server (say a > dual-cpu, 6-Core machine (12 cores, 24 native threads) with gobs of memory)? > > OK.....if that works how about two orders of magnitude (100x current size)? That would give us 2.5M documents, 250GB > dom.dbx and a structure.dbx in the 180GB range. Bit too big or practical to cache the whole structure.dbx in memory, > regardless of the size of the memory in the server. > > At what point do I start looking at alternative storage mechanisms, (RDBMS, Hadoop, memcached, etc.) or co-operating > distributed eXist instances? > > Thanks for any insights from those that have pushed big databases in eXist... > > -- > Andrzej Taramina > Chaeron Corporation: Enterprise System Solutions > http://www.chaeron.com > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Exist-development mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-development > -- Adam Retter eXist Developer { United Kingdom } ad...@ex... irc://irc.freenode.net/existdb |
From: Adam R. <ad...@ex...> - 2010-03-31 19:55:24
|
Sure, it may be that if we contact the Balisage people they might be able to help or recommend something. The reason I suggested it, is because it is an incredibly expensive proposition for us non-Americas folks to come over to the Americas unless we are being paid or funded. I though that if perhaps we combined it with the dates around Balisage it perhaps makes it a more acceptable cost for us and hence pushes up attendance and knowledge. Thanks Adam. On 31 March 2010 19:33, Loren Cahlander <lor...@gm...> wrote: > I already have some facilities tentatively arranged for in Minnesota. If we were to host this meeting in any other city than our own, then it would be too expensive for us to put it on. > On Mar 31, 2010, at 09:14 AM, Adam Retter wrote: > >> How about combining this with the Balisage dates. I really feel that >> eXist should have representation at Balisage this year, and regardless >> it looks like a pretty cool conference. Perhaps we could bundle the >> two up together? >> >> On 25 March 2010 15:48, Loren Cahlander <lor...@gm...> wrote: >>> >>> On Mar 25, 2010, at 09:24 AM, Wolfgang Meier wrote: >>> >>>>> There was talk of an eXistentialists mini-conference/get together in Minnesota sometime this summer. >>>> >>>> I think Loren is mainly waiting for an answer from me. If I fly to >>>> Minnesota, I would like to combine it with a private visit and/or a >>>> short holiday with my wife. Otherwise, the flight would be too >>>> expensive. I will probably have a chance to clarify this on the >>>> weekend. >>>> >>>> Wolfgang >>> >>> >>> That is correct. If Wolfgang is available to attend, then his schedule will be used to set the date(s) of the gathering. If he is not able to attend, then we will go ahead and make plans for sometime early this summer. >>> >>> It would also help to know how many people would be planning on attending. I know that not having a fixed date makes it difficult to commit, but please let me know directly (through 'reply' and not 'reply all') if you could be attending if schedules work out. It would also help to know what dates people would prefer, what topics they would like covered, and if they have any presentations for the community. >>> >>> Knowing how many people are planning on attending will help us to plan the size and scope of the gathering. >>> >>> I will consolidate all of the feedback and give the community a summary of the responses. >>> >>> Loren >>> +1 952-460-1675 >>> ------------------------------------------------------------------------------ >>> Download Intel® Parallel Studio Eval >>> Try the new software tools for yourself. Speed compiling, find bugs >>> proactively, and fine-tune applications for parallel performance. >>> See why Intel Parallel Studio got high marks during beta. >>> http://p.sf.net/sfu/intel-sw-dev >>> _______________________________________________ >>> Exist-development mailing list >>> Exi...@li... >>> https://lists.sourceforge.net/lists/listinfo/exist-development >>> >> >> >> >> -- >> Adam Retter >> >> eXist Developer >> { United Kingdom } >> ad...@ex... >> irc://irc.freenode.net/existdb > > -- Adam Retter eXist Developer { United Kingdom } ad...@ex... irc://irc.freenode.net/existdb |
From: Loren C. <lor...@gm...> - 2010-03-31 18:33:36
|
I already have some facilities tentatively arranged for in Minnesota. If we were to host this meeting in any other city than our own, then it would be too expensive for us to put it on. On Mar 31, 2010, at 09:14 AM, Adam Retter wrote: > How about combining this with the Balisage dates. I really feel that > eXist should have representation at Balisage this year, and regardless > it looks like a pretty cool conference. Perhaps we could bundle the > two up together? > > On 25 March 2010 15:48, Loren Cahlander <lor...@gm...> wrote: >> >> On Mar 25, 2010, at 09:24 AM, Wolfgang Meier wrote: >> >>>> There was talk of an eXistentialists mini-conference/get together in Minnesota sometime this summer. >>> >>> I think Loren is mainly waiting for an answer from me. If I fly to >>> Minnesota, I would like to combine it with a private visit and/or a >>> short holiday with my wife. Otherwise, the flight would be too >>> expensive. I will probably have a chance to clarify this on the >>> weekend. >>> >>> Wolfgang >> >> >> That is correct. If Wolfgang is available to attend, then his schedule will be used to set the date(s) of the gathering. If he is not able to attend, then we will go ahead and make plans for sometime early this summer. >> >> It would also help to know how many people would be planning on attending. I know that not having a fixed date makes it difficult to commit, but please let me know directly (through 'reply' and not 'reply all') if you could be attending if schedules work out. It would also help to know what dates people would prefer, what topics they would like covered, and if they have any presentations for the community. >> >> Knowing how many people are planning on attending will help us to plan the size and scope of the gathering. >> >> I will consolidate all of the feedback and give the community a summary of the responses. >> >> Loren >> +1 952-460-1675 >> ------------------------------------------------------------------------------ >> Download Intel® Parallel Studio Eval >> Try the new software tools for yourself. Speed compiling, find bugs >> proactively, and fine-tune applications for parallel performance. >> See why Intel Parallel Studio got high marks during beta. >> http://p.sf.net/sfu/intel-sw-dev >> _______________________________________________ >> Exist-development mailing list >> Exi...@li... >> https://lists.sourceforge.net/lists/listinfo/exist-development >> > > > > -- > Adam Retter > > eXist Developer > { United Kingdom } > ad...@ex... > irc://irc.freenode.net/existdb |
From: <jer...@gm...> - 2010-03-31 17:48:02
|
Thank you very much, There was a problem with the chmod & the code. It's working now. Best regards. Le Tue, 30 Mar 2010 17:36:05 +0200, Adam Retter <ad...@ex...> a écrit: > You need to specify the resource name, i.e. - > > import java.io.IOException; > import java.io.UnsupportedEncodingException; > import org.apache.commons.httpclient.HttpClient; > import org.apache.commons.httpclient.methods.PutMethod; > import org.apache.commons.httpclient.methods.RequestEntity; > import org.apache.commons.httpclient.methods.StringRequestEntity; > > public class App > { > public static void main( String[] args ) throws > UnsupportedEncodingException, IOException > { > HttpClient client = new HttpClient(); > PutMethod post = new > PutMethod("http://localhost:8080/exist/rest/db/test.xml"); > > RequestEntity entity = new StringRequestEntity("<hello/>", > "text/xml", "UTF-8"); > post.setRequestEntity(entity); > > int code = client.executeMethod(post); > > System.out.println("Response Code: " + code); > } > } > > On 30 March 2010 14:55, Jérémy Cazaux <jer...@gm...> wrote: >> Hello, >> >> I have a communication problem between android and eXist. I need to send >> information/file from android to eXist. >> >> My terminal in Android should be able to submit files of type XML to >> eXist >> database. I'm using android in VPN. >> I tried using the PUT&GET method of HTTP (i think GET&PUT method are >> aviable >> with eXIst, see=>http://exist.sourceforge.net/devguide_rest.htm ): >> >> private static String url = "http://localhost:8080/exist/rest/db/"; >> >> HttpClient httpclient = new DefaultHttpClient (); >> HttpPut httpput = new HttpPut (url); >> >> try ( >> httpput.setEntity (new StringEntity ( "<the goes here> xml")); >> httpput.setHeader ( "Content-Type", "text / xml"); >> HttpResponse response; >> response = httpclient.execute (httpput); >> >> ) Catch (ClientProtocolException e) ( >> e.printStackTrace (); >> >> ) Catch (IOException e) ( >> e.printStackTrace (); >> ) >> >> I get no result and no error in compilation or execution. >> >> I therefore ask you one example of Java code, or if an explanation of >> the >> "right way" to use. >> >> Thanks you for your help >> >> ------------------------------------------------------------------------------ >> Download Intel® Parallel Studio Eval >> Try the new software tools for yourself. Speed compiling, find bugs >> proactively, and fine-tune applications for parallel performance. >> See why Intel Parallel Studio got high marks during beta. >> http://p.sf.net/sfu/intel-sw-dev >> _______________________________________________ >> Exist-development mailing list >> Exi...@li... >> https://lists.sourceforge.net/lists/listinfo/exist-development >> >> > > > -- Utilisant le client e-mail révolutionnaire d'Opera : http://www.opera.com/mail/ |
From: Andrzej J. T. <an...@ch...> - 2010-03-31 15:44:16
|
Dannes: > Not sure about that statement :-) It took us quite some time to get > off the plain through the customs in Montreal. SFO was clearly > faster..... True...I forgot...Quebec is pretty much a separate country (at least in their own minds). ;-) -- Andrzej Taramina Chaeron Corporation: Enterprise System Solutions http://www.chaeron.com |
From: Dannes W. <da...@ex...> - 2010-03-31 15:34:57
|
Hi, On Wed, Mar 31, 2010 at 4:48 PM, Andrzej Jan Taramina <an...@ch...> wrote: > Given how nasty the US TSA and Homeland security are these days, it's probably easier and less hassle to get > into Canada than the USA anyway. Not sure about that statement :-) It took us quite some time to get off the plain through the customs in Montreal. SFO was clearly faster..... D. -- eXist-db Native XML Database - http://exist-db.org Join us on linked-in: http://www.linkedin.com/groups?gid=35624 |
From: Andrzej J. T. <an...@ch...> - 2010-03-31 14:49:08
|
Adam: > Yes, I did actually mean that we move it to Montreal. Its then perhaps > more accessible for everyone and for those of us that dont live in the > Americas. Given how nasty the US TSA and Homeland security are these days, it's probably easier and less hassle to get into Canada than the USA anyway. Though I must admit, I am looking forward to doing a long ride on one of the Harleys from the greater Toronto area (where I am located) to Minnesota for the mini-conf! ;-) -- Andrzej Taramina Chaeron Corporation: Enterprise System Solutions http://www.chaeron.com |
From: Adam R. <ad...@ex...> - 2010-03-31 14:39:18
|
> I've submitted a presentation proposal ("Open Source XML Databases in Healthcare") to the O'Reilly OSCON conference > (http://www.oscon.com/oscon2010), and intend to submit a similar proposal (though maybe slanted more towards the use of > XML markup in healthcare, HL7 and all that) for Balisage. Good stuff :-) > The problem with that is that Montreal and Minneapolis are pretty far apart. Almost 1300 miles according to Mapquest. > Hey thats pretty close - its 3249 miles from my house to Montreal (by plane)! And I am probably the closest core developer. > Bundling these together would only really work if the venue for the North American eXist mini-conf was changed to > Montreal. Otherwise it's a hell of a long drive or forces people to catch yet another plane flight. Yes, I did actually mean that we move it to Montreal. Its then perhaps more accessible for everyone and for those of us that dont live in the Americas. > -- > Andrzej Taramina > Chaeron Corporation: Enterprise System Solutions > http://www.chaeron.com > -- Adam Retter eXist Developer { United Kingdom } ad...@ex... irc://irc.freenode.net/existdb |
From: Andrzej J. T. <an...@ch...> - 2010-03-31 14:28:02
|
Adam: > Sorry I orginally meant 'collection.xconf' and not 'conf.xml' Oh...OK! That makes a lot more sense and would be a good place to put that kind of stuff! Great idea, Adam! -- Andrzej Taramina Chaeron Corporation: Enterprise System Solutions http://www.chaeron.com |
From: Andrzej J. T. <an...@ch...> - 2010-03-31 14:25:31
|
Adam: > How about combining this with the Balisage dates. I really feel that > eXist should have representation at Balisage this year, and regardless > it looks like a pretty cool conference. I've submitted a presentation proposal ("Open Source XML Databases in Healthcare") to the O'Reilly OSCON conference (http://www.oscon.com/oscon2010), and intend to submit a similar proposal (though maybe slanted more towards the use of XML markup in healthcare, HL7 and all that) for Balisage. So, fingers crossed, we will have some representation specific to eXist at both conferences, albeit specific to my interests in healthcare. > Perhaps we could bundle the > two up together? The problem with that is that Montreal and Minneapolis are pretty far apart. Almost 1300 miles according to Mapquest. Bundling these together would only really work if the venue for the North American eXist mini-conf was changed to Montreal. Otherwise it's a hell of a long drive or forces people to catch yet another plane flight. -- Andrzej Taramina Chaeron Corporation: Enterprise System Solutions http://www.chaeron.com |
From: Adam R. <ad...@ex...> - 2010-03-31 14:15:57
|
On 31 March 2010 15:14, Andrzej Jan Taramina <an...@ch...> wrote: > Adam said: > >> This is quite exciting - I like :-) > > I agree...the concept is intriguing. > >> And the database configuration could again be defined in conf.xml >> files in the /sql/system/sql collections. Caching and connection >> property characteristics could also be placed in conf.xml files in the >> appropriate sql configuration collection. > > I think this is not a feature that will be of general interest to all eXist users, and should not be part of the > core.....an external extension/plugin would make more sense, so that those that want to integrate with a RDBMS can just > install the plugin and get the integration functionality. > > In that regard, I think the configuration should stay out of the main conf.xml and be put somewhere else, specific to > the extension module. Sorry I orginally meant 'collection.xconf' and not 'conf.xml' -- Adam Retter eXist Developer { United Kingdom } ad...@ex... irc://irc.freenode.net/existdb |
From: Adam R. <ad...@ex...> - 2010-03-31 14:14:45
|
How about combining this with the Balisage dates. I really feel that eXist should have representation at Balisage this year, and regardless it looks like a pretty cool conference. Perhaps we could bundle the two up together? On 25 March 2010 15:48, Loren Cahlander <lor...@gm...> wrote: > > On Mar 25, 2010, at 09:24 AM, Wolfgang Meier wrote: > >>> There was talk of an eXistentialists mini-conference/get together in Minnesota sometime this summer. >> >> I think Loren is mainly waiting for an answer from me. If I fly to >> Minnesota, I would like to combine it with a private visit and/or a >> short holiday with my wife. Otherwise, the flight would be too >> expensive. I will probably have a chance to clarify this on the >> weekend. >> >> Wolfgang > > > That is correct. If Wolfgang is available to attend, then his schedule will be used to set the date(s) of the gathering. If he is not able to attend, then we will go ahead and make plans for sometime early this summer. > > It would also help to know how many people would be planning on attending. I know that not having a fixed date makes it difficult to commit, but please let me know directly (through 'reply' and not 'reply all') if you could be attending if schedules work out. It would also help to know what dates people would prefer, what topics they would like covered, and if they have any presentations for the community. > > Knowing how many people are planning on attending will help us to plan the size and scope of the gathering. > > I will consolidate all of the feedback and give the community a summary of the responses. > > Loren > +1 952-460-1675 > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Exist-development mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-development > -- Adam Retter eXist Developer { United Kingdom } ad...@ex... irc://irc.freenode.net/existdb |
From: Andrzej J. T. <an...@ch...> - 2010-03-31 14:14:07
|
Adam said: > This is quite exciting - I like :-) I agree...the concept is intriguing. > And the database configuration could again be defined in conf.xml > files in the /sql/system/sql collections. Caching and connection > property characteristics could also be placed in conf.xml files in the > appropriate sql configuration collection. I think this is not a feature that will be of general interest to all eXist users, and should not be part of the core.....an external extension/plugin would make more sense, so that those that want to integrate with a RDBMS can just install the plugin and get the integration functionality. In that regard, I think the configuration should stay out of the main conf.xml and be put somewhere else, specific to the extension module. My 2 cents worth. > Almost all of the existing infrastructure could be reused. You would > simply need to extend the Collection function so that in the case of a > "/sql" collection it defered the document set building to code that is > very similar to that of the sql module. You would also need to come up > with a collection xml serialization scheme for displaying the > available sql schemas and tables as sub-collections but thus should > not be very hard and should be as close to possible (if not the same) > as the existing ones for the db collections. One thing to keep in mind is that processing of in-memory fragments (which would be created from the xml data coming from the RDBMS) are much slower than native xml documents stored in eXist. Maybe this will provide some impetus to optimize processing of in-memory fragments a bit more, which would be a "good thing". ;-) -- Andrzej Taramina Chaeron Corporation: Enterprise System Solutions http://www.chaeron.com |
From: James F. <jam...@gm...> - 2010-03-31 14:03:31
|
ok this probably has to be a bit lower level then, because we want to make it as natural as possible for end users to apply XQuery ... probably best to work through this via chat. J On Wed, Mar 31, 2010 at 3:55 PM, Adam Retter <ad...@ex...> wrote: > Sure you could create a sql-collection() function using XQuery and > manually manage collection.xconf documents in a virtual hierarchy like > /db/system/sql/db/... > > On 31 March 2010 14:43, James Fuller <jam...@ex...> wrote: >> >> >> On 31 March 2010 15:03, Adam Retter <ad...@ex...> wrote: >>> >>> On 31 March 2010 06:45, James Fuller <jam...@ex...> wrote: >>> > whilst doing xproc development, I have had great fun integrating some of >>> > eXist existing extension functionality as extension steps ... a thought >>> > occurred to me about the current state of integration with RDBMS. >>> > Currently we have the nice SQL extension module (written by Adam Retter) >>> > which uses JDBC to query a relational database ... this means we can >>> > easily >>> > query an RDBMS through xquery using SQL statements. I think this module >>> > enables a lot of scenarios but misses off the primary scenario for the >>> > masses of people out there with data currently existing in RDBMS but >>> > want to >>> > use XML database. >>> > I would like to >>> > install eXist and provide in some registry a connection string which >>> > effectively 'mounts' the RDBMS as a collection ... from which here on >>> > out I >>> > can use pure xquery to work on. For now a read only view of the data is >>> > all >>> > I need but someday I would like to be able to do updates as well on the >>> > RDBMS data. Lastly I should be able to configure cache characteristics >>> > if >>> > the database connection is intermittent' >>> >>> Phew!!! For a moment then I thought you were going to ask for a SQL >>> layer atop eXist-db and a JDBC driver for eXist-db. >>> >>> This would actually be pretty easy to achieve :-) >>> >>> Imagine the collection namespace root starting at /sql as opposed to /db. >>> >>> Configuration could go into >>> /sql/system/sql/{database}/{schemaName}/{tableName} >>> >>> And the database configuration could again be defined in conf.xml >>> files in the /sql/system/sql collections. Caching and connection >>> property characteristics could also be placed in conf.xml files in the >>> appropriate sql configuration collection. >>> >>> Almost all of the existing infrastructure could be reused. You would >>> simply need to extend the Collection function so that in the case of a >>> "/sql" collection it defered the document set building to code that is >>> very similar to that of the sql module. You would also need to come up >>> with a collection xml serialization scheme for displaying the >>> available sql schemas and tables as sub-collections but thus should >>> not be very hard and should be as close to possible (if not the same) >>> as the existing ones for the db collections. >>> >>> This is quite exciting - I like :-) >>> >>> Can I help? >>> >> >> thx Adam for teasing out the detail ... this is a cool way to approach it >> ... I would appreciate a chat to discuss; my initial thoughts were to create >> an XQuery app (using sql extension module) to achieve the above ... its a >> bit high level for my tastes for this kind of thing, but perhaps the best >> first step. >> J >> J > > > > -- > Adam Retter > > eXist Developer > { United Kingdom } > ad...@ex... > irc://irc.freenode.net/existdb > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Exist-development mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-development > |
From: Adam R. <ad...@ex...> - 2010-03-31 13:55:44
|
Sure you could create a sql-collection() function using XQuery and manually manage collection.xconf documents in a virtual hierarchy like /db/system/sql/db/... On 31 March 2010 14:43, James Fuller <jam...@ex...> wrote: > > > On 31 March 2010 15:03, Adam Retter <ad...@ex...> wrote: >> >> On 31 March 2010 06:45, James Fuller <jam...@ex...> wrote: >> > whilst doing xproc development, I have had great fun integrating some of >> > eXist existing extension functionality as extension steps ... a thought >> > occurred to me about the current state of integration with RDBMS. >> > Currently we have the nice SQL extension module (written by Adam Retter) >> > which uses JDBC to query a relational database ... this means we can >> > easily >> > query an RDBMS through xquery using SQL statements. I think this module >> > enables a lot of scenarios but misses off the primary scenario for the >> > masses of people out there with data currently existing in RDBMS but >> > want to >> > use XML database. >> > I would like to >> > install eXist and provide in some registry a connection string which >> > effectively 'mounts' the RDBMS as a collection ... from which here on >> > out I >> > can use pure xquery to work on. For now a read only view of the data is >> > all >> > I need but someday I would like to be able to do updates as well on the >> > RDBMS data. Lastly I should be able to configure cache characteristics >> > if >> > the database connection is intermittent' >> >> Phew!!! For a moment then I thought you were going to ask for a SQL >> layer atop eXist-db and a JDBC driver for eXist-db. >> >> This would actually be pretty easy to achieve :-) >> >> Imagine the collection namespace root starting at /sql as opposed to /db. >> >> Configuration could go into >> /sql/system/sql/{database}/{schemaName}/{tableName} >> >> And the database configuration could again be defined in conf.xml >> files in the /sql/system/sql collections. Caching and connection >> property characteristics could also be placed in conf.xml files in the >> appropriate sql configuration collection. >> >> Almost all of the existing infrastructure could be reused. You would >> simply need to extend the Collection function so that in the case of a >> "/sql" collection it defered the document set building to code that is >> very similar to that of the sql module. You would also need to come up >> with a collection xml serialization scheme for displaying the >> available sql schemas and tables as sub-collections but thus should >> not be very hard and should be as close to possible (if not the same) >> as the existing ones for the db collections. >> >> This is quite exciting - I like :-) >> >> Can I help? >> > > thx Adam for teasing out the detail ... this is a cool way to approach it > ... I would appreciate a chat to discuss; my initial thoughts were to create > an XQuery app (using sql extension module) to achieve the above ... its a > bit high level for my tastes for this kind of thing, but perhaps the best > first step. > J > J -- Adam Retter eXist Developer { United Kingdom } ad...@ex... irc://irc.freenode.net/existdb |
From: Dan M. <dan...@gm...> - 2010-03-31 13:55:18
|
Great idea Jim! Your idea is another example of how we can make eXist more attractive for people that are stuck with older legacy RDBMS systems. Here is one other line of thought about how eXist might integrate with an RDBMS. Right now this was just a "design" and we never got full funding to finish the proof-of-concept. But I wanted to at least share our thinking. In the past I did quite a bit of work with porting XRX applications between eXist and DB2. Using eXist was easy but setting up a WebDAV connection so that we could make DB2 looks like an XML file system was a lot of work! We wanted to make it so simple for people do do drag-and-drop to import XML files into our system, but some of the collections in our WebDAV view would really be in a DB2 table. One thing we did was to try to map the notion of a "collection" into an RDBMS system. Here was our goal: We wanted eXist to "Create a new Collection but put the XML files in the DB2 system". We wanted our XQueries in eXist to access saved form data on a remote system. We wanted to keep our XQuery code portable free from *SQL pollution* but still use data in a remote RDBMS supported by another vendor. One design option we looked was to try to make specific *collections *map into an RDBMS table with a single column that contained XML data. Each XML file was then mapped to a row n the RDBMS table that had five columns. The time-stamps and users that created and updated the XML file and a single column for the RDBMS XML datatype. We could then think of any XML collection like: /db/db2/my_collection that *really *was running on DB2 in a SQL table called "my_collection" using the very fast IBM PureXML "viper" 9.1 system that does do nice work with the XML datatype and supported XQuery. We wanted the systems to sync up so that the b+trees on eXist would get updated with the RDBMS data was changed. That would have duplicated the indexes but we did not think that the disk space was a problem. IBM software was fast but the hoops we had to jump through to run XQuery with SQL made it 10x harder to work with. That was several years ago but I still think that eXist has a much cleaner XQuery interface. I should also note that we were asked (but not funded) if we could do this same thing with Microsoft-SQL server since the license costs were about 1/4th of the DB2 costs. I imagine that using the Oracle XML column type we could also do a similar thing. Most of this work was very low-level Java coding using the DB2 interfaces and I did not do the Java work myself. It was slow hard work and our little proof-of-architecture budget did not support it. But it was fun to think about how the system might work together and appear to the XQuery developer as a single seamless entity. The only restriction would be that distributed joins would be very slow. I don't know how difficult it would be to intercept all XQueries for a specific collection and route them to a remote RDBMS and get the results back. But if we did have a collection RDBMS collection plugin it might be an interesting test. Most of the work then like it would need to be done on each RDBMS. - Dan 2010/3/31 James Fuller <jam...@ex...> > 2010/3/31 Leif-Jöran Olsson <lj...@ex...> > >> Just remember that for a newbie this works best with non-structural data >> in the rdb fields. Otherwise the data massaging (possibly) needed to >> make it well-formed can as well scare them away. >> > > good point Leif, > > will need to apply 'principle of least surprise' everywhere possible with > this, lets see how the prototype I am cooking up goes, it maybe that high > level impl is enough as a 'taster' for people, > > J > > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Exist-development mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-development > > -- Dan McCreary Semantic Solutions Architect syntactica.com 952-460-1674 VOIP: 111@69.199.167.229 |
From: Thomas W. <tho...@gm...> - 2010-03-31 13:53:14
|
I was wandering if anybody can have a look at this issue. It is pretty serious to me because after a second file import eXist goes out of sync between the binary resource listed internally and the files that actually are stored in the file system at the location they are expected. And second, the fact that files stay in the file system after collections deletions is a security issue as well. I really hope somebody will be able to go to the bottom of this problem. Thomas ------ Thomas White Mobile:+44 7711 922 966 Skype: thomaswhite gTalk: thomas.0007 Linked-In:http://www.linkedin.com/in/thomaswhite0007 facebook: http://www.facebook.com/thomas.0007 On 25 March 2010 17:28, Thomas White <tho...@gm...> wrote: > EXIST_HOME is D:\eXist2 > JAVA SDK HOME C:\Java\jdk1.6.0_16 > > Has anyone been able to reproduce this bug? > > Thomas > |
From: James F. <jam...@ex...> - 2010-03-31 13:43:59
|
On 31 March 2010 15:03, Adam Retter <ad...@ex...> wrote: > On 31 March 2010 06:45, James Fuller <jam...@ex...> wrote: > > whilst doing xproc development, I have had great fun integrating some of > > eXist existing extension functionality as extension steps ... a thought > > occurred to me about the current state of integration with RDBMS. > > Currently we have the nice SQL extension module (written by Adam Retter) > > which uses JDBC to query a relational database ... this means we can > easily > > query an RDBMS through xquery using SQL statements. I think this module > > enables a lot of scenarios but misses off the primary scenario for the > > masses of people out there with data currently existing in RDBMS but want > to > > use XML database. > > I would like to > > install eXist and provide in some registry a connection string which > > effectively 'mounts' the RDBMS as a collection ... from which here on out > I > > can use pure xquery to work on. For now a read only view of the data is > all > > I need but someday I would like to be able to do updates as well on the > > RDBMS data. Lastly I should be able to configure cache characteristics if > > the database connection is intermittent' > > Phew!!! For a moment then I thought you were going to ask for a SQL > layer atop eXist-db and a JDBC driver for eXist-db. > > This would actually be pretty easy to achieve :-) > > Imagine the collection namespace root starting at /sql as opposed to /db. > > Configuration could go into > /sql/system/sql/{database}/{schemaName}/{tableName} > > And the database configuration could again be defined in conf.xml > files in the /sql/system/sql collections. Caching and connection > property characteristics could also be placed in conf.xml files in the > appropriate sql configuration collection. > > Almost all of the existing infrastructure could be reused. You would > simply need to extend the Collection function so that in the case of a > "/sql" collection it defered the document set building to code that is > very similar to that of the sql module. You would also need to come up > with a collection xml serialization scheme for displaying the > available sql schemas and tables as sub-collections but thus should > not be very hard and should be as close to possible (if not the same) > as the existing ones for the db collections. > > This is quite exciting - I like :-) > > Can I help? > > thx Adam for teasing out the detail ... this is a cool way to approach it ... I would appreciate a chat to discuss; my initial thoughts were to create an XQuery app (using sql extension module) to achieve the above ... its a bit high level for my tastes for this kind of thing, but perhaps the best first step. J J |
From: Adam R. <ad...@ex...> - 2010-03-31 13:03:43
|
On 31 March 2010 06:45, James Fuller <jam...@ex...> wrote: > whilst doing xproc development, I have had great fun integrating some of > eXist existing extension functionality as extension steps ... a thought > occurred to me about the current state of integration with RDBMS. > Currently we have the nice SQL extension module (written by Adam Retter) > which uses JDBC to query a relational database ... this means we can easily > query an RDBMS through xquery using SQL statements. I think this module > enables a lot of scenarios but misses off the primary scenario for the > masses of people out there with data currently existing in RDBMS but want to > use XML database. > I would like to > install eXist and provide in some registry a connection string which > effectively 'mounts' the RDBMS as a collection ... from which here on out I > can use pure xquery to work on. For now a read only view of the data is all > I need but someday I would like to be able to do updates as well on the > RDBMS data. Lastly I should be able to configure cache characteristics if > the database connection is intermittent' Phew!!! For a moment then I thought you were going to ask for a SQL layer atop eXist-db and a JDBC driver for eXist-db. This would actually be pretty easy to achieve :-) Imagine the collection namespace root starting at /sql as opposed to /db. Configuration could go into /sql/system/sql/{database}/{schemaName}/{tableName} And the database configuration could again be defined in conf.xml files in the /sql/system/sql collections. Caching and connection property characteristics could also be placed in conf.xml files in the appropriate sql configuration collection. Almost all of the existing infrastructure could be reused. You would simply need to extend the Collection function so that in the case of a "/sql" collection it defered the document set building to code that is very similar to that of the sql module. You would also need to come up with a collection xml serialization scheme for displaying the available sql schemas and tables as sub-collections but thus should not be very hard and should be as close to possible (if not the same) as the existing ones for the db collections. This is quite exciting - I like :-) Can I help? > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Exist-development mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-development > > -- Adam Retter eXist Developer { United Kingdom } ad...@ex... irc://irc.freenode.net/existdb |
From: James F. <jam...@ex...> - 2010-03-31 09:57:20
|
2010/3/31 Leif-Jöran Olsson <lj...@ex...> > Just remember that for a newbie this works best with non-structural data > in the rdb fields. Otherwise the data massaging (possibly) needed to > make it well-formed can as well scare them away. > good point Leif, will need to apply 'principle of least surprise' everywhere possible with this, lets see how the prototype I am cooking up goes, it maybe that high level impl is enough as a 'taster' for people, J |
From: Leif-Jöran O. <lj...@ex...> - 2010-03-31 09:52:09
|
Just remember that for a newbie this works best with non-structural data in the rdb fields. Otherwise the data massaging (possibly) needed to make it well-formed can as well scare them away. Leif-Jöran Den 2010-03-31 09:52, James Fuller skrev: > thx Thomas, all good thinking > > in the meantime I am going to do a proof of concept using sql extension > and xquery > > J > > > On 31 March 2010 09:19, Thomas White <tho...@gm... > <mailto:tho...@gm...>> wrote: > > James, > > I like your idea - it lowers the resistance about moving away from > the existing investments of time, code and money in the existing RDBMS. > > As always the first thing first - we need to name what we are doing. > I propose the working name "*XML gateway to RDBMS*". When we refer > to this functionality we can say *"eXist is a XML gateway to > existing RDBMS".* This sounds as a reasonable USP (Unique Selling > Proposition) as well. > > Let us explore the idea further: > Step 1. Mount a RDBMS as a collection. We may need to use a special > prefix to the mounterd name. > Step 2. Provide means for enumerating the existing db resources: > > * Enumerate all tables and views. > * Get the metadata for each table in two formats: a) as XML > document, b) as a schema document. We can use this to generate > XQuery for quering data. > * A function that returns data from a table receiving the name > of the table, a list of field names and number of records. > This will allow the newbies coming from RDBMS world to start > using eXist right out of the box before they become familiar > with XQuery. > > Step 3. We can even provide a web front end that will allow > developers with no XQuery experience to select the tables and fields > they need and then generate and save an XQuery for extracting the > data they need, giving them an URI that returns the data. This could > be very attractive functionality. > > Any other ideas? > > Thomas > > > ------ > > Thomas White > > Mobile:+44 7711 922 966 > Skype: thomaswhite > gTalk: thomas.0007 > Linked-In:http://www.linkedin.com/in/thomaswhite0007 > facebook: http://www.facebook.com/thomas.0007 > > > > > On 31 March 2010 06:45, James Fuller <jam...@ex... > <mailto:jam...@ex...>> wrote: > > whilst doing xproc development, I have had great fun integrating > some of > > eXist existing extension functionality as extension steps ... a > thought > > occurred to me about the current state of integration with RDBMS. > > Currently we have the nice SQL extension module (written by Adam > Retter) > > which uses JDBC to query a relational database ... this means we > can easily > > query an RDBMS through xquery using SQL statements. I think this > module > > enables a lot of scenarios but misses off the primary scenario for the > > masses of people out there with data currently existing in RDBMS > but want to > > use XML database. > > Take the following imaginary developer scenario: > > 'I have an existing RDBMS and I want to use XML database ... I am > worried > > about going 'all xml' because RDBMS has served me well, not to > mention I > > have a lot of code hanging off the existing database ... if I > migrate to an > > XML database I will have to migrate all those other bits ... this > is too > > hard and too much change to attempt in one risky step. I would like to > > install eXist and provide in some registry a connection string which > > effectively 'mounts' the RDBMS as a collection ... from which here > on out I > > can use pure xquery to work on. For now a read only view of the > data is all > > I need but someday I would like to be able to do updates as well > on the > > RDBMS data. Lastly I should be able to configure cache > characteristics if > > the database connection is intermittent' > > Thoughts ? Does anyone see this scenario as something that needs to be > > addressed ? > > James Fuller > > > > > ------------------------------------------------------------------------------ > > Download Intel® Parallel Studio Eval > > Try the new software tools for yourself. Speed compiling, find bugs > > proactively, and fine-tune applications for parallel performance. > > See why Intel Parallel Studio got high marks during beta. > > http://p.sf.net/sfu/intel-sw-dev > > _______________________________________________ > > Exist-development mailing list > > Exi...@li... > <mailto:Exi...@li...> > > https://lists.sourceforge.net/lists/listinfo/exist-development > > > > > > > > > > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > > > > _______________________________________________ > Exist-development mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-development |
From: James F. <jam...@ex...> - 2010-03-31 07:52:21
|
thx Thomas, all good thinking in the meantime I am going to do a proof of concept using sql extension and xquery J On 31 March 2010 09:19, Thomas White <tho...@gm...> wrote: > James, > > I like your idea - it lowers the resistance about moving away from the > existing investments of time, code and money in the existing RDBMS. > > As always the first thing first - we need to name what we are doing. I > propose the working name "*XML gateway to RDBMS*". When we refer to this > functionality we can say *"eXist is a XML gateway to existing RDBMS".*This sounds as a reasonable USP (Unique Selling Proposition) as well. > > Let us explore the idea further: > Step 1. Mount a RDBMS as a collection. We may need to use a special prefix > to the mounterd name. > Step 2. Provide means for enumerating the existing db resources: > > - Enumerate all tables and views. > - Get the metadata for each table in two formats: a) as XML document, > b) as a schema document. We can use this to generate XQuery for quering > data. > - A function that returns data from a table receiving the name of the > table, a list of field names and number of records. This will allow the > newbies coming from RDBMS world to start using eXist right out of the box > before they become familiar with XQuery. > > Step 3. We can even provide a web front end that will allow developers with > no XQuery experience to select the tables and fields they need and then > generate and save an XQuery for extracting the data they need, giving them > an URI that returns the data. This could be very attractive functionality. > > Any other ideas? > > Thomas > > > ------ > > Thomas White > > Mobile:+44 7711 922 966 > Skype: thomaswhite > gTalk: thomas.0007 > Linked-In:http://www.linkedin.com/in/thomaswhite0007 > facebook: http://www.facebook.com/thomas.0007 > > > > > On 31 March 2010 06:45, James Fuller <jam...@ex...> wrote: > > whilst doing xproc development, I have had great fun integrating some of > > eXist existing extension functionality as extension steps ... a thought > > occurred to me about the current state of integration with RDBMS. > > Currently we have the nice SQL extension module (written by Adam Retter) > > which uses JDBC to query a relational database ... this means we can > easily > > query an RDBMS through xquery using SQL statements. I think this module > > enables a lot of scenarios but misses off the primary scenario for the > > masses of people out there with data currently existing in RDBMS but want > to > > use XML database. > > Take the following imaginary developer scenario: > > 'I have an existing RDBMS and I want to use XML database ... I am worried > > about going 'all xml' because RDBMS has served me well, not to mention I > > have a lot of code hanging off the existing database ... if I migrate to > an > > XML database I will have to migrate all those other bits ... this is too > > hard and too much change to attempt in one risky step. I would like to > > install eXist and provide in some registry a connection string which > > effectively 'mounts' the RDBMS as a collection ... from which here on out > I > > can use pure xquery to work on. For now a read only view of the data is > all > > I need but someday I would like to be able to do updates as well on the > > RDBMS data. Lastly I should be able to configure cache characteristics if > > the database connection is intermittent' > > Thoughts ? Does anyone see this scenario as something that needs to be > > addressed ? > > James Fuller > > > > > ------------------------------------------------------------------------------ > > Download Intel® Parallel Studio Eval > > Try the new software tools for yourself. Speed compiling, find bugs > > proactively, and fine-tune applications for parallel performance. > > See why Intel Parallel Studio got high marks during beta. > > http://p.sf.net/sfu/intel-sw-dev > > _______________________________________________ > > Exist-development mailing list > > Exi...@li... > > https://lists.sourceforge.net/lists/listinfo/exist-development > > > > > > > |
From: Thomas W. <tho...@gm...> - 2010-03-31 07:20:17
|
James, I like your idea - it lowers the resistance about moving away from the existing investments of time, code and money in the existing RDBMS. As always the first thing first - we need to name what we are doing. I propose the working name "*XML gateway to RDBMS*". When we refer to this functionality we can say *"eXist is a XML gateway to existing RDBMS".* This sounds as a reasonable USP (Unique Selling Proposition) as well. Let us explore the idea further: Step 1. Mount a RDBMS as a collection. We may need to use a special prefix to the mounterd name. Step 2. Provide means for enumerating the existing db resources: - Enumerate all tables and views. - Get the metadata for each table in two formats: a) as XML document, b) as a schema document. We can use this to generate XQuery for quering data. - A function that returns data from a table receiving the name of the table, a list of field names and number of records. This will allow the newbies coming from RDBMS world to start using eXist right out of the box before they become familiar with XQuery. Step 3. We can even provide a web front end that will allow developers with no XQuery experience to select the tables and fields they need and then generate and save an XQuery for extracting the data they need, giving them an URI that returns the data. This could be very attractive functionality. Any other ideas? Thomas ------ Thomas White Mobile:+44 7711 922 966 Skype: thomaswhite gTalk: thomas.0007 Linked-In:http://www.linkedin.com/in/thomaswhite0007 facebook: http://www.facebook.com/thomas.0007 On 31 March 2010 06:45, James Fuller <jam...@ex...> wrote: > whilst doing xproc development, I have had great fun integrating some of > eXist existing extension functionality as extension steps ... a thought > occurred to me about the current state of integration with RDBMS. > Currently we have the nice SQL extension module (written by Adam Retter) > which uses JDBC to query a relational database ... this means we can easily > query an RDBMS through xquery using SQL statements. I think this module > enables a lot of scenarios but misses off the primary scenario for the > masses of people out there with data currently existing in RDBMS but want to > use XML database. > Take the following imaginary developer scenario: > 'I have an existing RDBMS and I want to use XML database ... I am worried > about going 'all xml' because RDBMS has served me well, not to mention I > have a lot of code hanging off the existing database ... if I migrate to an > XML database I will have to migrate all those other bits ... this is too > hard and too much change to attempt in one risky step. I would like to > install eXist and provide in some registry a connection string which > effectively 'mounts' the RDBMS as a collection ... from which here on out I > can use pure xquery to work on. For now a read only view of the data is all > I need but someday I would like to be able to do updates as well on the > RDBMS data. Lastly I should be able to configure cache characteristics if > the database connection is intermittent' > Thoughts ? Does anyone see this scenario as something that needs to be > addressed ? > James Fuller > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Exist-development mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-development > > |
From: Evgeny G. <gaz...@gm...> - 2010-03-31 05:53:56
|
> I would like to install eXist and provide in some registry a connection string which > effectively 'mounts' the RDBMS as a collection ... from which here on out I > can use pure xquery to work on. For now a read only view of the data is all > I need but someday I would like to be able to do updates as well on the > RDBMS data. Lastly I should be able to configure cache characteristics if > the database connection is intermittent' > Thoughts ? Does anyone see this scenario as something that needs to be > addressed ? > James Fuller > This is great idea to 'mounts' the RDBMS as a collection! ---------------- Evgeny |