exist-development

[Exist-development] thoughts about next steps with RDBMS integration

[James F. <jam...@ex...>]

whilst doing xproc development, I have had great fun integrating some of
eXist existing extension functionality as extension steps ... a thought
occurred to me about the current state of integration with RDBMS.

Currently we have the nice SQL extension module (written by Adam Retter)
which uses JDBC to query a relational database ... this means we can easily
query an RDBMS through xquery using SQL statements. I think this module
enables a lot of scenarios but misses off the primary scenario for the
masses of people out there with data currently existing in RDBMS but want to
use XML database.

Take the following imaginary developer scenario:

'I have an existing RDBMS and I want to use XML database ... I am worried
about going 'all xml' because RDBMS has served me well, not to mention I
have a lot of code hanging off the existing database ... if I migrate to an
XML database I will have to migrate all those other bits ... this is too
hard and too much change to attempt in one risky step. I would like to
install eXist and provide in some registry a connection string which
effectively 'mounts' the RDBMS as a collection ... from which here on out I
can use pure xquery to work on. For now a read only view of the data is all
I need but someday I would like to be able to do updates as well on the
RDBMS data. Lastly I should be able to configure cache characteristics if
the database connection is intermittent'

Thoughts ? Does anyone see this scenario as something that needs to be
addressed ?

James Fuller

Re: [Exist-development] thoughts about next steps with RDBMS integration

[Evgeny G. <gaz...@gm...>]

> I would like to install eXist and provide in some registry a connection string which
> effectively 'mounts' the RDBMS as a collection ... from which here on out I
> can use pure xquery to work on. For now a read only view of the data is all
> I need but someday I would like to be able to do updates as well on the
> RDBMS data. Lastly I should be able to configure cache characteristics if
> the database connection is intermittent'
> Thoughts ? Does anyone see this scenario as something that needs to be
> addressed ?
> James Fuller
>
This is great idea to 'mounts' the RDBMS as a collection!

----------------
Evgeny

Re: [Exist-development] thoughts about next steps with RDBMS integration

[Thomas W. <tho...@gm...>]

James,

I like your idea - it lowers the resistance about moving away from the
existing investments of time, code and money in the existing RDBMS.

As always the first thing first - we need to name what we are doing. I
propose the working name "*XML gateway to RDBMS*". When we refer to this
functionality we can say *"eXist is a XML gateway to existing RDBMS".* This
sounds as a reasonable USP (Unique Selling Proposition) as well.

Let us explore the idea further:
Step 1. Mount a RDBMS as a collection. We may need to use a special prefix
to the mounterd name.
Step 2. Provide means for enumerating the existing db resources:

   - Enumerate all tables and views.
   - Get the metadata for each table in two formats: a) as XML document, b)
   as a schema document. We can use this to generate XQuery for quering data.
   - A function that returns data from a table receiving the name of the
   table, a list of field names and number of records. This will allow the
   newbies coming from RDBMS world to start using eXist right out of the box
   before they become familiar with XQuery.

Step 3. We can even provide a web front end that will allow developers with
no XQuery experience to select the tables and fields they need and then
generate and save an XQuery for extracting the data they need, giving them
an URI that returns the data. This could be very attractive functionality.

Any other ideas?

Thomas


------

Thomas White

Mobile:+44 7711 922 966
Skype: thomaswhite
gTalk: thomas.0007
Linked-In:http://www.linkedin.com/in/thomaswhite0007
facebook: http://www.facebook.com/thomas.0007




On 31 March 2010 06:45, James Fuller <jam...@ex...> wrote:
> whilst doing xproc development, I have had great fun integrating some of
> eXist existing extension functionality as extension steps ... a thought
> occurred to me about the current state of integration with RDBMS.
> Currently we have the nice SQL extension module (written by Adam Retter)
> which uses JDBC to query a relational database ... this means we can
easily
> query an RDBMS through xquery using SQL statements. I think this module
> enables a lot of scenarios but misses off the primary scenario for the
> masses of people out there with data currently existing in RDBMS but want
to
> use XML database.
> Take the following imaginary developer scenario:
> 'I have an existing RDBMS and I want to use XML database ... I am worried
> about going 'all xml' because RDBMS has served me well, not to mention I
> have a lot of code hanging off the existing database ... if I migrate to
an
> XML database I will have to migrate all those other bits ... this is too
> hard and too much change to attempt in one risky step. I would like to
> install eXist and provide in some registry a connection string which
> effectively 'mounts' the RDBMS as a collection ... from which here on out
I
> can use pure xquery to work on. For now a read only view of the data is
all
> I need but someday I would like to be able to do updates as well on the
> RDBMS data. Lastly I should be able to configure cache characteristics if
> the database connection is intermittent'
> Thoughts ? Does anyone see this scenario as something that needs to be
> addressed ?
> James Fuller
>
>
------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Exist-development mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-development
>
>

Re: [Exist-development] thoughts about next steps with RDBMS integration

[James F. <jam...@ex...>]

thx Thomas, all good thinking

in the meantime I am going to do a proof of concept using sql extension and
xquery

J


On 31 March 2010 09:19, Thomas White <tho...@gm...> wrote:

> James,
>
> I like your idea - it lowers the resistance about moving away from the
> existing investments of time, code and money in the existing RDBMS.
>
> As always the first thing first - we need to name what we are doing. I
> propose the working name "*XML gateway to RDBMS*". When we refer to this
> functionality we can say *"eXist is a XML gateway to existing RDBMS".*This sounds as a reasonable USP (Unique Selling Proposition) as well.
>
> Let us explore the idea further:
> Step 1. Mount a RDBMS as a collection. We may need to use a special prefix
> to the mounterd name.
> Step 2. Provide means for enumerating the existing db resources:
>
>    - Enumerate all tables and views.
>    - Get the metadata for each table in two formats: a) as XML document,
>    b) as a schema document. We can use this to generate XQuery for quering
>    data.
>    - A function that returns data from a table receiving the name of the
>    table, a list of field names and number of records. This will allow the
>    newbies coming from RDBMS world to start using eXist right out of the box
>    before they become familiar with XQuery.
>
> Step 3. We can even provide a web front end that will allow developers with
> no XQuery experience to select the tables and fields they need and then
> generate and save an XQuery for extracting the data they need, giving them
> an URI that returns the data. This could be very attractive functionality.
>
> Any other ideas?
>
> Thomas
>
>
> ------
>
> Thomas White
>
> Mobile:+44 7711 922 966
> Skype: thomaswhite
> gTalk: thomas.0007
> Linked-In:http://www.linkedin.com/in/thomaswhite0007
> facebook: http://www.facebook.com/thomas.0007
>
>
>
>
> On 31 March 2010 06:45, James Fuller <jam...@ex...> wrote:
> > whilst doing xproc development, I have had great fun integrating some of
> > eXist existing extension functionality as extension steps ... a thought
> > occurred to me about the current state of integration with RDBMS.
> > Currently we have the nice SQL extension module (written by Adam Retter)
> > which uses JDBC to query a relational database ... this means we can
> easily
> > query an RDBMS through xquery using SQL statements. I think this module
> > enables a lot of scenarios but misses off the primary scenario for the
> > masses of people out there with data currently existing in RDBMS but want
> to
> > use XML database.
> > Take the following imaginary developer scenario:
> > 'I have an existing RDBMS and I want to use XML database ... I am worried
> > about going 'all xml' because RDBMS has served me well, not to mention I
> > have a lot of code hanging off the existing database ... if I migrate to
> an
> > XML database I will have to migrate all those other bits ... this is too
> > hard and too much change to attempt in one risky step. I would like to
> > install eXist and provide in some registry a connection string which
> > effectively 'mounts' the RDBMS as a collection ... from which here on out
> I
> > can use pure xquery to work on. For now a read only view of the data is
> all
> > I need but someday I would like to be able to do updates as well on the
> > RDBMS data. Lastly I should be able to configure cache characteristics if
> > the database connection is intermittent'
> > Thoughts ? Does anyone see this scenario as something that needs to be
> > addressed ?
> > James Fuller
> >
> >
> ------------------------------------------------------------------------------
> > Download Intel&#174; Parallel Studio Eval
> > Try the new software tools for yourself. Speed compiling, find bugs
> > proactively, and fine-tune applications for parallel performance.
> > See why Intel Parallel Studio got high marks during beta.
> > http://p.sf.net/sfu/intel-sw-dev
> > _______________________________________________
> > Exist-development mailing list
> > Exi...@li...
> > https://lists.sourceforge.net/lists/listinfo/exist-development
> >
> >
>
>
>

Re: [Exist-development] thoughts about next steps with RDBMS integration

[Leif-Jöran O. <lj...@ex...>]

Just remember that for a newbie this works best with non-structural data
in the rdb fields. Otherwise the data massaging (possibly) needed  to
make it well-formed can as well scare them away.

Leif-Jöran

Den 2010-03-31 09:52, James Fuller skrev:
> thx Thomas, all good thinking
> 
> in the meantime I am going to do a proof of concept using sql extension
> and xquery
> 
> J
> 
> 
> On 31 March 2010 09:19, Thomas White <tho...@gm...
> <mailto:tho...@gm...>> wrote:
> 
>     James,
>      
>     I like your idea - it lowers the resistance about moving away from
>     the existing investments of time, code and money in the existing RDBMS.
>      
>     As always the first thing first - we need to name what we are doing.
>     I propose the working name "*XML gateway to RDBMS*". When we refer
>     to this functionality we can say *"eXist is a XML gateway to
>     existing RDBMS".* This sounds as a reasonable USP (Unique Selling
>     Proposition) as well.
>      
>     Let us explore the idea further:
>     Step 1. Mount a RDBMS as a collection. We may need to use a special
>     prefix to the mounterd name.
>     Step 2. Provide means for enumerating the existing db resources:
> 
>         * Enumerate all tables and views.
>         * Get the metadata for each table in two formats: a) as XML
>           document, b) as a schema document. We can use this to generate
>           XQuery for quering data.
>         * A function that returns data from a table receiving the name
>           of the table, a list of field names and number of records.
>           This will allow the newbies coming from RDBMS world to start
>           using eXist right out of the box before they become familiar
>           with XQuery.
> 
>     Step 3. We can even provide a web front end that will allow
>     developers with no XQuery experience to select the tables and fields
>     they need and then generate and save an XQuery for extracting the
>     data they need, giving them an URI that returns the data. This could
>     be very attractive functionality.
>      
>     Any other ideas?
>      
>     Thomas
>      
>      
>     ------
>      
>     Thomas White
>      
>     Mobile:+44 7711 922 966
>     Skype: thomaswhite
>     gTalk: thomas.0007
>     Linked-In:http://www.linkedin.com/in/thomaswhite0007
>     facebook: http://www.facebook.com/thomas.0007
>      
>      
>      
>      
>     On 31 March 2010 06:45, James Fuller <jam...@ex...
>     <mailto:jam...@ex...>> wrote:
>     > whilst doing xproc development, I have had great fun integrating
>     some of
>     > eXist existing extension functionality as extension steps ... a
>     thought
>     > occurred to me about the current state of integration with RDBMS.
>     > Currently we have the nice SQL extension module (written by Adam
>     Retter)
>     > which uses JDBC to query a relational database ... this means we
>     can easily
>     > query an RDBMS through xquery using SQL statements. I think this
>     module
>     > enables a lot of scenarios but misses off the primary scenario for the
>     > masses of people out there with data currently existing in RDBMS
>     but want to
>     > use XML database.
>     > Take the following imaginary developer scenario:
>     > 'I have an existing RDBMS and I want to use XML database ... I am
>     worried
>     > about going 'all xml' because RDBMS has served me well, not to
>     mention I
>     > have a lot of code hanging off the existing database ... if I
>     migrate to an
>     > XML database I will have to migrate all those other bits ... this
>     is too
>     > hard and too much change to attempt in one risky step. I would like to
>     > install eXist and provide in some registry a connection string which
>     > effectively 'mounts' the RDBMS as a collection ... from which here
>     on out I
>     > can use pure xquery to work on. For now a read only view of the
>     data is all
>     > I need but someday I would like to be able to do updates as well
>     on the
>     > RDBMS data. Lastly I should be able to configure cache
>     characteristics if
>     > the database connection is intermittent'
>     > Thoughts ? Does anyone see this scenario as something that needs to be
>     > addressed ?
>     > James Fuller
>     >
>     >
>     ------------------------------------------------------------------------------
>     > Download Intel&#174; Parallel Studio Eval
>     > Try the new software tools for yourself. Speed compiling, find bugs
>     > proactively, and fine-tune applications for parallel performance.
>     > See why Intel Parallel Studio got high marks during beta.
>     > http://p.sf.net/sfu/intel-sw-dev
>     > _______________________________________________
>     > Exist-development mailing list
>     > Exi...@li...
>     <mailto:Exi...@li...>
>     > https://lists.sourceforge.net/lists/listinfo/exist-development
>     >
>     >
>      
>      
> 
> 
> 
> 
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> 
> 
> 
> _______________________________________________
> Exist-development mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-development

Re: [Exist-development] thoughts about next steps with RDBMS integration

[James F. <jam...@ex...>]

2010/3/31 Leif-Jöran Olsson <lj...@ex...>

> Just remember that for a newbie this works best with non-structural data
> in the rdb fields. Otherwise the data massaging (possibly) needed  to
> make it well-formed can as well scare them away.
>

good point Leif,

will need to apply 'principle of least surprise' everywhere possible with
this, lets see how the prototype I am cooking up goes, it maybe that high
level impl is enough as a 'taster' for people,

J

Re: [Exist-development] thoughts about next steps with RDBMS integration

[Dan M. <dan...@gm...>]

Great idea Jim!

Your idea is another example of how we can make eXist more attractive for
people that are stuck with older legacy RDBMS systems.

Here is one other line of thought about how eXist might integrate with an
RDBMS.  Right now this was just a "design" and we never got full funding to
finish the proof-of-concept.  But I wanted to at least share our thinking.

In the past I did quite a bit of work with porting XRX applications between
eXist and DB2.  Using eXist was easy but setting up a WebDAV connection so
that we could make DB2 looks like an XML file system was a lot of work!  We
wanted to make it so simple for people do do drag-and-drop to import XML
files into our system, but some of the collections in our WebDAV view would
really be in a DB2 table.

One thing we did was to try to map the notion of a "collection" into an
RDBMS system.  Here was our goal:  We wanted eXist to "Create a new
Collection but put the XML files in the DB2 system". We wanted our XQueries
in eXist to access saved form data on a remote system.  We wanted to keep
our XQuery code portable free from *SQL pollution* but still use data in a
remote RDBMS supported by another vendor.

One design option we looked was to try to make specific *collections *map
into an RDBMS table with a single column that contained XML data.  Each XML
file was then mapped to a row n the RDBMS table that had five columns.  The
time-stamps and users that created and updated the XML file and a single
column for the RDBMS XML datatype.  We could then think of any XML
collection like:

/db/db2/my_collection

that *really *was running on DB2 in a SQL table called "my_collection"
using the very fast IBM PureXML "viper" 9.1 system that does do nice work
with the XML datatype and supported XQuery.  We wanted the systems to sync
up so that the b+trees on eXist would get updated with the RDBMS data was
changed.  That would have duplicated the indexes but we did not think that
the disk space was a problem.  IBM software was fast but the hoops we had to
jump through to run XQuery with SQL made it 10x harder to work with.  That
was several years ago but I still think that eXist has a much cleaner XQuery
interface.

I should also note that we were asked (but not funded) if we could do this
same thing with Microsoft-SQL server since the license costs were about
1/4th of the DB2 costs.  I imagine that using the Oracle XML column type we
could also do a similar thing.

Most of this work was very low-level Java coding using the DB2 interfaces
and I did not do the Java work myself.  It was slow hard work and our little
proof-of-architecture budget did not support it.  But it was fun to think
about how the system might work together and appear to the XQuery developer
as a single seamless entity.  The only restriction would be that distributed
joins would be very slow.

I don't know how difficult it would be to intercept all XQueries for a
specific collection and route them to a remote RDBMS and get the results
back.  But if we did have a collection RDBMS collection plugin it might be
an interesting test.  Most of the work then like it would need to be done on
each RDBMS.

- Dan

2010/3/31 James Fuller <jam...@ex...>

> 2010/3/31 Leif-Jöran Olsson <lj...@ex...>
>
>> Just remember that for a newbie this works best with non-structural data
>> in the rdb fields. Otherwise the data massaging (possibly) needed  to
>> make it well-formed can as well scare them away.
>>
>
> good point Leif,
>
> will need to apply 'principle of least surprise' everywhere possible with
> this, lets see how the prototype I am cooking up goes, it maybe that high
> level impl is enough as a 'taster' for people,
>
> J
>
>
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Exist-development mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-development
>
>


-- 
Dan McCreary
Semantic Solutions Architect
syntactica.com
952-460-1674
VOIP: 111@69.199.167.229

Re: [Exist-development] thoughts about next steps with RDBMS integration

[Adam R. <ad...@ex...>]

On 31 March 2010 06:45, James Fuller <jam...@ex...> wrote:
> whilst doing xproc development, I have had great fun integrating some of
> eXist existing extension functionality as extension steps ... a thought
> occurred to me about the current state of integration with RDBMS.
> Currently we have the nice SQL extension module (written by Adam Retter)
> which uses JDBC to query a relational database ... this means we can easily
> query an RDBMS through xquery using SQL statements. I think this module
> enables a lot of scenarios but misses off the primary scenario for the
> masses of people out there with data currently existing in RDBMS but want to
> use XML database.
> I would like to
> install eXist and provide in some registry a connection string which
> effectively 'mounts' the RDBMS as a collection ... from which here on out I
> can use pure xquery to work on. For now a read only view of the data is all
> I need but someday I would like to be able to do updates as well on the
> RDBMS data. Lastly I should be able to configure cache characteristics if
> the database connection is intermittent'

Phew!!! For a moment then I thought you were going to ask for a SQL
layer atop eXist-db and a JDBC driver for eXist-db.

This would actually be pretty easy to achieve :-)

Imagine the collection namespace root starting at /sql as opposed to /db.

Configuration could go into /sql/system/sql/{database}/{schemaName}/{tableName}

And the database configuration could again be defined in conf.xml
files in the /sql/system/sql collections. Caching and connection
property characteristics could also be placed in conf.xml files in the
appropriate sql configuration collection.

Almost all of the existing infrastructure could be reused. You would
simply need to extend the Collection function so that in the case of a
"/sql" collection it defered the document set building to code that is
very similar to that of the sql module. You would also need to come up
with a collection xml serialization scheme for displaying the
available sql schemas and tables as sub-collections but thus should
not be very hard and should be as close to possible (if not the same)
as the existing ones for the db collections.

This is quite exciting - I like :-)

Can I help?


>
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Exist-development mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-development
>
>



-- 
Adam Retter

eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb

Re: [Exist-development] thoughts about next steps with RDBMS integration

[James F. <jam...@ex...>]

On 31 March 2010 15:03, Adam Retter <ad...@ex...> wrote:

> On 31 March 2010 06:45, James Fuller <jam...@ex...> wrote:
> > whilst doing xproc development, I have had great fun integrating some of
> > eXist existing extension functionality as extension steps ... a thought
> > occurred to me about the current state of integration with RDBMS.
> > Currently we have the nice SQL extension module (written by Adam Retter)
> > which uses JDBC to query a relational database ... this means we can
> easily
> > query an RDBMS through xquery using SQL statements. I think this module
> > enables a lot of scenarios but misses off the primary scenario for the
> > masses of people out there with data currently existing in RDBMS but want
> to
> > use XML database.
> > I would like to
> > install eXist and provide in some registry a connection string which
> > effectively 'mounts' the RDBMS as a collection ... from which here on out
> I
> > can use pure xquery to work on. For now a read only view of the data is
> all
> > I need but someday I would like to be able to do updates as well on the
> > RDBMS data. Lastly I should be able to configure cache characteristics if
> > the database connection is intermittent'
>
> Phew!!! For a moment then I thought you were going to ask for a SQL
> layer atop eXist-db and a JDBC driver for eXist-db.
>
> This would actually be pretty easy to achieve :-)
>
> Imagine the collection namespace root starting at /sql as opposed to /db.
>
> Configuration could go into
> /sql/system/sql/{database}/{schemaName}/{tableName}
>
> And the database configuration could again be defined in conf.xml
> files in the /sql/system/sql collections. Caching and connection
> property characteristics could also be placed in conf.xml files in the
> appropriate sql configuration collection.
>
> Almost all of the existing infrastructure could be reused. You would
> simply need to extend the Collection function so that in the case of a
> "/sql" collection it defered the document set building to code that is
> very similar to that of the sql module. You would also need to come up
> with a collection xml serialization scheme for displaying the
> available sql schemas and tables as sub-collections but thus should
> not be very hard and should be as close to possible (if not the same)
> as the existing ones for the db collections.
>
> This is quite exciting - I like :-)
>
> Can I help?
>
>
thx Adam for teasing out the detail ... this is a cool way to approach it
... I would appreciate a chat to discuss; my initial thoughts were to create
an XQuery app (using sql extension module) to achieve the above ... its a
bit high level for my tastes for this kind of thing, but perhaps the best
first step.

J
J

Re: [Exist-development] thoughts about next steps with RDBMS integration

[Adam R. <ad...@ex...>]

Sure you could create a sql-collection() function using XQuery and
manually manage collection.xconf documents in a virtual hierarchy like
/db/system/sql/db/...

On 31 March 2010 14:43, James Fuller <jam...@ex...> wrote:
>
>
> On 31 March 2010 15:03, Adam Retter <ad...@ex...> wrote:
>>
>> On 31 March 2010 06:45, James Fuller <jam...@ex...> wrote:
>> > whilst doing xproc development, I have had great fun integrating some of
>> > eXist existing extension functionality as extension steps ... a thought
>> > occurred to me about the current state of integration with RDBMS.
>> > Currently we have the nice SQL extension module (written by Adam Retter)
>> > which uses JDBC to query a relational database ... this means we can
>> > easily
>> > query an RDBMS through xquery using SQL statements. I think this module
>> > enables a lot of scenarios but misses off the primary scenario for the
>> > masses of people out there with data currently existing in RDBMS but
>> > want to
>> > use XML database.
>> > I would like to
>> > install eXist and provide in some registry a connection string which
>> > effectively 'mounts' the RDBMS as a collection ... from which here on
>> > out I
>> > can use pure xquery to work on. For now a read only view of the data is
>> > all
>> > I need but someday I would like to be able to do updates as well on the
>> > RDBMS data. Lastly I should be able to configure cache characteristics
>> > if
>> > the database connection is intermittent'
>>
>> Phew!!! For a moment then I thought you were going to ask for a SQL
>> layer atop eXist-db and a JDBC driver for eXist-db.
>>
>> This would actually be pretty easy to achieve :-)
>>
>> Imagine the collection namespace root starting at /sql as opposed to /db.
>>
>> Configuration could go into
>> /sql/system/sql/{database}/{schemaName}/{tableName}
>>
>> And the database configuration could again be defined in conf.xml
>> files in the /sql/system/sql collections. Caching and connection
>> property characteristics could also be placed in conf.xml files in the
>> appropriate sql configuration collection.
>>
>> Almost all of the existing infrastructure could be reused. You would
>> simply need to extend the Collection function so that in the case of a
>> "/sql" collection it defered the document set building to code that is
>> very similar to that of the sql module. You would also need to come up
>> with a collection xml serialization scheme for displaying the
>> available sql schemas and tables as sub-collections but thus should
>> not be very hard and should be as close to possible (if not the same)
>> as the existing ones for the db collections.
>>
>> This is quite exciting - I like :-)
>>
>> Can I help?
>>
>
> thx Adam for teasing out the detail ... this is a cool way to approach it
> ... I would appreciate a chat to discuss; my initial thoughts were to create
> an XQuery app (using sql extension module) to achieve the above ... its a
> bit high level for my tastes for this kind of thing, but perhaps the best
> first step.
> J
> J



-- 
Adam Retter

eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb

Re: [Exist-development] thoughts about next steps with RDBMS integration

[James F. <jam...@gm...>]

ok this probably has to be a bit lower level then, because we want to
make it as natural as possible for end users to apply XQuery ...
probably best to work through this via chat.

J

On Wed, Mar 31, 2010 at 3:55 PM, Adam Retter <ad...@ex...> wrote:
> Sure you could create a sql-collection() function using XQuery and
> manually manage collection.xconf documents in a virtual hierarchy like
> /db/system/sql/db/...
>
> On 31 March 2010 14:43, James Fuller <jam...@ex...> wrote:
>>
>>
>> On 31 March 2010 15:03, Adam Retter <ad...@ex...> wrote:
>>>
>>> On 31 March 2010 06:45, James Fuller <jam...@ex...> wrote:
>>> > whilst doing xproc development, I have had great fun integrating some of
>>> > eXist existing extension functionality as extension steps ... a thought
>>> > occurred to me about the current state of integration with RDBMS.
>>> > Currently we have the nice SQL extension module (written by Adam Retter)
>>> > which uses JDBC to query a relational database ... this means we can
>>> > easily
>>> > query an RDBMS through xquery using SQL statements. I think this module
>>> > enables a lot of scenarios but misses off the primary scenario for the
>>> > masses of people out there with data currently existing in RDBMS but
>>> > want to
>>> > use XML database.
>>> > I would like to
>>> > install eXist and provide in some registry a connection string which
>>> > effectively 'mounts' the RDBMS as a collection ... from which here on
>>> > out I
>>> > can use pure xquery to work on. For now a read only view of the data is
>>> > all
>>> > I need but someday I would like to be able to do updates as well on the
>>> > RDBMS data. Lastly I should be able to configure cache characteristics
>>> > if
>>> > the database connection is intermittent'
>>>
>>> Phew!!! For a moment then I thought you were going to ask for a SQL
>>> layer atop eXist-db and a JDBC driver for eXist-db.
>>>
>>> This would actually be pretty easy to achieve :-)
>>>
>>> Imagine the collection namespace root starting at /sql as opposed to /db.
>>>
>>> Configuration could go into
>>> /sql/system/sql/{database}/{schemaName}/{tableName}
>>>
>>> And the database configuration could again be defined in conf.xml
>>> files in the /sql/system/sql collections. Caching and connection
>>> property characteristics could also be placed in conf.xml files in the
>>> appropriate sql configuration collection.
>>>
>>> Almost all of the existing infrastructure could be reused. You would
>>> simply need to extend the Collection function so that in the case of a
>>> "/sql" collection it defered the document set building to code that is
>>> very similar to that of the sql module. You would also need to come up
>>> with a collection xml serialization scheme for displaying the
>>> available sql schemas and tables as sub-collections but thus should
>>> not be very hard and should be as close to possible (if not the same)
>>> as the existing ones for the db collections.
>>>
>>> This is quite exciting - I like :-)
>>>
>>> Can I help?
>>>
>>
>> thx Adam for teasing out the detail ... this is a cool way to approach it
>> ... I would appreciate a chat to discuss; my initial thoughts were to create
>> an XQuery app (using sql extension module) to achieve the above ... its a
>> bit high level for my tastes for this kind of thing, but perhaps the best
>> first step.
>> J
>> J
>
>
>
> --
> Adam Retter
>
> eXist Developer
> { United Kingdom }
> ad...@ex...
> irc://irc.freenode.net/existdb
>
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Exist-development mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-development
>