digir-protocol Mailing List for Distributed Generic Info Retrieval

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Your boss giving you a hard time, get a new job!

What would a "PhD" title do for your resume?

Call....

For US: 1-309-415-3595
Outside US: +1-309-415-3595

You don't have to spend 40k on a college degree today

Give your career a makeover with a new degree

http://degreenow.to/

Watches at wholesale prices, plus many other great deals as well.

Authorized dealer for more than 30 brands of watches.

http://fardspicresponsible.com/

Introducing the new tech giant in the making
DNC MultiMedia Corp
Symbol: DCNM

This is a hugely undervalued Gem with high profile clients such as
Microsoft, Samsung, LG, and Napster

the huge volume and price spike today on the news release is a good indicator of short and long term movement.

See the company's outstanding pr and pull the trigger on DCNM

Improve your resume in less than 40 days

What would an EMBA do for your career.

http://fastdegree4u.com/

Need A Business Loan? Reach Over 290 Lenders with One Easy Form.
5k-200k For Your Business!
http://yankih.cn/
If you have your own business and wish IMMEDIATE cash to spend ANY way you like or want Extra money to give the company a boost or require A low interest loan - NO STRINGS ATTACHED!
Do not worry about approval... your credit score will not disqualify you!
http://bearte.com.cn/
apices ashman artful
ampex anselm
algonquin alewife babcock arroyo

Hi Kevin,

Computing the total number of records per provider can indeed be 
risky. The protocol doesn't even try to deal with that notion - DiGIR 
metadata responses only show the number of records by resource (one 
cannot tell whether two resources under the same provider share the 
same database and "root table", or not).

TAPIR still needs some polishing before its first "official" release 
and it doesn't have a specification yet. A good starting point to get 
more information is:

http://ww3.bgbm.org/protocolwiki/FrequentlyAskedQuestions

Best regards,
--
Renato

On 7 Dec 2005 at 9:42, Kevin Webb wrote:

> Greetings Renato!
> 
> My thought was, and this might be a departure from DiGIR's operating
> model, that provider sites be classified by schema, and that a
> particular provider site would only offer resources based on a
> specific schema. If a data set was to be provided via more than one
> schema representation then there would be more than one provider, or
> service point, associated with that data set.
> 
> I might be trying to solve a problem that doesn't justify the effort,
> and in fact the only downside to mingling multiple schema
> representations that I can identify is record inflation due to
> multiply-defined resources. Take for example a data set containing
> 1,000,000 records. If I create a DiGIR provider resource that
> 'provides' this data in a strict Darwin Core schema, and then I create
> a second DiGIR provider resource that 'provides' this data using an
> extended schema, then the metadata response for that DiGIR provider
> shows an aggregate of 2,000,000 records when in reality there is only
> one set of 1,000,000 records that exist. Like I said earlier, this
> issue might not justify the effort.
> 
> Your comment about Tapir has caught my attention, as I have heard
> references made to the project. Other than knowing the extension of
> the acronym - TDWG Access Protocol for Information Retrieval, I can't
> find any development resources on line for the project. Are there any
> information or on-line resources that can be shared with respect to
> this project?
> 
> Thank you.
> 
> 
> KFW

Greetings Renato!

My thought was, and this might be a departure from DiGIR's operating
model, that provider sites be classified by schema, and that a particular
provider site would only offer resources based on a specific schema. If a
data set was to be provided via more than one schema representation
then there would be more than one provider, or service point, associated with
that data set.

I might be trying to solve a problem that doesn't justify the effort, and 
in fact
the only downside to mingling multiple schema representations that I can
identify is record inflation due to multiply-defined resources. Take for 
example
a data set containing 1,000,000 records. If I create a DiGIR provider resource
that 'provides' this data in a strict Darwin Core schema, and then I create 
a second
DiGIR provider resource that 'provides' this data using an extended schema, 
then
the metadata response for that DiGIR provider shows an aggregate of 2,000,000
records when in reality there is only one set of 1,000,000 records that exist.
Like I said earlier, this issue might not justify the effort.

Your comment about Tapir has caught my attention, as I have heard
references made to the project. Other than knowing the extension
of the acronym - TDWG Access Protocol for Information Retrieval,
I can't find any development resources on line for the project. Are there
any information or on-line resources that can be shared with respect
to this project?

Thank you.

KFW

At 09:04 AM 12/7/2005, Renato De Giovanni wrote:
>Hi Kevin,
>
>Even if we had separate tModels for each conceptual schema, it seems
>that there could be a problem for DiGIR networks to use that in UDDI.
>tModels are associated with a "bindingTemplate", or in other words,
>with a "real" service end point. And as you know, a DiGIR service can
>have several "resources" underneath, each one with potentially
>different conceptual schemas. So at this level, there's a granularity
>mismatch between DiGIR and UDDI, because your network might be
>interested only in a specific resource from a provider - not all of
>its resources - and UDDI will always return you service end points.
>
>We tried to avoid that in TAPIR by "forcing" each data source (same
>as resource in the DiGIR jargon) to have its own access point. So in
>this case, I believe it will be safe to use such a tModel strategy.
>
>Regards,
>--
>Renato
>
>On 6 Dec 2005 at 13:39, Kevin Webb wrote:
>
> > --Concept--
> > Should the DiGIR community consider using multiple tModel values
> > to differentiate between provider sites offering alternate or
> > extended schema representations of provider data?
> >
> > --Discussion--
> > As extensions to the Darwin Core schema proliferate within the DiGIR
> > provider community how are services based on alternate or extended
> > schemas supposed to differentiate themselves from strict DWC sites?
> > Should multiple schemas representing a provider data resource be
> > co-mingled?
> >
> > Portal code, or more specifically the presentation layer of the portal
> > engine, separates provider resource lists based on the various
> > conceptual schemas returned by discovery requests. This may be
> > confusing to the portal user not knowing which data resources were
> > available by schema. Automatons can resort to filtering based on
> > specific DiGIR-Resource tags, but discovery or metadata requests risk
> > overstated inventory counts if numerous resource schemas mapped to a
> > single data set are 'mixed' together.
> >
> > Separate tModel values based on schema enforce a stronger provider
> > identity which can result in more specific WSDL entries.
> >
> > Some may consider this a solution to a problem that doesn't exist,
> > while others may find that the separation of provider services based
> > on schema enhances the service.
> >
> > Comments?
> >
> >
> >
> > KFW
>
>
>
>-------------------------------------------------------
>This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
>for problems?  Stop!  Download the new AJAX search engine that makes
>searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
>http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
>_______________________________________________
>Digir-protocol mailing list
>Dig...@li...
>https://lists.sourceforge.net/lists/listinfo/digir-protocol

Hi Kevin,

Even if we had separate tModels for each conceptual schema, it seems 
that there could be a problem for DiGIR networks to use that in UDDI. 
tModels are associated with a "bindingTemplate", or in other words, 
with a "real" service end point. And as you know, a DiGIR service can 
have several "resources" underneath, each one with potentially 
different conceptual schemas. So at this level, there's a granularity 
mismatch between DiGIR and UDDI, because your network might be 
interested only in a specific resource from a provider - not all of 
its resources - and UDDI will always return you service end points.

We tried to avoid that in TAPIR by "forcing" each data source (same 
as resource in the DiGIR jargon) to have its own access point. So in 
this case, I believe it will be safe to use such a tModel strategy.

Regards,
--
Renato

On 6 Dec 2005 at 13:39, Kevin Webb wrote:

> --Concept--
> Should the DiGIR community consider using multiple tModel values
> to differentiate between provider sites offering alternate or
> extended schema representations of provider data?
> 
> --Discussion--
> As extensions to the Darwin Core schema proliferate within the DiGIR
> provider community how are services based on alternate or extended
> schemas supposed to differentiate themselves from strict DWC sites?
> Should multiple schemas representing a provider data resource be
> co-mingled?
> 
> Portal code, or more specifically the presentation layer of the portal
> engine, separates provider resource lists based on the various
> conceptual schemas returned by discovery requests. This may be
> confusing to the portal user not knowing which data resources were
> available by schema. Automatons can resort to filtering based on
> specific DiGIR-Resource tags, but discovery or metadata requests risk
> overstated inventory counts if numerous resource schemas mapped to a
> single data set are 'mixed' together.
> 
> Separate tModel values based on schema enforce a stronger provider
> identity which can result in more specific WSDL entries.
> 
> Some may consider this a solution to a problem that doesn't exist,
> while others may find that the separation of provider services based
> on schema enhances the service.
> 
> Comments?
> 
> 
> 
> KFW

--Concept--
Should the DiGIR community consider using multiple tModel values
to differentiate between provider sites offering alternate or
extended schema representations of provider data?

--Discussion--
As extensions to the Darwin Core schema proliferate within the DiGIR
provider community how are services based on alternate or extended
schemas supposed to differentiate themselves from strict DWC sites?
Should multiple schemas representing a provider data resource be
co-mingled?

Portal code, or more specifically the presentation layer of the portal
engine, separates provider resource lists based on the various conceptual
schemas returned by discovery requests. This may be confusing to the
portal user not knowing which data resources were available by schema.
Automatons can resort to filtering based on specific DiGIR-Resource
tags, but discovery or metadata requests risk overstated inventory
counts if numerous resource schemas mapped to a single data set are
'mixed' together.

Separate tModel values based on schema enforce a stronger provider
identity which can result in more specific WSDL entries.

Some may consider this a solution to a problem that doesn't exist,
while others may find that the separation of provider services based
on schema enhances the service.

Comments?

KFW

Greetings.

I am following up on my predecessor's question and the thoughtful
replies provided.

I understand the two methods discussed for defining extensions,
independent schemas and inheritance. The issue I am confronting
now is making a DiGIR portal work with either extension method.

--Independent Schemas--
On the DiGIR provider side, when defining a resource that is 'pulling'
from 2 independent schemas you have to define 2 <conceptualSchema>
data elements. Examination of the resource_name.xml file shows
2 <conceptualSchema> elements with 2 schemaLocation attributes.
On the DiGIR portal side, or more accurately in the presentation servlet,
the Information Domains are defined in presentation.xml. The issue as I
see it is that the Information Domain structure only allows for 1
<conceptualSchema> to be defined. I tried tests with adding a
<conceptualSchemas> wrapper around 2 <conceptualSchema> elements, and
tests with 2 stand-alone <conceptualSchema> elements both generating errors.
What/which schemaLocation namespace should be used in the Information Domain
document when trying to list provider resources that are defined with 2
<conceptualSchema>s? Or am I barking ( DiGIR-ing ) up the wrong tree?

--Inherited Schemas--
I tried tests using inheritance and received prefix label errors. These errors
were more due to my lack of grace in this realm than anything else, but my gut
feel was that the errors were due to a relationship between the <handle> value
defined in the Information Domain document and the tags used in the 
<resultSchema>
definition, etc. If I recall correctly the software was complaining about 
something
like "prefix bmde:darwin: needs to be defined."

Any suggestions to get extended-schema provider resources working through a 
DiGIR
portal for either extension technique would be greatly appreciated.

Thank you.

KFW

>Personally I remain somewhat uneasy with having different schemas for the
>different purposes since I would like it to be possible for software to
>validate all of our data mappings and I am not sure that we could easily do
>it between two or more arbitrary schemas.

yes - it would be nice to be able to specify the entire biodiversity data 
enterprise (or at least the biological collections component of it) in only 
one place...

>My hope (based on the combination of the DiGIR meetings and the ABCD
>sessions) is that any future version of Darwin Core will in fact model
>documents which are valid under the ABCD Schema (a small subset of the total
>set of fields, probably corresponding to what most people will in fact be
>able to populate within the schema).  I think that this should be possible
>and will in fact allow us in practice to maintain a single schema without
>any philosophical issues about how to perform correct mappings between the
>schemas.

Conceptually, will we not end up with a Darwin core that is simply a 
flattened subset of ABCD?    If that is the case, I think it would make a 
lot of sense to handle or the definitions and constraints in a single 
comprehensive schema.

>The problem is that even Darwin Core will
>include repeating subelements (e.g. for collector or higher taxon).  This
>will mean that we need to understand how to model these within DiGIR
>queries.

yes - but to start with a flat DC with no repeating elements would still be 
incredibly useful/powerful as a query framework that  we could live with 
that for the time being if it makes implementation any easier.  However, we 
would definitely want to deliver and receive repeating elements in the 
result set (determination histories, numeric/time ranges, etc.)

>The bottom line here is that I think our level of agreement in Brazil was in
>part because BioCASE needed to be able to separate the query and result
>schemas whereas I assumed that we would ultimately be able to combine both
>as different subsets of the same schema.

I quite agree.  The DC elements could be nailed down rather quickly so that 
all the mucking around that is going to take place for the wider ABCD will 
not hold up progress for the Species Analyst type queries that are going on 
at the moment.

>We ignored larger questions about
>what the DiGIR protocol may or may not be able to achieve because the
>specific case which interests us first can probably work without answering
>them.

yes - but,  in the Australian context at least, we are going to want 
delivery of repeating values for certain data elements as soon as possible.

>  Speaking both from my role in GBIF and as a software architect I
>would like to see DiGIR become a common transport and query protocol for use
>in any subject domain in which we want to exchange data.  For that reason I
>would like to resolve these more abstract issues before we go too far.

yes - DIGIR/ABCD should do it all...  nothing short of complete global 
biodiversity domination...  tomorrow, the world...

jim

~ Jim Croft ~ jr...@an... ~ 02-62465500 ~ www.anbg.gov.au/jrc/ ~

Le Mardi 29 Octobre 2002 21:12, Hobern, Donald a =E9crit :
> In the meetings in Brazil Guillaume suggested that we might standardise a=
ll
> our data storage as XML documents and then rely on XPath pattern matching.
> (If this is a misrepresentation, please forgive me.)  I argued that this
> might be undesirable in that it may be difficult to turn arbitrary XPath
> patterns into SQL statements for arbitrary database schemas and that this
> might therefore place an unnecessary technology constraint on provider
> implementers.
Actually, i only suggested to standardise external view of our data storage=
 as=20
virtual XML documents, whetever technology is used behind the scene, so as =
to=20
express our queries following a consensus model, the federative schema.

The question of how to translate an incoming xpath query into the most=20
efficient native-format query is just a technical issue. Anyone able to=20
produce the global DOM tree (or global SAX events set) from its dataset is=
=20
able to run the xpath statement on it anyway.

> However from what I have read of XQuery, it seems to be aiming to become a
> plausible standard for addressing our problem and I need to understand it
> further.  We certainly do not want to end up with a totally bespoke XML
> language for defining queries if a widely accepted standard for doing the
> same thing is likely to appear in the near future.
A rapid web research shows that XQuery is official W3C effort toward XML qu=
ery=20
language, and it is going to be largely incorporated in XPath 2.=20

But there are also other query languages developped elsewhere:
=2D XQL (http://www.ibiblio.org/xql)
=2D QUILT (http://www.almaden.ibm.com/cs/people/chamberlin/quilt.html)
=2D LOREL (http://www-db.stanford.edu/lore)
=2D XML-QL (http://www.w3.org/TR/1998/NOTE-xml-ql-19980819)

I guess however that XQuery is the safer bet.
=2D-=20
Guillaume Rousse <ro...@cc...>
GPG key http://lis.snv.jussieu.fr/~rousse/gpgkey.html

Dear Don,

> As Anton has mentioned via e-mail

In which e-mail?  How many mail-"societies" are there?

I suggest to use either DiGIR-protocol or DiGIR-dev. list but not both,
depending on talking about protocol or its implementation.  So I'm
sending only to DiGIR-prot. list...

Separation of query schema and result schema (are they agreed terms?)
sounds logically good for me.  However, I wonder that it makes session
between provider and portal stateful; return schema may depend on
providers. The separation also sounds "inclining" so called ANSI three
layer model of DB, because usind different schema in query and return
deferenciate level of abstraction in sense of view and model.

A realistic scenario is having multiple UDDI registry depending on
shema combination, e.g. UDDI registry for DC/DC, for DC/BioCASE,
for DC/EINSIN (and DiGIR meta-regitry giving URI of these
registries?).  In other words, solving schema issue outside of DiGIR
itself.  I'm not sure this will make us happier in total.

Cheeers,
James

--
Dr. Nozomi "James" Ytow
Institute of Biological Sciences / Gene research center
University of Tsukuba
Tsukuba, Ibaraki 305-8572
Japan

I would really like to see the current suite of And, Or, AndNot and OrNot
operators replaced with And, Or and Not, all nestable in any combinations
(with Not obviously being a unary operator).  I think that that would give a
much more natural way to model many queries and would be more natural for
most programmers.  (Indeed there are some queries which I believe cannot be
modeled with the existing suite of four operators.)

Donald

---------------------------------------------------------------
Donald Hobern
Programme Officer for Data Access and Database Interoperability
Global Biodiversity Information Facility Secretariat
Zoological Museum - University of Copenhagen
Universitetsparken 15, DK-2100 Copenhagen, Denmark
Tel: +45-35321483 Fax: +45-35321480 E-mail: dh...@gb...
---------------------------------------------------------------

(I need to think about this area much more than I have so far done.  I may
well do this in England over the next few days.  I also understand that many
of you have been thinking about this for far longer.)

There are clearly other (possibly simpler) ways for us to define the
relationships between schema elements than by using substitutable elements.
I am certainly tempted by the idea of just defining the federation schema
and then adding attributes from a DiGIR namespace, where these attributes
define the essential characteristics of each element as it relates to DiGIR
(e.g. whether it is queriable, and if so, what class of query operations is
appropriate).  Possibly a compiler could then be used to derive a more
natural query schema from the federation schema.  Secondary configuration
documents could still be used to define the mapping to the physical data
model.

In any case I believe that (whether we stick with using substitutable
elements or something more like attributes), we need to model more
information within the protocol schema.  In particular, can we lose Darwin
Core bounding box in favour of identifying that Location is an element which
is an instance of a 'CoordinateSubstitutableElement'?  We could then simply
allow software to deduce that a bounding box query construct can be applied
when querying it.  This would allow the concept to become a generalised part
of the protocol rather than a special case for one particular coordinate
element.

Donald

---------------------------------------------------------------
Donald Hobern
Programme Officer for Data Access and Database Interoperability
Global Biodiversity Information Facility Secretariat
Zoological Museum - University of Copenhagen
Universitetsparken 15, DK-2100 Copenhagen, Denmark
Tel: +45-35321483 Fax: +45-35321480 E-mail: dh...@gb...
---------------------------------------------------------------

As Anton has mentioned via e-mail we currently expect that the full data
model for collection data will be some version of the ABCD schema, which is
heavily nested, with multiple optional sections (including strict choices)
and with repeating subelements.  These characteristics do not seem easy to
fit in the current DiGIR substitutable element search model.  

In Brazil we therefore acknowledged that there does not necessarily have to
be a single common schema for both query and results.  The suggestion from
the meeting was that the query schema should be in effect Darwin Core 3, and
that the result schema should be the ABCD model.

In fact the current DiGIR model uses the federation schema for three
logically separable purposes: as a foundation from which to construct the
query, as a logical schema to which physical database schemas are mapped,
and as the record structure for results.  

Personally I remain somewhat uneasy with having different schemas for the
different purposes since I would like it to be possible for software to
validate all of our data mappings and I am not sure that we could easily do
it between two or more arbitrary schemas.  In effect the implementor of each
provider software application would be responsible for interpreting the
relationships between the different schemas.  In practice my qualms are
probably just a matter of philosophy but they remain.  

It also seems to me that separating the schemas ignores one of the major
benefits which may follow from the DiGIR protocol (although I acknowledge
that it is outside the concern of DiGIR as a query and transfer protocol).
That benefit would come from having standard methods for a software tool
(such as the PHP Provider) to automate the mapping between a query, a
physical data model and a result structure.  If all of these relate to the
same schema, the process may be tricky but it is logically just a matter of
programming.  If multiple schemas are used for the different aspects, at the
very least need extra definitions would be required to define how these
mappings are to be performed.  This seems unnecessarily complex.  (I believe
that this issue is less important for BioCASE, since there the protocol will
be used simply for transfer and there is therefore no need for DiGIR to be
able to define the other mappings internal to the provider.)

My hope (based on the combination of the DiGIR meetings and the ABCD
sessions) is that any future version of Darwin Core will in fact model
documents which are valid under the ABCD Schema (a small subset of the total
set of fields, probably corresponding to what most people will in fact be
able to populate within the schema).  I think that this should be possible
and will in fact allow us in practice to maintain a single schema without
any philosophical issues about how to perform correct mappings between the
schemas.  

We could define certain characteristics of schema elements which make them
unavailable as query elements (e.g. some combination of depth in document,
optionality and cardinality).  The problem is that even Darwin Core will
include repeating subelements (e.g. for collector or higher taxon).  This
will mean that we need to understand how to model these within DiGIR
queries.

The bottom line here is that I think our level of agreement in Brazil was in
part because BioCASE needed to be able to separate the query and result
schemas whereas I assumed that we would ultimately be able to combine both
as different subsets of the same schema.  We ignored larger questions about
what the DiGIR protocol may or may not be able to achieve because the
specific case which interests us first can probably work without answering
them.  Speaking both from my role in GBIF and as a software architect I
would like to see DiGIR become a common transport and query protocol for use
in any subject domain in which we want to exchange data.  For that reason I
would like to resolve these more abstract issues before we go too far.

Donald

---------------------------------------------------------------
Donald Hobern
Programme Officer for Data Access and Database Interoperability
Global Biodiversity Information Facility Secretariat
Zoological Museum - University of Copenhagen
Universitetsparken 15, DK-2100 Copenhagen, Denmark
Tel: +45-35321483 Fax: +45-35321480 E-mail: dh...@gb...
---------------------------------------------------------------

Should we take time out at this point to review the present state of
XPath/XQuery to see if they might provide us with a better language for
modeling queries like ours (which after all is the goal of those standards
activities)?  

In the meetings in Brazil Guillaume suggested that we might standardise all
our data storage as XML documents and then rely on XPath pattern matching.
(If this is a misrepresentation, please forgive me.)  I argued that this
might be undesirable in that it may be difficult to turn arbitrary XPath
patterns into SQL statements for arbitrary database schemas and that this
might therefore place an unnecessary technology constraint on provider
implementers. 

However from what I have read of XQuery, it seems to be aiming to become a
plausible standard for addressing our problem and I need to understand it
further.  We certainly do not want to end up with a totally bespoke XML
language for defining queries if a widely accepted standard for doing the
same thing is likely to appear in the near future.

Donald

---------------------------------------------------------------
Donald Hobern
Programme Officer for Data Access and Database Interoperability
Global Biodiversity Information Facility Secretariat
Zoological Museum - University of Copenhagen
Universitetsparken 15, DK-2100 Copenhagen, Denmark
Tel: +45-35321483 Fax: +45-35321480 E-mail: dh...@gb...
---------------------------------------------------------------

2002	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct (7)	Nov	Dec
2005	Jan	Feb	Mar	Apr	May	Jun (1)	Jul (1)	Aug	Sep (1)	Oct	Nov	Dec (4)
2006	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct (3)	Nov	Dec
2007	Jan	Feb	Mar	Apr	May	Jun (4)	Jul (3)	Aug (3)	Sep (3)	Oct	Nov (2)	Dec
2008	Jan	Feb (2)	Mar (4)	Apr (5)	May (2)	Jun	Jul	Aug (1)	Sep	Oct	Nov	Dec (1)

digir-protocol Mailing List for Distributed Generic Info Retrieval

digir-protocol — Public list to discuss DiGIR protocol.