Re: [GeoNetwork-users] ArcSDE harvester problems

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

hi Pat and all other (potential) ArcSDE harvesters,

thanks for posting the ESRI output, yes that worked :-)

I've run this through the transformations in ArcSDEHarvester, with the
following results :

(1) ARC_TO_ISO19115_TRANSFORMER_LOCATION :

produces well-formed XML, however it's not completely valid w.r.t. the
ISO19115 XSD in GeoNetwork's
<install-dir>/web/geonetwork/xml/schemas/iso19115/schema.xsd.

Now I'm not too sure what ISO19115 XSD that actually is, because as far as I
understood the ISO19139 XSD is the schema for the ISO19115 spec. I think it
was some ad-hoc XSD made for ISO19115 from before the existence of the
ISO19139 XSD, but as I said I'm not sure about this.

In any case, I repaired the output to make it valid for that ISO19115 XSD
(had to add an <mdContact> element, an <idCitation> element, and a <tpCat>
element).

Using this modified ISO19115, I ran it through the second transformation :

(2) ISO19115_TO_ISO19139_TRANSFORMER_LOCATION

Again this resulted in a well-formed XML document, but not totally valid
w.r.t. to the ISO19139 XSD (I used GeoNetwork's copy in
<install-dir>/web/geonetwork/xml/schemas/iso19139/schema.xsd).

To make it valid ISO19139 I had to change the format of
MD_Metadata/dateStamp/gco:DateTime to an actual date-time format (the value
produced in the transformation was something like "20040901"), and I had to
add a value to
MD_Metadata/identificationInfo/MD_DataIdentification/citation/CI_Citation/date/CI_Date/date/gco:DateTime
(it did not have a value).

So I think we can make these empirical observations:

* ArcSDE appears to put out FGDC-ish metadata, but there are also cases
where it appears to put out ISO19139 ;

* The FGDC output from version 9.3 is not transformed correctly by
ARC_TO_ISO19115_TRANSFORMER_LOCATION. This can be due to incomplete FGDC
output from ArcSDE, or due to some missing steps in that transformation. Or
it can be that the ISO19115 XSD in GeoNetwork is not 100% appropriate ;

* The ISO19139 output from the transformation in
ISO19115_TO_ISO19139_TRANSFORMER_LOCATION does not transform to a correct
ISO19139 format. This seems due to wrong assumptions about the input XML.

Now to make the ArcSDEHarvester work like a charm for all cases, I propose
the following :

(I) it should scan for "MD_Metadata" in the output, as it does now; these
metadata should not be subjected to the chain of transformations to turn
FGDC into ISO19139, obviously. It should be saved in GeoNetwork as it is,
assuming that this really is valid (or at least not too badly misformed)
ISO19139 ;

(II) it should also scan for "metadata" in the output. These metadata should
go through the chain of transformations to turn it into ISO19139, as that is
GN's preferred format. The transformations need to be made a bit more robust
to avoid putting out invalid formats -- even if the input XML may be
incomplete or a bit malformed.

(III) if the community really insists on it I could make it optional to
transform things to ISO19139, if you must save it in FGDC format

(IV) we should find out what exactly is that ISO19115 XSD in GeoNetwork. Who
made this, and for what purpose ? And it would surely perform better (and be
less error-prone) if we transformed the FGDC directly to ISO19139, in just a
single step. Should we aim for that ?

(V) we should find out if there are differences between various versions of
ArcSDE, for both the FGDC and ISO output, that we should take into account.
So I'd like to ask everybody who has some version of ArcSDE, could you
please share some example metadata output from your catalog ? Or is there an
ESRI employee in the audience who can shed light on this ?

(VI) as for accessing only your targeted database schema, I'll take a look
again at the documentation of ESRI's plug-in, but I do not have any ArcSDE
available for testing. Anyone who does, willing to help me test this ?

Looking forward to your reactions,

kind regards
Heikki Doeleman

On Mon, Jul 27, 2009 at 11:21 PM, PatB <pba...@gi...> wrote:

>
> Hi Heikki,
>
> I agree completely!  It’d be awesome to have geonetwork harvest metadata
> from arcsde regardless of the version or format.
>
> I uploaded (or at least tried to, let me know if it didn't work) a dummy
> metadata record exported from ArcCatalog 9.3.  The esri documentation
> (http://www.esri.com/metadata/) mentions several times that ‘by default’,
> catalog stores metadata as fgdc with the additional tags added by esri – it
> doesn’t mention how you would do it ‘not by default’.  There are also a
> number of stylesheets within catalog so you can display/export metadata in
> whatever format you prefer (they’re on the page referenced above) but the
> records are still stored as fgdc esri.  I haven’t done much with metadata
> in
> 9.2 but from the original post I answered it sounds like ISO metadata was
> being detected (it found ‘MD_Metadata’ in the xml blob and said cool :) so
> maybe it depends on what format it was in when created.  Not sure.  They’ve
> also started talking about a more detailed ESRI ISO format but I don’t
> think
> it’s being widely used (at least not yet).
>
> So anyway, the only other thing I ran into was that the ArcSDE query
> returns
> all records regardless of which tables the user has rights to.  I’m not
> sure
> if this is an across the board problem or specific to my setup but I solved
> it by adding a ‘Where Owner=…’  clause in the arcsdemetadataadapter.java
> file.
>
> Oh, and I replaced intermap with openlayers but I’ve already seen some
> posts
> about that.  In 2.4 RC1 the aoi search didn’t work, in RC2 it did  but then
> the search by category was broken, I haven’t checked it since the release
> of
> 2.4 but it might be worth looking into.
>
> Thanks!
>
>
> Heikki Doeleman wrote:
> >
> > hi Pat,
> >
> > glad to hear to hear that you got it working, and sorry to hear that you
> > had
> > to struggle to get there.
> >
> > The reason for this chain of transformations in the ArcSDE harvester is
> > that
> > we try to turn things into ISO19139 ("preferred" format) for storage in
> > GeoNetwork, as you correctly deduced.
> >
> > I think it would be in everybody's best interest to get this harvester
> > working more smoothly, also across different versions of ESRI's catalog.
> >
> > From what you write I gather that version 9.3 returns a FGDC flavour
> > that's
> > not correctly handled by the transformation in
> > FGDC_ISO_TRANSFORMER_LOCATION
> > in the harvester code ?
> >
> > If you could provide me with a spec on what it does actually return, I
> > could
> > add a different transformation for that one to make the code work. If
> it's
> > different from ESRI catalog version 8, we could make the ESRI version a
> > parameter in the harvesting settings, to get things right for both 8 and
> > 9.3
> > (and what about 9.2?)
> >
> > Thanks very much for sharing your analysis,
> >
> > kind regards
> > Heikki Doeleman
> >
> >
> > On Sat, Jul 25, 2009 at 6:17 PM, PatB <pba...@gi...>
> wrote:
> >
> >>
> >> I got it working with 9.3 - but it was a struggle.  First thing I'd do
> is
> >> verify the metadata you're getting out of sde 9.2 is the format you're
> >> expecting (ISO), by default 9.3 returns esri's flavor of fgdc.
> >>
> >> Next take a look at
> >>
> >>
> ...\geonetwork\src\org\fao\geonet\kernel\harvest\harvester\arcsde\ArcSDEHarvester.java
> >>
> >> around line 175 it has the following:
> >>
> >> // create JDOM element from String-XML
> >> Element metadataElement = Xml.loadString(metadata, false);
> >>
> >> // transform ESRI output to ISO19115
> >> Element iso19115 = Xml.transform(metadataElement,
> >> FGDC_ISO_TRANSFORMER_LOCATION);
> >>
> >> // transform ISO19115 to ISO19139
> >> Element iso19139 = Xml.transform(iso19115,
> >> ISO19115_TO_ISO19139_TRANSFORMER_LOCATION);
> >>
> >> So, if you track down the transformers and see what they're doing, it
> >> goes
> >> from ESRI Catalog 8 (which doesn't match anything I saw coming out of
> >> 9.3)
> >> to ISO19115 to ISO19139.  I got around it by using esri's metaid as the
> >> uuid
> >> and commenting out the transformers above so the metadata is stored as
> >> fgdc-std.  I'm still surprised that it worked.
> >>
> >> Good luck!  It sounds like you're close, you may also need to look at:
> >> ...\geonetwork\src\org\fao\geonet\arcgis\ArcSDEMetadataAdapter.java
> >>
> >> there's a line in there that sets ISO_METADATA_IDENTIFIER to look for
> >> MD_Metadata, which is what identifies the type of incoming metadata.  I
> >> had
> >> to change mine to "metadata" for fgdc.
> >>
> >> Hope this helps.  (oh, and you have to rebuild the jars using ant if you
> >> mess with the .java files)
> >>
> >>
> >> WSimonazzi wrote:
> >> >
> >> > Dear all,
> >> >
> >> > We are testing the ArcSDE harvester functionality using GN 2.4.0
> >> (final)
> >> > /jetty/McKoi.
> >> >
> >> > ArcSDE 9.2 (dummy library replaced by the 4 jars from ArcSDE SDK 9.2).
> >> >
> >> > dbtune: SDEbinary storage,
> >> >
> >> > Metadata: ISO 19115 and 19139.
> >> >
> >> > This is what we get:
> >> >
> >> >
> >>
> ----------------------------------------------------------------------------------------------------------------------------
> >> >
> >> > Connected to ArcSDE
> >> >
> >> > start retrieve metadata
> >> >
> >> > ISO metadata found
> >> >
> >> > cool
> >> >
> >> > Start of alignment for : REFDATABASE
> >> >
> >> > Recoverable error
> >> >
> >> >   XTRE0540: Ambiguous rule match for /
> >> >
> >> > Matches both "document-node()" on line 5 of
> >> >
> >> >
> >> >
> >>
> file:///C:/geonetwork/web/geonetwork/xsl/conversion/import/ISO19115-to-ISO1913
> >> >
> >> > 9.xsl
> >> >
> >> > and "document-node()" on line 30 of
> >> >
> >> >
> >> >
> >>
> file:/C:/geonetwork/web/geonetwork/xsl/conversion/19115to19139/19115-to-19139.
> >> >
> >> > xsl
> >> >
> >> > ArcSDE harvest finished
> >> >
> >>
> -------------------------------------------------------------------------------------------------------------------------------------
> >> > An no metadata is harvested, any clue/help can be rewarded with a beer
> >> in
> >> > Barcelona ;-).
> >> >
> >> > Many thanks,
> >> >
> >> > Walter & Maria Jose
> >> >
> >> > --
> >> > --
> >> > ================================================
> >> > Walter Simonazzi
> >> > European Topic Centre on Land Use and Spatial Information Torre C5-S,
> >> 4ª
> >> > planta
> >> > Edifici C- Facultat de Ciències Universitat Autònoma de Barcelona
> >> > 08193 Bellaterra (Barcelona)
> >> > Spain
> >> > ================================================
> >> > E-mail: walter.simonazzi (at) uab.es
> >> > Direct phone:  +34 93 581 3868
> >> > Fax:           +34 93 581 3545
> >> > Secretary:     +34 93 581 3518
> >> > http://terrestrial.eionet.europa.eu
> >> >
> >>
> ------------------------------------------------------------------------------
> >> > _______________________________________________
> >> > GeoNetwork-users mailing list
> >> > Geo...@li...
> >> > https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> >> > GeoNetwork OpenSource is maintained at
> >> > http://sourceforge.net/projects/geonetwork
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >> http://n2.nabble.com/ArcSDE-harvester-problems-tp3308994p3325857.html
> >> Sent from the GeoNetwork users mailing list archive at Nabble.com.
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> _______________________________________________
> >> GeoNetwork-users mailing list
> >> Geo...@li...
> >> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> >> GeoNetwork OpenSource is maintained at
> >> http://sourceforge.net/projects/geonetwork
> >>
> >
> ------------------------------------------------------------------------------
> > _______________________________________________
> > GeoNetwork-users mailing list
> > Geo...@li...
> > https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> > GeoNetwork OpenSource is maintained at
> > http://sourceforge.net/projects/geonetwork
> >
> >
> http://n2.nabble.com/file/n3336435/utilities_storm.xml utilities_storm.xml
> --
> View this message in context:
> http://n2.nabble.com/ArcSDE-harvester-problems-tp3308994p3336435.html
> Sent from the GeoNetwork users mailing list archive at Nabble.com.
>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus
> on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> GeoNetwork-users mailing list
> Geo...@li...
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
>