exist-development

[Exist-development] spatial index problem

[Loren C. <lor...@gm...>]

I forget who implemented the spatial index, but it was not compiling.  The libraries that are used for it are no longer where they were.  I updated the build to the current libraries location?

The dataset the is now downloading in the test has changed from a .gz to a .zip file.  The contents of the sample file has also changed.  I will be looking into this, but it will have to wait a week.  If someone else considers this a priority, then I would appreciate the unit test updates.

spatial:

prepare-libs:
     [echo] ------------------------------------------------------
     [echo] Downloading libraries required by the spatial index...
     [echo] ------------------------------------------------------
      [get] Getting: http://lists.refractions.net/m2/org/opengis/geoapi-nogenerics/2.1.1/geoapi-nogenerics-2.1.1.jar
      [get] To: /Users/lcahlander/Documents/workspace/test/extensions/indexes/spatial/lib/geoapi-nogenerics-2.1.1.jar
      [get] local file date : Fri Jun 13 04:47:48 CDT 2008
      [get] Not modified - so not downloaded
      [get] Getting: http://maven.geotools.fr/repository/org/geotools/gt2-api/2.4.5/gt2-api-2.4.5.jar
      [get] To: /Users/lcahlander/Documents/workspace/test/extensions/indexes/spatial/lib/gt2-api-2.4.5.jar
      [get] Error opening connection java.io.FileNotFoundException: http://maven.geotools.fr/repository/org/geotools/gt2-api/2.4.5/gt2-api-2.4.5.jar
      [get] Error opening connection java.io.FileNotFoundException: http://maven.geotools.fr/repository/org/geotools/gt2-api/2.4.5/gt2-api-2.4.5.jar
      [get] Error opening connection java.io.FileNotFoundException: http://maven.geotools.fr/repository/org/geotools/gt2-api/2.4.5/gt2-api-2.4.5.jar
      [get] Can't get http://maven.geotools.fr/repository/org/geotools/gt2-api/2.4.5/gt2-api-2.4.5.jar to /Users/lcahlander/Documents/workspace/test/extensions/indexes/spatial/lib/gt2-api-2.4.5.jar

BUILD FAILED
/Users/lcahlander/Documents/workspace/test/build/scripts/build-impl.xml:685: The following error occurred while executing this line:
/Users/lcahlander/Documents/workspace/test/extensions/indexes/build.xml:2: The following error occurred while executing this line:
/Users/lcahlander/Documents/workspace/test/extensions/indexes/build.xml:2: The following error occurred while executing this line:
/Users/lcahlander/Documents/workspace/test/extensions/indexes/build.xml:2: The following error occurred while executing this line:
/Users/lcahlander/Documents/workspace/test/extensions/indexes/spatial/build.xml:32: Can't get http://maven.geotools.fr/repository/org/geotools/gt2-api/2.4.5/gt2-api-2.4.5.jar to /Users/lcahlander/Documents/workspace/test/extensions/indexes/spatial/lib/gt2-api-2.4.5.jar

Re: [Exist-development] spatial index problem

[Pierrick B. <pie...@fr...>]

Hi,

Le 01/05/2011 02:51, Loren Cahlander a écrit :

> I forget who implemented the spatial index,

I did :-)

> but it was not compiling.

Yes, that's possible. I've asked to have a dedicated build on 
Teamcity... quite some time ago :-(

> The libraries that are used for it are no longer where they were. I
> updated the build to the current libraries location?

Feel free to update the build file with the correct locations. They 
regularly change...

> The dataset the is now downloading in the test has changed from a .gz to
> a .zip file.

Same remark.

> The contents of the sample file has also changed. I will be
> looking into this, but it will have to wait a week. If someone else
> considers this a priority, then I would appreciate the unit test updates.

What's wrong with the unit tests ? One of the them has been failing 
since the huge refactorings in winter 2009-2010. I'm afraid I wasn't
lucid enough to fix the issue :-)

Other tests were OK, even though they might be too generic (they were 
designed to validate the whole process).

I plan to add the possibility to bind the spatial engine with SQLite in 
the next weeks... It seems to be mature enough now.

Cheers,

p.b.

Re: [Exist-development] spatial index problem

[Dannes W. <da...@ex...>]

Hi,

On Wed, May 4, 2011 at 7:55 AM, Pierrick Brihaye
<pie...@fr...> wrote:
> What's wrong with the unit tests ? One of the them has been failing
> since the huge refactorings in winter 2009-2010. I'm afraid I wasn't
> lucid enough to fix the issue :-)

I run the spatial tests by default, they all run fine except one; this
one is caused by the introduction of SAXON as default XSLT engine (for
betterForm ?)

I agreed last week with LJO to reconsider the build process on
teamCity, I have some good ideas to improve the build a bit. In the
end, Maven could help us here. If we switch the spatial build to use
the "fetch" download  task, we can easily enable the cache function of
that ant task so we can limit the overhead of downloading the
dependencies over and over again.

'll push it some this/next week.......

D.

-- 
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

Re: [Exist-development] spatial index problem

[Loren C. <lor...@gm...>]

Hello Dannes,

Try clearing out the downloaded test data (extensions/indexes/spatial/test).  The build for test will fail:


spatial:

prepare:
     [echo] Building index module: spatial
     [echo] Building into classes
     [echo] ------------------------------------

compile:
     [echo] Compiling with Java 1.6 from 1.6 source to 1.6 target, debug on, optimize on, deprecation off

compile-tests:

prepare-test-tresources:
    [mkdir] Created dir: /Users/lcahlander/Documents/workspace/test/extensions/indexes/spatial/test/resources

download-15385-SS7886-5i1:
      [get] Getting: http://www.ordnancesurvey.co.uk/docs/sample-data/os-mastermap-topography-layer-sample-data.gz
      [get] To: /Users/lcahlander/Documents/workspace/test/extensions/indexes/spatial/test/resources/15385-SS7886-5i1.gz
      [get] http://www.ordnancesurvey.co.uk/docs/sample-data/os-mastermap-topography-layer-sample-data.gz moved to http://www.ordnancesurvey.co.uk/oswebsite/docs/sample-data/os-mastermap-topography-layer-sample-data.gz
      [get] http://www.ordnancesurvey.co.uk/oswebsite/docs/sample-data/os-mastermap-topography-layer-sample-data.gz moved to http://www.ordnancesurvey.co.uk/site/404.html
      [get] http://www.ordnancesurvey.co.uk/site/404.html moved to http://www.ordnancesurvey.co.uk/oswebsite/site/404.html
      [get] .................
   [gunzip] Expanding /Users/lcahlander/Documents/workspace/test/extensions/indexes/spatial/test/resources/15385-SS7886-5i1.gz to /Users/lcahlander/Documents/workspace/test/extensions/indexes/spatial/test/resources/15385-SS7886-5i1.gml

BUILD FAILED
/Users/lcahlander/Documents/workspace/test/build/scripts/junit.xml:221: The following error occurred while executing this line:
/Users/lcahlander/Documents/workspace/test/extensions/indexes/build.xml:2: The following error occurred while executing this line:
/Users/lcahlander/Documents/workspace/test/extensions/indexes/build.xml:2: The following error occurred while executing this line:
/Users/lcahlander/Documents/workspace/test/extensions/indexes/build.xml:2: The following error occurred while executing this line:
/Users/lcahlander/Documents/workspace/test/extensions/indexes/spatial/build.xml:67: The following error occurred while executing this line:
/Users/lcahlander/Documents/workspace/test/extensions/indexes/spatial/build.xml:76: Problem expanding gzip Not in GZIP format



the file that is downloaded is not a GZIP file.  It is a webpage with the title Sorry! We hope this doesn't spoil your experience of our site

Their sample data has changed.  You can find the current sample data at: http://www.ordnancesurvey.co.uk/oswebsite/products/try-now/sample-data.html

The new sample data is at http://www.ordnancesurvey.co.uk/docs/sample-data/os-mastermap-topography-layer-sample-data.zip and contains a GML file Initial/58116-SX9192-2c1.gz

I am wondering if we want to store a copy of the sample data that the tests use on demo.exist-db.org.  We would probably need to get their permission for that.




On May 4, 2011, at 1:22 AM, Dannes Wessels wrote:

> Hi,
> 
> On Wed, May 4, 2011 at 7:55 AM, Pierrick Brihaye
> <pie...@fr...> wrote:
>> What's wrong with the unit tests ? One of the them has been failing
>> since the huge refactorings in winter 2009-2010. I'm afraid I wasn't
>> lucid enough to fix the issue :-)
> 
> I run the spatial tests by default, they all run fine except one; this
> one is caused by the introduction of SAXON as default XSLT engine (for
> betterForm ?)
> 
> I agreed last week with LJO to reconsider the build process on
> teamCity, I have some good ideas to improve the build a bit. In the
> end, Maven could help us here. If we switch the spatial build to use
> the "fetch" download  task, we can easily enable the cache function of
> that ant task so we can limit the overhead of downloading the
> dependencies over and over again.
> 
> 'll push it some this/next week.......
> 
> D.
> 
> -- 
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

Re: [Exist-development] spatial index problem

[Dannes W. <da...@ex...>]

On 4 May 2011, at 8:22 , Dannes Wessels wrote:

> I run the spatial tests by default, they all run fine except one; this
> one is caused by the introduction of SAXON as default XSLT engine (for
> betterForm ?)


log4j:WARN No such property [append] in org.apache.log4j.ConsoleAppender.
May 4, 2011 6:46:06 PM org.geotools.gml.SubHandlerPolygon create
WARNING: Topology Error building polygon
May 4, 2011 6:46:13 PM org.geotools.gml.SubHandlerPolygon create
WARNING: Topology Error building polygon
Error 
  The SAX2 parser org.geotools.xml.transform.TransformerBase$XMLReaderSupport does not
  support setting the 'namespaces' feature to true: Feature: http://xml.org/sax/features/namespaces
org.exist.xquery.XPathException:  [source: String]
	at org.exist.xquery.modules.spatial.FunGMLProducers.eval(FunGMLProducers.java:426)
	at org.exist.xquery.BasicFunction.eval(BasicFunction.java:68)
	at org.exist.xquery.InternalFunctionCall.eval(InternalFunctionCall.java:55)
	at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:70)
......
	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
Caused by: org.exist.indexing.spatial.SpatialIndexException: net.sf.saxon.trans.XPathException: The SAX2 parser org.geotools.xml.transform.TransformerBase$XMLReaderSupport does not support setting the 'namespaces' feature to true
	at org.exist.indexing.spatial.AbstractGMLJDBCIndexWorker.streamGeometryToElement(AbstractGMLJDBCIndexWorker.java:581)
	at org.exist.xquery.modules.spatial.FunGMLProducers.eval(FunGMLProducers.java:419)
	... 35 more
Caused by: net.sf.saxon.trans.XPathException: The SAX2 parser org.geotools.xml.transform.TransformerBase$XMLReaderSupport does not support setting the 'namespaces' feature to true
	at net.sf.saxon.event.Sender.configureParser(Sender.java:541)
	at net.sf.saxon.event.Sender.sendSAXSource(Sender.java:305)
	at net.sf.saxon.event.Sender.send(Sender.java:156)
	at net.sf.saxon.IdentityTransformer.transform(IdentityTransformer.java:32)
	at org.geotools.xml.transform.TransformerBase$Task.run(TransformerBase.java:292)
	at org.geotools.xml.transform.TransformerBase.transform(TransformerBase.java:125)
	at org.geotools.xml.transform.TransformerBase.transform(TransformerBase.java:113)
	at org.geotools.xml.transform.TransformerBase.transform(TransformerBase.java:154)
	at org.exist.indexing.spatial.AbstractGMLJDBCIndexWorker.streamGeometryToElement(AbstractGMLJDBCIndexWorker.java:579)
	... 36 more
Caused by: org.xml.sax.SAXNotRecognizedException: Feature: http://xml.org/sax/features/namespaces
	at org.xml.sax.helpers.XMLFilterImpl.setFeature(XMLFilterImpl.java:156)
	at net.sf.saxon.event.Sender.configureParser(Sender.java:535)
	... 44 more


--
Dannes Wessels
eXist-db Open Source Native XML Database
e: da...@ex...
w: http://www.exist-db.org 

Re: [Exist-development] spatial index problem

[Pierrick B. <pie...@fr...>]

Hi,

Le 04/05/2011 20:30, Dannes Wessels a écrit :

> log4j:WARN No such property [append] in org.apache.log4j.ConsoleAppender.
> May 4, 2011 6:46:06 PM org.geotools.gml.SubHandlerPolygon create
> WARNING: Topology Error building polygon
> May 4, 2011 6:46:13 PM org.geotools.gml.SubHandlerPolygon create
> WARNING: Topology Error building polygon

Yes. There are a few polygons in the demo file that are not 
topologically correct.

> Error
>    The SAX2 parser org.geotools.xml.transform.TransformerBase$XMLReaderSupport does not
>    support setting the 'namespaces' feature to true: Feature:http://xml.org/sax/features/namespaces
> org.exist.xquery.XPathException:  [source: String]
> 	at org.exist.xquery.modules.spatial.FunGMLProducers.eval(FunGMLProducers.java:426)
> 	at org.exist.xquery.BasicFunction.eval(BasicFunction.java:68)
> 	at org.exist.xquery.InternalFunctionCall.eval(InternalFunctionCall.java:55)
> 	at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:70)
> ......
> 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
> Caused by: org.exist.indexing.spatial.SpatialIndexException: net.sf.saxon.trans.XPathException: The SAX2 parser org.geotools.xml.transform.TransformerBase$XMLReaderSupport does not support setting the 'namespaces' feature to true

This is the point. The previous parser used to work...

Any way to use another parser or to set the namespace feature ?

Cheers,

p.b.

Re: [Exist-development] spatial index problem

[Dannes W. <da...@ex...>]

On 4 May 2011, at 20:38 , Pierrick Brihaye wrote:

> This is the point. The previous parser used to work...
> 
> Any way to use another parser or to set the namespace feature ?

I had a look at the code some time ago, I am not sure if they do some special things there. There are not too many resources on the project that provide a clue...

in FunGMLProducers.java we need to figure out how we can pass a XMLFilterImpl object to the indexWorker#streamGeometryToElement method.... need to dive deeper inthis...... http://svn.geotools.org/tags/geoserver-1.7.0-beta1/modules/library/main/src/main/java/org/geotools/xml/transform/TransformerBase.java

:-(

D.

--
Dannes Wessels
eXist-db Open Source Native XML Database
e: da...@ex...
w: http://www.exist-db.org