[Geotools-devel] OSGi: first simple example with referencing factories working

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hello,

(long mail, formatted in HTML for better readability)

I just committed in the plugin branch a first version of an OSGi extension
which is scanning service files, publishing implementations as OSGi services
(registered by the providing bundle and instantiated with its classloader)
as well as a FactoryIteratorProvider implementation based on OSGi services.
*
## Overview*
A org.geotools.osgi bundle has been introduced which must be started (for
those familiar with Spring OSGi, this is similar to the
org.springframework.osgi.extender bundle). This module can be found in
osgisandbox/runtime/org.geotools.osgi.

When a bundle is started (status RESOLVED => STARTED), the org.geotools.osgi
bundle is notified and publishes the implementations defined in the
META-INF/services/* files under the related interfaces (if the same class is
defined for two interfaces, two services will be created each based on a
distinct instance). This is implemented by a SynchronousBundleListener
(org.geotools.osgi.factory.SpiToOsgiServiceBundleListener).

When org.geotools.osgi is started it also registers
org.geotools.osgi.factory.OsgiServiceFactoryIteratorProvider to the GeoTools
class. When it is called for a given category (<=> interface), it returns
the services published under the related interface.

This is basically the idea, and validates the approach that we had
discussed.
*
## Start order*
However, there is still some work to be done on how the bundle are started
and how dependencies between services are managed.

A problem I had was that, when starting gt-referencing some factories
needed, say DatumFactory, which was not yet processed.
I worked around this by catching FactoryNotFoundExceptions and retrying
until all factories are available. This works for a single bundle containing
a consistent set of factories (such as gt-referencing), but it could quickly
become messy with more modules depending on each other: in which order they
are started would yield a different state (especially since the service
lookup is cached and not performed each time, see below).

I have some ideas around this (like lazy start of bundles, wait for
dependencies as in Spring OSGi with timeout, etc.) but I preferred to come
back to you at this stage before adding more logic.

Now for the questions / request for comments. This dev was long and painful
and there is a lot to be said, but I will try to keep it not too long in
order to underline the main points that we should discuss.

At the end of this mail are some instruction to build and test all this.

*## Same package provided by different modules*
This is the main problem.

I have been struggling for a while at the beginning because I was stuck with
what is, in my opinion, the most horrendous kind of OSGi errors: 'Package
use conflicts.'

Without going into the details, that kind of issue can appear when two
bundles are exporting the same package to the OSGi runtime. If bundle A
references one, bundle B the other and bundle C references classes from A
and B, then C cannot be resolved (its dependency tree is disjoint).

I had the issue with gt-referencing which was exporting
org.geotools.resources while gt-metadata is exporting it as well.
In that case gt-referencing is bundle A and C, and gt-metadata is bundle B:
but gt-referencing needs itself as well as gt-metadata, hence the conflict.

No need to go through the pain of understanding this in detail, the point is
that this is much much easier if a given package is only exported by one
single module.

For the blocking case above, I fine-tuned the generation of the OSGi
metadata so that org.geotools.resources becomes private for gt-rendering and
is not exported. (hopefully not other module depends on the class of the
org.geotools.resources package of gt-referencing, because they are now only
visible by gt-referencing)

But I had a similar issue with gt-epsg-hsql which exports
org.geotools.referencing.factory.epsg just as gt-referencing does. I could
not find a workaround at the metdata level. But locally renamed the package
to org.geotools.referencing.factory.epsg.hsql and could have it working as
well.
(but unit tests where failing and anyhow I did not want to start messing
around on a large scale, so I reverted this, so epsg-hsql is not yet
working).

*Question*: would we consider impacting the code in order to reduce (or
completely remove) that kind of packages declared in many modules?
*
## Implicit classes dependencies in hints*
In gt-metadata, need to add all the packages of the implicit classes in
org.geotools.factory.Hints in the OSGi metadata (done)

*## "Plugins" cached*
I realized that the instances where actually somehow cached and that the
discovery mechanism was not executed each time a category is requested.
This perfectly makes sense in a non-dynamic environment, but in OSGi one
needs to refresh each time a new service is declared.

The only way I found was to use the static
ReferencingFactoryFinder.scanForPlugins().
The problem is that it creates a dependency on gt-referencing not
gt-metadata, which doesn't feel right.
Moreover, since I have tested only referencing related factories, I don't
know if it would work well for other kind of factories.

*Question*: is there a better way to refresh the registry?

What could be much better in a dynamic environment would be to be able to
configure that the factory iterator is queried each time.
(I won't go into the details, but for the next steps this would simplify a
lot).

*## Build and test*
I did my best to make this portable, but maybe something will fail at some
point, just let me know and that should be easy to fix.

# go to branch working copy (checked out from
http://svn.osgeo.org/geotools/branches/plugin)
cd ~/dev/src/geotools-branches-plugin
# update sources
svn up
# build geotools
mvn clean install
# go to OSGi sandbox
cd osgisandbox
# build OSGi sandbox
mvn clean install
# go to OSGi sandbox demo
cd demo
# start OSGi runtime
mvn -o org.argeo.maven.plugins:maven-argeo-osgi-plugin:equinox

(in OSGi console:)

# look for the test bundle
osgi> ss test
id      State       Bundle
2       RESOLVED    org.geotools.osgisandbox.referencing.test_8.0.0.SNAPSHOT

# start it using its id
# start <bundle id>
osgi> start 2
DefaultAuthorityFactory["All"] (crs, buffered)
└───ManyAuthoritiesFactory["All"] (crs, cs, datum, operation, optional)

# look for the gt-referencing bundle
osgi> ss gt-referencing
id      State       Bundle
4       RESOLVED    org.geotools.gt-referencing_8.0.0.SNAPSHOT

# start it using its id
# start <bundle id>
osgi> start 4
org.geotools.gt-referencing_8.0.0.SNAPSHOT [4]
org.opengis.referencing.datum.DatumFactory
org.geotools.referencing.factory.DatumAliases
org.geotools.gt-referencing_8.0.0.SNAPSHOT [4]
org.opengis.referencing.datum.DatumFactory
org.geotools.referencing.factory.ReferencingObjectFactory
...

# restart the test bundle
osgi> stop 2

osgi> start 2
DefaultAuthorityFactory["All"] (crs, buffered)
└───ManyAuthoritiesFactory["All"] (crs, cs, datum, operation, optional)
    ├───CartesianAuthorityFactory["EPSG"] (crs, registered)
    │   ├───ReferencingObjectFactory[direct] (crs, cs, datum, buffered,
registered)
    │   └───DatumAliases[direct] (datum, registered)
    ├───URN_AuthorityFactory["urn:ogc:def", "urn:x-ogc:def"] (crs, cs,
datum, operation, optional, registered)
    │   └───AllAuthoritiesFactory["All"] (crs, cs, datum, operation,
optional)
    │       ├───CartesianAuthorityFactory["EPSG"] (crs, registered)
    │       │   ├───ReferencingObjectFactory[direct] (crs, cs, datum,
buffered, registered)
    │       │   └───DatumAliases[direct] (datum, registered)
    │       ├───HTTP_AuthorityFactory["http://www.opengis.net"] (crs, cs,
datum, operation, optional, registered)
    │       ├───AutoCRSFactory["AUTO2", "AUTO"] (crs, registered)
    │       │   ├───ReferencingObjectFactory[direct] (crs, cs, datum,
buffered, registered)
    │       │   └───DatumAliases[direct] (datum, registered)
    │       └───WebCRSFactory["CRS"] (crs, registered)
    │           ├───ReferencingObjectFactory[direct] (crs, cs, datum,
buffered, registered)
    │           └───DatumAliases[direct] (datum, registered)
    ├───HTTP_AuthorityFactory["http://www.opengis.net"] (crs, cs, datum,
operation, optional, registered)
    │   └───AllAuthoritiesFactory["All"] (crs, cs, datum, operation,
optional)
    │       ├───CartesianAuthorityFactory["EPSG"] (crs, registered)
    │       │   ├───ReferencingObjectFactory[direct] (crs, cs, datum,
buffered, registered)
    │       │   └───DatumAliases[direct] (datum, registered)
    │       ├───URN_AuthorityFactory["urn:ogc:def", "urn:x-ogc:def"] (crs,
cs, datum, operation, optional, registered)
    │       ├───AutoCRSFactory["AUTO2", "AUTO"] (crs, registered)
    │       │   ├───ReferencingObjectFactory[direct] (crs, cs, datum,
buffered, registered)
    │       │   └───DatumAliases[direct] (datum, registered)
    │       └───WebCRSFactory["CRS"] (crs, registered)
    │           ├───ReferencingObjectFactory[direct] (crs, cs, datum,
buffered, registered)
    │           └───DatumAliases[direct] (datum, registered)
    ├───AutoCRSFactory["AUTO2", "AUTO"] (crs, registered)
    │   ├───ReferencingObjectFactory[direct] (crs, cs, datum, buffered,
registered)
    │   └───DatumAliases[direct] (datum, registered)
    └───WebCRSFactory["CRS"] (crs, registered)
        ├───ReferencingObjectFactory[direct] (crs, cs, datum, buffered,
registered)
        └───DatumAliases[direct] (datum, registered)

# close the runtime gracefully
osgi> close

Note: the test bundle does the following when it is started:

    public void start(BundleContext arg0) throws Exception {
        String[] args = { "-dependencies" };
        CRS.main(args);
    }
*
## OSGi development environment*
(Eclipse RCP cann be downloaded from here:
http://www.eclipse.org/downloads/packages/eclipse-rcp-and-rap-developers/heliossr2
)

For those familiar with OSGi:

- an Eclipse PDE target platform (based on the Maven dependencies) is
generated under osgisandbox/dep/org.geotools.osgisandbox.dep.referencing.
So, if you have this directory in your workspace, this target platform will
be listed under Window > Preferences > Plug-in Development > Target
Platform. You can select it and use it as a basis to develop or extend some
bundles

- if you go to osgisandbox/dep/org.geotools.osgisandbox.dep.referencing and
run
mvn -o dependency:copy-dependencies
you will have all the bundles in the target/dependency directory (you can
create a target platform based on this dir if you don't want to use the
automatically generated one, see above)

[mbaudier@alma org.geotools.osgisandbox.dep.referencing]$ ll
target/dependency/
total 9564
-rw-rw-r-- 1 mbaudier mbaudier  259959 May 22 21:29
com.springsource.javax.media.jai.codec-1.1.3.jar
-rw-rw-r-- 1 mbaudier mbaudier 1914524 May 22 21:29
com.springsource.javax.media.jai.core-1.1.3.jar
-rw-rw-r-- 1 mbaudier mbaudier   97124 May 22 21:29
com.springsource.org.apache.commons.pool-1.5.3.jar
-rw-rw-r-- 1 mbaudier mbaudier  396713 May 22 21:29
com.springsource.org.apache.log4j-1.2.15.jar
-rw-rw-r-- 1 mbaudier mbaudier  713050 May 22 21:29
com.springsource.org.hsqldb-1.8.0.10.jar
-rw-rw-r-- 1 mbaudier mbaudier   24098 May 22 21:29
com.springsource.slf4j.api-1.5.10.jar
-rw-rw-r-- 1 mbaudier mbaudier    9939 May 22 21:29
com.springsource.slf4j.log4j-1.5.10.jar
-rw-rw-r-- 1 mbaudier mbaudier   17578 May 22 21:29
com.springsource.slf4j.org.apache.commons.logging-1.5.10.jar
-rw-rw-r-- 1 mbaudier mbaudier 1856248 May 22 21:29
gt-epsg-hsql-8-SNAPSHOT.jar
-rw-rw-r-- 1 mbaudier mbaudier  489062 May 22 21:29
gt-metadata-8-SNAPSHOT.jar
-rw-rw-r-- 1 mbaudier mbaudier  329035 May 22 21:29
gt-opengis-8-SNAPSHOT.jar
-rw-rw-r-- 1 mbaudier mbaudier 1057237 May 22 21:29
gt-referencing-8-SNAPSHOT.jar
-rw-rw-r-- 1 mbaudier mbaudier 1154773 May 22 21:29
org.argeo.dep.osgi.jai.imageio-1.1.0.0001.jar
-rw-rw-r-- 1 mbaudier mbaudier   90457 May 22 21:29
org.argeo.dep.osgi.java3d-1.3.2.0002.jar
-rw-rw-r-- 1 mbaudier mbaudier   92839 May 22 21:29
org.argeo.dep.osgi.jsr275-1.0.0.0002beta2.jar
-rw-rw-r-- 1 mbaudier mbaudier   36521 May 22 21:29
org.argeo.osgi.boot-0.3.2-SNAPSHOT.jar
-rw-rw-r-- 1 mbaudier mbaudier 1148372 May 22 21:29
org.eclipse.osgi-3.6.2.jar
-rw-rw-r-- 1 mbaudier mbaudier   10174 May 22 21:29
org.geotools.osgi-8-SNAPSHOT.jar
-rw-rw-r-- 1 mbaudier mbaudier    3855 May 22 21:29
org.geotools.osgisandbox.referencing.test-8-SNAPSHOT.jar

- we (Argeo) have a small Eclipse plugin which simplifies generating PDE
launch configurations. This would be a bit OT and not really necessary to
describe how to install it, but just let me know if you are interested in
reproducing exactly my development environment going further.

- it is pretty easy to declare the GeoTools module as PDE projects in
Eclipse (that is as OSGi bundles).
In a given project, add the following files (adapt where necessary):

# .project

<?xml version="1.0" encoding="UTF-8"?>
<projectDescription>
    <name>gt-referencing</name>
    <comment></comment>
    <projects>
    </projects>
    <buildSpec>
        <buildCommand>
            <name>org.eclipse.jdt.core.javabuilder</name>
            <arguments>
            </arguments>
        </buildCommand>
        <buildCommand>
            <name>org.eclipse.pde.ManifestBuilder</name>
            <arguments>
            </arguments>
        </buildCommand>
        <buildCommand>
            <name>org.eclipse.pde.SchemaBuilder</name>
            <arguments>
            </arguments>
        </buildCommand>
    </buildSpec>
    <natures>
        <nature>org.eclipse.jdt.core.javanature</nature>
        <nature>org.eclipse.pde.PluginNature</nature>
    </natures>
</projectDescription>

# .classpath

<?xml version="1.0" encoding="UTF-8"?>
<classpath>
    <classpathentry kind="src" output="target/classes"
path="src/main/java"/>
    <classpathentry kind="src" output="target/classes"
path="src/main/resources"/>
    <classpathentry kind="src" output="target/test-classes"
path="src/test/java"/>
    <classpathentry kind="src" output="target/test-classes"
path="src/test/resources"/>
    <classpathentry kind="con"
path="org.eclipse.jdt.launching.JRE_CONTAINER/org.eclipse.jdt.internal.debug.ui.launcher.StandardVMType/J2SE-1.5"/>
    <classpathentry kind="con" path="org.eclipse.pde.core.requiredPlugins"/>
    <classpathentry kind="output" path="target/classes"/>
</classpath>

And copy the files from target/classes/META-INF (also the services files!)
under a META-INF subdirectory at the root of the project.

Cheers,

Mathieu

[Geotools-devel] OSGi: first simple example with referencing factories working

Toolkit for working with and mapping geospatial data

[Geotools-devel] OSGi: first simple example with referencing factories working