Re: [Geotools-devel] OSGi: first simple example with referencing factories working

[Justin D. <jde...@op...>]

Sounds like you are making great progress Mathieu, good stuff. Some comments
inline.

On Sun, May 22, 2011 at 1:52 PM, Mathieu Baudier <mba...@ar...> wrote:

> Hello,
>
> (long mail, formatted in HTML for better readability)
>
> I just committed in the plugin branch a first version of an OSGi extension
> which is scanning service files, publishing implementations as OSGi services
> (registered by the providing bundle and instantiated with its classloader)
> as well as a FactoryIteratorProvider implementation based on OSGi services.
> *
> ## Overview*
> A org.geotools.osgi bundle has been introduced which must be started (for
> those familiar with Spring OSGi, this is similar to the
> org.springframework.osgi.extender bundle). This module can be found in
> osgisandbox/runtime/org.geotools.osgi.
>
> When a bundle is started (status RESOLVED => STARTED), the
> org.geotools.osgi bundle is notified and publishes the implementations
> defined in the META-INF/services/* files under the related interfaces (if
> the same class is defined for two interfaces, two services will be created
> each based on a distinct instance). This is implemented by a
> SynchronousBundleListener
> (org.geotools.osgi.factory.SpiToOsgiServiceBundleListener).
>
> When org.geotools.osgi is started it also registers
> org.geotools.osgi.factory.OsgiServiceFactoryIteratorProvider to the GeoTools
> class. When it is called for a given category (<=> interface), it returns
> the services published under the related interface.
>
> This is basically the idea, and validates the approach that we had
> discussed.
> *
> ## Start order*
> However, there is still some work to be done on how the bundle are started
> and how dependencies between services are managed.
>
> A problem I had was that, when starting gt-referencing some factories
> needed, say DatumFactory, which was not yet processed.
> I worked around this by catching FactoryNotFoundExceptions and retrying
> until all factories are available. This works for a single bundle containing
> a consistent set of factories (such as gt-referencing), but it could quickly
> become messy with more modules depending on each other: in which order they
> are started would yield a different state (especially since the service
> lookup is cached and not performed each time, see below).
>
> I have some ideas around this (like lazy start of bundles, wait for
> dependencies as in Spring OSGi with timeout, etc.) but I preferred to come
> back to you at this stage before adding more logic.
>
>
> Now for the questions / request for comments. This dev was long and painful
> and there is a lot to be said, but I will try to keep it not too long in
> order to underline the main points that we should discuss.
>
> At the end of this mail are some instruction to build and test all this.
>
> *## Same package provided by different modules*
> This is the main problem.
>
> I have been struggling for a while at the beginning because I was stuck
> with what is, in my opinion, the most horrendous kind of OSGi errors:
> 'Package use conflicts.'
>
> Without going into the details, that kind of issue can appear when two
> bundles are exporting the same package to the OSGi runtime. If bundle A
> references one, bundle B the other and bundle C references classes from A
> and B, then C cannot be resolved (its dependency tree is disjoint).
>
> I had the issue with gt-referencing which was exporting
> org.geotools.resources while gt-metadata is exporting it as well.
> In that case gt-referencing is bundle A and C, and gt-metadata is bundle B:
> but gt-referencing needs itself as well as gt-metadata, hence the conflict.
>
> No need to go through the pain of understanding this in detail, the point
> is that this is much much easier if a given package is only exported by one
> single module.
>
> For the blocking case above, I fine-tuned the generation of the OSGi
> metadata so that org.geotools.resources becomes private for gt-rendering and
> is not exported. (hopefully not other module depends on the class of the
> org.geotools.resources package of gt-referencing, because they are now only
> visible by gt-referencing)
>
> But I had a similar issue with gt-epsg-hsql which exports
> org.geotools.referencing.factory.epsg just as gt-referencing does. I could
> not find a workaround at the metdata level. But locally renamed the package
> to org.geotools.referencing.factory.epsg.hsql and could have it working as
> well.
> (but unit tests where failing and anyhow I did not want to start messing
> around on a large scale, so I reverted this, so epsg-hsql is not yet
> working).
>
> *Question*: would we consider impacting the code in order to reduce (or
> completely remove) that kind of packages declared in many modules?
>

I think long term it would be ok to try to enforce the policy that we make
packages unique to modules/bundles. But in any attempt to do so we will have
to worry about maintaining backwards compatibility... and any packages that
we want to move we will have to first go through the normal deprecation
cycle. So a possible way forward would be:

  1. identify all the packages/modules now that are in violation
  2. figure out how to move/reorganize all the package structures, and do
all the necessary deprecations on the current 8.x branch
  3. once 9.x realizes that will be the first osgi compatible branch

> *
> ## Implicit classes dependencies in hints*
> In gt-metadata, need to add all the packages of the implicit classes in
> org.geotools.factory.Hints in the OSGi metadata (done)
>
> *## "Plugins" cached*
> I realized that the instances where actually somehow cached and that the
> discovery mechanism was not executed each time a category is requested.
> This perfectly makes sense in a non-dynamic environment, but in OSGi one
> needs to refresh each time a new service is declared.
>
> The only way I found was to use the static
> ReferencingFactoryFinder.scanForPlugins().
> The problem is that it creates a dependency on gt-referencing not
> gt-metadata, which doesn't feel right.
> Moreover, since I have tested only referencing related factories, I don't
> know if it would work well for other kind of factories.
>
> *Question*: is there a better way to refresh the registry?
>

I ran into this same problem when trying to do some jython scripting with
geotools and to be able to add the geotools core libs dynamically to the
classpath. And the solution was to simply reload all the necessary factories
manually, referencing, common, etc...

I wonder if we could add some sort of callback to the GeoTools class
(similar to the listener list) in which the various factory finders could
register themselves with and then user/client code could just call a
GeoTools.scanForPlugins() or something and that would iterate through all
the registered factory finders invoking the scanForPlugins() method.

>
> What could be much better in a dynamic environment would be to be able to
> configure that the factory iterator is queried each time.
> (I won't go into the details, but for the next steps this would simplify a
> lot).
>
> *## Build and test*
> I did my best to make this portable, but maybe something will fail at some
> point, just let me know and that should be easy to fix.
>
> # go to branch working copy (checked out from
> http://svn.osgeo.org/geotools/branches/plugin)
> cd ~/dev/src/geotools-branches-plugin
> # update sources
> svn up
> # build geotools
> mvn clean install
> # go to OSGi sandbox
> cd osgisandbox
> # build OSGi sandbox
> mvn clean install
> # go to OSGi sandbox demo
> cd demo
> # start OSGi runtime
> mvn -o org.argeo.maven.plugins:maven-argeo-osgi-plugin:equinox
>
> (in OSGi console:)
>
> # look for the test bundle
> osgi> ss test
> id      State       Bundle
> 2       RESOLVED
>  org.geotools.osgisandbox.referencing.test_8.0.0.SNAPSHOT
>
> # start it using its id
> # start <bundle id>
> osgi> start 2
> DefaultAuthorityFactory["All"] (crs, buffered)
> └───ManyAuthoritiesFactory["All"] (crs, cs, datum, operation, optional)
>
> # look for the gt-referencing bundle
> osgi> ss gt-referencing
> id      State       Bundle
> 4       RESOLVED    org.geotools.gt-referencing_8.0.0.SNAPSHOT
>
> # start it using its id
> # start <bundle id>
> osgi> start 4
> org.geotools.gt-referencing_8.0.0.SNAPSHOT [4]
> org.opengis.referencing.datum.DatumFactory
> org.geotools.referencing.factory.DatumAliases
> org.geotools.gt-referencing_8.0.0.SNAPSHOT [4]
> org.opengis.referencing.datum.DatumFactory
> org.geotools.referencing.factory.ReferencingObjectFactory
> ...
>
> # restart the test bundle
> osgi> stop 2
>
> osgi> start 2
> DefaultAuthorityFactory["All"] (crs, buffered)
> └───ManyAuthoritiesFactory["All"] (crs, cs, datum, operation, optional)
>     ├───CartesianAuthorityFactory["EPSG"] (crs, registered)
>     │   ├───ReferencingObjectFactory[direct] (crs, cs, datum, buffered,
> registered)
>     │   └───DatumAliases[direct] (datum, registered)
>     ├───URN_AuthorityFactory["urn:ogc:def", "urn:x-ogc:def"] (crs, cs,
> datum, operation, optional, registered)
>     │   └───AllAuthoritiesFactory["All"] (crs, cs, datum, operation,
> optional)
>     │       ├───CartesianAuthorityFactory["EPSG"] (crs, registered)
>     │       │   ├───ReferencingObjectFactory[direct] (crs, cs, datum,
> buffered, registered)
>     │       │   └───DatumAliases[direct] (datum, registered)
>     │       ├───HTTP_AuthorityFactory["http://www.opengis.net"] (crs, cs,
> datum, operation, optional, registered)
>     │       ├───AutoCRSFactory["AUTO2", "AUTO"] (crs, registered)
>     │       │   ├───ReferencingObjectFactory[direct] (crs, cs, datum,
> buffered, registered)
>     │       │   └───DatumAliases[direct] (datum, registered)
>     │       └───WebCRSFactory["CRS"] (crs, registered)
>     │           ├───ReferencingObjectFactory[direct] (crs, cs, datum,
> buffered, registered)
>     │           └───DatumAliases[direct] (datum, registered)
>     ├───HTTP_AuthorityFactory["http://www.opengis.net"] (crs, cs, datum,
> operation, optional, registered)
>     │   └───AllAuthoritiesFactory["All"] (crs, cs, datum, operation,
> optional)
>     │       ├───CartesianAuthorityFactory["EPSG"] (crs, registered)
>     │       │   ├───ReferencingObjectFactory[direct] (crs, cs, datum,
> buffered, registered)
>     │       │   └───DatumAliases[direct] (datum, registered)
>     │       ├───URN_AuthorityFactory["urn:ogc:def", "urn:x-ogc:def"] (crs,
> cs, datum, operation, optional, registered)
>     │       ├───AutoCRSFactory["AUTO2", "AUTO"] (crs, registered)
>     │       │   ├───ReferencingObjectFactory[direct] (crs, cs, datum,
> buffered, registered)
>     │       │   └───DatumAliases[direct] (datum, registered)
>     │       └───WebCRSFactory["CRS"] (crs, registered)
>     │           ├───ReferencingObjectFactory[direct] (crs, cs, datum,
> buffered, registered)
>     │           └───DatumAliases[direct] (datum, registered)
>     ├───AutoCRSFactory["AUTO2", "AUTO"] (crs, registered)
>     │   ├───ReferencingObjectFactory[direct] (crs, cs, datum, buffered,
> registered)
>     │   └───DatumAliases[direct] (datum, registered)
>     └───WebCRSFactory["CRS"] (crs, registered)
>         ├───ReferencingObjectFactory[direct] (crs, cs, datum, buffered,
> registered)
>         └───DatumAliases[direct] (datum, registered)
>
> # close the runtime gracefully
> osgi> close
>
> Note: the test bundle does the following when it is started:
>
>     public void start(BundleContext arg0) throws Exception {
>         String[] args = { "-dependencies" };
>         CRS.main(args);
>     }
> *
> ## OSGi development environment*
> (Eclipse RCP cann be downloaded from here:
> http://www.eclipse.org/downloads/packages/eclipse-rcp-and-rap-developers/heliossr2
> )
>
> For those familiar with OSGi:
>
> - an Eclipse PDE target platform (based on the Maven dependencies) is
> generated under osgisandbox/dep/org.geotools.osgisandbox.dep.referencing.
> So, if you have this directory in your workspace, this target platform will
> be listed under Window > Preferences > Plug-in Development > Target
> Platform. You can select it and use it as a basis to develop or extend some
> bundles
>
> - if you go to osgisandbox/dep/org.geotools.osgisandbox.dep.referencing and
> run
> mvn -o dependency:copy-dependencies
> you will have all the bundles in the target/dependency directory (you can
> create a target platform based on this dir if you don't want to use the
> automatically generated one, see above)
>
> [mbaudier@alma org.geotools.osgisandbox.dep.referencing]$ ll
> target/dependency/
> total 9564
> -rw-rw-r-- 1 mbaudier mbaudier  259959 May 22 21:29
> com.springsource.javax.media.jai.codec-1.1.3.jar
> -rw-rw-r-- 1 mbaudier mbaudier 1914524 May 22 21:29
> com.springsource.javax.media.jai.core-1.1.3.jar
> -rw-rw-r-- 1 mbaudier mbaudier   97124 May 22 21:29
> com.springsource.org.apache.commons.pool-1.5.3.jar
> -rw-rw-r-- 1 mbaudier mbaudier  396713 May 22 21:29
> com.springsource.org.apache.log4j-1.2.15.jar
> -rw-rw-r-- 1 mbaudier mbaudier  713050 May 22 21:29
> com.springsource.org.hsqldb-1.8.0.10.jar
> -rw-rw-r-- 1 mbaudier mbaudier   24098 May 22 21:29
> com.springsource.slf4j.api-1.5.10.jar
> -rw-rw-r-- 1 mbaudier mbaudier    9939 May 22 21:29
> com.springsource.slf4j.log4j-1.5.10.jar
> -rw-rw-r-- 1 mbaudier mbaudier   17578 May 22 21:29
> com.springsource.slf4j.org.apache.commons.logging-1.5.10.jar
> -rw-rw-r-- 1 mbaudier mbaudier 1856248 May 22 21:29
> gt-epsg-hsql-8-SNAPSHOT.jar
> -rw-rw-r-- 1 mbaudier mbaudier  489062 May 22 21:29
> gt-metadata-8-SNAPSHOT.jar
> -rw-rw-r-- 1 mbaudier mbaudier  329035 May 22 21:29
> gt-opengis-8-SNAPSHOT.jar
> -rw-rw-r-- 1 mbaudier mbaudier 1057237 May 22 21:29
> gt-referencing-8-SNAPSHOT.jar
> -rw-rw-r-- 1 mbaudier mbaudier 1154773 May 22 21:29
> org.argeo.dep.osgi.jai.imageio-1.1.0.0001.jar
> -rw-rw-r-- 1 mbaudier mbaudier   90457 May 22 21:29
> org.argeo.dep.osgi.java3d-1.3.2.0002.jar
> -rw-rw-r-- 1 mbaudier mbaudier   92839 May 22 21:29
> org.argeo.dep.osgi.jsr275-1.0.0.0002beta2.jar
> -rw-rw-r-- 1 mbaudier mbaudier   36521 May 22 21:29
> org.argeo.osgi.boot-0.3.2-SNAPSHOT.jar
> -rw-rw-r-- 1 mbaudier mbaudier 1148372 May 22 21:29
> org.eclipse.osgi-3.6.2.jar
> -rw-rw-r-- 1 mbaudier mbaudier   10174 May 22 21:29
> org.geotools.osgi-8-SNAPSHOT.jar
> -rw-rw-r-- 1 mbaudier mbaudier    3855 May 22 21:29
> org.geotools.osgisandbox.referencing.test-8-SNAPSHOT.jar
>
> - we (Argeo) have a small Eclipse plugin which simplifies generating PDE
> launch configurations. This would be a bit OT and not really necessary to
> describe how to install it, but just let me know if you are interested in
> reproducing exactly my development environment going further.
>
> - it is pretty easy to declare the GeoTools module as PDE projects in
> Eclipse (that is as OSGi bundles).
> In a given project, add the following files (adapt where necessary):
>
> # .project
>
> <?xml version="1.0" encoding="UTF-8"?>
> <projectDescription>
>     <name>gt-referencing</name>
>     <comment></comment>
>     <projects>
>     </projects>
>     <buildSpec>
>         <buildCommand>
>             <name>org.eclipse.jdt.core.javabuilder</name>
>             <arguments>
>             </arguments>
>         </buildCommand>
>         <buildCommand>
>             <name>org.eclipse.pde.ManifestBuilder</name>
>             <arguments>
>             </arguments>
>         </buildCommand>
>         <buildCommand>
>             <name>org.eclipse.pde.SchemaBuilder</name>
>             <arguments>
>             </arguments>
>         </buildCommand>
>     </buildSpec>
>     <natures>
>         <nature>org.eclipse.jdt.core.javanature</nature>
>         <nature>org.eclipse.pde.PluginNature</nature>
>     </natures>
> </projectDescription>
>
> # .classpath
>
> <?xml version="1.0" encoding="UTF-8"?>
> <classpath>
>     <classpathentry kind="src" output="target/classes"
> path="src/main/java"/>
>     <classpathentry kind="src" output="target/classes"
> path="src/main/resources"/>
>     <classpathentry kind="src" output="target/test-classes"
> path="src/test/java"/>
>     <classpathentry kind="src" output="target/test-classes"
> path="src/test/resources"/>
>     <classpathentry kind="con"
> path="org.eclipse.jdt.launching.JRE_CONTAINER/org.eclipse.jdt.internal.debug.ui.launcher.StandardVMType/J2SE-1.5"/>
>     <classpathentry kind="con"
> path="org.eclipse.pde.core.requiredPlugins"/>
>     <classpathentry kind="output" path="target/classes"/>
> </classpath>
>
> And copy the files from target/classes/META-INF (also the services files!)
> under a META-INF subdirectory at the root of the project.
>
> Cheers,
>
> Mathieu
>
>
> ------------------------------------------------------------------------------
> What Every C/C++ and Fortran developer Should Know!
> Read this article and learn how Intel has extended the reach of its
> next-generation tools to help Windows* and Linux* C/C++ and Fortran
> developers boost performance applications - including clusters.
> http://p.sf.net/sfu/intel-dev2devmay
> _______________________________________________
> Geotools-devel mailing list
> Geo...@li...
> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>
>


-- 
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.