Re: [Aperture-devel] Aperture Mavenization proposal

[Herko t. H. <her...@ad...>]

Antoni Myłka wrote:
> First of all, great proposal. That's something we've been talking about 
> since November 2006 - my notes are in the text.

Talk has been going on too long, it's time for action :D

And it looks like interest in Maven artifacts is there now (also outside 
Aduna ;)), so it seems like a good time to do something about it.

>> P: each module defines its own dependencies, yet inherits version
>>       information from the parent (as proposed above)
>> J: Ensures modules with overlapping dependencies automatically use the
>>       same version of that dependency
>>     Ensure separation of dependencies
>>     Avoids issues where different modules within the project would use
>>       different versions of dependencies
> 
> I didn't know it was possible. I thought that depencencies inherited 
> from a parent module are an integral part of the child modules i.e. if 
> we specify 30 dependencies in the master pom of aperture, then each 
> build of a submodule (i.e. an aperture-extractors.jar) would ship with 
> all 30 jars (also those for crawlers). But, if it is then go for it.

Yep, it's possible. Using the <dependencyManagement> section of the pom 
file, a parent pom can specify default versions (and scopes) of 
dependencies for its children without forcing them to actually depend on 
all those dependencies. So for example, the parent would specify:

   <dependencyManagement>
     <dependencies>
       <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.4</version>
        <scope>test</test>
       </dependency>
     </dependencies>
   </dependencyManagement>

And then all the child would need to do is specify it want to use that 
dependency:

   <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
    </dependency>
   </dependency>

No need to specify the version or the scope (although it could still 
override both, if needed).

>> P: module granularity will be high (i.e. many modules: one for each API,
>>       one for each implementation)
>> J: Enforces separation of concerns
>>     Enforces separation of dependencies
>>     It's easy to combine modules, hard to separate them
>>     Supports the use of Aperture in constrained environments where only a
>>       small part of the total functionality is required (see also the
>>       next 3 points on aggregation modules, which basically consist of
>>       nothing more than a Maven pom file)
> 
> This may not be easy in some cases. Many extractors depend on the 
> extractor.util package, which in turn references POI and htmlparser. 1. 
> copy-paste the PoiUtil code into each extractor that uses it, 2. include 
> extractor.util into extractor-api bundle (and make extractor-api depend 
> on poi and htmlparser) or 3. create a separate extractor-util bundle.
> 
> And there is also the 15-months old issue of the MailUtil used to be in 
> aperture.util (and that included a dependency on javamail on the core 
> bundle), now it has two copies , one with the MimeExtractor, one with 
> the IMAPCrawler - not to introduce a dependency between them. Creating a 
> separate bundle for a single class with two methods is not a good idea.

We obviously need to be sensible about the exact granularity. However, I 
don't think the number of classes or methods is necessarily relevant to 
that choice. If that one class with two methods helps avoid dependencies 
on a couple of external libraries in other parts of the code, I think 
that might be a good enough reason to put it in its own module.

>> P: create aggregation modules to provide support for use cases where
>>       less granularity is desired
>> J: Strong use cases exists to use Aperture as a single "traditional"
>>       library or comprehensive service: no need to worry about managing
>>       dependencies or missing functionality
>>     Many different levels of granularity can be supported by just writing
>>       a new Maven pom file
> 
> Not "just by writing a maven pom file" but "by writing a maven pom file 
> and a detailed, step by step, foolproof manual what to type and what to 
> click in order to achieve the result we want", including the things that 
> could go wrong in the process (advice like "delete the 
> org.semanticdesktop folder from .m2/repository" or "delete .m2/repository")
> 
> :-)

I agree we need to document the advanced options that this approach 
gives us. However, I think in many cases people will be able to just use 
the end result of the top level build (which would be similar to the 
current Aperture release: one jar/zip/whatever).

>> P: there will also be another aggregation module that is basically the
>>       same as the current artifact: contains all Aperture code
>> J: Allows people to continue to use Aperture like they use it now, with
>>       the added benefit of being OSGi compatible
> 
> I.e. the overall aperture aggregate will be an osgi bundle, without any 
> embedded jars, but WITH the Activators. It will have all imported 
> packages listed in the Import-Package entry in the manifest. That 
> manifest is going to be HUGE and it definitely can't be done by hand.

Agreed.

>> P: create all jar files as OSGi bundles by default => put Activators in
>>       the regular src dir
>> J: Enables the use of Aperture in an OSGi context
>>     Does not prevent the use of Aperture in other situations
> 
> This is tricky. If we want to implement the proposal above, i.e. one 
> aperture jar with everything, all dependencies listed in ImportPackage 
> and no embedded jars, then we must provide osgi-enabled versions of all 
> dependencies, otherwise it doesn't make sense. Right now the jars that 
> get included in the aperture-osgi.zip have been hacked by myself to be 
> osgi-compatible (i.e. httpclient and commons-codec). That's not the way 
> to go in the long run. I don't know what's the status of the pdfbox, 
> poi, bouncycastle etc. as far as the osgi entries in their manifests are 
> concerned.

IMO, we don't necessarily need to strictly go the "no embedded jars" 
route. It might make sense to embed some jars. Having said that, the 
Apache Felix Commons project is an ongoing process to provide 
(Mavenized) OSGi bundles for "non-OSGi" jars.

> Some research is necessary in this. I could clarify the requirements. 
> The build tool should be able to:
> 
> 1. determine the imported packages
> 2. find out in which dependencies the imported packages occur
> 3. determine the versions of those dependencies
> 4. create a Import-Package entry for the manifests that lists all 
> imported packages (with their correct versions if needed)
> 5. provide a way to customize this behavior (maybe, the version 
> constraints could be relaxed in osgi The modules could specify the 
> versions they use, those versions will be used for compilation and those 
> versions will be shipped with a release, but they could also specify the 
> range of versions they could accept in an osgi environment, I don't know 
> how useful it would be, I'd start with having an Import-Package list 
> without versions, and add it if it's needed)
> 6. create an exported-packages entry with all packages in the bundle (I 
> guess it doesn't make sense for us to differentiate between 'public' and 
> 'published' API.
> 7. aggregate these entries in aggregate bundles
> 8. the aggregate bundles must have separate aggregators, these must 
> invoke all aggregators from the child bundles, compiling them might be 
> tricky, because the compiler for the aggregation module would need to 
> see the classes for the child modules. These are simple and could 
> theoretically be generated automatically (see the 
> CoarsegrainedCoreActivator for instance), but I don't know if the bnd 
> plugin can do this.
> 9. it needs to interoperate somehow with the eclipse PDE, to allow for 
> testing and debugging the aperture-as-bundle setup.
> 10. and it would be nice to have a switch that would embed some 
> dependencies inside the generated bundle (i say "some" since that would 
> apply to poi and pdfbox, but definitely not to rdf2go and sesame). Such 
> a switch might facilitate deployment in some cases (e.g. in Nepomuk), 
> where you'd need to add only one jar to the osgi environment and not 30).

I think most, if not all of this is doable.

Thanks for these comments, very useful. Keep 'em coming :)

Cheers,

Herko
-- 
Aduna - Guided Exploration

Herko ter Horst
Senior Software Engineer

Prinses Julianaplein 14b
3817 CS Amersfoort
The Netherlands

+31 33 465 9987 office
www.aduna-software.com