[Farrago-developers] maven, junit4 and initBuild

[Hunter P. <hun...@sq...>]

Hello all,
   Due to some special testing needs I have for SQLStream, I need do
some work on our build system.  Taking a long look at initBuild and our
entire build infrastructure, its clear that it is long overdue for an
overhaul.  Most of the tools and practices are circa 2004 and its very
complex.  Just to add a jar requires several types of changes to several
files in different directories.

   So I've done some in-depth research into how our build works and the
tools used by other open and closed source projects.  Obviously some of
what initBuild does is quite standard.  So it makes sense to focus on
what is different about our build:
- Repos/MDR (UML -> Java code)
- Resgen (needs Eigenbase exception classes built first)
- release properties and other generated java code
- running SQL scripts
- diffing SQL results
- need for a special driver jar
- two different server main classes

Things that we do that are out of date/no longer best practice:
- we still use JUnit3
- most of our logic is bound up in ant and shell scripts

Things we do that are simply a bad idea:
- mixing code used by the build process, testing code and product code
into the same source directory
- using testing code as part of the build (FarragoSqlTest)
- not using a tool to do dependency resolution (Ivy or Maven)
- testing results and dynamically generated documentation and reports
not available in a user-friendly format
- managing our own classpathes
- not taking advantage of Jar manifests
- launching the build from a shell script (because Jenkins/Hudson can't
understand, take advantage of knowing about each part of the build)
- distributing jars (especially ones that are not our own) in scm

The net net of all of this research is:
- we need to upgrade to JUnit4 which uses annotations to mark and
generate tests (no more special lists of test in the build.xml)
- upgrading to JUnit 4 is actually quite easy and doesn't require any
other changes to the build system
- we need to switch to Maven (http://maven.apache.org/)
- the build should produce one type of catalog/repo at a time
  Jenkins/Hudson can generate each type of build (for different
platforms/architectures, opto/debug, and different repos types) and the
installer can take artifacts produced by Hudson/Jenkins and create
installers any way you choose

The advantages of Maven are:
- Dependency management and resolution (this is the big one)
- Better integration with Hudson/Jenkins
- Powerful reporting framework for analysis of code and automatic
generation of documentation suitable for external developers to
learn/gain confidence about Farrago.
- Automatic generation of IDE configurations
- Better integration with popular IDEs
- Shorter, simpler build configuration (instead of build logic)
- Archetypes for generation of plugin and customer projects
- Easier integration of build tools such as code coverage tools, etc
- Platform independent build scripts
- Makes it easier for new developers to build the first time
- Fewer env variables
- You can google for maven issues when you have a problem (not so for
initBuild.sh)
- Easier deployment of build artifacts to central repositories
- Makes possible the parallelization of the build

   Switching over to Maven is a big task but not as bad as it first
appears.  To do so, we need to break Farrago into a series of
sub-projects:

- farrago-util (org.eigenbase.util.property.*,
net.sf.farrago.util.FarragoProperties, net.sf.farrago.util.MdrUtil,
net.sf.farrago.catalog.FarragoModelLoader,
org.eigenbase.util.Eigenbase*Exception,
org.eigenbase.sql.SqlValidatorException,
net.sf.farrago.catalog.codegen.*, net.sf.farrago.trace.FarragoTrace, and
a couple of static methods org.eigenbase.util.Util.newInternal() and
org.eigenbase.jmi.JmiObjUtil.getObjectId() which will be moved into a
BuildUtil.java in org.eigenbase.util)
  This subproject contains utility code used in all parts of the build
system (build, test, and product) and all of the code used by the build
(FactoryGen and FarragoReposUtil).

- farrago-mdr maven plugin (uses farrago-util)
  A maven plugin that provides the MDR functionality used to build the
catalog and generate Java and C++ code

- farrago-resgen maven plugin (uses farrago-util)
  A maven plugin that calls resgen to generate resource files

- farrago-sql-maven-plugin
  A maven plugin that starts a server and then runs a list of sql files.
This replaces the initsql and defaults to running the .sql files in
src/main/sql.  If diff files are found, it diffs them against the output
of the .sql files.

- farrago-sql-test-maven-plugin
  Extends the farrago-sql-maven-plugin runs and diffs all .sql tests in
a directory which defaults to src/test/sql

- driver project (depends on farrago-util, and contains
net.sf.farrago.jdbc.* and org.eigenbase.util14.ConnectStringParser)

- catalog project (might rename to metamodel)
  Uses the farrago-mdr-maven-plugin to process the FEM and generates the
Java and C++ repo/JMI code.  This one is weird because each repo type
produces extra files that need to be used/packaged/distributed possibly
as a zip or tar/gzip.

- farrago-core
  The meat of the project.  Contains the bulk of the product code, the
initsql scripts and tests which either don't need a server or use the
engine driver.  This can obviously be broken down further if you wish.

- farrago-vjdbc-rmi-server
  Contains the net.sf.farrago.server.FarragoVjdbcServer class (plus a
couple of dependent classes) and produces a jar file with a manifest
that will start the server using the 'java -jar Farrago.jar' syntax.
Contains the tests which need a server or uses the client driver.

- farrago-vjdbc-http-server
  Same as farrago-vjdbc-rmi-server but for the
net.sf.farrago.server.FarragoJettyEmbedding class

- farrago-plugin-maven-archetype (optional)
  An archetype for creating a simple farrago plugin project.  Possibly
this will split into various types of archetypes for different types of
Farrago extensions.

- farrago-plugin
  - farrago-dmv-plugin
  - farrago-miniplan-plugin
  - farrago-plannerviz-plugin
  - farrago-rng-plugin
  Building and testing of the example plugins


  We can break this up into several stages by calling maven from
initBuild.sh at first and doing some simple separation of selected
source files one maven component at a time.  However, it might be better
for people to get this change in one big bang as the intermediate points
won't really contain changes interesting to most developers.

Questions, comments?


Hunter

PS Yes I know I need to fix FNL-84 too but this is a more pressing issue
in my opinion.