jboss-dev-forums

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<ad...@jb...>]

First some history.

The initial jbossbuild worked by having three different models.

1) The top level build model that defined all artifacts, these are the
build/component/artifact elements
2) The component build model that defined how those artifacts were constructed
componentdef/artifactdef/sourcedef
3) The tasks.xml that defined the artifact types and the macros that provide
foreach type behaviour on the model above

Assumptions:
1) The top level build information was shared by all the components and defined
common/consistent properties. It also defined the release structure
2) The top level/component level model could be combined where you just had one
project
3) Components were treated equivalently whether they were binary or source
the only difference being whether an artifact's location was resolved to
thirdparty/component or the ROOT/component/output
4) The build should be as declarative as possible, potentially allowing custom tasks
to other build scripts to mine the build structure to provide other behaviour,
e.g. cruisecontrol

View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3878064#3878064

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3878064

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<ad...@jb...>]

Overtime, the requirements have changed and I believe we have a current
structure something like the following:

1) A top level build that just defines the list of primary components
2) The component level build is made up of two files
component-info which describes the external view of the component
component build which desribes how the component is built
3) The tasks/macros as before

The component-info is intended to describe what this component provides,
but also what it uses. i.e. it could import other components (secondary components)
e.g. the test module will import junit, there is no need in the top level build to define
that we use junit because the test module provides that point of integration.

View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3878066#3878066

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3878066

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<ad...@jb...>]

Further requirements that have been picked up along the way:

1) Being able to federate builds.
e.g. jboss-portal includes jbossas
i.e. one project is not based on a specfic list of components, it is based on a list
provided by another project
2) Being able to version components to check their consistency
3) Provide a more rich definition of the component, e.g. licenses what "spec" it implements
4) Allow for custom release structures, e.g. being able to build testsuite configs using 
similar information that is used to build the main release
5) Making a lot of the component information available at runtime

View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3878067#3878067

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3878067

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<ad...@jb...>]

What I propose we do is clearly define the current requirements with
use cases explaining how they will be used.
Then people can show counter examples or argue why it should be done a different
way and we will more aware of why we are doing something in a particular way.

View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3878068#3878068

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3878068

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<sco...@jb...>]

Agreed. We need a real project to prototype the requirements and the interaction of components in both source and binary form, as well as getting to the release aspect of the build system. Matching artifacts on the legacy build was an ok start, but its not really getting us where we need to be.

I would like to see the jbossas component used for this purpose. We can prototype this to point of building a release of the microkernel, ejb3, tomcat + dependencies to flesh out usecases and implementation.


View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3878580#3878580

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3878580

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<ad...@jb...>]

I was thinking more of writing a regression testsuite for the new build.

This could be based on a local cvs repository and binary repository inside the
testsuite's output folder.

The tests could be based on one of the oldest forms of regression tests
where you start with an initial config and apply diffs/commands in steps/stages
comparing the result with the expected output after each stage.

test1/initial
test1/step1/patch (the diffs)
test1/step1/commands(ant build invocations)
test1/step1/expected
etc.


View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3878583#3878583

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3878583

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<sco...@jb...>]

That certainly should be part of the jbossbuild project tests to validate the implementation, but I don't see that it allows project developers to interact easily with the usecase driven design we are asking for.

View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3878601#3878601

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3878601

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<ad...@jb...>]

"sco...@jb..." wrote : That certainly should be part of the jbossbuild project tests to validate the implementation, but I don't see that it allows project developers to interact easily with the usecase driven design we are asking for.

Yes, but that process is stuck in a deadlock where people request features
without actually trying it. 
The continuous addition of new/changed features means it is not in a state 
where it is finished or being used.

I'd be prepared to cutover jboss-head to the new build so that people can try it
(leaving a deprecated-build.xml in place for people who need to get work done)
But only, because this is likely to move things forward in terms of feedback and stabilization.

View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3878605#3878605

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3878605

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<sco...@jb...>]

I would like to see a full release build working before cutting over to the sink or swim strategy ;)

Bang on a standalone remoting, aop and small jbossas release to better flush out some of the current thirdparty/versioning/release usecases before taking the plunge.


View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3878607#3878607

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3878607

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<rya...@jb...>]

Ok, I would like to propose the following plan:

1.  Get agreement on the following use cases (by mid next week):
     - versioning/materialize
     - top-level build structure
     - release process/markup

2.  Create a partial build of jbossas which implements the above use cases 
     - in jbossas/jbossbuild.xml on HEAD.
     - have a task breakdown/roadmap complete by end of next week
     - each task tied to an approved use case
   
Objections/approval/acquiescence?


View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3878976#3878976

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3878976

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<ad...@jb...>]

My preferred solution is to try to define upfront what we want it do
(I'm not sure what this is anymore ;-)

I think Scott wants to prototype so he is confident it is correctly managing thirdparty/versioning.

But I'm in favour of anything that speeds up the ditching of buildmagic and enables us
to get useful information out of the build so we can more easily manage the process!

We don't have any information at the moment other than reading the build scripts
or refering to the manually maintained xml files in thirdparty.

View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3878981#3878981

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3878981

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<sco...@jb...>]

I'm not sure the release process can be defined in much detail as there are issues to look into regarding how the installer fits into this. I'll create a seperate thread on issues with creating a 4.0.x installer. The long term release features should not be holding up getting the new build working so that projects can get migrated over to it.

The plan seems fine, but I want to see it validated with a minimal prototype build as we have to get the features validated as soon as possible. The approach of migrating a large existing project like jboss-head for each feature change will simply take too long.


View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3878983#3878983

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3878983

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<sco...@jb...>]

So the first thirdparty test I need to get working is something like this:


  | <?xml version="1.0"?>
  | 
  | <project name="main.build"
  |           default="synchronize"
  |           basedir=".">
  | 
  |    <property file="local.properties"/>
  |    <!-- Import the types -->
  |    <import file="../tools/etc/jbossbuild/tasks.xml"/>
  | 
  |    <property file="synchronize.properties"/>
  | 
  |    <build id="jbossas-thirdparty"
  |           impltitle="JBossAS"
  |           implversion="4.0.2"
  |           implvendor="JBoss Inc."
  |           implurl="http://www.jboss.org"
  |           description="JBoss Application Server"
  |           cvsroot="${cvs.prefix}@cvs.forge.jboss.com:/cvsroot/jboss"
  |           thirdpartypath="../thirdparty/"
  |           location="http://cruisecontrol.jboss.com/repository"
  |           targetdefs="targets"
  |           componentVisitor="com.acme.SomeGraphVisitor"
  |       >
  | 
  |       <component id="hibernate"
  |                  version="3.0.3">
  |       </component>
  |    </build>
  | 
  |    <!-- Generate the targets -->
  |    <generate generate="jbossas-thirdparty"/>
  | 
  |    <target name="test" depends="synchronize">
  |       <fail>
  |      <condition>
  |        <not>
  |           <available file="../thirdparty/hibernate/3.0.3/hibernate3.jar" />
  |           <available file="../thirdparty/hibernate/3.0.3/hibernate-metadata.jar" />
  |           <available file="../thirdparty/antlr/2.7.5H3/antlr.jar" />
  |           <available file="../thirdparty/asm/1.5.3/asm.jar" />
  |        </not>
  |      </condition>
  |    </fail>
  |    </target>
  | </project>
  | 

This tests pulling in the hibernate dependencies. A new feature is the ability to specify a vistor that will have access to the graph of components and their dependencies. This will allow for the generation of the license-info.xml for example.

I have added a org.jboss.ant.util.graph to jbossbuild as a start for graph visitation and algorithms.


View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3879238#3879238

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3879238

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<rya...@jb...>]

So in the case of the synchronization cycle, we will want the visitor to traverse the graph and collect a set of unresolved dependencies (ie, no local component-info), download those component-info's and repeat until all component-info's are downloaded.

Once the component-info's are complete, another visitor can traverse the graph and generate synchornize targets for each component.




View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3879326#3879326

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3879326

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<sco...@jb...>]

Yes, something like that.

View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3879397#3879397

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3879397

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<rl...@jb...>]

I have been working on implementing the functionality of the third party test which Scott defined.  

Currently,  I am able to set up a graph based upon the components defined in the top-level build and then traverse it and pull down the necessary component-info.xml files from the repository (in this case, the component-info.xml for hibernate).   The component-info from the repository currently looks like this:

  | 
  | <project name="hibernate-component-info">
  | 
  |    <component id="hibernate"
  |               licenseType="lgpl"
  |               version="3.0.5"
  |               projectHome="http://hibernate.org/"
  |               description="ultra-high performance object/relational persistence">
  | 
  |       <artifact id="hibernate3.jar"/>
  |       <artifact id="hibernate-annotations.jar"/>
  |       <includes id="thirdparty">
  |          <include component="asm" />
  |          <include component="antlr" />
  |       </includes>
  |       <export>
  |          <include input="hibernate3.jar"/>
  |          <include input="hibernate-annotations.jar"/>
  |       </export>
  |    </component>
  | 
  | 
  | </project>

My next step is to resolve the secondary dependencies, however version information is currently not defined at the component-info level for dependencies.  I would like to confirm what this should look like and if it is acceptable.  Is the following suitable to what you have in mind?

<project name="hibernate-component-info">
  | 
  |    <component id="hibernate"
  |               licenseType="lgpl"
  |               version="3.0.5"
  |               projectHome="http://hibernate.org/"
  |               description="ultra-high performance object/relational persistence">
  | 
  |       <artifact id="hibernate3.jar"/>
  |       <artifact id="hibernate-annotations.jar"/>
  |       <import component="asm">
  |       	<compatible version="1.3.4"/>
  |       </import>
  |       <import> component="antlr">
  |       	<compatible version="3.3.2"/>
  |       </import>
  |       
  |       <export>
  |          <include input="hibernate3.jar"/>
  |          <include input="hibernate-annotations.jar"/>
  |       </export>
  |    </component>
  | 
  | 
  | </project>


Thanks!

Ruel Loehr
JBoss QA

View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3880163#3880163

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3880163

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<sco...@jb...>]

See the following thread for the far from complete discussion on this. 

http://www.jboss.org/index.html?module=bb&op=viewtopic&t=64049

I don't have a problem with the proposed syntax as a starting point, but we should have a notion of an overriding version coming in from the top level build as an example.


View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3880196#3880196

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3880196

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<sco...@jb...>]

The other more likely source of the overriding dependency component version info would be the  xxx/version-info.xml descriptor mentioned in the thread I referenced.

It does make sense to state some base dependent version info in the component-info.xml of the dependee, but the actual version pulled into a given project has to be a compatible union of all dependees and the current thinking is that this is coming either from the master build directly or by a label that is mapped onto a version using the xxx/xxx/version-info.xml descriptor.


View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3880201#3880201

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3880201

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<rl...@jb...>]

I have implemented the functionality required to demonstrate the third party test which was described earlier in this thread. 

mkdir test
cd test
cvs checkout jbossas
cvs checkout tools
cd jbossas
ant -f secondary-dependency-test.xml testdependencies

The secondary-dependency-test.xml contains a task, synchronizeinfo, which will construct a graph containing the componentref's which are defined in the top level build.  The task resolves each componentref into a component, resolves its dependencies as defined in the corresponding component-info.xml file, and adds the component to the build.  This process is repeated until all dependencies are resolved.

The target testdependencies then performs a check to ensure the proper jar files have been downloaded.  In this case, after synchronization has been performed the following will have been retrieved from the repository:

/thirdparty/hibernate/3.0.5/component-info.xml
/thirdparty/hibernate/3.0.5/lib/hibernate3.jar                                                                   
/thirdparty/hibernate/3.0.5/lib/hibernate-annotations.jar

/thirdparty/antlr/2.7.5H3/component-info.xml
/thirdparty/antlr/2.7.5H3/lib/antlr.jar

/thirdparty/asm/1.5.3/component-info.xml
/thirdparty/asm/1.5.3/lib/asm.jar
/thirdparty/asm/1.5.3/lib/asm-attrs.jar

Ruel Loehr
JBoss QA



View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3880973#3880973

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3880973

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<sco...@jb...>]

Looks good. I'll move forward with trying to setup the jboss-4.0/thirdparty structure from a new build script.


View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3880987#3880987

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3880987

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<sco...@jb...>]

In working on generating thirdparty from the repository I have run into this problem. If I update the secondary-dependency-test.xml to include multiple component-refs that depend on apache-logging (here jgroups and tomcat):


  |       <componentref name="hibernate"
  |                  version="3.0.5" />
  |       <componentref name="jgroups"
  |                  version="2.2.7" />
  |       <componentref name="tomcat"
  |                  version="5.5.9" />
  | 

the build fails because there are multiple synchronize.apache-logging targets:


  | [starksm@banshee9100 jbossas]$ ant -buildfile secondary-dependency-test.xml
  | Buildfile: secondary-dependency-test.xml
  |      [null] Created dir: C:\cvs\JBossHead\jboss-build\thirdparty\jgroups\2.2.7
  |      [null] Getting: http://cruisecontrol.jboss.com/repository/jgroups/2.2.7/component-info.xml
  |      [null] To: C:\cvs\JBossHead\jboss-build\thirdparty\jgroups\2.2.7\component-info.xml
  |      [null] Created dir: C:\cvs\JBossHead\jboss-build\thirdparty\tomcat\5.5.9
  |      [null] Getting: http://cruisecontrol.jboss.com/repository/tomcat/5.5.9/component-info.xml
  |      [null] To: C:\cvs\JBossHead\jboss-build\thirdparty\tomcat\5.5.9\component-info.xml
  |      [null] Created dir: C:\cvs\JBossHead\jboss-build\thirdparty\apache-logging\1.0.4jboss
  |      [null] Getting: http://cruisecontrol.jboss.com/repository/apache-logging/1.0.4jboss/component-info.xml
  |      [null] To: C:\cvs\JBossHead\jboss-build\thirdparty\apache-logging\1.0.4jboss\component-info.xml.
  |      [null] Getting: http://cruisecontrol.jboss.com/repository/apache-logging/1.0.4jboss/component-info.xml
  |      [null] To: C:\cvs\JBossHead\jboss-build\thirdparty\apache-logging\1.0.4jboss\component-info.xml.
  |      [null] Created dir: C:\cvs\JBossHead\jboss-build\thirdparty\apache-modeler\1.1
  |      [null] Getting: http://cruisecontrol.jboss.com/repository/apache-modeler/1.1/component-info.xml
  |      [null] To: C:\cvs\JBossHead\jboss-build\thirdparty\apache-modeler\1.1\component-info.xml.
  |      [null] Created dir: C:\cvs\JBossHead\jboss-build\thirdparty\commons-el\1.0
  |      [null] Getting: http://cruisecontrol.jboss.com/repository/commons-el/1.0/component-info.xml
  |      [null] To: C:\cvs\JBossHead\jboss-build\thirdparty\commons-el\1.0\component-info.xml
  | .
  | BUILD FAILED
  | C:\cvs\JBossHead\jboss-build\jbossas\secondary-dependency-test.xml:38: Duplicate
  |  target: `synchronize.apache-logging'
  |  
  | 

There needs to be some tracking of whether a synchronization target has already been added.

View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3881510#3881510

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3881510

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<rl...@jb...>]

The problem was occurring when the same dependency occurred more than once in the graph.   Each time the dependency appeared, we would import its component-info.xml file, which of course, led to the duplicate components being present in the build and ultimately this caused duplicate targets to be created.

I have modified the functionality so that a check is performed before importing a component info.xml file.   If the component is already present in the ant project (e.g. we have already imported it's component-info.xml), then a new one is not created.   The dependency tree is still accurate, we are simply not adding the component to the project if it already exists.

I have already committed these changes so that work on the thirdparty dir can proceed.  The visitor attribute functionality is next on the task list.

View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3881677#3881677

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3881677

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<rl...@jb...>]

I have implemented the functionality which will allow a visitor to be defined as an attribute and then sent to the graph.  

A new target has been created in the secondary-dependency-test.xml file to support this.

<target name="generate-license-xml"  depends="synchronize">
  |             <visit-componentref-graph
  | componentVisitor="org.jboss.ant.util.graph.ComponentRefGraphLicenseVisitor"/>
  | </target>   

When this target is executed, the class defined by componentVisitor will be instantiated and sent to the graph for a visit.  Currently, this visitor does nothing more than print out the component id which is contained in the vertex it is visiting.

To implement the full use case of creating the thirdparty-licenses.xml, the visit method of class org.jboss.ant.util.graph.ComponentRefGraphLicenseVisitor and possibly the execute method of class: org.jboss.ant.tasks.build.VisitComponentRefGraphTask will need to be modified, depending on the final implementation.

To see this in action execute the following commands:

mkdir test
cd test
cvs checkout jbossas
cvs checkout tools
cd jbossas
ant -f secondary-dependency-test.xml generate-license-xml

Ruel Loehr
JBoss QA

View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3881831#3881831

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3881831

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<sco...@jb...>]

So this visit-componentref-graph task is something I can use for any navigation task with this update? There is an additional need to deal with a mismatch between the new build thirdparty repository that includes version information in the output structure, and the thirdparty legacy structure that has no such notion. The problem being that all the existing classpaths have no version notion and I don't want to have to propgate the version info other than automatically.

I'll look into the new visitor capability to see if this can be handled as part of the license-info.xml generation as well.

View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3881847#3881847

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3881847

[Jboss-dev-forums] [Design of JBoss Build System] - Re: New Build - Restate requirements

[<rl...@jb...>]

In it's current form, yes.   The task takes whatever vistitor was defined, instantiates it, and sends it to the graph where it will visit each node.  The visitor will perform the exact same set of operations on each vertex.

That being said...considering the case of license generation for example.   If the visitor were to handle doing all the work then for each vertex it encounters it would need to check if a license file already exists/or create one, get the component version information and then append some information to the license file.  

Another approach, if we didn't use a generic task, would be for the visitor to only be responsible for compiling a list of version information.   Once it returned to the task, the task itself would use the information the visitor gathered to create the license xml.  This is the approach I used for the synchronize info task.

Ultimately, I think there should be a generic task to handle any and all visitors who will be doing simple things which do not requrire any pre or post processing.  For complex functionality that we do not wish to repeat at each vertex, we will need more specific tasks.

View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3881852#3881852

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3881852