Again on mavenizing adempiere

FreePath
2011-08-16
2013-03-07
1 2 > >> (Page 1 of 2)
  • FreePath
    FreePath
    2011-08-16

    Hi all,
    we know in past there was some interest on mavenizing adempiere. We use maven internally, and even if we are far from being maven guru, we are interested in volunteering the effort needed to port Adempiere from ant to maven.
    We think such effort is only useful if there is a strong interest and if, once working, it will became the standard building method.

    As of now the plan is (details of the process are subject to change, should problems arise during the port):

    1. Clone the hg tip
    2. Keep the current sub-projects structure (base, client, ecc)
    3. Refactor every sub-project to be compliant with standard maven structure (this will be the most intrusive change)
    4. Dont split the test from the main line of code: Maven has separate source folder for tests, and they are executed at every build. This require checking every single test to be sure they work in a compatible way, and this effort is out of the main scope.
    5. Create a master pom, and a pom for every relevant module, to take advantage of project inheritance and aggregation.

    Whats the PMC (and community) opinion about that porting ?

    Best regards,
    Silvano
    www.freepath.it

    PS: I would also like to know if such plan can be useful for iDempiere, since our knowledge of osgi is close to zero.

     
  • Hi Silvano,
    here is my opinion: I think mavenizing adempiere would be a great thing. I am only a maven beginner, but from what I leanred so far, I think that it can be done.

    As a matter of fact, yesterday I talked with Mark about making a workshop at the conference to actually get it done.
    …well, or at least to push it to a point where it's worth finishing the result after the conference, instead of letting them dangle when daily business kicks in once more :-).

    Here is your plan with comments and additions:

    1. Clone the hg tip

    …I recomment adding a feature branch to the clone (see http://www.adempiere.com/Software_Development_Procedure#Feature_branches for what I mean)

    2. Keep the current sub-projects structure (base, client, ecc)

    …so every subproject would be a maven project/module, right?

    3. Refactor every sub-project to be compliant with standard maven structure (this will be the most intrusive change)

    …the good thing is that HG is able to track file renames and changes of directory strutuce. This means that concurrent changes from the developement branch can still be merged into the "mavenizing"-branch, and the result can be merged back.

    4. Dont split the test from the main line of code: Maven has separate source folder for tests, and they are executed at every build. This require checking every single test to be sure they work in a compatible way, and this effort is out of the main scope.

    …OK..if it turns out to be practical, it could still be done along with the main effort

    5. Create a master pom, and a pom for every relevant module, to take advantage of project inheritance and aggregation.

    …agree, this is one of the best things of maven

    There are two more aspects which I think are important:
    6. I think there needs be a maven project (pom) whose only job is to assemble the artifacts of the other sub projects into the actual application. This pom would also have to build the ear's application.xml and the adempiere.jnlp file.
    This pom could then be used to tailor the resulting adempiere binary to your needs. E.g. if you don't want to include posterita: no problem, just kick it out of the pom. You want to use mobileUI: no problem, add it to the pom. It will make sure that all dempendenices are present and include them in the xml descriptors.
    There can be a multitude of those poms, each one defining a different ADempiere package (similar to the different eclipse packages).

    7. Once mavenizing rolls, PMC should discuss providing an adempiere maven-repository on the adev-server. That way, we could make sure that all dependencies required to build adempiere are *really* available in the net. Otherwise there migh always be a risk that adempiere doesn't build because maven can't get some arcane.jar file.

    PS: I would also like to know if such plan can be useful for iDempiere, since our knowledge of osgi is close to zero.

    Last thing I know is that iDempiere (or "ADempiere-OSGI", as I like to call it:  :-P) uses buckminster to do the building, so I don't think that it's an osgi-issue. Can anybody tell us more or point to some good resources?

    Best regards
    Tobi

     
  • Hello Freepath , Tobi!

    It would be great to migrate our project to maven, the current approach based on ANT Script does not facilitate the handling of dependency and the updating them.

    I think we can start with small steps have to evolving.

    first phase

    1 .- Create a list of dependencies and remove some jars
    2 .- Migrating construction project ANT to MAVEN

    second Phase

    1 .- Update dependencies and validate the impact
    2 .- Automatize the tests and install a continuous integration tools
    3 .- Structure of construction based on extensions or components
    4 .- Facilitate the release of new versions

    So we can go with small steps by measuring the risk and go forward.

    regards
    Victor Perez
    www.e-evolution.com

     
  • FreePath
    FreePath
    2011-08-17

    Hi Tobi and Victor,
    thanks for the prompt answer.

    I recomment adding a feature branch to the clone

    I checked the document you mention, what kind of permission is needed to do (and work on) such a branch ? (the only one i know of is ssh access to the project)

    so every subproject would be a maven project/module, right?

    Yes

    6. I think there needs be a maven project (pom) whose only job is to assemble the artifacts

    You are right, assembling an ear require a specific pom.
    Jnlp is not supported natively, but there external plugins for it (this is an area that may raise problems).
    Some part, like the final assembling of the installation zip and tar.gz most probably will still be managed by ant.

    So, at a first glance, there will be three 'assembling' pom needed (or two, and use the parent pom with ant tasks to manage the zip files).

    Once mavenizing rolls, PMC should discuss providing an adempiere maven-repository…

    This will be of great help. If i can give a suggestion, a tool like Sonar may be of great help too.

    Regards,
    Silvano
    www.freepath.it

     
  • FreePath
    FreePath
    2011-08-26

    Hi all,

    about making a workshop at the conference to actually get it done. …well, or at least to push it to a point where it's worth finishing the result after the conference, instead of letting them dangle when daily business kicks in once more

    I will be at the conference, i would be happy to participate or to talk about it (and to future architectural direction of adempiere) in more detail.

    Best regards,
    Silvano
    www.freepath.it

     
  • Hi Silvano,

    I will be at the conference

    That's great, I'm looking forward to meet you there :-)

    Best regards
    Tobi

     
  • Hi all,
    first I would like to say that I enjoyed the conference very much (particularly the bbq-evening, not so much the next morning…but that's adifferent story:-) ).

    I was also very happy to learn that so many of you are willing to take part and push forward the the mavenizing effort.

    I thought about what was discussed in the mavenizing-workshop which we had in the conference and for me it turned out that there are much more aspects to consider than I actually expected. So I would like to thank you all for this valuable input.

    Given this, I want suggest a formulation of the first goal of the mavenizing effort:

    Mavenize the 3rd-party libs that are used by adempiere
    

    Roughly outlined this means:
    *Introduce poms for cctools, cstools and so on
    *Search the existing ant build files for their jars and add them all as dependencies to the pom
    *Then we can remove thes jars from both the build files and the adempiere hg-repo.
    *Now, the important point is here: The first goal is only to kick the jars out of the build files and remove them from the various lib folders. There are ways to process the poms and download the 3rd-party libs from within ant (google for "maven ant tasks" for further info).

    The goal of this step is not:
    *replace the ant files and the ant-based build system
    *upgrade the current 3rd party libs to newer versions

    Why so meager, you might ask.
    The reason is, that I would like the result of this first step to be merged into the development branch for the next release!
    So, no matter about how we go on from this point, I think it is clear that taking control of the 3rd-party libs "the maven way" is a sensible first step. Then we can still discuss the best approach for the next steps.

    Imho a very impotant tool for this step is the maven-repository that we have here: http://nexus.adempiere.de/nexus/ (note that http://nexus.adempiere.de doesn't work yet)

    It is hosted by ADeV and has been up by banym, so thanks a lot to both!

    This maven repository is supposed to serve two purposes:

    1. Be a proxy for all other maven repositories. That way, we can use only this repository. If a required 3rd-party library is e.g. hosted on codehaus, maven2, google-code etc, then this adempiere-maven-repository will find it there, download the library from there and forward it to the caller.

    2. Be a host for those libraries that are used in adempiere, but can't be found on any other repo. This can be the case, because the library is already very old, or because the lib is just a jar without any version info.

    If you want to take part in this effort, I would be very glad for you feedback and help.

    I started on the task wiki page by trying and sorting out the subprojects with their required jars and ant files. Then I started with CCTools.jar and added a littly pom to the feature branch.
    I further started to identify and add the first jars from the build.xml to the cctools.pom. You can see the current state on the wiki page (not a lot, right now).

    WDYT?
    Maybe we should schedule a skype meeting to get everybody to the same level and distribute some tasks?

    Best regards
    Tobi

     
  • FreePath
    FreePath
    2011-09-07

    Hi Tobi,
    if i understand it right, you want to use maven to generate the *tools jar.
    So, a pom for each, that will just be an 'empty' project to collect the dependencies.
    This will still require ant to assemble them, but removing them from the project will require every single 'subproject' (base,client,etc) to
    to have the appropriate *tool.jar on the classpath (instead of the jars).
    I dont think this is a good approach, because it will force users to use both maven and ant, with maven used for nearly nothing.
    IDE project will depend on a big jar instead on the single ones, wich does not look like a good idea to me.

    I think this can be a backup plan if at a certain point it became clear maven migration will not be ready in time. But, as a backup plan, maybe simply adding Ivy will obtain a similar result without needing maven.

    But, since now there is high interest (and as said at starting of thread, we as Freepath have the will and time to work on it) i think its better to try and replace all of the building process, still leaving out the deployment process (i mean RUN_setup and friends).
    So i'll suggest to start this way to start:
    1. migrate base to use maven
    2. prepare the parent pom using current ant files/.classpath files to gather them, enough to make base compiling
    3. peer review on that, improve if needed.

    At the some time, we can decide on the final subproject structures, and then go on migrating.
    Last step will be to prepare the ant script to assemble the final zip, and any other artifact out of maven scope (assembled jars, zips, and so on).

    Btw, i have sent you my skype contact, so we keep in touc.

    WDYT ?

    Regards,
    silvano
    www.freepath.it

     
  • FreePath
    FreePath
    2011-09-07

    Sorry, while reviewing the post i cut out a piece:

    3. peer review on that, improve if needed.

    4. Use 'base' as skeleton for the other subprojects

     
  • Hi Silvano,
    I follow your reasoning. I also see the point that using maven only to get the jars and then mashing them into *tools.jars is not satisfactory.

    The point I try to make is this: whatever we do, the first step would be to mavenize the 3rd-party libs, by identifying them in the build files and making sure that they can be obtained from our maven-repo.
    I think this alone will be quite an effort. That's why I suggested to concentrate on this first, before thinking too much about the more sophisticated aspects.

    However, I didn't want to imply that we should stop after that first step.

    Best regards
    Tobi

    PS: I got you contact data, I will send you mine shortly.

     
  • FreePath
    FreePath
    2011-09-08

    I agree, building the master pom and making sure all the jar are available are on top priority list.
    I'll update the wiki page with this first steps.

    Regards,
    Silvano
    www.freepath.it

     
  • Banym
    Banym
    2011-09-21

    Thank you Phil and Tobi for this effort. If I had some time I would join this effort with some more resources but daily business is keeping me busy at the moment.
    The Nexus installation should be ready for use now after implementing a workaround for the proxy configuration and it should be able to upload all the necessary artifacts and libs to an own repository.

    Regards,

    Dominik

     
  • FreePath
    FreePath
    2011-09-21

    Hello,
    i pushed the first batch of the mavenization: master pom, base and tools ported. I made some mistake while committing on mercurial, and that resullted in 3 versions instead of one.
    On maven repository i have uploaded the missing artifact.
    Let me know if you find issues building it, i will update the wiki page with the status and next steps.

    Regards,
    Silvano
    www.freepath.it

     
  • Wait wait wait … slow down …

    You guys are completely changing the directory structure in Trunk (Adempiere Development Branch).
    None of our patch files are working anymore for our own development.

    According to software development procedure:

    # structural changes (e.g. file, directory, rename, move, delete) should be announced and approved by Technical team before applying it

    Are you comitting to the wrong repository or branch by mistake?

    Stefan

     
  • Hi Stefan,
    the changes of directory structure have been made to the respective feature-branch(es), not to the "development"-branch!

    # structural changes (e.g. file, directory, rename, move, delete) should be announced and approved by Technical team before applying it

    "Applying" means that those changes are merged from the feature-branch to the branch that is named "development". Pls check out http://www.adempiere.com/Software_Development_Procedure#Supporting_branches for further details.

    If I misunderstood your point, feel free to clarify further.
    From my point of view it's very important to say (and i just checked it to be 100% sure) that those structural change have not been made to the development branch. The head of development branch is currently at rev "525874b80b44" (last changed on sep 14th).

    Best regards
    Tobi

     
  • Hi Tobi,

    Were you not supposed to be on holiday?

    You are right. I was still thinking the CVS/SVN way. I thought I was working with the development branch, but actually I was using the tip of the repository, which now I understand would put me into any branch that was last modified.

    I guess I will need to see a brain doctor to fully understand how hg works.

    Sorry for any accusations, and I confirm that you have NOT applied your changes to the development branch.
    Feel free to do what you want in your feature branch, and I will have to see how we correct our local patch algorithms to correctly handle the mercurial branching concept.

    (I liked your cleanup of the ant scripts, though).

    Best regards,
    Stefn

     
  • Hi,

    Were you not supposed to be on holiday?

    Not yet, but soon enough :-P

    (I liked your cleanup of the ant scripts, though).

    That was done by Philip Gosweiler (ackh). I just created the branch

     
  • FreePath
    FreePath
    2011-09-23

    Hi all,
    after a series of tuning to the pom, now on the new mavenization branch there is a working master pom, plus 'base' and 'tools' migrated to compile with

    maven.
    Now, im beginning to re-think the choice of having a master pom with all the depencies, for two reasons:
    1. When project 'client' will be migrated, and will depend on 'base', all 'base' depencies will be published to client anyway.
    2. Libraries needed just for 'client' will be visible to 'base' too, and considering there are ejb, jasperreport, web-related dependencies, this can

    become very messy.

    Second point opens another problem: i believe every project should just have the minumun needed dependencies, so its easy to spot what impact has a

    library, its easy to spot architectural issues (for example, base has dependencies on swing libraries, wich should just be used from the swing client),

    and its generally speaking a more clean approach.

    First point should guarantee that dependencies inheritance will still be in place even if we move from one big pom with all dependencies to smaller POMs

    with the the minimun set.

    What do you think about it ?

    Once that is set, the next step is to decide how to structure the sub-project, both to have a cleaner division, and to have a better layout to work with

    when working on the new ant tasks to assemble it all.
    I was thinking about doing something like that (just two examples to show the general approach i was thinking to apply):

    Now we have:
    client
    looks
    posterita
    webCM
    webStore
    zkwebui

    They can became:

    desktop-client
    - client
    - looks

    web-client
    - zkwebui-lib
    - zkwebui-webapp
    - webStore-lib
    - webStore-webapp
    - posterita-lib
    - posterita-webapp

    Or:

    web-client
    -zkwebui
    ---lib
    ---webapp
    -webStore
    ---lib
    ---webapp
    -posterita
    ---lib
    ---webapp

    In both case, the idea is to keep the library divided from the assembly, to allow for modules to depend on it (a zk package will very probably need to

    depend on the base zkclasses, but i dont think this can be done if zkwebui is just a war, while can be done if the zk code is packaged as a jar)

    Second one is cleaner, but more deep on directory structure, wich can become more difficult to navigate.

    What do you think about it ?
    And, from an assembly/perspective Will it be better/worse then the current structure ?

    Redards,
    Silvano
    www.freepath.it

     
  • From my point of view we should come up with individual files that define a module (jar, war, or ear) and its dependencies instead of having one master file listing all dependencies of all modules. For example there will be a file defining the module "base", i.e. contains a list of all dependencies of "base". Since "client" depends on "base" it will reference it but does not need to care about the dependencies of "base" since these will be handled by "base" itself.

    If I understand the second part of your posting correctly you are proposing that we introduce subdirectories instead of just having one directory containing all output files. Generally speaking this seems like a good idea to me. The problem we have right now is that we cannot really separate the desktop-client and the web-client since both the web-client and the desktop client depends on classes of the "base" project, i.e. besides the " desktop-client" and " web-client" it would be necessary to have a "common" (or something like that folder) that contains classes used by both.

    The first thing to resolve that I propose is to eliminate the redundancies of CSTools.jar and CCTools.jar. I guess that the original intention was to have a package containing all external dependencies of the server (hence "CSTools", probably short for "Compiere Server Tools") and one package containing the external dependencies of the client (hence "CCTools", probably short for "Compiere Client Tools"). Since both packages contain almost exactly the same libraries it is useless to maintain both.

    Regards,

    Phil

     
  • FreePath
    FreePath
    2011-09-27

    If I understand the second part of your posting correctly you are proposing that we introduce subdirectories instead of just having one directory containing all output files

    Yes, but they are not only directory, they are maven projects, so every directory will result in a maven artifact.
    A deeper refactoring, as you mention, would be even better, but most probably that will require a refactoring of the code. As a first step, i believe moving all to maven and reviewing the assemly process will a big enough effort, so to avoid not finishing it in time i plan to just 'move things around' without really changing things.

    he first thing to resolve that I propose is to eliminate the redundancies of CSTools.jar and CCTools.jar..

    To the building process, they will be not needed anymore: maven will take care of dependencies, and eliminate the need for them.
    So , they only remain interesting/neded for the assembly process. I hope a refactoring of the assembly process will eliminate the need for them.

    Regards,
    Silvano
    www.freepath.it

     
  • FreePath
    FreePath
    2011-10-11

    Hi all,
    i have committed the first version of adempiere compiling with maven (all of adempiere, with the exceptino of posterita, wich i dont know if its still supported or not).  As of now all compiles, but its not ready to be deployed because the depend on how the adempiere installable package (distribution) will be assembled.
    On that, i would like to ask for help from someone with better ant skills then mine.

    I have updated the wiki page with the new layout structure:
    http://www.adempiere.com/Feature:_Mavenize_ADempiere

    Best regards,
    www.freepath.it

     
1 2 > >> (Page 1 of 2)