I'd like to talk about AoI's build process once more. :)
First of all, I still wonder why the ant build scripts are not available via SVN. When I do a checkout, I have no means to build AoI. There's just a bunch of .java files - no makefile, no SConstruct, no build.xml. That's a little cumbersome. :) Why not just include the build files in the SVN repository?
How do *you* actually build AoI? Do you use Eclipse or another IDE?
Second, while trying to build the latest SVN revision, I realized that all external JAR files are hardcoded in the build script. That means I have to manually edit the build script in order to add "svgSalamander"'s JARs to the classpath. Couldn't we just move all needed JARs into a "lib" subdirectory and tell the build script to add all JARs in this directory to the classpath?
(On a side note, this applies to launcher scripts like "aoi.sh", too. All JAR files are hardcoded, so you also have to edit that file for svgSalamander.)
Finally, there's this thread which lists further improvements:
Wow, that was an amazingly timely post. As part of implementing the SVG support, I restructured the build script to not hardcode the libraries, and literally just finished checking in all the libraries and build scripts. Do an update and see if it better fits your expectations. :)
Peter
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Well, uhm, *cough*, there's a little problem though.
Okay, the "zipgroupfileset" task basically is what I thought of. But svgSalamander breaks it: The file svgSalamander.jar includes a digital signature which gets copied, too. That's sort of a problem because Java then thinks this signature belongs to the newly created ArtOfIllusion.jar. Thus, when launching AoI I get this:
Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
Dunno, maybe Apple's Java implementation doesn't check signatures, so maybe that's why this didn't happen to you? For the sake of completeness, I'm using this version of Java:
$ java -version
java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.4)(ArchLinux-6.b20_1.9.4-1-i686)
OpenJDK Server VM (build 17.0-b16, mixed mode)
To be more precise, everything in svgSalamander's META-INF directory should not get copied to ArtOfIllusion.jar. Those last two KITFOX files should not be present:
$ jar tf ArtOfIllusion.jar | grep META
META-INF/
META-INF/MANIFEST.MF
META-INF/KITFOX.DSA
META-INF/KITFOX.SF
The sad thing is, it's impossible to filter the contents of JAR files when using "zipgroupfileset". Looks like this is a common problem, see for example:
Ironically, the "common solution" is not to use "zipgroupfileset" but either hardcode the filenames or do it the way you did it in the past (extract JARs into a temporary directory, then re-jarring everything but META-INF/* …). Well, yes, that's what we wanted to get rid of. :(
I don't know how to fix this, yet. I'll see if I can find a suitable workaround.
Cheers,
TroY
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Sigh. Well, the easiest solution is just to edit svgSalamander.jar to remove those two files. The only real problem I see is that I'll have to remember to do it every time I update to a new version of SVG Salamander.
Nik, could you explain your suggestions in more detail?
Peter
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
a) You're still able to include all JAR files using globbing, b) no need to manually edit any files.
Still a dirty hack, though…
By the way, the manifest in ArtOfIllusion.jar specifies a class path which includes "lib/jogl.jar" and "lib/gluegen-rt.jar". This is obsolete, isn't it?
(While we're at it: SVN now includes a newer version of JOGL, doesn't it? AoI refused to start on my machine until I updated all the native libraries (*.so) to JOGL 1.1.1.)
Cheers!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Q1: Why do you need to add a jar to the AOI jar file itself?
Q2: Is svgSalamander required for the core AOI code, or for a plugin?
My understanding is that the svgSalamander jar is probably required for a plugin (in this case a renderer) - am I correct?
If so, the svgSalamander jar should not be added to the core AOI jar, but to the dependencies of the plugin itself.
(I'm currently in a hotel, teaching a course, and am using a bandwidth-restricted internet connection, so I am reluctant to do an SVN update on the AOI code until I'm back home at the end of the week.)
Cheers!
Nik
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
As far as I understood, svgSalamander is now needed in core because we can use SVG textures.
Well, repackaging everything into ArtOfIllusion.jar was originally introduced by Peter (see SVN revision 286). :) But I, too, think that this is the easiest way to distribute AoI - (almost…) everything you need is in ArtOfIllusion.jar and that's it. Buoy, for example, is repackaged, too. It's easy to use. Moreover, when a new external library gets added to AoI core, I can just do a "git svn rebase" (or "svn up"), rebuild AoI and I'm done.
It's not a very "clean" way to distribute software, I admit. Actually, AoI should be AoI and nothing more. Everything else (JOGL, Bouy, Jama, svgSalamander, Beanshell, …) should be different packages. But how do you deal with all those different platforms? On GNU/Linux, people are used to use a package manager. Same for *BSD. On Windows, it's complete chaos because every piece of software has its own "package manager" (aka setup.exe). I know nothing about MacOS, but I'm sure Steve™ invented his very own system. ;)
But for me as a lazy person, the current solution is fine. :) AoI's dependencies (JOGL, Buoy, …) are only used by AoI itself, so it's okay.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
ArtOfIllusion uses multiple ClassLoaders: one for the AOI core, and then a separate one for each plugin.
This ensures isolation between plugins, so, for example, 2 plugins can depend on 2 different versions of the same library, and AOI won't explode with class conflicts (there is a compromise in AOI on this in the scripting environment…)
So, if the svgSalamander library is called by core AOI code, then it is a dependency of the AOI core, and should be loaded by the core AOI classloader.
If, however, svgSalamander is called by plugin code (eg a renderer) then it should be loaded by the classloader for that plugin.
I agree the easiest way to get a library into the core ClassLoader is to include the classes into the ArtOfIllusion jar.
I also agree that it is probably "cleaner" to keep the jars isolated, and get the Ant file to build the correct ClassPath entry in the manifest (I have Ant code that does this).
However, before we concern ourselves over how we make the jar available to the classloader, we really need to decide which classloader should be loading it.
Cheers!
Nik
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
SVG Salamander is used by the AoI core. It's used for a feature I implemented recently, to let you use SVG images in textures.
The approach Nik suggests - moving all the different jars into the lib directory and not merging any of them with ArtOfIllusion.jar - is a reasonable approach, and I would have no objection to it, if no one else sees a problem.
Peter
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi there. My son asked me to look into making Art of Illusion multicore/OpenGL/OpenCL-aware, and before I dive in on that (which is really a "learn to swim by being thrown in the deep end" affair-I have a lot of experience with concurrent programming, but precious little with OpenGL/OpenCL), I figured it might be worth trying to get the build into Maven (which I also have a bit of experience at-I'm an SCM/build/release guy at $DAYJOB). This would get all of the 3rdparty jars out of the tree, would make it easier to consume updates from them, would speed up checkouts, plus all of the good things that Maven generally brings.
Would the project appreciate such an effort?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
What did you have in mind by "making Art of Illusion multicore/OpenGL/OpenCL-aware"? It's already pretty heavily multithreaded, so it makes good use of multiple cores, and the UI already uses OpenGL. The raytracer doesn't have any GPU acceleration, but see this thread for some recent discussion of that: https://sourceforge.net/projects/aoi/forums/forum/47782/topic/6259676. If you feel motivated to write an OpenCL raytracer, that would be really cool!
Regarding switching the build system to Maven, what would be the goal of it? I've never found it to be a problem keeping copies of the 3rd party libraries in the repository. What problem would it be intended to solve?
Peter
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
There was some effort by someone some years ago to put AOI into Maven.
He actually found it more difficult than he had anticipated (I'm not really sure why, Maven isn't really that hard to get along with);
and there was really little benefit from the result.
However, it is not rue to say there is no benefit.
It is certainly true that when people first come to develop with AOI, they have trouble understanding how to set up the build environment, and so a build system that could configure itself automatically would certainly benefit them.
Given that AOI uses ANT for its build system, it would almost certainly be easier to add Ivy to the build process to handle downloading libraries for those who want things automated. The nice thing about Ivy in this situation is that it can be made optional, whereas with Maven either you use it or you don't.
In addition, Ant/Ivy will happily support any source tree structure, whereas Maven doesn't (to my knowledge), and AOI's source tree is currently incompatible with Maven.
The bigger issue I encountered when I looked at this last (with Ivy) was that AOI relies on the JMF jar to build, and the JMF licence requires that each user download the jar manually. At that time I could not find the JMF jar in any Ivy or Maven repository, and according to the licence, we were not allowed to put it into a repository of our own either.
- Although it's possible the JMF licence has changed since?
Perhaps one of the AOI V3 tasks might be to remove the reliance on JMF, and move to a different library that:
a) supported current video codecs;
b) did not have such restrictive licencing requirements.
Maybe the OpenJDK project has something suitable?
The other issue we have all encountered is that some of the libraries that AOI relies on (eg JOGL) completely reimplemented their APIs over different releases, so any dependency definition would need to be very specific about the library versions.
At this stage, I would say that once you have downloaded the necessary (and very few) libraries for the AOI build, the build is very simple and very stable.
Meaning you can focus almost immediately on what you are trying to change.
I have an enhanced set of Ant files, which have included Ivy support (without support for JMF), which I would like to contribute to AOI - probably for V3 and beyond.
If you wish to create a Maven POM file for AOI, then feel free - although at the moment, the structure of the AOI tree is not compatible with Maven, as far as I know - and so your Maven build would need to copy/move the source files into a maven-compatible structure.
Just my thoughts - let us know what you decide to do.
Nik
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Perhaps one of the AOI V3 tasks might be to remove the reliance on JMF
Beat you to it. :) I got rid of JMF in 2.9.1. (That's separate from the question of supporting more video formats, of course, which would be useful in its own right.)
I'm not familiar with Ivy, but clearly I should take a look at it.
Peter
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Regarding switching the build system to Maven, what would be the goal of it?
You could taken advantage of Maven's dependency management and analysis, which includes transitive dependency analysis. You could also take advantage of the built-in class path computation, and get your unit tests almost for free (you still have to write them, but Maven will run them for you). You can use any of a number of static analysis tools. You can nail down your dependencies at the top level and have your sub-builds inherit them. You can have a standard way of doing things that any developer familiar with Maven can grasp instantly, without having to spend time to grok your custom ant build (as ALL ant builds are one-offs).
The major win is that you don't have to program your build system, as you do in ant. With ant, there are as many ways to do things as there are people who want to do them, and everyone re-invents their own wheel. With Maven, by following conventions, your build recipes end up being tiny, and sub-builds can inherit a great deal from their parent POM files. Most of the Java world appears to use Maven these days, and there's a huge amount of software already published in the two major public repositories (Central and Codehaus). It becomes fairly easy to program with interfaces and a number of optional implementation backends.
From my perspective as a by-day, professional CM guy, checking in prebuilt dependencies is poor practice: it bloats your workspace, you cannot diff the checked-in binaries anyway, it bloats your repo, and it's terribly easy to end up with multiple copies of the same dependency (sometimes with different, incompatible versions). Maven lets you specify (and enforce) what you're using, will fetch (and cache) all of your dependencies and transitive dependencies, and will catch dependency conflicts.
As for Ivy, I've been down that road, too. I don't recommend it. For one thing, it doesn't get you away from ant and all of ant's attendant nastiness/kludginess. For another, in order to take advantage of transitive dependency management, you end up having to write your own Ivy files for everything, rather than just being able to take advantage of publicly-published Maven artifacts, thereby getting your transitive dependency management for free.
And yes, Maven 2 and Maven 3 allow for build aggregation, with the ability to dispatch to sub-module builds, in dependency order.
Having suffered with ant for years with a MUCH larger project than AoI, and having tried Ivy, my employer is moving to Maven. All of our developers are enthusiastic about it, and we just can't get there fast enough!
If you're willing, I'll make a fork and Mavenize it, and you can take a look and see what you think.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
By all means, go ahead and try creating a Maven version of the build. Just be aware that I'm a big believer in the grammatically correct saying, "If it isn't broken, don't fix it!" The ant scripts were all written years ago and have continued to "just work" ever since. I probably spend an average of about one hour a year doing anything with the build scripts. So if you can produce drop in replacements that work better, that's great. But if it takes significant work to adopt them, such as reorganizing the source repository, I'll be more inclined to just stick with what's already working well.
Peter
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Firstly, my apologies if Peter's or my responses seemed negative or dismissive - I believe we are both trying to be as helpful as possible by giving accurate responses.
Whilst definitely trying to avoid a Maven vs Ivy flame war, I will respond to your previous post.
Firstly, I'll clarify my standpoint: I used to much prefer Ant/Ivy - it fitted much better to the model of simple tasks that do one thing well, and which can be used in combination; and it supported a nice model of private/public repos.
In contrast, I found Maven's dictatorial style annoying.
However, I currently stand pretty much in the middle.
I deal regularly with Maven projects, Ant projects, and Ant/Ivy projects, and experience tells me that Maven isn't as bad as I once thought, and Ivy isn't as good as I once thought.
I find some things in Ivy difficult to understand and drive, whilst I have seen others struggle to understand or modify, or just control, a Maven build.
Currently, it seems to me that both are good tools, each with respective benefits and weaknesses.
Having suffered with ant for years with a MUCH larger project than AoI
I think this is the most significant sentence in your reply.
AOI is such a relatively simple project (in terms of structure and build complexity), that I suspect the incremental improvement from using Maven or Ivy compared to Ant on its own is quite small.
You could taken advantage of Maven's dependency management and analysis, which includes transitive dependency analysis. You could also take advantage of the built-in class path computation,
All of these are supported by Ivy as well, as I presume you know.
and get your unit tests almost for free (you still have to write them, but Maven will run them for you). You can use any of a number of static analysis tools.
[/quote[
Ant can do as many of these on its own.
You can nail down your dependencies at the top level and have your sub-builds inherit them.
If I've understood what you've said correctly, then Ivy can do this also.
You can have a standard way of doing things that any developer familiar with Maven can grasp instantly, without having to spend time to grok your custom ant build
Now we get to the meat of what makes Maven special. :o)
I agree that this is a strength of Maven. I find very often that I can simply tell people "it's a Maven project", and they, by and large, understand how it will be built.
However, this is not to say that someone cannot create a one-off build using Maven - I've seen it, and I've seen the confusion it can cause. The build conforms to all the Maven conventions, but has to do something special, and all the Maven people get really confused when it does.
And of course, if someone comes to build AOI and is *not* familiar with Maven, then that will be a problem.
The project structure that Maven dictates is a very good one, so users unfamiliar with Maven will have little trouble finding things - certainly no more effort than understanding the current structure of AOI.
But, the build process will most likely be a mystery to them.
Maven is effectively a black box, and without learning how Maven thinks, a Maven build is pretty difficult to understand or modify.
Judging only by the posts on the forums, those developers who ask about how the build process of AOI works have never heard of Maven or Ant - and many of them have only just heard of Java and Eclipse.
(as ALL ant builds are one-offs).
Yes-ish.
There are many Ant build systems that coalesce common tasks into inherited Ant file(s), and reuse those inherited targets.
So Ant is effectively a programming language: used poorly, with lots of copy-and-paste code, it can end up being a mess and a maintenance nightmare.
Used well with good reuse, Ant scripts can be elegant and simple to maintain.
The major win is that you don't have to program your build system, as you do in ant. With ant, there are as many ways to do things as there are people who want to do them, and everyone re-invents their own wheel. With Maven, by following conventions, your build recipes end up being tiny, and sub-builds can inherit a great deal from their parent POM files.
Agreed - in general.
However, AOI is a very simple system; its build system is very simple; the Ant scripts are already simple and tiny, with basically no programming inside them.
Most of the Java world appears to use Maven these days, and there's a huge amount of software already published in the two major public repositories (Central and Codehaus). It becomes fairly easy to program with interfaces and a number of optional implementation backends. From my perspective as a by-day, professional CM guy, checking in prebuilt dependencies is poor practice: it bloats your workspace, you cannot diff the checked-in binaries anyway, it bloats your repo, and it's terribly easy to end up with multiple copies of the same dependency (sometimes with different, incompatible versions). Maven lets you specify (and enforce) what you're using, will fetch (and cache) all of your dependencies and transitive dependencies, and will catch dependency conflicts.
I don't see anything in this that marks Maven apart from Ant/Ivy.
For one thing, it (Ivy) doesn't get you away from ant and all of ant's attendant nastiness/kludginess.
I don't see that the AOI Ant files have any of the nastiness or kludginess that can happen in more complex Ant files.
For another, in order to take advantage of transitive dependency management, you end up having to write your own Ivy files for everything, rather than just being able to take advantage of publicly-published Maven artifacts, thereby getting your transitive dependency management for free.
I'm not sure I understand what you're saying here.
If you're criticising the need to write the Ivy dependency rules in separate ivy.xml files, then I accept that this is true; but I've never found it to be a problem to write them.
If you're saying that there are no public Ivy repositories such as Maven's Central or Codehaus, then I'm puzzled. Ivy Roundup is a public Ivy repository; Ivy can use Maven repos as well as its own; and Ivy has the ability to download raw installation, ZIP, or JAR files and create the Ivy metadata automatically.
In addition, Ivy supports internal enterprise-wide repos really well.
So I don't think I have understood your point on this one.
And yes, Maven 2 and Maven 3 allow for build aggregation, with the ability to dispatch to sub-module builds, in dependency order.
As far as I understand it, Ant/Ivy has this ability as well - although I've not used it.
my employer is moving to Maven. All of our developers are enthusiastic about it, and we just can't get there fast enough!
In my experience, the difference has not been so clear cut.
I've seen people wanting the more convention-based approach, and wanting to switch to Maven.
I've also seen people struggling to add a new artifact to an existing Maven build, and failing to understand why the Maven black box just won't do what they want.
I've actually been paid to build an Ant/Ivy build system as an alternative to an existing Maven build system.
They asked for the Ant/Ivy alternative because it would run significantly faster, and be easier to control.
(They had regular problems with Maven attempting to download the internet when all they wanted to do was run a demo.)
In short, building large systems is complex, and any system that helps with that must also be complex, and so at some stage, configuring the build system will entail creating a complex solution.
I too have witnessed large projects switch from Ant/Ivy to Maven.
The process was long, and for some of them at least, the nett benefit was small to non-existent.
I suspect that the better the Ant/Ivy system was to begin with, the smaller the overall benefit.
I also suspect that switching from a good Maven system to a good Ant/Ivy system would often realise little benefit.
Since you pretty clearly have good experience with Ant and Ivy, I presume that your project has enough complex issues that the move to Maven will be a definite benefit, and the greater control that Ant/Ivy provides does not outweigh those benefits.
Which seems to me to reconfirm my opinion that AOI is so simple that any of Ant, Ant/Ivy, or Maven will produce a good result.
If you're willing, I'll make a fork and Mavenize it, and you can take a look and see what you think.
I'm certainly very interested to see the result. I am have no problem with being proven wrong on this issue.
I would also be very interested in comparing your Maven build with a fairly simple Ant/Ivy build.
Bear in mind Peter's comment about avoiding wholesale reorganising of the source repository.
I suspect that your maven build may need to move the AOI source tree into the required Maven structure before it can perform the other steps. I'm happy to help you with that that task if you need.
When running the build scripts, I find that the distribution artifacts (ArtOfIllusion.jar, the libraries tree, the plugins folder, etc. are created in the source tree. Is this the way it is supposed to work, or do I have something messed up?
ljsails
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi all,
I'd like to talk about AoI's build process once more. :)
First of all, I still wonder why the ant build scripts are not available via SVN. When I do a checkout, I have no means to build AoI. There's just a bunch of .java files - no makefile, no SConstruct, no build.xml. That's a little cumbersome. :) Why not just include the build files in the SVN repository?
How do *you* actually build AoI? Do you use Eclipse or another IDE?
Second, while trying to build the latest SVN revision, I realized that all external JAR files are hardcoded in the build script. That means I have to manually edit the build script in order to add "svgSalamander"'s JARs to the classpath. Couldn't we just move all needed JARs into a "lib" subdirectory and tell the build script to add all JARs in this directory to the classpath?
(On a side note, this applies to launcher scripts like "aoi.sh", too. All JAR files are hardcoded, so you also have to edit that file for svgSalamander.)
Finally, there's this thread which lists further improvements:
https://sourceforge.net/projects/aoi/forums/forum/47784/topic/3176847
Cheers,
TroY
Wow, that was an amazingly timely post. As part of implementing the SVG support, I restructured the build script to not hardcode the libraries, and literally just finished checking in all the libraries and build scripts. Do an update and see if it better fits your expectations. :)
Peter
Haha, nice. ;) Thank you!
Well, uhm, *cough*, there's a little problem though.
Okay, the "zipgroupfileset" task basically is what I thought of. But svgSalamander breaks it: The file svgSalamander.jar includes a digital signature which gets copied, too. That's sort of a problem because Java then thinks this signature belongs to the newly created ArtOfIllusion.jar. Thus, when launching AoI I get this:
Dunno, maybe Apple's Java implementation doesn't check signatures, so maybe that's why this didn't happen to you? For the sake of completeness, I'm using this version of Java:
To be more precise, everything in svgSalamander's META-INF directory should not get copied to ArtOfIllusion.jar. Those last two KITFOX files should not be present:
$ jar tf ArtOfIllusion.jar | grep META META-INF/ META-INF/MANIFEST.MF META-INF/KITFOX.DSA META-INF/KITFOX.SFThe sad thing is, it's impossible to filter the contents of JAR files when using "zipgroupfileset". Looks like this is a common problem, see for example:
http://stackoverflow.com/questions/1274879/ant-exclude-files-from-merged-jar-file
Ironically, the "common solution" is not to use "zipgroupfileset" but either hardcode the filenames or do it the way you did it in the past (extract JARs into a temporary directory, then re-jarring everything but META-INF/* …). Well, yes, that's what we wanted to get rid of. :(
I don't know how to fix this, yet. I'll see if I can find a suitable workaround.
Cheers,
TroY
Hi,
I had been going to propose a number of changes for the build process - but I was waiting until AOI 3.0 to suggest them.
However … given the changes currently underfoot, I can offer:
1. some Ant tasks that build a classpath dynamically, and can use it for (among other things) creating the correct manifest entries.
2. Add some Ant tasks that use Ivy to manage the dependencies (which is what Ivy is for).
Cheers!
Nik
Sigh. Well, the easiest solution is just to edit svgSalamander.jar to remove those two files. The only real problem I see is that I'll have to remember to do it every time I update to a new version of SVG Salamander.
Nik, could you explain your suggestions in more detail?
Peter
Hi again,
the best solution I can find is this one:
http://java.randgestalten.de/index.php/2008/10/combine-multiple-jar-files-and-remove-signatures/
The following changes do the trick:
a) You're still able to include all JAR files using globbing, b) no need to manually edit any files.
Still a dirty hack, though…
By the way, the manifest in ArtOfIllusion.jar specifies a class path which includes "lib/jogl.jar" and "lib/gluegen-rt.jar". This is obsolete, isn't it?
(While we're at it: SVN now includes a newer version of JOGL, doesn't it? AoI refused to start on my machine until I updated all the native libraries (*.so) to JOGL 1.1.1.)
Cheers!
Hi Troy,
Firstly, may I ask:
Q1: Why do you need to add a jar to the AOI jar file itself?
Q2: Is svgSalamander required for the core AOI code, or for a plugin?
My understanding is that the svgSalamander jar is probably required for a plugin (in this case a renderer) - am I correct?
If so, the svgSalamander jar should not be added to the core AOI jar, but to the dependencies of the plugin itself.
(I'm currently in a hotel, teaching a course, and am using a bandwidth-restricted internet connection, so I am reluctant to do an SVN update on the AOI code until I'm back home at the end of the week.)
Cheers!
Nik
Hi Nik :)
As far as I understood, svgSalamander is now needed in core because we can use SVG textures.
Well, repackaging everything into ArtOfIllusion.jar was originally introduced by Peter (see SVN revision 286). :) But I, too, think that this is the easiest way to distribute AoI - (almost…) everything you need is in ArtOfIllusion.jar and that's it. Buoy, for example, is repackaged, too. It's easy to use. Moreover, when a new external library gets added to AoI core, I can just do a "git svn rebase" (or "svn up"), rebuild AoI and I'm done.
It's not a very "clean" way to distribute software, I admit. Actually, AoI should be AoI and nothing more. Everything else (JOGL, Bouy, Jama, svgSalamander, Beanshell, …) should be different packages. But how do you deal with all those different platforms? On GNU/Linux, people are used to use a package manager. Same for *BSD. On Windows, it's complete chaos because every piece of software has its own "package manager" (aka setup.exe). I know nothing about MacOS, but I'm sure Steve™ invented his very own system. ;)
But for me as a lazy person, the current solution is fine. :) AoI's dependencies (JOGL, Buoy, …) are only used by AoI itself, so it's okay.
Hi Troy,
My apologies - I didn't make myself clear enough:
ArtOfIllusion uses multiple ClassLoaders: one for the AOI core, and then a separate one for each plugin.
This ensures isolation between plugins, so, for example, 2 plugins can depend on 2 different versions of the same library, and AOI won't explode with class conflicts (there is a compromise in AOI on this in the scripting environment…)
So, if the svgSalamander library is called by core AOI code, then it is a dependency of the AOI core, and should be loaded by the core AOI classloader.
If, however, svgSalamander is called by plugin code (eg a renderer) then it should be loaded by the classloader for that plugin.
I agree the easiest way to get a library into the core ClassLoader is to include the classes into the ArtOfIllusion jar.
I also agree that it is probably "cleaner" to keep the jars isolated, and get the Ant file to build the correct ClassPath entry in the manifest (I have Ant code that does this).
However, before we concern ourselves over how we make the jar available to the classloader, we really need to decide which classloader should be loading it.
Cheers!
Nik
Actually, some of these jars are shipped separately.
Look in <AOI-home>/libs. YOu'll see jogl.jar, gluegen.jar and others in there.
So AOI already uses exteral jars for some packages, and has a classpath entry in the manifest to support this.
Cheers!
Nik
SVG Salamander is used by the AoI core. It's used for a feature I implemented recently, to let you use SVG images in textures.
The approach Nik suggests - moving all the different jars into the lib directory and not merging any of them with ArtOfIllusion.jar - is a reasonable approach, and I would have no objection to it, if no one else sees a problem.
Peter
Hi,
sorry for being late.
Yes, not merging anything into ArtOfIllusion.jar is most probably the best thing to do. ;)
Building AoI currently fails for me, though. Don't know about your Ant, but mine doesn't work with wildcards in "classpath". Hence, I need this patch:
Greetings,
TroY
Sorry about that. I guess you must be compiling with Java 5? I've just fixed it.
Peter
Hi there. My son asked me to look into making Art of Illusion multicore/OpenGL/OpenCL-aware, and before I dive in on that (which is really a "learn to swim by being thrown in the deep end" affair-I have a lot of experience with concurrent programming, but precious little with OpenGL/OpenCL), I figured it might be worth trying to get the build into Maven (which I also have a bit of experience at-I'm an SCM/build/release guy at $DAYJOB). This would get all of the 3rdparty jars out of the tree, would make it easier to consume updates from them, would speed up checkouts, plus all of the good things that Maven generally brings.
Would the project appreciate such an effort?
Hi,
Welcome to the forum!
What did you have in mind by "making Art of Illusion multicore/OpenGL/OpenCL-aware"? It's already pretty heavily multithreaded, so it makes good use of multiple cores, and the UI already uses OpenGL. The raytracer doesn't have any GPU acceleration, but see this thread for some recent discussion of that: https://sourceforge.net/projects/aoi/forums/forum/47782/topic/6259676. If you feel motivated to write an OpenCL raytracer, that would be really cool!
Regarding switching the build system to Maven, what would be the goal of it? I've never found it to be a problem keeping copies of the 3rd party libraries in the repository. What problem would it be intended to solve?
Peter
Hi Extrachrispy :o)
There was some effort by someone some years ago to put AOI into Maven.
He actually found it more difficult than he had anticipated (I'm not really sure why, Maven isn't really that hard to get along with);
and there was really little benefit from the result.
However, it is not rue to say there is no benefit.
It is certainly true that when people first come to develop with AOI, they have trouble understanding how to set up the build environment, and so a build system that could configure itself automatically would certainly benefit them.
Given that AOI uses ANT for its build system, it would almost certainly be easier to add Ivy to the build process to handle downloading libraries for those who want things automated. The nice thing about Ivy in this situation is that it can be made optional, whereas with Maven either you use it or you don't.
In addition, Ant/Ivy will happily support any source tree structure, whereas Maven doesn't (to my knowledge), and AOI's source tree is currently incompatible with Maven.
The bigger issue I encountered when I looked at this last (with Ivy) was that AOI relies on the JMF jar to build, and the JMF licence requires that each user download the jar manually. At that time I could not find the JMF jar in any Ivy or Maven repository, and according to the licence, we were not allowed to put it into a repository of our own either.
- Although it's possible the JMF licence has changed since?
Perhaps one of the AOI V3 tasks might be to remove the reliance on JMF, and move to a different library that:
a) supported current video codecs;
b) did not have such restrictive licencing requirements.
Maybe the OpenJDK project has something suitable?
The other issue we have all encountered is that some of the libraries that AOI relies on (eg JOGL) completely reimplemented their APIs over different releases, so any dependency definition would need to be very specific about the library versions.
At this stage, I would say that once you have downloaded the necessary (and very few) libraries for the AOI build, the build is very simple and very stable.
Meaning you can focus almost immediately on what you are trying to change.
I have an enhanced set of Ant files, which have included Ivy support (without support for JMF), which I would like to contribute to AOI - probably for V3 and beyond.
If you wish to create a Maven POM file for AOI, then feel free - although at the moment, the structure of the AOI tree is not compatible with Maven, as far as I know - and so your Maven build would need to copy/move the source files into a maven-compatible structure.
Just my thoughts - let us know what you decide to do.
Nik
Beat you to it. :) I got rid of JMF in 2.9.1. (That's separate from the question of supporting more video formats, of course, which would be useful in its own right.)
I'm not familiar with Ivy, but clearly I should take a look at it.
Peter
You could taken advantage of Maven's dependency management and analysis, which includes transitive dependency analysis. You could also take advantage of the built-in class path computation, and get your unit tests almost for free (you still have to write them, but Maven will run them for you). You can use any of a number of static analysis tools. You can nail down your dependencies at the top level and have your sub-builds inherit them. You can have a standard way of doing things that any developer familiar with Maven can grasp instantly, without having to spend time to grok your custom ant build (as ALL ant builds are one-offs).
The major win is that you don't have to program your build system, as you do in ant. With ant, there are as many ways to do things as there are people who want to do them, and everyone re-invents their own wheel. With Maven, by following conventions, your build recipes end up being tiny, and sub-builds can inherit a great deal from their parent POM files. Most of the Java world appears to use Maven these days, and there's a huge amount of software already published in the two major public repositories (Central and Codehaus). It becomes fairly easy to program with interfaces and a number of optional implementation backends.
From my perspective as a by-day, professional CM guy, checking in prebuilt dependencies is poor practice: it bloats your workspace, you cannot diff the checked-in binaries anyway, it bloats your repo, and it's terribly easy to end up with multiple copies of the same dependency (sometimes with different, incompatible versions). Maven lets you specify (and enforce) what you're using, will fetch (and cache) all of your dependencies and transitive dependencies, and will catch dependency conflicts.
As for Ivy, I've been down that road, too. I don't recommend it. For one thing, it doesn't get you away from ant and all of ant's attendant nastiness/kludginess. For another, in order to take advantage of transitive dependency management, you end up having to write your own Ivy files for everything, rather than just being able to take advantage of publicly-published Maven artifacts, thereby getting your transitive dependency management for free.
And yes, Maven 2 and Maven 3 allow for build aggregation, with the ability to dispatch to sub-module builds, in dependency order.
Having suffered with ant for years with a MUCH larger project than AoI, and having tried Ivy, my employer is moving to Maven. All of our developers are enthusiastic about it, and we just can't get there fast enough!
If you're willing, I'll make a fork and Mavenize it, and you can take a look and see what you think.
By all means, go ahead and try creating a Maven version of the build. Just be aware that I'm a big believer in the grammatically correct saying, "If it isn't broken, don't fix it!" The ant scripts were all written years ago and have continued to "just work" ever since. I probably spend an average of about one hour a year doing anything with the build scripts. So if you can produce drop in replacements that work better, that's great. But if it takes significant work to adopt them, such as reorganizing the source repository, I'll be more inclined to just stick with what's already working well.
Peter
Hi Extra,
Firstly, my apologies if Peter's or my responses seemed negative or dismissive - I believe we are both trying to be as helpful as possible by giving accurate responses.
Whilst definitely trying to avoid a Maven vs Ivy flame war, I will respond to your previous post.
Firstly, I'll clarify my standpoint: I used to much prefer Ant/Ivy - it fitted much better to the model of simple tasks that do one thing well, and which can be used in combination; and it supported a nice model of private/public repos.
In contrast, I found Maven's dictatorial style annoying.
However, I currently stand pretty much in the middle.
I deal regularly with Maven projects, Ant projects, and Ant/Ivy projects, and experience tells me that Maven isn't as bad as I once thought, and Ivy isn't as good as I once thought.
I find some things in Ivy difficult to understand and drive, whilst I have seen others struggle to understand or modify, or just control, a Maven build.
Currently, it seems to me that both are good tools, each with respective benefits and weaknesses.
I think this is the most significant sentence in your reply.
AOI is such a relatively simple project (in terms of structure and build complexity), that I suspect the incremental improvement from using Maven or Ivy compared to Ant on its own is quite small.
All of these are supported by Ivy as well, as I presume you know.
However, this is not to say that someone cannot create a one-off build using Maven - I've seen it, and I've seen the confusion it can cause. The build conforms to all the Maven conventions, but has to do something special, and all the Maven people get really confused when it does.
And of course, if someone comes to build AOI and is *not* familiar with Maven, then that will be a problem.
The project structure that Maven dictates is a very good one, so users unfamiliar with Maven will have little trouble finding things - certainly no more effort than understanding the current structure of AOI.
But, the build process will most likely be a mystery to them.
Maven is effectively a black box, and without learning how Maven thinks, a Maven build is pretty difficult to understand or modify.
Judging only by the posts on the forums, those developers who ask about how the build process of AOI works have never heard of Maven or Ant - and many of them have only just heard of Java and Eclipse.
If you're saying that there are no public Ivy repositories such as Maven's Central or Codehaus, then I'm puzzled. Ivy Roundup is a public Ivy repository; Ivy can use Maven repos as well as its own; and Ivy has the ability to download raw installation, ZIP, or JAR files and create the Ivy metadata automatically.
In addition, Ivy supports internal enterprise-wide repos really well.
So I don't think I have understood your point on this one.
I've actually been paid to build an Ant/Ivy build system as an alternative to an existing Maven build system.
They asked for the Ant/Ivy alternative because it would run significantly faster, and be easier to control.
(They had regular problems with Maven attempting to download the internet when all they wanted to do was run a demo.)
In short, building large systems is complex, and any system that helps with that must also be complex, and so at some stage, configuring the build system will entail creating a complex solution.
I too have witnessed large projects switch from Ant/Ivy to Maven.
The process was long, and for some of them at least, the nett benefit was small to non-existent.
I suspect that the better the Ant/Ivy system was to begin with, the smaller the overall benefit.
I also suspect that switching from a good Maven system to a good Ant/Ivy system would often realise little benefit.
Since you pretty clearly have good experience with Ant and Ivy, I presume that your project has enough complex issues that the move to Maven will be a definite benefit, and the greater control that Ant/Ivy provides does not outweigh those benefits.
Which seems to me to reconfirm my opinion that AOI is so simple that any of Ant, Ant/Ivy, or Maven will produce a good result.
Bear in mind Peter's comment about avoiding wholesale reorganising of the source repository.
I suspect that your maven build may need to move the AOI source tree into the required Maven structure before it can perform the other steps. I'm happy to help you with that that task if you need.
If you're interested, here is a link to the older thread on mavenising AOI:
http://sourceforge.net/projects/aoi/forums/forum/47784/topic/1600368
Looking forward to your thoughts.
Cheers!
Nik
Question about the current build process:
When running the build scripts, I find that the distribution artifacts (ArtOfIllusion.jar, the libraries tree, the plugins folder, etc. are created in the source tree. Is this the way it is supposed to work, or do I have something messed up?
ljsails