Plugins repository: Imagineering thread

Luke S
2 days ago
  • Luke S

    Luke S - 2016-09-29

    So, the migration of the core source repository to github has gone pretty well. We've even received our first pull requests from a third party.

    Now its time to start talking a little about how to handle the future of the plugins repo, and any enhancements to it. This will be a much larger project than the source migration, so take your time to consider your thoughts. This will not happen overnight.

    The following discussion will get a bit technical. For an end user, the goal is to keep the process of getting or downloading plugins simple and available in-app, either through the existing SPManager or a successor.

    Plugin re-builds

    In the past, we have been bitten by non-functional plugins that have not been compiled against current AOI build. We need to look at building a new copy of every plugin every time we release a new version of AOI.

    Several automated tools do this, and ideally, they would taek care of uploading teh new versions with minimal supervision or interferance.

    Plugin source code

    It would seem to make sense to migrate all of the plugins that we have source for over to git. We could do this two ways:

    • All plugins in one repository, each in a subdirectory.
      • Advantage: all plugins built from one project. Perhaps a bit simpler to set up.
      • Disadvantage: Any collaborator that needs access to one plugin gets acess to all.
    • Since adding new repositories to a github organization is simple and cheap, have a separate source repository for each plugin.
      • Advantage: Simple to give push access priviledges to someone for one plugin only.
      • Advantage: Each plugin can build very simply. No need to have some conditional build-script, etc.for all of the plugins

    If I've missed anything, or you have thoughts on either of the above, or any alternatives, post a reply below.

    Plugin binaries

    Its been proposed that compiled plugins be distributed through a git repository themselves.

    Presumably, something like This stackoverflow post to prevent the user from having to pull down the entire plugin database to get one plugin.

    The internt here is be able to track several versions of each plugin efficiently. This is a bit of an abuse of git for a non-standard purpose.

    We could also continue to use a standard web server as we currently do. Alternatives/suggestions welcome

    And beyond

    One idea that's been around a bit has been adding a repository for (User-contributed) Textures, Materials, and possibly full-up objects or scenes.

    What would you like to see in this space, and how do your see it working?

  • Luke S

    Luke S - 2016-12-15

    Since We're in the home stretch for version 3.0.3 (Done, other than issues with lauching on OS X) I'd like to figure out where to go next.

    My current thought is that the plugin situation should be addressed next. We still need to figure out what the future of plugin management looks like, but in the mean time, we can start combing through the plugins, getting them set up with good, repeatable builds, and updatiing them for compatibility with core.

    Question: What plugins should get priority? Personally, I think that the following are fairly universally installed:

    • Polymesh
    • Log Plugin
    • DisplayModeIcons

    These have usefull functionality, but need some major re-work to be used with current AOI:

    • AdvancedRendering
    • EffectsCatcher

    What say you? What plugins would you most like to see updated or improved?

  • Pete

    Pete - 2016-12-15

    What plugins would you most like to see updated or improved?

    That's a good list to start with. As I recall interoperability with other software was ranked pretty high in some previous discussions, so import-export tools might be a good addition.

    Plugin source code

    I don't know what the technical side of it would be but it'd probably be the most user frienly approach to have a common acces point to everyting -- the core AoI, plugins, textures/materials, themes.... It seems to me that now when there is/has been a different SF-project for plugins, not many people have really used it. Issues with plugins are usually discussed in the AoI main project.

    And beyond

    YardStick asked about updating the textures and materials library some time ago (and if it could be done through GitHub). I don't know if he had anythig specific on his mind.... :)

  • Luke S

    Luke S - 2016-12-15

    re: Interop: we should probably add the existing translators to the list. Part of the problem here is that the various importers are, AFAIK, somewhat limited in how much of their format they support, and updating or completing them is non-trivial.

    As a side project, I'm working on building a plugin that will interface AOI with the assimp project, which is a c++ library that supports a lot of formats. I'd evetually like to see this, or something like it be the primary AOI format support.

    Plugin code, I think, should end up as separate projects under the ArtOfIllusion 'organization' on github. ArtOfIllusion/polymesh, ArtOfIllusion/EffectsCatcher, etc. Each one could be forked, and Pull Requests created against it, just like we have begun doing with the core project.

    I see what you mean regarding a one-stop shopping experience from the user perspective. Perhaps it would make sense to add a couple of forums to the discussion board here for plugins and texturing.

    We still need to have a technical discussion about where/how to host plugin binaries and texture packs, but as long as we have web space, either through Sourceforge, Github, or something different, we can set up the user interface however we like.

  • Pete

    Pete - 2016-12-18

    Plugin code, I think, should end up as separate projects under the ArtOfIllusion 'organization' on github. ArtOfIllusion/polymesh, ArtOfIllusion/EffectsCatcher, etc. Each one could be forked, and Pull Requests created against it, just like we have begun doing with the core project.

    That sounds about right, but I came to think, if it would be possible to it somehow like:

    • ArtOfIllusion/ArtOfIllusion
    • ArtOfIllusion/plugins/EffectsCatcher
    • ArtOfIllusion/plugins/PolyMesh
    • ArtOfIllusion/themes/NightFall
    • ArtOfIllusion/templatefiles/MedievalCastle
    • ArtOfIllusion/templatefiles/GardeningTools

    Looking at all the material there is now and all that there potentially could be, the library would soon begin to look rather desperate to read if every plugin (& al) is their own project directly under the main organization. Though it certainly would be a good thing to have them all in one place.

    The above contains the idea, that I'd hope to have the UI-themes separated from the plugins... Now (as it turned out) some of the users don't even know about themes.

    I don't know, what is possible in GH, but just contemplating the future practices.

  • Luke S

    Luke S - 2016-12-22

    Due to the nature of how git and github work, it's not possible to have a nested directory structure like that.

    At this point, I'm only looking at having plugin source set up as separate projects, since as you note, a huge list of items would get to be unwieldy.

    I have some very serious concerns about (ab)useing git for binary files, such as any future library. However, If we do chose to use it for that anyway, I'd imagine that we would use a single, reserved project for each type of binary:

    • Plugin jars would be in the "Plugins" project, but the individual plugin sources would be separate.
    • Scripts would all go in a "Scripts" project - they are all small enough, and should not need as much updating, etc. that this would seem to make more sense. These are the most suited to being served over git.
    • Textures, Materials, and Template models would have a project per category, but these are the least suited to being served over git, and the fact that these are the categories where you are going to want contributions from non-programmers makes it worse. Any such library would need a very good, simple interface for more artistically-inclined types (@pete, your engineering background makes you much better prepared to work with the programing type tools than most users will ever be.)
  • Pete

    Pete - 2016-12-22

    @pete, your engineering background makes you much better prepared to work with the programing type tools than most users will ever be.

    Hehe... :D I guess so.

    But what ever, I do agree that the UI should be as simple as possible ... Somehow many of the things are of the kind that a "natural" address from a user's point of view would be the homepages of the software. -- But a lot of work and thinking needed there too and at least I am lost with the www-page tools that the world uses nowadays.

    Last edit: Pete 2017-01-16
  • Luke S

    Luke S - 2017-01-17

    I've gotten a few of the plugins' sources uploaded to github, and will be continuing with the others. In this process, I've checked in some updates to the Hologram plugin that I made some time ago.

    This has lead me to a question that is not quite clear to me:

    How does SPManager know what version of a plugin it should load? Does it always load the latest version, or does it have an ability to check version compatibility? If so, how?

    This also leads to a broader question: How should we handle version support? Plugins change for two reasons:

    • Bugfixes and feature enhancements
    • Updates to remain compatible with the latest version of the app

    How many versions of AOI should we support in the plugin repository?

    • Just the latest release, with a test build to check compatibility with the most recent checkins
    • Back a certain number of releases
    • Latest release and one semi-major release back (Would mean that we currently would support AOI 2.9 API)
    • Other

    If we chose the first, part of the AOI release process would be making sure that all of the plugins will compile against the current release candidate. If we were to try to support older AOI, we would run into a situation where the current plugin source is not compatible with the old AOI. We would need to have a way to mark such plugins to tell the build process and SPManager or Librarian which version of the source to use.

  • Pete

    Pete - 5 days ago

    AFAIK the SPM supports only the last version, whether it is compatipble to your version of AoI or not. If someone knows otherwise, then please correct.

    I feel that it'd be reasonable to keep it supporting the last releastes, but it should be made sure, that the versions of plugins, that are available are confirmed to be compatible with core AoI (which is not the case currently). -- And then some other channel for older verions to those who need them.

    At least to me that would sound like an acceptable basis to the plan.

  • Nik Trevallyn-Jones

    Hi Luke, and All,

    Actally ... SPM maintains as many versions of a plugin as you choose to keep in the repository - similar to any source control system such as GIT or Subversion.

    The correct version of any plugin is chosen based on the version of AoI that makes the request.

    SPM actually understands version ranges, so if version 1.1 of MyPlugin is flagged as being compatible with AoI 2.5, and version 1.2 of MyPlugin is flagged as being compatible with AoI 2.8, then SPM understands that MyPlugin 1.1 is compatible with AoI versions 2.5-2.7.

    The metadata for the version association is kept in a very simple text file.
    Let's use the (recently updated - thanks Luke!) Hologram plugin as an example.

    The current respository can be viewed from the web via the SourceForce AOISP project:

    Clicking on the "Browse Repository", and following links we get to the Hologram plugin folder:

    There is a file in here called Hologram.txt which contains the version information.
    **Note that in a major update to the plugin system - at AoI 2.5, I think - this file can now also be called: versions.txt.

    Looking inside Hologram.txt, we see:
    3.0 : Hologram-3_0.jar
    2.7 : Hologram-2_7.jar
    2.6 : Hologram-2_6.jar
    2.5 : Hologram-2_5.jar
    2.4 : Hologram-2_4.jar

    In the case of the Hologram plugin, I have named the Hologram Jar file after the version of AoI that it is compatible with, but there is no requirement to do this.

    Looking in the HologramPlugin repository folder, all the different installable files are kept in there:
    AdvancedRendering.jar 17-Oct-2007 09:48 1.1M
    Hologram-2_4.jar 12-May-2007 06:32 5.5K
    Hologram-2_4.xml 12-May-2007 06:32 1.0K
    Hologram-2_5.jar 29-Oct-2007 00:24 405K
    Hologram-2_5.xml.old 02-Jun-2007 10:19 1.3K
    Hologram-2_6.jar 23-Aug-2008 14:23 423K
    Hologram-2_7.jar 27-Nov-2008 08:24 425K
    Hologram-3_0.jar 23-May-2015 04:48 423K 12-May-2007 06:32 751K
    Hologram.txt 23-May-2015 04:50 115
    Hologram.xml 17-Oct-2007 09:49 1.0K 12-May-2007 06:33 399K
    Renderers.jar 17-Oct-2007 09:49 200K

    For a different example, look at the Physics plugin:

    It's versions.txt reads as follows:
    2.9 : Physics-0_7.jar
    2.4 : Physics-0_6.jar

    And finally, a plugin can be marked as no longer supported, with a '-' character - see RenderUtils for an example:

    It's RenderUtils.txt reads as follows:
    2.3 : -
    1.9 : RenderUtils_1.9.jar

    While we're on the topic of Plugins, versions, and the capabilities of SPM, I'll point out a few other supported features:
    * transitive dependencies
    * A plugin can specify other plugin(s) that it depends on, and SPM will ensure those are installed (after prompting the user).

    • additional files
      • a plugin can specify a "fileset" of additional files that are part of the plugin, and must be downloaded when installing the plugin.
        • for an example, see the HIDPlugin which requires additional Native library files to be installed:
          • see the and files in the HIDPlugin repository folder.

    The metadata for these features is in the plugin metadata XML (extensions.xml) file inside the plugin JAR.

    I really need to write some documentation on the available features in the extensions.xml file...


    Last edit: Nik Trevallyn-Jones 5 days ago
  • Nik Trevallyn-Jones

    Out of interest - I started a re-design of SPM intended to use GIT and the jGIT library to store both the source and the binary artifacts...

    I didn't get very far, though...


  • Luke S

    Luke S - 4 days ago

    Thanks for the info, Nik.

    Some documentation about what SPManager can do might be handy. I've not been able to follow it all in code, though I suspect that is because the working version relies on an on-server script or program, which I've never gotten access to.

    One of the reasons I'm trying to lock down a workable framework for versioning and plugin service is that I intend to upgrade the build systems a bit. Given AOI's rather slow development cycle, I really think that the project would benefit from a 'nightly build' of some sort, so that we can battle-test new features and bugfixes more thoroughly before an actual release.

    One thing that this will require to be practical is an automated build, and I want to make sure that we can build all of the plugins against the latest nightly, so that they can get an early test as well. However, I want to be able to discard all of the interim nightly artifacts once we have released a stable build.

    I'd also like to enable using secondary repositories with less fuss. Most plugins, I suspect will end up in the official repository, but I might see custom texture libraries or models, being made available by third parties that wish to maintain distribution control (Perhaps selling access to their work?)

    • Pete

      Pete - 3 days ago

      One thing that this will require to be practical is an automated build, and I want to make sure that we can build all of the plugins against the latest nightly, ....

      Well, this would be (at least a step in) the way of solving the issue that, when plugin is marked to work with a certain AoI-version up, the SPM still finds it even if it is not working anymore unless it has been marked to be uncompatible.... Which can be confusing at times. Q.E.D. :)

  • Nik Trevallyn-Jones

    Hi Luke,

    The on-server program is avaiable somewhere - I remember pushing changes to it years ago when SourceForge changed something about how the webservers worked.

    I'll have to look around to find which repo I pushed it to - otherwise I'll just dig it up from my old local repository.

    However, the server-side component really only processes the versions.txt file, and does very little with the extensions.xml file in the JAR.

    Regrading nightly builds, any of the CI tools such as Jenkins or Hudson is able to do what you want with few or no probems.

    I'm happy to set up a Hudson server and project to get something started.

    The current SPM uses a parameterised definition for the download repo(s). I don't believe it allows simtansous access to multiple repos, but it certainly allows switching between different repos.
    - Perhaps this is what you mean by "less fuss"? :)


  • Nik Trevallyn-Jones

    Hi Luke,

    Ok, the latest source of the on-server code (actually a C program which is run as a CGI script in the SF web-servers) is in the aoisp file download area here:

    As you can (probably) see, it simply parses versions.txt - or the same file with the provious file convention; and then extracts the plugin or script metadata (eg extensions.xml), and sends that back to the client (SPManager).

    All the features of the extensions.xml file are implemented in the client-side Java code of SPManager.

    However, due to the history of SPManager, the code in there is fairly convoluted, so I will happily write some doco on it.


  • Peter Eastman

    Peter Eastman - 3 days ago

    I'm happy to set up a Hudson server and project to get something started.

    Rather than running our own server, it might be easier to use one of the online CI services. I've used travis for other projects, and it's free for open source projects.


  • Luke S

    Luke S - 3 days ago

    I was thinking Travis as well. Not only is it free, but it also is pretty neatly tied in to github. It can even deploy binary packages for actual releases. Just need to refactor the build scripts a bit - Either put the packagin steps into the ant definition, or hand-code it for travis. Neither should be hard.

    Thanks for the link, Nik, I'll have to take a look. I learn more about the SPManager every time it comes up. Some documentation of supported options would be great, as I've had challenges getting anywhere in the source. I haven't even been able to add support for extracting XML headers from gradle scripts, in addition to the beanshell scripts that it currently handles.

    By "Less Fuss" I was thinking more on the hosting end. Any customization or special setup is fuss, especially for people who do not spend a lot of time on system admin.

    Speaking of nightly builds: I've never really done them before. Is it typical to only have the latest nightly available, or does one keep available every nightly back until the latest stable release?

  • Nik Trevallyn-Jones

    Hi Luke,

    • I'm not sure I see any need to modify the AoI build scripts.

      • anything you need beyond what is in there already can be defined in most CI tools' build steps;
      • that being said, it would certainly help if AoI could be built with a single ant command;
    • I'll take a look at Travis then :)

    • for nightly builds, I normally keep the artifacts around for some fixed amount of time - eg 10 days.

    • what were you trying to extract from the Gradle files?

      • I'm happy to help with that;
  • Nik Trevallyn-Jones

    Ok, to set up a Travis build for the ArtOfIllusion GitHub repository, I would need someone to give me admin access to the repository.

    Otherwies, I'll need to let someone else set up the Travis build...



Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

No, thanks