Plugin with native dependency

[Luke S]

I'm looking at building as a plugin an Importer-Exporter harness that would make use of the Assimp project. This brings up an interesting issue:

The assimp library is written in C++, and needs to be compiled separately for each platform and bitwidth. The compiled library is >4MB for the windows variants, and I'm trying to figure out the best way to handle it. I've come up with several options:

• Bundle all libraries in one .jar, have the plugin unpack and install the version that matches the running platform/jvm.
  • Pros: Simple, no thought required from user, just works
  • Cons: ~20MB for various libs: a plugin the size of AOI core package!
• Separate plugins for each platform, possibly bitwidth as well.
  • Pros: smaller plugins, plugin does not have to try to detect platform.
  • Cons: plugin management just got harder - User has to select correct platform, or the plugin system needs to do platform detection/matching. Seems very hacky.
• Build this feature in core, no Plugin. Manage native libraries like several others, placed during install process.
  • Pros: no giant downloads, no reason for user management.
  • Cons: Is this really a core feature? Possibly. Obtaining and building the correct libraries is now part of the AOI build process.

Any thoughts/ideas on how best to handle this would be appreciated.

[Nik Trevallyn-Jones]

Hi Luke,

The AOI plugin system already caters for plugins with native libraries.
The HIDPlugin does exactly that - it uses the jinput (open-source) native library.

SPManager downloads the jars, plus the native library files automatically.
I did look at adding code in SPManager to only download the files for the current OS and architecture (bitwidth), but many users are in the habit of just copying AOI installations from one machine to another, so SPManager currently copies all listed files unconditionally.

If you want, I could implement filter logic in SPManager, which would enable you to restrict SPManager to only download a file if it matched the filter requirements, such as OS type and architecture.

You should provide (at least) versions of the native library for: Windows (32 and 64); Linux (32 and 64); and OSX (32 and 64 - although these are built into a single file).

If you browse to the HIDPlugin directory in the AOISP repository, you'll see the native libraries in there as individual files:
http://aoisp.sourceforge.net/AoIRepository/Plugins/HIDPlugin/

If you extract the "extensions.xml" file from the HIDPlugin jar, you will find a <fileset> element that contains the <file> elements for each of the the native libraries. The "todir" attribute is used to tell SPManager where to copy the downloaded file to. Note the use of the "$APP_DIRECTORY" meta-directory name.

Finally, you'll need to make arrangements for your native libraries to be found and loaded by Java.
The simplest approach is to follow what the HIDPlugin does:
* package the JNI classes in a separate JAR (this is how most such libraries are packaged anyway);
* include this JNI jar in the <fileset> element (for HIDPlugin, this is "jinput.jar");
* add an <import> element for this file (eg jinput.jar) in extensions.xml which causes SPManager to add this file to the classloader for the plugin;
* ensure all the native libraries are put into the same directory (for HIDPlugin: $APP_DIRECTORY/lib)
* ie ensure the "todir" attribute is the same for the JNI jar and all the native library files.

The steps above will ensure that the classloader for your plugin will be able to find the JNI jar file (eg jinput.jar) plus all of its native library files.

Let me know if you want further clarification - or features added to SPManager. :)

Finally, I agree that the Import/Export functions are plugin functions and not part of the AOI core - as per the current structure.

Good luck, and feel free to ask for help! :)

Cheers!
Nik

[Luke S]

The AOI plugin system already caters for plugins with native libraries.

Ooh... Learned someting new.

many users are in the habit of just copying AOI installations from one machine to another, so SPManager currently copies all listed files unconditionally.
If you want, I could implement filter logic in SPManager, which would enable you to restrict SPManager to only download a file if it matched the filter requirements,

Might be worth considering, especially with larger libraries. I'm not sure why Assimp is as large as it is, but its...significant.

versions of the native library for: Windows (32 and 64); Linux (32 and 64); and OSX (32 and 64 - although these are built into a single file).

I'm aware. That's going to be a whole new adventure for me, as I have not done native compilation in 15 years, and that was just kid stuff out of "C++ for dumies" (Not kidding here)

Assimp is old-school OSS, assumes all users are set up to build their project, and since cross-compiling on a Windows box seems like a mess (most how-tos, etc. basically say dual-boot to linux or run a series of kind-of ported GNU tools on Cygwin + extra packages...) I'm for the first time in a while seriously thinking about getting a penguin party running. Not yet sure if I'm going dual-boot or VM. Thoughts on that would be appreciated, if you've ever tried it.

package the JNI classes in a separate JAR (this is how most such libraries are packaged anyway);

Definitely true. Assimp will actually have three artifacts:

• Standard Assimp shared library, w/c & c++ interface
• Hand-written "JNI friendly" c interface - translates assimp internals into something more java-ish
• The java end of the JNI pairing. scene-graph-ish interface w/abilities to load and process various files. This is what I would write translations to AOI structures against. Want to triangulate and enforce winding order on an arbitrary mesh? Done before any data makes it into java land!

This, however, does not get me out of messing in native code. Items 2 and 3 above are only partly completed - basically, import as is, no export or processing. So I will need to help flesh out those bits of the assimp API as I go. (And may need to ask for help along the way) Is this getting scary yet?

Let me know if you want further clarification - or features added to SPManager.

You were very clear. I do have a wishlist for the future, but for now I need to learn how to get gcc up and running...

[Nik Trevallyn-Jones]

Ooh... Learned someting new.

:) Actually, SPManager supports a number of "special" options to deal with various downloading and classloader requirements.

Might be worth considering, especially with larger libraries.

Ok. The syntax of both the existing <filter> and <assert> plugin elements are attributes in the form key="[op]value[op]". Eg <filter beta="confirm"/> or <assert java.version="1.5+"/>.
So how about I add support to the <file> element such that any attribute with a dot (".") in its name will be treated as a filter on that file?
Ex: <file name="jinput.dll" todir="..." os.name="Windows" os.arch="i386"/>

Alternatively, I could add a single "filter" attribute:
<file name="jinput.dll" todir="..." filter="os.name=Windows; os.arch=i386"/>

I personally prefer the first syntax, but I am happy to hear other opinions and thoughts.

Not yet sure if I'm going dual-boot or VM. Thoughts on that would be appreciated, if you've ever tried it.

I am no fan of dual-boot, so unless your laptop/desktop is criminally low on CPU and RAM, I would use VMs.

I used to develop AOI on a Linux box, so compiling the linux libs for HIDPlugin was easy. I resurected an old Mac Cube to compile the OSX binaries, and I installed the "free" VisualStudio to compile the Windows binaries on a VM (I think).

Given how much effort it takes to boot OSX in a VM, you may need to find an old Mac around somewhere to do the OSX compiles.

I now have a powerful OSX box wih VMs for both linux and Windows, so I that problem is solved pretty well now.
Worst comes to worst, someone like Peter E or myself can run the OSX compile for you once you have the Linux and Windows compiles running.

It is possible to write C code that calls C++, C++ code that can be easily called from C, and C++ code that calls C, so your interface code shouldn't be that difficult. :)
Feel free to ask for help :)

Cheers!
Nik

[Nik Trevallyn-Jones]

Hi Luke,

From your posts, it seems you may have limited experience running VMs. If that is the case, then here are the basic tips that enable VMs and help keep them fast:

• Virtualisation instruction set - VTx, etc. You have to ensure the virtualisation features are enabled in your BIOS as well.
  You can run (32bit) guest VMs on a 32bit host, but in general, its best to run VMs on a 64bit host, even though 64bit is so wasteful of RAM.

• multiple CPU cores. The minimum you need is 2: 1 for the host and 1 for the guest.
  More is better.
  In this situation, I do not regard Intel's Hyperthreading to be worth much, if anything - in my experience a hyperthreaded "core" is nowhere near an additional CPU - physical cores are far better for VMs.
  So if your hardware has 2 physical cores which are also hyperthreaded (which Intel then counts as 4 cores), in terms of being able to run VMs, you have a 2 core box.

• RAM - enough for both host and guest.
  More is (much) better.
  VMWare and VirtualBox will usually default a newly-created guest to 512MB RAM, so a 4GB 32bit box, or a 6GB 64bit box should be able to run at least 1 VM (at a time).
  8, 12, or 16GB is way better - and pretty cheap to install these days :)

• SSD. Solid-State DIsks are so much faster, and make a really noticeable difference when running VMs.
  I don't believe there is any benefit to running an SSD externally over USB, so either an internal SSD, or external SSD over eSATA would be my approach. I've not tried SSD over USB3, and am dubious - but it could yield a benefit.

• In general, my experience is that VMWare Player is still the fastest for running VMs, (fairly closely) followed by VirtualBox. Unless you enjoy torture and self-mutilation, do not even consider QEMU (it is meant for completely different use-cases anyway).

VMWare player is also the easiest one when creating and configuring VMs. The downside is that the free VMWare Player is oh-so buggy. Even with the latest version, I am still dealing with idiot bugs - my current installation supposedly syncs its date from my host, but is suddenly getting it wrong by about 6 months, causing all sorts of errors - including when browsing to websites - with things appearing to be happening "in the future".

VirtualBox is messier to install; create VMs; and install VM tools; but I've not had anywhere near the amount of bugs. But it does run somewhat slower...

Hope this helps! :)

Cheers!
Nik

[Luke S]

It is possible to write C code that calls C++, C++ code that can be easily called from C, and C++ code that calls C

Yep, Assimp actually has a C-API for language interop purposes, the JNI and java layers are built on top of that.

how about I add support to the <file> element such that any attribute with a dot (".") in its name will be treated as a filter on that file?
Ex: <file name="jinput.dll" todir="..." os.name="Windows" os.arch="i386"/>
Alternatively, I could add a single "filter" attribute:
<file name="jinput.dll" todir="..." filter="os.name=Windows; os.arch=i386"/>

Either format looks usable to me. I'd wonder if it might make more sense to nest the selectors:

<fileset>
    <file todir="$APP_DIRECTORY/lib">jinput.jar</file> <--all systems -->
  <file-filter os.name="Windows">
     <file todir="$APP_DIRECTORY/lib">jinput-wintab.dll</file> <--all windows systems -->

    <file-filter os.arch="i386">
      <file todir="$APP_DIRECTORY/lib">jinput-raw.dll</file>  
      <file todir="$APP_DIRECTORY/lib">jinput-dx8.dll</file>
    </file-filter>

    <file-filter os.arch="x64">
      <file todir="$APP_DIRECTORY/lib">jinput-raw_64.dll</file>
      <file todir="$APP_DIRECTORY/lib">jinput-dx8_64.dll</file>      
    </file-filter>
  </file-filter>

  <file-filter os.name="linux" os.arch="i386">
    <file todir="$APP_DIRECTORY/lib">libjinput-linux.so</file>  
  </file-filter>

  <file-filter os.name="linux" os.arch="x64">
    <file todir="$APP_DIRECTORY/lib">libjinput-linux64.so</file>
  </file-filter>

  <file-filter os.name="OS X">
    <file todir="$APP_DIRECTORY/lib">libjinput-osx.jnilib</file>
  </file-filter>

</fileset>

Looks rather verbose, but allows for inclusion of a single file on several platforms. Or would you solve that by placing several entries like so:

<fileset>
<--good for all windows platforms -->
  <file todir="$APP_DIRECTORY/lib">MyAwesomeNonNativefile.custom os.name="Windows"</file>
<-- for some reason is only good for 64 bit linux -->  
  <file todir="$APP_DIRECTORY/lib">MyAwesomeNonNativefile.custom os.name="linux" os.arch="x64"</file>
</fileset>

Perhaps I'm over-thinking this, I've never designed or build an XML Schema before, so I'm probably making every newbie mistake in the book...

Also, we need to keep in mind that the OS arch is not the critical data. We need to know whether we are running 32 or 64 bit JVM (on windows, at least, it's very easy to end up running 32 bit java on 64 bit hardware/OS, and you need to match the Native libraries to the Java environment. A missing file here is what caused the long-running windows install bug...)

may need to find an old Mac around somewhere to do the OSX compiles.

A quick look on craigslist shows that the only sort of thing I am likely to find locally within my budget is an ancient Powermac G4 or similar (500 MHZ, and a whole 512MB ram! Wow! what a system!)

This may not actually be a problem, as the library is really mostly self contained, and the only interfacing it does with the outside world are through JNI and asking the OS for a file. I may be able to cross-compile with GCC or similar - will need to try it!

Thanks for the hints about VMs. I have an i7 Q 720, so 4 real cores, with virtualization support, and 16 GB ram. I'm currently trying to clean out my SSD to make room - Must really just keep a bunch of my archival stuff on my secondary drive. So I think I'm set there.

I had virtualbox running at one point, but I think that was before the guest extensions were mature. Should consider again. I may simply start by trying Hyper-V, as I have win 10 Pro (got one of the free upgrades) and it is installed by default.

[Nik Trevallyn-Jones]

Hey Luke,

Also, we need to keep in mind that the OS arch is not the critical data. We need to know whether we are running 32 or 64 bit JVM (on windows, at least, it's very easy to end up running 32 bit java on 64 bit hardware/OS, and you need to match the Native libraries to the Java environment.

This is very true!
I've spent too much time when teaching students to install Eclipse having to explain this very point (eclipse uses SWT, which uses native libraries...).

We'll have to test further, but looking quickly at documentation, it looks like sun.arch.data.model shows the VM architecture - but it may only be set by Sun (now Oracle) VMs...

I'd wonder if it might make more sense to nest the selectors:

I'm trying to avoid nested elements - either with filter outside or inside the <file> element.
Not that there's anything incorrect with that, it's just annoyingly verbose - and more prone to XML syntax error.

Another solution for large downloads that is already coded in SPManager is that the plugin author can specify a message to be displayed, and a confirmation dialog displayed before the plugin is downloaded. The HelpPlugin used to use this feature, because its download was quite large.
The feature is still supported, but the HelpPlugin no longer uses it. :)

Cheers!
Nik

[Peter Eastman]

I wouldn't worry too much about hardware. If you're just using a VM to do compilation, you'll start it up, compile, copy the files out, and shut it down again. If the compilation takes a little while and your computer is slowed down while it's happening, that's not a big deal. I've used VirtualBox for this purpose before. It works fine and it's free, which is about all you really need.

I've never tried running OS X in a VM (I was already working on a Mac), but it is possible. Here's the first likely looking page that came up when I did a search for information about it: http://lifehacker.com/5938332/how-to-run-mac-os-x-on-any-windows-pc-using-virtualbox

Peter

[Nik Trevallyn-Jones]

In fact, VirtualBox, VMWare Player, and QEMU are all free, so there is reasonable choice. :)

Cheers!
Nik

[Nik Trevallyn-Jones]

Luke suggested:

<fileset>
<--good for all windows platforms -->
  <file todir="$APP_DIRECTORY/lib">MyAwesomeNonNativefile.custom os.name="Windows"</file>
<-- for some reason is only good for 64 bit linux -->  
  <file todir="$APP_DIRECTORY/lib">MyAwesomeNonNativefile.custom os.name="linux" os.arch="x64"</file>
</fileset>

Except the attributes would be inside the element, not the content:

<fileset>
<--good for all windows platforms -->
  <file todir="$APP_DIRECTORY/lib" os.name="Windows">MyAwesomeFile.dat</file>
<-- for some reason is only good for 64 bit linux -->  
  <file todir="$APP_DIRECTORY/lib" os.name="linux" os.arch="x64">MyAwesomeFile.bin</file>
</fileset>

[Luke S]

Thanks for the correction on the syntax. Told you I'm new to this.

Just had a couple of further thoughts on how to get the info needed. The trick is, one would need to first find out what vendor the JRE is from, since the system properties are stored a bit differently. You can find the vendor with System.getProperty("java.vendor");

• Oracle and IBM (does anyone use anymore?) you can query System.getProperty("sun.arch.data.model");
• OpenJDK I have not found the info, but they are very tightly tied to the oracle releases. The above may work as well.
• GNUJava does not have that property, wonder where the equivalent hides?
• AppleJava <= java 1.6 does not matter. Recent OSX use oracle as the standard install.

This should handle >95% of cases, and if the vendor is not recognized, why not just ask the user?

Also, If for some strange reason we do not have a matching binary, the manager should alert this, and either prevent download, or allow a manual override.

Also, should we consider ARM support? AOI is probably not best used on a phone, even if there is an OS/JVM compatible, but that may change.

[Peter Eastman]

Also, should we consider ARM support? AOI is probably not best used on a phone, even if there is an OS/JVM compatible, but that may change.

People occassionally ask about creating an Android or iOS version to run on tablets. But really, that would mean a complete rewrite of the UI! Things are changing so fast, though, who knows what we might want in a few years?

Peter

[Nik Trevallyn-Jones]

Also, should we consider ARM support? AOI is probably not best used on a phone, even if there is an OS/JVM compatible, but that may change.

I agree that AoI on iOS or Android is probably far too much UI work; but AoI on linux on an ARM processor, such as a Raspberry Pie, CueBox, etc would be workable.

I can plug my CueBox into a monitor or TV using HDMI, and connect USB or bluetooth keyboard and mouse.
I have also been experimenting with a cluster of CueBox machines, so the renderfarm project would be handy there as well :)

[Nik Trevallyn-Jones]

Just had a couple of further thoughts on how to get the info needed.

Another alternative would be to ship 2 tiny JNI libraries which support a single method call. A 64bit version of the library that returns the string "x64", and a 32bit one that returns the string "x32".

We then call the JNI method, let the JVM decide which of our implementations it calls, and we then just use the string it returns.

[Luke S]

linux on an ARM processor, such as a Raspberry Pie, CueBox, etc

There is also a report going around that there will be a version of Win 10 compatible with Raspberry PI. (FREE!) No idea if it will support standard desktop apps, but worth thinking about.

Definitely don't see Android Support any time soon, most larger tablets, even, would have trouble rendering. (More due to battery and heat, rather than raw power)

cluster of CueBox machines

Could you point me out what these are? The only google results I got are for the AV industry.

call the JNI method, let the JVM decide which of our implementations it calls

I like it, but it'll be more:

for (binary bin : systemBinaries)
{
    bool correct = true;
    try 
    {
        JNIinvoke()
    }
    catch (exception e)
    {
        correct = false;
    }

    if (correct) return bin;
}

Flow control by exception, but when you are deliberately trying to break an abstraction that the inventors insist does not need to be worked around...

[Luke S]

Okay, been poking at this every now and then, and I've run into a snag.

First, thanks for the hints about VM use. I have a successful running virtualbox instance, with Arch linux. (I know, bit of a glutton for punishment, doing everything from the command line so far. I'm actually liking some aspects of vim.)

I've been compiling Assimp through gcc (And trying to cross-compile with mingw32, trouble there is a different story.)

My problem is that, at least on my system, the .so is turing out huge! 17.5 MB for 64 bit! The assimp team apparently has no idea why that may be, so far.

Would one of you be willing to run a quick test? If you have a linux build environment that includes cmake, could you check out and build Assimp? (Just the default options) Let me know how large the libassimp.so.3.2.0 file ends up being. I'd like to be able to tell if this is something unique to my system.

[Nik Trevallyn-Jones]

Hey Luke,

I'm happy to run a build for you!
I used to do all my development and day-to-day work on linux, but that laptop is almost dead, and I haven't replaced it yet.
I have been issued a "MacTop" (which is fine except for the woeful keybaord and WiFi :) and I have a Stella and CentOS 7 on VMs there, so it should be no problem.

I'll let you know the results!

cluster of CueBox machines

Could you point me out what these are? The only google results I got are for the AV industry.

https://www.solid-run.com/
Sorry - I miss-spelled the product name - I have a cluster of 3 CuBox-i machines:
https://www.solid-run.com/freescale-imx6-family/cubox-i/

[Nik Trevallyn-Jones]

Ok, turns out I didn't have a cmake environment set up, so a quick-n-dirty:

yum install cmake gcc-* zlib zlib-devel boost-* pth-devel
export CMAKE_CXX_COMPILER=gcc
cmakeCMakeLists.txt -G 'Unix Makefiles'
make

yields:
21M libassimp.so.3.2.0

[Nik Trevallyn-Jones]

So I've spent some time trawling through the Jinput library to see how they handle the 32 vs 64 bit library issue.

It turns out Jinput relies on the "os.arch" system property.

Looking at the jassimp code, it doesn't even check that, meaning you have to ensure that the correct version of the library is in your library path.

AOI, on the other hand, already has code to help automate the loading of native libraries (SearchlistClassLoader), so we could add some further logic in there if we decide to.

So, we can proceed in a number of ways:
- explicitly check the "os.arch" property in the plugin, and manage the library loading manually (clunky);
- add some code to AOI to check "os.arch", and adjust how it locates a library (possibly limiting);
- add some code to AOI to automatically detect the 64 vs 32 bit architecture (more generally successful)

Cheers!
Nik

[Luke S]

Thank you for checking that for me. Hmm. This requires more research, as the comunity-provided package of assimp for my system installs at 8 MB.

Edit: Akk! can't trust pacman's installed size estimate. The packaged version, which is a bit behind, comes out at 15.7 MB.

So, we are actually in the ballpark. This makes being able to distribute just one platform's library even more important.

[Luke S]

Just had a (Crazy?) thought. It is possible to use whereis (where on windows) to find location of installed commands and libraries. Would it make sense to try to find such installations, and then offer the choice between using them and downloading the portable version?

[Nik Trevallyn-Jones]

Hi Luke,

Actually, I really wouldn't worry about the lib size - certainly not initially.
80GB still isn't a long wait on many (most??) home internet connections...

Looking forward, I am happy to add filtering on additional files using various predicates, including os.name and vm architecture; so the user only downloads those native libs that apply to their platform.

That would require a user to perform an "update" of plugins after any copy of the Plugins folder to another machine - or manually download and install the native libs.

I should be able to write a check so that SPManager flags missing native libs as a missing dependency.

Cheers!
Nik

[Nik Trevallyn-Jones]

If you're still looking for the cause of the difference in sizes, then I would propose you investigate the following:

1. stripped lib;
   I ran "strip" on the assimp....so, and it went from20M to 16M.

2. dynamically linked dependencies (boost);
   I haven't been able to confirm whether assimp dynamically links to Boost, or includes it into the assimp lib (perhaps either is possible).
   IF boost is linked into the linux lib, but the Windows lib links boost dynamically, then that would account for a lot of difference in size - the boost lib(s) are multiple Megabytes in size.

Of course, dynically linking to boost would require that a user had installed boost independently prior to using your plugin, so it would be a fairly fragile solution, I would think.

Cheers!
Nik

[Luke S]

Hey, missed your post a couple of days ago.

• os.arch is just kind of useless, as there are still plenty of people using 32 bit java on 64 bit OS/hardware.
• Did not realize that SearchListClassLoader could do that.

You have a point about dependency sizes, I was just rather shocked at the difference... we'll see how that holds up once I'm actually getting a windows build to work. BTW, if you were building for Windows, would you use the mingw-w64 compilers, or would you use msvc on your windows virtual machine?

• Stripping my own build results in an 11 MB file. Various compiler versions must be part of the difference.
• Boost: by default, CMake for assimp builds a thin boost replacement into the structure of the library. You lose a little flexibility, (No concurrent imports on different threads) and in return do not have to link in the entire boost library. The assimp team does not recommend using the full boost library unless you need it for other parts of your application, or reallyreallyreally want the multi-thread access capacity.

[Luke S]

Hey, I know its been a while. I got bust with other projects. I've realized that we should probably keep SPManager simple, and download all palatforms of the native library by default.

This leaves the question of how to select the correct version at runtime. This appears to be a non-trivial problem. I was pokin around the current version of JogAmp's JOGL, and found that the newer versions of Gluegen have several classes of code devoted to just this problem, as well as other "Platform/VM spec" type stuff. Perhaps I should bend some of my effort to updating the version of JOGL that AOI uses?

@Nik OT for the discussion, but how well do those CuBox systems perform? I've been checking the specs, and comparing them to an ODROID C2. For the price, the C2 seems the better deal, the only places where it falls short is: No e-SATA, and a rather wussy GPU. (To use the Mali 450 GPU for computations, one would have to fall back to old-school legacy style GPGPU)