#375 Shared library sonames and symlinks

[Anonymous]

I have been working with Irrlicht SVN head and it seems a number of bad changes have been made since the release of 1.7.2 which break the shared libraries on Linux and other Unix-like platforms. I'm putting them all together into one bug since they are closely related. None of these bugs affect 1.7.2, only the SVN head.

Firstly I want to clarify some terminology, from http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html. Every library has three names: the "linker name", the "soname" and the "real name". For Irrlicht, these are "libIrrlicht.so", "libIrrlicht.so.1.7" and "libIrrlicht.so.1.7.2". The linker name is the name you specify on the command-line when linking (e.g., "-lIrrlicht" means the linker name *must be* "libIrrlicht.so"). The soname is the name that all binary-compatible releases have in common. Usually it only has a major version number (e.g., libz.so.1) but Irrlicht seems to be binary-incompatible between minor versions, so Irrlicht's sonames must be "libIrrlicht.so.$(MAJOR).$(MINOR)". Finally, the real name is the full version number, so Irrlicht's real names are "libIrrlicht.so.$(MAJOR).$(MINOR).$(REVISION)". I'm new to the Irrlicht project, so I may have this wrong, but that's the versioning scheme used by Irrlicht as I see it.

In Linux and other Unices, the scheme for setting up shared libraries (as described in the above link) is to have the library file named with the real name (e.g., libIrrlicht.so.1.7.2), and have a symlink from the soname to the real name (e.g., libIrrlicht.so.1.7 -> libIrrlicht.so.1.7.2) and another symlink from the linker name to the soname (e.g., libIrrlicht.so -> libIrrlicht.so.1.7). This way, the linker can look up the linker name, while the loader can look up the soname, and the real name can change without anybody noticing (since changes to the revision version should not break ABI compatibility). Now, onto the bugs.

Firstly, when you run "make install", the soname symlink has the wrong name. Rather than creating a symlink called "libIrrlicht.so.1.8", it creates a symlink called "1.8". This was introduced in r3578 (http://irrlicht.svn.sourceforge.net/viewvc/irrlicht?view=revision&revision=3578). The line of the Makefile:
cd $(INSTALL_DIR) && ln -s -f $(SHARED_LIB).$(VERSION) $(COMPATIBILITY_VERSION)
should read:
cd $(INSTALL_DIR) && ln -s -f $(SHARED_LIB).$(VERSION) $(SHARED_LIB).$(COMPATIBILITY_VERSION)

Secondly, "make install" no longer creates the linker name symlink (also as of r3578). This was deliberately removed, and it is explained in the changelog for that revision: "Binary compatibility is only confirmed between minor releases, so the only useful symlink is from libIrrlicht.so.1.8 to libIrrlicht.so.1.8.0; others should rightly fail." This seems to be a misunderstanding of what the symlinks are for. The symlinks do not represent binary compatibility, that is what the soname is for. As long as two releases do not share the same soname, they are allowed to be ABI-incompatible, and the loader will not get confused. Removing the symlink makes it impossible to link against Irrlicht. The gcc command "-lIrrlicht" will search for a symlink called "libIrrlicht.so", so that symlink needs to be there. So the "make install" needs another action:
cd $(INSTALL_DIR) && ln -s -f $(SHARED_LIB).$(COMPATIBILITY_VERSION) $(SHARED_LIB)

Thirdly, why is "ldconfig -n $(INSTALL_DIR)" commented out in the Makefile? That command is necessary to complete installation of the library, but I haven't fixed that as it might be commented out for a reason. (It was commented out in r2721 (http://irrlicht.svn.sourceforge.net/viewvc/irrlicht?view=revision&revision=2721)).

The last problem is that, the soname of the shared library generated by "make sharedlib" is too specific. This was deliberately changed in r3620 (http://irrlicht.svn.sourceforge.net/viewvc/irrlicht?view=revision&revision=3620), which is more recent than 1.7.2, so this change hasn't actually been released yet. The commit for that revision only says "Add revision version to soname in shared lib for Linux." I'm not sure why this is, and I think it was a mistake.

Previously, the soname was "libIrrlicht.so.$(MAJOR).$(MINOR)" (e.g., libIrrlicht.so.1.7). Right now, they are "libIrrlicht.so.$(MAJOR).$(MINOR).$(REVISION)" (e.g., libIrrlicht.so.1.7.2).

As I explained above, the soname is a name shared by binary-compatible releases. If the soname doesn't change between releases, then it is safe for programs linked against the old release to run against the new release. If the soname changes between releases, all programs must be recompiled for the new version. This means that from now on, every single revision of Irrlicht will require programs compiled against the previous revision to be recompiled. Again, I am assuming that Irrlicht won't break binary compatibility between revisions (e.g., from 1.7.2 to a hypothetical 1.7.3), in which case, recompilation shouldn't be required.

To demonstrate the problem:
1. Grab the latest SVN head, and fix the above two problems first, or you won't be able to install the library.
2. The Makefile should have VERSION_MAJOR = 1, VERSION_MINOR = 8, VERSION_RELEASE = 0-SVN. Change VERSION_RELEASE to 0.
3. Build and install Irrlicht 1.8.0, and run ldconfig. This creates symlinks libIrrlicht.so.1.8 -> libIrrlicht.so.1.8.0 and libIrrlicht.so -> libIrrlicht.so.1.8.
4. Create an Irrlicht program 'test.cpp'.
5. gcc -otest -lIrrlicht test.cpp
6. ldd test. Note the soname:
libIrrlicht.so.1.8.0 => /usr/local/lib/libIrrlicht.so.1.8.0
7. Test that the program works fine.
8. Delete /usr/local/lib/libIrrlicht.so.1.8.0.
9. Edit the Makefile, changing VERSION_RELEASE to 1.
10. Build and install Irrlicht 1.8.1, and run ldconfig. This creates symlinks libIrrlicht.so.1.8 -> libIrrlicht.so.1.8.1 and libIrrlicht.so -> libIrrlicht.so.1.8.
11. ldd test. Note the soname is still 1.8.0 and it can not find this version of the library:
libIrrlicht.so.1.8.0 => not found
12. Run the program:
./test: error while loading shared libraries: libIrrlicht.so.1.8.0: cannot open shared object file: No such file or directory

Don't forget to delete your phony "future releases" of Irrlicht and revert the Makefile or you may have problems!

Assuming that only binary-compatible changes are made between version 1.8.0 and 1.8.1, there is no reason why steps 11 and 12 should fail like that. What should happen is that on step 6, you see the following:
libIrrlicht.so.1.8 => /usr/local/lib/libIrrlicht.so.1.8
such that the linked executable is not looking for a specific revision, only a specific minor version. Then, when version 1.8.0 is replaced with version 1.8.1, the executable looks for the 1.8 symlink, which points to the updated version. This is important, as it allows you to roll out bug fixes and have all existing binaries automatically point to the fixed version. Note that if you were to create a version 1.9.0 (breaking binary compatibility), the executable would rightly still load version 1.8.1. If you deleted all versions of 1.8, the executable would rightly fail to load until you recompile it.

So I would revert the change made in r3620. This code is correct:
-soname,$(SHARED_LIB).$(VERSION_MAJOR).$(VERSION_MINOR)
or
-soname,$(SHARED_LIB).$(COMPATIBILITY_VERSION)

Lastly, I've noticed that the terms $(SHARED_LIB).$(COMPATIBILITY_VERSION) and $(SHARED_LIB).$(VERSION) show up many times in the Makefile. In all cases, the former refers to the soname and the latter refers to the real name. Therefore, I have created symbolic names $(SONAME) and $(SHARED_FULLNAME) for these two names. That should avoid more of these bugs from cropping up in the future.

I have attached a full patch which fixes all of these issues (except the commented-out call to ldconfig) and makes the Makefile simpler by using defined constants for these names. If you would like to commit the four fixes as separate commits, I have a Bazaar repository here: https://code.launchpad.net/~mgiuca/irrlicht/soname. The individual diffs can be downloaded here: http://bazaar.launchpad.net/~mgiuca/irrlicht/soname/changes/2220?start_revid=2220.

[Anonymous]

Patch to the Makefile, fixing the symlink filenames, library soname and cleaning up the Makefile

[Christian Stehno]

Fixed in latest SVN revision. It will be part of the next
stable Irrlicht release.