White Paper
From ogre4j
Contents |
Abstract
This document is a technical report about the ogre4j project hosted at Sourceforge.net. It covers the questions why this project exist, how it works, which alternatives are available in the open source universe and what are the future plans for it. This paper is also available as PDF.
If you have any questions about this document or want to give us some feedback about it and the project itself, please send an email to the authors
or visit the web site http://ogre4j.sourceforge.net and make use of the forums or trackers.
Introduction
ogre4j is a project that enables the use of the C++ library OGRE3D (Object-Oriented Graphics Rendering Engine) in Java applications. More precisely it is a thin layer of C and Java code on top of the public API of OGRE. This layer consists of the Java Native Interfaces to access the C/C++ code. JNI was introduced with the release of Java version 1.2 and gives developers the opportunity to reuse C/C++ code in their Java applications.
The first approach was made by Ivica Aracic. He developed the first version of ogre4j in the year 2002. At that time OGRE was in the fledgling stages too. Today OGRE is one of the most popular and feature rich open source 3D render engines. The current version consists of about 540 classes and 18.000 lines of code for the main library plus os dependent and sample code.
In 2005 the netAllied GmbH decided to use OGRE as 3D render engine in their product portfolio which is based on the Eclipse RCP (Rich Client Platform) framework. This framework is mostly developed in Java but the main UI (User Interface) tool kit is based on the C/C++ APIs (Advanced Programming Interface) of the operating system.
The Standard Widget Toolkit, SWT, uses Java Native Interfaces to access the native (Native to the operating system.) UI APIs to create a look and feel that isn't different to real C/C++ applications. SWT as good example and the performance it demonstrates assured the decision to enhance ogre4j.
OGRE3D
OGRE is a scene-oriented, flexible 3D rendering engine (as opposed to a game engine) written in C++ designed to make it easier and intuitive for developers to produce applications utilising hardware-accelerated 3D graphics. The class library abstracts the details of using the underlying system libraries like Direct3D and OpenGL and provides an interface based on world objects and other high level classes.
The engine is free software, licensed under the LGPL and has a very active community. It has been used in some commercial games like Ankh and Pacific Storm. It was Sourceforge's project of the month in March 2005.
1.0.0 "Azathoth" was released in February 2005, the current release in the 1.x.y series is 1.4.3 "Eihort" (July 2007).
General
As its name states, OGRE is "just" a rendering engine. As thus, its main purpose is to provide a general solution for graphics rendering. Though it also comes with some facilities (vector and matrix classes, memory handling, et cetera) they must be thought as helpers only. It is not an all-in-one solution in terms of game development or simulation as it doesn't provide audio or physics support, for instance.
Generally, this is thought of as the main drawback of OGRE, but it could also be seen as a feature of the engine. The choice of OGRE as a graphics engine allows developers the freedom to use whatever physics, input, audio and other libraries they want and allows the OGRE development team to focus on graphics rather than distribute their efforts amongst several systems. OGRE explicitly supports the OIS, SDL and CEGUI libraries, and includes the Cg toolkit.
Currently OGRE is published under a dual license (One being LGPL, the other one called OGRE Unrestricted License (OUL)), to make it possible to be chosen for console development as well, because most of the publishers reject using free/open-source software in that particular market.
Features
- object oriented design
- plug-in architecture
- scene graph based
- support of a wide variety of scene managers
- fully multi-platform
- support of GLSL, HLSL, Cg and asm.
- full support for hardware weighted multiple bone skinning
- support for effects such as HDR, blooming, saturation, brightness, blurring and noise
- particle system with extensible rendering and customizable effectors and emitters
A full overview of the features provided by OGRE can be found on the web site http://www.ogre3d.org.
Java Native Interface
Java Native Interface (JNI) is a programming framework that allows Java code running in the Java virtual machine (VM) to call and be called by native applications (Programs specific to a hardware and operating system platform) and libraries written in other languages, such as C, C++ and assembly.
The JNI is used to write native methods to handle situations when an application cannot be written entirely in the Java programming language such as when the standard Java class library does not support the platform-specific features or program library. It is also used to modify an existing application, written in another programming language, to be accessible to Java applications. Many of the standard library classes depend on the JNI to provide functionality to the developer and the user, e.g. I/O file reading and sound capabilities. Including performance- and platform-sensitive API implementations in the standard library allows all Java applications to access this functionality in a safe and platform-independent manner.
The JNI framework lets a native method utilize Java objects in the same way that Java code uses these objects. A native method can create Java objects and then inspect and use these objects to perform its tasks. A native method can also inspect and use objects created by Java application code.
JNI is sometimes referred to as the "escape valve" for Java developers because it allows them to add functionality to their Java Application that the Java API can't provide. It can be used to interface with code written in other languages, like C++. It is also used for time-critical calculations or operations like solving complicated mathematical equations, since native code can potentially be faster than JVM code.
The JNI is not trivial and requires a considerable effort to learn, and some people recommend that only advanced programmers should use the JNI. However, the capability for Java to communicate with C++ and assembly removes any limitations on what function Java programs can perform. Programmers considering using the JNI should be aware that
- as mentioned before, the JNI is not an easy API to learn;
- only applications and signed applets can invoke the JNI;
- an application that relies on JNI loses the platform portability Java offers (a workaround is to write a separate implementation of the JNI code for each platform and have Java detect the Operating System and load the correct one at runtime)
- there is no garbage collection for the JNI side (JNI code must do explicit deallocation);
- error checking is a MUST or it has the potential to crash the JNI side and the JVM.
Alternative 3D APIs
Java3D
Java 3D is a scene graph-based 3D application programming interface for the Java platform. It runs on top of either OpenGL or Direct3D. Since version 1.2, Java 3D is developed under the Java Community Process. JSR 926 specifies Java 3D 1.4; as of 2007, the current version is 1.5.1 (released in June 2007).
Compared to other solutions, Java 3D is not only a wrapper around these graphics APIs, but an interface that encapsulates the graphics programming using a real, object-oriented concept. Here a scene is constructed using a scene graph that is a representation of the objects that have to be shown. This scene graph is structured as a tree containing several elements that are necessary to display the objects. Additionally, Java 3D offers extensive specialized sound support.
Java 3D and its documentation are available for download separately. They are not part of the JDK 6. However future versions of the JDK are expected to include a API package for Java 3D.
JOGL
Java OpenGL is a wrapper library that allows OpenGL (Open Graphics Library) to be used in the Java programming language. It is currently being developed by the Game Technology Group at Sun Microsystems, and is the reference implementation for JSR-231 (Java Bindings for OpenGL).
JOGL allows access to most features available to C programming language programmers, with the notable exception of window-system related calls in GLUT (OpenGL Utility Toolkit), and some extensions.
LWJGL
The Lightweight Java Game Library is a solution aimed directly at professional and amateur Java programmers alike to enable commercial quality games to be written in Java. LWJGL provides developers access to high performance cross-platform libraries such as OpenGL and OpenAL (Open Audio Library) allowing for state of the art 3D games and 3D sound. Additionally LWJGL provides access to controllers such as Gamepads, Steering wheel and Joysticks. All in a simple and straight forward API. The LWJGL project also has bindings for FMOD and DevIL, formerly known as OpenIL. LWJGL 1.0 was released February 4th, 2007.
LWJGL is not meant to make writing games particularly easy; it is primarily an enabling technology which allows developers to get at resources that are simply otherwise unavailable or poorly implemented on the existing Java platform. It is anticipated that the LWJGL will, through evolution and extension, become the foundation for more complete game libraries and "game engines" as they have popularly become known, and hide some of the new evils have had to been exposed in the APIs. One such example is the jMonkey Engine.
LWJGL is available under a BSD license, which means it's open source and freely available.
Xith3D
Xith3D is a scene graph based 3D engine written in Java. It uses JOGL or LWJGL for rendering, but it is designed to be not tied up with any underlying API. Sound is designed independetly of any API as well and there is support for OpenAL and JavaSound.
Like Java3D it is not only a wrapper. The Contents of a 3D scene are described as a tree-like structure of Java-objects. Xith3D can be considered an alternative solution to Java3D as both use the same basic architecture.
Xith3D is released under the BSD license allowing commercial and non-commercial projects.
jME
jMonkeyEngine is also a scene graph based 3D engine written in Java. Due to an abstraction layer, any rendering system can be plugged in but currently (Version 0.11) there is only LWJGL supported. It supports OpenAL and FMOD.
Based on a 3D game engine design it is apparently more oriented to game development.
Like Xith3D, jME is under BSD license.
Jirr
Jirr is a JNI wrapper for the Irrlicht Engine.
The wrapper code is generated by SWIG (Simplified Wrapper and Interface Generator, see also 17).Irrlicht is an open source 3D engine written in C++. For rendering it uses Direct3D, OpenGL and its own software renderer. Compared to Ogre the Irrlich Engine is generally considered more beginner-friendly but less feature-rich. For example Irrlicht has:
- no multipass rendering
- no unified file format
- no advanced animation system
Furthermore, some features of the Irrlicht Engine are not accessible by Jirr.
Summary
JOGL and LWJGL are or contain simple JNI wrappers of OpenGL. OpenGL is a standard specification defining a cross-language cross-platform API for writing applications that produce 2D and 3D computer graphics. The interface consists of over 250 different function calls which can be used to draw complex three-dimensional scenes from simple primitives. OpenGL was designed to be graphic output-only: it provides only rendering functions to draw 3D lines, vertices and bodies. E.g. scene graph based handling of 3D data is not supported. Because of that OpenGL is in comparison with OGRE a low level render API which leaves a lot of work to the application developer.
Java3D, Xith3D and jME are high level render APIs too but their high level functionalities are implemented in Java and because of that they lag performance in comparison to OGRE. E.g. the implementation of the rendering pipeline. The Java3D rendering pipeline must process the scene graph for each frame and invokes JNI calls to draw every single vertex, line and geometry. Xith3D and jME have even more overhead as they pass calls first to JOGL or LWJGL which in turn invoke JNI calls to draw every single vertex, line and geometry.OGRE does this all without the JNI bottleneck and should be faster accordingly. Even if the render loop is implemented in Java using ogre4j there is just one JNI call necessary to trigger the rendering of a frame.
Jirr and ogre4j have the same approach but Jirr has some drawbacks: It uses SWIG for code generation which causes several problems as explained in 17. Jirr also supports fewer rendering and animation features than ogre4j due to the use of the Irrlicht Engine.
So what do we get from the OGRE - ogre4j bundle? We get a high level feature-rich 3D render API in Java with nearly same performance as the native variant.
Concept
The basic concept behind ogre4j is to provide one-to-one bindings for each public interface of the OGRE library. The JNI wrapper should not hide any features of the underlaying API and should not contain any black magic. Therefore the JNI bindings do nothing else than calling the corresponding native function.
Memory Allocation
If a developer creates a new ogre4j object he will create a native OGRE object as well and all data will be kept in it. The Java object is only the interface to access this data through JNI. Creating a native object means to call the constructor and allocate memory for it on the native heap (new). The problem of this memory allocation is that Javas garbage collector doesn't know anything about it and on the native side doesn't exist an equivalent system. Accounting to that the developer has to be sure to deallocate everything he allocated on his own! The result of this is a major rule that every developer must remember when using ogre4j:
"If you create a ogre4j object with 'new' you have to delete it!"
Objects created by OGRE are in the responsibility of the native library and should be never deleted by a Java program. To strictly adhere to the mentioned rule we decided to introduce some API changes to the bindings which should help to make it clear which objects should be deleted. C/C++ has three different ways to pass variables:
- by value: A copy of the variable on the stack.
- by reference: A copy of the reference of the variable on the stack.
- by pointer: A copy of the pointer to the variable on the stack.
By contrast Java passes variables by reference (Except for primitive variables.) only. For primitive variables like integers this problem is irrelevant because primitives can be passed by value in JNI calls without loss of performance. Non primitive objects should never be passed by value through JNI calls because the size of the data that must be copied exceeds the size of the corresponding reference which is sufficient to access the native data. Because of that ogre4j passes heavy weight objects by reference only. This behavior has one big disadvantage. If a OGRE function returns a non-primitive object by value this object is only valid in the scope of the JNI call because it is copied on the stack and will be removed if the scope is left. To be able to return the reference to such an object the wrapper has to copy the object to the heap and returns the reference of the copy which will be valid until it is deleted. On Java side the wrapper does not know when to delete the returned object because the wrapper doesn't include a smart pointer or garbage collector implementation. Because of that the developer which calls the corresponding function has to delete the returned object if not longer needed. This conflicts with the "new" rule and can lead to misunderstandings. To put straight that the developer assumes responsibility for the returned object the API is changed so that the developer has to pass a previously created object to receive the return value. Now the "new" rule applies and the developers knows that he has to delete object.
JNI Code Generation
One big point for every project that implements wrapper(s) for a library in different programming languages is the effort to maintain the wrapper code. The target library has their own release cycles and most major releases introduce API breaking changes. Most of the projects deal with this issue by using code generators which create the necessary bindings automatically. We evaluated the application of several code generators such as SWIG (Simplified Wrapper and Interface Generator) and NoddleGlue but none of the tested tools met our requirements. SWIG needs very much effort beforehand because every interface that should be wrapped needs a interface description file. Both tools miss full support of C/C++ templates which are used quite often in OGRE. For this and other reasons we decided in autumn 2005 to implement our own generator based on the same technologies as NoodleGlue. Since autumn 2006 the JNI code generator project is forked from ogre4j under the name XBiG (XSLT Bindings Generator) and got its own project space on Sourceforge.net.
NoodleGlue is the wrapper generation tool of "noodle heaven" and uses doxygen to extract the API information from the library's source code. This approach had the advantage that parsing and analyzing is done by a tool that is widely-used and tested with different input languages. So the first step to our generator was already available for free. Besides the usual outputs like a HTML documentation Doxygen provides a XML output of the analysed source code. This output is specialized for the Doxygen task to generate documentation, contains a lot of information that isn't necessary to generate wrapper code and is structured in a flat (E.g. name spaces are not nested as child XML elements.) hierarchy. For these reasons and to have the possibility to replace Doxygen with another tool, we decided to implement a meta layer that is represented in XML too.
To convert the Doxygen output to our meta layer we're using XSLT (Extensible Stylesheet Language Transformations) which is designed to describe conversions or transformations of XML code with XML code. One big advantage of XSLT is that it is an interpreted language and therefore OS (Operation System) independent. The generation of the meta layer and the layer itself should be independent from any OS or platform to make it possible to generate bindings for "every" language on every platform. To have a consistent tool chain the generation of the wrapper code is done with XSLT too. This reduces the usage of different tools and technologies to one major aspect: XML/XSLT. As mentioned before, Doxygen could be replaced with another tool that is capable of parsing source code and generating a XML representation of the parsed input.
Meta Description of APIs
The meta XML layer should be independent from any OS or platform and additional to this requirement the goal was to create a meta description of programming interfaces that is independent from the input language. To reach this goal the meta layer must be able to describe features of the most common programming languages in an abstract and common way.
The meta layer is a XML tree of name spaces, classes, functions andincludes the support for the description of templates. These basic features shouldfulfill the requirements of most modern programming languages.
Doxygen
Doxygen is a documentation generator for C++, C, Java, Objective-C, Python, IDL (Corba and Microsoft flavors) and to some extent PHP, C#, D and ActionScript. It runs on most Unix-like systems as well as on Windows and Mac OS X. Most of the Doxygen code was written by Dimitri van Heesch. Like Javadoc, Doxygen extracts documentation from source file comments. In addition to the Javadoc syntax it supports the documentation tags used in the Qt toolkit and can generate output in HTML as well as in CHM, RTF, PDF, LaTeX, PostScript or man pages.
XSLT
Extensible Stylesheet Language Transformations is an XML-based language used for the transformation of XML documents. XSLT is designed to transform XML documents into other XML or "human-readable" documents. The original document is not changed; rather, a new document is created based on the content of an existing one. The new document may be serialized (output) by the processor in standard XML syntax or in another format, such as HTML or plain text. XSLT is most often used to convert data between different XML schemes or to convert XML data into HTML or XHTML documents for web pages, creating a Dynamic web page, or into an intermediate XML format that can be converted to PDF documents.
As a language, XSLT is influenced by functional languages, and by text-based pattern matching languages like SNOBOL and awk. Its most direct predecessor was DSSSL, a language that performed the same function for SGML that XSLT performs for XML. XSLT can also be considered as a template processor.
Advantages
- ogre4j provides all features of OGRE.
- ogre4j is faster than Java3D.
- Development with Java is platform independent (Only if platform and OS independent libraries are used).
- Java has rich and clean APIs.
- Eclipse can be used as application platform.
- Eclipse is one of the most powerful IDEs.
Summary
The development of ogre4j can now be done in parallel to the development/release process of OGRE. The automatic code generation of the JNI bindings reduces the effort to maintain API breaking changes of OGRE and reduces the possibility of bugs in the wrapper. The development of the necessary XSLT style sheets resulted in a stand alone project called XBiG which is maintained on its project space at Sourceforge.net. The usage of XSLT has a lot of of advantages but for big APIs the generation of the bindings is slow because it is a interpreted language.
The current release "ogre4j 1.4.3 beta2" is available on Sourceforge.net andshould be downloaded by interested developers to test the runtime behavior ofthe bindings. The bindings are yet not finished but the first samples anddemos show that it is possible to create 3D applications with it.
Outlook
So, what are the future plans for ogre4j and XBiG? At the moment there are still some API parts of OGRE on the ignore list of the generation process. One of the next goals will be to reduce this ignore list to generated the OGRE bindings up to 99% all OGRE demos to Java to have the same suite of applications which will show the capabilities of OGRE in Java. Parallel to the development of more samples and demonstration we are working at a 3D UI toolkit based on ogre4j which will be called ogreface. ogreface is inspired by the UI toolkit JFace and will provide developers a higher level API to create windowed ogre4j applications. Additionally there will be Eclipse plug-ins which integrate ogre4j and ogreface in the popular application framework.
Bibliography
[1] Steve Streeting et. al., Object-Oriented Graphics Render Engine, Torus Knot Software, 2007: http://www.ogre3d.org
[2] Dimitri v. Heesch, Doxygen, 2007: http://www.stack.nl/~dimitri/doxygen/
[3] Sun Microsystems, Java Native Interface, Sun Microsystems, 2004: http://java.sun.com/j2se/1.5.0/docs/guide/jni
[4] Gregory Junker, Pro OGRE 3D Programming, Apress, 2006
[5] Michael Kay, XSLT 2nd Edition - Programmer's Reference, Wiley Publishing, 2003
[6] ogre4j, Java Native Interfaces for OGRE, 2007: http://ogre4j.sourceforge.net
[7] XBiG, XSLT Bindings Generator, 2007: http://xbig.sourceforge.net
[8] Wikipedia
