[Pyode-user] Initial (re)design questions

[Matthias B. <ba...@ir...>]

Hi,

it's good to see that there's still interest to get to a decent ODE 
binding that's up to date and will be maintained.
I suppose there are a couple of questions and decisions that have to be 
made:

1) Which tool to use for creating the wrappers?

I think the most common tools are Boost.Python, SWIG, Pyrex and ctypes. 
It seems you've already agreed on Pyrex. Currently, I'm using 
Boost.Python for another project, so I also have a little experience 
here as well (by the way, that other project might eventually turn into 
the second version of cgkit, my other Open Source project at 
cgkit.sf.net. I even have plans to integrate ODE in that project at some 
time, however not as 1:1 binding but as an internal dynamics system, so 
it's not really a replacement for PyODE).
I have only very little experience with SWIG. But when I tested it 
(which was already quite a while ago), it didn't appear to be that well 
suited to wrap C++ code. I'm aware that there have been new versions 
since then so the situation might be better now.
But at least I can comment a bit on Pyrex vs Boost.Python. When using 
Pyrex you have to write your wrapper code in a "special language" 
(*.pyx) which is compiled by the Pyrex compiler into C source code. Then 
you compile this C source code with your C compiler and you get your 
Python module. This "special language" closely follows the Python syntax 
and adds some more constructs that allow to declare C variables or call 
C functions. So you can say the Pyrex language is just Python spiced up 
with C. That's why it's so easy to learn, you already know most of the 
syntax. Programming in Pyrex feels very "Pythonic" and you can freely 
mix Python code and C calls (sometimes without noticing), Pyrex will 
take care of reference counts and type conversions between C and Python 
types. The major drawback with the current version of Python is in my 
opinion the lack of a tool that processes header files. Because you can 
call C functions, Pyrex has to know an "approximate" signature of the 
function so that it can do proper type conversions. And that's the 
tedious part because you still have to do that manually. Of course, if 
we build on my binding we already have a great deal of the declarations. 
But if there are going to be interface changes in ODE we have to go 
through the declarations and adapt them to reflect the changes. This is 
not automated! Another disadvantage of Pyrex is that it has no special 
support for C++ yet. If you want to wrap classes, you first have to 
write a C wrapper file.
Now to Boost.Python: it actually took me to while to get started with 
it. Before you can actually do the very first tests you have to download 
a ~13MB package (compared to ~180KB for Pyrex) and compile the 
Boost.Python runtime (Pyrex is implemented in pure Python and uses the 
distutils). But once you've everything set up you can really do spiffy 
things, especially when you want to wrap C++ code. The first thing, that 
I found remarkable is that it doesn't need a special compiler that 
translates interface files or whatever. Everything you need is your C++ 
compiler and you just define the interfaces in "pure" C++ and directly 
compile that code into a Python module. From my experience, the learning 
curve is much steeper than for Pyrex. It took me a while to get those 
call policies right and understand some of the compile errors. But the 
more I use Boost.Python, the more I'm in awe of what you can do with it, 
or rather, what it does for you! Once you've defined your wrapper 
classes Boost.Python takes care of converting the types between C++ and 
Python and of course, you don't have to deal with reference counts and 
that low level stuff from the pure Python/C API. The manual declaration 
of the ODE headers is not necessary here, because you simply include the 
original headers right away.
Personally, I like both tools and I think they both have their 
advantages and disadvantages and it depends on the project which tool is 
better suited.
As ODE has a rather small API, and the "official" API is C anyway, I'd 
probably tend to stick to Pyrex. As this is an Open Source project we 
should encourage other people to have a look at the code so that they 
might contribute something. And I think by using Pyrex we make it much 
easier to enable them to compile the module themselves. Pyrex is a small 
tool and it's easy to install and try out.
But I also want to mention one open issue I have with Pyrex in my 
binding. For the collision callback mechanism it's necessary to pass C 
pointers (geoms) back to Python which means we have to be able to map a 
C pointer to its corresponding Python object. This is *not* done 
automatically in Pyrex as it is done in Boost.Python. That's why I have 
a lookup table called "_geom_c2py_lut" in the module. This is just a 
dictionary that maps C pointers to Python objects. The problem now is, 
that Pyrex didn't support weak references at the time I was writing the 
module. As a result, I wasn't able to remove entries from this lookup 
table when the corresponding object was destroyed. I've reported this to 
the author of Pyrex but I'm not sure if support for weak references has 
really been added in the meantime. However, I didn't find that problem 
really restricting for my purposes because when I did a new simulation I 
was usually restarting the program anyway, and even if I wouldn't I 
could still delete the entries in the dictionary manually as you have 
access to it from Python. But I wanted to bring it up anyway, because 
this is really a flaw in my binding.

Well, sorry, this was a lot of text for just one point, I try to be more 
concise with the other ones:

2) Should we wrap the C functions or the C++ classes?

Here's my opinion: I'd go for the C functions because they belong to the 
official and *documented* API.

3) How do we map the ODE API to Python?

Well, you see my preference if you look at my binding... ;)
I've put everything in classes and the functions became methods. Every 
joint type got its own class (instead of just *one* joint class). The 
same holds for geoms. In general, I'm always in favour of a "Pythonic" 
interface, even if that would mean that we would have to change the 
original interface slightly.
Oh, what's also nice is the ability to pass any Python 3-sequence where 
ODE actually expects a vector. This way, you can still choose to use the 
vector type of your choice (or just plain tuples).

4) How do we implement the collision callback mechanism?

In my version, I tried to do it just as it is done in C with the near 
callback that can now be implemented in Python.
However, there's the above mentioned problem with the lookup table that 
so far can only grow, but never gets smaller again.
What might also be desireable is the ability to use a "default" callback 
that's implemented in C so that no Python code has to be called during a 
simulation. If you have a lot of objects this Python callback can 
actually slow down the simulation too much.

5) Trimesh support?

I suppose, everyone wants to have that, right? ;-)
But how do we pass the data?

6) double vs float. What type do we use?

I've always used doubles and really would like to continue doing so as I 
belief it can postpone numerical problems.
However, I don't know if trimesh support is possible with doubles. This 
was a problem in the beginning, wasn't it? This is actually what has 
always prevented me from trying it out.

7) What build system do we use?

I would really recommend using the distutils as this is the standard way 
for distributing Python extension modules. It has the advantage that 
building the module works the same on every platform.

8) Shall we try to implement an XODE import/export in Python?

This might raise the popularity of the Python version...

9) Which operating systems can we support directly?

Or in other words: What systems are you working on? I'm mainly on 
Windows XP and can occasionally do tests on Linux (SuSE 8.2).

10) ...what else?...


- Matthias -

PS: I'm yet from another corner of the world, I'm from Germany...