[Jython-dev] JyNI development - question about exception handling

[Stefan R. <Ste...@gm...>]

Hello Jython developers,
my JyNI development reached a state where I can call an "import numpy" without
getting undefined-symbol-errors (this version is not yet at github).
However it still needs much work to be reallyusable.
I hope someone can give me a hint about the following issue.

The numpy __init__.py script starts with the following lines (initial doc comments collapsed):


# We first need to detect if we're being called as part of the numpy setup
# procedure itself in a reliable manner.
try:
    __NUMPY_SETUP__
except NameError:
    __NUMPY_SETUP__ = False


if __NUMPY_SETUP__:
    import sys as _sys
    _sys.stderr.write('Running from numpy source directory.\n')
    del _sys
else:

...actual import code...


The code goes fine until it reaches the numpy.core __init__.py script, particularly
the line "import multiarray". However, up to there Jython would have done it also without JyNI anyway.
import multiarray is the first place, where a native file (multiarray.so) must be loaded.
Now JyNI comes in. The code calling the multiarray init method looks like the following:

p = _PyImport_GetDynLoadFunc(name, shortname, pathname, fp);
if (PyErr_Occurred())
    return NULL;
...
(*p)();

In CPython everything works fine, PyErr_Occurred() returns NULL.
In JyNI, the code fails because PyErr_Occurred() returns non-NULL.
I found out, this is not due to an actual import error, but that the
Name-error from the beginning, where numpy checks for setup-mode, persists:

try:
    __NUMPY_SETUP__
except NameError:
    __NUMPY_SETUP__ = False



The JyNI implementation of PyErr_Occurred() is as follows:

PyObject *
PyErr_Occurred(void)
{
    env(NULL);
    return JyNI_PyObject_FromJythonPyObject((*env)->CallStaticObjectMethod(env, JyNIClass, JyNIPyErr_Occurred));
}

JyNI_PyObject_FromJythonPyObject is responsible for converting jobject to PyObject*. To discuss it
here would get far out of scope. However it maps null to NULL and also handles the None singleton correctly.
env is the following macro responsible for retrieving the JNI environment pointer:
(The jvm pointer was previously cached in *java)

#define env(errRet) \
    JNIEnv *env;\
    if ((*java)->GetEnv(java, (void **)&env, JNI_VERSION_1_2))\
        return errRet


The backing Java method goes as follows:

public static PyObject PyErr_Occurred()
{
    ThreadState tstate = Py.getThreadState();
    return tstate.exception == null ? null : tstate.exception.type;
}


In comparision to that, take a look at the CPython implementation:

PyObject *
PyErr_Occurred(void)
{
    PyThreadState *tstate = PyThreadState_GET();
    return tstate->curexc_type;
}

My JyNI implementation should do exactly the same in my eyes, but behaves subtly different.
While in CPython the Name error from the beginning is cleared somehow
before imports are performed, it persists in Jython/JyNI case.
I read that exceptions are scoped on stackframes and threads. I think there
are no multiple threads involved in CPython case, so is it due to different
stackframes? I could find no call to an active exception-clearing method like
PyErr_Clear or PyErr_Fetch in any CPython code relevant for importing.

Of course I could adjust the JyNI loading-code to clear any previous exceptions
before loading external modules (i.e. I would not just clear them but do PyErr_Fetch
and PyErr_Restore later to restore the previous state after loading is done).
Probably this will be the solution in the end anyway, but I would really like to
understand the cause of this cumbersome behavior so I can fix it "the right way".


Thanks in advance for any help on this problem!

Cheers

Stefan