pyopengl-users

[PyOpenGL-Users] Loading PyOpenGL from embedded Python interpreter

[Patrick H. <pa...@vr...>]

[Note: This is a reposting of a message I misdirected to pyopengl-devel 
last night.  I apologize for the reposting, but I wanted to be sure this 
reaches the right audience.]

My group is having some problems trying to load a Python module that
uses PyOpenGL (2.0.0.44) from C++ code that includes an embedded Python
interpreter.  This is happening on Red Hat Linux 8 and FreeBSD 5.

The first issue we ran into was unresolved Python/C symbols in various
.so files in the PyOpenGL installation.  I attempted to fix this by
changing the PyOpenGL build so that all the .so's are linked with
-lpython2.2 and -lutil.  This appears to address the unresovled symbol
problem, but now I have a new problem that I can't seem to figure out.

When I run the C++ application that loads the Python module, I get this
error:

Fatal Python error: Interpreter not initialized (version mismatch?)

I am certain that the interpreter is initialized because I added an
assertion that calls Py_IsInitialized() immediately before trying to
load the Python module.  However, within OpenGL/GL/__init___.so,
Py_IsInitialized() is invoked by Py_InitModule4(), and here,
Py_IsInitialized() returns false.  I am not sure why this is, but I have
a theory.

In Python 2.2.2 (and in most other versions, I presume), there is a
static variable that is used to keep track of whether the interpreter is
initialized (see Python-2.2.2/Python/pythonrun.c).  If there were
different verions of that static variable in memory, there would be
problems in determining whether the interpreter is actually initialized. 
  On FreeBSD and Red Hat Linux, there is no dynamically loadable version
of the Python runtime library--it only exists as libpython2.2.a.  With
the code I am using and the rebuilt PyOpenGL, there are at least
two--possibly three--places where libpython2.2.a is linked.  One is our
C++ code since it is using the Python/C API, and the other is in my
build of PyOpenGL's .so files.  The third place is in Boost.Python, the
library we're using to help our C++ code talk to Python.  All of those
PyOpenGL .so files are loaded at runtime, so is it possible that the
runtime loader isn't handling the symbol resolution correctly?  In other
words, is the runtime loader allowing the existence two or more versions
of various static variables within libpython2.2.a?  I'm not an expert in
how linkers or loaders work, so I could be way off here.  I have read a 
little bit saying that Python modules in C and C++ may not get along 
well unless they are both linked with a C++ compiler, and my next effort 
is going to be linking the PyOpenGL .so's with a C++ compiler.

There are many more details I am leaving out, but I'm trying to include
only the relevant pieces in hopes of zeroing in on a solution more
quickly.  To cause the crash, however, it is sufficient for the Python
module loaded at runtime by the C++ code to be only the following:

from OpenGL.GL import *

The (Boost.Python) code that is doing the module loading looks like this:

std::string module("someModule");
python::handle<>
cur_name(python::borrowed(PyString_FromString(module.c_str())));
python::handle<>
cur_module(python::borrowed(PyImport_Import(cur_name.get())));

I believe that the Python interpreter is initialized correctly prior to
the above calls because everything works fine until the contents of
OpenGL.GL are imported.

I have the stack trace (from FreeBSD 5) leading up to the above error
message being printed, but because I don't have a debugging build of the
Python runtime, it may not be very useful.  Here it is nonetheless:

#0  0x286f9353 in kill () from /usr/lib/libc.so.5
#1  0x287627cc in abort () from /usr/lib/libc.so.5
#2  0x289e8267 in Py_FatalError ()
    from /home/patrick/lib/python/OpenGL/GL/__init___.so
#3  0x289e4ea7 in Py_InitModule4 ()
    from /home/patrick/lib/python/OpenGL/GL/__init___.so
#4  0x2899961f in init__init___ ()
    from /home/patrick/lib/python/OpenGL/GL/__init___.so
#5  0x288393ee in _PyImport_LoadDynamicModule ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#6  0x2883762c in load_module ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#7  0x28838303 in import_submodule ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#8  0x28837ed4 in load_next ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#9  0x28837afd in import_module_ex ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#10 0x28837c4e in PyImport_ImportModuleEx ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#11 0x28855cb3 in builtin___import__ ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#12 0x28804e6c in PyCFunction_Call ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#13 0x287efb19 in PyObject_Call ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#14 0x28826f53 in PyEval_CallObjectWithKeywords ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#15 0x28824f82 in eval_frame ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#16 0x28826339 in PyEval_EvalCodeEx ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#17 0x28823417 in PyEval_EvalCode ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#18 0x28836760 in PyImport_ExecCodeModuleEx ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#19 0x28836d66 in load_source_module ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#20 0x28837602 in load_module ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#21 0x28836ef2 in load_package ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#22 0x2883763d in load_module ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#23 0x28838303 in import_submodule ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#24 0x28837ed4 in load_next ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#25 0x28837afd in import_module_ex ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#26 0x28837c4e in PyImport_ImportModuleEx ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#27 0x28855cb3 in builtin___import__ ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#28 0x28804e6c in PyCFunction_Call ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#29 0x287efb19 in PyObject_Call ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#30 0x28826f53 in PyEval_CallObjectWithKeywords ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#31 0x28824f82 in eval_frame ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#32 0x28826339 in PyEval_EvalCodeEx ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#33 0x28823417 in PyEval_EvalCode ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#34 0x28836760 in PyImport_ExecCodeModuleEx ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#35 0x28836d66 in load_source_module ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#36 0x28837602 in load_module ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#37 0x28836ef2 in load_package ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#38 0x2883763d in load_module ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#39 0x28838303 in import_submodule ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#40 0x28837ed4 in load_next ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#41 0x28837afd in import_module_ex ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#42 0x28837c4e in PyImport_ImportModuleEx ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#43 0x28855cb3 in builtin___import__ ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#44 0x28804e6c in PyCFunction_Call ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#45 0x287efb19 in PyObject_Call ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#46 0x28826f53 in PyEval_CallObjectWithKeywords ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#47 0x28824f82 in eval_frame ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#48 0x28826339 in PyEval_EvalCodeEx ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#49 0x28823417 in PyEval_EvalCode ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#50 0x28836760 in PyImport_ExecCodeModuleEx ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#51 0x28836d66 in load_source_module ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#52 0x28837602 in load_module ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#53 0x28838303 in import_submodule ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#54 0x28837ed4 in load_next ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#55 0x28837afd in import_module_ex ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#56 0x28837c4e in PyImport_ImportModuleEx ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#57 0x28855cb3 in builtin___import__ ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#58 0x28804e6c in PyCFunction_Call ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#59 0x287efb19 in PyObject_Call ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#60 0x28826f53 in PyEval_CallObjectWithKeywords ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#61 0x287efae0 in PyObject_CallObject ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#62 0x287efc2c in PyObject_CallFunction ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#63 0x28838722 in PyImport_Import ()
    from /home/patrick/src/Juggler/Toolbox/simulators/PySim/PySim_drv.so
#64 0x287dace5 in simulators::PySim::switchDriver(std::string const&) (
     this=0x817b200, module=@0x817b37c) at PySim.cpp:350
#65 0x287d9e66 in
simulators::PySim::config(boost::shared_ptr<jccl::ConfigChunk>)
(this=0x817b200, c={px = 0x8157740, pn = {pi_ = 0x8157760}}) at
PySim.cpp:139
#66 0x280c38f3 in
vrj::SimViewport::config(boost::shared_ptr<jccl::ConfigChunk>)
(this=0x8177a00, chunk={px = 0x81576e0, pn = {pi_ = 0x8157720}})
     at
/home/patrick/src/Juggler/main/juggler/modules/vrjuggler/vrj/Display/SimViewport.cpp:94
#67 0x280a8006 in
vrj::Display::configViewports(boost::shared_ptr<jccl::ConfigChunk>)
(this=0x8172540, chunk={px = 0x811d740, pn = {pi_ = 0x811d760}})
     at
/home/patrick/src/Juggler/main/juggler/modules/vrjuggler/vrj/Display/Display.cpp:130
#68 0x280a7298 in
vrj::Display::config(boost::shared_ptr<jccl::ConfigChunk>) (
     this=0x8172540, chunk={px = 0x811d740, pn = {pi_ = 0x811d760}})
     at
/home/patrick/src/Juggler/main/juggler/modules/vrjuggler/vrj/Display/Display.cpp:58
#69 0x280acfff in
vrj::DisplayManager::configAddDisplay(boost::shared_ptr<jccl::ConfigChunk>)
(this=0x81461c0, chunk={px = 0x811d740, pn = {pi_ = 0x811d760}})
     at
/home/patrick/src/Juggler/main/juggler/modules/vrjuggler/vrj/Display/DisplayManager.cpp:210
#70 0x280abca9 in
vrj::DisplayManager::configAdd(boost::shared_ptr<jccl::ConfigChunk>)
(this=0x81461c0, chunk={px = 0x811d740, pn = {pi_ = 0x811d760}})
     at
/home/patrick/src/Juggler/main/juggler/modules/vrjuggler/vrj/Display/DisplayManager.cpp:119
#71 0x2838f91f in jccl::ConfigChunkHandler::configProcessPending() (
     this=0x81461c0)
     at
/home/patrick/src/Juggler/main/juggler/modules/jackal/rtrc/jccl/RTRC/ConfigChunkHandler.cpp:95
#72 0x283961d5 in jccl::ConfigManager::attemptReconfiguration() (
     this=0x8071440)
     at
/home/patrick/src/Juggler/main/juggler/modules/jackal/rtrc/jccl/RTRC/ConfigManager.cpp:477
#73 0x280b6756 in vrj::Kernel::checkForReconfig() (this=0x8070180)
     at
/home/patrick/src/Juggler/main/juggler/modules/vrjuggler/vrj/Kernel/Kernel.cpp:235
#74 0x280b61af in vrj::Kernel::controlLoop(void*) (this=0x8070180,
     nullParam=0x0)
     at
/home/patrick/src/Juggler/main/juggler/modules/vrjuggler/vrj/Kernel/Kernel.cpp:184
#75 0x280bdf4d in vpr::ThreadMemberFunctor<vrj::Kernel>::operator()() (
     this=0x811de40) at Thread/ThreadFunctor.h:127
#76 0x2840bfd8 in vpr::ThreadPosix::startThread(void*) (this=0x8144940,
     null_param=0x0)
     at
/home/patrick/src/Juggler/main/juggler/modules/vapor/vpr/md/POSIX/Thread/ThreadPosix.cpp:299
#77 0x2840cafd in vpr::ThreadMemberFunctor<vpr::ThreadPosix>::operator()() (
     this=0x811de60) at Thread/ThreadFunctor.h:127
#78 0x28407cd5 in vprThreadFunctorFunction (args=0x811de60)
#79 0x286944ee in _thread_start () from /usr/lib/libc_r.so.5

Any help or insight would be greatly appreciated.

  -Patrick


-- 
Patrick L. Hartling                     | Research Assistant, VRAC
pa...@vr...                | 2624 Howe Hall: 1.515.294.4916
http://www.137.org/patrick/             | http://www.vrac.iastate.edu/

Re: [PyOpenGL-Users] Loading PyOpenGL from embedded Python interpreter

[Thomas W. <th...@xs...>]

On Fri, Apr 25, 2003 at 06:59:39AM -0500, Patrick Hartling wrote:

> My group is having some problems trying to load a Python module that
> uses PyOpenGL (2.0.0.44) from C++ code that includes an embedded Python
> interpreter.  This is happening on Red Hat Linux 8 and FreeBSD 5.

> The first issue we ran into was unresolved Python/C symbols in various
> .so files in the PyOpenGL installation.  I attempted to fix this by
> changing the PyOpenGL build so that all the .so's are linked with
> -lpython2.2 and -lutil.  This appears to address the unresovled symbol
> problem, but now I have a new problem that I can't seem to figure out.

I think this 'solution' actually caused your new problem. Instead, you need
to build your application in a dlopen()-friendly way: tell your linker to
export all symbols in your application to the dynamic symbol table. This is
not the default behaviour, but practically always necessary for applications
that want to dlopen().

I haven't used Boost myself, nor built C++ applications, nor embedded Python
using distutils or other more 'modern' built systems, but way back when
(cough) you were supposed to use the Makefile.pre that came with the Python
against which you were building. Makefile.pre is a template makefile that
contains the right variables to embed Python into your program -- which
libraries to link to, for instance, which compiler to use, et cetera. It
still gets built, and indeed, on my Linux system, it contains (among other
things):

CCSHARED=       -fPIC
LINKFORSHARED=  -Xlinker -export-dynamic
LIBS=           -lrt -ldl  -lpthread -lutil

I can't imagine e.g. distutils and whatever-Boost-does not providing this,
as it's quite crucial information for embedding, and it can vary per
platform (for instance, -lutil is not always necessary or available.)

> When I run the C++ application that loads the Python module, I get this
> error:

> Fatal Python error: Interpreter not initialized (version mismatch?)

Because you only have libpython2.2.a, using -lpython2.2 means that Python
2.2.2 is linked statically into the PyOpenGL .so's (and each seperately, at
that.) You should not link those .so's with -lpython2.2, and preferably not
with -lutil either (but that one probably is shared, so might turn out
okay.) If you really don't want to do the above, you can built
libpython2.2.so using one of the patches out there (many Python
distributions do that) or you can wait for Python 2.3, which can build
libpython2.3.so for more platforms (using --enable-shared).

-- 
Thomas Wouters <th...@xs...>

Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

Re: [PyOpenGL-Users] Loading PyOpenGL from embedded Python interpreter

[Patrick H. <pa...@vr...>]

Thomas,
	Your suggestions all seem right on target, but for some reason, they 
don't work for me.  I've verified that all the options I know about are 
being used, including the ones you mention.  I'm testing with my original 
build of PyOpenGL that doesn't have any of the modifications I described 
in the original message.  I guess I'll keep scanning the ld(1) manpage in 
case there's something I am missing.
	Regarding Python 2.3, unfortunately, I have a deadline in two weeks, so 
I can't wait for it--though I may try an alpha or a beta version if I get 
totally frustrated.

  -Patrick

Thomas Wouters wrote:
> On Fri, Apr 25, 2003 at 06:59:39AM -0500, Patrick Hartling wrote:
> 
> 
>>My group is having some problems trying to load a Python module that
>>uses PyOpenGL (2.0.0.44) from C++ code that includes an embedded Python
>>interpreter.  This is happening on Red Hat Linux 8 and FreeBSD 5.
> 
> 
>>The first issue we ran into was unresolved Python/C symbols in various
>>.so files in the PyOpenGL installation.  I attempted to fix this by
>>changing the PyOpenGL build so that all the .so's are linked with
>>-lpython2.2 and -lutil.  This appears to address the unresovled symbol
>>problem, but now I have a new problem that I can't seem to figure out.
> 
> 
> I think this 'solution' actually caused your new problem. Instead, you need
> to build your application in a dlopen()-friendly way: tell your linker to
> export all symbols in your application to the dynamic symbol table. This is
> not the default behaviour, but practically always necessary for applications
> that want to dlopen().
> 
> I haven't used Boost myself, nor built C++ applications, nor embedded Python
> using distutils or other more 'modern' built systems, but way back when
> (cough) you were supposed to use the Makefile.pre that came with the Python
> against which you were building. Makefile.pre is a template makefile that
> contains the right variables to embed Python into your program -- which
> libraries to link to, for instance, which compiler to use, et cetera. It
> still gets built, and indeed, on my Linux system, it contains (among other
> things):
> 
> CCSHARED=       -fPIC
> LINKFORSHARED=  -Xlinker -export-dynamic
> LIBS=           -lrt -ldl  -lpthread -lutil
> 
> I can't imagine e.g. distutils and whatever-Boost-does not providing this,
> as it's quite crucial information for embedding, and it can vary per
> platform (for instance, -lutil is not always necessary or available.)
> 
> 
>>When I run the C++ application that loads the Python module, I get this
>>error:
> 
> 
>>Fatal Python error: Interpreter not initialized (version mismatch?)
> 
> 
> Because you only have libpython2.2.a, using -lpython2.2 means that Python
> 2.2.2 is linked statically into the PyOpenGL .so's (and each seperately, at
> that.) You should not link those .so's with -lpython2.2, and preferably not
> with -lutil either (but that one probably is shared, so might turn out
> okay.) If you really don't want to do the above, you can built
> libpython2.2.so using one of the patches out there (many Python
> distributions do that) or you can wait for Python 2.3, which can build
> libpython2.3.so for more platforms (using --enable-shared).
> 



-- 
Patrick L. Hartling                     | Research Assistant, VRAC
pa...@vr...                | 2624 Howe Hall: 1.515.294.4916
http://www.137.org/patrick/             | http://www.vrac.iastate.edu/