QtConcurrent threading + PythonQt

[Joe Connellan]

Hello,

I am getting a crash when running PythonQtObjectPtr::evalScript() from two different threads each using their own PythonQtObjectPtr created inside each thread with PythonQt::self()->createUniqueModule().

I have read through:
https://sourceforge.net/p/pythonqt/discussion/631392/thread/28c4d2bd/

and older threads but I wasn't sure exactly what process I need to go through to get it to work, eg if any explicit GIL handling is required or if it's all being handled by PythonQt now. I'm using the current HEAD of the svn PythonQt trunk as of this writing.

I'm reasonably sure the Python libs I am building against have threading on as this works:

[Hitch:~] joe% /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python
Python 2.7.10 (default, Aug 17 2018, 19:45:58)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import threading
>>> import thread
>>>

Here is what I'm doing in my C++:

void runPython(QFutureWatcher<void>& watcher)
{
    QFuture<void> future = QtConcurrent::run(std::function<void()>([]()
        {
            PythonQtObjectPtr context = PythonQt::self()->createUniqueModule();
            context.evalScript(QString("import time  \n"
                                        "time.sleep(3)\n"));
        }));

    watcher.setFuture(future);
}

int main(int argc, char *argv[])
{
    QApplication application(argc, argv);

    PythonQt::init(PythonQt::IgnoreSiteModule);
    PythonQt::setEnableThreadSupport(true);

    QFutureWatcher<void> watcherA;
    runPython(watcherA);

    QFutureWatcher<void> watcherB;
    runPython(watcherB);

    return 0;
}

What do I need to do to get the above to work?

I've attached the stack traces of both threads, as well as the full project.

Thanks.

[Florian Link]

Since you are entering Python from C++, you need to make sure that you hold the GiL,
which you can do with placing the PYTHONQT_GIL_SCOPE into the scope where you communicate with Python or PythonQt (and even when you talk to a PythonQtObjectPtr, because it will do non-thread-safe ref counting of the PyObject).

So you have to place PYTHONQT_GIL_SCOPE into your run method.

An important thing to understand is that even it is multi threading, only one thread can talk to Python at a time, which is why you need to hold the GiL to call into Python.

BUT, this leaves us with a problem: When you startup PythonQt in the main thread, you implicitly hold the GiL on the main thread (because Python locks it when being initialized)

So what you are also missing is to release the GIL on the main thread, before you fire your threads, using:

PyEval_SaveThread()

This will release the GIL on the main thread and will feel a bit strange to call, because you never call the PyEval_RestoreTread(), but it is the way to release the GiL after initialization.

When you want to talk to PythonQt from the main thread later on, you have to use
PYTHONQT_GIL_SCOPE again.

Hope this helps, threading with Python from the C++ side is a complex topic...

[Joe Connellan]

Ah great, thank you very much, I can confirm that these changes do work for me.

Thanks again.