Python GIL Handling with PythonQtAll
Dynamic Python binding for Qt Applications
Brought to you by:
florianlink,
marcusbarann
Hi!
I have an issue with the PythonQtAll Wrappers in case of
Use PythonQtAll bindings that require and event loop
Use PythonQtAll bindings that use gui ( and therefore an eventloop).
Python is Build with Thread Support on
PythonQt has Threading enabled
As i have seen the generated code overloads the event handlers and executues python code without trying to lock the GIL. This leads to immediate crashes.
So the gil should be locked before python is access from c land.
Fortunately the code is generated by an generator, so it should be enough to patch the generator and provide new bindings. Due to a other problems with the bindings (WindowsOkButtonHint) the bindings maybe required to be updated the maintainers.
Here are my patches. The patch regarding the ShellImplGenerator should be integrated, the second is just an example for ONE binding file i patched manually to examine and fix this issue for a certain use case
And here the example binding path
@Florian: The ShellImplGenerator.patch was applied, but the generator NEVER called. You have to prove if it generates code like int the com_trolltech_qt_gui12.patch or if i made a logical mistake. I hope so that not, but only a human :-)
Sorry to disappoint you, but PythonQt currently does not support multithreading at all. Years ago someone (I think there is a post here on the forum) tried implementing it, but it never reached a state which I could merge in.
To support multithreading in PythonQt, there are two sides to it:
ensure that the GIL is ensured whenever we enter into Python from C++ code
and ensure that the GIL is released and retaken when we enter C++ code from Python
This includes things like the PythonQtObjectPtr, which needs to take the GIL before doing a PY_DECREF().
Another question is the PythonQt singleton api itself. If you want to talk to the PythonQt C++ api from C++ from multiple threads, PythonQt would need to support locking of global state as well.
I understand that people want this, but since we don't need it in my company's products, I have no resources to work on this.
Hi!
some of the threads are from myself and i made several patches to support it. I agree with you, that a total complete multithreaded and independent thread support with PythonQt is not possible at the moment and maybe will never and you will need many hours to do such stuff. Also this singleton stuff.
But we have to be aware that if PYTHON itself is compiled with THREAD support on, python will take use of the GIL,. This is there my patch comes into account.
Currently we do all this things you mention: ensure that we have GIL when we go from C++ to Python via PythonQt and we also ensure that we release GIL the other way around. But this was perfect use case for me, cause i wrapped this in my code before acessing python Qt to call scripts or code or whatever, and in all my Objects/Classes Slots that are called from python. This could also be done e.g. in PythonQtSlot.cpp more easily, but i can also doit for myself not forcing you to implement it.
So currently everything is perfect, i can run multiple scripts with python QT in my QThreads, not FULLY Parallel, but pseudo parallel (like in python threads). I do not need more.
Problem is now with those C++ PythonQtAll Stuff , which now will be run from Python Context.
This is a point i cannot wrap, an there we could simply support GIL support with this generator patch.
The only think i want to avoid is to maintain patches againts pythonqt for the rest of my life to support it :-)
Maybe we could do a check with PyEval_ThreadsInitialized() to ensure that Thread usage was enabled. This may be helpfull. But with Python 3.7 i read in the docs this will always be the case :-)
Do you want to call me by phone to shorten the discussion? you can call the phone number from www.procitec.de and ask for my name? or can i call you at mevislab?
Last edit: Joerg Kreuzberger 2018-08-23
I discussed this with a colleague and we think the best way will be to add some helper scope classes which ensure/release the GIL. Then we can have two implementations for the scope classes, an empty one which gets optimized away and one with threading enabled which does the GIl work. We would still use PythonQt without the GIL scopes in my company and you can use it with the GIL. And all the GIL stuff would be in a single file. I would place the scope objects at the central places were PythonQt is entered and vice versa, including in the wrapper generator.
What do you think, would that solve the problem for you?
So this would be the very luxury solution for me. Currently i have no issues with it using PythonQt from my C++ Application cause i can do the gil lock and release from my application and in the classes i write to be used from python. But this is currently not so important for me, but would really be helpful in future for PythonQt users to avoid deadlocks and segfaults.
My real issue that i can't fix is only this PythonQt_QtAll issue. Using the wrapper classes is good, i also use own implementations. You should provide if possible two implementations, A scope locked one and one with dedicated take/release, to be also used by PythonQt user to access python code directly. I attached my solutions. Regarding the docu of python in 3.7 threads default handling seems to be changed (enabled by default as i understood)
Tell me if you need any further support or help. Maybe you want to integrate a first solution in a branch and we support testing?
Hi!
I have seen you have done the GIL integration with r487 now. Thank you VERY MUCH for this!
Just a little question anyway: Do you plan to generate new bindings after r487? Then i would wait for it, if not i have to think about generating them for myself
I'm still working on it, so please wait a bit, I will post when I'm finished.
I could regenerate Qt 5.6 and Qt 5.11 wrappers. If you need older wrappers, you need to run the pythonqt_generator. (It is really not hard to run it, just build it and set your QTDIR to the Qt version that you want to wrap, then run the generator without further commandline arguments inside of the "generator" directory).
ok, i wait and 5.6 is sufficient cause i use it currently also for 5.9.
But i will integrate it into my build steps. to be more independent in future. Thanks!
Hi!
I have a question regarding r489 and PythonQtSlot.cpp(188)
Here you leave the python scope and enter c++ scope.
Therefore you call your PYTHONQT_ALLOW_THREADS_SCOPE macro.
I think all calls in you catch statements (e.g. line 194 PyErr_SetString) will fail cause your dtor of the object behind the makro is not called. Don't you think so?
As far as I know C++ the local variables in the try block are all destroyed before getting into the catch clause, which means that the scope object gets deleted in time and restores the thread state.
See mistake #2
http://www.acodersjourney.com/2016/08/top-15-c-exception-handling-mistakes-avoid/
Great! learned something new :-)
I now updated the wrappers for 5.6 and 5.11 with the new generator containing the GIL macros.
I also added override keyword parsing to the C++ parser, since Qt uses these a lot nowadays.
Regarding thread support, the state is as this:
For your project, the PythonQtObjectPtr is probably not a problem, except if your users store Python objects on QObjects and the QObjects get deleted without the GIL being taken.
Last edit: Florian Link 2018-08-29
Hi!
i integrated now several revisions and after some minor changes (regarding gil handling in my slots, now all removed) everythings works fine,
A slightfull migration is possible every time. So thanks for your great support and time you spent on it!
I did some more work on this. The thread support is now compiled in by default, but needs to be enabled using:
Is there any relation to PythonQt::init()
i assume directly after PythonQt::init()
You can call it before or after init (because it only stores static bools).
But you have to call it at least before releasing the GIL and before
starting threads.
On Thu, Sep 6, 2018 at 9:45 AM Joerg Kreuzberger kreuzberger@users.sourceforge.net wrote: