#649 Deadlocks using GIL and PythonCOM locking

[Stefan Schukat]

If running in a multithreaded environment deadlocks could occur due to the fact that two locks are used (GIL and PythonCOM lock).

E.g. three thread scenario:
First thread has the GIL and tries to get an interface from the GIT.

Callstack:
...
18f5e798 09ea9b8b 09eb62d4 09e4696c 09e88c08 ntdll!RtlEnterCriticalSection+0x150
18f5e7a0 09e4696c 09e88c08 18f5e7c4 09e64228 pywintypes25!PyWin_AcquireGlobalLock+0xb
18f5e7b4 09e3879e 13828118 13828118 18f5e7e8 pythoncom25!PyCom_DLLAddRef+0x1c
18f5e7cc 09e371fd 0c718804 0c718804 09e3139b pythoncom25!PyIUnknown::PyIUnknown+0x4e
18f5e7d8 09e3139b 0c718804 13828118 18f5ff78 pythoncom25!PyIDispatch::PyIDispatch+0xd
18f5e7f0 09e345d2 0c718804 12940418 00000000 pythoncom25!PyIDispatch::PyObConstruct+0x3b
18f5e804 09e53024 0c718804 18f5e830 00000000 pythoncom25!PyCom_PyObjectFromIUnknown+0xc2
18f5e83c 09c4e317 0a0ed3bc 16c09f58 18f5e8ac pythoncom25!PyIGlobalInterfaceTable::GetInterfaceFromGlobal+0xb4

To achieve this it needs the PythonCOM lock.

A second thread has the PythonCOM lock while calling CoUninitialize which wants to release a COM interface in the main thread (ole32!RemoteReleaseRifRef and ole32!SwitchSTA).

Callstack:
…
187af0e0 76f7a7c9 00a6b360 187af1d8 187af1ec ole32!SwitchSTA+0x21
..
187af7ac 76e7b172 0c711104 00a6b360 00000001 ole32!RemoteReleaseRifRef+0xb0
…
187afad4 09e46ce5 09e3edae 0a1f4828 09c4e317 ole32!CoUninitialize+0x72
187afad8 09e3edae 0a1f4828 09c4e317 00000000 pythoncom25!PyCom_CoUninitialize+0x35

The main thread wants to execute Python code and waits for the GIL and hence blocking the message loop.

Callstack.
…
0018f850 09c8e0c7 0a0e6d78 ffffffff 09bf8f28 python25!EnterNonRecursiveMutex+0x3d
0018f85c 09bf8f28 0a0e6d78 00000001 00000000 python25!PyThread_acquire_lock+0x17
0018f874 09c63f1c 0a0e24c0 ffffffff 0261f870 python25!PyEval_RestoreThread+0x38
0018f884 09b85588 0261f870 099b58fc 02642d10 python25!PyGILState_Ensure+0x5c

So first thread holds the GIL while waiting for the PythonCOM lock. Second thread holds the PythonCOM lock while synchronizing with the main thread. The main thread waits for the GIL, i.e. deadlock.

Two thread scenario.
Almost as above only exception the GIL is not involved. A second thread wants to call CoUninitialize while holding the PythonCOM lock triggers a synchronization with the main thread due to the STA model. The main thread blocks the message loop since it waits for the PythonCOM lock while calling PyCom_DLLAddRef.

A attach a patch which is based on following ideas.
- Calling PyCom_DllAddref does not need the GIL so before calling the method the GIL must be released.
- CoUnitialize synchronizes threads, hence the PythonCOM lock must be released before CoIUnitialize is called.

With these changes the deadlock did not occur anymore in our product.