usePythonQt to create python extension modules

[Thomas]

Hi,
I'm not sure whether I missed something in the documentation but to me it is unclear whether PythonQt can be used to create python extension modules that can be used by python (standalone) or whether the dynamic libraries can only be used inside an application after PythonQt has initialized an embedded python interpreter.

[Thomas]

Unfortunately nobody answered (but maybe I'm just too impatient, after all it was weekend) so I eventually just tried it out myself. I created a minimal extension module whose init module function basically consists only of

PythonQt::init( PythonQt::PythonAlreadyInitialized );

The resulting extension module can be loaded by the python interpreter and seems to be working.

[Florian Link]

Yes, PythonQt can be used as a Python extension, but I guess up to now, nobody has used it that way, because it is mainly thought for being embedded into a C++ application.

[Thomas]

Unfortunately that is not enough fro my scenario. I am basically hoping to use PythonQt to create several individual dlls and corresponding externsion modules. The idea is modularity. Subsets of the dlls can the be used in different c++ exe's and also with the python interpreter (scripts) for prototypes, tooling, testing, or whatever. The key to the interoperability of the modules is that they use the same 'wrapping' mechanism. I'm hoping that splitting the init routine (at least) two parts, one that initializes the module and one that does the default / builtin initialization will get me there.

[Thomas]

I've created an pythonQt::init_module() function that just does:

_self = new PythonQt( flags, pythonQtModuleName );

In the init module function of the extension I then init PythonQt with it and register my own type:

PythonQt::init_module( PythonQt::PythonAlreadyInitialized, "Config" );
PythonQt::priv()->registerClass( &CustomWidget::staticMetaObject );

The module is loadable, but CustomWidget is not there. This is what I get for a dir.

['BoolResult', '__doc__', '__file__', '__name__', '__package__', 'private']

which is the same as without the registerClass. Any ideas?

[Thomas]

Ok, I found the CustomWidget, it appears to be under private

>>> dir( module.private )
['CustomWidget', 'QObject', 'QWidget', '__doc__', '__name__', '__package__']

So that's a step in the right direction. However, when I try to create an instance I get:

ValueError: No constructors available for CustomWidget

even thought I declared the following constructor

public:
   CustomWidget( QWidget *parent = nullptr );

Why is that? Do I need to add an extra wrapper like all the Qt wrapped stuff does?

class CustomWidget_Wrapper : public QObject
{ 
   Q_OBJECT
public slots:
   CustomWidget* new_CustomWidget();
   void delete_CustomWidget( CustomWidget* );
} 

I thought that due to the meta information, QObject based stuff could be used directly?

[Florian Link]

Signals, slots and properties are available via the QMetaObject, but constructors and destructor need a decorator class. An alternative is to provide a factory class which has slots to create the individual instances.

Regarding your extensions, I think you should do this differently!
Create a normal Python C extension for each of your Dlls and call the PythonQt init only when PythonQt was not yet initialized, which you can detect by PythonQt::self being null. Then use the PythonQt::priv registerClass method that takes a Python module as argument.
That will place the wrapped class into your c extension module instead of PythonQt.private. Since PythonQt is a Dll, you need to initialize it only once and all you c extensions need to use the same PythonQt dll.

QMetaObject supports constructors for a while, but I did not have time to add support for that into PythonQt. It could be added to PythonQtClassInfo, in addition to the decorators. Constructors require the Q_INVOKABLE define before each constructor and they can be queried by QMetaObject::constructor().

[Thomas]

I think you should do this differently!

This is basically what I am doing in init_module. I check the self just as in the original init function.

That will place the wrapped class into your c extension module instead of PythonQt.private

I don't quite I understand, esp the "takes a Python module as argument".

but constructors and destructor need a decorator class.

I already though as much. The generic wrapping mechanism probably uses naming convention to infer semantics.

Using the wrapper is ok. One usually wants to differentiate between the c++ api and the scripting api anyway, but decorating with Q_INVOKABLE and adding support for that sounds like a good idea.

[Florian Link]

In PythonQt::priv(), there is:

void registerCPPClass(const char typeName, const char parentTypeName = NULL, const char package = NULL, PythonQtQObjectCreatorFunctionCB wrapperCreator = NULL, PythonQtShellSetInstanceWrapperCB shell = NULL, PyObject module = NULL, int typeSlots = 0);

void registerClass(const QMetaObject metaobject, const char package = NULL, PythonQtQObjectCreatorFunctionCB wrapperCreator = NULL, PythonQtShellSetInstanceWrapperCB shell = NULL, PyObject* module = NULL, int typeSlots = 0);

If you pass you C-Extention Python module as "PyObject* module", your classes are added to your C-Extension module instead of the PythonQt.private package.

Supporting Q_INVOKABLE constructors should be straight forward, but I don't have resources to do that right now. Feel free to try adding it, it should not be that difficult, since constructors and static decorator slots should be quite similiar.

[Thomas]

OK, I think I'm beginning to understand, but that means changing some more stuff as "my module" doesn't really exist right now. The only module is the one registered by PythonQt when the first extension module causes self to be instantiated.

I'm uncertain about the best way to achieve what I want. The use case to support is quite simple. Create a CustomWidget and place it in a dynamic lib. Create an extension module that creates a python wrapped CustomWidget. Have the possibility of loading an extension module that wraps Qt and give it my wrapped CustomWidget.

Then everybody wins. On the C++ side the application can use the dynamic libs directly or make them scriptable with PythonQt and on the python side people script together other things using only the extension modules (and the dynamic libs).

If that is too much effort to achieve with PythonQt, then I might have to experiment with PyQt/SIP. I'm a big fan of exploiting meta-information for scripting, so right now PythonQt gets my preference. (At least to my knowledge SIP does not use the moc generated meta-information, but I may be wrong ...)

Do you think that changing PythonQt to make it more modular way is desirable? (PythonQtCore, PythonQtGui, ...)

[Florian Link]

All this is already possible using PythonQt, but you need to create an individual C-Python extension for each DLL, initialize PythonQt in it and add the wrapped classes to the C-Python extension.

In MeVisLab (for which I developed PythonQt), we use it exactly in this way, we have

QtCore, QtGui, QtOpenGL, ... C-Extensions, which are located inside of a PythonQt package.

I never found time to add these extensions to the PythonQt source forge repository, since it involves more complex build steps, renaming DLLs to *.pyd etc.

The init code for such a module is pretty simple, attached is a sketch for QtCore.

[Thomas]

All this is already possible using PythonQt

That's good news. I was starting to think that I would need to invest more time than I'd care to. I'll try following your pointers.

[Thomas]

It seems that I managed to get what I wanted working. Thanks a lot for the guidance. The python interpreter can now run the following script:

import PythonQt
app = PythonQt.QApplication()
import Custom
wnd = Custom.CustomWidget()
wnd.show()
app.exec_()

To get there I had to patch PythonQtWrapper_QApplication so that exec is callable and a constructor is available.

Still, my code looks a bit different than the QtCore.cpp you provided. Parts of your code did not work for me. The equivalent of

PyObject* builtins = PyImport_ImportModule("_PythonQt.QtCore")

resulted in a nullptr. Also I don't seem to need any of the symbol copying. Everything is already top level, and there is no _PythonQt submodule.

My initCustom is just

PythonQt::init( PythonQt::IgnoreSiteModule, "_PythonQt" );
PyObject* module = Py_InitModule( "Config", NoMethods );
PythonQt::priv()->registerClass( &CustomWidget::staticMetaObject, nullptr, PythonQtCreateObject<CustomWidgetWrapper>, nullptr, module );

The initPythonQt is equally minimalistic. I simply added a module parameter to PythonQt_QtAll::init and used

PythonQt::init( PythonQt::IgnoreSiteModule, "_PythonQt" );
PyObject* module = Py_InitModule( "PythonQt", NoMethods );
PythonQt_QtAll::init( module );