Menu
▾
▴
#15 Python interface
closed
9
2014-03-20
2012-08-15
No
A Python interface should be provided by exposing the necessary classes throught Boost.Python
Building the interface should be a separate build target.
The exposing code should be put in the same files which define the c++ classes, but it should be guarded by #ifdef-s, so that they come into play only when the python-interface target is built.
As discussed earlier, the problem is of course the templates. Not very deeply templated classes do not present much problem, as they can be explicitly instantiated for several cases (cf. the RANK of objects).
This is not possible for deeply templated classes, like Composite.
I now believe that C++ compilation should be made part of the Python interface. When a new instantiation of e.g. Composite is needed, some Python routine should write the corresponding C++ code in a file whose name identifies the given instantiation. Then, it has to compile it, and somehow present it to Python.
This way, the first run of the corresponding Python script will take a long init time (usual for Python anyway), but in later runs the routine in question sees that the necessary instantiation is already there and is compiled.
Setting up (and maintaining) such a compilation infrastructure is a highly non-trivial task, even if we consider just portability across different _Linux_ distributions. This is why I would suggest to aim for scipy.weave, which is certainly better tested and matured than anything we could come up with.
Scipy.weave enables us to compile python extensions on the fly and immediatly import them in python. The problem: at the moment it is only possible to wrap functions (and modules of several functions) in this way, not classes. There has been some efforts by Eric Jones (the author of weave) back in 2002 to also support classes, but nothing about this is in the documentation.
The easiest would probably be to expose a wrapper function to evolve via scipy.weave.
In any case compile time consistency checks are not available anymore, consistency checks would have to be shifted to python, right?
Just as a bookmark, a post by Eric Jones with an interesting answer, maybe I have to get in touch with these guys:
http://mail.python.org/pipermail/cplusplus-sig/2002-January/000429.html
I do not see the problem: the compilation infrastructure has to be in place anyway, since the user first has to compile the python-interface version of the framework. Then, relying on this very infrastructure (which can either be CMake or Boost.Build, where we already add these potential python-created targets), the scripts can do their own compilation.
This would probably work.
I head a Python interface in mind which doesn't necessarily depend on the build infrastructure on runtime, something Python users could ideally "just use" without the burdon to compile anything: e.g. a Ubuntu precompiled Python extension bundled together with libC++QED, libutils, libel and some headers which uses weave for instantiations internally.
But that is of course a design choice, and yours has some obvious advantages.
The reason a binary C++QED package isn't too useful at the moment is that scripts have to be compiled anyway. But with the interface scripts would be written in Python.
As a first test (also to test template instantiation) I wrapped parameters::Parameter<int> into a python extension and it worked quite well. I found using cmake to be the easiest for the moment.
Probably it is not possible (or at least not practical) to include the wrapper code into the files which define the classes. The build system builds a standalone <extension>.so shared library from the wrapper code and links it with libel.so, libC++QED.so etc. When linking <extension>.so we don't want to have all the symbols of the framework in the object files for the extension.
If you want to try it out, check out raimar/private/python and compile it using cmake. Finally make install which installs everything to /usr/local, but you can use -DCMAKE_INSTALL_PREFIX= to overwrite (and then add <prefix>/lib/python2.7/site-packages to $PYTHONPATH and <prefix>/lib to $LD_LIBRARY_PATH). In python try
import CPyPyQED.testmodule as test
a = test.ParameterInt("foo","bar",3)
a.do_print(5,5,5)
I could not use "print" as the function name because print is a statement in python2 and something like a.print always gives a SyntaxError.
Made some progress:
test.py:
import sys
from CPyPyQED.utils import *
from CPyPyQED.elements import *
p = ParameterTable()
q = QbitParsPumpedLossy(p)
update(p,sys.argv,"--")
p.printList()
can be called with "python test.py --deltaA -4 --gamma 2 --qbitInit "(1,0)" --etat 2" and prints correctly:
help display this list
version display version information
*** Qbit
qbitInit St7complexIdE Qbit initial condition excited (1,0)
deltaA d Qbit detuning -4
*** PumpedQbit
etat St7complexIdE Qbit pump (2,0)
*** LossyQbit
gamma d Qbit decay rate 2
We cannot wrap parameters::ParameterTable::add directly: primitive data types are immutable in Python, so we cannot return a non-const reference to the parameter from add. On the other hand we probably don't need "add": we could use Python's optparse to pass parameters to the script, remove everything script-specific from argv and let update handle the rest. What do you think?
I have a first functional prototype python script in raimar/private/python, it is the QbitMode_C++QED script. Could you maybe test if it builds and runs on your system?
The build directory can be used directly as a python package: after the cmake build finished, stay in the build directory and call
PYTHONPATH=. ../cpypyqed/scripts/QbitMode.py --dc 0 --Dt .01 --T 2 --etat '(1,3)' --gamma 1.2 --deltaA -1 --qbitInit '(.4,.3)' --minit '(1,-.5)' --eta '(30,10)' --deltaC 100 --cutoff 30 --g 0 --evol single
The script QbitMode.py is very closely related to the original:
from cpypyqed import *
import sys
p = ParameterTable()
pm=ParsPumpedLossyMode(p)
pq=ParsPumpedLossyQbit(p)
pe=ParsEvolution(p)
pjc=ParsJaynesCummings(p)
update(p,sys.argv,'--')
m=modeMake(pm,QM_Picture.IP)
q=qbitMake(pq,QM_Picture.IP)
jc=JaynesCummings(q,m,pjc)
psi = (qbitInit(pq)*modeInit(pm))
psi.renorm()
evolve(psi, binaryMake(jc), pe)
It has identical output and runtime compared with the C++ version.
At the moment only BinarySystem is implemented. Now starts the fun part with composite::make.
Now that I understand a little bit better how Boost.python works we should discuss soon about some general design decisions. Boost.python is great, but the extension needs quite a lot of wrapping code. We need to get the structure right so that it will be efficiently maintainable.
It compiles and runs straightforwardly. (Is there a single target alias that builds the whole python extension?)
It looks very promising, but I still need to understand the structure.
It is always possible to call make in a subdirectory of the build dir to build all targets specific to this sub-directory. But now I also added a cpypyqed meta-target which builds the extension.
The structure is not obvious, which is usually not a good sign. Any structure approach I tried had some advantages and some drawbacks, and I have not found an optimum yet.
In a nutshell: cpypyqed is a python package (by the existence of __init__.py) with three sub-packages (utils, cppqed, elements) corresponding to the three C++-libraries. In each of these sub-packages there is a python module in shared library form (_utils.so, _cppqed.so and _elements.so), but later there might also be other modules as .py files, e.g. wrappers written in python to supplement the interface.
Every class that we want to expose has a .cc file in the relevant sub-package, with the same basename as the .h file in which the class is declared. Here the actual code which wraps the class is inside an pythonext::export_Something function. The various export_ functions are called by a build_build_<modulename>.cc file to create the actual module shared library. This is a recipe Boost.python recommends to distribute the module over several files, as module creation must be in a single file, but compiling everything in one unit would be too expensive in terms of memory. In my case, the build_<modulename>.cc is generated by cmake automatically and copied to the build directory. This way we don't have to keep track of the export_ functions in several places.
I also thought about including the wrapper code in the actual C++Qed library source files, but then we are again introducing template definitions everywhere which we have just tried to eliminate as best as possible.
Let's discuss this on skype.
I have pushed a first prototype for a composite system 2particles1mode.py. The template classes are still pre-instantiated for this particular case, but at least I know how to do it and can now work on an automatic instantiation.
Wrapping composite::make gave me some headache. Returning the composite object by value as it is done in the framework resulted in dangling pointers to the frees (don't ask why). I introduced a new maker function composite::makeNew which returns a shared_ptr to a newly created composite object. This is then automatically handled (and eventually deleted) by Python.
Prototype on-demand template instantiation can be tested from the new cpypyqed binary package in my ppa.
For example, after installing the python interface, with the attached script 2particles1mode.py:
2particles1mode.py --fin 3 --T 0.1 <---- (takes quite long to compile)
2particles1mode.py --fin 3 --T 0.1 <---- (second time simulation starts immediately)
Compilation is done in ~/.cpypyqed
I've played a bit with the python interface, and I am very impressed.
The interface of composite is extremely nice:
{(0,2):poc,(0,1):poc}
(This {…} is the dictionary data structure, right, so that the order of the elements do not matter? And what kind of data structure is this (0,2) for example? )
I see that in-place creation of frees also works (without a separate named variable). It's also very nice that in case there are named variables, their names can be deleted after everything gets stored in Composite.
I think it's a very good idea to collect all the resulting files into a per-user central directory .cpypyqed.
Some problems:
------------------------
* the libboost-python-dev is not pulled in as dependency of cpypyqed, even though it's necessary for on-demand compilation
* exception names are not propagated through the python interface (it just gives RuntimeError: unidentifiable C++ exception). Is this because the exception classes are not exposed?
* it would be nice to adopt more consistent and modular naming, e.g.: makeComposite => composite.make; modeMake => mode.make
* (I'm not sure, but) perhaps the user should be notified about (the necessity and progress of) on-demand compilation by streaming its output to cerr (so that it do not interfere with the usual output on cout).
You are right, {...} is a dictionary where the order does not matter. The (0,2), which serves as dictionary key in this case, is a tuple: an immutable ordered collection of items in Python.
* Thanks for the hint, I will fix the package dependencies.
* At the moment no exception classes are exposed. Putting it on my TODO list, I will have to learn how this is done.
* I also agree about the naming problem. At some point I even tried it the way you are suggesting but discarded it for some reaseon which I cannot recall now, will check that again.
* More verbose output is a good idea, otherwise impatient users will abort the compilation.
Branch python of repository "complete", "cpypyqed" and "core" in private repository: Brought the python interface up to date with the current development and improved many things. Overall the adaption to the development branch was quite unproblematic and most of the time went into implementing new stuff.
How great. One feature is more useful than the other. I’m looking forward to seeing this documented and integrated into the distribution.
Cpypyqed is in quite good shape now, of course still with limited coverage but self-consistent. In principle it could be merged to Development soon.
However we might consider leaving it in a feature branch. The wrappers are a bit fragile and could break if the libraries are changed. That might leave the repository in an inconsistent state. On the other hand, changes breaking cpypyqed are of course noticed faster if it is merged. This is your call.
And my answer is that I don’t understand the problem :) . I mean, cpypyqed is in its own repository, and it will remain there, won’t it?
You mean it might break monolithic builds if the python branch of cpypyqed gets merged into Development of cpypyqed? But monolithic builds will mostly be used by end-users of packages anyway.
I suggest to merge.
Of course you are right that problems can only occur for monolithic builds, however I would disagree that they are used mostly by end-users. I would encourage a work flow where, before pushing to the repository, the test suite is run, which requires a monolithic build and of course also tests cpypyqed (the io module already now, and python scripts / on-demand compilation in the python branch).
And even from the perspective of the end user, a consistent monolithic repository is important, because it is always frustrating for new users if they clone a project (and probably the monolithic build will be the preferred method) and it doesn't even compile :)
Agreed. In principle I pretty much agree with running the testsuite (perhaps not the whole, since after the inclusion of TestsuitePhysics it will be rather expensive) before each push, and I will try to conform to this principle in practice as well. And to be sure this means that one will be forced to immediately hack away with cpypyqed if it goes out of sync. But in case of too much difficulty, we can always use an intermediary branch and ask the other the sort it out :) .
And this is exactly what I suggest.
As to the end-user, she will always have the option to fall back to a (more or less supported) stable release.
I merged the python branch into Development. For now only on the private repository, maybe you could check if the testsuite runs successfully for you, too. There is a new check target check_fast which checks everything except cpypyqed, because the on-demand compilation takes quite some time.
For the cpypyqed documentation to build, you need sphinxcontrib.doxylink (e.g. install python2-pip and run 'pip2 install --user sphinxcontrib-doxylink'. This dependency is of course not a required one for C++QED.
Building the documentation in the monolithic project should then generate pages which have links to the other components on top, both in doxygen and sphinx.