Re: [Quantlib-dev] Building python wheels

[Achilleas M. <mic...@gm...>]

Hi Luigi,

Normally on a CI pipeline you can control when the various artifacts are
built. So for example you can launch the Python wheels build as soon as a
new release is tagged on the master branch.
So you can use it also officially. This way you will be able to reduce the
intervals between releases thanks to that automation.

I've looked into it and it's quite some work, so I'll have to park it for
the moment until I get some more resources on my project. It should be
around October. Will keep you updated.

Kind regards,
Achilleas



On Wed, 7 Aug 2019, 17:04 Luigi Ballabio, <lui...@gm...> wrote:

> Hello Achilleas,
>      if you do, please by all means notify the community here.  However, I
> think I'd keep them separate from the official releases (mostly to avoid
> supporting daily versions).
>
> Thanks,
>     Luigi
>
>
> On Wed, Jul 10, 2019 at 11:34 AM Achilleas Michos <
> mic...@gm...> wrote:
>
>> I see, it is because the last image in the multi-stage actually gets the
>> wheel from the previous images. So itself does not have the wheel package.
>>
>> Will try as CI pipeline based on those instructions then. Any insights
>> about windows builds (e.g. AppVeyor)?
>>
>> If I manage to create a pipeline that produces the wheels binary, is it
>> something that would interest the community?
>>
>> On Wed, Jul 10, 2019 at 11:27 AM Luigi Ballabio <lui...@gm...>
>> wrote:
>>
>>> Those instructions should work, yes.  The devenv image doesn't have the
>>> wheel package, because it doesn't build wheels.  You need to install it
>>> with pip, if you want to use that image for the build.  However, if you
>>> build your wheel in that image, it will only work in a corresponding Ubuntu
>>> distribution and it will need libQuantLib.so installed.  To build a
>>> generic, self-contained wheel you need to use the manylinux image instead.
>>>
>>> Luigi
>>>
>>>
>>> On Wed, Jul 10, 2019 at 11:17 AM Achilleas Michos <
>>> mic...@gm...> wrote:
>>>
>>>> Thank you both for your input!
>>>>
>>>> So for the build I should follow the instructions in
>>>> https://github.com/lballabio/QuantLib-SWIG/issues/103 or there are
>>>> more recent?
>>>>
>>>> I'm still quite perplexed from the fact that when I run it in the
>>>> ballabio/quantlib-swig-devenv:python3 it does not work, as the image itself
>>>> runs the same command internally.
>>>> I do:
>>>> docker run --rm -ti -v /tmp/QuantLib-SWIG/:/build -w /build
>>>> lballabio/quantlib-swig-devenv:python3 ./.travis/python3.build
>>>>
>>>> Where ./.travis/python3.build is
>>>> #!/bin/bash
>>>>
>>>> ./autogen.sh \
>>>> && ./configure PYTHON=/usr/bin/python3 CXXFLAGS='-O1' \
>>>> && make -C Python \
>>>> && make -C Python check \
>>>> && make -C Python install \
>>>> && for i in Python/examples/*.py ; do echo $i && /usr/bin/python3 $i
>>>> || break -1 ; done \
>>>> && cd Python && CXXFLAGS='-g0 -O3' /usr/bin/python3 setup.py
>>>> bdist_wheel
>>>>
>>>> OUTCOME:
>>>> usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
>>>>    or: setup.py --help [cmd1 cmd2 ...]
>>>>    or: setup.py --help-commands
>>>>    or: setup.py cmd --help
>>>>
>>>> error: invalid command 'bdist_wheel'
>>>>
>>>> On Wed, Jul 10, 2019 at 10:51 AM Luigi Ballabio <
>>>> lui...@gm...> wrote:
>>>>
>>>>> Hello,
>>>>>     the process to build the wheels on PyPI is still manual.  They're
>>>>> manylinux wheels (i.e., compatible with most Linux distributions), which
>>>>> means that both the library and the wrappers have to be compiled inside the
>>>>> Docker images provided at <https://github.com/pypa/manylinux> (you'll
>>>>> also have to install Boost inside them) and that they have to be processed
>>>>> by the auditwheel tool in order to include the shared C++ library
>>>>> referenced by the wrappers.  More details at the link above.
>>>>>
>>>>> Luigi
>>>>>
>>>>>
>>>>> On Wed, Jul 10, 2019 at 10:02 AM Achilleas Michos <
>>>>> mic...@gm...> wrote:
>>>>>
>>>>>> Hi Alix,
>>>>>>
>>>>>> Thanks for coming back! Yes, on my machine it is installed and
>>>>>> building the wheels works fine.
>>>>>> I'm trying to do the same using the script that Travis runs for
>>>>>> Quantlib-SWIG. That script is run inside
>>>>>> the lballabio/quantlib-swig-devenv:python3 docker image.
>>>>>> Does this mean that the images does not have wheel installed? i.e. if
>>>>>> I want to build wheels I have to make my own custom image to run on my CI?
>>>>>>
>>>>>>
>>>>>> On Wed, Jul 10, 2019 at 9:58 AM ALIX LASSAUZET <al...@la...>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Achilleas,
>>>>>>>
>>>>>>> Have you installed the wheel package on your machine (orin the
>>>>>>> virtual env) first?
>>>>>>> > pip install wheel
>>>>>>>
>>>>>>> Alix
>>>>>>>
>>>>>>> Le mer. 10 juil. 2019 à 09:54, Achilleas Michos <
>>>>>>> mic...@gm...> a écrit :
>>>>>>>
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> I'm trying to expose in Python features of the QuantLib that are
>>>>>>>> not yet in the Quantlib-SWIG.
>>>>>>>> For that I've forked the later and I've done my changes. Now I need
>>>>>>>> to compile a python wheel and provide it as a pip install-able package, to
>>>>>>>> use in my project.
>>>>>>>> I want, pretty much, to replicate the process that creates
>>>>>>>> the QuantLib-Python deliveries in PyPI.
>>>>>>>>
>>>>>>>> I've noticed that the Quantlib-SWIG has a CI configuration setup
>>>>>>>> for testing, though not for building wheels for PyPi. The build/test script
>>>>>>>> looks like this:
>>>>>>>>
>>>>>>>> ./autogen.sh \
>>>>>>>> && ./configure PYTHON=/usr/bin/python3 CXXFLAGS='-O1' \
>>>>>>>> && make -C Python \
>>>>>>>> && make -C Python check \
>>>>>>>> && make -C Python install \
>>>>>>>> && for i in Python/examples/*.py ; do echo $i && /usr/bin/python3
>>>>>>>> $i || break -1 ; done
>>>>>>>>
>>>>>>>> I added the following commands at the end:
>>>>>>>>
>>>>>>>> && cd Python &&  /usr/bin/python3 setup.py bdist_wheel
>>>>>>>>
>>>>>>>> Aiming to recover the .whl in the travis worker and publish it.
>>>>>>>> Unfortunately, the command gives me the following error even though
>>>>>>>> the same works as expected on my machine (Ubuntu 18.04).
>>>>>>>>
>>>>>>>> usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts]
>>>>>>>> ...]
>>>>>>>>    or: setup.py --help [cmd1 cmd2 ...]
>>>>>>>>    or: setup.py --help-commands
>>>>>>>>    or: setup.py cmd --help
>>>>>>>>
>>>>>>>> error: invalid command 'bdist_wheel'
>>>>>>>>
>>>>>>>> Could someone help me figure out why in the docker container the
>>>>>>>> command is not recognized but it is on my machine?
>>>>>>>> By the way, how are the wheels posted on PyPi get generated? Is it
>>>>>>>> manually or there is a CI process?
>>>>>>>>
>>>>>>>> Kind regards,
>>>>>>>> Achilleas
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> QuantLib-dev mailing list
>>>>>>>> Qua...@li...
>>>>>>>> https://lists.sourceforge.net/lists/listinfo/quantlib-dev
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>> QuantLib-dev mailing list
>>>>>> Qua...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/quantlib-dev
>>>>>>
>>>>>