Re: [tauruslib-devel] Taurus Continuous Integration and Documentation (was Re: Taurus-CI)

[Vincent M. <vin...@ma...>]

On 10/25/2016 01:53 PM, Carlos Pascual wrote
> Note: this is a reply to a private email from Vincent Michel, but it touches a
> topic that I wanted to open for public discussion, so I CC'ed the taurus-devel
> list. Vincent: I hope you do not mind it...

No problem!

> Now, with the Conda support (and with the work you did in getting the stack
> for PyTango in Conda), I guess we could get rid of most of the mocks, but
> probably not all of them (I haven't checked, but I do not think that e.g.
> epics and pyepics will have cunda packages).

I just checked and they're available on the lightsource2 anaconda 
channel: https://anaconda.org/lightsource2

> And, since we *plan* to use Travis for Continuous Deployment of packages, it
> seems natural to make the docs just another artifact from the Travis builds
> and have them delivered to e.g. github pages or pythonhosted, or wherever.
> Building the docs in Travis with docker also has the advantage of considering
> the success in building the docs as one more test during the CI.

Yes that makes a lot of sense. It's too bad that readthedocs doesn't 
provide an API to upload the documentation directly from travis (though 
it has been discussed at some point: 
https://github.com/rtfd/readthedocs.org/issues/1083)

> - For us, the main reason for going "the docker way" is that it enables us to
> introduce CI tests for configurations that closely match those used by
> different institutions: e.g. if MaxIV was interested in having every commit
> tested on  a system like the ones used in production, all that would be
> required would be for MaxIV to provide the link to a suitable  container in
> dockerhub. This way we could create a large set of test environments while
> keeping the maintenance work distributed (each interested party provides and
> updates its own container(s)).

That's a very good point!

> I hope I managed to cover your questions. Now I would be very interested in
> learning about your reasons, choices and concerns as well. I am sure there is
> a lot of experience than we can share.

Thanks for the thorough reply. In the case of pytango we're still 
experimenting, though the readthedocs + mock and travis + conda 
solutions seem to be promising.

Mocking is an acceptable hack for the pytango documentation since it is 
only one module to mock (tango._tango, see file [1]). Also having, the 
documentation separated from the regular build is good, since building 
the boost layer of pytango takes 10 minutes. That plus the collaborative 
features of ReadTheDocs should make it easy for people to contribute to 
the documentation, hopefully :)

Now the travis builds. I am actually not so sure conda is the best 
candidate for that, docker might actually be a better choice. But conda 
is interesting for several reasons:

- it's cross platform (run transparently on all linux OS)
- its installation doesn't require root access
- the anaconda cloud lets you upload your own packages
- the anaconda cloud lets you access many packages built by other users
- the packages don't have to be python packages
- though it integrates well with pip
- it's very simple to manage several environments with specific packages
- it's very simple to build and upload packages to anaconda
- conda-forge can build and upload packages for all platforms on its own 
channel

That means in the future, you might be able to run:

$ conda create --channel conda-forge --name tango python=3 taurus

to create a tango-ready environment with taurus and python3 that works 
on any platform, from scratch, without admin rights and without building 
anything like pip would do. We're not here yet, but I would like to see 
conda becoming the default solution for tutorial, trainings, short-term 
clients, etc.

Now back to the builds. One issue is that travis doesn't provide any 
integrated support for conda, like it does with docker. That means conda 
needs to be installed from scratch (see the pytango travis file [2]). 
Also, it would be nice for travis to support anaconda as a deployment 
provider. But otherwise, it works fine.

To conclude, I don't think docker and conda are mutually exclusive since 
we'll keep building platform-specific packages anyway. So I'd say it's 
good that we keep working on providing more resources for both technologies.

I have additional question regarding docker. One thing that I'd like to 
do in travis is to cache the build objects so that the boost layer 
doesn't build every time, but only if something changed on the boost 
side (which is pretty rare). Do you know if there is a way to achieve 
that using travis and/or docker capabilities?

Thanks,
/Vincent


[1] 
https://github.com/tango-cs/pytango/blob/master/doc/mock_tango_extension.py
[2] https://github.com/tango-cs/pytango/blob/develop/.travis.yml