Re: [tauruslib-devel] Taurus Continuous Integration and Documentation (was Re: Taurus-CI)

[Carlos P. <cpa...@ce...>]

On Wednesday, October 26, 2016 12:01:14 PM CEST Vincent Michel wrote:
> On 10/25/2016 01:53 PM, Carlos Pascual wrote
>(...)
> >(I haven't checked, but I do not think that e.g.
> > epics and pyepics will have cunda packages).
> 
> I just checked and they're available on the lightsource2 anaconda
> channel: https://anaconda.org/lightsource2

good to know!

>  (...) It's too bad that readthedocs doesn't
> provide an API to upload the documentation directly from travis (though
> it has been discussed at some point:
> https://github.com/rtfd/readthedocs.org/issues/1083)

Yes, that would be awesome...

> Mocking is an acceptable hack for the pytango documentation since it is
> only one module to mock (tango._tango, see file [1]). 

I agree. In Taurus I worry about more dependencies, though

> Also having, the
> documentation separated from the regular build is good, since building
> the boost layer of pytango takes 10 minutes. That plus the collaborative
> features of ReadTheDocs should make it easy for people to contribute to
> the documentation, hopefully :)

I suppose this could be alleviated if you manage to setup the caching (see 
below)
 
> Now the travis builds. I am actually not so sure conda is the best
> candidate for that, docker might actually be a better choice. But conda
> is interesting for several reasons:
> 
> - it's cross platform (run transparently on all linux OS)
> - its installation doesn't require root access
> - the anaconda cloud lets you upload your own packages
> - the anaconda cloud lets you access many packages built by other users
> - the packages don't have to be python packages
> - though it integrates well with pip
> - it's very simple to manage several environments with specific packages
> - it's very simple to build and upload packages to anaconda
> - conda-forge can build and upload packages for all platforms on its own
> channel

I definitely need to learn more about conda... Johan Forsberg already sold me 
some of those points last week ;)

> 
> That means in the future, you might be able to run:
> 
> $ conda create --channel conda-forge --name tango python=3 taurus
> 
> to create a tango-ready environment with taurus and python3 that works
> on any platform, from scratch, without admin rights and without building
> anything like pip would do. We're not here yet, but I would like to see
> conda becoming the default solution for tutorial, trainings, short-term
> clients, etc.

That's nice, yes... although it is more or less what you get with Docker 
(except for the admin permissions part)

> 
> Now back to the builds. One issue is that travis doesn't provide any
> integrated support for conda, like it does with docker. That means conda
> needs to be installed from scratch (see the pytango travis file [2]).
> Also, it would be nice for travis to support anaconda as a deployment
> provider. But otherwise, it works fine.
> 
> To conclude, I don't think docker and conda are mutually exclusive since
> we'll keep building platform-specific packages anyway. So I'd say it's
> good that we keep working on providing more resources for both technologies.

I agree, in some cases, I would consider useful even to use Conda within a 
docker...
Besides, even if we base our solution on docker, I think we should automate 
the deployment of Conda  packages...

> 
> I have additional question regarding docker. One thing that I'd like to
> do in travis is to cache the build objects so that the boost layer
> doesn't build every time, but only if something changed on the boost
> side (which is pretty rare). Do you know if there is a way to achieve
> that using travis and/or docker capabilities?

I haven't worked with caching... 
I guess you 've already looked here: https://docs.travis-ci.com/user/caching/

If the above does not work,  maybe you can instruct travis to deploy your 
build objects as a tar on every successful build to some ad-hoc repo (or some 
ftp server or whatever) and then download and untar it in the  before_install 
phase. This would work both with the Travis VM directly with docker (via a 
mounted volume).

Well, thanks for all the food for thought. Let's keep this flowing...

Cheers!

Carlos


-- 
+----------------------------------------------------+
 Carlos Pascual Izarra
 Scientific Software Coordinator
 Computing Division
 ALBA Synchrotron  [http://www.albasynchrotron.es]
 Carrer de la Llum 2-26
 E-08290 Cerdanyola del Valles (Barcelona), Spain
 E-mail: cpa...@ce...
 Phone: +34 93 592 4428
+----------------------------------------------------+