Automated build

[nthiery]

It would be nice to have dockerhub build automatically our images. This would save us from having to maintain the infrastructure to do those builds on a regular basis. As a nice side effect, the docker image description on dockerhub would be automatically filled from the README (see #5)

The show stopper may bethat there are resource limits that dockerhub impose on the build process. See:
http://stackoverflow.com/questions/34440753/docker-hub-timeout-in-automated-build. We may not fit in them.

[saraedum]

We can work around the resource limit by using linked repositories. It's the officially proposed workaround. I am trying to setup a prototype.

[saraedum]

This is more limited than I thought. It appears that you can only do this when you only care about a tag such as latest. In more general setups this does not seem to work (linked repositories ignore rebuild rules that use a regex.)

[saraedum]

It's a bit counter to the way a Dockerfile usually works, but starting compilation from a sage/ directory containing all the temporary files of a previous build, compilation finishes in well under one hour. I wrote a multi-stage Dockerfile which has a switch for such compilation from a previous run at https://github.com/saraedum/sage. I still have to try it out with dockerhub but it could do the trick.
Note that we could copy over ccache like that as well (see #21.)

[saraedum]

So dockerhub runs a quite old version of docker, so I run into errors. It seems that they are not really developing dockerhub anymore. And docker cloud has no free autobuilds for open source projects anymore.

[saraedum]

Ok, I now have a working version (using CircleCI for the build) which pushes the images and the READMEs automatically at https://github.com/saraedum/sage/tree/circleci which builds to https://hub.docker.com/r/saraedum/sagemath/ and https://hub.docker.com/r/saraedum/sagemath-dev/.

[saraedum]

@nthiery, @embray: Are you zulip, so we could discuss this there?

[embray]

No, but I could try to be. But I am leaving in ~10 minutes. Give me chance to look over what you've been doing. It sounds impressive!

[embray]

I see you've completely reworked the Dockerfile using multi-stage builds. That feature didn't exist when I first started working on it, but it looks like it will make things simpler than my Makefile. I'm still not 100% sure how I feel about some of the fine details but I'm going to take a closer look tomorrow.

[embray]

I haven't used Circle-CI yet personally, though I've seen a few other projects that do. What do they offer that makes it such that there's enough time to actually perform the builds?

[saraedum]

They don't have time limits actually, i.e., you can set the timeout. They just limit open source projects to four concurrent builds. I think I misread something here. Not sure if you can get over the two hours limit.

[saraedum]

The current Dockerfile runs in less than one hour actually. The limiting factor of docker hub was that I ran out of space (inodes?) and that their version of docker is quite old (surprisingly.)

[saraedum]

About multi-stage builds: yes, it's somewhat new and makes the builder pattern mostly unnecessary. I like the idea of having everything in one Dockerfile that actually works, so people can just type docker build . to try it out. Of course, I copied many of the tricky parts from the existing Dockerfiles and I also don't think that my approach is perfect :)

[saraedum]

I am also not sure whether I forgot to copy something over. My dev image is 4GB 3GB and the non-dev is just 800 MB (compressed.)

[saraedum]

From 8.1 to 8.2.beta3 things have changed in the build system it seems, so CircleCI is now essentially rebuilding all of Sage's dependencies. It won't manage to do that on one core in 120 minutes. :(
I am not sure what we would want to do about things like that. We could build and push manually in cases like that but that's quite annoying.
An alternative would be Google Cloud's container builder which has no limits. It's free for the first 120 minutes per day and after that costs about $0.20 per hour. The main problem is that we don't get publicly visible logs I think and it's harder for people to set up themselves.
Another option is GitlabCI (which I like anyway.) I can set the runtime to 600 minutes per job and it does not complain and they say that there are no limits for open source projects (at the moment.)

[saraedum]

I'll try to transfer the circleCI build to gitlab and see if that's more convenient. They give us two cores, 50GB of disk, 2GB of RAM, and a recent docker.