Documentation missing some points

[richarvey]

Hi all,

I've been using buildx on osx-desktop (edge version) to build some multiarch images. I'm now trying to replicate the buildx experience on linux. In this case debian stretch, QEMU installed and docker 19.03. I've had limited success, but I'm really looking forward to getting this working.

First thing is that if starting from scratch we need to add a list of required installs on the host to make this work (if there are any). Secondly in order to make the build on 19.03+ work i had to enable experimental on docker. It might be nice if this could be added to the docs.

Now its build I can get this working:

$ docker buildx build .
[+] Building 8.4s (23/32)
 => ...

my docker buildx ls however shows only support for linux/amd64 despite my having QEMU installed. I'm guessing i need to link this somehow. I tried:

docker buildx create --platform linux/amd64/,linux/arm64,linux/arm/v7 --name mybuilder

This seems to work:

NAME/NODE    DRIVER/ENDPOINT             STATUS   PLATFORMS
mybuilder *  docker-container
  mybuilder0 unix:///var/run/docker.sock inactive linux/amd64, linux/arm64, linux/arm/v7

However i can't build on those target platforms :(

root@ip-10-100-0-29:/etc/docker# docker buildx build --platform linux/arm64 -t richarvey/nginx-demo --push .
[+] Building 2.0s (3/3) FINISHED
 => [internal] booting buildkit                                                                                                                                                                                    1.8s
 => => pulling image moby/buildkit:master                                                                                                                                                                          1.3s
 => => creating container buildx_buildkit_mybuilder0                                                                                                                                                               0.5s
 => [internal] load .dockerignore                                                                                                                                                                                  0.0s
 => => transferring context: 2B                                                                                                                                                                                    0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                               0.1s
 => => transferring dockerfile: 2B                                                                                                                                                                                 0.0s
failed to solve: rpc error: code = Unknown desc = failed to read dockerfile: open /tmp/buildkit-mount550903397/Dockerfile: no such file or directory

I'm probably missing a real simple step but couldn't find it int he guide. Any help really appreciated.

[tonistiigi]

my docker buildx ls however shows only support for linux/amd64 despite my having QEMU installed. I'm guessing i need to link this somehow. I tried:

Buildx will not detect the platforms before the builder starts. So either run docker buildx inspect --bootstrap and look at the detected builds or just start a build and that will start the builder instance.

There shouldn't be a need to specify platforms manually unless you want to give some node higher priority as manual platforms always override the automatic ones.

failed to solve: rpc error: code = Unknown desc = failed to read dockerfile: open /tmp/buildkit-mount550903397/Dockerfile: no such file or directory

You seem to be building from /etc/docker. You probably do not have a Dockerfile in that directory.

[thomas-riccardi]

After some digging and some guesswork (reading: I don't really know what I'm doing) I finally got it working on ubuntu 18.04 !

Sources:

• Qemu instruction translation for ARM for-linux#56
• https://github.com/computermouth/qemu-static-conf

Context

• desktop PC amd64
• ubuntu 18.04
• installed docker-ce 19.03-beta3 via "test" apt repo (sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) test"
• installed buildx 0.2.0 via github binary release as described in the README

Install and configure qemu

I'm using systemd-binfmt instead of binfmt-support (but maybe it could work too; the F fix-binary flag seems important).

# install qemu in static mode (without `binfmt-support`)
sudo apt install qemu qemu-user-static binfmt-support-
# install binfmt config files
git clone https://github.com/computermouth/qemu-static-conf.git
sudo mkdir -p /lib/binfmt.d
sudo cp qemu-static-conf/qemu-{aarch64,arm}-static.conf  /etc/binfmt.d
#  or enable more emulated architectures:
# sudo cp qemu-static-conf/*.conf /lib/binfmt.d/
sudo systemctl restart systemd-binfmt.service

Use buildx

Now that qemu and binfmt are properly configured, you can finally use buildx docker-container builder: - https://github.com/docker/buildx/#building-multi-platform-images

• https://engineering.docker.com/2019/04/multi-arch-images/
  (If you created your docker-container builder before configuring qemu it doesn't seem to detect the change: rm and re-createing it seems to work.)

Result

$ docker buildx ls
NAME/NODE    DRIVER/ENDPOINT             STATUS  PLATFORMS
mybuilder *  docker-container                    
  mybuilder0 unix:///var/run/docker.sock running linux/amd64, linux/arm64, linux/arm/v7, linux/arm/v6
default      docker                              
  default    default                     running linux/amd64

$ cat Dockerfile 
FROM ubuntu
RUN uname -a

$ docker buildx build --platform linux/amd64,linux/arm64 . --progress plain
WARN[0000] No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load 
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 136B done
#2 DONE 0.0s

#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s

#4 [linux/amd64 internal] load metadata for docker.io/library/ubuntu:latest
#4 DONE 1.0s

#3 [linux/arm64 internal] load metadata for docker.io/library/ubuntu:latest
#3 DONE 1.0s

#5 [linux/amd64 1/2] FROM docker.io/library/ubuntu@sha256:d26d529daa4d85671...
#5 resolve docker.io/library/ubuntu@sha256:d26d529daa4d8567167181d9d569f2a85da3c5ecaf539cace2c6223355d69981 done
#5 CACHED

#7 [linux/arm64 1/2] FROM docker.io/library/ubuntu@sha256:d26d529daa4d85671...
#7 resolve docker.io/library/ubuntu@sha256:d26d529daa4d8567167181d9d569f2a85da3c5ecaf539cace2c6223355d69981 done
#7 CACHED

#8 [linux/arm64 2/2] RUN uname -a
#8 ...

#6 [linux/amd64 2/2] RUN uname -a
#6 0.234 Linux buildkitsandbox 4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13 10:44:52 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
#6 DONE 0.4s

#8 [linux/arm64 2/2] RUN uname -a
#8 0.265 Linux buildkitsandbox 4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13 10:44:52 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
#8 DONE 0.4s

[tonistiigi]

Please create a PR with proposed docs changes.

The easiest way to setup qemu should be to run docker run --privileged linuxkit/binfmt:v0.7

[thomas-riccardi]

@tonistiigi

The easiest way to setup qemu should be to run docker run --privileged linuxkit/binfmt:v0.7

I was not aware of that...

Please create a PR with proposed docs changes.

I am not sure at all what I documented is the best way to go; I was just trying to quickly get it working from random sources of information.
It seems you have much more information than me about all that; I don't think I am qualified to write documentation for that...

[chewi]

I can't figure out for the life of me how you push to a private registry. No matter what I do, I get the following and I suspect it's trying to push to docker.io instead. If it's possible, can we also document this?

Error: failed to solve: rpc error: code = Unknown desc = server message: insufficient_scope: authorization failed

[tonistiigi]

@chewi You need to provide more details on what is failing and what you are trying and preferably a reproducer. buildx uses credentials set with docker login.

» gcloud auth configure-docker                                    
gcloud credential helpers already registered correctly.
 # docker buildx build --platform=linux/amd64,linux/arm64 -t gcr.io/myprojectid/hello --push .
[+] Building 14.2s (13/13) FINISHED
 => [internal] booting buildkit                                                                   4.8s
 => => pulling image moby/buildkit:master                                                         4.2s
 => => creating container buildx_buildkit_cranky_goodall0                                         0.7s
 => [internal] load build definition from Dockerfile                                              0.0s
 => => transferring dockerfile: 83B                                                               0.0s
 => [internal] load .dockerignore                                                                 0.0s
 => => transferring context: 2B                                                                   0.0s
 => [linux/arm64 internal] load metadata for docker.io/library/alpine:latest                      2.0s
 => [linux/amd64 internal] load metadata for docker.io/library/alpine:latest                      2.0s
 => [linux/amd64 1/3] FROM docker.io/library/alpine@sha256:28ef97b8686a0b5399129e9b763d5b7e5ff03  0.9s
 => => resolve docker.io/library/alpine@sha256:28ef97b8686a0b5399129e9b763d5b7e5ff03576aa5580d6f  0.0s
 => => sha256:28ef97b8686a0b5399129e9b763d5b7e5ff03576aa5580d6f4182a49c5fe1913 2.36kB / 2.36kB    0.0s
 => => sha256:5c40b3c27b9f13c873fefb2139765c56ce97fd50230f1f2d5c91e55dec171907 528B / 528B        0.0s
 => => sha256:bdf0201b3a056acc4d6062cc88cd8a4ad5979983bfb640f15a145e09ed985f92 2.76MB / 2.76MB    0.8s
 => => sha256:cdf98d1859c1beb33ec70507249d34bacf888d59c24df3204057f9a6c758dddb 1.51kB / 1.51kB    0.0s
 => => unpacking docker.io/library/alpine@sha256:28ef97b8686a0b5399129e9b763d5b7e5ff03576aa5580d  0.1s
 => [internal] load build context                                                                 0.0s
 => => transferring context: 83B                                                                  0.0s
 => [linux/arm64 1/3] FROM docker.io/library/alpine@sha256:28ef97b8686a0b5399129e9b763d5b7e5ff03  0.9s
 => => resolve docker.io/library/alpine@sha256:28ef97b8686a0b5399129e9b763d5b7e5ff03576aa5580d6f  0.0s
 => => sha256:bc6e6ad08312deb806ff4bf805c2e24f422859ff3f2082b68336e9b983fbc2f7 528B / 528B        0.0s
 => => sha256:6f37394be673296a0fdc21b819c5df40431baf7d3af121bee451726dd1457493 2.69MB / 2.69MB    0.8s
 => => sha256:660030ffd296125c8af2f25a34ab2c12ac3d7656b31ac40ccf52eb14cd1c4bf0 1.51kB / 1.51kB    0.0s
 => => sha256:28ef97b8686a0b5399129e9b763d5b7e5ff03576aa5580d6f4182a49c5fe1913 2.36kB / 2.36kB    0.0s
 => => unpacking docker.io/library/alpine@sha256:28ef97b8686a0b5399129e9b763d5b7e5ff03576aa5580d  0.1s
 => [linux/amd64 2/3] RUN touch hello                                                             0.3s
 => [linux/arm64 2/3] RUN touch hello                                                             0.3s
 => [linux/amd64 3/3] COPY Dockerfile .                                                           0.1s
 => [linux/arm64 3/3] COPY Dockerfile .                                                           0.1s
 => exporting to image                                                                            5.8s
 => => exporting layers                                                                           0.1s
 => => exporting manifest sha256:01b16d9a88ca1a77809326722129a18c7ee77699b019cb91e63767269fef65a  0.0s
 => => exporting config sha256:c2451d1ba9f2d2cc8f5b75b1b68cc407abd7ba8e5bba74cc6b71f050845dd091   0.0s
 => => exporting manifest sha256:9b72c62a1fb85d8d4b8153e6e27d53a0454ec9ec787fceb3af40e621069b247  0.0s
 => => exporting config sha256:d4c534aa9fc5e4d2d559718e86173ed0630db7d7fbbcb285ba1481db36d84941   0.0s
 => => exporting manifest list sha256:c34beebaa024aa1f9a098a874c166b24bc528e5a5872537359be57588c  0.0s
 => => pushing layers                                                                             3.1s
 => => pushing manifest for gcr.io/myprojectid/hello:latest                                 2.6s
 # docker buildx imagetools inspect gcr.io/myprojectid/hello:latest
Name:      gcr.io/myprojectid/hello:latest
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:c34beebaa024aa1f9a098a874c166b24bc528e5a5872537359be57588c3e74d8

Manifests:
  Name:      gcr.io/myprojectid/hello:latest@sha256:01b16d9a88ca1a77809326722129a18c7ee77699b019cb91e63767269fef65a4
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64

  Name:      gcr.io/myprojectid/hello:latest@sha256:9b72c62a1fb85d8d4b8153e6e27d53a0454ec9ec787fceb3af40e621069b2474
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64

Also, a good idea to create a new issue if you have a specific problem.

[chewi]

Thanks for showing that it can work. I thought it was merely a docs issue but perhaps not. I was trying to push to Artifactory, which works without buildx, but no luck here. I've since decided to just wait until the BuildKit stuff officially hits Docker as I'm having to jump through far too many hoops right now.

[vielmetti]

According to

https://community.arm.com/developer/tools-software/tools/b/tools-software-ides-blog/posts/getting-started-with-docker-for-arm-on-linux

There is a need to set the environment variable

export DOCKER_CLI_EXPERIMENTAL=enabled

to get buildx to work on arm64.

[thomas-riccardi]

According to

https://community.arm.com/developer/tools-software/tools/b/tools-software-ides-blog/posts/getting-started-with-docker-for-arm-on-linux

There is a need to set the environment variable

export DOCKER_CLI_EXPERIMENTAL=enabled

to get buildx to work on arm64.

Or add to ~/.docker/config.json:

"experimental": "enabled"

(related to #81, only seems to happen with buildx shipped with docker-cli, not the github releases here)
(this could be added to https://beta-stage.docs.docker.com/buildx/working-with-buildx/ ...)

[yiblet]

Thank you so much for this, I hit this bug too.

[mcnaveen]

Thanks, @thomas-riccardi 💯 it worked

[emretapci]

You seem to be building from /etc/docker. You probably do not have a Dockerfile in that directory.

I am trying the same thing on Windows 10, with docker desktop. In the current directory that I run the following command;

docker buildx build --platform linux/arm/v7 -t opc_ua_open62541 .

I have a nice Dockerfile, and the above command gives the same result:

failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to read dockerfile: open /tmp/buildkit-mount199750085/Dockerfile: no such file or directory

What might be the reason?