Custom registry, push error on self-signed cert

[minusdelta]

'buildx' errors, while 'docker build' succeeds:

cat <<'EOD' > Dockerfile
FROM alpine
RUN touch /test
EOD

docker buildx build  \
  -t img.service.consul/alpine:test  \
 --platform=linux/amd64,linux/arm64,linux/arm  \
 --push  \
 .

[+] Building 2.1s (12/12) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                       0.0s
 => => transferring dockerfile: 65B                                                                                                                                                                        0.0s
 => [internal] load .dockerignore                                                                                                                                                                          0.1s
 => => transferring context: 2B                                                                                                                                                                            0.0s
 => [linux/arm/v7 internal] load metadata for docker.io/library/alpine:latest                                                                                                                              1.5s
 => [linux/amd64 internal] load metadata for docker.io/library/alpine:latest                                                                                                                               1.3s
 => [linux/arm64 internal] load metadata for docker.io/library/alpine:latest                                                                                                                               1.3s
 => CACHED [linux/arm64 1/2] FROM docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6                                                                         0.0s
 => => resolve docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6                                                                                            0.0s
 => [linux/arm64 2/2] RUN touch /test                                                                                                                                                                      0.1s
 => CACHED [linux/amd64 1/2] FROM docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6                                                                         0.0s
 => => resolve docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6                                                                                            0.0s
 => [linux/amd64 2/2] RUN touch /test                                                                                                                                                                      0.1s
 => CACHED [linux/arm/v7 1/2] FROM docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6                                                                        0.0s
 => => resolve docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6                                                                                            0.0s
 => [linux/arm/v7 2/2] RUN touch /test                                                                                                                                                                     0.2s
 => ERROR exporting to image                                                                                                                                                                               0.4s
 => => exporting layers                                                                                                                                                                                    0.2s
 => => exporting manifest sha256:879ac4adf9493121ff9bb12f8566ed993fa7079c59ae02b516a9287d6de7daea                                                                                                          0.0s
 => => exporting config sha256:42c2e158eb64d21ea6832e4842a3c11c9fd70c89afbf2fd0fbe2f16dd2698453                                                                                                            0.0s
 => => exporting manifest sha256:64b17d691e5c5ab257ad37b622c9ed219e50ea637bf5f7aa25e2f65f0bd0c26d                                                                                                          0.0s
 => => exporting config sha256:14af510e076d60389743c6fc7c99e2777ba56bdad11cbbffacb438e7c68f6321                                                                                                            0.0s
 => => exporting manifest sha256:2e87dbf064ba1c829a9d18525fa38e77baa26b654c04659b0fa3e75d6ea34ea5                                                                                                          0.0s
 => => exporting config sha256:f1b89d61d625bff13e65a679d2bdb1c513289999789ec2e13fe7acefca39adfd                                                                                                            0.0s
 => => exporting manifest list sha256:53f07aa12e20079138de3650629277928313e7bfdc59c3f22c93834fe11ba9f3                                                                                                     0.0s
 => => pushing layers                                                                                                                                                                                      0.0s
------
 > exporting to image:
------
failed to solve: rpc error: code = Unknown desc = failed to do request: Head https://img.service.consul/v2/alpine/blobs/sha256:8dc302c06141b7124ea05ccf2fdde10013ce594c28e5fe980047b0740891e398: x509: certificate signed by unknown authority

x509: certificate signed by unknown authority, but certificate chain is ok.

test:
: |openssl s_client -connect img.service.consul:443 [...] Verify return code: 0 (ok)

docker build + push works also:

docker build  \
 -t img.service.consul/x86_64/alpine:test  \
 .

Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM alpine
 ---> 055936d39205
Step 2/2 : RUN touch /test
 ---> Using cache
 ---> 7bd5dcd02d4c
Successfully built 7bd5dcd02d4c
Successfully tagged img.service.consul/x86_64/alpine:test

docker push img.service.consul/x86_64/alpine:test

The push refers to repository [img.service.consul/x86_64/alpine]
b13e8440598c: Pushed
f1b5933fe4b5: Pushed
test: digest: sha256:e9c2e8f188d0bedc6d3c26b39a6a75c36be5b4cbeedb9defc4f3b48953b4ef45 size: 734

buildx imagetools inspect again works:

docker buildx imagetools inspect img.service.consul/x86_64/alpine:test

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1645,
      "digest": "sha256:7bd5dcd02d4c340892fe431a40a39badf5695af58b669a33bd21b61159f4ffe5"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2757034,
         "digest": "sha256:e7c96db7181be991f19a9fb6975cdbbd73c65f4a2681348e63a141a2192a5f10"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 97,
         "digest": "sha256:6e0d312d9ebb1961db61366af2a5a323ad84155db2018457d2c5168c4f86e410"
      }
   ]
}

perhaps related to #57 (comment)

tested:

• docker beta-4 + 20190517171028-f4f33ba16d nightly,
• 'buildx' release + current master.

uname -mrov
4.19.44 #1-NixOS SMP Thu May 16 17:41:32 UTC 2019 x86_64 GNU/Linux

[chewi]

In my case, it wasn't a self-signed cert but a company-wide CA cert. It's very possible that CA cert wasn't present in the environment I was running buildx from though so I could look into that.

[sudo-bmitch]

I'm seeing similar with a standalone registry server and locally created CA/certificates. I'm able to run a docker login after adding a /etc/docker/certs.d/<hostname>:5000/ca.crt file, but buildx doesn't appear to use this.

[klo2k]

I managed to work-around this - by adding my own CA's cert to the generated container, and restarting it.

$ docker ps|grep 'moby/buildkit'
ee110c9e6dfc        moby/buildkit:buildx-stable-1   "buildkitd"              7 minutes ago       Up 7 minutes                                                                                       buildx_buildkit_mybuilder0

$ docker exec -it ee110c9e6dfc sh
$$ cat >> /etc/ssl/certs/ca-certificates.crt <<'EOF'
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
EOF
$$ exit

$ docker restart ee110c9e6dfc

It aint pretty, but it works!

[mjan-jansson]

A marginally more robust work-around, but still not pretty (no error checking etc):

BUILDER=$(sudo docker ps | grep buildkitd | cut -f1 -d' ')
sudo docker cp YOUR-CA.crt $BUILDER:/usr/local/share/ca-certificates/
sudo docker exec $BUILDER update-ca-certificates
sudo docker restart $BUILDER

A possible solution/suggestion would be to allow for something like:
docker secret create a-ca-secret my-ca-file.crt
docker buildx create ---name builder --driver-opt private-ca-cert=/run/secrets/a-ca-secret
...and then have the e.g. the docker-container builder pull in the cert and call update-ca-certificates when it starts.

[fopina]

as I just bumped into this issue, I'd like to add one more option instead of mangling an existing builder container. driver-opt has a image option (for docker-container driver).

I went with the option to have a 2 line Dockerfile that adds my internal CAs to moby/buildkit and use that image when creating the builder.

[bdharrington7]

@fopina can you elaborate on how you did that? I have a 2 line dockerfile where I copy a cert into /usr/local/share/ca-certificates/, and I can run it and exec into it to verify that it's there. I create the image locally, tagging it like testbuilder:latest

When I run docker buildx create --name tester --builder testbuilder:latest, the resulting container that's spun up doesn't have the cert in that folder:

Create and verify the image:

% docker run --rm -ti testbuilder sh
INFO[0000] auto snapshotter: using native               
WARN[0000] using host network as the default            
INFO[0000] found worker "t3jdqv8o0ov117tdipqjmfnly", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:000f6536acfa org.mobyproject.buildkit.worker.snapshotter:native], platforms=[linux/amd64 linux/arm64 linux/riscv64 linux/ppc64le linux/s390x linux/386 linux/arm/v7 linux/arm/v6] 
WARN[0000] skipping containerd worker, as "/run/containerd/containerd.sock" does not exist 
INFO[0000] found 1 workers, default="t3jdqv8o0ov117tdipqjmfnly" 
WARN[0000] currently, only the default worker can be used. 
INFO[0000] running server on /run/buildkit/buildkitd.sock

# In a different terminal window:
% docker ps
CONTAINER ID   IMAGE         COMMAND          CREATED          STATUS          PORTS     NAMES
000f6536acfa   testbuilder   "buildkitd sh"   18 seconds ago   Up 17 seconds             jovial_blackwell

% docker exec -it jovial_blackwell sh
/ # cd /usr/local/share/ca-certificates/
/usr/local/share/ca-certificates # ls
certs.pem

using it as a builder?

% docker buildx create --name tester --builder testbuilder:latest
% docker buildx use tester
% docker buildx inspect --bootstrap
[+] Building 1.0s (1/1) FINISHED                                                                                                       
 => [internal] booting buildkit                                                                                                   1.0s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                0.4s
 => => creating container buildx_buildkit_tester0                                                                                 0.6s
Name:   tester
Driver: docker-container

Nodes:
Name:      tester0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

% docker ps
CONTAINER ID   IMAGE                           COMMAND       CREATED         STATUS         PORTS     NAMES
d9c5e77418fc   moby/buildkit:buildx-stable-1   "buildkitd"   9 seconds ago   Up 8 seconds             buildx_buildkit_tester0

docker exec -ti d9c5e77418fc sh
/ # ls /usr/local/share/ca-certificates/
/ # %                                                     < -- nothing here

[fopina]

@bdharrington7 have you tried curl in that image (on the registry endpoint) to validate that they have been properly installed?
I believe you have to run update-ca command unless you already pushed the total file.

[bdharrington7]

I didn't try curl, but I did try using the --push option when I built another image using buildx, and it still failed, because the registry I'm pushing to uses an internal cert. I assumed that the image I specified with the --builder flag wasn't being used, since it didn't have the same information in the one I actually built when I spun up testbuilder:latest manually (the cert I baked into the testbuilder image at /usr/local/share/ca-certificates/ isn't even there)

[hemna]

A marginally more robust work-around, but still not pretty (no error checking etc):

BUILDER=$(sudo docker ps | grep buildkitd | cut -f1 -d' ')
sudo docker cp YOUR-CA.crt $BUILDER:/usr/local/share/ca-certificates/
sudo docker exec $BUILDER update-ca-certificates
sudo docker restart $BUILDER

A possible solution/suggestion would be to allow for something like:
docker secret create a-ca-secret my-ca-file.crt
docker buildx create ---name builder --driver-opt private-ca-cert=/run/secrets/a-ca-secret
...and then have the e.g. the docker-container builder pull in the cert and call update-ca-certificates when it starts.

Tried that, but it didn't work.