Support --secret and --ssh flags in docker build args for buildkit

[bvtujo]

Similar to #1555, we should consider supporting the --secret and --ssh flags in our calls to Docker Build. This is a feature of buildkit which can help customers with builds requiring authentication to private repositories or private key files.

[ColeDCrawford]

This would definitely be useful. Supporting both --secret and --ssh flags would allow for projects to use deploy keys for private repositories without leaving traces in the final image.

[rmarapp-dio]

+1. This one is much needed. Thanks, we have been struggling with a bunch of workarounds. This would help.

[RobinVds]

+1

[nbarrera]

I need to access a private github repo when building the docker image.

I 've read the most secure way would be to use the --ssh flag.

Can you point me to any workaround to this?

Is it possible to pass build arguments using --build-arg? I wouldn't pass the key as an environment variable as I 'm afraid it ends up available on the running container.

Maybe I end up using subst on the Dockerfile and then using multi-stage build to "erase" the value

[sribas]

What's currently the best way to do this? Pass an SSH key using the image.build.args? The SSH key can be stored and referenced from AWS Secrets Manager so it's not in the manifest (and therefore not visible in the repo) but it will still show in the Dockerfile build history, correct?

[Kylir]

Hey there!
Any update on that feature please?
We are now using private npm modules and we need the --secret option in the docker build command.
Any idea how to do that?

[ColeDCrawford]

I was able to open source the two private dependency repos we had, but the way I handled this previously was to build the image with buildkit and the SSH args and then reference the image in the copilot manifest.yml. Not ideal but it works. Setting up CD with that would be a bit trickier though - I have a nice flow for projects with Github Actions and you'd have to do some extra steps there rather than just copilot svc deploy.

[Kylir]

Thanks @ColeDCrawford for the answer.
In my case I have a Copilot Pipeline and I want to build on AWS pulling private NPM modules hosted by Github.
Really not sure how to handle that.

[Lou1415926]

Hi @Kylir ! Thank you very much for the inquiry

Apologies 🙇🏼‍♀️ ! At this moment, this feature isn't in our short term roadmap yet. Here is our sprint board that shows the ongoing features that we are currently working on.

For the time being, would customizing the buildspec.yml work for your use case? You can add --secret flag at this line in your buildspec.

[Kylir]

Hi @Lou1415926 ,
Thanks for the post. You are very correct. It is exactly what I'm doing right now!
I declared some parameters taken from Parameter Store and tweaked the docker build command.
It's still work in progress... But it looks promising.

[Kylir]

Update! It's working! Yeah! I've been able to pull private NPM modules from Github.

In case it is useful to anyone, here is what I did:

Secret in the Parameter Store

First I created my secret in the AWS Parameter Store. Let's name it SUPER_SECRET

Edit build file

Then, I created a Copilot pipeline: copilot pipeline init
This created a buildspec.yml file that I edited.
At the top I declared an env variable from the parameter store

# Buildspec runs in the build stage of your pipeline.
version: 0.2
env:
    parameter-store:
        SUPER_SECRET: SUPER_SECRET
phases:
  install:
    runtime-versions:
      docker: 18

Then, almost at the bottom I changed the docker build command for those 3 lines:

export DOCKER_BUILDKIT=1
echo $SUPER_SECRET > secret.txt
docker build --secret id=secret,src=secret.txt -t $workload:$tag $build_args-f $df_path $df_dir_path;          

Use the secret

Finally I'm using this secret in the Dockerfile to set an env variable that contains my secret.

RUN --mount=type=secret,id=secret export NPM_TOKEN=$(cat /run/secrets/secret) && npm install --production

Source: Docker

[codekitchen]

I have a use case where we need the --ssh flag, I think I will take this on since it seems like a straightforward first task to get familiar with the copilot codebase. However, I was wondering if a better approach would be to support arbitrary extra flags doing something like:

image:
  build:
    dockerfile: Dockerfile
    build_flags: ['--ssh', 'default']

Rather than

image:
  build:
    dockerfile: Dockerfile
    ssh: default

I just worry that it will be a never-ending game of whack-a-mole trying to support every possible docker build parameter that somebody might need.

A potential counter-argument to build_flags is that so far this yaml config seems to be trying to mirror the same compose build: functionality. Unfortunately compose doesn't support --ssh yet either so there's not a pattern to follow there. I see a few unresolved issues such as compose-spec/compose-spec#81 that have been discussed for a couple years without resolution.