Getting certificates from step-ca + a reverse proxy that uses a Web PKI certificate #1017

achernya · 2022-08-01T16:53:43Z

achernya
Aug 1, 2022

Steps to Reproduce

Set up caddy with Let's Encrypt certificates
Set up step-ca behind the caddy, with its own root certificate. Configure caddy to accept step-ca's certificates.
Try to provision a host ssh certificate with step ssh certificate ${FQDN} /etc/ssh/ssh_host_ecdsa_key.pub --host --sign --principal ${FQDN} --principal ${HOSTNAME}

Your Environment

OS - 3.16.0 (docker container)
step-ca Version - Smallstep CA/0.21.0 (linux/amd64)

Expected Behavior

SSH certificate provisioning succeeds, after JWT password is entered.

Actual Behavior

Get "https://${CA_URL}/provisioners?limit=100": x509: certificate signed by unknown authority
github.com/smallstep/certificates/ca.(*Client).Provisioners
        github.com/smallstep/[email protected]/ca/client.go:826
github.com/smallstep/certificates/pki.GetProvisioners
        github.com/smallstep/[email protected]/pki/pki.go:148
github.com/smallstep/cli/utils/cautils.NewTokenFlow
        github.com/smallstep/cli/utils/cautils/token_flow.go:100
github.com/smallstep/cli/utils/cautils.(*CertificateFlow).GenerateSSHToken
        github.com/smallstep/cli/utils/cautils/certificate_flow.go:163
github.com/smallstep/cli/command/ssh.certificateAction
        github.com/smallstep/cli/command/ssh/certificate.go:273
go.step.sm/cli-utils/command.ActionFunc.func1
        go.step.sm/[email protected]/command/command.go:37
github.com/urfave/cli.HandleAction
        github.com/urfave/[email protected]/app.go:522
github.com/urfave/cli.Command.Run
        github.com/urfave/[email protected]/command.go:173
github.com/urfave/cli.(*App).RunAsSubcommand
        github.com/urfave/[email protected]/app.go:405
github.com/urfave/cli.Command.startApp
        github.com/urfave/[email protected]/command.go:372
github.com/urfave/cli.Command.Run
        github.com/urfave/[email protected]/command.go:102
github.com/urfave/cli.(*App).Run
        github.com/urfave/[email protected]/app.go:277
main.main
        ./main.go:113
runtime.main
        runtime/proc.go:250
runtime.goexit
        runtime/asm_amd64.s:1571

If you instead add --root=/etc/ssl/certs/ca-certificates.crt to the command line, the error becomes

Please enter the password to decrypt the provisioner key: 
error decoding /etc/ssl/certs/ca-certificates.crt: contains more than one PEM encoded block
github.com/smallstep/cli/crypto/pemutil.Parse
        github.com/smallstep/cli/crypto/pemutil/pem.go:243
github.com/smallstep/cli/crypto/pemutil.Read
        github.com/smallstep/cli/crypto/pemutil/pem.go:318
github.com/smallstep/cli/crypto/pemutil.ReadCertificate
        github.com/smallstep/cli/crypto/pemutil/pem.go:171
github.com/smallstep/cli/token.WithRootCA.func1
        github.com/smallstep/cli/token/options.go:37
github.com/smallstep/cli/token.NewClaims
        github.com/smallstep/cli/token/token.go:103
github.com/smallstep/cli/token/provision.New
        github.com/smallstep/cli/token/provision/provision.go:25
github.com/smallstep/cli/utils/cautils.(*TokenGenerator).Token
        github.com/smallstep/cli/utils/cautils/token_generator.go:83
github.com/smallstep/cli/utils/cautils.(*TokenGenerator).SignSSHToken
        github.com/smallstep/cli/utils/cautils/token_generator.go:115
github.com/smallstep/cli/utils/cautils.generateJWKToken
        github.com/smallstep/cli/utils/cautils/token_generator.go:394
github.com/smallstep/cli/utils/cautils.NewTokenFlow
        github.com/smallstep/cli/utils/cautils/token_flow.go:138
github.com/smallstep/cli/utils/cautils.(*CertificateFlow).GenerateSSHToken
        github.com/smallstep/cli/utils/cautils/certificate_flow.go:163
github.com/smallstep/cli/command/ssh.certificateAction
        github.com/smallstep/cli/command/ssh/certificate.go:273
go.step.sm/cli-utils/command.ActionFunc.func1
        go.step.sm/[email protected]/command/command.go:37
github.com/urfave/cli.HandleAction
        github.com/urfave/[email protected]/app.go:522
github.com/urfave/cli.Command.Run
        github.com/urfave/[email protected]/command.go:173
github.com/urfave/cli.(*App).RunAsSubcommand
        github.com/urfave/[email protected]/app.go:405
github.com/urfave/cli.Command.startApp
        github.com/urfave/[email protected]/command.go:372
github.com/urfave/cli.Command.Run
        github.com/urfave/[email protected]/command.go:102
github.com/urfave/cli.(*App).Run
        github.com/urfave/[email protected]/app.go:277
main.main
        ./main.go:113
runtime.main
        runtime/proc.go:250
runtime.goexit
        runtime/asm_amd64.s:1571

Additional Context

Apologies if this issue should have been filed on smallstep/cli instead.

I've set up an instance of step-ca behind caddy, with Caddy having a Let's Encrypt certificate, proxying over TLS to the step-ca instance for external access. I'm mostly interested in the SSH CA, getting assertions over OIDC. I'm aware of other issues such as #193 and #246 where some of the philosophy of why not to do this was discussed.

The root issue seems to be that --root is sometimes used as the trust store, with 1 or more certificate PEMs in it, but in other code paths it is used as the true root.

Best I can tell, the root is loaded at https://github.com/smallstep/cli/blob/master/utils/cautils/client.go#L49-L71, and the same ClientOption is used both for the Transport as it is for step operations. I think it should be possible to patch this to specify the ca-bundle separately.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

Answered by maraino

Aug 16, 2022

Let me put here the default behavior of step. The cli uses different ways to load its own root certificate, this is the list by priority:

On JWT tokens, we add the claim "sha" with the fingerprint of the root certificate, if present, step will use an "insecure" connection to get the root from step-ca (/root/<fingerprint>), and after verifying the fingerprint, it will create a secure connection with that root.
Flag --root
Environment variable STEP_ROOT
"root" property on $(step path)/defaults.json
Certificate file at $(step path)/certs/root_ca.crt

Now to your use case, assuming you're running this:

step ssh certificate ${FQDN} /etc/ssh/ssh_host_ecdsa_key.pub --host --sign --principal ${F…

View full answer

tashian · 2022-08-01T19:36:24Z

tashian
Aug 1, 2022
Collaborator

Normally step CLI doesn't trust the system trust store (which includes the entire Web PKI) for its connection to the CA. It only trusts the root it is configured with. This keeps the trust constrained to our own PKI.

For an X.509 CA, the cert renewal flows sometimes use mutual TLS. That's why we don't recommend putting it behind a layer 7 load balancer. But, this has been addressed with the new --mtls=false flag. And if you're just using the SSH CA then you will use the SSHPOP provisioner to renew host certs, and that's a different kind of renewal flow that doesn't rely on mTLS.

Some ways forward in the short term...

Only pass the Let's Encrypt root PEM that issued your leaf into --root
Or, have your caddy server get its cert from your step-ca instead of Let's Encrypt.
Or, Caddy has a layer 4 proxy plugin you could use to pass step-ca TCP traffic along directly.

As for this issue...

Maybe it would be nice if --root accepted multiple certs (--roots?).
Or, we could potentially add a flag (behind --insecure or --subtle) that get step to trust the system trust store

0 replies

achernya · 2022-08-01T20:24:08Z

achernya
Aug 1, 2022
Author

Thanks for the reply! I've tried the workarounds that you've listed:

If I pass in --root=/etc/ssl/certs/ISRG_Root_X1.pem, I get the error message

The requested resource could not be found. Please see the certificate authority logs for more info.

from step-cli. On step-ca, I see

time="2022-08-01T20:10:11Z" level=warning duration="61.09µs" duration-ns=61090 error="/root/96bcec06264976f37460779acf28c5a7cfe8a3c0aae11a8ffcee05c0bddf08c6 was not found: certificate with fingerprint 96bcec06264976f37460779acf28c5a7cfe8a3c0aae11a8ffcee05c0bddf08c6 was not found" fields.time="2022-08-01T20:10:11Z" method=GET name=ca path=/root/96bcec06264976f37460779acf28c5a7cfe8a3c0aae11a8ffcee05c0bddf08c6 protocol=HTTP/2.0 referer= remote-address=[some docker IP redacted] request-id=cbk368q346fc73c299g0 size=127 status=404 user-agent="Smallstep CLI/0.21.0 (linux/amd64)" user-id=

This appears to be the SHA-256 fingerprint of the ISRG X1 root:

# openssl x509 -in /etc/ssl/certs/ISRG_Root_X1.pem -noout -fingerprint -sha256
SHA256 Fingerprint=96:BC:EC:06:26:49:76:F3:74:60:77:9A:CF:28:C5:A7:CF:E8:A3:C0:AA:E1:1A:8F:FC:EE:05:C0:BD:DF:08:C6

which leads me to believe the observation I made before about --root sometimes being used as a trust store and sometimes as the true root from which issuance is requested to persist.

I suppose I could configure the CA endpoint to use Caddy ACME back to step-ca, but I was really hoping to have just have Let's Encrypt certificates in the Caddy endpoint.
Ideally, yes, the layer 4 plugin would be great. Unfortunately, I'm using caddy-docker-proxy which only supports Caddyfile-based configuration and mholt/caddy-l4 says "Since this app does not support Caddyfile".

I could probably work around this in my environment by having the step-ca port be exposed on the internal network, and have Caddy only for external ingress. It does however occur to me that the URL printed in step ssh login doesn't ever touch any CA endpoints, so the browser doesn't need to trust the step-ca root. So it's not actually important for that to be covered by the Web PKI.

For the suggestions you made,

I was personally thinking --tls-trust-store=, rather than --roots=, to avoid confusion between what is a root for what.
You could then easily make --tls-trust-store=system require --insecure or --subtle to auto-detect the system trust store.

0 replies

tashian · 2022-08-01T20:33:07Z

tashian
Aug 1, 2022
Collaborator

Oh! You might try appending the Let's Encrypt root PEM to your certs/root_ca.crt file on step-ca.

Alex, if you don't mind jumping into our Discord, we could debug this in real time: https://bit.ly/step-discord

0 replies

achernya · 2022-08-01T21:09:33Z

achernya
Aug 1, 2022
Author

As requested during our chat on Discord, here's the snippet (redacted) from the Caddyfile

*.example.com {
        @ca host ca.example.com
        handle {
                respond 404
        }
        handle @ca {
                reverse_proxy https://[docker-assigned-ip-of-step-ca]:9000 {
                        transport http {
                                tls_insecure_skip_verify
                        }
                }
        }       
}

I haven't yet tried to make Caddy verify the step-ca certificate, so tls_insecure_skip_verify is in use.

0 replies

maraino · 2022-08-16T01:35:16Z

maraino
Aug 16, 2022
Maintainer

Let me put here the default behavior of step. The cli uses different ways to load its own root certificate, this is the list by priority:

On JWT tokens, we add the claim "sha" with the fingerprint of the root certificate, if present, step will use an "insecure" connection to get the root from step-ca (/root/<fingerprint>), and after verifying the fingerprint, it will create a secure connection with that root.
Flag --root
Environment variable STEP_ROOT
"root" property on $(step path)/defaults.json
Certificate file at $(step path)/certs/root_ca.crt

Now to your use case, assuming you're running this:

step ssh certificate ${FQDN} /etc/ssh/ssh_host_ecdsa_key.pub --host --sign --principal ${FQDN} --principal ${HOSTNAME}

In most cases, it will get the root from 1, and it will properly get from the proxy the root because it won't check the TLS cert, but then it will fail to establish the connection because the proxy, caddy, in this case, is using a let's encrypt cert instead of a step-ca. And that "sha" from one is coming from a file, and in this case, it will probably be 4 or 5.

One solution is to create JWT tokens without the "sha", but that gets complicated, but I think we can do the following:

Add in ca.json the root of let's encrypt. The "root" property accepts both a string and an array, so we will add the let's encrypt root to it, so it will look like:

"root": ["/path/to/step/certs/root_ca.crt", "/path/to/ISRG_Root_X1.pem"]

In the client, change the defaults.json to use let's encrypt root:

"root": "/path/to/ISRG_Root_X1.pem"

With this, I'm trying to add the "sha" of the let's encrypt root, and because of 1 it will be present in step-ca, so it will be able to retrieve it. And with 2, the token created will have that "sha" and hopefully, everything will work.

@achernya @tashian I haven't tested this setup, so let me know if it works.

1 reply

maxx1e Oct 31, 2023

Hello, @maraino , any ideas on how to achieve the same in case this is a K8 deployment? How we can specify several root certs in the config if we generating the config for the helm?

tashian · 2022-08-16T17:32:15Z

tashian
Aug 16, 2022
Collaborator

Ahh, ok. @maraino @achernya I'm going to convert this issue into a discussion so we can have the notes for posterity.

0 replies

achernya · 2022-08-23T02:18:14Z

achernya
Aug 23, 2022
Author

Hello @maraino, @tashian! I just tested this by putting

        "root": ["/home/step/certs/root_ca.crt", "/etc/ssl/certs/ca-cert-ISRG_Root_X1.pem"],

in ca.json and running the aforementioned step ssh certificate command. The output was

✔ Would you like to overwrite /etc/ssh/ssh_host_ecdsa_key-cert.pub [y/n]: y
✔ Certificate: /etc/ssh/ssh_host_ecdsa_key-cert.pub

Which appears to work!

0 replies

Getting certificates from step-ca + a reverse proxy that uses a Web PKI certificate #1017

Uh oh!

achernya Aug 1, 2022

Steps to Reproduce

Your Environment

Expected Behavior

Actual Behavior

Additional Context

Contributing

Replies: 7 comments · 1 reply

Uh oh!

Uh oh!

tashian Aug 1, 2022 Collaborator

Uh oh!

achernya Aug 1, 2022 Author

Uh oh!

tashian Aug 1, 2022 Collaborator

Uh oh!

achernya Aug 1, 2022 Author

Uh oh!

Uh oh!

maraino Aug 16, 2022 Maintainer

Uh oh!

maxx1e Oct 31, 2023

Uh oh!

tashian Aug 16, 2022 Collaborator

Uh oh!

achernya Aug 23, 2022 Author

achernya
Aug 1, 2022

Replies: 7 comments 1 reply

tashian
Aug 1, 2022
Collaborator

achernya
Aug 1, 2022
Author

tashian
Aug 1, 2022
Collaborator

achernya
Aug 1, 2022
Author

maraino
Aug 16, 2022
Maintainer

tashian
Aug 16, 2022
Collaborator

achernya
Aug 23, 2022
Author