Can't get ACME Cloudflare challenge to succeed

[bitblt]

I'm trying to integrate Step CA into an existing framework that uses ACME DNS-01 challenges to issue server and wildcard certificates as needed for the consuming application. My approach was to set up Step CA and validate it was properly configured before I dropped it into the framework. I used step cli first as a base case for validation and then moved to the ACME clients. For ACME, I tested HTTP-01 followed by DNS-01 based on the document https://smallstep.com/docs/tutorials/acme-protocol-acme-clients. Testing was successful with HTTP-01 challenge but not DNS-01.

I decided to set up a repeatable, minimialist environment for troubleshooting. I set up a test environment with two nodes - the CA VM and a client VM running Ubuntu 24.04 and the latest patches directly from Ubuntu.com. On both VMs, I installed Step CLI, and I installed Step CA on the CA VM.

On the CA, I created the CA using --acme --remote-management and launched the CA from the command line. No other changes were made on the CA VM.

On the client VM, I also installed certbot, acme.sh, and configured Cloudflare DNS. I set Let's Encrypt as the default for acme.sh (acme.sh --set-default-ca --server letsencrypt) to keep testing consistent. I ran step ca bootstrap and step certificate install to establish the trust.

When I attempt to issue a certificate using the ACME clients and DNS-01 challenge, it failed during the DNS verification phase. The CA logs showed the same error for both ACME clients:
"error\":{\"type\":\"urn:ietf:params:acme:error:dns\",\"detail\":\"There was a problem with a DNS query during identifier validation\"}}.

Here are the command lines for the ACME clients:
certbot certonly --dns-cloudflare --dns-cloudflare-propagation-seconds 20 --dns-cloudflare-credentials ~provision/cloudflare.ini -d tcaclient.<mydomain> --server https://tstepca.<mydomain>/acme/acme/directory
acme.sh --issue -d tcaclient.<mydomain> --fullchain-file x.crt --key-file x.key --dns dns_cf --server https://tstepca<mydomain>/acme/acme/directory

If I drop the --server flag and make no other changes, I can successfully issue the certificate from Let's Encrypt.

What am I missing?

[hslatman]

Hey @bitblt, could you verify/show the DNS records that certbot sets during the request?

[bitblt]

@hslatman,
The following TXT record was created:
_acme-challenge.tcaclient.=0RUuTa1rN1SVcUVeOZTscx2ISbWTmAqs3g23iVtWhPM

I attached a file from the client side that contains the output from the certbot command with --verbose --debug-challenges and the /var/log/letsencrypt/letsencrypt.log. I also attached the output from step-ca with STEPDEBUG=1.

I did some light sanitizing of the files by replacing my actual domain name with . I also truncated both files after certbot received the error from step-ca but continued to send requests to step-ca to confirm the DNS validation was successful.

client.log
step-ca.log

[bitblt]

@hslatman, this was a self-inflicted wound. TL;DR my dev/test BIND9 split-horizon setup, by design, does not query my Cloudflare DNS instance; therefore, step-ca did not find the generated TXT record.

The VM for the CA was pointing to the internal dev (BIND9) DNS instance instead of the Clouldflare DNS. The dev DNS instance uses the same domain name but does not resolve split-horizon entries from my Cloudflare DNS instance. Everything worked with LE because their CA is external and went directly to Cloudflare DNS.

Short term, I configured the DNS resolver for step-ca to use Cloudflare. This will fix the DNS-01 challenge issue. However, HTTP-01 will break for internal services because Cloudflare is strictly for public-accessible services. I need to rearchitect the internal DNS to cover all of the use cases (DNS-01, HTTP-01, SSH certs, etc).