Can't get ACME Cloudflare challenge to succeed

[bitblt]

I'm trying to integrate Step CA into an existing framework that uses ACME DNS-01 challenges to issue server and wildcard certificates as needed for the consuming application. My approach was to set up Step CA and validate it was properly configured before I dropped it into the framework. I used step cli first as a base case for validation and then moved to the ACME clients. For ACME, I tested HTTP-01 followed by DNS-01 based on the document https://smallstep.com/docs/tutorials/acme-protocol-acme-clients. Testing was successful with HTTP-01 challenge but not DNS-01.

I decided to set up a repeatable, minimialist environment for troubleshooting. I set up a test environment with two nodes - the CA VM and a client VM running Ubuntu 24.04 and the latest patches directly from Ubuntu.com. On both VMs, I installed Step CLI, and I installed Step CA on the CA VM.

On the CA, I created the CA using --acme --remote-management and launched the CA from the command line. No other changes were made on the CA VM.

On the client VM, I also installed certbot, acme.sh, and configured Cloudflare DNS. I set Let's Encrypt as the default for acme.sh (acme.sh --set-default-ca --server letsencrypt) to keep testing consistent. I ran step ca bootstrap and step certificate install to establish the trust.

When I attempt to issue a certificate using the ACME clients and DNS-01 challenge, it failed during the DNS verification phase. The CA logs showed the same error for both ACME clients:
"error\":{\"type\":\"urn:ietf:params:acme:error:dns\",\"detail\":\"There was a problem with a DNS query during identifier validation\"}}.

Here are the command lines for the ACME clients:
certbot certonly --dns-cloudflare --dns-cloudflare-propagation-seconds 20 --dns-cloudflare-credentials ~provision/cloudflare.ini -d tcaclient.<mydomain> --server https://tstepca.<mydomain>/acme/acme/directory
acme.sh --issue -d tcaclient.<mydomain> --fullchain-file x.crt --key-file x.key --dns dns_cf --server https://tstepca<mydomain>/acme/acme/directory

If I drop the --server flag and make no other changes, I can successfully issue the certificate from Let's Encrypt.

What am I missing?

[hslatman]

Hey @bitblt, could you verify/show the DNS records that certbot sets during the request?

[bitblt]

@hslatman,
The following TXT record was created:
_acme-challenge.tcaclient.=0RUuTa1rN1SVcUVeOZTscx2ISbWTmAqs3g23iVtWhPM

I attached a file from the client side that contains the output from the certbot command with --verbose --debug-challenges and the /var/log/letsencrypt/letsencrypt.log. I also attached the output from step-ca with STEPDEBUG=1.

I did some light sanitizing of the files by replacing my actual domain name with . I also truncated both files after certbot received the error from step-ca but continued to send requests to step-ca to confirm the DNS validation was successful.

client.log
step-ca.log