Route53 DNS01 cannot process due to refused error

[nmiculinic]

Describe the bug:
In kubernetes I have cert-manager providing TLS certificated. It correctly created the TXT entry on the route53, but it cannot fetch it for some reason.

Expected behaviour:
Correctly fetch the _acme-challenge.ascalia.io. TXT entry which has successfuly setup in an earlier step

A concise description of what you expected to happen.

Steps to reproduce the bug:

• install cert-manager via helm
• create needed secrets for accessing route53
• attempt to create cerficate

Anything else we need to know?:

The route53 credentials have full route53 admin permissions

helm setup:

fullnameOverride: cert-manager
securityContext:
  enabled: true
ingressShim:
  defaultIssuerName: kraken
  defaultIssuerKind: ClusterIssuer
  defaultACMEChallengeType:	dns01
  defaultACMEDNS01ChallengeProvider: route53
webhook:
  enabled: false

Certificate in question:

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: wild-ascalia-io
  namespace: ingress
spec:
  acme:
    config:
      - dns01:
          provider: route53
        domains:
          - '*.ascalia.io'
  dnsNames:
    - '*.ascalia.io'
  issuerRef:
    kind: ClusterIssuer
    name: kraken
  secretName: 'wild-ascalia-io'

Logs:

Alias tip: kl cert-manager-656476c4dd-28sl5
I0502 13:15:31.789219       1 start.go:81] starting cert-manager v0.6.2 (revision f5e1477bd7ced69e53a233484905fea16bf4102f)
I0502 13:15:31.790877       1 controller.go:141] Using the following nameservers for DNS01 checks: [10.233.0.3:53]
I0502 13:15:31.791953       1 leaderelection.go:193] attempting to acquire leader lease  ingress/cert-manager-controller...
I0502 13:16:41.249629       1 leaderelection.go:202] successfully acquired lease ingress/cert-manager-controller
I0502 13:16:41.250088       1 controller.go:82] Starting certificates controller
I0502 13:16:41.250208       1 controller.go:82] Starting clusterissuers controller
I0502 13:16:41.254078       1 metrics.go:145] Listening on http://0.0.0.0:9402
I0502 13:16:41.254239       1 controller.go:82] Starting issuers controller
I0502 13:16:41.254212       1 controller.go:82] Starting ingress-shim controller
I0502 13:16:41.257175       1 controller.go:82] Starting orders controller
I0502 13:16:41.257534       1 controller.go:82] Starting challenges controller
I0502 13:16:41.353258       1 controller.go:145] certificates controller: syncing item 'ingress/wild-ascalia-io'
I0502 13:16:41.353410       1 controller.go:141] clusterissuers controller: syncing item 'kraken'
I0502 13:16:41.353793       1 setup.go:149] Skipping re-verifying ACME account as cached registration details look sufficient.
I0502 13:16:41.353819       1 controller.go:147] clusterissuers controller: Finished processing work item "kraken"
I0502 13:16:41.353902       1 issue.go:154] Order ingress/wild-ascalia-io-2419620338 is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate.
I0502 13:16:41.353931       1 controller.go:151] certificates controller: Finished processing work item "ingress/wild-ascalia-io"
I0502 13:16:41.357468       1 controller.go:173] ingress-shim controller: syncing item 'kube-system/prom-grafana'
I0502 13:16:41.357487       1 sync.go:64] Not syncing ingress kube-system/prom-grafana as it does not contain necessary annotations
I0502 13:16:41.357491       1 controller.go:179] ingress-shim controller: Finished processing work item "kube-system/prom-grafana"
I0502 13:16:41.357495       1 controller.go:173] ingress-shim controller: syncing item 'kube-system/prom-prometheus-operator-alertmanager'
I0502 13:16:41.357504       1 controller.go:173] ingress-shim controller: syncing item 'kube-system/prom-prometheus-operator-prometheus'
I0502 13:16:41.357512       1 sync.go:64] Not syncing ingress kube-system/prom-prometheus-operator-prometheus as it does not contain necessary annotations
I0502 13:16:41.357517       1 controller.go:179] ingress-shim controller: Finished processing work item "kube-system/prom-prometheus-operator-prometheus"
I0502 13:16:41.357528       1 controller.go:173] ingress-shim controller: syncing item 'kube-system/kubernetes-dashboard'
I0502 13:16:41.357545       1 sync.go:64] Not syncing ingress kube-system/kubernetes-dashboard as it does not contain necessary annotations
I0502 13:16:41.357550       1 controller.go:179] ingress-shim controller: Finished processing work item "kube-system/kubernetes-dashboard"
I0502 13:16:41.357512       1 sync.go:64] Not syncing ingress kube-system/prom-prometheus-operator-alertmanager as it does not contain necessary annotations
I0502 13:16:41.357584       1 controller.go:179] ingress-shim controller: Finished processing work item "kube-system/prom-prometheus-operator-alertmanager"
I0502 13:16:41.357530       1 controller.go:173] ingress-shim controller: syncing item 'kube-system/elk-kibana'
I0502 13:16:41.357622       1 sync.go:64] Not syncing ingress kube-system/elk-kibana as it does not contain necessary annotations
I0502 13:16:41.357628       1 controller.go:179] ingress-shim controller: Finished processing work item "kube-system/elk-kibana"
I0502 13:16:41.357728       1 controller.go:183] orders controller: syncing item 'ingress/wild-ascalia-io-2419620338'
I0502 13:16:41.357746       1 controller.go:205] challenges controller: syncing item 'ingress/wild-ascalia-io-2419620338-0'
I0502 13:16:41.357906       1 dns.go:110] Checking DNS propagation for "ascalia.io" using name servers: [10.233.0.3:53]
I0502 13:16:41.357928       1 sync.go:274] Need to create 0 challenges
I0502 13:16:41.357939       1 sync.go:323] Waiting for all challenges for order "wild-ascalia-io-2419620338" to enter 'valid' state
I0502 13:16:41.357974       1 controller.go:189] orders controller: Finished processing work item "ingress/wild-ascalia-io-2419620338"
E0502 13:16:41.389396       1 controller.go:207] challenges controller: Re-queuing item "ingress/wild-ascalia-io-2419620338-0" due to error processing: NS ns-1024.awsdns-00.org.:53 returned REFUSED for _acme-challenge.ascalia.io.
I0502 13:16:46.318806       1 controller.go:141] clusterissuers controller: syncing item 'kraken'
I0502 13:16:46.319024       1 setup.go:149] Skipping re-verifying ACME account as cached registration details look sufficient.
I0502 13:16:46.319048       1 controller.go:147] clusterissuers controller: Finished processing work item "kraken"
I0502 13:16:46.389597       1 controller.go:205] challenges controller: syncing item 'ingress/wild-ascalia-io-2419620338-0'
I0502 13:16:46.389758       1 dns.go:110] Checking DNS propagation for "ascalia.io" using name servers: [10.233.0.3:53]
E0502 13:16:46.405789       1 controller.go:207] challenges controller: Re-queuing item "ingress/wild-ascalia-io-2419620338-0" due to error processing: NS ns-1024.awsdns-00.org.:53 returned REFUSED for _acme-challenge.ascalia.io.
I0502 13:16:56.406051       1 controller.go:205] challenges controller: syncing item 'ingress/wild-ascalia-io-2419620338-0'
I0502 13:16:56.406239       1 dns.go:110] Checking DNS propagation for "ascalia.io" using name servers: [10.233.0.3:53]
E0502 13:16:56.432658       1 controller.go:207] challenges controller: Re-queuing item "ingress/wild-ascalia-io-2419620338-0" due to error processing: NS ns-1024.awsdns-00.org.:53 returned REFUSED for _acme-challenge.ascalia.io.
I0502 13:17:16.432842       1 controller.go:205] challenges controller: syncing item 'ingress/wild-ascalia-io-2419620338-0'
I0502 13:17:16.433023       1 dns.go:110] Checking DNS propagation for "ascalia.io" using name servers: [10.233.0.3:53]
E0502 13:17:16.450047       1 controller.go:207] challenges controller: Re-queuing item "ingress/wild-ascalia-io-2419620338-0" due to error processing: NS ns-1024.awsdns-00.org.:53 returned REFUSED for _acme-challenge.ascalia.io.
I0502 13:17:56.450226       1 controller.go:205] challenges controller: syncing item 'ingress/wild-ascalia-io-2419620338-0'
I0502 13:17:56.450477       1 dns.go:110] Checking DNS propagation for "ascalia.io" using name servers: [10.233.0.3:53]
E0502 13:17:56.467009       1 controller.go:207] challenges controller: Re-queuing item "ingress/wild-ascalia-io-2419620338-0" due to error processing: NS ns-1024.awsdns-00.org.:53 returned REFUSED for _acme-challenge.ascalia.io.
I0502 13:19:16.467194       1 controller.go:205] challenges controller: syncing item 'ingress/wild-ascalia-io-2419620338-0'
I0502 13:19:16.467374       1 dns.go:110] Checking DNS propagation for "ascalia.io" using name servers: [10.233.0.3:53]
E0502 13:19:16.487313       1 controller.go:207] challenges controller: Re-queuing item "ingress/wild-ascalia-io-2419620338-0" due to error processing: NS ns-1536.awsdns-00.co.uk.:53 returned REFUSED for _acme-challenge.ascalia.io.
I0502 13:21:56.487496       1 controller.go:205] challenges controller: syncing item 'ingress/wild-ascalia-io-2419620338-0'
I0502 13:21:56.487668       1 dns.go:110] Checking DNS propagation for "ascalia.io" using name servers: [10.233.0.3:53]
E0502 13:21:56.505097       1 controller.go:207] challenges controller: Re-queuing item "ingress/wild-ascalia-io-2419620338-0" due to error processing: NS ns-1024.awsdns-00.org.:53 returned REFUSED for _acme-challenge.ascalia.io.
I0502 13:27:16.505337       1 controller.go:205] challenges controller: syncing item 'ingress/wild-ascalia-io-2419620338-0'
I0502 13:27:16.505516       1 dns.go:110] Checking DNS propagation for "ascalia.io" using name servers: [10.233.0.3:53]
E0502 13:27:16.523518       1 controller.go:207] challenges controller: Re-queuing item "ingress/wild-ascalia-io-2419620338-0" due to error processing: NS ns-1024.awsdns-00.org.:53 returned REFUSED for _acme-challenge.ascalia.io.
I0502 13:37:56.523716       1 controller.go:205] challenges controller: syncing item 'ingress/wild-ascalia-io-2419620338-0'
I0502 13:37:56.523901       1 dns.go:110] Checking DNS propagation for "ascalia.io" using name servers: [10.233.0.3:53]
E0502 13:37:56.543110       1 controller.go:207] challenges controller: Re-queuing item "ingress/wild-ascalia-io-2419620338-0" due to error processing: NS ns-1024.awsdns-00.org.:53 returned REFUSED for _acme-challenge.ascalia.io.

There exists a private zone ascalia.io, though I see cert-manager create the DNS TXT entry in the proper public zone.

Environment details::

• Kubernetes version (e.g. v1.10.2): v1.13.5
• Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): AWS
• cert-manager version (e.g. v0.4.0): v0.6.2
• Install method (e.g. helm or static manifests): helm

/kind bug

[wiltonfelix]

hi @nmiculinic , this is a problem in k8s internal cluster, to solve just add to your cert-manager config yaml this:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: cert-manager
  namespace: "cert-manager"
  labels:
    app: cert-manager
    chart: cert-manager-v0.7.2
    release: cert-manager
    heritage: Tiller
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cert-manager
      release: cert-manager
  template:
    metadata:
      labels:
        app: cert-manager
        release: cert-manager
      annotations:
        prometheus.io/path: "/metrics"
        prometheus.io/scrape: 'true'
        prometheus.io/port: '9402'
    spec:
      serviceAccountName: cert-manager
      containers:
        - name: cert-manager
          image: "quay.io/jetstack/cert-manager-controller:v0.7.2"
          imagePullPolicy: IfNotPresent
          args:
          - --cluster-resource-namespace=$(POD_NAMESPACE)
          - --leader-election-namespace=$(POD_NAMESPACE)
          - --dns01-recursive-nameservers="8.8.8.8:53"
          - --dns01-recursive-nameservers-only
          ports:
          - containerPort: 9402
          env:
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          resources:
            requests:
              cpu: 10m
              memory: 32Mi

or in helm values config:

extraArg:
--dns01-recursive-nameservers-only
--dns01-self-check-nameservers=8.8.8.8:53

for me this is an information problem in the cert-manager documentation, but I will open a PR for this.

[PabloCastellano]

@wiltonfelix Thanks! I've noticed that you have used --dns01-self-check-nameservers=8.8.8.8:53 in the helm example and --dns01-recursive-nameservers="8.8.8.8:53" in the manifest. Is it a typo?

Also, could you edit your comment and add markdown format to it by using triple backticks? (check this)

[TheKangaroo]

@PabloCastellano, --dns01-self-check-nameservers is deprecated in favor of --dns01-recursive-nameservers, but as of 0.8.0 both versions still work.

[retest-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[retest-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle rotten
/remove-lifecycle stale

[retest-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to jetstack.
/close

[jetstack-bot]

@retest-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to jetstack.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[chrislovecnm]

/reopen

Do we have documentation around this problem?

[jetstack-bot]

@chrislovecnm: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Do we have documentation around this problem?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[chrislovecnm]

#1627 (comment) fixed the problem on EKS