HTTP-01 IPv6 to IPv4 fallback not working properly

[cpu]

A user in IRC noticed that they were suffering HTTP-01 validation failures for a domain that previously worked. Investigating it appears the domain had an AAAA record and an A record but the AAAA address wasn't working. I expected the IPv6 to IPv4 fallback code would have masked this issue but looking at the validation records it did not, there is no addressTried, and the addressUsed is the v6 address:

    "validationRecord": [
      {
        "url": "http://xxxxx/.well-known/acme-challenge/XXXXXXXXXXXXX",
        "hostname": "XXXXX",
        "port": "80",
        "addressesResolved": [
          "92.XXX.XXX.XXX",
          "2001:XXXX:XXXX:XXXX::111"
        ],
        "addressUsed": "2001:XXXX:XXXX:XXXX::111",
        "addressesTried": null
      }
    ]

The VA logged:

HTTP request to http://xxxxx/.well-known/acme-challenge/XXXXXXXXXXXXX failed. err=[&url.Error{Op:"Get", URL:"http://xxxxx/.well-known/acme-challenge/XXXXXXXXXXXXX", Err:(*http.httpError)(0xc420c89260)}] errStr=[Get http://xxxxx/.well-known/acme-challenge/XXXXXXXXXXXXX: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)]

The root cause is the VA's HTTP-01 dialer wrapper is re-using the same underlying net.Dialer with an expended timeout between the initial and subsequent fallback connection.

[cpu]

One false-positive for this issue I've seen so far is a host with an A and AAAA record failing an HTTP-01 challenge because the webserver on the AAAA IP returned a 404 while the A webserver had the correct webroot configured. This doesn't meet the conditions for the retry because the failure is at the HTTP challenge validation level and not the IP connectivity level.

[jsha]

That seems like a situation where it's consistent with our other behavior to treat the validation as failed due to a misconfigured server.

[cpu]

@jsha I agree, that's why I called it a false positive.

[jangrewe]

This also happened to me, although with dehydrated.
The issue for me was that the IPv6 address in my AAAA wasn't properly routed by my ISP, but the IPv4 still worked fine.

Processing pokemap.berlin with alternative names: www.pokemap.berlin dev.pokemap.berlin
 + Checking domain name(s) of existing cert... changed!
 + Domain name(s) are not matching!
 + Names in old certificate: dev.pokemap.berlin pmg.faked.org pokemap.berlin www.pokemap.berlin
 + Configured names: dev.pokemap.berlin pokemap.berlin www.pokemap.berlin
 + Forcing renew.
 + Checking expire date of existing cert...
 + Valid till Jun 14 22:01:00 2017 GMT Certificate will expire
(Less than 30 days). Renewing!
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting challenge for pokemap.berlin...
 + Requesting challenge for www.pokemap.berlin...
 + Requesting challenge for dev.pokemap.berlin...
 + Responding to challenge for pokemap.berlin...
 + Responding to challenge for www.pokemap.berlin...
 + Responding to challenge for dev.pokemap.berlin...
ERROR: Challenge is invalid! (returned: invalid) (result: {
  "type": "http-01",
  "status": "invalid",
  "error": {
    "type": "urn:acme:error:connection",
    "detail": "Could not connect to dev.pokemap.berlin",
    "status": 400
  },
  "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/aL5EsF2NuL8zBQHzgBw8qFxKFmnrlt91RdfNxm30lAk/1210320188",
  "token": "MQyfKns3ATGGPMS-pKSieBlrWpjE86FIj2pnuYcAFfo",
  "keyAuthorization": "MQyfKns3ATGGPMS-pKSieBlrWpjE86FIj2pnuYcAFfo.ymn7rrjFsLBQUTzWYgdoacDjsIe-B36saKrAYkAh2Tk",
  "validationRecord": [
    {
      "url": "http://dev.pokemap.berlin/.well-known/acme-challenge/MQyfKns3ATGGPMS-pKSieBlrWpjE86FIj2pnuYcAFfo",
      "hostname": "dev.pokemap.berlin",
      "port": "80",
      "addressesResolved": [
        "87.128.111.190",
        "2003:a:37f:ef4f::"
      ],
      "addressUsed": "2003:a:37f:ef4f::",
      "addressesTried": []
    }
  ]
})

[sahsanu]

Regarding this ipv6 preference https://community.letsencrypt.org/t/certbot-ipv6-address-on-domain-misconfigured-and-challenges-fail-prefer-ipv6/34626

In this case the user have a domain with both records, A and AAAA, but the web server is only configured for ipv4, the ipv6 reachs the web server but not the right virtualhost, in this case, obviously, the challenge fails. I don't know whether it is worth to fallback to ipv4 in this case.

[cpu]

@sahsanu - Thanks for commenting. That therad is the same one I mentioned earlier in this thread as a false positive (I should have linked to it, apologies). In this case I don't expect a fallback and everything appears to be working as intended.

[jsha]

Got it! I didn't understand that part. :-)

[sahsanu]

Just another "false positive" https://community.letsencrypt.org/t/404-on-well-known-acme-challenge-but-accessable-from-browser/34730

I can understand the decision to prefer AAAA if both records are available for a domain, but sadly we are still living in an ipv4 world. The use case for ipv6 is very limited as majority of domestic ISPs doesn't provide an ipv6 to their customers. Also, there are a lot of people getting a dedicated, vps and shared hosting and theirs hosters auto conf the DNS providing both ips (ipv4 and ipv6) but people doesn't care about ipv6 (yet) and don't configure their services tu use it properly so I'm afraid we will see a lot of cases with this "false positive" issue ;).

[cpu]

so I'm afraid we will see a lot of cases with this "false positive" issue ;).

I will be posting an announcement in the community forum about the IPv6 preference today. Hopefully that will help clear up the confusion.

Ultimately if you run a website that publishes an AAAA address that doesn't work you're going to run into problems sooner or later!

[jsha]

From the entry in https://community.letsencrypt.org/t/unable-to-update-challenge-the-challenge-is-not-pending/35118/3, looking in the logs, it appears that the problem was a timeout. So one possibility is that the fallback doesn't happen correctly on timeouts, perhaps because the first try uses up all of the available time?

[cpu]

@jsha That's indeed a possibility. We didn't increase the timeout to accommodate making two back-to-back requests.

[mhofman]

Same problem here. The IPv6 address times out since the HTTP server isn't listening on that interface, and the IPv4 is never checked ("addressesTried": []).

Why not run both in parallel and let the faster one win? If you want to give IPv6 a preference, you can start that check a second before the IPv4.