v24.11rc4: Calling `connect` fails with `Unable to connect, no address known for peer`

[JssDWt]

Issue and Steps to Reproduce

• node l1 and l2 are connected via a channel
• node l2 restarts
• node l2 calls connect to l1 with explicit host and port

Connect fails with Unable to connect, no address known for peer, while the connection details were in the request.

getinfo output

v24.11rc4

[JssDWt]

I can fix my integration tests by omitting the connect call after restart. The error seems wrong to me though. Possible change that caused it: 23dc10c.

[rustyrussell]

OK, this is fascinating. Here's what's happening:

1. On restart, l2's gossipd aggressively tries to connect to l1 (it's the only node it knows about, and it wants gossip). It doesn't have an address though, so as a last-ditch effort, it tries to do a DNS lookup.
2. You tell it to connect, with an address. It's already connecting, so it simply appends to the addresses.
3. The DNS lookup doesn't work (those servers don't answer, and anyway we're not on mainnet, so we shouldn't even try)
4. Because that was our "last ditch" effort, we don't re-check to see if any addresses have been added.

Of course, l1 knows how to connect to l2, and eventually it does.

You can simply ask l1 to connect to l2 not vice-versa and it will work. But I shall queue a few fixes for this case next release.

[grubles]

I think this might be preventing channels from being opened? The connection at some point drops, then when manually trying to reconnect withlightning-cli connect <pubkey> there is some kind of error reconnect loop that occurs. This is with v24.11 release BTW, not a release candidate.

Here is a log snippet of this happening:

2024-12-09T18:50:04.442Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Connected out, starting crypto                                                                       
2024-12-09T18:50:04.462Z DEBUG   036e680823a129ac9a2eeb8c43514a5b6cb09ae28e3e698a0e34aaa8865d6cc344-hsmd: Got WIRE_HSMD_ECDH_REQ                                                                                   
2024-12-09T18:50:04.462Z DEBUG   hsmd: Client: Received message 1 from client                                                                                                                                      
2024-12-09T18:50:04.462Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Connect OUT                                                                                          
2024-12-09T18:50:04.462Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: peer_out WIRE_INIT                                                                                   
2024-12-09T18:50:04.463Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: peer_in WIRE_INIT                                                                                    
2024-12-09T18:50:04.463Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Handed peer, entering loop                                                                           
2024-12-09T18:50:04.463Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: peer_out WIRE_GOSSIP_TIMESTAMP_FILTER                                                                
2024-12-09T18:50:04.463Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-gossipd: seeker: chosen as startup peer                                                                        
2024-12-09T18:50:04.463Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-gossipd: seeker: starting gossip (EVERYTHING)                                                                  
2024-12-09T18:50:04.463Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-chan#3: Peer has reconnected, state CHANNELD_AWAITING_LOCKIN: connecting subd                                  
2024-12-09T18:50:04.465Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-channeld-chan#3: pid 171612, msgfd 85                                                                          
2024-12-09T18:50:04.465Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-chan#3: Already have funding locked in                                                                         
2024-12-09T18:50:04.465Z UNUSUAL 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-chan#3: Ignoring fee limits!                                                                                   
2024-12-09T18:50:04.465Z DEBUG   hsmd: new_client: 3                                                                                                                                                               
2024-12-09T18:50:04.465Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-channeld-chan#3: option_static_remotekey = 1, option_anchor_outputs = 0 option_anchors_zero_fee_htlc_tx = 1    
2024-12-09T18:50:04.465Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-channeld-chan#3: init LOCAL: remote_per_commit = 028ac29a7a2256e00be60fe343b28c0cd15eff7b129c9b16edd7d298da03b3
a3f6, old_remote_per_commit = 028ac29a7a2256e00be60fe343b28c0cd15eff7b129c9b16edd7d298da03b3a3f6 next_idx_local = 1 next_idx_remote = 1 revocations_received = 0 feerates { SENT_ADD_ACK_REVOCATION:1250 } range 1-
4294967295 blockheights { SENT_ADD_ACK_REVOCATION:0 }, our current 225659                                                                                                                                          
2024-12-09T18:50:04.466Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-hsmd: Got WIRE_HSMD_GET_PER_COMMITMENT_POINT                                                                   
2024-12-09T18:50:04.466Z DEBUG   hsmd: Client: Received message 18 from client                                                                                                                                     
2024-12-09T18:50:04.466Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-hsmd: Got WIRE_HSMD_GET_PER_COMMITMENT_POINT                                                                   
2024-12-09T18:50:04.466Z DEBUG   hsmd: Client: Received message 18 from client                                                                                                                                     
2024-12-09T18:50:04.466Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-channeld-chan#3: peer_out WIRE_CHANNEL_REESTABLISH                                                             
2024-12-09T18:50:04.466Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-channeld-chan#3: billboard: Sent reestablish, waiting for theirs                                               
2024-12-09T18:50:04.468Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Activating for message WIRE_CHANNEL_REESTABLISH                                                      
2024-12-09T18:50:04.468Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-lightningd: Telling connectd to send error 00111c675c099a12ff9f34256bcfd6d4030939b17889c52940437f9b7a069314b8e3
00226368616e6e656c20696e20737461746520434c4f53494e47445f434f4d504c455445                                                                                                                                           
2024-12-09T18:50:04.468Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: peer_out WIRE_CHANNEL_REESTABLISH                                                                    
2024-12-09T18:50:04.468Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: peer_out WIRE_ERROR                                                                                  
2024-12-09T18:50:04.468Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: disconnect_peer                                                                                      
2024-12-09T18:50:04.468Z DEBUG   connectd: drain_peer                                                                                                                                                              
2024-12-09T18:50:04.468Z DEBUG   connectd: drain_peer draining subd!                                                                                                                                               
2024-12-09T18:50:04.468Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-lightningd: peer_disconnect_done                                                                               
2024-12-09T18:50:04.468Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds                                                                    
2024-12-09T18:50:04.468Z DEBUG   plugin-funder: Cleaning up inflights for peer id 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923                                                               
2024-12-09T18:50:08.494Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Connected out, starting crypto                                                                       
2024-12-09T18:50:08.504Z DEBUG   gossipd: seeker: no peers, waiting                                                                                                                                                
2024-12-09T18:50:08.504Z DEBUG   gossipd: seeker: need more peers for gossip (have 0)                                                                                                                              
2024-12-09T18:50:08.504Z DEBUG   lightningd: attempting connection to 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923 for additional gossip                                                     
2024-12-09T18:50:08.505Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Adding 0 addresses to important peer                                                                 
2024-12-09T18:50:08.514Z DEBUG   036e680823a129ac9a2eeb8c43514a5b6cb09ae28e3e698a0e34aaa8865d6cc344-hsmd: Got WIRE_HSMD_ECDH_REQ                                                                                   
2024-12-09T18:50:08.514Z DEBUG   hsmd: Client: Received message 1 from client                                                                                                                                      
2024-12-09T18:50:08.515Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Connect OUT                                                                                          
2024-12-09T18:50:08.515Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: peer_out WIRE_INIT                                                                                   
2024-12-09T18:50:08.621Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: peer_in WIRE_INIT                                                                                    
2024-12-09T18:50:08.621Z INFO    035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-chan#3: Peer transient failure in CHANNELD_AWAITING_LOCKIN: Disconnected                                       

This is a brand new signet channel with an LND v0.18.3-beta node and seems stuck at CHANNELD_AWAITING_LOCKIN.

      {
         "peer_id": "035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923",
         "connected": false,
         "state": "CHANNELD_AWAITING_LOCKIN",
         "channel_id": "bc7b355dbb04076c370e64e92ecc5d5e9c4f3eebbd67e3a396526439bdc55fdb",
         "short_channel_id": "225649x270x0",
         "our_amount_msat": 1000000000000,
         "amount_msat": 1000000000000,
         "funding_txid": "db5fc5bd39645296a3e367bdeb3e4f9c5e5dcc2ee9640e376c0704bb5d357bbc",
         "funding_output": 0
      }

[grubles]

Still experiencing this with a direct channel to an LND node running on the same machine:

2025-01-06T15:30:31.449Z DEBUG   lightningd: attempting connection to 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923 for additional gossip
2025-01-06T15:30:31.449Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Adding 0 addresses to important peer
2025-01-06T15:30:31.449Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:31.449Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:32.037Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:32.037Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:32.428Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:32.428Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:32.680Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:32.681Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:33.468Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:33.469Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:33.471Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:33.471Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:33.935Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:33.935Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:34.125Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:34.125Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:35.496Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:35.497Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:35.623Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:35.623Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:35.751Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:35.751Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:35.892Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:35.892Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds

Only manually doing a lightning-cli connect <peer id> works to reconnect.

[grubles]

Maybe this is a different issue considering the original problem was related to connect not working to reconnect?

[rustyrussell]

Yes, this is a different problem. I'm going to defer this to next release, unfortunately. There are a few issues here:

1. We shouldn't be disconnecting after sending ERROR; it disrupts other channels with the same peer unnecessarily. Changing this will no doubt break a pile of tests which assumed this behavior though.
2. connectd itself (not just the db) should remember the last successful connection address. The db is useful for restarts, but in the no-restart case it doesn't really help.
3. Original issue: we shouldn't do DNS lookups if we're not on mainnet. And we should check if someone has given us an address in the meantime, after we fail DNS (though that's a corner case).

[northben]

is there any workaround for this issue? lightning-cli connect doesn't work for me, I get the same "Unable to connect, no address known for peer".

[vincenzopalazzo]

@northben the work around is specify the address of the peer, or let the peer (if you know) connect to you and then you are fine.

Sorry for requiring this hack :/ I described a similar problem here #7995

[grubles]

Letting the peer connect to you doesn't work if you're firewalled, so if CLN can't connect out then the channel is stuck disconnected.