Trying to remove tower from client causes rpc error: code = Unknown desc = tower has unacked updates

[iangregsondev]

Background

Trying to remove a tower wtclient but it gives me an issue

Your environment

• version of lnd : lnd version 0.10.99-beta commit=clock/v1.0.0-106-gc1ef5bb908606343d2636c8cd345169e064bdc91
• which operating system (uname -a on *Nix) : ubuntu (in container)
• version of btcd, bitcoind, or other backend bitcoins 0.19.1
• any other relevant environment details

Steps to reproduce

I have a wtclient setup to the wrong address, in fact in the logs it is telling me it can’t dial.
I want to remove it but it doesn’t let me

bash-5.0# lncli --macaroonpath /lnd/chain/bitcoin/testnet/admin.macaroon   --tlscertpath /shared/tls.cert wtclient remove 038f7f36689b9d7274702f5bce3a5d8bc4596d4894d9985c5203604fff4daef425
[lncli] rpc error: code = Unknown desc = tower has unacked updates

this is what the towers command retuns

   "towers": [
        {
            "pubkey": "038f7f36689b9d7274702f5bce3a5d8bc4596d4894d9985c5203604fff4daef425",
            "addresses": [
                "157.245.68.69:9911"
            ],
            "active_session_candidate": true,
            "num_sessions": 1,
            "sessions": [
            ]
        },

Expected behaviour

Allow me to remove a tower or allow some way to FORCE removal

Actual behaviour

It won't let me, please see the issue above.

[wpaulino]

I believe removing a tower when a backup hasn't been fully processed isn't safe at the moment because it won't be replayed to any other existing towers cc @cfromknecht.

I have a wtclient setup to the wrong address, in fact in the logs it is telling me it can’t dial.
I want to remove it but it doesn’t let me

If you just want to modify the address, then you can specify it in the lncli wtclient remove command, and add the new one with the lncli wtclient add command.

[iangregsondev]

Hi @wpaulino thanks for the response.

That is true, I did want to modify the address but I wasn't aware I could.

I have just tried the following but it did not work

lncli --macaroonpath /lnd/chain/bitcoin/testnet/admin.macaroon --tlscertpath /shared/tls.cert wtclient remove 038f7f36689b9d7274702f5bce3a5d8bc4596d4894d9985c5203604fff4daef425 021f6fddf84ccaf1a87c99634770e2b7fb25eac890f8f4e5501abbf1a60b25d4fc

so the address starting with 038 is what I wish to remove OR change with the address starting with 02a

I can easily add a new one which I can do, but removing OR changing the one mentioned above doesn't work.

Maybe I have the syntax of "wtclient remove" wrong

Can you confirm ?

Thanks

[wpaulino]

Ah, I thought you were referring to a network address. If you want to modify the public key, then you had the correct command the first time which resulted in the error (lncli wtclient remove PUBLIC_KEY). Are you able to add the new tower without needing to remove the stale one?

[iangregsondev]

@wpaulino sure - that's what I have done, so I have a new watchtower client connecting to the correct URL. All good.

But the old one still persists and although it's not causing any issues, it does output errors of WTC into the logs.

I just thought it would be possible to force remove it.

[Roasbeef]

I think this has bene resolved?

[wpaulino]

The root issue hasn't, which is replaying any pending backups to new towers. That would make the removal of a stale tower possible.

[djkazic]

Is there any way to force remove a watchtower? I've got pending backups that won't go through, so I get the same error here. In the meantime I've added a new watchtower but I still see RPC log spam from the unremovable old WT.

[djkazic]

FWIW when I added a new watchtower it uploaded backup states without an issue. I suspect the underlying issue is that when the channel.db was corrupted by a power failure, I restored the most up to date version of it that wasn't corrupt. Could this "travel back in time" cause this broken state for watchtowers?

[guggero]

You should never restore the channel.db file from a backup! You'll not only put your funds at risk but cause all sorts of problems. The watchtower not being able to send backups just being one of them.

[djkazic]

I didn't want to lose all my channels. I saved 12 of them by doing so, and some others force closed.

[djkazic]

In addition, in this case I confirmed no payments had been routed from the time of the backup to the time of corruption.

Considering that, I don't think there was substantial risk versus guaranteed closing all channels (thus having to pay two on chain tx fees -- one to close and one to reopen after restoring from an SCB).