Lnd crashing when watchtower client active

tokriha · September 16, 2021, 4:11pm

Background

Hey guys!

When configuring a watchtower client on my umbrel node, I run into issues with lnd not starting. The last thing the lnd log prints is: unable to create server: tower not found

Previously, I have tried to add some watchtowers through lncli and I’m not sure if something went wrong there. I moved/backed up the lnd/data/watchtower/bitcoin/mainnet/watchtower.db and let it be created on a lnd reboot, so I’m assuming the db should be empty and there is no other location where the active watchtowers are tracked.

I’d appreciate any help a lot, I’m a bit uneasy to run my node without a watchtower.

Your environment

Umbrel 0.4.2
On what device / OS you are running Umbrel: Raspberry Pi
How is your Umbrel connected to the Internet: Ethernet
What type of drive you are using: SSD
version of lnd: v0.13.1-beta

Steps to reproduce

In the lnd.conf, I have activated the watchtower and lnd restarts without any issues.

[watchtower]
watchtower.active=1

When trying to activate the client with

[wtclient]
wtclient.active=1

and restarting lnd (docker-compose restart lnd), I get the following lnd log:

2021-09-16 15:12:28.523 [INF] LTND: Version: 0.13.1-beta commit=v0.13.1-beta, build=production, logging=default, debuglevel=info
2021-09-16 15:12:28.523 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2021-09-16 15:12:28.560 [INF] RPCS: RPC server listening on 0.0.0.0:10009
2021-09-16 15:12:28.562 [INF] RPCS: gRPC proxy started at 0.0.0.0:8080
2021-09-16 15:12:28.569 [INF] LTND: Opening the main database, this might take a few minutes...
2021-09-16 15:12:28.569 [INF] LTND: Opening bbolt database, sync_freelist=false, auto_compact=false
2021-09-16 15:12:29.042 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: waiting to start, RPC services not available
2021-09-16 15:12:29.180 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: waiting to start, RPC services not available
2021-09-16 15:12:29.563 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: waiting to start, RPC services not available
2021-09-16 15:12:30.930 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: waiting to start, RPC services not available
2021-09-16 15:12:31.671 [INF] CHDB: Checking for schema update: latest_version=22, db_version=22
2021-09-16 15:12:31.671 [INF] LTND: Database now open (time_to_open=3.101283509s)!
2021-09-16 15:12:31.671 [INF] LTND: Waiting for wallet encryption password. Use `lncli create` to create a wallet, `lncli unlock` to unlock an existing wallet, or `lncli changepassword` to change the password of an existing wallet and unlock it.
2021-09-16 15:12:32.420 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access
2021-09-16 15:12:34.257 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access
2021-09-16 15:12:36.859 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access
2021-09-16 15:12:36.859 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: wallet locked, unlock it to enable full RPC access
2021-09-16 15:12:36.873 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access
2021-09-16 15:12:38.948 [ERR] RPCS: [/lnrpc.Lightning/WalletBalance]: wallet locked, unlock it to enable full RPC access
2021-09-16 15:12:38.948 [ERR] RPCS: [/lnrpc.Lightning/GetTransactions]: wallet locked, unlock it to enable full RPC access
2021-09-16 15:12:38.950 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access
2021-09-16 15:12:38.961 [ERR] RPCS: [/lnrpc.Lightning/ListInvoices]: wallet locked, unlock it to enable full RPC access
2021-09-16 15:12:38.962 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: wallet locked, unlock it to enable full RPC access
2021-09-16 15:12:38.963 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: wallet locked, unlock it to enable full RPC access
2021-09-16 15:12:39.056 [ERR] RPCS: [/lnrpc.Lightning/ListPayments]: wallet locked, unlock it to enable full RPC access
2021-09-16 15:12:39.201 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access
2021-09-16 15:12:39.261 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access
2021-09-16 15:12:41.396 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access
2021-09-16 15:12:42.078 [INF] LNWL: Opened wallet
2021-09-16 15:12:42.255 [INF] CHRE: Primary chain is set to: bitcoin
2021-09-16 15:12:42.285 [INF] CHRE: Initializing bitcoind backed fee estimator in CONSERVATIVE mode
2021-09-16 15:12:42.285 [INF] LNWL: Started listening for bitcoind transaction notifications via ZMQ on 10.21.21.8:28333
2021-09-16 15:12:42.285 [INF] LNWL: Started listening for bitcoind block notifications via ZMQ on 10.21.21.8:28332
2021-09-16 15:12:42.293 [INF] CHRE: Using external fee estimator https://nodes.lightning.computer/fees/v1/btc-fee-estimates.json: cached=true
2021-09-16 15:12:42.294 [INF] LNWL: Starting web API fee estimator
2021-09-16 15:12:45.218 [INF] LNWL: The wallet has been unlocked without a time limit
2021-09-16 15:12:45.224 [INF] CHRE: LightningWallet opened
2021-09-16 15:12:45.229 [INF] SRVR: Proxying all network traffic via Tor (stream_isolation=false)! NOTE: Ensure the backend node is proxying over Tor as well
2021-09-16 15:12:45.250 [INF] HSWC: Restoring in-memory circuit state from disk
2021-09-16 15:12:45.252 [INF] HSWC: Payment circuits loaded: num_pending=0, num_open=0
2021-09-16 15:12:45.279 [INF] HSWC: Trimming open circuits for chan_id=700215:1561:0, start_htlc_id=1
2021-09-16 15:12:45.280 [INF] HSWC: Trimming open circuits for chan_id=697402:2025:1, start_htlc_id=0
2021-09-16 15:12:45.280 [INF] HSWC: Trimming open circuits for chan_id=697335:1350:0, start_htlc_id=3
2021-09-16 15:12:45.280 [INF] HSWC: Trimming open circuits for chan_id=697360:88:1, start_htlc_id=0
2021-09-16 15:12:45.280 [INF] HSWC: Trimming open circuits for chan_id=699092:1129:0, start_htlc_id=0
2021-09-16 15:12:45.280 [INF] HSWC: Trimming open circuits for chan_id=700787:559:1, start_htlc_id=20
2021-09-16 15:12:45.280 [INF] HSWC: Trimming open circuits for chan_id=697341:2104:1, start_htlc_id=19
2021-09-16 15:12:45.280 [INF] HSWC: Trimming open circuits for chan_id=698968:871:1, start_htlc_id=2
2021-09-16 15:12:45.280 [INF] HSWC: Trimming open circuits for chan_id=700096:282:0, start_htlc_id=0
2021-09-16 15:12:45.280 [INF] HSWC: Trimming open circuits for chan_id=698420:2478:1, start_htlc_id=0
2021-09-16 15:12:45.280 [INF] HSWC: Trimming open circuits for chan_id=697853:1215:1, start_htlc_id=15
2021-09-16 15:12:45.280 [INF] HSWC: Trimming open circuits for chan_id=697801:2794:1, start_htlc_id=1
2021-09-16 15:12:45.281 [INF] HSWC: Trimming open circuits for chan_id=700809:1666:0, start_htlc_id=0
2021-09-16 15:12:45.281 [INF] HSWC: Trimming open circuits for chan_id=699537:533:0, start_htlc_id=1
2021-09-16 15:12:45.334 [INF] LTND: Channel backup proxy channel notifier starting
2021-09-16 15:12:45.335 [ERR] LTND: unable to create server: tower not found
2021-09-16 15:12:45.336 [INF] LNWL: Stopping web API fee estimator
2021-09-16 15:12:45.360 [INF] LTND: Shutdown complete

unable to create server: tower not found

Expected behaviour

lnd boots and actually starts a communicating with configured watchtowers

Actual behaviour

lnd crashes and restarts in a loop.

Any ideas why the server cant be created?

DarthCoin · September 16, 2021, 4:30pm

To get your WTClient address:

Run this:

docker exec -it lnd lncli tower info

To add someone else’s node to you, have them run that command above (on an umbrel) and get the URI value, and you run this on your umbrel:

docker exec -it lnd lncli wtclient add

and before you press return, add their URI value. It should look like this:

02cef4bdea…@lyfay2xy…onion:9911

If you do not indicate to your node to which WT to connect, for sure is crashing.

tokriha · September 16, 2021, 5:06pm

I activated the wtclient in the lnd.conf. Now when I try to execute

docker exec lnd lncli wtclient add xxx@yyy.onion:9911 I get the following error message:

[lncli] rpc error: code = Unknown desc = waiting to start, RPC services not available

this is no surprise, as the lnd container still cannot boot when wtclient is activated.

and, regarding your last sentence, it would be weirdly written code, if your lnd only starts if you BOTH activate the client AND configure at least one watchtower. this should be independent.

anything else I might be missing?

DarthCoin · September 16, 2021, 5:09pm

No need for both to activate. Actually will be better not to use each others WTs.

tokriha · September 16, 2021, 5:13pm

First of all, thx for your replies of course, @DarthCoin!
However, I don’t get the last statement. Who is “each others” WTs and why is it better not to use them?

DarthCoin · September 16, 2021, 5:17pm

each other = you and your peer/friend that provide a WT server to watch on your channels.
I said that is not good to have connected to each others as WT clients. Means you set his WT server and he set your WT server address.
I read about this some time ago, some LN developers talking about this, that can have some issues.
But connecting only you from your peer/friend WT server is OK. He will have to use another WT server.

tokriha · September 16, 2021, 5:23pm

ok, cool, thx. but, from a functional perspective, it shouldn’t matter to which watchtower I connect, right?

However, the issue at hand is still that I cannot even add a WT, because the lnd crashes when I activate wtclient in the lnd.conf

DarthCoin · September 16, 2021, 6:58pm

maybe you put it into the wrong place and/or add too much, be careful with spaces and line breaks.
A full sample lnd.conf file with explanations here

DarthCoin · February 16, 2022, 9:27am

Do you still have this behavior ?

Masterno2 · April 21, 2022, 5:56pm

I followed the discussion, but i have the same issue, i simply add
[wtclient]

wtclient.active=true

[watchtower]

watchtower.active=true

[wtclient]

wtclient.active=true

[watchtower]

watchtower.active=true

After restart, each time i tried a docker command, i always got an “waiting to start, RPC services not available”

I am on unbuntu server

Masterno2 · April 21, 2022, 6:08pm

Ok i found, i made a copy paste from

And i removed the empty line

[wtclient]
wtclient.active=true

[watchtower]
watchtower.active=true

That was it !

Topic		Replies	Views
"watchtower client not active" on Umbrel 0.5.4 Bitcoin and Lightning	2	195	October 5, 2023
Umbrel as watchtower Support and Troubleshooting	2	696	March 18, 2022
Added Watchtower not working and WTClient apparently not activating Support and Troubleshooting	1	616	April 5, 2024
Watchtower Tutorial Bitcoin and Lightning	4	1323	August 22, 2021
Adding watchtower to Umbrel node? General Discussions	8	927	October 30, 2021