Lightning apps (lnd, lndg, etc...) stuck in "starting"

Support and Troubleshooting
1

Hello everyone.
I had a really bad luck some days ago.
Internet went down in the whole area.
I was updating Lightning Node, LNDg and NoLooking when this happened.
Since the connection is not going to be restored anytime soon, I had to move my Pi to my friend’s house.
I managed to start it up.
Umbrel started fine, the Bitcoin Node app reached 100% synchronization again.
However, Lightning Node and every other dependent app is stuck in “starting”. They have been stuck for about 1 hour now.
I checked the Umbrel troubleshoot and it just reports “Attaching to” for these apps.
I am afraid to uninstall and reinstall these apps, because I am not sure I have a backup of my channels at all.
Even if I had any, it might be too old and it could not include every channel.
What can I do?

Thank you

2

you should try to shutdown the Umbrel and turn it on again.
Also check if your Bitcoin node is 100% sync?

3

Yes, I have already tried it with no success.

However, I managed to unlock at least the lightning wallet app.

I used this command via ssh:

sudo ~/umbrel/scripts/app restart lightning

It seems the app managed to update and start.
I was able to access it through the umbrel dashboard/gui.
For a moment, the app loaded smoothly. By clicking on the three dots on the top right connect, I was able to see that my node name was there and the settings were unaltered since last time.
Nevertheless, everything was stuck on loading. The onchain balance and lightning wallet were not displaying any amount. Also the channel list was loading.
I closed the related browser tab and went away, hoping everything would be restored soon.

This morning I reopened the app, but it remained stuck in the app loading screen at about 60%.
It has been in this situation for 2 hours now.

The logs are these (I censored some transaction hash):

 lightning

Attaching to lightning_app_proxy_1, lightning_tor_1, lightning_tor_server_1, lightning_app_1, lightning_lnd_1
app_1         | [backup-monitor] Sleeping...
app_1         | [backup-monitor] Checking channel backup...
app_1         | [backup-monitor] Sleeping...
app_1         | ::ffff:10.21.0.26 - - [Sun, 07 May 2023 07:57:08 GMT] "GET /v1/lnd/lightning/payments HTTP/1.1" 200 2880713 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
app_1         | 
app_1         | umbrel-lightning
app_1         | [backup-monitor] Checking channel backup...
app_1         | [backup-monitor] Sleeping...
app_1         | [backup-monitor] Checking channel backup...
app_1         | [backup-monitor] Sleeping...
app_proxy_1   | Validating token: bb42445fa4bf ...
app_proxy_1   | Validating token: bb42445fa4bf ...
app_proxy_1   | Validating token: bb42445fa4bf ...
app_proxy_1   | Validating token: bb42445fa4bf ...
app_proxy_1   | Validating token: bb42445fa4bf ...
app_proxy_1   | Validating token: bb42445fa4bf ...
app_proxy_1   | Validating token: bb42445fa4bf ...
app_proxy_1   | Validating token: bb42445fa4bf ...
app_proxy_1   | Validating token: bb42445fa4bf ...
app_proxy_1   | Validating token: bb42445fa4bf ...
lnd_1         | 2023-05-07 07:48:29.904 [WRN] LNWL: unable to fetch transaction XXX for mempool: -5: No such mempool or blockchain transaction. Use gettransaction for wallet transactions.
lnd_1         | 2023-05-07 07:48:29.922 [WRN] LNWL: unable to fetch transaction XXX for mempool: -5: No such mempool or blockchain transaction. Use gettransaction for wallet transactions.
lnd_1         | 2023-05-07 07:49:03.012 [WRN] LNWL: unable to fetch transaction XXX for mempool: -5: No such mempool or blockchain transaction. Use gettransaction for wallet transactions.
lnd_1         | 2023-05-07 07:49:04.311 [WRN] LNWL: unable to fetch transaction XXX for mempool: -5: No such mempool or blockchain transaction. Use gettransaction for wallet transactions.
lnd_1         | 2023-05-07 07:49:19.979 [WRN] LNWL: unable to fetch transaction XXX for mempool: -5: No such mempool or blockchain transaction. Use gettransaction for wallet transactions.
lnd_1         | 2023-05-07 07:50:34.888 [WRN] LNWL: unable to fetch transaction XXX for mempool: -5: No such mempool or blockchain transaction. Use gettransaction for wallet transactions.
tor_1         | May 06 22:02:25.000 [notice] Bootstrapped 95% (circuit_create): Establishing a Tor circuit
tor_1         | May 06 22:02:27.000 [notice] Bootstrapped 100% (done): Done
tor_1         | May 06 23:29:21.000 [notice] No circuits are opened. Relaxed timeout for circuit 139 (a Hidden service: Uploading HS descriptor 4-hop circuit in state doing handshakes with channel state open) to 60000ms. However, it appears the circuit has timed out anyway.
tor_1         | May 06 23:30:21.000 [warn] Unable to decompress HTTP body (tried gzipped, on Directory connection (client reading) with 138.2.165.161:9001).
tor_1         | May 07 01:50:30.000 [notice] No circuits are opened. Relaxed timeout for circuit 392 (a Hidden service: Uploading HS descriptor 4-hop circuit in state doing handshakes with channel state open) to 60000ms. However, it appears the circuit has timed out anyway. [26 similar message(s) suppressed in last 8520 seconds]
tor_1         | May 07 04:01:05.000 [notice] Heartbeat: Tor's uptime is 6:00 hours, with 22 circuits open. I've sent 6.91 MB and received 12.60 MB. I've received 0 connections on IPv4 and 0 on IPv6. I've made 39 connections with IPv4 and 0 with IPv6.
tor_1         | May 07 04:01:05.000 [notice] While bootstrapping, fetched this many bytes: 636606 (consensus network-status fetch); 25190 (authority cert fetch); 7562397 (microdescriptor fetch)
tor_1         | May 07 04:01:05.000 [notice] While not bootstrapping, fetched this many bytes: 207090 (consensus network-status fetch); 7096 (authority cert fetch); 142348 (microdescriptor fetch)
tor_1         | May 07 06:49:24.000 [notice] No circuits are opened. Relaxed timeout for circuit 603 (a Hidden service: Uploading HS descriptor 4-hop circuit in state doing handshakes with channel state open) to 60000ms. However, it appears the circuit has timed out anyway. [1 similar message(s) suppressed in last 17940 seconds]
lnd_1         | 2023-05-07 07:52:18.161 [WRN] LNWL: unable to fetch transaction 43ae04ac875f8251ee404d47b0900c6512abd0084bcd4d59597aff80a99da82d for mempool: -5: No such mempool or blockchain transaction. Use gettransaction for wallet transactions.
lnd_1         | 2023-05-07 07:52:45.763 [WRN] LNWL: unable to fetch transaction XXX for mempool: -5: No such mempool or blockchain transaction. Use gettransaction for wallet transactions.
lnd_1         | 2023-05-07 07:53:23.627 [WRN] LNWL: unable to fetch transaction XXX for mempool: -5: No such mempool or blockchain transaction. Use gettransaction for wallet transactions.
lnd_1         | 2023-05-07 07:53:53.139 [WRN] LNWL: unable to fetch transaction XXX for mempool: -5: No such mempool or blockchain transaction. Use gettransaction for wallet transactions.

I do not know whether to try another restart or not.
I hope my old channel.db file is still in the current drive!
I tried to access following the path showed here: http://community.umbrel.com/t/how-to-reduce-size-of-channel-db-file-for-lnd/6439 but I cannot get there through ssh.

4

I was using Umbrel Automatic Channel Recovery and was able to recover satoshi without a recent backup channel file. It, however, will force close all of your existing channels.
https://twitter.com/umbrel/status/1562099972547690501

Is your Bitcoin node 100% synced now?

Anyway, I suggest you would try to fix the apps without having to perform a recovery. The Bitcoin fees now are very high.

1 Like
5

Indeed, the Bitcoin Node is 100% synced.

Thank you for pointing that guide! I will keep it as last option for now.
If I perform a recovery, the last automatic backup should be date 1st May (not too long ago).

I will keep this thread updated. Thank you again.

6

Just an additional question: did you really mean that all existing channels would be force-closed after a recovery, or maybe there is a possibility that only those opened after the last backup date will be closed?

7

For me, all channels were force closed. I don’t think we have an option to selective close some channels.

8

this may be related problem… Upgrade LND stuck at "starting" v 0.16.2

9

I finally solved it.
I think it was also due to my friend’s router or the ethernet cable he gave me.

Basically, I shut down umbrel, shut down my friend’s router, changed AC wall adapter for the router, changed ethernet cable, started the pi/umbrel again.
Everything started correctly again, except for NoLooking and Lndg, which needed an ssh restart using

sudo ~/umbrel/scripts/app restart *

Where * stands for the app name.

The Lightning Wallet took about an hour to be fully operational. The longest part was the reconnection to lightning peers.