Upgrade to 0.5.4 Crashed Node, All Channels Closed

I installed the 0.5.4 upgrade. It crashed my bitcoin/lightning node - wouldn’t start up. I wound up having to flash the SD card, delete the known-hosts file, find my channel backup and install it (it didn’t just show up when clicking “Recover Channels”). Once I did that, it started closing every lightning channel. I had no idea that “Recover Channels” meant closing all channels. With the closing fees this has been a costly and time-consuming upgrade to 0.5.4. If I did something wrong it would be nice to know what I did wrong so this never happens to me again.

1 Like

I would need to confirm, but I think it depends on the back-up.
If it’s via the Static channel backup, LND automatically force-closes. Disaster recovery - Builder's Guide

The Automatic Backups slider was enabled - there’s no option for anything else for backups - how would I know if its a static channel backup, or if something else should have been selected?

Did you resolve your issue? If yes, how? Thanks

Almost, I used chantools and force closed the rest of the channels that Umbrel did not automatically force close, except one channel. That node is dead and it’s on Tor so chantools won’t force close it. I can’t get an answer from Umbrel how to install the channel backup file they sent me like this 1692500000000.tar.gz.pgp so I can maybe try to force close it myself.

At least you were able to get apps to reinstall after a fresh re-install of everything. I can’t even get apps to install again. My Umbrel is a powered paperweight at the moment. All my logs say is everything is working fine except when I try to install any app it’s like the repository it’s trying to pull from isn’t there. One would think a fresh full re-install would add the correct missing repositories needed but the only thing it has me keep redoing is setting up my user name and password. Then on the homepage where is says “Install your first app.” :unamused:

It always one of these not connecting:

ERROR: for app_proxy Get “https://registry-1.docker.io/v2/getumbrel/app-proxy/manifests/sha256:130fb1572fa23e740b4b7b97bce2c206ead6b138d321ff152b1ba35ead61e67a”: dial tcp [2600:1f18:2148:bc02:cfd8:db68:ea1f:277c]:443: connect: network is unreachable
Get “https://registry-1.docker.io/v2/getumbrel/app-proxy/manifests/sha256:130fb1572fa23e740b4b7b97bce2c206ead6b138d321ff152b1ba35ead61e67a”: dial tcp [2600:1f18:2148:bc02:cfd8:db68:ea1f:277c]:443: connect: network is unreachable

Hey @gotagreatdog, if your Umbrel is online you can click Download Channel Backup file and grab your channel.backup,

That is your SCB file,
You can also ask your peer to cooperative close, are they online?

The tar.gz.pgp is encrypted by your own machine, referenced here “The backups are encrypted client side,” to use that file you must recover with your seed and then can get all the channels back, which will force close as noted in disaster recovery - closing channels is currently recommended as the safest and most reliable way to return funds - as offchain channel migration and disaster recovery as we see here has many things that can go wrong and we want to avoid having to do a disaster recovery,

I’m sorry you had that issue, flashing your MicroSD card shouldn’t have caused you to reinstall Lightning Node App and do a recovery (automated recovery is a great step if someone has completely relocated for example and have no access to the old device), is the Pi 4 still functioning?

You don’t need to have done anything differently, except by troubleshooting the Pi 4 we should have been able to get it right back online and secure funds with lightning channels still open,

More details on the SCB file:

Many of the steps are quite a bit of hassle to manually download a channel.db file and reinitiliaze LND natively, a lot can go wrong here if we broadcast a stale channel for instance we risk losing all funds, so we should ensure we have some redundancies in place on our setup like plugging in with official power supply, extra MicroSD card on hand, using an uninterruptible power supply etc. and being confident in our setup and recovery process steps before storing any significant funds,

triggerforcelose command would be the chantools command on LND if it is a CLN peer that would be an issue on sometimes only on Tor AFAIK, have we identified the peer on your last channel to confirm?
That channel is offline? We can double check on steps here to get this last channel closed for you

You’re having an IPv6 error on the 443 port needed to be open on Umbrel, we can further investigate,

May I confirm that we have taken any additional steps to block traffic or have any custom rule set to block more ports or devices on the network? Such as any custom IPs for your device, custom DNS, or any other manual settings configured you may share?

Any additional details provided about the firewall configuration may help. Is it blocking communication from other devices within the local network or restricting access from the internet? By default, most firewalls are set to allow all ports for local network communication and have no ports forwarded to the internet, which is typically a secure configuration,

As an example, Umbrel utilizes the main ports 80 and 443 locally so default configuration on most routers should work,

And there are any conflicts here with custom ports or anything you can think of? We can keep looking into this if you’re having any issue… Feel free to e-mail us directly as well on our official channel to investigate and help further @RyukenKai for this separate issue https://www.umbrel.com/support

Happy to address any other questions please let me know if you were able to fix this yet and happy to assist get you all up and running!

Hi,

The last remaining open channel is 39cd769e5c8df97e77ea2f70c70b80fa750cb7a14d0eaab46962fb2794bc03b5:1 All other channels were force closed either during recovery or with chantools.

https://1ml.com/channel/862485508159438849
https://amboss.space/node/028d3abdb56f8b2675d87fcf15e79e3f4df19d0de3a4a811684be95785fa565581?section=Channels

The peer has been offline for months. I tried force close the channel with chantools but it’s Tor so chantools didn’t work.

None of the channel backups you guys provided from before the crash on Aug 19 worked unfortunately. I have removed the Umbrel app, reinstalled the node with the same seed, and every time I try to recover channels I get an error, unable to unpack chan backup
I tried
1692386276718.tar.gz.pgp
1692400274984.tar.gz.pgp
1692439225443.tar.gz.pgp
1692500044648.tar.gz.pgp
1692504935725.tar.gz.pgp
1692545142009.tar.gz.pgp

My node is online, 028d3abdb56f8b2675d87fcf15e79e3f4df19d0de3a4a811684be95785fa565581@3avjv4r3seyl6lumo4cbcx4zkzvngiu3b5nfdxf57ok3b5c5ak4cmnid.onion:9735

I’m not having any IP error or firewall issues. I have no idea what to try now, please let me know.

Thank you,
Clint Farrell

Thank you. I appreciate your response. I need to build up enough sats to actually open a stable channel. 1,000,000,000 sats minimum for most stable channel openings is a bit steep for me at the moment. My router ports are set at default. I don’t have any extra open or closed. I am able to get apps to install but I have to use cli and make several attempts to force the install but it’s a hassle. I was able to recover most of my funds, aside from dust and fees lost in channel connection failures. I’ll give it another try in the future but for now I’m going to have to shelve it.

1 Like

Gotchya to confirm this channel is appearing in your pendingchannels output?

When running

~/umbrel/scripts/app compose lightning exec lnd lncli pendingchannels

or it is not there at all?

Feel free to provide output to this command as well

Hi Jim,

It shows nothing there at all, even though there is one channel open as I showed.
{
“total_limbo_balance”: “0”,
“pending_open_channels”: [
],
“pending_closing_channels”: [
],
“pending_force_closing_channels”: [
],
“waiting_close_channels”: [
]
}

https://1ml.com/channel/862485508159438849 shows channel is closed (it’s not) and my Gotagreatdog node is inactive (it’s not)
https://amboss.space/node/02defb1fcf3d8e6254abffe422327ab049848b80d2198bf41afb39d86ee70c4797?section=Channels whereas Amboss shows the channel as open

No idea what’s going on. As I said, I tried force close and sweep in chantools but no luck.

Thank you,
Clint Farrell