Stuck backing up after failed OS update

[lowa]

Yo team,

My OS stalled following an update a few weeks ago, and was stuck on “backing up” splash screen. I waited for over 24h initially and after that concluded something was clearly NQR
and now i can’t actually remember to which OS update it was… 0.4.something…

I’ve tried flashing the SD to the latest, and so far that has not worked. i still get the same backing up splash screen.

I’ve been trying to get debug logs for hrs but cannot seem to find the link to post. I’ve been up and down the forum to try and avoid wasting anyones time, but am at a genuine loss as to wtf to do…

I would be happy to just reinstall from seed, but i tried that also, and that has not worked so far either.

greatly appreciate any assistance

[lowa]

just noticed this in ‘backup-monitor’ log

/home/umbrel/umbrel/scripts/backup/…/…/lnd/data/chain/bitcoin/mainnet/channel.backup doesn’t exist, waiting for it to be created…

[Hakuna]

Check this out.
Manually updating Umbrel node

To manually update your node using SSH access.

Make a channels backup. Go to Umbrel dashboard - Lightning and on the right top side, 3 dots, save backup on your local PC.

A. For Rasp Pi users:

Enter using SSH into your Umbrel node, then run:

cd ~/umbrel && sudo ./scripts/update/update --repo getumbrel/umbrel#v0.4.8

Replace v0.4.8 with the version you want to update to.

If the update was stuck in update, run this before the above command:

sudo rm statuses/update-in-progress
sudo rm statuses/update-status.json

Restart your node if is done OK:

sudo reboot

[lowa]

Thanks Hakuna - much appreciated.

This seemed to do something. now am back at the “loading” splash screen.

[lowa]

looking very positive. first time seeing the login being possible via the GUI for weeks. Waiting to see if things sync… previous login worked, so looking promising

[Hakuna]

Keep us posted, good luck!

[lowa]

Am back into the GUI with same account (good), but while the blockchain is 100% synced (also good) - bitcoin and lightning wallets have not yet… just downloaded the logs (which ive attached)umbrel-1641374153850-debug.log (38.5 KB)

also kind of weird i flashed the sd with 0.4.8 but it showed 0.4.1 as the new release to install (which i did) and it hasn’t suggested updated further yet.

[Hakuna]

I’m not sure if there are things still catching up.
It is known that the electr server needs up to 24hrs to catch up with the blockchain. But I don’t know if that’s the reason it can’t find your lnd-wallet:

lnd | 2022-01-05 09:14:55.391 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: wallet not created, create one to enable full RPC access

So we have 3 choices I think

1. Wait a few more hours, keep an eye on this debug log and check whether things are changing in the LND and electr section
2. Reflash the SD card via getumbrel.com and try the update process again (do a graceful shutdown via UI, remove the SD, flash, reinsert it and boot => check the logfiles)
3. Tinker with the wallet via lncli (most advanced stuff)

I’d move from 1-3 in the ranked order.
Hope this helps

[lowa]

very much appreciate your help. noticed that in the logs too. will do as you have suggested and update afterwards… will give it a few more hrs incase before continuing

[lowa]

Waited a few hrs. no dice
Reflashed 0.4.9 after a graceful shutdown

new logs umbrel-1641443197067-debug.log (34.2 KB)

Same issue still on LND with no wallet found.

I’m still confused as to why it shows umbrel V 0.4.1 when i flashed the SD with 0.4.9
It this normal ?

moving to step 3 now…

[Hakuna]

Did you ever do a restart? Logfile shows some containers running for 5 weeks already, is that true?

For the wallet pondering in Step 3, you may want to check out this help-entry:
Wallet is locked. How to unlock it in command line

Credits to @Hakuna that helps writing it.

How to verify: login to the Umbrel dashboard page > settings > Debug log and look out for the LND section. If you find a repetitive entry like the below, your wallet is locked and LND can’t start properly:
RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access

Part 1)
Login to your node with an SSH client / Power terminal via

ssh -t umbrel@umbrel.local

The prompted password is your dashboard password.
Then execute:

cd ~/umbrel && docker-compose exec lnd /bin/bash

Then:

lncli unlock

When prompted, enter the password you have used logging into your Umbrel user interface.
Done. Now open Umbrel dashboard page and login. Verify like above if that was successful.

Part 2)
In case the above doesn’t work and your wallet is still locked, add the following steps

Login to your node with an SSH client / Power terminal via

ssh -t umbrel@umbrel.local

Make a backup of your current lnd.conf

cp ~/umbrel/lnd/lnd.conf ~/umbrel/lnd/lnd.bak

Delete your current lnd.conf, re-configure Umbrel and restart the node

rm ~/umbrel/lnd/lnd.conf && ~/umbrel/scripts/configure && sudo reboot

After restart, your LND will take a few minutes to catch up again.
Go into the Umbrel Dashboard > Settings > Debug again and validate the wallet is unlocked this time.

In case of success, carefully add the lines of customization from your lnd.bak and copy those into your lnd.conf. Don’t copy the whole file, as it may contains a wrong configuration which caused your wallet to be locked in the first place.

[lowa]

have done multiple restarts. have reflashed the sd 4 seperate times trying different os versions…

[Hakuna]

Okay, so I guess a restart and reflash is not sufficient.
What about the wallet adjustments I shared above, did you move on to this?

[lowa]

just tried lncli unlock, but password not working… onto part 2

[lowa]

part 2 done, but still no change. LND wallet not found…

umbrel-1641979244756-debug.log (37.9 KB)

[Hakuna]

@lowa

part 2 done, but still no change. LND wallet not found…

Could you copy and paste the results of the commands please, from step 1 and 2 again:

1. cd ~/umbrel && docker-compose exec lnd /bin/bash

Then:

lncli unlock
and especially
2) for every single command, instead of one in a row

• rm ~/umbrel/lnd/lnd.conf
• ~/umbrel/scripts/configure

I’m particulary interested what it states when you kick off the configure command

[lowa]

@Hakuna

1. cd ~/umbrel && docker-compose exec lnd /bin/bash

1. Last login: Thu Jan 13 01:13:40 2022 from fe80::1837:41e7:6326:7b20%eth0
   umbrel@umbrel:~ $ cd umbrel
   umbrel@umbrel:~/umbrel $ docker-compose exec lnd /bin/bash
   bash-5.1$ lncli unlock
   Input wallet password:
   [lncli] rpc error: code = Unknown desc = wallet not found
   bash-5.1$ cp ~/umbrel/lnd/lnd.conf ~/umbrel/lnd/lnd.bak
   cp: can’t stat ‘/data/umbrel/lnd/lnd.conf’: No such file or directory

lncli unlock = (password does not work)

umbrel@umbrel:~ $ cp ~/umbrel/lnd/lnd.conf ~/umbrel/lnd/lnd.bak
umbrel@umbrel:~ $
umbrel@umbrel:~ $ rm ~/umbrel/lnd/lnd.conf
umbrel@umbrel:~ $ ~/umbrel/scripts/configure

======================================
=========== RECONFIGURING ============
========= UMBREL (mainnet) ===========

Configuring permissions…

chown: changing ownership of ‘/home/umbrel/umbrel/events/signals/shutdown’: Operation not permitted
Configuration successful
You can now start Umbrel by running:
sudo ./scripts/start

umbrel@umbrel:~ $ sudo reboot

[lowa]

@Hakuna I’m kind of confused at this point, as to what the issue is, and how best to solve ?

If i have LN channel backup and seed, would it not be easiest just to restore from seed?

[Hakuna]

Wait - not yet. A channel backup would trigger a force-close of all your channels, let’s try to keep that as a last resort.

Could be the file user permissions are off. This one threw me off, even though I never done the reconfig myself, I fear the configuration isn’t successfully completed, hence you’re stuck.
chown: changing ownership of ‘/home/umbrel/umbrel/events/signals/shutdown’: Operation not permitted

Could you try the 3, but especially the last command ~/umbrel/scripts/configure with sudo please
sudo ~/umbrel/scripts/configure

[lowa]

@Hakuna

sudo ~/umbrel/scripts/configure

ah OK. I did not realise it triggered forced closure of channel. That is sub-optimal.

umbrel@umbrel:~/umbrel $ sudo ~/umbrel/scripts/configure
[sudo] password for umbrel:

======================================
=========== RECONFIGURING ============
========= UMBREL (mainnet) ===========

Skipping lnd.conf update to preserve user changes…
Configuring permissions…

Configuration successful
You can now start Umbrel by running:
sudo ./scripts/start

umbrel@umbrel:~/umbrel $