Lighting Not starting after power outage/restart

LND not available, logs attached.

ightning_app_1 | Waiting for LND…
lightning_app_1 | Checking LND status…
lightning_app_1 | [backup-monitor] Checking channel backup…
lightning_app_1 | [backup-monitor] Sleeping…
lightning_tor_1 | Apr 21 06:47:37.000 [warn] Received http status code 404 (“Not found”) from server 212.227.135.115:443 while fetching “/tor/keys/fp/D586D18309DED4CD6D57C18FDB97EFA96D330566+14C131DFC5C6F93646BE72FA1401C02A8DF2E8B4+EFCBE720AB3A82B99F9E953CD5BF50F7EEFC7B97”.
lightning_tor_1 | Apr 21 06:48:38.000 [notice] No circuits are opened. Relaxed timeout for circuit 3595 (a Hidden service: Establishing introduction point 4-hop circuit in state doing handshakes with channel state open) to 60000ms. However, it appears the circuit has timed out anyway. [9 similar message(s) suppressed in last 9960 seconds]
lightning_tor_1 | Apr 21 08:41:38.000 [warn] Received http status code 404 (“Not found”) from server 212.227.135.115:443 while fetching “/tor/keys/fp/D586D18309DED4CD6D57C18FDB97EFA96D330566+EFCBE720AB3A82B99F9E953CD5BF50F7EEFC7B97”.
lightning_tor_1 | Apr 21 08:42:41.000 [notice] No circuits are opened. Relaxed timeout for circuit 3714 (a Hidden service: Uploading HS descriptor 4-hop circuit in state doing handshakes with channel state open) to 60000ms. However, it appears the circuit has timed out anyway. [5 similar message(s) suppressed in last 6900 seconds]
lightning_tor_1 | Apr 21 09:46:38.000 [notice] No circuits are opened. Relaxed timeout for circuit 3765 (a Hidden service: Establishing introduction point 4-hop circuit in state doing handshakes with channel state open) to 60000ms. However, it appears the circuit has timed out anyway. [11 similar message(s) suppressed in last 3840 seconds]
lightning_tor_1 | Apr 21 10:22:14.000 [notice] Heartbeat: Tor’s uptime is 2 days 12:00 hours, with 16 circuits open. I’ve sent 49.48 MB and received 58.61 MB. I’ve received 0 connections on IPv4 and 0 on IPv6. I’ve made 18 connections with IPv4 and 0 with IPv6.
lightning_tor_1 | Apr 21 10:22:14.000 [notice] While bootstrapping, fetched this many bytes: 763818 (consensus network-status fetch); 14104 (authority cert fetch); 16978027 (microdescriptor fetch)
lightning_tor_1 | Apr 21 10:22:14.000 [notice] While not bootstrapping, fetched this many bytes: 514862 (consensus network-status fetch); 93769 (authority cert fetch); 3884639 (
umbrel_lightning_2025-04-21_09-11.log (104.4 KB)

ok

I haven’t had this happen yet or needed to do it, so proceed at your own risk here. Have a full read through before proceeding. :slight_smile:

The logs provide critical insight into why LND is not available: LND is repeatedly crashing due to a panic error in the bbolt database, specifically with the error freepages: failed to get all reachable pages (page 34104: multiple references). This indicates database corruption in LND’s bbolt database, which is used to store channel states, wallet data, and other critical information. The Tor issues noted previously are secondary and likely not the root cause of LND’s failure to start. Below are focused actions to address this issue, prioritizing the database corruption.

Analysis

  1. LND Crash Cause:
  • The error panic: freepages: failed to get all reachable pages (page 34104: multiple references) occurs when LND attempts to open its bbolt database (channel.db). This is a known issue with bbolt (a key-value store used by LND) and typically indicates corruption, often caused by:
    • Improper shutdown: Power loss or abrupt termination of LND.
    • Disk issues: Hardware failure, full disk, or file system errors.
    • Resource constraints: Insufficient memory or CPU causing incomplete writes.
  • The repeated crashes (every minute from 12:25 to 13:02 on April 21, 2025) show that LND is stuck in a restart loop, unable to proceed past opening the database.
  1. Tor Issues:
  • The lightning_tor_1 logs still show 404 errors and circuit timeouts, but these are likely irrelevant to the LND crash. Tor is bootstrapping successfully (reaching 100% in the logs), and the lack of INTRODUCE2 cells suggests LND isn’t even attempting to establish connections due to the crash.
  1. Configuration Warning:
  • The warning Config ‘bitcoin.active’ is deprecated, please remove it indicates an outdated configuration in lnd.conf. While not the cause of the crash, it should be addressed to ensure compatibility with LND 0.18.5-beta.
  1. Impact:
  • LND’s failure to start means your Umbrel Lightning node is offline, preventing channel operations, payments, or routing.
  • The database corruption could risk channel funds if not handled carefully, especially if channels need to be closed or restored.

Focused Actions

The primary goal is to resolve the bbolt database corruption and get LND running again. Below are targeted steps, ordered by priority. Proceed with caution and ensure you have backups of critical files (especially the channel.backup file) before making changes.

1. Back Up Critical Files

Before attempting repairs, back up LND’s data to prevent potential loss of channel funds:

  • SSH into your Umbrel node:

ssh umbrel@umbrel.local

  • Copy the LND data directory:

cp -r ~/umbrel/app-data/lightning/data/lnd ~/umbrel/lnd-backup-$(date +%F-%H%M%S)

  • Locate and back up the channel.backup file (Static Channel Backup, critical for recovering funds):

find ~/umbrel/app-data/lightning/data/lnd -name channel.backup cp ~/umbrel/app-data/lightning/data/lnd/channel.backup ~/umbrel/channel-backup-$(date +%F-%H%M%S).backup

  • Copy the backup to an external device (e.g., USB drive or another computer) to ensure it’s safe.

2. Check Disk and System Health

Database corruption is often linked to disk or resource issues. Verify the system’s health:

  • Check disk space:

df -h

Ensure the partition hosting ~/umbrel has at least a few GB free. If full, free up space by removing unnecessary files or apps via the Umbrel dashboard.

  • Check disk errors:

sudo dmesg | grep -i error sudo fsck /dev/sdX

Replace /dev/sdX with your disk (e.g., /dev/sda1). Run fsck only if the disk is unmounted or the system is in recovery mode.

  • Check system resources:

top free -m

Ensure memory and CPU usage are not maxed out. If resources are low, stop non-essential Umbrel apps via the dashboard.

3. Attempt to Repair the bbolt Database

The channel.db file is likely corrupted. You can attempt to repair it using bbolt tools or by compacting the database.

  • Install bbolt CLI tool (if not already present):

go install go.etcd.io/bbolt/cmd/bbolt@latest

Ensure $GOPATH/bin is in your PATH:

export PATH=$PATH:~/go/bin

  • Stop Umbrel to prevent LND from accessing the database:

sudo ~/umbrel/scripts/stop

  • Locate the channel.db file: The database is typically at:

~/umbrel/app-data/lightning/data/lnd/data/graph/mainnet/channel.db

  • Back up channel.db:

cp ~/umbrel/app-data/lightning/data/lnd/data/graph/mainnet/channel.db ~/umbrel/channel.db.backup

  • Check the database for errors:

bbolt check ~/umbrel/app-data/lightning/data/lnd/data/graph/mainnet/channel.db

This will report any inconsistencies. If errors are found, proceed to compacting.

  • Compact the database (to attempt repair):

bbolt compact -o ~/umbrel/channel.db.compacted ~/umbrel/app-data/lightning/data/lnd/data/graph/mainnet/channel.db

If successful, replace the original database:

mv ~/umbrel/app-data/lightning/data/lnd/data/graph/mainnet/channel.db ~/umbrel/channel.db.corrupted mv ~/umbrel/channel.db.compacted ~/umbrel/app-data/lightning/data/lnd/data/graph/mainnet/channel.db

  • Restart Umbrel:

sudo ~/umbrel/scripts/start

  • Monitor LND logs:

docker logs -f lightning_lnd_1

Check if LND starts without the freepages panic. If it still crashes, proceed to the next step.

4. Rebuild the channel.db Using Static Channel Backup (SCB)

If the database cannot be repaired, you may need to rebuild channel.db and restore channels using the channel.backup file. This step risks force-closing channels, which could result in loss of funds if not done correctly, so proceed only if necessary.

  • Stop Umbrel:

sudo ~/umbrel/scripts/stop

  • Move the corrupted channel.db:

mv ~/umbrel/app-data/lightning/data/lnd/data/graph/mainnet/channel.db ~/umbrel/channel.db.corrupted

  • Start Umbrel to create a new channel.db:

sudo ~/umbrel/scripts/start

LND will create a new, empty channel.db.

  • Restore channels using SCB:
    • Access the LND container:

docker exec -it lightning_lnd_1 sh

  • Use lncli to initiate recovery:

lncli restorechanbackup --multi_file /lnd/channel.backup

Replace /lnd/channel.backup with the path to your channel.backup file inside the container (check with find /lnd -name channel.backup).

  • This will attempt to force-close channels and recover funds to your on-chain wallet. Monitor the process:

lncli pendingchannels lncli closedchannels

  • Monitor logs:

docker logs -f lightning_lnd_1

Ensure LND starts and channels are being restored.

  • Warning: Channel restoration may take time and requires peers to be online. If channels don’t close properly, you may need to contact channel peers or use tools like chantools for advanced recovery (see step 6).

5. Address Deprecated Configuration

The bitcoin.active warning should be resolved to ensure compatibility:

  • Edit lnd.conf:

It’s either here,

nano ~/umbrel/app-data/lightning/data/lnd/lnd.conf or
nano ~/umbrel/app-data/lightning/data/lightning/settings.json
I suggest making a backup of the file first.

  • Remove or comment out the bitcoin.active line:

;bitcoin.active=1

  • Save and restart Umbrel:

sudo ~/umbrel/scripts/stop sudo ~/umbrel/scripts/start

See later comment, as this prob doesn’t work any longer.

6. Use chantools for Advanced Recovery (if Necessary)

If the above steps fail or channel restoration is incomplete, use chantools to recover funds from the corrupted database or SCB:

  • Install chantools:

go install github.com/lightninglabs/chantools@latest. ## If this doesn’t work, see comments at bottom.

  • Dump data from corrupted channel.db:

chantools dumpchanneldb --db ~/umbrel/channel.db.corrupted

This extracts channel information for analysis.

  • Force-close channels:

chantools forceclose --channeldb ~/umbrel/channel.db.corrupted --publish

This attempts to close channels using the corrupted database.

  • Recover funds with SCB:

chantools rescueclosed --backup ~/umbrel/channel-backup-<date>.backup

Follow the chantools prompts to recover funds to your on-chain wallet.

7. Reset LND (Last Resort)

If all else fails, reset LND and start fresh. This will lose channel state unless you have a valid SCB:

  • Stop Umbrel:

sudo ~/umbrel/scripts/stop

  • Delete LND data (excluding wallet.db and channel.backup):

mv ~/umbrel/app-data/lightning/data/lnd ~/umbrel/lnd-backup-$(date +%F-%H%M%S) mkdir -p ~/umbrel/app-data/lightning/data/lnd

  • Restore wallet.db and channel.backup (if available):

cp ~/umbrel/lnd-backup-<date>/wallet.db ~/umbrel/app-data/lightning/data/lnd/ cp ~/umbrel/lnd-backup-<date>/channel.backup ~/umbrel/app-data/lightning/data/lnd/

  • Restart Umbrel:

sudo ~/umbrel/scripts/start

  • Restore channels (as in step 4).

8. Address Tor Issues (Secondary)

The Tor circuit timeouts and 404 errors may persist but are not the primary issue. If LND starts successfully, test connectivity:

  • Check LND’s Tor address:

cat ~/umbrel/app-data/lightning/tor/data/hostname

  • Test peer connectivity:

docker exec lightning_lnd_1 lncli getinfo docker exec lightning_lnd_1 lncli listpeers

  • If Tor issues persist, configure Tor bridges (as outlined in the previous response) or consider switching to clearnet (requires a public IP).

Monitoring and Validation

After each step, monitor logs to confirm progress:

docker logs -f lightning_lnd_1 docker logs -f lightning_app_1

Look for:

  • LND starting without the freepages panic.
  • Successful blockchain sync (lncli getinfo shows synced_to_chain: true).
  • Channel restoration progress (if using SCB).

Important Notes

  • Channel Funds: If you have open channels, do not skip the backup step. A valid channel.backup file is critical for recovering funds. If channels are force-closed, you may incur on-chain fees, and funds may be locked in timelocks (CLTV).
  • Community Support: If you encounter issues, share sanitized logs (remove public keys, addresses) on the Lightning Labs GitHub (GitHub · Where software is built).
  • Hardware Check: If corruption persists, test your storage device (e.g., SD card or SSD) for failures, as hardware issues are a common cause.

Next Steps

Start with steps 1–3 (backup, system health, database repair). If the bbolt compact fails, proceed to step 4 (SCB restore) or step 6 (chantools). Avoid resetting LND (step 7) unless absolutely necessary.

Thank you for the detailed steps. Due to the risk and effort I am leaning towards re-creating the wallet from seed phrase and channels from a backup file- then setup on another identical setup that is unused that I have available. Do you know where the latest recovery steps are documented?, this system is on Rasp pi4 OS 1.4

attempted the fix with bbolt. getting this module 5 issue. any thoughts? changed the “link” ,
would not let me post more than two .
umbrel@umbrel:~/go/pkg/mod$ go install goetcdio/bbolt/cmd/bbolt@latest
go: goetcdio/bbolt/cmd/bbolt@latest (in go.etcd.io/bbolt@v1.4.0): go.mod:5: unknown directive: toolchain

okay, crud, maybe an older go verion not working:

Steps to Install the bbolt CLI Tool

1. Verify and Remove Old Go Version

Umbrel’s Debian OS may have an outdated Go version installed (e.g., via apt-get install golang). Check and remove it to avoid conflicts:

go version sudo apt-get remove -y golang golang-go sudo rm -rf /usr/local/go

2. Install Go 1.22

The bbolt module at @latest (currently v1.4.0) requires Go 1.21+. Install Go 1.22 for compatibility and stability:

wget https://go.dev/dl/go1.22.8.linux-amd64.tar.gz sudo tar -C /usr/local -xzf go1.22.8.linux-amd64.tar.gz rm go1.22.8.linux-amd64.tar.gz

Add Go to your PATH by adding the following line to ~/.bashrc:

export PATH=$PATH:/usr/local/go/bin

Apply the change:

source ~/.bashrc

Verify the installation:

go version

You should see go version go1.22.8 linux/amd64 or similar.

3. Set Up GOPATH

The go install command installs binaries to $GOPATH/bin. Ensure $GOPATH is set and included in your PATH:

export GOPATH=$HOME/go export PATH=$PATH:$GOPATH/bin

Add these lines to ~/.bashrc to make them persistent:

echo 'export GOPATH=$HOME/go' >> ~/.bashrc echo 'export PATH=$PATH:$GOPATH/bin' >> ~/.bashrc source ~/.bashrc

4. Install the bbolt CLI Tool

Now install the bbolt CLI tool:

go install go.etcd.io/bbolt/cmd/bbolt@latest

Verify the installation:

bbolt --version

You should see output like bbolt version v1.4.0 (or the latest version). The binary is now located at ~/go/bin/bbolt.

5. Test Accessibility

Ensure you can run bbolt commands from any directory:

bbolt help

If this fails, double-check that ~/go/bin is in your PATH (echo $PATH should include /home/umbrel/go/bin).

Troubleshooting

  • Command Not Found: If bbolt isn’t found after installation, ensure $GOPATH/bin is in your PATH. Run:

export PATH=$PATH:$HOME/go/bin

and retry.

  • Permission Issues: If you get permission errors, ensure you have write access to ~/go/bin:

mkdir -p ~/go/bin chmod -R u+rw ~/go

also, this sop command may not work:

4. Rebuild the channel.db Using Static Channel Backup (SCB)

If the database cannot be repaired, …

  • Stop Umbrel:

sudo ~/umbrel/scripts/stop. <<<<<<<<<<<

Instead, use:

sudo systemctl stop umbrel-startup ## and start later on

Verify that the LND container is stopped:

docker ps

If the lightning_lnd_1 container is still running, stop it explicitly:

sudo docker stop lightning_lnd_1 

yep. seeing old go, will remove and install new .. thanks.

umbrel@umbrel:~/go/pkg/mod$ go version
go version go1.19.8 linux/arm64