Repair suggestion

[Green2J]

I made a previous post (if anyone cares to look), and I have determined I will start replacing stuff. My question is, having a synced node before that has lightning channels and funds on it, should I replace the hard drive or the actual Pi first? And if the Pi should be replaced, should I go with the Pi, or try one of the other machines (like the Orange Pi)?

[Green2J]

To clarify, I am hesitant to replace the hard drive because I am not sure I would be able to get my old node info back - can anyone alleviate my fears about this?

[chiptuner]

So I was following your old thread. So pleas help my clarify your setup.

You are using a Rpi 4 correct?
You have an SD card for umbrelOS installed?
You have an external drive storing all your data?

If I have the above correct, you mentioned your SD card was giving you issues right? If you are using the recommended setup for your external drive, then your node data should be safe. I will confirm that for you but that should be the case!

[Green2J]

I am using a RPI 4, replaced the SD card multiple times, and yes I have an external SSD…I can’t be sure that it is the SD card that is giving me issues; I have reflashed and replaced it multiple times

[chiptuner]

When you say “replaced the SD card multiple times” do you mean the issue with your bitcoin node synchronization is the only issue that seems to be related to the SD card, and it seems to reappear after some time?

[Green2J]

It seems like it isn’t the sd card…im not sure but I’m sure since I’ve changed it multiple times, it wouldn’t be that…that’s why I’m thinking it would be the Pi or the SSD itself

[chiptuner]

Since your node’s data should be storing data on your external storage using bind mounts I’d probably be more concerned with that storage for issues with node performance. It may be worth sharing system logs to see if you can find any IO/Disk related errors.

[Green2J]

Shouldn’t I be able to disconnect my SSD from the node and hook it to my computer and run CHKDSK (sp?) to identify any errors? I’m not comfortable posting my logs since I can’t seem to locate all instances of my onion address

[Green2J]

And if it is the SSD, replacing it means what exactly for all my previous node data (and funds)?

[chiptuner]

Entirely understandable! All of your node data should be stored in your SSD. You should be able to list your mounts to see where everything is located on your SSD’s filesystem. After that you can backup everything safely. What lightning node app are you using? I’m assuming it’s one on LND?

[Green2J]

I just have the regular lightning node (purple background with the lightning bolt in the center) I got from the Umbrel Store. Can you explain “listing my mounts”? That’s why I’m hesitant to replace the SSD…it is working SOMEWHAT, but it just gets to a point and stops doing whatever it is supposed to do (syncing). I’d need to get the data off of it first before I’d be comfortable with replacing it…at least that’s my hope

[chiptuner]

Yeah sorry, Lightning Node

Can you explain “listing my mounts”?

Yes, so you run a command from your terminal called lsblk which will show what partition’s are mapped to a given path.

It will look kind of like this.

umbrel@umbrel:~$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda      8:0    0   24G  0 disk
|-sda1   8:1    0  200M  0 part /mnt/root/boot/efi
|                               /boot/efi
|-sda2   8:2    0  9.8G  0 part /mnt/root
|                               /
|-sda3   8:3    0  9.8G  0 part
`-sda4   8:4    0  4.1G  0 part /mnt/root/var/log
                                /var/log
                                /mnt/root/var/lib/systemd/timesync
                                /var/lib/systemd/timesync
                                /mnt/root/var/lib/docker
                                /var/lib/docker
                                /mnt/root/home
                                /home
                                /mnt/root/data
                                /data

This will show you any mount points that are aliased to a given partition. This may help you narrow down where your Lighting Node data is stored, so you can properly create a backup. Remember all apps on umbrel are containerized so as long as these mounts point the same place on another drive, all of your node data will be restored.

I’m not sure if any of this was too advanced or I might be too confusing, so let me know if and where I lose you!

[Green2J]

So I turned off my node and connected my ssd to my laptop; what kind of tests can I run to see if it is still viable? In Disk Management, it says it is a healthy partition (I can’t run CHKDSK because it doesn’t have a drive letter)

[chiptuner]

First, what OS is your laptop running? Were you able to successfully mount it to your laptop?

I would suggest starting with taking backup as soon as you can access the filesystem before you do any kind of work on the disk.

[Green2J]

I am on Windows 11…can you go more in depth of how to do a backup? Or point me to a thread that explains it?

[chiptuner]

I found a couple threads that might be relevant in understanding lightning and btc related states

What are the recommended backup procedures?

No worries and, correct, you will be starting again.

You should be able to mount your SSD and all of it’s partitions, then manually copy your files. It’s not ideal if you have a lot on the disk, or your disk has many partitions. Without knowing your lsblk I can’t hold your hand.

The important thing is that you can start by mounting your drive in Windows. Unfortunately I have not had to use a Windows 11 machine, so I’m going to give some instruction based off Windows 10 and hopefully they haven’t changed it too much.

You should be able to do the following

• press windows key + x (or run and type diskmgmt.msc and hit enter button)
• Click Disk Management
• You should see your SSD appear on the bottom pane on the left hand side, it will probably show a little red icon saying the drive could not be mounted.
• Right click, on it and an option should be to mount the disk
• Do not click any buttons that suggest formatting or deleting your volume This is very important, if Windows cannot mount the drive do not force it because you may accidentally delete everything.

Start here, make sure you can mount your disk, add a volume label to it and you can start accessing the drive. If the drive successfully mounts follow these steps if a volume label isn’t done automatically.

• Right click a partition on your drive
• Click Change Drive Letter and Paths
• Assign a volume label

I want to repeat, do not under any circumstances click okay on any prompt that suggests formatting or deleting a volume or partition.

This thread is relevant but it’s probably a bit outdated. It’s is relevant because a block-level copy is important if you choose to migrate to a new SSD in the event yours is in fact failing. I’m not the biggest fan telling people to install of Freemium Windows software.

[GUIDE] Cloning or Migrating SSD/HDD (Windows)

This guide is for anyone who needs to migrate from an HDD to a SSD, or vice-versa… You can also clone the drive to move it to a bigger drive. Currently I only have Windows instructions. So if anyone would like to make instructions for Linux, or Mac, feel free to do so. TL;DR: version – scroll all the way down. Things needed: Windows PC (of course.) 2 USB 3.0 enclosures or whatever interfaces/enclosures you use for your drives. The drive you’re migrating/cloning from (any size) The drive y…

Let me know if and where you get stuck, feel free to ask for any explanations here.

[Green2J]

So I got as far as being able to “see” my SSD in disk management, but the option to change the drive letter and path is greyed out…the ONLY option available to me is to delete the volume…lol…go figure

[chiptuner]

Okay, so at some point I though I read that Windows added support for ext4 drives, that seems to be incorrect, so that’s why you can’t mount it. However you may be able to create a Linux VM and add this disk using Hyper-V.

Hyper-V disk example

If you want to follow up, this SuperUser thread has quite a few options that might work for you, none of which I have tested or used myself.

Can we start with a few things. So I went back and reread your other thread, and I can’t see any mention where you had any evidence of disk failure. Is that correct? Maybe we should try troubleshooting your disk issues instead? You should be looking at your system logs to see if you can find any evidence of disk related errors. You should be able to share your system logs without leaking sensitive data if you want to. I also updated my bio to include my PGP public key if you want to encrypt them and send it to me.

Otherwise maybe consider taking a look at this thread for some info on checking the disk health on Linux (loading umbrelOS back up and installing your disk again). I cannot fully endorse this link as I have little experience with disk testing on Linux, so use your best judgment.

[Green2J]

When I was in the telegram group, I posted my logs and they said it COULD be my hard drive (after changing out the SD card). I tried to post my logs but apparently my onion address was in there (I tried to search what I thought was my onion address, but it didn’t find any instances of it, so I assume it wasn’t it). I’m not opposed to spinning up a virtual machine…it’s something I need to get more practice with anyway. I’ll send you my logs though so that you can look at it.

[chiptuner]

Okay, maybe we just went down a bit of a rabbit hole. Thanks for the logs, I took a look and couldn’t find anything that signaled a clear disk related issue. As far as your bitcoin node is concerned, I noticed the bitcoin node crashed because if a corrupted index, which can happen for many reasons. I think you should be able to set your node to re-index and let it chew through it. I think I may have been too advanced with my instructions, so I will try to meet you where you’re at!

However I learned that the only thing you really need to copy in a backup is your lighting node data. Your bitcoin node can just re-download and resync in the future if you ever need to recover.

If you aren’t using a recommended RaspberryPi power supply it’s recommended to get that swapped out because power issues can appear with external storage.

Re-index

Sometimes the bitcoin-core software can get out of sync with it’s block data and indexes, you can tell to re-index it’s blocks to recover from a corrupted index like this.

1. Stop your bitcoin & lighting node apps
2. Go to Settings > Advanced Settings > Terminal
3. Type sudo nano umbrel/app-data/bitcoin/data/bitcoin/bitcoin.conf
4. Add reindex=1 at the end of the file
5. Restart your bitcoin node and monitor your log file again
6. After a re-index has completed, stop your node again, re-open the bitcoin.conf file again and remove the reindex=1 from the file and restart your node again

Backup lighting

1. Stop your lighting node
2. Go to Settings > Advanced Settings > Terminal
3. Run sudo tar -czf ln-backup.tgz umbrel/app-data/lightning/ (creates a tar archive copy of all your files into a single file, sort of like .zip on Windows)
4. Copy the ln-backup.tgz file off the machine

Getting your backup file off the machine

1. If you have another Linux computer that you can ssh into you can use scp
2. You can install a usb flash-drive with enough space to copy to it, but that might be a bit more involved on your terminal
3. You can install the File Browser app

Using file browser

1. Install the app from the Umbrel app store
2. Open your terminal again: Settings > Advanced Settings > Terminal
3. run: mv ln-backup.tgz umbrel/data/storage/downloads/
4. Open File Browser in your web browser and download the ln-backup.tgz file

Windows now supports tar too if you ever need to unpack your archive and poke around. Make sure your store that file in a safe place as you wouldn’t want to leak that or have it stolen.

Untar: ‘tar -xzf ln-backup.tgz’

Tar will preserve all of the folders in the path too, so if you don’t want that, you can cd umbrel/app-data/lighting and then use tar -czf ln-backup.tgz . So whatever folder you take the tar from, make sure you are in the same place when you un-tar it!