Node "bricked"?

Hello everyone,

I was running my node on v.0.38 with a raspberry pi 4 with 4GB and a 1TB HDD. I updated it with the lastest version and since then I haven’t been able to connect to it, nor through the browser nor through ssh. In fact, it’s not just that I can’t connect to it, it doesn’t seem to be running at all. Up until that moment the read/write LED in the HDD was blinking at all times as long as the node was on, since the attempted update it doesn’t blink any more, so the Raspberry pi is not reading/writing anything.

I attempted to reboot it by unplugging and plugging the power source, didn’t solve the issue.

I tried to re-flash the sd card with the latest version and balena etcher wasn’t able to complete the validation step (it would flash successfully but then crash during validation, this happened several times). I nonetheless tried to put the sd back in the raspberry and switch on the node, and the same happened as before: the raspberry seems to start, all leds blink for a while (including the HDD) but after a minute it all goes dark and does nothing again. Not accessible via umbrel.local, neither with SSH.

Any idea how I can fix it?

I don’t mind to resync, altough it would be great if I didn’t have to, but most of all I want to have the node running again.

Thanks a lot!

Hey, sorry about that!

What hardware are you using (drive and power supply)? I would be useful for us if you were able to get the logs.

Open a terminal (Powershell on Windows, Terminal app on macOS) and execute:

ssh -t umbrel@umbrel.local "~/umbrel/scripts/debug --upload"

The prompted password is your dashboard password or “moneyprintergobrrr” (without quotes).

Hi Lounes!

thanks a lot for the quick reply.

I’m using an official raspberry pi power supply and as drive I have a 1TB HDD (Toshiba HDTB410EK3AA) connected via USB 3.0

I have tried the command you suggest in powershell, I get “ssh: could not resolve hostname umbrel.local: host unknown”.

I took care of switching off my VPN before, just in case.

Hi Aaron,

as I replied to Lounes, it did not work. My node is not reachable via ssh.

Is your computer well connected to the same local network as your Pi? Are you able to find the IP of your Umbrel using Angry IP Scanner?

Hi Lounes,

yes, the computer is well connected. In general, every single piece of hardware and connection at home is exactly in the same way as it has been in the last months while I was able to connect to my node without problems.

Angry IP Scanner does find the umbrel node in the network (I tried to attach a picture below, I hope you can see it).

However the node is non-responsive via ssh. Even if Angry IP scanner finds it the result of the ssh comand is still “ssh: could not resolve hostname umbrel.local: host unknown”

image

Hi Aaron,

as I described in my first post:

I attempted to reboot it by unplugging and plugging the power source, didn’t solve the issue.

I tried to re-flash the sd card with the latest version and balena etcher wasn’t able to complete the validation step (it would flash successfully but then crash during validation, this happened several times). I nonetheless tried to put the sd back in the raspberry and switch on the node, and the same happened as before: the raspberry seems to start, all leds blink for a while (including the HDD) but after a minute it all goes dark and does nothing again. Not accessible via umbrel.local, neither with SSH.

Hello,

I just wanted to give an update that tinkering a little with formating the SD card and balena I managed to 100% successfully reflash the SD card. It changed nothing in the situation, but at least now I know the problem is not with the OS flashed in the SD card.

My node is still not reachable in any way. Not by web, not by SSH. And even angry IP explorer doesn’t find it anymore. It’s like the node simply doesn’t boot.

Can anyone please help me?

Hi again,

so I just wanted to add some additional information after I have followed the steps in the [GUIDE] Umbrel.local - not accessible - what to do (awesome guide, by the way).

  • I entered my router configuration and the node was visible there. I assigned a fixed IP to the node.
  • I added the node IP to the hosts file
  • I tried to SSH again and got the info that the device was pinging back an ECDSA fingerprint that contradicted the info saved in the list of known hosts
  • I found said list of known hosts and deleted the content

And now I finally can ssh my node!!!

I executed the debug command, I paste below the result. Can you help me understand what is wrong?

=====================
= Umbrel debug info =

Umbrel version

0.3.10

Umbrel OS version

v0.3.10

Raspberry Pi Model

Revision : c03111
Serial : 10000000a53edea2
Model : Raspberry Pi 4 Model B Rev 1.1

Firmware

Feb 25 2021 12:10:40
Copyright © 2012 Broadcom
version 564e5f9b852b23a330b1764bcf0b2d022a20afd0 (clean) (release) (start)

Temperature

temp=37.9’C

Throttling

throttled=0x0

Memory usage

          total        used        free      shared  buff/cache   available

Mem: 3.8G 140M 3.2G 8.0M 475M 3.6G
Swap: 0B 0B 0B

total: 3.7%
system: 3.7%
tor: 0%
lnd: 0%
electrs: 0%
bitcoin: 0%

Filesystem information

Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 2.7G 26G 10% /
/dev/root 29G 2.7G 26G 10% /

Startup service logs

– Logs begin at Mon 2021-05-03 13:28:10 UTC, end at Thu 2021-05-13 16:55:48 UTC. –
May 03 13:28:17 umbrel systemd[1]: Dependency failed for Umbrel Startup Service.
May 03 13:28:17 umbrel systemd[1]: umbrel-startup.service: Job umbrel-startup.service/start failed with result ‘dependency’.

External storage service logs

– Logs begin at Mon 2021-05-03 13:28:10 UTC, end at Thu 2021-05-13 16:55:48 UTC. –
May 03 13:28:13 umbrel systemd[1]: Starting External Storage Mounter…
May 03 13:28:13 umbrel external storage mounter[352]: Running external storage mount script…
May 03 13:28:14 umbrel external storage mounter[352]: Found device “TOSHIBA External USB 3.0”
May 03 13:28:14 umbrel external storage mounter[352]: Blacklisting USB device IDs against UAS driver…
May 03 13:28:14 umbrel external storage mounter[352]: Rebinding USB drivers…
May 03 13:28:14 umbrel external storage mounter[352]: Checking USB devices are back…
May 03 13:28:14 umbrel external storage mounter[352]: Waiting for USB devices…
May 03 13:28:15 umbrel external storage mounter[352]: Waiting for USB devices…
May 03 13:28:16 umbrel external storage mounter[352]: Checking if the device is ext4…
May 03 13:28:16 umbrel external storage mounter[352]: Yes, it is ext4
May 03 13:28:17 umbrel external storage mounter[352]: mount: /mnt/data: mount(2) system call failed: Structure needs cleaning.
May 03 13:28:17 umbrel systemd[1]: umbrel-external-storage.service: Main process exited, code=exited, status=32/n/a
May 03 13:28:17 umbrel systemd[1]: umbrel-external-storage.service: Failed with result ‘exit-code’.
May 03 13:28:17 umbrel systemd[1]: Failed to start External Storage Mounter.

External storage SD card update service logs

– Logs begin at Mon 2021-05-03 13:28:10 UTC, end at Thu 2021-05-13 16:55:48 UTC. –
May 03 13:28:17 umbrel systemd[1]: Dependency failed for External Storage SDcard Updater.
May 03 13:28:17 umbrel systemd[1]: umbrel-external-storage-sdcard-update.service: Job umbrel-external-storage-sdcard-update.service/start failed with result ‘dependency’.

Karen logs

tail: cannot open ‘logs/karen.log’ for reading: No such file or directory

Docker containers

NAMES STATUS

Bitcoin Core logs

ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements

LND logs

ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements

Tor logs

ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements

==== Result ====

This script could not automatically detect an issue with your Umbrel.
Please share the following links and paste it in the Umbrel Telegram group (https://t.me/getumbrel) so we can help you with your problem.
tail: cannot open ‘logs/karen.log’ for reading: No such file or directory
ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements
ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements
ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements

1 Like

I have this same issue since update! whats going on people? A lot of nodes going down!

Possibly something to do with the HDD? doesn’t seem like the umbrel can access it for some reason?!

Can I get the blockchain download off the HDD? don’t really want to dl 350Gb again!

It worked, thank you!

2 Likes

NO way this worked… thanks so much!

2 Likes

Spoke too soon…kinda fixed it…

BTC core is 100% synced but BTC wallet and lightening wallet are not syncing…

where you able to get past this point? did you just wait?

Nope rebooted several times left for a few days nothing…wiped the drive and started again…360GB later it’s still acting up! My wallets arent synced and I can’t see the wallet address only the QR code…
It’s annoying because this is so close to being good!

cd ~/umbrel && sudo ./scripts/update/update --repo getumbrel/umbrel#v0.3.7

https://umbrelinfo.gitlab.io/troubleshooting.html

run that command after sudo fsck -y /dev/sda1 and sudo reboot

its worked for me twice. pretty much fixes anything ive messed up so far

I can’t connect to my SSD and the issue seems to be related to /dev/sda1

I posted in another thread before I saw your post. Running the debug script gives the following output:

Jul 29 21:16:09 umbrel external storage mounter[474]: Formatting device…
Jul 29 21:16:26 umbrel external storage mounter[474]: /dev/sda: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
Jul 29 21:16:26 umbrel external storage mounter[474]: /dev/sda: 8 bytes were erased at offset 0xee77a55e00 (gpt): 45 46 49 20 50 41 52 54
Jul 29 21:16:26 umbrel external storage mounter[474]: /dev/sda: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
Jul 29 21:16:26 umbrel external storage mounter[474]: /dev/sda: calling ioctl to re-read partition table: Success
Jul 29 21:19:00 umbrel external storage mounter[474]: Warning: Error fsyncing/closing /dev/sda: Input/output error
Jul 29 21:19:00 umbrel external storage mounter[474]: mke2fs 1.44.5 (15-Dec-2018)
Jul 29 21:19:00 umbrel external storage mounter[474]: The file /dev/sda1 does not exist and no size was specified.

I then tried doing an fsck of /dev/sda1 (recommended elsewhere) and got this error:

fsck.ext2: No such file or directory while trying to open /dev/sda1
Possibly non-existent device?

At this point I can’t go any further…

Anyone have any suggestions for what might be done? Thanks!

Now fixed. See here for details: Failed to connect external drive (error on power up)