I’ve been running Umbrel on a Raspberry Pi (made by The Bitcoin Machines) since about April with no issues, including updating to 0.5.1. It’s connected via Ethernet to my local network, and has a 1 TB SSD with plenty of space.
A few weeks ago, I lost web and SSH access to the node.
- On the browser, I get “Error: System service failed.”
- When I try to login via SSH on PowerShell (with Admin privileges), it returns “Connection closed by 2604:3d08:2c7e:1300:540f:8bcc:c82f:6fcc port 22”.
The node itself appears to be working fine: the screen shows the current BTC price and transaction costs. I can ping it no problem. It shows up as “umbrel” on my internet router. I just can’t log in to do any diagnostics.
Things I’ve tried:
- Setting a static IP -> same error
- Deleting my Windows .ssh folder and trying to SSH in -> same error
-
ssh -t umbrel[at]umbrel.local “~/umbrel/scripts/debug” -> same error
- Tried to SSH using local IP address and IP address in the above error -> same error
- ssh -v -v -v -v umbrel[at]umbrel.local -> The resulting log below.
What else should I try? Reflash the SD card even though the node appears to work fine under the hood? Something else?
This is the log for the SSH -v -v -v -v command:
Similar. 5.1 worked fine. Updated from 5.1 to 5.3 today and now no web or ssh access 
I (partially) resolved this issue by flashing 0.5.1 directly to the SD using Balena Etcher. After that, I can now SSH into Umbrel using the default password. I later flashed 0.5.2, and I can SSH into that no problem as well.
However, the web interface still shows “System service failed”. Looking into the debug logs, it first seemed like I was having problems with the docker daemon:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
…but looking more closely, the root cause of the problem seems to be that the 1 TB SSD is completely full:
– Logs begin at Mon 2022-10-31 18:14:36 PDT, end at Mon 2022-10-31 18:20:49 PDT. –
Oct 31 18:14:41 umbrel systemd[1]: Starting External Storage Mounter…
Oct 31 18:14:41 umbrel external storage mounter[562]: Running external storage mount script…
Oct 31 18:14:41 umbrel external storage mounter[562]: Found device "JMicron "
Oct 31 18:14:41 umbrel external storage mounter[562]: Blacklisting USB device IDs against UAS driver…
Oct 31 18:14:41 umbrel external storage mounter[562]: Rebinding USB drivers…
Oct 31 18:14:41 umbrel external storage mounter[562]: Checking USB devices are back…
Oct 31 18:14:41 umbrel external storage mounter[562]: Waiting for USB devices…
Oct 31 18:14:42 umbrel external storage mounter[562]: Waiting for USB devices…
Oct 31 18:14:43 umbrel external storage mounter[562]: Waiting for USB devices…
Oct 31 18:14:44 umbrel external storage mounter[562]: Checking if the device is ext4…
Oct 31 18:14:44 umbrel external storage mounter[562]: Yes, it is ext4
Oct 31 18:14:44 umbrel external storage mounter[562]: Checking filesystem for corruption…
Oct 31 18:14:44 umbrel external storage mounter[562]: e2fsck 1.44.5 (15-Dec-2018)
Oct 31 18:14:45 umbrel external storage mounter[562]: umbrel: clean, 1474390/61054976 files, 244186112/244190208 blocks
Oct 31 18:14:45 umbrel external storage mounter[562]: Mounting partition…
Oct 31 18:14:45 umbrel external storage mounter[562]: Checking if device contains an Umbrel install…
Oct 31 18:14:45 umbrel external storage mounter[562]: Yes, it contains an Umbrel install
Oct 31 18:14:45 umbrel external storage mounter[562]: Bind mounting external storage over local Umbrel installation…
Oct 31 18:14:45 umbrel external storage mounter[562]: Bind mounting external storage over local Docker data dir…
Oct 31 18:14:45 umbrel external storage mounter[562]: Bind mounting external storage to /swap
Oct 31 18:14:45 umbrel external storage mounter[562]: Bind mounting SD card root at /sd-card…
Oct 31 18:14:45 umbrel external storage mounter[562]: Checking Umbrel root is now on external storage…
Oct 31 18:14:46 umbrel external storage mounter[562]: Checking /var/lib/docker is now on external storage…
Oct 31 18:14:46 umbrel external storage mounter[562]: Checking /swap is now on external storage…
Oct 31 18:14:46 umbrel external storage mounter[562]: Setting up swapfile
Oct 31 18:14:49 umbrel external storage mounter[562]: fallocate: fallocate failed: No space left on device
Oct 31 18:14:49 umbrel systemd[1]: umbrel-external-storage.service: Main process exited, code=exited, status=1/FAILURE
Oct 31 18:14:49 umbrel systemd[1]: umbrel-external-storage.service: Failed with result ‘exit-code’.
Oct 31 18:14:49 umbrel systemd[1]: Failed to start External Storage Mounter.
I’ve found a couple of threads on this forum for the same issue, and I gather the problem is often a rogue log file filling up the SSD, but I haven’t been able to find the commands to get into the SSD and check.