Error: system service failed - debug interpretation

Support and Troubleshooting
1

Hi all,

I am receiving the above error with the following debug log. I am not an expert but wanted to confirm whether my assessment is right that my device is full? What’s the appropriate solution if so? Just get a new 2 TB SSD and re-flash?

=====================
= Umbrel debug info =

Umbrel version

0.5.3

Flashed OS version

v0.3.10

Raspberry Pi Model

Revision : c03112
Serial : 1000000037104886
Model : Raspberry Pi 4 Model B Rev 1.2

Firmware

Feb 25 2021 12:10:40
Copyright © 2012 Broadcom
version 564e5f9b852b23a330b1764bcf0b2d022a20afd0 (clean) (release) (start)

Temperature

temp=41.8’C

Throttling

throttled=0x0

Memory usage

          total        used        free      shared  buff/cache   available

Mem: 3.8G 155M 3.2G 40M 415M 3.5G
Swap: 4.1G 0B 4.1G

total: 4.1%
system: 4.1%
total: 4.1%
system: 4.1%
lightning: 0%
bitcoin: 0%

Memory monitor logs

2019-02-14 10:24:09 Memory monitor running!
2019-02-14 10:24:23 Memory monitor running!
2019-02-14 10:24:36 Memory monitor running!
2019-02-14 10:24:50 Memory monitor running!
2019-02-14 10:25:03 Memory monitor running!
2019-02-14 10:25:16 Memory monitor running!
2019-02-14 10:25:30 Memory monitor running!
2019-02-14 10:25:43 Memory monitor running!
2019-02-14 10:25:56 Memory monitor running!
2019-02-14 10:26:10 Memory monitor running!

Filesystem information

Filesystem Size Used Avail Use% Mounted on
/dev/root 59G 59G 0 100% /
/dev/sda1 916G 562G 309G 65% /home/umbrel/umbrel

Startup service logs

Feb 14 10:26:09 umbrel umbrel startup[7138]: Starting status monitors…
Feb 14 10:26:09 umbrel umbrel startup[7138]: Starting memory monitor…
Feb 14 10:26:09 umbrel umbrel startup[7138]: Starting backup monitor…
Feb 14 10:26:09 umbrel umbrel startup[7138]: Starting decoy backup trigger…
Feb 14 10:26:09 umbrel umbrel startup[7138]: Starting Docker services…
Feb 14 10:26:12 umbrel umbrel startup[7138]: Starting dashboard …
Feb 14 10:26:12 umbrel umbrel startup[7138]: Starting manager …
Feb 14 10:26:12 umbrel umbrel startup[7138]: Starting tor_proxy …
Feb 14 10:26:12 umbrel umbrel startup[7138]: Starting tor_server …
Feb 14 10:26:12 umbrel umbrel startup[7138]: Starting auth …
Feb 14 10:26:12 umbrel umbrel startup[7138]: Starting dashboard … error
Feb 14 10:26:13 umbrel umbrel startup[7138]: ERROR: for dashboard Cannot start service dashboard: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/113e98074074d3eab07f4a54ca859c7fcf8d90f8418df58b3962ea6d88ec3b76: no space left on device: unknown
Feb 14 10:26:13 umbrel umbrel startup[7138]: Starting manager … error
Feb 14 10:26:13 umbrel umbrel startup[7138]: ERROR: for manager Cannot start service manager: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/0a76a56a68120be8a8a8ad5ac767ab17133da676bec2d57b616205387ae138a7: no space left on device: unknown
Feb 14 10:26:13 umbrel umbrel startup[7138]: Starting tor_proxy … error
Feb 14 10:26:13 umbrel umbrel startup[7138]: ERROR: for tor_proxy Cannot start service tor_proxy: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/ec4c0d46948248632fd6c25b8a63e5d6b95dc096df600049a79fc33fbc1706d8: no space left on device: unknown
Feb 14 10:26:13 umbrel umbrel startup[7138]: Starting tor_server … error
Feb 14 10:26:13 umbrel umbrel startup[7138]: ERROR: for tor_server Cannot start service tor_server: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/ebd6e401e11b22235d2dc9c48a0cbbd25efb827a665a243c0cc78659cf6de0d6: no space left on device: unknown
Feb 14 10:26:13 umbrel umbrel startup[7138]: Starting auth … error
Feb 14 10:26:13 umbrel umbrel startup[7138]: ERROR: for auth Cannot start service auth: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/fd5f55afcac5268a12db62d78549d526ac60e4c172148ff3af0dd65c77d475cb: no space left on device: unknown
Feb 14 10:26:13 umbrel umbrel startup[7138]: ERROR: for dashboard Cannot start service dashboard: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/113e98074074d3eab07f4a54ca859c7fcf8d90f8418df58b3962ea6d88ec3b76: no space left on device: unknown
Feb 14 10:26:13 umbrel umbrel startup[7138]: ERROR: for manager Cannot start service manager: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/0a76a56a68120be8a8a8ad5ac767ab17133da676bec2d57b616205387ae138a7: no space left on device: unknown
Feb 14 10:26:13 umbrel umbrel startup[7138]: ERROR: for tor_proxy Cannot start service tor_proxy: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/ec4c0d46948248632fd6c25b8a63e5d6b95dc096df600049a79fc33fbc1706d8: no space left on device: unknown
Feb 14 10:26:13 umbrel umbrel startup[7138]: ERROR: for tor_server Cannot start service tor_server: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/ebd6e401e11b22235d2dc9c48a0cbbd25efb827a665a243c0cc78659cf6de0d6: no space left on device: unknown
Feb 14 10:26:13 umbrel umbrel startup[7138]: ERROR: for auth Cannot start service auth: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/fd5f55afcac5268a12db62d78549d526ac60e4c172148ff3af0dd65c77d475cb: no space left on device: unknown
Feb 14 10:26:13 umbrel umbrel startup[7138]: Encountered errors while bringing up the project.
Feb 14 10:26:13 umbrel umbrel startup[7138]: Failed to start containers
Feb 14 10:26:13 umbrel systemd[1]: umbrel-startup.service: Control process exited, code=exited, status=1/FAILURE
Feb 14 10:26:13 umbrel systemd[1]: umbrel-startup.service: Failed with result ‘exit-code’.
Feb 14 10:26:13 umbrel systemd[1]: Failed to start Umbrel Startup Service.

External storage service logs

Feb 14 10:12:03 umbrel external storage mounter[500]: Rebinding USB drivers…
Feb 14 10:12:03 umbrel external storage mounter[500]: Checking USB devices are back…
Feb 14 10:12:03 umbrel external storage mounter[500]: Waiting for USB devices…
Feb 14 10:12:04 umbrel external storage mounter[500]: Waiting for USB devices…
Feb 14 10:12:05 umbrel external storage mounter[500]: Checking if the device is ext4…
Feb 14 10:12:05 umbrel external storage mounter[500]: Yes, it is ext4
Feb 14 10:12:05 umbrel external storage mounter[500]: Checking filesystem for corruption…
Feb 14 10:12:05 umbrel external storage mounter[500]: e2fsck 1.44.5 (15-Dec-2018)
Feb 14 10:12:06 umbrel external storage mounter[500]: Superblock last mount time is in the future.
Feb 14 10:12:06 umbrel external storage mounter[500]: (by less than a day, probably due to the hardware clock being incorrectly set)
Feb 14 10:12:06 umbrel external storage mounter[500]: Superblock last write time is in the future.
Feb 14 10:12:06 umbrel external storage mounter[500]: (by less than a day, probably due to the hardware clock being incorrectly set)
Feb 14 10:12:06 umbrel external storage mounter[500]: umbrel: clean, 180914/61054976 files, 151162952/244190208 blocks
Feb 14 10:12:06 umbrel external storage mounter[500]: Mounting partition…
Feb 14 10:12:06 umbrel external storage mounter[500]: Checking if device contains an Umbrel install…
Feb 14 10:12:06 umbrel external storage mounter[500]: Yes, it contains an Umbrel install
Feb 14 10:12:06 umbrel external storage mounter[500]: Bind mounting external storage over local Umbrel installation…
Feb 14 10:12:06 umbrel external storage mounter[500]: Bind mounting external storage over local Docker data dir…
Feb 14 10:12:06 umbrel external storage mounter[500]: Bind mounting external storage to /swap
Feb 14 10:12:06 umbrel external storage mounter[500]: Bind mounting SD card root at /sd-card…
Feb 14 10:12:06 umbrel external storage mounter[500]: Checking Umbrel root is now on external storage…
Feb 14 10:12:07 umbrel external storage mounter[500]: Checking /var/lib/docker is now on external storage…
Feb 14 10:12:07 umbrel external storage mounter[500]: Checking /swap is now on external storage…
Feb 14 10:12:07 umbrel external storage mounter[500]: Setting up swapfile
Feb 14 10:12:07 umbrel external storage mounter[500]: Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
Feb 14 10:12:07 umbrel external storage mounter[500]: no label, UUID=34590490-55de-4c4d-a606-3f55f914a1b8
Feb 14 10:12:07 umbrel external storage mounter[500]: Checking SD Card root is bind mounted at /sd-root…
Feb 14 10:12:07 umbrel external storage mounter[500]: Starting external drive mount monitor…
Feb 14 10:12:07 umbrel external storage mounter[500]: Mount script completed successfully!
Feb 14 10:12:07 umbrel systemd[1]: Started External Storage Mounter.

External storage SD card update service logs

Feb 14 10:12:17 umbrel external storage updater[1018]: == Begin Update Script 01-run.sh ==
Feb 14 10:12:17 umbrel external storage updater[1018]: =======================================
Feb 14 10:12:17 umbrel external storage updater[1018]: =============== UPDATE ================
Feb 14 10:12:17 umbrel external storage updater[1018]: =======================================
Feb 14 10:12:17 umbrel external storage updater[1018]: =========== Stage: Install ============
Feb 14 10:12:17 umbrel external storage updater[1018]: =======================================
Feb 14 10:12:17 umbrel external storage updater[1018]: =============================================
Feb 14 10:12:17 umbrel external storage updater[1018]: Installing on Umbrel OS v0.3.10
Feb 14 10:12:17 umbrel external storage updater[1018]: =============================================
Feb 14 10:12:17 umbrel external storage updater[1018]: ./01-run.sh: line 25: cannot create temp file for here-document: No space left on device
Feb 14 10:12:17 umbrel external storage updater[1018]: == End Update Script 01-run.sh ==
Feb 14 10:12:17 umbrel external storage updater[1018]: == Begin Update Script 02-run.sh ==
Feb 14 10:12:17 umbrel external storage updater[1018]: =======================================
Feb 14 10:12:17 umbrel external storage updater[1018]: =============== UPDATE ================
Feb 14 10:12:17 umbrel external storage updater[1018]: =======================================
Feb 14 10:12:17 umbrel external storage updater[1018]: ========= Stage: Post-update ==========
Feb 14 10:12:17 umbrel external storage updater[1018]: =======================================
Feb 14 10:12:17 umbrel external storage updater[1018]: == End Update Script 02-run.sh ==
Feb 14 10:12:17 umbrel external storage updater[1018]: == Begin Update Script 03-run.sh ==
Feb 14 10:12:17 umbrel external storage updater[1018]: =======================================
Feb 14 10:12:17 umbrel external storage updater[1018]: =============== UPDATE ================
Feb 14 10:12:17 umbrel external storage updater[1018]: =======================================
Feb 14 10:12:17 umbrel external storage updater[1018]: =========== Stage: Success ============
Feb 14 10:12:17 umbrel external storage updater[1018]: =======================================
Feb 14 10:12:17 umbrel external storage updater[1018]: Removing backup
Feb 14 10:12:17 umbrel external storage updater[1018]: ./03-run.sh: line 19: cannot create temp file for here-document: No space left on device
Feb 14 10:12:17 umbrel external storage updater[1018]: == End Update Script 03-run.sh ==
Feb 14 10:12:17 umbrel external storage updater[1018]: Deleting cloned repository
Feb 14 10:12:17 umbrel external storage updater[1018]: Removing lock
Feb 14 10:12:17 umbrel systemd[1]: Started External Storage SDcard Updater.

2

it looks like its trying to write to your sd card instead of your ssd

3

Do you have any idea how to fix it?

Afaik I did not change anything, just noticed that my umbrel dashboard wasn’t loading.

4

Bumping this up again as I am not finding any solution. Would you recommend getting new hardware to try out fixing it?

5

by the way when SSHing into the umbrel and trying to access lnd or bitcoind, I get the following feedback:

/home/umbrel/umbrel/scripts/app: line 79: NETWORK_IP: unbound variable
umbrel@umbrel:~ $ sudo ~/umbrel/scripts/app compose lightning exec lnd lndcli getinfo
/home/umbrel/umbrel/scripts/app: line 79: NETWORK_IP: unbound variable
umbrel@umbrel:~ $ sudo ~/umbrel/scripts/app compose bitcoin exec bitcoind bitcoin-cli getblockchaininfo
/home/umbrel/umbrel/scripts/app: line 79: NETWORK_IP: unbound variable

6

Found the solution. The SD card was broken. I noticed (through another thread), that the green LED was not on when the node was on but I could SSH into the umbrel.

Solution: Re-flashed umbrelOS onto a new 16GB SD card and could log back on.

Curiously, the bitcoin and lnd interface do not work/load in the GUI but I will investigate this further once I have more time.