No space left on device

Support and Troubleshooting
1

Hi everyone,

first post here and requesting support or help.

Umbrel stopped booting. I tried to stop scripts, got this warnings that ended with:
“Defaulting to a blank string”

When asking to reconfigure possibly broken container, got this message:
afbeelding

So I asked for a debug log. This called my attention as well. Possibly when things has gone south:

Memory monitor logs

2023-05-26 21:16:34 Warning memory usage at 91%
2023-05-26 22:48:36 Warning memory usage at 91%
2023-05-27 00:07:10 Warning memory usage at 94%
2023-05-27 00:15:16 Warning memory usage at 94%
2023-06-02 16:02:37 Warning memory usage at 92%
2023-06-05 12:54:46 Warning memory usage at 92%
2023-06-05 12:57:31 Warning memory usage at 91%
2019-02-14 10:13:07 Memory monitor running!
2019-02-14 10:13:22 Memory monitor running!
2019-02-14 10:13:37 Memory monitor running!

I flashed a new card and that didn’t work. The new system couldn’t recognize the hard disk. Dates on SSD folders are also changed to 2019(?)

Any suggestion on how to recover it?

Here the full log:

=====================
= Umbrel debug info =

Umbrel version

0.5.3

Flashed OS version

v0.4.4

Raspberry Pi Model

Revision : d03114
Serial : 10000000263003c6
Model : Raspberry Pi 4 Model B Rev 1.4

Firmware

Sep 28 2021 11:27:35
Copyright (c) 2012 Broadcom
version 778b6a4f3c7d8d48bb63c02c47bcfbac79417bea (clean) (release) (start)

Temperature

temp=40.4’C

Throttling

throttled=0x0

Memory usage

          total        used        free      shared  buff/cache   available

Mem: 7.8G 180M 7.3G 8.0M 362M 7.5G
Swap: 4.1G 0B 4.1G

total: 2.3%
system: 2.3%
thunderhub: 0%
samourai-server: 0%
ride-the-lightning: 0%
pi-hole: 0%
passky-server: 0%
passky-client: 0%
mempool: 0%
lnplus: 0%
lightning-terminal: 0%
lightning: 0%
electrs: 0%
core-lightning-rtl: 0%
core-lightning: 0%
btc-rpc-explorer: 0%
bitfeed: 0%
bitcoin: 0%

Memory monitor logs

2023-05-26 21:16:34 Warning memory usage at 91%
2023-05-26 22:48:36 Warning memory usage at 91%
2023-05-27 00:07:10 Warning memory usage at 94%
2023-05-27 00:15:16 Warning memory usage at 94%
2023-06-02 16:02:37 Warning memory usage at 92%
2023-06-05 12:54:46 Warning memory usage at 92%
2023-06-05 12:57:31 Warning memory usage at 91%
2019-02-14 10:13:07 Memory monitor running!
2019-02-14 10:13:22 Memory monitor running!
2019-02-14 10:13:37 Memory monitor running!

Filesystem information

Filesystem Size Used Avail Use% Mounted on
/dev/root 15G 15G 0 100% /
/dev/sda1 916G 626G 244G 72% /home/umbrel/umbrel

Startup service logs

Feb 14 10:13:26 umbrel umbrel startup[5241]: ERROR: for tor_server Cannot start service tor_server: failed to start shim : mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/b2f33b090a903e1172b2dc95f1e4fb73cc2ce9e0cb6c3b6c1f215f1c3e 03be79: no space left on device: unknown
Feb 14 10:13:26 umbrel umbrel startup[5241]: Starting dashboard … error
Feb 14 10:13:26 umbrel umbrel startup[5241]: ERROR: for dashboard Cannot start service dashboard: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/1832d677188768f2107deb5ce40cd50cab134efca83b8d325e3171ea6b5b cf3e: no space left on device: unknown
Feb 14 10:13:26 umbrel umbrel startup[5241]: Starting auth … error
Feb 14 10:13:26 umbrel umbrel startup[5241]: ERROR: for auth Cannot start service auth: failed to start shim: mkdir /var /lib/containerd/io.containerd.runtime.v2.task/moby/c9e7e0165611d6976ac38ecac3fd52429a8f08c06cad59f16af71a4da63256ff: no s pace left on device: unknown
Feb 14 10:13:26 umbrel umbrel startup[5241]: ERROR: for tor_proxy Cannot start service tor_proxy: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/afed7929885584a221ca83d1a59615bf3894cabcaf2e5571640234328b39 d82a: no space left on device: unknown
Feb 14 10:13:26 umbrel umbrel startup[5241]: ERROR: for manager Cannot start service manager: failed to start shim: mkdi r /var/lib/containerd/io.containerd.runtime.v2.task/moby/fee176559e5ba2302bc76d49e4d85801ca5a56bb3d2036acb4bc1c2312fe0364 : no space left on device: unknown
Feb 14 10:13:26 umbrel umbrel startup[5241]: ERROR: for tor_server Cannot start service tor_server: failed to start shim : mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/b2f33b090a903e1172b2dc95f1e4fb73cc2ce9e0cb6c3b6c1f215f1c3e 03be79: no space left on device: unknown
Feb 14 10:13:26 umbrel umbrel startup[5241]: ERROR: for dashboard Cannot start service dashboard: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/1832d677188768f2107deb5ce40cd50cab134efca83b8d325e3171ea6b5b cf3e: no space left on device: unknown
Feb 14 10:13:26 umbrel umbrel startup[5241]: ERROR: for auth Cannot start service auth: failed to start shim: mkdir /var /lib/containerd/io.containerd.runtime.v2.task/moby/c9e7e0165611d6976ac38ecac3fd52429a8f08c06cad59f16af71a4da63256ff: no s pace left on device: unknown
Feb 14 10:13:26 umbrel umbrel startup[5241]: Encountered errors while bringing up the project.
Feb 14 10:13:26 umbrel umbrel startup[5241]: Failed to start containers
Feb 14 10:13:26 umbrel systemd[1]: umbrel-startup.service: Control process exited, code=exited, status=1/FAILURE
Feb 14 10:13:26 umbrel systemd[1]: umbrel-startup.service: Failed with result ‘exit-code’.
Feb 14 10:13:26 umbrel systemd[1]: Failed to start Umbrel Startup Service.
Feb 14 10:13:36 umbrel systemd[1]: umbrel-startup.service: Service RestartSec=10s expired, scheduling restart.
Feb 14 10:13:36 umbrel systemd[1]: umbrel-startup.service: Scheduled restart job, restart counter is at 2.
Feb 14 10:13:36 umbrel systemd[1]: Stopped Umbrel Startup Service.
Feb 14 10:13:36 umbrel systemd[1]: Starting Umbrel Startup Service…
Feb 14 10:13:36 umbrel umbrel startup[6748]: ======================================
Feb 14 10:13:36 umbrel umbrel startup[6748]: ============= STARTING ===============
Feb 14 10:13:36 umbrel umbrel startup[6748]: ============== UMBREL ================
Feb 14 10:13:36 umbrel umbrel startup[6748]: ======================================
Feb 14 10:13:36 umbrel umbrel startup[6748]: Setting environment variables…
Feb 14 10:13:36 umbrel umbrel startup[6748]: Starting karen…
Feb 14 10:13:36 umbrel umbrel startup[6748]: Starting status monitors…
Feb 14 10:13:36 umbrel umbrel startup[6748]: Starting memory monitor…
Feb 14 10:13:36 umbrel umbrel startup[6748]: Starting backup monitor…
Feb 14 10:13:36 umbrel umbrel startup[6748]: Starting decoy backup trigger…
Feb 14 10:13:36 umbrel umbrel startup[6748]: Starting Docker services…

External storage service logs

Feb 14 10:12:14 umbrel external storage mounter[531]: now = Thu Feb 14 10:12:07 2019) is in the future.
Feb 14 10:12:14 umbrel external storage mounter[531]: Fix? yes
Feb 14 10:12:14 umbrel external storage mounter[531]: Pass 1: Checking inodes, blocks, and sizes
Feb 14 10:12:23 umbrel external storage mounter[531]: Pass 2: Checking directory structure
Feb 14 10:12:28 umbrel external storage mounter[531]: Pass 3: Checking directory connectivity
Feb 14 10:12:28 umbrel external storage mounter[531]: Pass 4: Checking reference counts
Feb 14 10:12:32 umbrel external storage mounter[531]: Pass 5: Checking group summary information
Feb 14 10:12:32 umbrel external storage mounter[531]: Free blocks count wrong (82341205, counted=4097).
Feb 14 10:12:32 umbrel external storage mounter[531]: Fix? yes
Feb 14 10:12:32 umbrel external storage mounter[531]: Free inodes count wrong (60558023, counted=60458267).
Feb 14 10:12:32 umbrel external storage mounter[531]: Fix? yes
Feb 14 10:12:32 umbrel external storage mounter[531]: umbrel: ***** FILE SYSTEM WAS MODIFIED *****
Feb 14 10:12:32 umbrel external storage mounter[531]: umbrel: 596709/61054976 files (3.7% non-contiguous), 244186111/2441 90208 blocks
Feb 14 10:12:33 umbrel external storage mounter[531]: Mounting partition…
Feb 14 10:12:33 umbrel external storage mounter[531]: Checking if device contains an Umbrel install…
Feb 14 10:12:33 umbrel external storage mounter[531]: Yes, it contains an Umbrel install
Feb 14 10:12:33 umbrel external storage mounter[531]: Bind mounting external storage over local Umbrel installation…
Feb 14 10:12:33 umbrel external storage mounter[531]: Bind mounting external storage over local Docker data dir…
Feb 14 10:12:33 umbrel external storage mounter[531]: Bind mounting external storage to /swap
Feb 14 10:12:33 umbrel external storage mounter[531]: Bind mounting SD card root at /sd-card…
Feb 14 10:12:33 umbrel external storage mounter[531]: Checking Umbrel root is now on external storage…
Feb 14 10:12:34 umbrel external storage mounter[531]: Checking /var/lib/docker is now on external storage…
Feb 14 10:12:34 umbrel external storage mounter[531]: Checking /swap is now on external storage…
Feb 14 10:12:34 umbrel external storage mounter[531]: Setting up swapfile
Feb 14 10:12:34 umbrel external storage mounter[531]: Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
Feb 14 10:12:34 umbrel external storage mounter[531]: no label, UUID=bda64779-ce1e-405f-bd54-4b4fac5cba67
Feb 14 10:12:34 umbrel external storage mounter[531]: Checking SD Card root is bind mounted at /sd-root…
Feb 14 10:12:34 umbrel external storage mounter[531]: Starting external drive mount monitor…
Feb 14 10:12:34 umbrel external storage mounter[531]: Mount script completed successfully!
Feb 14 10:12:34 umbrel systemd[1]: Started External Storage Mounter.

External storage SD card update service logs

Feb 14 10:13:07 umbrel external storage updater[3472]: == Begin Update Script 01-run.sh ==
Feb 14 10:13:07 umbrel external storage updater[3472]: =======================================
Feb 14 10:13:07 umbrel external storage updater[3472]: =============== UPDATE ================
Feb 14 10:13:07 umbrel external storage updater[3472]: =======================================
Feb 14 10:13:07 umbrel external storage updater[3472]: =========== Stage: Install ============
Feb 14 10:13:07 umbrel external storage updater[3472]: =======================================
Feb 14 10:13:07 umbrel external storage updater[3472]: =============================================
Feb 14 10:13:07 umbrel external storage updater[3472]: Installing on Umbrel OS v0.4.4
Feb 14 10:13:07 umbrel external storage updater[3472]: =============================================
Feb 14 10:13:07 umbrel external storage updater[3472]: ./01-run.sh: line 25: cannot create temp file for here-document: N o space left on device
Feb 14 10:13:07 umbrel external storage updater[3472]: == End Update Script 01-run.sh ==
Feb 14 10:13:07 umbrel external storage updater[3472]: == Begin Update Script 02-run.sh ==
Feb 14 10:13:07 umbrel external storage updater[3472]: =======================================
Feb 14 10:13:07 umbrel external storage updater[3472]: =============== UPDATE ================
Feb 14 10:13:07 umbrel external storage updater[3472]: =======================================
Feb 14 10:13:07 umbrel external storage updater[3472]: ========= Stage: Post-update ==========
Feb 14 10:13:07 umbrel external storage updater[3472]: =======================================
Feb 14 10:13:07 umbrel external storage updater[3472]: == End Update Script 02-run.sh ==
Feb 14 10:13:07 umbrel external storage updater[3472]: == Begin Update Script 03-run.sh ==
Feb 14 10:13:07 umbrel external storage updater[3472]: =======================================
Feb 14 10:13:07 umbrel external storage updater[3472]: =============== UPDATE ================
Feb 14 10:13:07 umbrel external storage updater[3472]: =======================================
Feb 14 10:13:07 umbrel external storage updater[3472]: =========== Stage: Success ============
Feb 14 10:13:07 umbrel external storage updater[3472]: =======================================
Feb 14 10:13:07 umbrel external storage updater[3472]: Removing backup
Feb 14 10:13:07 umbrel external storage updater[3472]: ./03-run.sh: line 19: cannot create temp file for here-document: N o space left on device
Feb 14 10:13:07 umbrel external storage updater[3472]: == End Update Script 03-run.sh ==
Feb 14 10:13:07 umbrel external storage updater[3472]: Deleting cloned repository
Feb 14 10:13:07 umbrel external storage updater[3472]: Removing lock
Feb 14 10:13:07 umbrel systemd[1]: Started External Storage SDcard Updater.

Karen logs

Deriving keys…
Creating backup…
mkdir: cannot create directory ‘/home/umbrel/umbrel/.backup/30566’: No space left on device
Got signal: backup
karen is getting triggered!
Deriving keys…
Creating backup…
mkdir: cannot create directory ‘/home/umbrel/umbrel/.backup/25599’: No space left on device
Got signal: backup
karen is getting triggered!
Deriving keys…
Creating backup…
mkdir: cannot create directory ‘/home/umbrel/umbrel/.backup/15260’: No space left on device
Got signal: backup
karen is getting triggered!
Deriving keys…
Creating backup…
mkdir: cannot create directory ‘/home/umbrel/umbrel/.backup/27741’: No space left on device
Got signal: backup
karen is getting triggered!
Deriving keys…
Creating backup…
mkdir: cannot create directory ‘/home/umbrel/umbrel/.backup/7000’: No space left on device
Got signal: backup
karen is getting triggered!
Deriving keys…
Creating backup…
mkdir: cannot create directory ‘/home/umbrel/umbrel/.backup/13121’: No space left on device
Got signal: backup
karen is getting triggered!
Deriving keys…
Creating backup…
mkdir: cannot create directory ‘/home/umbrel/umbrel/.backup/8410’: No space left on device
Got signal: backup
karen is getting triggered!
Deriving keys…
Creating backup…
mkdir: cannot create directory ‘/home/umbrel/umbrel/.backup/12476’: No space left on device
Got signal: backup
karen is getting triggered!
Deriving keys…
Creating backup…
mkdir: cannot create directory ‘/home/umbrel/umbrel/.backup/27567’: No space left on device
Got signal: backup
karen is getting triggered!
Deriving keys…
Creating backup…
mkdir: cannot create directory ‘/home/umbrel/umbrel/.karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events

Docker containers

NAMES STATUS

Umbrel logs

Attaching to manager

Tor Proxy logs

Attaching to tor_proxy

App logs

bitcoin

Attaching to

bitfeed

Attaching to

btc-rpc-explorer

Attaching to

core-lightning

Attaching to

core-lightning-rtl

Attaching to

electrs

Attaching to

lightning

Attaching to

lightning-terminal

Attaching to

lnplus

Attaching to

mempool

Attaching to

passky-client

Attaching to

passky-server

Attaching to

pi-hole

Attaching to

ride-the-lightning

Attaching to

samourai-server

Attaching to

thunderhub

Attaching to

==== Result ====

The debug script did not automatically detect any issues with your Umbrel.

2

get a bigger SD, flash the newest Umbrel and swap the SD Card. This should help.

3

Thank you for your reply!

I’ve done it, on a 32Gb one.

The new system didn’t recognize the SSD which has files now dated from 2019.
afbeelding

4

I couldn’t yet find a way to fix this issue.

And I really would appreciate any support or help.
It seems the only option here is tabula rasa.
Clean it all up and start over.

Is it really so?

5

Solution was found. Although the issue was not clearly identified.
As suggested above, a new SD card ended up partially solving it.
Funny enough, the first attempt on a 32Gb card resulted on an also corrupted card.

So I bought a 3rd one and Umbrel finally loaded.

But still not fully initializing. Blockchain wasn’t synchronizing.

So I thought some other files seemed to be corrupted as well.
settings.json and peers.dat deleted.

System restarted and, after a while, it’s syncing again.

Thanks @rieger_san