Umbrel Not Operational: Error System Service Failed - No Space Left on Device

I have read for hours over the course of a couple of weeks and there is a very long post here with debug logs and more. I have a 1TB SSD; 64GB mSD running Umbrel on a Raspberry Pi. Everything was set up exactly as recommended on GetUmbrel.com. My node has been running for almost one year until SHTF a few weeks ago. My question is regarding the data usage of certain directories listed. When reviewing the usage report below it appears to my untrained eyes that the reason I am unable to update to latest version or run ~/umbrel $ sudo ./scripts/configure is because something has caused a duplication of files to fill all of the space on my SSD.

/mnt /data/umbrel is using 517G of data.  
/home/umbrel/umbrel is also using 517G of data.

with 482G of that being in the /mnt/data/umbrel/bitcoin directory and
another 482G in the /home/umbrel/umbrel/bitcoin directory.

So my question is two-fold. Am I correct that this is two separate directories? and if so, which one of them would it be safest to delete which will allow me to still SSH into the device, login to umbrel via ssh and then run the script to update?

Hey @CryptoMistro :wave:! Sorry for the delay,
/mnt/data/umbrel
/home/umbrel/umbrel
are the same folder, Umbrel uses something called ‘bind mount’ to make the /mnt/data/umbrel folder accessible in ‘common places’ like /home/umbrel

375G /var/lib/docker/containers/fb66b3fc74454f5f8c2db8e0cfa542bbf55e5141691be2e96844bc22e0b71fc6/fb66b3fc74454f5f8c2db8e0cfa542bbf55e5141691be2e96844bc22e0b71fc6-json.log

Here we can see that there are 375G of docker logs. It means that a container is probably writing a ton of logs.
It’d be nice to see which container is causing the issue and check which logs are written.
You can get them using:

sudo cat /var/lib/docker/containers/fb66b3fc74454f5f8c2db8e0cfa542bbf55e5141691be2e96844bc22e0b71fc6/fb66b3fc74454f5f8c2db8e0cfa542bbf55e5141691be2e96844bc22e0b71fc6-json.log | tail -n 100
1 Like

Thanks. Because I grew impatient, I purchased an additional SSD and am currently downloading the blockchain to it. I haven’t purchased another raspberry pi yet, but I most likely will do so and eventually run two umbrel servers. At any rate, once the other Raspberry pi comes in I will run the command you suggested on the original SSD.:

sudo cat /var/lib/docker/containers/fb66b3fc74454f5f8c2db8e0cfa542bbf55e5141691be2e96844bc22e0b71fc6/fb66b3fc74454f5f8c2db8e0cfa542bbf55e5141691be2e96844bc22e0b71fc6-json.log | tail -n 100

I’ll post the results…

2 Likes

I’m running into the same issue. I found the container which is creating the log. It is 384gb. Is it okay if I delete that log file or will that cause further issues?

1 Like