MicroSD could fail soon/Error Failed to start containers

[wealthcollector]

Hi guys,

Last night had a storm and the power went out. Woke up to a MicroSD could fail soon message. I read some messages on here from the past and it said that the SD card is likely not the issue. I reflashed the SD card with umbrel and put it back into the rasberryPi. It still doesn’t work but now I’m getting an “Error: Failed to start containers” message when i fire up umbrel. Not sure what to do now.

I read some other threads and people got a fix by doing some commands in terminal. I’m not that knowledgeable with that kind of stuff, so could appreciate it if someone could walk me through some attempts at fixing. I don’t even know how to access the logs, etc. I’m using a mac fwiw.

I also bought a new SD card and tried that, but still getting the “Error: Failed to start containers” message.

Hopefully someone can help. Thanks in advance.

[wealthcollector]

So I figured out how to do some things in terminal. I have followed all these steps below. However when I get to step 3, I get this message:

docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.

See ‘docker run --help’.

Step 1 - Enter using SSH into your Umbrel node:

Step 2 - stop Umbrel node with:

cd umbrel

sudo ./scripts/stop

Step 3 - clean / reconfigure the broken container:

sudo rm -f .env && sudo ./scripts/configure

[smolgrrr]

Have you tried this method?

@mayank

SSH (Secure Shell) is a way to access and control a device, such as your Umbrel server running on a Umbrel Home or Raspberry Pi, via the command line. To access Umbrel via SSH, follow these steps:

1. Open a terminal window on your computer. On macOS, you can open the Terminal app that’s installed by default on every Mac. On Windows, you can open Command Prompt or the PowerShell app.
2. Type in the following command* and press the Enter key:ssh -t umbrel@umbrel.local*In the command given above, you can replace umbrel.local with the local IP of your Umbrel if you prefer. If you are using PowerShell on Windows 10, you may need to run ssh umbrel@umbrel instead of the command given above.
3. You will then be prompted to enter your password. This is the same password you use to access your Umbrel through a browser. The default password before you have created an account on your Umbrel is moneyprintergobrrr on Raspberry Pi, and umbrel on Umbrel Home. You will not be able to see your password as you type it into the terminal. Once you have typed in your password, press the Enter key.

This will established an SSH connection to your Umbrel server. You can now begin using the command line interface to perform various actions and troubleshoot issues that require SSH.

and then

@mayank

If your umbrelOS logs show an error message that reads “failed to start containers”, follow these steps:

1. Restart your Umbrel server from settings.
2. If the problem persists, connect to your Umbrel server by following the instructions in How to SSH into umbrelOS on Umbrel Home or Rasberry Pi or by accessing a terminal on your Linux machine.
3. Run the following command:sudo systemctl stop docker && sudo rm -rf /var/lib/docker/* && sudo reboot

This will stop and reset your docker service files and then it will reboot your device, which may take several minutes.

[wealthcollector]

Yup rebooted then back to the “Error: failed to start containers” page.

[smolgrrr]

Are you able to SSH into your node? Can you please run the following and share the output

sudo ~/umbrel/scripts/debug

[wealthcollector]

@smolgrrr

sudo ~/umbrel/scripts/debug

= Umbrel debug info =

=====================

Umbrel version

---

0.5.4

Flashed OS version

---

v0.5.4-patch-1

Raspberry Pi Model

---

Revision : d03115

Serial : 10000000babb75e7

Model : Raspberry Pi 4 Model B Rev 1.5

Firmware

---

May 9 2023 12:16:34

Copyright (c) 2012 Broadcom

version 30aa0d70ab280427ba04ebc718c81d4350b9d394 (clean) (release) (start)

Temperature

---

temp=53.5’C

Throttling

---

throttled=0x0

Memory usage

---

total used free shared buff/cache available

Mem: 7.8G 214M 7.1G 8.0M 547M 7.5G

Swap: 4.1G 0B 4.1G

total: 2.8%

system: 2.8%

thunderhub: 0%

tailscale: 0%

snowflake: 0%

snort: 0%

robosats: 0%

mempool: 0%

lnplus: 0%

lndg: 0%

lnbits: 0%

lightning-terminal: 0%

lightning: 0%

electrs: 0%

btc-rpc-explorer: 0%

bitcoin: 0%

Memory monitor logs

---

2023-10-30 11:02:03 Memory monitor running!

2023-10-30 11:02:15 Memory monitor running!

2023-10-30 11:02:27 Memory monitor running!

2023-10-30 11:02:40 Memory monitor running!

2023-10-31 11:36:16 Memory monitor running!

2023-10-31 11:36:48 Memory monitor running!

2023-10-31 11:37:00 Memory monitor running!

2023-10-31 11:37:13 Memory monitor running!

2023-10-31 11:37:25 Memory monitor running!

2023-10-31 11:37:38 Memory monitor running!

Filesystem information

---

Filesystem Size Used Avail Use% Mounted on

/dev/root 30G 3.6G 25G 13% /

/dev/sda1 1.8T 670G 1.1T 39% /home/umbrel/umbrel

Startup service logs

---

Oct 31 11:37:38 umbrel umbrel startup[5245]: Starting status monitors…

Oct 31 11:37:38 umbrel umbrel startup[5245]: Starting memory monitor…

Oct 31 11:37:38 umbrel umbrel startup[5245]: Starting backup monitor…

Oct 31 11:37:38 umbrel umbrel startup[5245]: Starting decoy backup trigger…

Oct 31 11:37:38 umbrel umbrel startup[5245]: Starting umbreld…

Oct 31 11:37:38 umbrel umbrel startup[5245]: Starting Docker services…

Oct 31 11:37:40 umbrel umbrel startup[5245]: The TOR_PROXY_PORT variable is not set. Defaulting to a blank string.

Oct 31 11:37:40 umbrel umbrel startup[5245]: The TOR_PROXY_IP variable is not set. Defaulting to a blank string.

Oct 31 11:37:40 umbrel umbrel startup[5245]: The NGINX_PORT variable is not set. Defaulting to a blank string.

Oct 31 11:37:40 umbrel umbrel startup[5245]: The NGINX_IP variable is not set. Defaulting to a blank string.

Oct 31 11:37:40 umbrel umbrel startup[5245]: The DASHBOARD_IP variable is not set. Defaulting to a blank string.

Oct 31 11:37:40 umbrel umbrel startup[5245]: The UMBREL_AUTH_SECRET variable is not set. Defaulting to a blank string.

Oct 31 11:37:40 umbrel umbrel startup[5245]: The BITCOIN_P2P_PORT variable is not set. Defaulting to a blank string.

Oct 31 11:37:40 umbrel umbrel startup[5245]: The BITCOIN_RPC_PORT variable is not set. Defaulting to a blank string.

Oct 31 11:37:40 umbrel umbrel startup[5245]: The BITCOIN_RPC_USER variable is not set. Defaulting to a blank string.

Oct 31 11:37:40 umbrel umbrel startup[5245]: The BITCOIN_RPC_PASS variable is not set. Defaulting to a blank string.

Oct 31 11:37:40 umbrel umbrel startup[5245]: The BITCOIN_NETWORK variable is not set. Defaulting to a blank string.

Oct 31 11:37:40 umbrel umbrel startup[5245]: The MANAGER_IP variable is not set. Defaulting to a blank string.

Oct 31 11:37:40 umbrel umbrel startup[5245]: The AUTH_PORT variable is not set. Defaulting to a blank string.

Oct 31 11:37:40 umbrel umbrel startup[5245]: The AUTH_IP variable is not set. Defaulting to a blank string.

Oct 31 11:37:40 umbrel umbrel startup[5245]: The NETWORK_IP variable is not set. Defaulting to a blank string.

Oct 31 11:37:40 umbrel umbrel startup[5245]: The Compose file ‘./docker-compose.yml’ is invalid because:

Oct 31 11:37:40 umbrel umbrel startup[5245]: networks.default.ipam.config.subnet is invalid: should use the CIDR format

Oct 31 11:37:40 umbrel umbrel startup[5245]: services.tor_proxy.ports contains an invalid type, it should be a number, or an object

Oct 31 11:37:40 umbrel umbrel startup[5245]: services.auth.ports contains an invalid type, it should be a number, or an object

Oct 31 11:37:40 umbrel umbrel startup[5245]: services.nginx.ports contains an invalid type, it should be a number, or an object

Oct 31 11:37:41 umbrel umbrel startup[5245]: Failed to start containers

Oct 31 11:37:41 umbrel systemd[1]: umbrel-startup.service: Control process exited, code=exited, status=1/FAILURE

Oct 31 11:37:41 umbrel systemd[1]: umbrel-startup.service: Failed with result ‘exit-code’.

Oct 31 11:37:41 umbrel systemd[1]: Failed to start Umbrel Startup Service.

External storage service logs

---

Oct 31 11:35:56 umbrel external storage mounter[485]: Rebinding USB drivers…

Oct 31 11:35:56 umbrel external storage mounter[485]: Checking USB devices are back…

Oct 31 11:35:56 umbrel external storage mounter[485]: Waiting for USB devices…

Oct 31 11:35:57 umbrel external storage mounter[485]: Waiting for USB devices…

Oct 31 11:35:58 umbrel external storage mounter[485]: Waiting for USB devices…

Oct 31 11:35:59 umbrel external storage mounter[485]: Checking if the device is ext4…

Oct 31 11:35:59 umbrel external storage mounter[485]: Yes, it is ext4

Oct 31 11:35:59 umbrel external storage mounter[485]: Checking filesystem for corruption…

Oct 31 11:35:59 umbrel external storage mounter[485]: e2fsck 1.44.5 (15-Dec-2018)

Oct 31 11:36:00 umbrel external storage mounter[485]: umbrel: recovering journal

Oct 31 11:36:03 umbrel external storage mounter[485]: Setting free inodes count to 121614642 (was 121614650)

Oct 31 11:36:03 umbrel external storage mounter[485]: Setting free blocks count to 305071773 (was 305071999)

Oct 31 11:36:03 umbrel external storage mounter[485]: umbrel: clean, 487118/122101760 files, 183306339/488378112 blocks

Oct 31 11:36:03 umbrel external storage mounter[485]: Mounting partition…

Oct 31 11:36:04 umbrel external storage mounter[485]: Checking if device contains an Umbrel install…

Oct 31 11:36:04 umbrel external storage mounter[485]: Yes, it contains an Umbrel install

Oct 31 11:36:04 umbrel external storage mounter[485]: Bind mounting external storage over local Umbrel installation…

Oct 31 11:36:04 umbrel external storage mounter[485]: Bind mounting external storage over local Docker data dir…

Oct 31 11:36:04 umbrel external storage mounter[485]: Bind mounting external storage to /swap

Oct 31 11:36:04 umbrel external storage mounter[485]: Bind mounting SD card root at /sd-card…

Oct 31 11:36:04 umbrel external storage mounter[485]: Checking Umbrel root is now on external storage…

Oct 31 11:36:05 umbrel external storage mounter[485]: Checking /var/lib/docker is now on external storage…

Oct 31 11:36:05 umbrel external storage mounter[485]: Checking /swap is now on external storage…

Oct 31 11:36:05 umbrel external storage mounter[485]: Setting up swapfile

Oct 31 11:36:05 umbrel external storage mounter[485]: Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)

Oct 31 11:36:05 umbrel external storage mounter[485]: no label, UUID=ed27a193-4178-4c78-a444-836f4d0abec4

Oct 31 11:36:05 umbrel external storage mounter[485]: Checking SD Card root is bind mounted at /sd-root…

Oct 31 11:36:05 umbrel external storage mounter[485]: Starting external drive mount monitor…

Oct 31 11:36:05 umbrel external storage mounter[485]: Mount script completed successfully!

Oct 31 11:36:05 umbrel systemd[1]: Started External Storage Mounter.

External storage SD card update service logs

---

– Logs begin at Tue 2023-10-31 11:35:51 UTC, end at Tue 2023-10-31 11:37:41 UTC. –

Oct 31 11:36:16 umbrel systemd[1]: Starting External Storage SDcard Updater…

Oct 31 11:36:16 umbrel external storage updater[935]: Checking if SD card Umbrel is newer than external storage…

Oct 31 11:36:16 umbrel external storage updater[935]: No, SD version is not newer, exiting.

Oct 31 11:36:16 umbrel systemd[1]: Started External Storage SDcard Updater.

Karen logs

---

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

karen is running in /home/umbrel/umbrel/events

Docker containers

---

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Umbrel logs

---

ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:

networks.default.ipam.config.subnet is invalid: should use the CIDR format

services.tor_proxy.ports contains an invalid type, it should be a number, or an object

services.auth.ports contains an invalid type, it should be a number, or an object

services.nginx.ports contains an invalid type, it should be a number, or an object

Tor Proxy logs

---

ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:

networks.default.ipam.config.subnet is invalid: should use the CIDR format

services.tor_proxy.ports contains an invalid type, it should be a number, or an object

services.auth.ports contains an invalid type, it should be a number, or an object

services.nginx.ports contains an invalid type, it should be a number, or an object

App logs

---

bitcoin

./scripts/app: line 79: NETWORK_IP: unbound variable

btc-rpc-explorer

./scripts/app: line 79: NETWORK_IP: unbound variable

electrs

./scripts/app: line 79: NETWORK_IP: unbound variable

lightning

./scripts/app: line 79: NETWORK_IP: unbound variable

lightning-terminal

./scripts/app: line 79: NETWORK_IP: unbound variable

lnbits

./scripts/app: line 79: NETWORK_IP: unbound variable

lndg

./scripts/app: line 79: NETWORK_IP: unbound variable

lnplus

./scripts/app: line 79: NETWORK_IP: unbound variable

mempool

./scripts/app: line 79: NETWORK_IP: unbound variable

robosats

./scripts/app: line 79: NETWORK_IP: unbound variable

snort

./scripts/app: line 79: NETWORK_IP: unbound variable

snowflake

./scripts/app: line 79: NETWORK_IP: unbound variable

tailscale

./scripts/app: line 79: NETWORK_IP: unbound variable

thunderhub

./scripts/app: line 79: NETWORK_IP: unbound variable

================

==== Result ====

================

The debug script did not automatically detect any issues with your Umbrel.

[wealthcollector]

Any thoughts @smolgrrr

[JohnvonHorn]

I’m having exactly the same problem. I’ve reflashed the SD card but have the same problem.

Docker is down and consistently fails to start.

Process: 22173 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=2)

From scripts/debug I’ve got:

Docker containers

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

And there seems to be problems with the Docker config file:

with ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.auth.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object

I think the docker-compose.yml file is trying to pick up environmental variables which are not being set.

Also From systelctl.status (and ps-aux) I can see that containerd is running but docker is not. Is Docker failing because of a bad docker-compose.yml file OR are the environmental values in the docker-compose.yml file not being set properly by earlier config files?

Sorry, I’m just an average Joe trying to get to grips with all of this.

[cavemanf16]

I too am having this problem, although I can see in your logs that the temperature on your Pi4 seems really high:

temp=53.5’C

(for comparison, my Pi4 is running at just 34.0 C, and I just have a simple fan sitting on top of mine, so the hottest it ever gets is ~46 C)

Your log about mounting the external file storage system appears in order, as does my own, so a bad external drive doesn’t appear to be a problem.

I’ve tried the other suggestions of stopping Docker, clearing it out, and restarting, but to no avail as well. Given that there is a good deal of discussion on this board about the flakiness of the Pi4 hardware, I’m going to try to spin up Umbrel in a VM on proxmox which is installed on another home server I setup, and then I’ll connect my Umbrel VM to my external drive to see if the problem is the Pi4 board or hardware vs. some type of software problem. I’ll do my best to report back if I have any luck.

IF you have access to another machine or can spin up a virtual machine with your external drive connected, such as using VirtualBox on a Windows laptop or desktop, you may want to try doing so.

After all, I’ve had three power outages over the course of running my Umbrel on a Raspberry Pi4, and each time the power outage has really jacked up the Pi4, even though I have them on a UPS and sinoidal wave power supply which shouldn’t cause disruptions to the hardware like it seems to do.

[Tyler]

When I type in ssh -tumbrel@umrel.local* it tells me “ssh: illegal option – u” and give me a bunch of different code that I don’t understand. Am I entering this incorrectly? It never asked me for my umbrel password. @wealthcollector @smolgrrr you seem to be having some of the same issues I’m having. Any idea what I’m doing incorrectly? I have had my node for about a year.

[Tyler]

Ok I did more research and got it to take the next step. (Mac user)

It’s stating the following: The authenticity of host 'umbrel.local cannot be established. This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])?

Is it safe to move forward? I don’t want to mess anything up.

[smolgrrr]

hey @Tyler
Yes this is a standard process when SSHing into a new a server!
Let me know how that goes.

[Tyler]

@smolgrrr it gave me the following message:

Warning: Permanently added ‘umbrel.local’ to the list of known hosts.

ssh_dispatch_run_fatal: Connection to 192.etc port 22: Broken pipe

I’m not sure if I should be putting the full IP address online so I took part of it off.

I took the node apart and reflashed the SD card, put it all back together, restarted umbrel, and I’m still getting the red Umbrel sign and " Error: Failed to start containers"

I’m not sure what else to do at this point.

[StevieD]

I got this message as well (MicroSD could fail soon) and thought my SD card was a fake. I confirmed it was an original and unplugged the device is it would not shut down, then removed the SD card, unplugged the cable from the box, reinserted the SD card, then plugged everything back in.
It started working again.

[marukopollo]

My SD volume was full

ran the following command to check

df -h
cd /var/log
sudo du -sh *

found out that couple of logs were taking up the space. Truncated them

sudo truncate -s 0 /var/log/syslog.1
sudo truncate -s 0 /var/log/daemon.log.1

[redone]

@marukopollo works perfectly, thx for sharing

[JELI]

@marukopollo thank you!

[BTC_KS]

Worked for me also. Thank you! Seems like MicroSD issues are an ongoing problem (running since 2021). Maybe a dumb question, but… any way to prevent/limit recurrence of MicroSD issues by, for example, buying a bigger, better, badder MicroSD card? I recall buying the one suggested on Umbrel page, but would gladly pay more if it would make a difference. Thanks again.

[Gox]

This worked for me as well! Brilliant! Thank you for that, I was tearing my hair out for a bit.

[Gox]

I too wonder if this would be worthwhile. I have an extra 128gb TF card lying around from an old dashcam I sold, wonder if that would fix this problem for a while.