Failed to Start Docker Application Container Engine

I began running Umbrel successfully nearly a year ago with minimal issues on several nodes. I recently lost power and one of my nodes returned as normal, but the other did not. I have a bitcoin wallet and lightning channels on this node. I have my seed and backups. Connecting via browser, I receive the following Error: Failed to start containers.

I am able to connect via ssh and verify that the SD card with the OS and SSD drive are functional, but continually receive this error when connecting via any browser. I bought a new micro SD card and flashed it to Umbrel v0.5.0. One interesting note is that my password reset to the default password when I began receiving this error.


Here is my setup:

  • Raspberry Pi 4B - 8GB
  • Samsung Portable SSD T7 - 2TB
  • SanDisk 32GB Ultra microSDHC UHS-I Memory Card
  • GeeekPi Armor Lite Aluminum Heatsink with PWM Speed Control Fan for Raspberry Pi 4 Model B

Here is the debug log from this Umbrel node.


=====================
= Umbrel debug info =

Umbrel version

0.5.0

Flashed OS version

v0.5.0

Raspberry Pi Model

Revision : d03114
Serial : 10000000678aebdd
Model : Raspberry Pi 4 Model B Rev 1.4

Firmware

Dec 1 2021 15:01:54
Copyright (c) 2012 Broadcom
version 71bd3109023a0c8575585ba87cbb374d2eeb038f (clean) (release) (start)

Temperature

temp=37.4’C

Throttling

throttled=0x0

Memory usage

          total        used        free      shared  buff/cache   available

Mem: 7.8G 115M 7.3G 8.0M 361M 7.6G
Swap: 4.1G 0B 4.1G

total: 1.5%
system: 1.5%
tor: 0%
ride-the-lightning: 0%
mempool: 0%
lnd: 0%
lightning: 0%
electrs: 0%
electrs: 0%
btcpay-server: 0%
bitcoin: 0%
bitcoin: 0%

Memory monitor logs

2022-07-21 16:02:17 Memory monitor running!
2022-07-21 16:02:29 Memory monitor running!
2022-07-21 16:02:40 Memory monitor running!
2022-07-21 16:02:52 Memory monitor running!
2022-07-21 16:03:04 Memory monitor running!
2022-07-21 16:03:16 Memory monitor running!
2022-07-21 16:03:28 Memory monitor running!
2022-07-21 16:03:40 Memory monitor running!
2022-07-21 16:03:52 Memory monitor running!
2022-07-21 16:04:04 Memory monitor running!

Filesystem information

Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 3.2G 25G 12% /
/dev/sda1 1.8T 534G 1.2T 31% /home/umbrel/umbrel

Startup service logs

Jul 21 16:04:04 umbrel umbrel startup[32709]: Starting backup monitor…
Jul 21 16:04:04 umbrel umbrel startup[32709]: Starting decoy backup trigger…
Jul 21 16:04:04 umbrel umbrel startup[32709]: Starting Docker services…
Jul 21 16:04:05 umbrel umbrel startup[32709]: The TOR_PROXY_PORT variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The TOR_PROXY_IP variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The NGINX_PORT variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The NGINX_IP variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The DASHBOARD_IP variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The MIDDLEWARE_IP variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The UMBREL_AUTH_SECRET variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The BITCOIN_P2P_PORT variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The BITCOIN_RPC_PORT variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The BITCOIN_RPC_USER variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The BITCOIN_RPC_PASS variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The BITCOIN_NETWORK variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The MANAGER_IP variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The BITCOIN_IP variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The LND_IP variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The AUTH_PORT variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The AUTH_IP variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The NETWORK_IP variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The Compose file ‘./docker-compose.yml’ is invalid because:
Jul 21 16:04:05 umbrel umbrel startup[32709]: networks.default.ipam.config.subnet is invalid: should use the CIDR format
Jul 21 16:04:05 umbrel umbrel startup[32709]: services.tor_proxy.ports contains an invalid type, it should be a number, or an object
Jul 21 16:04:05 umbrel umbrel startup[32709]: services.auth.ports contains an invalid type, it should be a number, or an object
Jul 21 16:04:05 umbrel umbrel startup[32709]: services.nginx.ports contains an invalid type, it should be a number, or an object
Jul 21 16:04:05 umbrel umbrel startup[32709]: Failed to start containers
Jul 21 16:04:05 umbrel systemd[1]: umbrel-startup.service: Control process exited, code=exited, status=1/FAILURE
Jul 21 16:04:05 umbrel systemd[1]: umbrel-startup.service: Failed with result ‘exit-code’.
Jul 21 16:04:05 umbrel systemd[1]: Failed to start Umbrel Startup Service.

External storage service logs

– Logs begin at Thu 2022-07-21 15:17:01 UTC, end at Thu 2022-07-21 16:04:05 UTC. –
Jul 21 15:17:06 umbrel systemd[1]: Starting External Storage Mounter…
Jul 21 15:17:06 umbrel external storage mounter[486]: Running external storage mount script…
Jul 21 15:17:06 umbrel external storage mounter[486]: Found device “Samsung PSSD T7”
Jul 21 15:17:06 umbrel external storage mounter[486]: Blacklisting USB device IDs against UAS driver…
Jul 21 15:17:06 umbrel external storage mounter[486]: Rebinding USB drivers…
Jul 21 15:17:06 umbrel external storage mounter[486]: Checking USB devices are back…
Jul 21 15:17:06 umbrel external storage mounter[486]: Waiting for USB devices…
Jul 21 15:17:07 umbrel external storage mounter[486]: Waiting for USB devices…
Jul 21 15:17:08 umbrel external storage mounter[486]: Waiting for USB devices…
Jul 21 15:17:09 umbrel external storage mounter[486]: Checking if the device is ext4…
Jul 21 15:17:09 umbrel external storage mounter[486]: Yes, it is ext4
Jul 21 15:17:09 umbrel external storage mounter[486]: Checking if device contains an Umbrel install…
Jul 21 15:17:09 umbrel external storage mounter[486]: Yes, it contains an Umbrel install
Jul 21 15:17:09 umbrel external storage mounter[486]: Bind mounting external storage over local Umbrel installation…
Jul 21 15:17:09 umbrel external storage mounter[486]: Bind mounting external storage over local Docker data dir…
Jul 21 15:17:09 umbrel external storage mounter[486]: Bind mounting external storage to /swap
Jul 21 15:17:10 umbrel external storage mounter[486]: Bind mounting SD card root at /sd-card…
Jul 21 15:17:10 umbrel external storage mounter[486]: Checking Umbrel root is now on external storage…
Jul 21 15:17:11 umbrel external storage mounter[486]: Checking /var/lib/docker is now on external storage…
Jul 21 15:17:11 umbrel external storage mounter[486]: Checking /swap is now on external storage…
Jul 21 15:17:11 umbrel external storage mounter[486]: Setting up swapfile
Jul 21 15:17:11 umbrel external storage mounter[486]: Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
Jul 21 15:17:11 umbrel external storage mounter[486]: no label, UUID=74b6f1ed-4f73-4531-9e47-7fb91be3ddf0
Jul 21 15:17:11 umbrel external storage mounter[486]: Checking SD Card root is bind mounted at /sd-root…
Jul 21 15:17:11 umbrel external storage mounter[486]: Starting external drive mount monitor…
Jul 21 15:17:11 umbrel external storage mounter[486]: Mount script completed successfully!
Jul 21 15:17:11 umbrel systemd[1]: Started External Storage Mounter.

External storage SD card update service logs

– Logs begin at Thu 2022-07-21 15:17:01 UTC, end at Thu 2022-07-21 16:04:05 UTC. –
Jul 21 15:17:19 umbrel systemd[1]: Starting External Storage SDcard Updater…
Jul 21 15:17:19 umbrel external storage updater[938]: Checking if SD card Umbrel is newer than external storage…
Jul 21 15:17:19 umbrel external storage updater[938]: No, SD version is not newer, exiting.
Jul 21 15:17:19 umbrel systemd[1]: Started External Storage SDcard Updater.

Karen logs

karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events

Docker containers

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Umbrel logs

ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.auth.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object

Tor logs

ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.auth.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object

App logs

bitcoin

./scripts/app: line 81: NETWORK_IP: unbound variable

btcpay-server

./scripts/app: line 81: NETWORK_IP: unbound variable

electrs

./scripts/app: line 81: NETWORK_IP: unbound variable

lightning

./scripts/app: line 81: NETWORK_IP: unbound variable

mempool

./scripts/app: line 81: NETWORK_IP: unbound variable

ride-the-lightning

./scripts/app: line 81: NETWORK_IP: unbound variable

==== Result ====

The debug script did not automatically detect any issues with your Umbrel.


Here is the docker.service status.

umbrel@umbrel:/var/run $ sudo systemctl status docker.service
[sudo] password for umbrel:
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Thu 2022-07-21 16:08:14 UTC; 1s ago
Docs: https://docs.docker.com
Process: 11295 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=2
Main PID: 11295 (code=exited, status=2)


Here is the journalctl -xe.

umbrel@umbrel:/var/run $ journalctl -xe
Jul 21 16:16:03 umbrel dockerd[31831]: /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go
Jul 21 16:16:03 umbrel dockerd[31831]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x4000
Jul 21 16:16:03 umbrel dockerd[31831]: /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go
Jul 21 16:16:03 umbrel dockerd[31831]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(...)
Jul 21 16:16:03 umbrel dockerd[31831]: /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go
Jul 21 16:16:03 umbrel dockerd[31831]: main.main()
Jul 21 16:16:03 umbrel dockerd[31831]: /go/src/github.com/docker/docker/cmd/dockerd/docker.go:97 +0x1b4
Jul 21 16:16:03 umbrel systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
– Subject: Unit process exited
– Defined-By: systemd
– Support: Debian -- User Support

– An ExecStart= process belonging to unit docker.service has exited.

– The process’ exit code is ‘exited’ and its exit status is 2.
Jul 21 16:16:03 umbrel systemd[1]: docker.service: Failed with result ‘exit-code’.
– Subject: Unit failed
– Defined-By: systemd
– Support: Debian -- User Support

– The unit docker.service has entered the ‘failed’ state with result ‘exit-code’.
Jul 21 16:16:03 umbrel systemd[1]: Failed to start Docker Application Container Engine.
– Subject: A start job for unit docker.service has failed
– Defined-By: systemd
– Support: Debian -- User Support

– A start job for unit docker.service has finished with a failure.

– The job identifier is 12539 and the job result is failed.


I admit I am out of my depth here, but it appears that the Docker Application Container Engine is the issue. I am just not sure how to troubleshoot from here. I appreciate any helpful posts, links, or advice. Thank you for your time and consideration.

Jaes aka OpenWaterSwim

Hi
I got the same issue and i’ve just solved it!
I’ve first flashed the micro sd. But nothing changed.
Then I followed this steps :

Step 1 - Enter using SSH into your Umbrel node:

Step 2 - stop Umbrel node with:

cd umbrel
sudo systemctl stop umbrel-startup

Step 3 - clean / reconfigure the broken container:

sudo rm .env --> got an error because .env did not existed. No probs
sudo ./scripts/configure

Step 4 - apply latest Umbrel update (replace the version number with the latest):

sudo ./scripts/update/update --repo getumbrel/umbrel#v0.5.0

Restart your node:

UmbrelOS (RPi) use sudo systemctl start umbrel-startup

Worked fine for me! I hope it’s gonna help you
Cheers
Pierre

1 Like

Pierre:

Thank you for your reply. I appreciate any and all suggestions as I feel like I have tried plenty. At least through all this troubleshooting, I learn with each new attempt to fix my issue. I have also been able to help others while learning, so this is all a plus as I believe I am then able to give back to the community.

When I get to step 3 and run the following command
sudo .scripts/configure

I receive the error message regarding the Docker daemon.


umbrel@umbrel:~/umbrel $ sudo ./scripts/configure

======================================
=========== RECONFIGURING ============
========= UMBREL (mainnet) ===========

Generating auth credentials

Generating Tor password

docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
See ‘docker run --help’.


I tried the following to see if I could fix the Docker, but there still seems to be a problem with my docker-compose.yml file.

cd umbrel
sudo systemctl stop docker.service
sudo systemctl stop docker.socket
sudo rm -rf /var/lib/docker/* && sudo umount /var/lib/docker || true && sudo rm -rf /mnt/data/docker/*
sudo reboot


However, this likewise did not correct my issue. Thank you for your suggestion. We will figure it out.

2 Likes

Merci Pierre. This helped fixed a separate issue I had with the dockers not being able to fully shut down and reboot.

1 Like

Also having the same issue. Did you figure it out?

I am also having the same issue. Did anyone figure this out yet?

hello, i am stuck here with this same issue…

Hi @Groove, have you tried the following, and if so could you please share your troubleshooting logs?

Step 1 - Enter using SSH into your Umbrel node:

Step 2 - stop Umbrel node with:

cd umbrel

sudo ./scripts/stop

Step 3 - clean / reconfigure the broken container:

sudo rm -f .env && sudo ./scripts/configure

Step 4 - Restart your node:

sudo ./scripts/start

Please share them from here https://pastebin.com/

=====================
= Umbrel debug info =

Umbrel version

0.5.0

Memory usage

           total       usada       livre    compart.  buff/cache  disponível

Mem.: 4,1G 1,8G 141M 167M 2,6G 2,3G
Swap: 4,1G 610M 3,5G

total: 42,9%
tor: 0%
system: %
mempool: 0%
lndg: 0%
lnd: 0%
lightning: 0%
electrs: 0%
electrs: 0%
bluewallet: 0%
bitcoin: 0%
bitcoin: 0%

Memory monitor logs

3333 ? S 0:00 bash ./scripts/memory-monitor
Memory monitor is already running
2023-08-11 10:20:23 Warning memory usage at 91%
2023-08-11 10:21:23 Warning memory usage at 93%
2023-08-12 10:16:38 Memory monitor running!
2023-08-12 11:23:51 Warning memory usage at 92%
2023-08-12 11:24:52 Warning memory usage at 92%
2023-08-12 11:25:52 Warning memory usage at 91%
2023-08-12 12:24:00 Warning memory usage at 93%
2023-08-12 12:32:02 Warning memory usage at 96%

Filesystem information

Sist. Arq. Tam. Usado Disp. Uso% Montado em
/dev/sda2 916G 654G 216G 76% /
/dev/sda2 916G 654G 216G 76% /

Karen logs

3924 bytes (3,9 kB, 3,8 KiB) copied, 0,0118845 s, 330 kB/s
Creating encrypted tarball…
backup/
backup/.padding
backup/channel.backup
Uploading backup…
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5123 100 146 100 4977 31 1078 0:00:04 0:00:04 --:–:-- 1559
{“message”:“Successfully uploaded backup 1691752226465.tar.gz.pgp for backup ID 00eeacdf540898c21482f02811819f2c7367399f01b5610a4c192ef1944a81f7”}

====== Backup success =======

Got signal:
karen is getting triggered!
./karen: linha 68: /home/diogo/umbrel/events/triggers/: É um diretório
Got signal:
karen is getting triggered!
Got signal:
karen is getting triggered!
./karen: linha 68: /home/diogo/umbrel/events/triggers/: É um diretório
./karen: linha 68: /home/diogo/umbrel/events/triggers/: É um diretório
3321 ? S 0:00 bash ./karen
3376 ? S 0:00 bash ./karen
karen is already running
Got signal:
karen is getting triggered!
Got signal:
karen is getting triggered!
./karen: linha 68: /home/diogo/umbrel/events/triggers/: É um diretório
./karen: linha 68: /home/diogo/umbrel/events/triggers/: É um diretório
Got signal:
karen is getting triggered!
./karen: linha 68: /home/diogo/umbrel/events/triggers/: É um diretório
Got signal:
karen is getting triggered!
Got signal:
karen is getting triggered!
./karen: linha 68: /home/diogo/umbrel/events/triggers/: É um diretório
./karen: linha 68: /home/diogo/umbrel/events/triggers/: É um diretório
3321 ? S 0:00 bash ./karen
3376 ? S 0:00 bash ./karen
karen is already running
karen is running in /home/diogo/umbrel/events
Got signal:
karen is getting triggered!
Got signal:
karen is getting triggered!
./karen: linha 68: /home/diogo/umbrel/events/triggers/: É um diretório
./karen: linha 68: /home/diogo/umbrel/events/triggers/: É um diretório

Docker containers

NAMES STATUS

Umbrel logs

Attaching to

Tor logs

Attaching to

App logs

bitcoin

Attaching to

bluewallet

Attaching to

electrs

Attaching to

lightning

Attaching to

lndg

Attaching to

mempool

Attaching to

==== Result ====

The debug script did not automatically detect any issues with your Umbrel.

stopping

It appears your Umbrel is not up-to-date- this may resolve the issue.

Try running the following

sudo ~/umbrel/scripts/update/update --repo getumbrel/umbrel#v0.5.4