I began running Umbrel successfully nearly a year ago with minimal issues on several nodes. I recently lost power and one of my nodes returned as normal, but the other did not. I have a bitcoin wallet and lightning channels on this node. I have my seed and backups. Connecting via browser, I receive the following Error: Failed to start containers.
I am able to connect via ssh and verify that the SD card with the OS and SSD drive are functional, but continually receive this error when connecting via any browser. I bought a new micro SD card and flashed it to Umbrel v0.5.0. One interesting note is that my password reset to the default password when I began receiving this error.
Here is my setup:
- Raspberry Pi 4B - 8GB
- Samsung Portable SSD T7 - 2TB
- SanDisk 32GB Ultra microSDHC UHS-I Memory Card
- GeeekPi Armor Lite Aluminum Heatsink with PWM Speed Control Fan for Raspberry Pi 4 Model B
Here is the debug log from this Umbrel node.
=====================
= Umbrel debug info =
Umbrel version
0.5.0
Flashed OS version
v0.5.0
Raspberry Pi Model
Revision : d03114
Serial : 10000000678aebdd
Model : Raspberry Pi 4 Model B Rev 1.4
Firmware
Dec 1 2021 15:01:54
Copyright (c) 2012 Broadcom
version 71bd3109023a0c8575585ba87cbb374d2eeb038f (clean) (release) (start)
Temperature
temp=37.4’C
Throttling
throttled=0x0
Memory usage
total used free shared buff/cache available
Mem: 7.8G 115M 7.3G 8.0M 361M 7.6G
Swap: 4.1G 0B 4.1G
total: 1.5%
system: 1.5%
tor: 0%
ride-the-lightning: 0%
mempool: 0%
lnd: 0%
lightning: 0%
electrs: 0%
electrs: 0%
btcpay-server: 0%
bitcoin: 0%
bitcoin: 0%
Memory monitor logs
2022-07-21 16:02:17 Memory monitor running!
2022-07-21 16:02:29 Memory monitor running!
2022-07-21 16:02:40 Memory monitor running!
2022-07-21 16:02:52 Memory monitor running!
2022-07-21 16:03:04 Memory monitor running!
2022-07-21 16:03:16 Memory monitor running!
2022-07-21 16:03:28 Memory monitor running!
2022-07-21 16:03:40 Memory monitor running!
2022-07-21 16:03:52 Memory monitor running!
2022-07-21 16:04:04 Memory monitor running!
Filesystem information
Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 3.2G 25G 12% /
/dev/sda1 1.8T 534G 1.2T 31% /home/umbrel/umbrel
Startup service logs
Jul 21 16:04:04 umbrel umbrel startup[32709]: Starting backup monitor…
Jul 21 16:04:04 umbrel umbrel startup[32709]: Starting decoy backup trigger…
Jul 21 16:04:04 umbrel umbrel startup[32709]: Starting Docker services…
Jul 21 16:04:05 umbrel umbrel startup[32709]: The TOR_PROXY_PORT variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The TOR_PROXY_IP variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The NGINX_PORT variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The NGINX_IP variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The DASHBOARD_IP variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The MIDDLEWARE_IP variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The UMBREL_AUTH_SECRET variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The BITCOIN_P2P_PORT variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The BITCOIN_RPC_PORT variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The BITCOIN_RPC_USER variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The BITCOIN_RPC_PASS variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The BITCOIN_NETWORK variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The MANAGER_IP variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The BITCOIN_IP variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The LND_IP variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The AUTH_PORT variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The AUTH_IP variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The NETWORK_IP variable is not set. Defaulting to a blank string.
Jul 21 16:04:05 umbrel umbrel startup[32709]: The Compose file ‘./docker-compose.yml’ is invalid because:
Jul 21 16:04:05 umbrel umbrel startup[32709]: networks.default.ipam.config.subnet is invalid: should use the CIDR format
Jul 21 16:04:05 umbrel umbrel startup[32709]: services.tor_proxy.ports contains an invalid type, it should be a number, or an object
Jul 21 16:04:05 umbrel umbrel startup[32709]: services.auth.ports contains an invalid type, it should be a number, or an object
Jul 21 16:04:05 umbrel umbrel startup[32709]: services.nginx.ports contains an invalid type, it should be a number, or an object
Jul 21 16:04:05 umbrel umbrel startup[32709]: Failed to start containers
Jul 21 16:04:05 umbrel systemd[1]: umbrel-startup.service: Control process exited, code=exited, status=1/FAILURE
Jul 21 16:04:05 umbrel systemd[1]: umbrel-startup.service: Failed with result ‘exit-code’.
Jul 21 16:04:05 umbrel systemd[1]: Failed to start Umbrel Startup Service.
External storage service logs
– Logs begin at Thu 2022-07-21 15:17:01 UTC, end at Thu 2022-07-21 16:04:05 UTC. –
Jul 21 15:17:06 umbrel systemd[1]: Starting External Storage Mounter…
Jul 21 15:17:06 umbrel external storage mounter[486]: Running external storage mount script…
Jul 21 15:17:06 umbrel external storage mounter[486]: Found device “Samsung PSSD T7”
Jul 21 15:17:06 umbrel external storage mounter[486]: Blacklisting USB device IDs against UAS driver…
Jul 21 15:17:06 umbrel external storage mounter[486]: Rebinding USB drivers…
Jul 21 15:17:06 umbrel external storage mounter[486]: Checking USB devices are back…
Jul 21 15:17:06 umbrel external storage mounter[486]: Waiting for USB devices…
Jul 21 15:17:07 umbrel external storage mounter[486]: Waiting for USB devices…
Jul 21 15:17:08 umbrel external storage mounter[486]: Waiting for USB devices…
Jul 21 15:17:09 umbrel external storage mounter[486]: Checking if the device is ext4…
Jul 21 15:17:09 umbrel external storage mounter[486]: Yes, it is ext4
Jul 21 15:17:09 umbrel external storage mounter[486]: Checking if device contains an Umbrel install…
Jul 21 15:17:09 umbrel external storage mounter[486]: Yes, it contains an Umbrel install
Jul 21 15:17:09 umbrel external storage mounter[486]: Bind mounting external storage over local Umbrel installation…
Jul 21 15:17:09 umbrel external storage mounter[486]: Bind mounting external storage over local Docker data dir…
Jul 21 15:17:09 umbrel external storage mounter[486]: Bind mounting external storage to /swap
Jul 21 15:17:10 umbrel external storage mounter[486]: Bind mounting SD card root at /sd-card…
Jul 21 15:17:10 umbrel external storage mounter[486]: Checking Umbrel root is now on external storage…
Jul 21 15:17:11 umbrel external storage mounter[486]: Checking /var/lib/docker is now on external storage…
Jul 21 15:17:11 umbrel external storage mounter[486]: Checking /swap is now on external storage…
Jul 21 15:17:11 umbrel external storage mounter[486]: Setting up swapfile
Jul 21 15:17:11 umbrel external storage mounter[486]: Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
Jul 21 15:17:11 umbrel external storage mounter[486]: no label, UUID=74b6f1ed-4f73-4531-9e47-7fb91be3ddf0
Jul 21 15:17:11 umbrel external storage mounter[486]: Checking SD Card root is bind mounted at /sd-root…
Jul 21 15:17:11 umbrel external storage mounter[486]: Starting external drive mount monitor…
Jul 21 15:17:11 umbrel external storage mounter[486]: Mount script completed successfully!
Jul 21 15:17:11 umbrel systemd[1]: Started External Storage Mounter.
External storage SD card update service logs
– Logs begin at Thu 2022-07-21 15:17:01 UTC, end at Thu 2022-07-21 16:04:05 UTC. –
Jul 21 15:17:19 umbrel systemd[1]: Starting External Storage SDcard Updater…
Jul 21 15:17:19 umbrel external storage updater[938]: Checking if SD card Umbrel is newer than external storage…
Jul 21 15:17:19 umbrel external storage updater[938]: No, SD version is not newer, exiting.
Jul 21 15:17:19 umbrel systemd[1]: Started External Storage SDcard Updater.
Karen logs
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
Docker containers
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Umbrel logs
ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.auth.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
Tor logs
ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.auth.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
App logs
bitcoin
./scripts/app: line 81: NETWORK_IP: unbound variable
btcpay-server
./scripts/app: line 81: NETWORK_IP: unbound variable
electrs
./scripts/app: line 81: NETWORK_IP: unbound variable
lightning
./scripts/app: line 81: NETWORK_IP: unbound variable
mempool
./scripts/app: line 81: NETWORK_IP: unbound variable
ride-the-lightning
./scripts/app: line 81: NETWORK_IP: unbound variable
==== Result ====
The debug script did not automatically detect any issues with your Umbrel.
Here is the docker.service status.
umbrel@umbrel:/var/run $ sudo systemctl status docker.service
[sudo] password for umbrel:
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Thu 2022-07-21 16:08:14 UTC; 1s ago
Docs: https://docs.docker.com
Process: 11295 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=2
Main PID: 11295 (code=exited, status=2)
Here is the journalctl -xe.
umbrel@umbrel:/var/run $ journalctl -xe
Jul 21 16:16:03 umbrel dockerd[31831]: /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go
Jul 21 16:16:03 umbrel dockerd[31831]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x4000
Jul 21 16:16:03 umbrel dockerd[31831]: /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go
Jul 21 16:16:03 umbrel dockerd[31831]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(...)
Jul 21 16:16:03 umbrel dockerd[31831]: /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go
Jul 21 16:16:03 umbrel dockerd[31831]: main.main()
Jul 21 16:16:03 umbrel dockerd[31831]: /go/src/github.com/docker/docker/cmd/dockerd/docker.go:97 +0x1b4
Jul 21 16:16:03 umbrel systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
– Subject: Unit process exited
– Defined-By: systemd
– Support: Debian -- User Support
– An ExecStart= process belonging to unit docker.service has exited.
– The process’ exit code is ‘exited’ and its exit status is 2.
Jul 21 16:16:03 umbrel systemd[1]: docker.service: Failed with result ‘exit-code’.
– Subject: Unit failed
– Defined-By: systemd
– Support: Debian -- User Support
– The unit docker.service has entered the ‘failed’ state with result ‘exit-code’.
Jul 21 16:16:03 umbrel systemd[1]: Failed to start Docker Application Container Engine.
– Subject: A start job for unit docker.service has failed
– Defined-By: systemd
– Support: Debian -- User Support
– A start job for unit docker.service has finished with a failure.
– The job identifier is 12539 and the job result is failed.
I admit I am out of my depth here, but it appears that the Docker Application Container Engine is the issue. I am just not sure how to troubleshoot from here. I appreciate any helpful posts, links, or advice. Thank you for your time and consideration.
Jaes aka OpenWaterSwim