Umbrel on Ubuntu 22.04 not starting all containers correctly on reboot

Support and Troubleshooting
1

I have come across some behavior that I can’t seem to solve. On reboot of the Umbrel server (Ubuntu 22.0.4) the node (Umbrel 0.5.3) does not start all the docker containers automatically and gets stuck in a container restart loop. If I do a simple stop (sudo ./stop) and then restart (sudo ./start) from the ~/scripts folder the containers come online and everything works as normal until the next server restart. I believe I’m missing something in the initial server docker start process that is causing this behavior. Any ideas to fix this? Debug below is after initial reboot of server.

=====================
= Umbrel debug info =

Umbrel version

0.5.3

Memory usage

           total        used        free      shared  buff/cache   available

Mem: 7.9G 463M 6.6G 1.0M 876M 7.2G
Swap: 4.1G 0B 4.1G

total: 5.8%
system: 5.8%
mempool: 0%
electrs: 0%
bitcoin: 0%

Memory monitor logs

43815 ? S 0:00 bash ./scripts/memory-monitor
Memory monitor is already running
43815 ? S 0:00 bash ./scripts/memory-monitor
Memory monitor is already running
2023-02-15 19:30:53 Memory monitor running!
2023-02-17 10:32:16 Memory monitor running!
2023-02-17 11:13:12 Memory monitor running!
2023-02-17 11:16:37 Memory monitor running!
2023-02-17 11:22:29 Memory monitor running!
2023-02-17 11:31:22 Memory monitor running!

Filesystem information

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu–vg-ubuntu–lv 1.3T 591G 652G 48% /
/dev/mapper/ubuntu–vg-ubuntu–lv 1.3T 591G 652G 48% /

Karen logs

Removing mempool_api_1 …
Removing mempool_mariadb_1 …
Removing mempool_tor_server_1 …
Removing mempool_web_1 …
Removing mempool_mariadb_1 … done
Removing mempool_web_1 … done
Removing mempool_api_1 … done
Removing mempool_tor_server_1 … done
Removing mempool_app_proxy_1 … done
Going to remove mempool_app_proxy_1, mempool_api_1, mempool_mariadb_1, mempool_tor_server_1, mempool_web_1
Stopping electrs_electrs_1 … done
Removing electrs_app_1 …
Removing electrs_app_proxy_1 …
Removing electrs_tor_server_1 …
Removing electrs_tor_1 …
Removing electrs_electrs_1 …
Removing electrs_electrs_1 … done
Removing electrs_tor_server_1 … done
Removing electrs_tor_1 … done
Removing electrs_app_1 … done
Removing electrs_app_proxy_1 … done
Going to remove electrs_app_1, electrs_app_proxy_1, electrs_tor_server_1, electrs_tor_1, electrs_electrs_1

Stopping Docker services…

Stopping nginx …
Stopping manager …
Stopping tor_server …
Stopping dashboard …
Stopping tor_proxy …
Stopping auth …
Stopping tor_server … done
Stopping nginx … done
Stopping auth … done
Stopping tor_proxy … done
Stopping manager … done
Stopping dashboard … done
Removing nginx …
Removing manager …
Removing tor_server …
Removing dashboard …
Removing tor_proxy …
Removing auth …
Removing dashboard … done
Removing tor_proxy … done
Removing nginx … done
Removing auth … done
Removing tor_server … done
Removing manager … done
Removing network umbrel_main_network

Docker containers

NAMES STATUS
nginx Up About a minute
tor_proxy Up Less than a second
manager Up Less than a second
auth Up Less than a second
dashboard Up Less than a second

Umbrel logs

Attaching to manager
manager | at attempt (/app/node_modules/passport/lib/middleware/authenticate.js:180:28)
manager | at JwtStrategy.strategy.fail (/app/node_modules/passport/lib/middleware/authenticate.js:302:9)
manager | at JwtStrategy.authenticate (/app/node_modules/passport-jwt/lib/strategy.js:96:21)
manager | at attempt (/app/node_modules/passport/lib/middleware/authenticate.js:366:16)
manager | at authenticate (/app/node_modules/passport/lib/middleware/authenticate.js:367:7)
manager | at jwt (/app/middlewares/auth.js:146:5)
manager | at Layer.handle [as handle_request] (/app/node_modules/express/lib/router/layer.js:95:5)
manager | at next (/app/node_modules/express/lib/router/route.js:137:13)
manager | ::ffff:10.21.21.2 - - [Fri, 17 Feb 2023 17:35:59 GMT] “POST /v1/account/refresh HTTP/1.0” 401 13 “-” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Safari/605.1.15”
manager |
manager | umbrel-manager
manager | Invalid JWT
manager | /v1/account/refresh
manager | NodeError: Invalid JWT
manager | at /app/middlewares/auth.js:137:19
manager | at allFailed (/app/node_modules/passport/lib/middleware/authenticate.js:107:18)
manager | at attempt (/app/node_modules/passport/lib/middleware/authenticate.js:180:28)
manager | at JwtStrategy.strategy.fail (/app/node_modules/passport/lib/middleware/authenticate.js:302:9)
manager | at JwtStrategy.authenticate (/app/node_modules/passport-jwt/lib/strategy.js:96:21)
manager | at attempt (/app/node_modules/passport/lib/middleware/authenticate.js:366:16)
manager | at authenticate (/app/node_modules/passport/lib/middleware/authenticate.js:367:7)
manager | at jwt (/app/middlewares/auth.js:146:5)
manager | at Layer.handle [as handle_request] (/app/node_modules/express/lib/router/layer.js:95:5)
manager | at next (/app/node_modules/express/lib/router/route.js:137:13)
manager | ::ffff:10.21.21.2 - - [Fri, 17 Feb 2023 17:35:59 GMT] “POST /v1/account/refresh HTTP/1.0” 401 13 “-” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Safari/605.1.15”
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Fri, 17 Feb 2023 17:36:38 GMT] “GET /v1/system/update-status HTTP/1.0” 200 65 “-” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Safari/605.1.15”
manager |
manager | umbrel-manager

Tor Proxy logs

Attaching to tor_proxy
tor_proxy | Feb 17 17:36:11.000 [notice] Bootstrapped 55% (loading_descriptors): Loading relay descriptors
tor_proxy | Feb 17 17:36:12.000 [notice] Bootstrapped 63% (loading_descriptors): Loading relay descriptors
tor_proxy | Feb 17 17:36:13.000 [notice] Bootstrapped 68% (loading_descriptors): Loading relay descriptors
tor_proxy | Feb 17 17:36:13.000 [notice] Bootstrapped 75% (enough_dirinfo): Loaded enough directory info to build circuits
tor_proxy | Feb 17 17:36:13.000 [notice] Bootstrapped 80% (ap_conn): Connecting to a relay to build circuits
tor_proxy | Feb 17 17:36:13.000 [notice] Bootstrapped 85% (ap_conn_done): Connected to a relay to build circuits
tor_proxy | Feb 17 17:36:13.000 [notice] Bootstrapped 89% (ap_handshake): Finishing handshake with a relay to build circuits
tor_proxy | Feb 17 17:36:13.000 [notice] Bootstrapped 90% (ap_handshake_done): Handshake finished with a relay to build circuits
tor_proxy | Feb 17 17:36:13.000 [notice] Bootstrapped 95% (circuit_create): Establishing a Tor circuit
tor_proxy | Feb 17 17:36:14.000 [notice] Bootstrapped 100% (done): Done

App logs

bitcoin

Attaching to

electrs

Attaching to

mempool

Attaching to

==== Result ====

The debug script did not automatically detect any issues with your Umbrel.

2

As luck would have it I solved my own problem hrs after posting this even though I’ve been troubleshooting for a few days.

● umbrel-startup.service
     Loaded: loaded (/etc/systemd/system/umbrel-startup.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2023-02-17 16:24:31 CST; 6s ago
     Process: 4099 ExecStart=/root/umbrel/scripts/start (code=exited, status=0/SUCCESS)

My systemctl unit file was pointing to the wrong ~umbrel/scripts/start location. After I corrected my service file to point to the correct script everything came back as normal.

● umbrel-startup.service
     Loaded: loaded (/etc/systemd/system/umbrel-startup.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2023-02-17 16:31:39 CST; 4s ago
     Process: 1082 ExecStart=/home/duckman/umbrel/scripts/start (code=exited, status=0/SUCCESS)