Error: Failed to start containers, alternative solutions?

I’ve come back around to checking in on my umbrel node after maaaany months leaving it unattended and after initially being able to log in, I’m not receiving this error.

I have tried:

  • reflashing SD card. (initially worked for a bit, seems to have broken again after updating all the apps I had)
  • power off / power on
  • some of the solutions found in threads with this similar issue: truncating logs, stop/reset docker

continuing to have the issue. Curiously, I can now only get in through SSH with the default password instead of my usual password. No idea what happened there; mildly concerning. Running the debug log has produced the following.

=====================
= Umbrel debug info =
=====================

Umbrel version
--------------
0.5.4

Flashed OS version
-----------------
v0.5.4-patch-1

Raspberry Pi Model
------------------
Revision        : c03112
Serial          : 1000000012b7590f
Model           : Raspberry Pi 4 Model B Rev 1.2

Firmware
--------
May  9 2023 12:16:34
Copyright (c) 2012 Broadcom
version 30aa0d70ab280427ba04ebc718c81d4350b9d394 (clean) (release) (start)

Temperature
-----------
temp=44.3'C

Throttling
----------
throttled=0x0

Memory usage
------------
              total        used        free      shared  buff/cache   available
Mem:           3.8G        146M        152M        8.0M        3.5G        3.6G
Swap:          4.1G        0.0K        4.1G

total: 3.9%
system: 3.9%
thunderhub: 0%
samourai-server: 0%
ride-the-lightning: 0%
mempool: 0%
lndg: 0%
lnbits: 0%
lightning-terminal: 0%
lightning: 0%
electrs: 0%
bitcoin: 0%

Memory monitor logs
-------------------
2024-02-29 02:25:47 Memory monitor running!
2024-02-29 02:26:00 Memory monitor running!
2024-02-29 02:26:13 Memory monitor running!
2024-02-29 02:26:26 Memory monitor running!
2024-02-29 02:26:39 Memory monitor running!
2024-02-29 02:26:52 Memory monitor running!
2024-02-29 02:27:06 Memory monitor running!
2024-02-29 02:27:19 Memory monitor running!
2024-02-29 02:27:32 Memory monitor running!
2024-02-29 02:27:45 Memory monitor running!

Filesystem information
----------------------
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        30G  7.4G   21G  27% /
/dev/sda1       916G  642G  229G  74% /home/umbrel/umbrel

Startup service logs
--------------------
Feb 29 02:27:32 umbrel umbrel startup[29143]: Starting umbreld...
Feb 29 02:27:32 umbrel umbrel startup[29143]: Starting Docker services...
Feb 29 02:27:34 umbrel umbrel startup[29143]: Pulling dashboard (getumbrel/dashboard:0.5.4@sha256:0662567cc1b0ba2179f65bfae0472d085a7e79f53f502e8d1789a1ecba58744b)...
Feb 29 02:27:35 umbrel umbrel startup[29143]: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Feb 29 02:27:35 umbrel umbrel startup[29143]: Failed to start containers
Feb 29 02:27:35 umbrel systemd[1]: umbrel-startup.service: Control process exited, code=exited, status=1/FAILURE
Feb 29 02:27:35 umbrel systemd[1]: umbrel-startup.service: Failed with result 'exit-code'.
Feb 29 02:27:35 umbrel systemd[1]: Failed to start Umbrel Startup Service.
Feb 29 02:27:45 umbrel systemd[1]: umbrel-startup.service: Service RestartSec=10s expired, scheduling restart.
Feb 29 02:27:45 umbrel systemd[1]: umbrel-startup.service: Scheduled restart job, restart counter is at 55.
Feb 29 02:27:45 umbrel systemd[1]: Stopped Umbrel Startup Service.
Feb 29 02:27:45 umbrel systemd[1]: Starting Umbrel Startup Service...
Feb 29 02:27:45 umbrel umbrel startup[30390]: ======================================
Feb 29 02:27:45 umbrel umbrel startup[30390]: ============= STARTING ===============
Feb 29 02:27:45 umbrel umbrel startup[30390]: ============== UMBREL ================
Feb 29 02:27:45 umbrel umbrel startup[30390]: ======================================
Feb 29 02:27:45 umbrel umbrel startup[30390]: Setting environment variables...
Feb 29 02:27:45 umbrel umbrel startup[30390]: Starting karen...
Feb 29 02:27:45 umbrel umbrel startup[30390]: Starting status monitors...
Feb 29 02:27:45 umbrel umbrel startup[30390]: Starting memory monitor...
Feb 29 02:27:45 umbrel umbrel startup[30390]: Starting backup monitor...
Feb 29 02:27:45 umbrel umbrel startup[30390]: Starting decoy backup trigger...
Feb 29 02:27:45 umbrel umbrel startup[30390]: Starting umbreld...
Feb 29 02:27:45 umbrel umbrel startup[30390]: Starting Docker services...
Feb 29 02:27:47 umbrel umbrel startup[30390]: Pulling manager (getumbrel/manager:0.5.4@sha256:400559b951ec19b9e79c5eec96bf2a11efd57a7f9c65f9a8cc03e6ff833e9a59)...
Feb 29 02:27:48 umbrel umbrel startup[30390]: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Feb 29 02:27:48 umbrel umbrel startup[30390]: Failed to start containers
Feb 29 02:27:48 umbrel systemd[1]: umbrel-startup.service: Control process exited, code=exited, status=1/FAILURE
Feb 29 02:27:48 umbrel systemd[1]: umbrel-startup.service: Failed with result 'exit-code'.
Feb 29 02:27:48 umbrel systemd[1]: Failed to start Umbrel Startup Service.

External storage service logs
-----------------------------
Feb 29 02:15:05 umbrel systemd[1]: Starting External Storage Mounter...
Feb 29 02:15:05 umbrel external storage mounter[496]: Running external storage mount script...
Feb 29 02:15:05 umbrel external storage mounter[496]: Found device "SanDisk SDSSDA-1T00"
Feb 29 02:15:05 umbrel external storage mounter[496]: Blacklisting USB device IDs against UAS driver...
Feb 29 02:15:05 umbrel external storage mounter[496]: Rebinding USB drivers...
Feb 29 02:15:06 umbrel external storage mounter[496]: Checking USB devices are back...
Feb 29 02:15:06 umbrel external storage mounter[496]: Waiting for USB devices...
Feb 29 02:15:07 umbrel external storage mounter[496]: Waiting for USB devices...
Feb 29 02:15:08 umbrel external storage mounter[496]: Checking if the device is ext4...
Feb 29 02:15:08 umbrel external storage mounter[496]: Yes, it is ext4
Feb 29 02:15:08 umbrel external storage mounter[496]: Checking filesystem for corruption...
Feb 29 02:15:08 umbrel external storage mounter[496]: e2fsck 1.44.5 (15-Dec-2018)
Feb 29 02:15:08 umbrel external storage mounter[496]: umbrel: clean, 477172/61046784 files, 172144481/244175218 blocks
Feb 29 02:15:08 umbrel external storage mounter[496]: Mounting partition...
Feb 29 02:15:08 umbrel external storage mounter[496]: Checking if device contains an Umbrel install...
Feb 29 02:15:09 umbrel external storage mounter[496]: Yes, it contains an Umbrel install
Feb 29 02:15:09 umbrel external storage mounter[496]: Bind mounting external storage over local Umbrel installation...
Feb 29 02:15:09 umbrel external storage mounter[496]: Bind mounting external storage over local Docker data dir...
Feb 29 02:15:09 umbrel external storage mounter[496]: Bind mounting external storage to /swap
Feb 29 02:15:09 umbrel external storage mounter[496]: Bind mounting SD card root at /sd-card...
Feb 29 02:15:09 umbrel external storage mounter[496]: Checking Umbrel root is now on external storage...
Feb 29 02:15:10 umbrel external storage mounter[496]: Checking /var/lib/docker is now on external storage...
Feb 29 02:15:10 umbrel external storage mounter[496]: Checking /swap is now on external storage...
Feb 29 02:15:10 umbrel external storage mounter[496]: Setting up swapfile
Feb 29 02:15:10 umbrel external storage mounter[496]: Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
Feb 29 02:15:10 umbrel external storage mounter[496]: no label, UUID=d9f151bd-1ca8-4569-821d-fc622eed7b78
Feb 29 02:15:10 umbrel external storage mounter[496]: Checking SD Card root is bind mounted at /sd-root...
Feb 29 02:15:10 umbrel external storage mounter[496]: Starting external drive mount monitor...
Feb 29 02:15:10 umbrel external storage mounter[496]: Mount script completed successfully!
Feb 29 02:15:10 umbrel systemd[1]: Started External Storage Mounter.

External storage SD card update service logs
--------------------------------------------
-- Logs begin at Thu 2024-02-29 02:14:59 UTC, end at Thu 2024-02-29 02:27:48 UTC. --
Feb 29 02:15:25 umbrel systemd[1]: Starting External Storage SDcard Updater...
Feb 29 02:15:25 umbrel external storage updater[949]: Checking if SD card Umbrel is newer than external storage...
Feb 29 02:15:25 umbrel external storage updater[949]: No, SD version is not newer, exiting.
Feb 29 02:15:25 umbrel systemd[1]: Started External Storage SDcard Updater.

Karen logs
----------

karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events

Docker containers
-----------------
NAMES     STATUS

Umbrel logs
-----------

Attaching to

Tor Proxy logs
--------

Attaching to

App logs
--------

bitcoin

Attaching to

electrs

Attaching to

lightning

Attaching to

lightning-terminal

Attaching to

lnbits

Attaching to

lndg

Attaching to

mempool

Attaching to

ride-the-lightning

Attaching to

samourai-server

Attaching to

thunderhub

Attaching to
================
==== Result ====
================
The debug script did not automatically detect any issues with your Umbrel.

Yeah I just had to do a restart tonight and I’m getting both “Error: System service failed” and “Failed to Start Containers” - I even flashed a brand new SD card and still no luck.

On a Raspberry Pi 4 with 8GB of RAM.

Would really like to get this sorted out.

hey @Marky5

Sorry to hear about your issue here. I assume you have tried the following?

Is there a way to uninstall/remove apps via ssh?

Yes I have :frowning:

Sure you can use this:

Run sudo ~/umbrel/scripts/app uninstall app-name

Can you please try generating some troubleshooting logs that we can look at:

alright well that didn’t work. I uninstalled everything except bitcoin, electrs, lightning, and lightning-terminal. Running the debus log again has it hang in the app logs section after it attaches to those four with “attaching to.”

Can I safely uninstall the rest of those apps without losing the key bits of information regarding my wallet/node/channels? OR can I get a backup off ssh and recover it after starting completely from scratch?

It’s a big one…

=====================
= Umbrel debug info =
=====================

Umbrel version
--------------
0.5.4

Flashed OS version
-----------------
v0.5.1

Raspberry Pi Model
------------------
Revision	: d03115
Serial		: 1000000073XXXXXX
Model		: Raspberry Pi 4 Model B Rev 1.5

Firmware
--------
May  9 2023 12:16:34 
Copyright (c) 2012 Broadcom
version 30aa0d70ab280427ba04ebc718c81d4350b9d394 (clean) (release) (start)

Temperature
-----------
temp=43.3'C

Throttling
----------
throttled=0x0

Memory usage
------------
              total        used        free      shared  buff/cache   available
Mem:           7.8G        109M        6.9G        8.0M        848M        7.6G
Swap:          4.1G          0B        4.1G

total: 1.4%
system: 1.4%
trilium-notes: 0%
tailscale: 0%
syncthing: 0%
overseerr: 0%
nextcloud: 0%
jackett: 0%
ipfs-podcasting: 0%
invidious: 0%
homebox: 0%
heimdall: 0%
file-browser: 0%
element: 0%
databag: 0%
cloudflared: 0%
back-that-mac-up: 0%
autobrr: 0%
audiobookshelf: 0%

Memory monitor logs
-------------------
2024-03-01 16:11:48 Memory monitor running!
2024-03-01 16:12:05 Memory monitor running!
2024-03-01 16:12:18 Memory monitor running!
2024-03-01 16:12:31 Memory monitor running!
2024-03-01 16:12:43 Memory monitor running!
2024-03-01 16:12:56 Memory monitor running!
2024-03-01 16:13:13 Memory monitor running!
2024-03-01 16:13:26 Memory monitor running!
2024-03-01 16:13:39 Memory monitor running!
2024-03-01 16:13:52 Memory monitor running!

Filesystem information
----------------------
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        29G  3.7G   25G  14% /
/dev/sda1       916G   69G  802G   8% /home/umbrel/umbrel

Startup service logs
--------------------
Mar 01 16:13:55 umbrel umbrel startup[12448]:   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 648, in send
Mar 01 16:13:55 umbrel umbrel startup[12448]:     r = adapter.send(request, **kwargs)
Mar 01 16:13:55 umbrel umbrel startup[12448]:   File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
Mar 01 16:13:55 umbrel umbrel startup[12448]:     raise ConnectionError(err, request=request)
Mar 01 16:13:55 umbrel umbrel startup[12448]: requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))
Mar 01 16:13:55 umbrel umbrel startup[12448]: During handling of the above exception, another exception occurred:
Mar 01 16:13:55 umbrel umbrel startup[12448]: Traceback (most recent call last):
Mar 01 16:13:55 umbrel umbrel startup[12448]:   File "/usr/local/bin/docker-compose", line 10, in <module>
Mar 01 16:13:55 umbrel umbrel startup[12448]:     sys.exit(main())
Mar 01 16:13:55 umbrel umbrel startup[12448]:   File "/usr/local/lib/python3.7/dist-packages/compose/cli/main.py", line 81, in main
Mar 01 16:13:55 umbrel umbrel startup[12448]:     command_func()
Mar 01 16:13:55 umbrel umbrel startup[12448]:   File "/usr/local/lib/python3.7/dist-packages/compose/cli/main.py", line 200, in perform_command
Mar 01 16:13:55 umbrel umbrel startup[12448]:     project = project_from_options('.', options)
Mar 01 16:13:55 umbrel umbrel startup[12448]:   File "/usr/local/lib/python3.7/dist-packages/compose/cli/command.py", line 70, in project_from_options
Mar 01 16:13:55 umbrel umbrel startup[12448]:     enabled_profiles=get_profiles_from_options(options, environment)
Mar 01 16:13:55 umbrel umbrel startup[12448]:   File "/usr/local/lib/python3.7/dist-packages/compose/cli/command.py", line 153, in get_project
Mar 01 16:13:55 umbrel umbrel startup[12448]:     verbose=verbose, version=api_version, context=context, environment=environment
Mar 01 16:13:55 umbrel umbrel startup[12448]:   File "/usr/local/lib/python3.7/dist-packages/compose/cli/docker_client.py", line 43, in get_client
Mar 01 16:13:55 umbrel umbrel startup[12448]:     environment=environment, tls_version=get_tls_version(environment)
Mar 01 16:13:55 umbrel umbrel startup[12448]:   File "/usr/local/lib/python3.7/dist-packages/compose/cli/docker_client.py", line 170, in docker_client
Mar 01 16:13:55 umbrel umbrel startup[12448]:     client = APIClient(use_ssh_client=not use_paramiko_ssh, **kwargs)
Mar 01 16:13:55 umbrel umbrel startup[12448]:   File "/usr/local/lib/python3.7/dist-packages/docker/api/client.py", line 197, in __init__
Mar 01 16:13:55 umbrel umbrel startup[12448]:     self._version = self._retrieve_server_version()
Mar 01 16:13:55 umbrel umbrel startup[12448]:   File "/usr/local/lib/python3.7/dist-packages/docker/api/client.py", line 222, in _retrieve_server_version
Mar 01 16:13:55 umbrel umbrel startup[12448]:     f'Error while fetching server API version: {e}'
Mar 01 16:13:55 umbrel umbrel startup[12448]: docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))
Mar 01 16:13:55 umbrel umbrel startup[12448]: Failed to start containers
Mar 01 16:13:55 umbrel systemd[1]: umbrel-startup.service: Control process exited, code=exited, status=1/FAILURE
Mar 01 16:13:55 umbrel systemd[1]: umbrel-startup.service: Failed with result 'exit-code'.
Mar 01 16:13:55 umbrel systemd[1]: Failed to start Umbrel Startup Service.

External storage service logs
-----------------------------
Mar 01 03:47:26 umbrel external storage mounter[490]: Checking USB devices are back...
Mar 01 03:47:26 umbrel external storage mounter[490]: Waiting for USB devices...
Mar 01 03:47:27 umbrel external storage mounter[490]: Waiting for USB devices...
Mar 01 03:47:28 umbrel external storage mounter[490]: Waiting for USB devices...
Mar 01 03:47:29 umbrel external storage mounter[490]: Checking if the device is ext4...
Mar 01 03:47:29 umbrel external storage mounter[490]: Yes, it is ext4
Mar 01 03:47:29 umbrel external storage mounter[490]: Checking filesystem for corruption...
Mar 01 03:47:29 umbrel external storage mounter[490]: e2fsck 1.44.5 (15-Dec-2018)
Mar 01 03:47:30 umbrel external storage mounter[490]: Superblock last mount time is in the future.
Mar 01 03:47:30 umbrel external storage mounter[490]:         (by less than a day, probably due to the hardware clock being incorrectly set)
Mar 01 03:47:30 umbrel external storage mounter[490]: Superblock last write time is in the future.
Mar 01 03:47:30 umbrel external storage mounter[490]:         (by less than a day, probably due to the hardware clock being incorrectly set)
Mar 01 03:47:30 umbrel external storage mounter[490]: umbrel: clean, 638449/61046784 files, 21952288/244182016 blocks
Mar 01 03:47:30 umbrel external storage mounter[490]: Mounting partition...
Mar 01 03:47:30 umbrel external storage mounter[490]: Checking if device contains an Umbrel install...
Mar 01 03:47:30 umbrel external storage mounter[490]: Yes, it contains an Umbrel install
Mar 01 03:47:30 umbrel external storage mounter[490]: Bind mounting external storage over local Umbrel installation...
Mar 01 03:47:30 umbrel external storage mounter[490]: Bind mounting external storage over local Docker data dir...
Mar 01 03:47:30 umbrel external storage mounter[490]: Bind mounting external storage to /swap
Mar 01 03:47:30 umbrel external storage mounter[490]: Bind mounting SD card root at /sd-card...
Mar 01 03:47:30 umbrel external storage mounter[490]: Checking Umbrel root is now on external storage...
Mar 01 03:47:31 umbrel external storage mounter[490]: Checking /var/lib/docker is now on external storage...
Mar 01 03:47:31 umbrel external storage mounter[490]: Checking /swap is now on external storage...
Mar 01 03:47:31 umbrel external storage mounter[490]: Setting up swapfile
Mar 01 03:47:31 umbrel external storage mounter[490]: Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
Mar 01 03:47:31 umbrel external storage mounter[490]: no label, UUID=248372db-df09-485e-9f5f-74b5ad1c7ebd
Mar 01 03:47:31 umbrel external storage mounter[490]: Checking SD Card root is bind mounted at /sd-root...
Mar 01 03:47:31 umbrel external storage mounter[490]: Starting external drive mount monitor...
Mar 01 03:47:31 umbrel external storage mounter[490]: Mount script completed successfully!
Mar 01 03:47:31 umbrel systemd[1]: Started External Storage Mounter.

External storage SD card update service logs
--------------------------------------------
-- Logs begin at Fri 2024-03-01 03:47:20 UTC, end at Fri 2024-03-01 16:13:55 UTC. --
Mar 01 03:47:38 umbrel systemd[1]: Starting External Storage SDcard Updater...
Mar 01 03:47:38 umbrel external storage updater[761]: Checking if SD card Umbrel is newer than external storage...
Mar 01 03:47:39 umbrel external storage updater[761]: No, SD version is not newer, exiting.
Mar 01 03:47:39 umbrel systemd[1]: Started External Storage SDcard Updater.

Karen logs
----------

karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events

Docker containers
-----------------
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Umbrel logs
-----------

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 355, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.7/http/client.py", line 1264, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1310, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1259, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1034, in _send_output
    self.send(msg)
  File "/usr/lib/python3.7/http/client.py", line 974, in send
    self.connect()
  File "/usr/local/lib/python3.7/dist-packages/docker/transport/unixconn.py", line 30, in connect
    sock.connect(self.unix_socket)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 368, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 692, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 355, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.7/http/client.py", line 1264, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1310, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1259, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1034, in _send_output
    self.send(msg)
  File "/usr/lib/python3.7/http/client.py", line 974, in send
    self.connect()
  File "/usr/local/lib/python3.7/dist-packages/docker/transport/unixconn.py", line 30, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/usr/local/lib/python3.7/dist-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/usr/local/lib/python3.7/dist-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 548, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 535, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 648, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/docker-compose", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/main.py", line 81, in main
    command_func()
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/main.py", line 200, in perform_command
    project = project_from_options('.', options)
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/command.py", line 70, in project_from_options
    enabled_profiles=get_profiles_from_options(options, environment)
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/command.py", line 153, in get_project
    verbose=verbose, version=api_version, context=context, environment=environment
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/docker_client.py", line 43, in get_client
    environment=environment, tls_version=get_tls_version(environment)
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/docker_client.py", line 170, in docker_client
    client = APIClient(use_ssh_client=not use_paramiko_ssh, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/usr/local/lib/python3.7/dist-packages/docker/api/client.py", line 222, in _retrieve_server_version
    f'Error while fetching server API version: {e}'
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

Tor Proxy logs
--------

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 385, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 381, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.7/http/client.py", line 1356, in getresponse
    response.begin()
  File "/usr/lib/python3.7/http/client.py", line 319, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.7/http/client.py", line 280, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 368, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 692, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 385, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 381, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.7/http/client.py", line 1356, in getresponse
    response.begin()
  File "/usr/lib/python3.7/http/client.py", line 319, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.7/http/client.py", line 280, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/usr/local/lib/python3.7/dist-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/usr/local/lib/python3.7/dist-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 548, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 535, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 648, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/docker-compose", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/main.py", line 81, in main
    command_func()
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/main.py", line 200, in perform_command
    project = project_from_options('.', options)
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/command.py", line 70, in project_from_options
    enabled_profiles=get_profiles_from_options(options, environment)
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/command.py", line 153, in get_project
    verbose=verbose, version=api_version, context=context, environment=environment
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/docker_client.py", line 43, in get_client
    environment=environment, tls_version=get_tls_version(environment)
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/docker_client.py", line 170, in docker_client
    client = APIClient(use_ssh_client=not use_paramiko_ssh, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/usr/local/lib/python3.7/dist-packages/docker/api/client.py", line 222, in _retrieve_server_version
    f'Error while fetching server API version: {e}'
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

App logs
--------

audiobookshelf

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 355, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.7/http/client.py", line 1264, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1310, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1259, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1034, in _send_output
    self.send(msg)
  File "/usr/lib/python3.7/http/client.py", line 974, in send
    self.connect()
  File "/usr/local/lib/python3.7/dist-packages/docker/transport/unixconn.py", line 30, in connect
    sock.connect(self.unix_socket)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 368, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 692, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 355, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.7/http/client.py", line 1264, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1310, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1259, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1034, in _send_output
    self.send(msg)
  File "/usr/lib/python3.7/http/client.py", line 974, in send
    self.connect()
  File "/usr/local/lib/python3.7/dist-packages/docker/transport/unixconn.py", line 30, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/usr/local/lib/python3.7/dist-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/usr/local/lib/python3.7/dist-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 548, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 535, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 648, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/docker-compose", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/main.py", line 81, in main
    command_func()
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/main.py", line 200, in perform_command
    project = project_from_options('.', options)
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/command.py", line 70, in project_from_options
    enabled_profiles=get_profiles_from_options(options, environment)
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/command.py", line 153, in get_project
    verbose=verbose, version=api_version, context=context, environment=environment
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/docker_client.py", line 43, in get_client
    environment=environment, tls_version=get_tls_version(environment)
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/docker_client.py", line 170, in docker_client
    client = APIClient(use_ssh_client=not use_paramiko_ssh, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/usr/local/lib/python3.7/dist-packages/docker/api/client.py", line 222, in _retrieve_server_version
    f'Error while fetching server API version: {e}'
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

================
==== Result ====
================
The debug script did not automatically detect any issues with your Umbrel.

Had to delete some of the log because it was too long to post. But it’s the same errors over and over again.

Kairos, did you install Docker on your system yourself OR did you let the Umbrel install script install Docker? Just a thought, since that could be a possible cause.

I’m not a Linux guy so I’ve tried to let Umbrel handle everything.

is there a verbose output I could see if I try to relaunch Docker from SSH? Maybe get a better idea of where it’s breaking?

If you install portainer-ce (not portainer-be and NOT the portainer umbrel plugin), you can use that to view logs and or login to the container you think is having a problem (with little effort). Portainer will tell you exactly which one is failing

I don’t have to do anything special in my case, do I? I can just follow the installation directions here, log in through the web server it creates, and then presumably poke around until I find out how to find the logs?

yep, sounds right.

installation fails on the second step.

docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest

results in:

Unable to find image 'portainer/portainer-ce:latest' locally
docker: Error response from daemon: Get "https://registry-1.docker.io/v2/portainer/portainer-ce/manifests/sha256:908d04d20e86f07a50b0f1a029a111b89aa3089b7fc7fdf68ec1c71b025f36cd": dial tcp [2600:1f18:2148:bc01:571f:e759:a87a:2961]:443: connect: network is unreachable.
See 'docker run --help'.

Looks like this error has occured for others outside of Umbrel. Resolution seems to either be something with the docker proxy settings or by running

docker logout
docker login

but the latter then prompts for credentials which I have no idea about.

In lieu of no new responses in the last couple days, I’ve been looking for a way to maybe see if I can find something in the docker logs. This is what’s in it; no idea if its relevant:

Mar 06 03:38:08 umbrel dockerd[22489]: time="2024-03-06T03:38:08.181758041Z" level=error msg="Handler for POST /v1.43/images/create returned error: toomanyrequests: Yo
Mar 06 03:38:21 umbrel dockerd[22489]: time="2024-03-06T03:38:21.204014814Z" level=warning msg="reference for unknown type: " digest="sha256:2ace83f22501f58857fa9b4030
Mar 06 03:38:21 umbrel dockerd[22489]: time="2024-03-06T03:38:21.495072986Z" level=error msg="Not continuing with pull after error: toomanyrequests: You have reached y
Mar 06 03:38:21 umbrel dockerd[22489]: time="2024-03-06T03:38:21.517244997Z" level=error msg="Handler for POST /v1.43/images/create returned error: toomanyrequests: Yo
Mar 06 03:38:34 umbrel dockerd[22489]: time="2024-03-06T03:38:34.278739646Z" level=warning msg="reference for unknown type: " digest="sha256:400559b951ec19b9e79c5eec96
Mar 06 03:38:34 umbrel dockerd[22489]: time="2024-03-06T03:38:34.597498550Z" level=error msg="Not continuing with pull after error: toomanyrequests: You have reached y
Mar 06 03:38:34 umbrel dockerd[22489]: time="2024-03-06T03:38:34.621557326Z" level=error msg="Handler for POST /v1.43/images/create returned error: toomanyrequests: Yo
Mar 06 03:38:47 umbrel dockerd[22489]: time="2024-03-06T03:38:47.172765838Z" level=warning msg="reference for unknown type: " digest="sha256:2ace83f22501f58857fa9b4030
Mar 06 03:38:47 umbrel dockerd[22489]: time="2024-03-06T03:38:47.487244038Z" level=error msg="Not continuing with pull after error: toomanyrequests: You have reached y
Mar 06 03:38:47 umbrel dockerd[22489]: time="2024-03-06T03:38:47.515704793Z" level=error msg="Handler for POST /v1.43/images/create returned error: toomanyrequests: Yo
~

I tried to list docker’s containers, but “docker ps -a” or the equivalent commands to list containers return an empty list. That doesn’t seem good?

No solutions, huh? My node is just going to stay stuck? So what do I do at this point, unplug the node, let the channels force close, and then recover the on-chain wallet?

Yeah, I don’t know. I still have some issues (as I cannot connect Specter to my Node anymore). I saw on the Umbrel page, that they’re probably busy with a big update on the OS. So my hope lies in there. As a last resort like you said: recover it from scratch.

All these recovery instructions assume you can get a channel backup via the UI but that’s not an option if Umbrel is borked. the typical recovery instructions imply I should be able to grab one with the scp command via SSH but I’ve ended up down a rabbit hole on how to get/operate that from a windows machine.

Preferably it would be nice to also find a way to back up the bitcoin ledger and restore it to a new umbrel install so I can save myself the time of letting it slowly re-download.

Agreed, same here. Troubleshooting teaches you a lot and still, a long way to go. Increasing the cache size saved me a lot of time downloading the blockchain. It took me a little longer than a day.