I was having issues accessing Thunderhub and the nostr relay on my node and I decided to reboot it. After nothing happened for over 1hr, I tried turning the node on and off a few times, but each time, the light on the hard disk went off after a few minutes and my browser is stuck on “Starting Umbrel…”.
Would appreciate any help. I have a basic setup with a raspberry pi 4 and a 1TB SSD. I’m running the latest version of Umbrel OS
We need a log to help you.
I would recommend to connect a display to your PC/PI/etc. and tell us what the linux shell want to tell us.
Thanks for getting back. I ran sudo ~/umbrel/scripts/debug
Please see the logs below. The SSH is super slow and shaky and after 1 hr, it was stuck at “Docker containers”. The logs below are everything until that point:
0.5.3
v0.4.14
Revision : c03114
Serial : 10000000c57f18cc
Model : Raspberry Pi 4 Model B Rev 1.4
Dec 1 2021 15:01:54
Copyright © 2012 Broadcom
version 71bd3109023a0c8575585ba87cbb374d2eeb038f (clean) (release) (start)
temp=28.2’C
throttled=0x0
total used free shared buff/cache available
Mem: 3.8G 137M 3.3G 8.0M 339M 3.6G
Swap: 4.1G 0B 4.1G
total: 3.6%
system: 3.6%
thunderhub: 0%
tailscale: 0%
robosats: 0%
ride-the-lightning: 0%
nostr-relay: 0%
mempool: 0%
lnbits: 0%
lightning-terminal: 0%
lightning-shell: 0%
lightning: 0%
electrs: 0%
btc-rpc-explorer: 0%
btcpay-server: 0%
bluewallet: 0%
bitcoin: 0%
2023-02-19 22:35:32 Memory monitor running!
2023-02-19 22:43:21 Memory monitor running!
2023-02-19 22:47:33 Memory monitor running!
2023-02-19 22:53:06 Memory monitor running!
2023-02-19 22:57:18 Memory monitor running!
2023-02-20 12:32:37 Memory monitor running!
2023-02-20 12:44:07 Memory monitor running!
2023-02-20 12:48:19 Memory monitor running!
2023-02-20 12:52:32 Memory monitor running!
2023-02-20 12:56:44 Memory monitor running!
Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 3.4G 25G 13% /
/dev/sda1 916G 588G 283G 68% /home/umbrel/umbrel
Feb 20 12:56:33 umbrel umbrel startup[5998]: File “/usr/local/lib/python3.7/dist-packages/compose/cli/command.py”, line 153, in get_project
Feb 20 12:56:33 umbrel umbrel startup[5998]: verbose=verbose, version=api_version, context=context, environment=environment
Feb 20 12:56:33 umbrel umbrel startup[5998]: File “/usr/local/lib/python3.7/dist-packages/compose/cli/docker_client.py”, line 43, in get_client
Feb 20 12:56:33 umbrel umbrel startup[5998]: environment=environment, tls_version=get_tls_version(environment)
Feb 20 12:56:33 umbrel umbrel startup[5998]: File “/usr/local/lib/python3.7/dist-packages/compose/cli/docker_client.py”, line 170, in docker_client
Feb 20 12:56:33 umbrel umbrel startup[5998]: client = APIClient(use_ssh_client=not use_paramiko_ssh, **kwargs)
Feb 20 12:56:33 umbrel umbrel startup[5998]: File “/usr/local/lib/python3.7/dist-packages/docker/api/client.py”, line 197, in init
Feb 20 12:56:33 umbrel umbrel startup[5998]: self._version = self._retrieve_server_version()
Feb 20 12:56:33 umbrel umbrel startup[5998]: File “/usr/local/lib/python3.7/dist-packages/docker/api/client.py”, line 222, in _retrieve_server_version
Feb 20 12:56:33 umbrel umbrel startup[5998]: f’Error while fetching server API version: {e}’
Feb 20 12:56:33 umbrel umbrel startup[5998]: docker.errors.DockerException: Error while fetching server API version: UnixHTTPConnectionPool(host=‘localhost’, port=None): Read timed out. (read timeout=240)
Feb 20 12:56:33 umbrel umbrel startup[5998]: Failed to start containers
Feb 20 12:56:33 umbrel systemd[1]: umbrel-startup.service: Control process exited, code=exited, status=1/FAILURE
Feb 20 12:56:33 umbrel systemd[1]: umbrel-startup.service: Failed with result ‘exit-code’.
Feb 20 12:56:33 umbrel systemd[1]: Failed to start Umbrel Startup Service.
Feb 20 12:56:44 umbrel systemd[1]: umbrel-startup.service: Service RestartSec=10s expired, scheduling restart.
Feb 20 12:56:44 umbrel systemd[1]: umbrel-startup.service: Scheduled restart job, restart counter is at 4.
Feb 20 12:56:44 umbrel systemd[1]: Stopped Umbrel Startup Service.
Feb 20 12:56:44 umbrel systemd[1]: Starting Umbrel Startup Service…
Feb 20 12:56:44 umbrel umbrel startup[8779]: ======================================
Feb 20 12:56:44 umbrel umbrel startup[8779]: ============= STARTING ===============
Feb 20 12:56:44 umbrel umbrel startup[8779]: ============== UMBREL ================
Feb 20 12:56:44 umbrel umbrel startup[8779]: ======================================
Feb 20 12:56:44 umbrel umbrel startup[8779]: Setting environment variables…
Feb 20 12:56:44 umbrel umbrel startup[8779]: Starting karen…
Feb 20 12:56:44 umbrel umbrel startup[8779]: Starting status monitors…
Feb 20 12:56:44 umbrel umbrel startup[8779]: Starting memory monitor…
Feb 20 12:56:44 umbrel umbrel startup[8779]: Starting backup monitor…
Feb 20 12:56:44 umbrel umbrel startup[8779]: Starting decoy backup trigger…
Feb 20 12:56:44 umbrel umbrel startup[8779]: Starting Docker services…
Feb 20 10:24:39 umbrel external storage mounter[491]: Running external storage mount script…
Feb 20 10:24:39 umbrel external storage mounter[491]: Found device “WDC WDS 100T2B0A-00SM50”
Feb 20 10:24:39 umbrel external storage mounter[491]: Blacklisting USB device IDs against UAS driver…
Feb 20 10:24:39 umbrel external storage mounter[491]: Rebinding USB drivers…
Feb 20 10:24:39 umbrel external storage mounter[491]: Checking USB devices are back…
Feb 20 10:24:39 umbrel external storage mounter[491]: Waiting for USB devices…
Feb 20 10:24:40 umbrel external storage mounter[491]: Waiting for USB devices…
Feb 20 10:24:41 umbrel external storage mounter[491]: Waiting for USB devices…
Feb 20 10:24:42 umbrel external storage mounter[491]: Checking if the device is ext4…
Feb 20 10:24:43 umbrel external storage mounter[491]: Yes, it is ext4
Feb 20 10:24:43 umbrel external storage mounter[491]: Checking filesystem for corruption…
Feb 20 10:24:43 umbrel external storage mounter[491]: e2fsck 1.44.5 (15-Dec-2018)
Feb 20 10:24:43 umbrel external storage mounter[491]: umbrel: clean, 573775/61054976 files, 158051971/244190208 blocks
Feb 20 10:24:43 umbrel external storage mounter[491]: Mounting partition…
Feb 20 10:24:43 umbrel external storage mounter[491]: Checking if device contains an Umbrel install…
Feb 20 10:24:43 umbrel external storage mounter[491]: Yes, it contains an Umbrel install
Feb 20 10:24:43 umbrel external storage mounter[491]: Bind mounting external storage over local Umbrel installation…
Feb 20 10:24:43 umbrel external storage mounter[491]: Bind mounting external storage over local Docker data dir…
Feb 20 10:24:43 umbrel external storage mounter[491]: Bind mounting external storage to /swap
Feb 20 10:24:43 umbrel external storage mounter[491]: Bind mounting SD card root at /sd-card…
Feb 20 10:24:43 umbrel external storage mounter[491]: Checking Umbrel root is now on external storage…
Feb 20 10:24:44 umbrel external storage mounter[491]: Checking /var/lib/docker is now on external storage…
Feb 20 10:24:44 umbrel external storage mounter[491]: Checking /swap is now on external storage…
Feb 20 10:24:44 umbrel external storage mounter[491]: Setting up swapfile
Feb 20 10:24:45 umbrel external storage mounter[491]: Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
Feb 20 10:24:45 umbrel external storage mounter[491]: no label, UUID=734eb542-6ef8-45e9-91e4-6858bd233e6e
Feb 20 10:24:45 umbrel external storage mounter[491]: Checking SD Card root is bind mounted at /sd-root…
Feb 20 10:24:45 umbrel external storage mounter[491]: Starting external drive mount monitor…
Feb 20 10:24:45 umbrel external storage mounter[491]: Mount script completed successfully!
Feb 20 10:24:45 umbrel systemd[1]: Started External Storage Mounter.
– Logs begin at Mon 2023-02-20 10:24:34 UTC, end at Mon 2023-02-20 12:58:19 UTC. –
Feb 20 12:32:37 umbrel systemd[1]: Starting External Storage SDcard Updater…
Feb 20 12:32:37 umbrel external storage updater[25805]: Checking if SD card Umbrel is newer than external storage…
Feb 20 12:32:37 umbrel external storage updater[25805]: No, SD version is not newer, exiting.
Feb 20 12:32:37 umbrel systemd[1]: Started External Storage SDcard Updater.
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
Okay, let’s have a look:
From the Filesystem information we can see the following:
Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 3.4G 25G 13% / → That’s your SD Card
/dev/sda1 916G 588G 283G 68% /home/umbrel/umbrel → That’s your SSD
This means that SD Card and SSD are working.
I found the following line:
Feb 20 12:56:33 umbrel umbrel startup[5998]: docker.errors.DockerException: Error while fetching server API version: UnixHTTPConnectionPool(host=‘localhost’, port=None): Read timed out. (read timeout=240)
This line tells us that umbrel was not able to connect to the docker demon which runs all the umbrel services.
I would therefore suspect that the Docker Service is not starting properly. This would also explain why we don’t see any logs from the Docker containers.
You can check if the Docker deamon is running with the following command:
systemctl status docker
Can you please send the output of the command so that we can be sure that docker is running correctly.
Please also run
docker ps
which should give us a list of all containers running.
Here’s what I get with systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabl
Active: active (running) since Mon 2023-02-20 15:58:03 UTC; 14min ago
Docs: https://docs.docker.com
Main PID: 734 (dockerd)
Tasks: 13
CGroup: /system.slice/docker.service
├─ 734 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.
├─2058 runc --version
└─2059 runc --versionFeb 20 14:45:14 umbrel dockerd[734]: time=“2023-02-20T14:45:14.805280559Z” level=war
Feb 20 14:45:14 umbrel dockerd[734]: time=“2023-02-20T14:45:14.805303800Z” level=war
Feb 20 14:45:14 umbrel dockerd[734]: time=“2023-02-20T14:45:14.805945744Z” level=inf
Feb 20 14:45:15 umbrel dockerd[734]: time=“2023-02-20T14:45:15.328812133Z” level=inf
Feb 20 14:45:15 umbrel dockerd[734]: time=“2023-02-20T14:45:15.489319410Z” level=inf
Feb 20 15:58:03 umbrel dockerd[734]: time=“2023-02-20T15:58:03.601952212Z” level=war
Feb 20 15:58:03 umbrel dockerd[734]: time=“2023-02-20T15:58:03.694522148Z” level=inf
Feb 20 15:58:03 umbrel dockerd[734]: time=“2023-02-20T15:58:03.709798324Z” level=inf
Feb 20 15:58:03 umbrel systemd[1]: Started Docker Application Container Engine.
Feb 20 15:58:03 umbrel dockerd[734]: time=“2023-02-20T15:58:03.783617406Z” level=inf
lines 1-21/21 (END)
docker ps
is stuck with no response for 1hr now
update: docker ps finally returned Bus error
Okay, we can see that Docker is running:
Active: active (running)
But we can also see that he doesn’t started the umbrel containers. otherwise the output would be longer.
Im not really sure how to fix this problem without a completely fresh instal of umbrel.
You can try to check if docker is running in the newest version with the following command.
sudo apt-get upgrade docker-engine
If the version is the newest you can try to start the bitcoin container manually with:
umbrel/scripts/app restart bitcoin
It’s definitely a strange problem. 
here’s the output of sudo apt-get upgrade docker-engine
Reading package lists… Done
Building dependency tree
Reading state information… Done
Package docker-engine is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
docker-ceE: Package ‘docker-engine’ has no installation candidate
what’s best next step?
Shit, wrong package name! my fault 
The correct package name is docker-ce not docker-engine
sudo apt-get upgrade docker-ce
after the update you should do a restart of your pi.
have you tried the second command from my last post?
here’s the output of sudo apt-get upgrade docker-ce
E: Could not get lock /var/lib/dpkg/lock-frontend - open (11: Resource temporarily unavailable)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?
and here’s what happens when I run umbrel/scripts/app restart bitcoin
Stopping app bitcoin…
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/urllib3/connectionpool.py”, line 384, in _make_request
six.raise_from(e, None)
File “”, line 3, in raise_from
File “/usr/lib/python3/dist-packages/urllib3/connectionpool.py”, line 380, in _make_request
httplib_response = conn.getresponse()
File “/usr/lib/python3.7/http/client.py”, line 1352, in getresponse
response.begin()
File “/usr/lib/python3.7/http/client.py”, line 310, in begin
version, status, reason = self._read_status()
File “/usr/lib/python3.7/http/client.py”, line 271, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), “iso-8859-1”)
File “/usr/lib/python3.7/socket.py”, line 589, in readinto
return self._sock.recv_into(b)
socket.timeout: timed outDuring handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/requests/adapters.py”, line 449, in send
timeout=timeout
File “/usr/lib/python3/dist-packages/urllib3/connectionpool.py”, line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File “/usr/lib/python3/dist-packages/urllib3/util/retry.py”, line 367, in increment
raise six.reraise(type(error), error, _stacktrace)
File “/usr/lib/python3/dist-packages/six.py”, line 693, in reraise
raise value
File “/usr/lib/python3/dist-packages/urllib3/connectionpool.py”, line 600, in urlopen
chunked=chunked)
File “/usr/lib/python3/dist-packages/urllib3/connectionpool.py”, line 386, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File “/usr/lib/python3/dist-packages/urllib3/connectionpool.py”, line 306, in _raise_timeout
raise ReadTimeoutError(self, url, “Read timed out. (read timeout=%s)” % timeout_value)
urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host=‘localhost’, port=None): Read timed out. (read timeout=60)During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/usr/local/lib/python3.7/dist-packages/docker/api/client.py”, line 214, in _retrieve_server_version
return self.version(api_version=False)[“ApiVersion”]
File “/usr/local/lib/python3.7/dist-packages/docker/api/daemon.py”, line 181, in version
return self._result(self._get(url), json=True)
File “/usr/local/lib/python3.7/dist-packages/docker/utils/decorators.py”, line 46, in inner
return f(self, *args, **kwargs)
File “/usr/local/lib/python3.7/dist-packages/docker/api/client.py”, line 237, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File “/usr/lib/python3/dist-packages/requests/sessions.py”, line 546, in get
return self.request(‘GET’, url, **kwargs)
File “/usr/lib/python3/dist-packages/requests/sessions.py”, line 533, in request
resp = self.send(prep, **send_kwargs)
File “/usr/lib/python3/dist-packages/requests/sessions.py”, line 646, in send
r = adapter.send(request, **kwargs)
File “/usr/lib/python3/dist-packages/requests/adapters.py”, line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host=‘localhost’, port=None): Read timed out. (read timeout=60)During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/usr/local/bin/docker-compose”, line 10, in
sys.exit(main())
File “/usr/local/lib/python3.7/dist-packages/compose/cli/main.py”, line 81, in main
command_func()
File “/usr/local/lib/python3.7/dist-packages/compose/cli/main.py”, line 200, in perform_command
project = project_from_options(‘.’, options)
File “/usr/local/lib/python3.7/dist-packages/compose/cli/command.py”, line 70, in project_from_options
enabled_profiles=get_profiles_from_options(options, environment)
File “/usr/local/lib/python3.7/dist-packages/compose/cli/command.py”, line 153, in get_project
verbose=verbose, version=api_version, context=context, environment=environment
File “/usr/local/lib/python3.7/dist-packages/compose/cli/docker_client.py”, line 43, in get_client
environment=environment, tls_version=get_tls_version(environment)
File “/usr/local/lib/python3.7/dist-packages/compose/cli/docker_client.py”, line 170, in docker_client
client = APIClient(use_ssh_client=not use_paramiko_ssh, **kwargs)
File “/usr/local/lib/python3.7/dist-packages/docker/api/client.py”, line 197, in init
self._version = self._retrieve_server_version()
File “/usr/local/lib/python3.7/dist-packages/docker/api/client.py”, line 222, in _retrieve_server_version
f’Error while fetching server API version: {e}’
docker.errors.DockerException: Error while fetching server API version: UnixHTTPConnectionPool(host=‘localhost’, port=None): Read timed out. (read timeout=60)
okay that’s weird!
The first command should install the newest version of docker, but it was not able to do so, because there was already a lock.
looks like a installation process was not able to finish it’s job which causes this deadlock.
The second command shows that docker is not able to start the umbrel container.
We can try to break this lock and update everything but there is no guarantee that this will fix the problem.
I would recommend you to perform a fresh installation and load a backup.
Thanks. Could you point me to the instructions for a fresh install and loading a backup? Also, what would that mean for my existing lightning channels?
Haven’t done this before.
But you should have a look at this:
Got it. Before I go that far, I’d like to try one more thing you mentioned
We can try to break this lock and update everything but there is no guarantee that this will fix the problem.
How can I do this?
Okay, then let’s try this.
Run the three commands to remove all linux packaging locks.
sudo rm /var/lib/apt/lists/lock
sudo rm /var/cache/apt/archives/lock
sudo rm /var/lib/dpkg/lock
After that you should be able to update all linux packages.
At first look for updates with
sudo apt-get update
Then you can update all packages with
sudo apt-get upgrade
this will update all installed linux dependencies.
after that you can perform a reboot with
sudo reboot
after that reboot you can check if the logs again.
sudo ~/umbrel/scripts/debug