Error: System service failed - no restart

Flolucious · May 18, 2023, 7:30pm

Hi everyone. after having issues to install apps - i tried to restart the umbrel but received an error.
Here is my logs. Anyone know how to fix this?

= Umbrel debug info =

=====================

Umbrel version

0.5.3

Flashed OS version

v0.3.9-patch-2

Raspberry Pi Model

Model : Raspberry Pi 4 Model B Rev 1.2

Firmware

Feb 25 2021 12:10:40

version 564e5f9b852b23a330b1764bcf0b2d022a20afd0 (clean) (release) (start)

Temperature

temp=53.0’C

Throttling

throttled=0x0

Memory usage

total used free shared buff/cache available

Mem: 3.8G 157M 3.3G 8.0M 352M 3.6G

Swap: 4.1G 0B 4.1G

bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)

total: 4.1%

system: 4.1%

thunderhub: 0%

robosats: 0%

ln-visualizer: 0%

lightning: 0%

bitcoin: 0%

Memory monitor logs

2019-02-14 10:14:59 Memory monitor running!

2019-02-14 10:12:18 Memory monitor running!

2019-02-14 10:12:33 Memory monitor running!

2019-02-14 10:12:47 Memory monitor running!

2019-02-14 10:13:00 Memory monitor running!

2019-02-14 10:13:14 Memory monitor running!

2019-02-14 10:13:27 Memory monitor running!

2019-02-14 10:13:40 Memory monitor running!

2019-02-14 10:13:54 Memory monitor running!

2019-02-14 10:14:07 Memory monitor running!

Filesystem information

Filesystem Size Used Avail Use% Mounted on

/dev/root 29G 29G 0 100% /

/dev/sda1 916G 565G 305G 65% /home/umbrel/umbrel

Startup service logs

Feb 14 10:14:07 umbrel umbrel startup[5993]: Starting status monitors…

Feb 14 10:14:07 umbrel umbrel startup[5993]: Starting memory monitor…

Feb 14 10:14:07 umbrel umbrel startup[5993]: Starting backup monitor…

Feb 14 10:14:07 umbrel umbrel startup[5993]: Starting decoy backup trigger…

Feb 14 10:14:07 umbrel umbrel startup[5993]: Starting Docker services…

Feb 14 10:14:09 umbrel umbrel startup[5993]: Starting dashboard …

Feb 14 10:14:09 umbrel umbrel startup[5993]: Starting auth …

Feb 14 10:14:09 umbrel umbrel startup[5993]: Starting tor_server …

Feb 14 10:14:09 umbrel umbrel startup[5993]: Starting manager …

Feb 14 10:14:09 umbrel umbrel startup[5993]: Starting tor_proxy …

Feb 14 10:14:10 umbrel umbrel startup[5993]: Starting tor_server … error

Feb 14 10:14:10 umbrel umbrel startup[5993]: ERROR: for tor_server Cannot start service tor_server: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/287950d10b8c2b55a2e55b1ab09e1adddda0

f41389366474ea5059417075b994: no space left on device: unknown

Feb 14 10:14:10 umbrel umbrel startup[5993]: Starting auth … error

Feb 14 10:14:10 umbrel umbrel startup[5993]: ERROR: for auth Cannot start service auth: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/0cae9edd612117270441c4580bb1057ed14b714ff0aa9722

5ab9fabf4c49f50f: no space left on device: unknown

Feb 14 10:14:10 umbrel umbrel startup[5993]: Starting manager … error

Feb 14 10:14:10 umbrel umbrel startup[5993]: ERROR: for manager Cannot start service manager: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/fb2f52ad0e1cc7291b72014e224a796d5d5b76a16a

b24218112b3d91c5f3c7d5: no space left on device: unknown

Feb 14 10:14:10 umbrel umbrel startup[5993]: Starting tor_proxy … error

Feb 14 10:14:10 umbrel umbrel startup[5993]: ERROR: for tor_proxy Cannot start service tor_proxy: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/5ae4189a4584cb671e3edd36f41ab3aea79775

be207fc86d6871341497726ee5: no space left on device: unknown

Feb 14 10:14:10 umbrel umbrel startup[5993]: Starting dashboard … error

Feb 14 10:14:10 umbrel umbrel startup[5993]: ERROR: for dashboard Cannot start service dashboard: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/0e67cfa3442c3d818f1b231d0ee59e92e2e34d

c7aab26751152a888e2747037e: no space left on device: unknown

Feb 14 10:14:10 umbrel umbrel startup[5993]: ERROR: for tor_server Cannot start service tor_server: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/287950d10b8c2b55a2e55b1ab09e1adddda0

f41389366474ea5059417075b994: no space left on device: unknown

Feb 14 10:14:10 umbrel umbrel startup[5993]: ERROR: for auth Cannot start service auth: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/0cae9edd612117270441c4580bb1057ed14b714ff0aa9722

5ab9fabf4c49f50f: no space left on device: unknown

Feb 14 10:14:10 umbrel umbrel startup[5993]: ERROR: for manager Cannot start service manager: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/fb2f52ad0e1cc7291b72014e224a796d5d5b76a16a

b24218112b3d91c5f3c7d5: no space left on device: unknown

Feb 14 10:14:10 umbrel umbrel startup[5993]: ERROR: for tor_proxy Cannot start service tor_proxy: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/5ae4189a4584cb671e3edd36f41ab3aea79775

be207fc86d6871341497726ee5: no space left on device: unknown

Feb 14 10:14:10 umbrel umbrel startup[5993]: ERROR: for dashboard Cannot start service dashboard: failed to start shim: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/0e67cfa3442c3d818f1b231d0ee59e92e2e34d

c7aab26751152a888e2747037e: no space left on device: unknown

Feb 14 10:14:10 umbrel umbrel startup[5993]: Encountered errors while bringing up the project.

Feb 14 10:14:10 umbrel umbrel startup[5993]: Failed to start containers

Feb 14 10:14:10 umbrel systemd[1]: umbrel-startup.service: Control process exited, code=exited, status=1/FAILURE

Feb 14 10:14:10 umbrel systemd[1]: umbrel-startup.service: Failed with result ‘exit-code’.

Feb 14 10:14:10 umbrel systemd[1]: Failed to start Umbrel Startup Service.

External storage service logs

Feb 14 10:12:04 umbrel external storage mounter[508]: Rebinding USB drivers…

Feb 14 10:12:04 umbrel external storage mounter[508]: Checking USB devices are back…

Feb 14 10:12:04 umbrel external storage mounter[508]: Waiting for USB devices…

Feb 14 10:12:05 umbrel external storage mounter[508]: Waiting for USB devices…

Feb 14 10:12:06 umbrel external storage mounter[508]: Checking if the device is ext4…

Feb 14 10:12:06 umbrel external storage mounter[508]: Yes, it is ext4

Feb 14 10:12:06 umbrel external storage mounter[508]: Checking filesystem for corruption…

Feb 14 10:12:06 umbrel external storage mounter[508]: e2fsck 1.44.5 (15-Dec-2018)

Feb 14 10:12:07 umbrel external storage mounter[508]: Superblock last mount time is in the future.

Feb 14 10:12:07 umbrel external storage mounter[508]: (by less than a day, probably due to the hardware clock being incorrectly set)

Feb 14 10:12:07 umbrel external storage mounter[508]: Superblock last write time is in the future.

Feb 14 10:12:07 umbrel external storage mounter[508]: (by less than a day, probably due to the hardware clock being incorrectly set)

Feb 14 10:12:07 umbrel external storage mounter[508]: umbrel: clean, 328160/61054976 files, 152045179/244190208 blocks

Feb 14 10:12:07 umbrel external storage mounter[508]: Mounting partition…

Feb 14 10:12:07 umbrel external storage mounter[508]: Checking if device contains an Umbrel install…

Feb 14 10:12:07 umbrel external storage mounter[508]: Yes, it contains an Umbrel install

Feb 14 10:12:07 umbrel external storage mounter[508]: Bind mounting external storage over local Umbrel installation…

Feb 14 10:12:07 umbrel external storage mounter[508]: Bind mounting external storage over local Docker data dir…

Feb 14 10:12:07 umbrel external storage mounter[508]: Bind mounting external storage to /swap

Feb 14 10:12:07 umbrel external storage mounter[508]: Bind mounting SD card root at /sd-card…

Feb 14 10:12:07 umbrel external storage mounter[508]: Checking Umbrel root is now on external storage…

Feb 14 10:12:08 umbrel external storage mounter[508]: Checking /var/lib/docker is now on external storage…

Feb 14 10:12:08 umbrel external storage mounter[508]: Checking /swap is now on external storage…

Feb 14 10:12:08 umbrel external storage mounter[508]: Setting up swapfile

Feb 14 10:12:08 umbrel external storage mounter[508]: Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)

Feb 14 10:12:08 umbrel external storage mounter[508]: no label, UUID=d3e136f3-5091-4f9a-8303-4bfd3a5b3b89

Feb 14 10:12:08 umbrel external storage mounter[508]: Checking SD Card root is bind mounted at /sd-root…

Feb 14 10:12:08 umbrel external storage mounter[508]: Starting external drive mount monitor…

Feb 14 10:12:08 umbrel external storage mounter[508]: Mount script completed successfully!

Feb 14 10:12:08 umbrel systemd[1]: Started External Storage Mounter.

External storage SD card update service logs

Feb 14 10:12:18 umbrel external storage updater[1129]: == Begin Update Script 01-run.sh ==

Feb 14 10:12:18 umbrel external storage updater[1129]: =======================================

Feb 14 10:12:18 umbrel external storage updater[1129]: =============== UPDATE ================

Feb 14 10:12:18 umbrel external storage updater[1129]: =======================================

Feb 14 10:12:18 umbrel external storage updater[1129]: =========== Stage: Install ============

Feb 14 10:12:18 umbrel external storage updater[1129]: =======================================

Feb 14 10:12:18 umbrel external storage updater[1129]: =============================================

Feb 14 10:12:18 umbrel external storage updater[1129]: Installing on Umbrel OS v0.3.9-patch-2

Feb 14 10:12:18 umbrel external storage updater[1129]: =============================================

Feb 14 10:12:18 umbrel external storage updater[1129]: ./01-run.sh: line 25: cannot create temp file for here-document: No space left on device

Feb 14 10:12:18 umbrel external storage updater[1129]: == End Update Script 01-run.sh ==

Feb 14 10:12:18 umbrel external storage updater[1129]: == Begin Update Script 02-run.sh ==

Feb 14 10:12:18 umbrel external storage updater[1129]: =======================================

Feb 14 10:12:18 umbrel external storage updater[1129]: =============== UPDATE ================

Feb 14 10:12:18 umbrel external storage updater[1129]: =======================================

Feb 14 10:12:18 umbrel external storage updater[1129]: ========= Stage: Post-update ==========

Feb 14 10:12:18 umbrel external storage updater[1129]: =======================================

Feb 14 10:12:18 umbrel external storage updater[1129]: == End Update Script 02-run.sh ==

Feb 14 10:12:18 umbrel external storage updater[1129]: == Begin Update Script 03-run.sh ==

Feb 14 10:12:18 umbrel external storage updater[1129]: =======================================

Feb 14 10:12:18 umbrel external storage updater[1129]: =============== UPDATE ================

Feb 14 10:12:18 umbrel external storage updater[1129]: =======================================

Feb 14 10:12:18 umbrel external storage updater[1129]: =========== Stage: Success ============

Feb 14 10:12:18 umbrel external storage updater[1129]: =======================================

Feb 14 10:12:18 umbrel external storage updater[1129]: Removing backup

Feb 14 10:12:18 umbrel external storage updater[1129]: ./03-run.sh: line 19: cannot create temp file for here-document: No space left on device

Feb 14 10:12:18 umbrel external storage updater[1129]: == End Update Script 03-run.sh ==

Feb 14 10:12:18 umbrel external storage updater[1129]: Deleting cloned repository

Feb 14 10:12:18 umbrel external storage updater[1129]: Removing lock

Feb 14 10:12:18 umbrel systemd[1]: Started External Storage SDcard Updater.

Karen logs

karen is running in /home/umbrel/umbrel/events

Docker containers

NAMES STATUS

Umbrel logs

Attaching to manager

Tor Proxy logs

Attaching to tor_proxy

bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)

App logs

bitcoin

bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)

Attaching to

lightning

bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)

Attaching to

ln-visualizer

bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)

Attaching to

robosats

bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)

Attaching to

thunderhub

bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)

Attaching to

================

==== Result ====

================
The debug script did not automatically detect any issues with your Umbrel.

usernameisJim · May 19, 2023, 7:40am

As a disclaimer please always make sure you have secured your 24 word seed phrase and backed up your funds prior to deleting any data on your storage drive, SSD/HDD

This error is the result of a failing read/write issue with the installed drive, it can also be caused by a failing storage device or a storage device that has not been formatted correctly in ext4, has become corrupted, or had data on it prior to installation.

The most reliable fix for this issue if you are on the Rasperry Pi 4 platform is to reflash your MicroSD card.

For reference, reflashing will not result in any data loss because all user data is stored on your external storage drive (SSD or HDD). To reflash your MicroSD card, follow the steps on our website to “Install on a Rasberry Pi 4”

If it is your first time setting up Umbrel, and you don’t have any data on it yet please try reformatting your storage device again and make sure all partitions are completely clear- this is also a common cause if you already had a prior installation or any data on your drive and are trying to setup like new once more. You can delete all partitions and format in any default format file type, upon a successful Umbrel installation- during the Umbrel setup it will then automatically format again in the correct ext4.

It looks like you haven’t flashed since v0.3.9 and thanks for being a legacy user!

Steps to SSH can be found in our FAQ here under How to SSH into Umbrel on a Raspberry Pi

Flolucious · May 19, 2023, 11:03am

Thank you! The reflashing worked well. Only my Bitcoin & Lightning node is still in startup mode. Can this take more time than usual?

amadeus · May 19, 2023, 7:48pm

I had the exact same problem here: System service failed

Still haven’t solved the problem, so my node is down and I can’t watch the mempool locally

Still hoping somebody notices this issue, I suspect we’re not the only ones.

usernameisJim · May 20, 2023, 10:19am

Hey @amadeus as the steps in my last post worked for @Flolucious can you run through them? I know it may seem redundant but it may take multiple reflashing, if you can SSH - try SSHing and running command: sudo shutdown

Then reflash MicroSD card, then plug everything back in to your Raspberry Pi 4 and the power last - let me know if the error on the dashboard changes - and if possible it may be worth trying another MicroSD card if available.

@Flolucious can you share a debug log now? You can attach it to a post here after clicking this icon
and can generate it by going to your System page > Troubleshoot > Start

If for some reason you can’t generate it through the dashboard follow the steps on Generating Troubleshooting Logs here in the FAQ

You basically run this command: sudo ~/umbrel/scripts/debug

Then can paste the output to a text file and attach it or paste it somewhere like pastebin.com and share the link from there

This will likely let us know why they Bitcoin Node is not loading.

Let me know if you have any questions

Flolucious · May 20, 2023, 12:56pm

umbrel-1684587324483-debug.log (31.3 KB)

usernameisJim · May 22, 2023, 6:55pm

If you check your debug logs you have the following error:
“Error: Failed loading settings file:
bitcoind_1 | - Unable to parse settings file /data/.bitcoin/settings.json”

To resolve, first connect to your Umbrel device by following the instructions in How to SSH into Umbrel on a Raspberry Pi or by accessing a terminal on your Linux machine

This command is the quickest way to resolve the error, by deleting the file first:

rm ~umbrel/app-data/bitcoin/data/bitcoin/settings.json

Then you should restart Bitcoin:
~/umbrel/scripts/app restart bitcoin

If for any reason when attempting to delete the file you face a permission or other errors that prevent you from doing so, we can also edit the file with this command:

sudo nano umbrel/app-data/bitcoin/data/bitcoin/settings.json

Then it will take you to a blank file and just type and add two lines like this, with the two curly braces

{
}

then hit CTRL+X to exit, Y to save, and enter to continue.

After completing the edit don’t forget the last step of restarting Bitcoin (you can also do this from the UI by going to Advanced Settings, scrolling down and clicking "Save and Restart Node):
~/umbrel/scripts/app restart bitcoin

Upon restarting wait ~5 minutes and this will resolve the error with the settings.json file.

Flolucious · May 23, 2023, 5:35pm

Thanks worked smoothly!

amadeus · June 6, 2023, 6:07pm

Re-flashing a second time didn’t solve the problem, but I did run the debug log and discovered a similar problem with the settings.json. I followed the instructions you gave Flolucious and we’re back in business. Thanks!

Topic		Replies	Views
Booting problem: System service failed Support and Troubleshooting	16	2219	November 18, 2021
System service failed Support and Troubleshooting	3	535	May 15, 2023
Error, System service failed Support and Troubleshooting	18	849	March 16, 2022
Unable to Start - Error: System service failed Support and Troubleshooting	4	1061	December 7, 2022
Error: "System service failed" Support and Troubleshooting	6	611	August 15, 2023

Error: System service failed - no restart

Hi everyone. after having issues to install apps - i tried to restart the umbrel but received an error. Here is my logs. Anyone know how to fix this?

Related topics

Hi everyone. after having issues to install apps - i tried to restart the umbrel but received an error.
Here is my logs. Anyone know how to fix this?