Error: System service failed --- I/O Error

I usually log into my node once per week to check on things, this past week I got nothing. I was not able to reach it via its IP like normal. I logged into my router and it did not show it online either. I pulled the plug the following day and got the " Error: System service failed" message. After logging in via ssh I tried to get debug info and only got an Input Output error. Fast forward a few days and I tried again. Trying to figure out if I need to reflash and start over, or something else can be done to get recovered

The following is the output:

=====================
= Umbrel debug info =

Umbrel version

0.4.15

Flashed OS version

v0.3.14

Raspberry Pi Model

Revision : c03112
Serial : 100000007dd563ad
Model : Raspberry Pi 4 Model B Rev 1.2

Firmware

May 27 2021 14:02:15
Copyright © 2012 Broadcom
version 7d9a298cda813f747b51fe17e1e417e7bf5ca94d (clean) (release) (start)

Temperature

temp=31.1’C

Throttling

throttled=0x0

Memory usage

          total        used        free      shared  buff/cache   available

Mem: 3.8G 146M 3.2G 8.0M 475M 3.6G
Swap: 0B 0B 0B

total: 3.9%
system: 3.9%
tor: 0%
lnd: 0%
electrs: 0%
bitcoin: 0%

Memory monitor logs

tail: cannot open ‘/home/umbrel/umbrel/scripts/…/logs/memory-monitor.log’ for reading: No such file or directory

Filesystem information

Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 2.9G 25G 11% /
/dev/root 29G 2.9G 25G 11% /

Startup service logs

– Logs begin at Fri 2022-03-11 15:25:11 UTC, end at Sat 2022-03-12 21:40:04 UTC. –
Mar 11 15:25:32 umbrel systemd[1]: Dependency failed for Umbrel Startup Service.
Mar 11 15:25:32 umbrel systemd[1]: umbrel-startup.service: Job umbrel-startup.service/start failed with result ‘dependency’.

External storage service logs

– Logs begin at Fri 2022-03-11 15:25:11 UTC, end at Sat 2022-03-12 21:40:04 UTC. –
Mar 11 15:25:16 umbrel systemd[1]: Starting External Storage Mounter…
Mar 11 15:25:16 umbrel external storage mounter[503]: Running external storage mount script…
Mar 11 15:25:16 umbrel external storage mounter[503]: Found device “SAMSUNG MZ7LM960HCHP-0E0”
Mar 11 15:25:16 umbrel external storage mounter[503]: Blacklisting USB device IDs against UAS driver…
Mar 11 15:25:30 umbrel external storage mounter[503]: Rebinding USB drivers…
Mar 11 15:25:30 umbrel external storage mounter[503]: Checking USB devices are back…
Mar 11 15:25:30 umbrel external storage mounter[503]: Waiting for USB devices…
Mar 11 15:25:31 umbrel external storage mounter[503]: Waiting for USB devices…
Mar 11 15:25:32 umbrel external storage mounter[503]: Checking if the device is ext4…
Mar 11 15:25:32 umbrel external storage mounter[503]: No, it’s not ext4
Mar 11 15:25:32 umbrel external storage mounter[503]: Formatting device…
Mar 11 15:25:32 umbrel external storage mounter[503]: Error: Input/output error during read on /dev/sda
Mar 11 15:25:32 umbrel external storage mounter[503]: Error: Input/output error during write on /dev/sda
Mar 11 15:25:32 umbrel external storage mounter[503]: Warning: Error fsyncing/closing /dev/sda: Input/output error
Mar 11 15:25:32 umbrel systemd[1]: umbrel-external-storage.service: Main process exited, code=exited, status=1/FAILURE
Mar 11 15:25:32 umbrel systemd[1]: umbrel-external-storage.service: Failed with result ‘exit-code’.
Mar 11 15:25:32 umbrel systemd[1]: Failed to start External Storage Mounter.

External storage SD card update service logs

– Logs begin at Fri 2022-03-11 15:25:11 UTC, end at Sat 2022-03-12 21:40:04 UTC. –
Mar 11 15:25:32 umbrel systemd[1]: Dependency failed for External Storage SDcard Updater.
Mar 11 15:25:32 umbrel systemd[1]: umbrel-external-storage-sdcard-update.service: Job umbrel-external-storage-sdcard-update.service/start failed with result ‘dependency’.

Karen logs

tail: cannot open ‘logs/karen.log’ for reading: No such file or directory

Docker containers

NAMES STATUS

Umbrel logs

ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements

Bitcoin Core logs

ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements

LND logs

ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements

electrs logs

ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements

Tor logs

ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements

==== Result ====

The debug script did not automatically detect any issues with your Umbrel.

Your external drive is fucked and/ot have cable/case issues.

Whats the best way to repair/rebuild?

check first the cabling and or replace it

I reseated the cable I had and nothing. I replaced it with an old (non-powered) adapter and looks like it’s working!

Thank you!

1 Like