For some months I have been monitoring my Umbrel node using Lightning Watch bot and figured out that a few times per weeks my node get offline for like more or less 5min and then come back online again. If I manage to connect to Umbrel GUI right during these down time it seams partially or totally unresponsive. Until then I mostly don’t have issue for about a year or so, or didn’t notice that it was going randomly offline as I was not monitoring it actively like now.
First I ignored it thinking it’s just some background tasks that overload the CPU or memory for a couple of minutes but now it’s getting more and more often and sometimes for longer period like 10 or 15min. Today it happens almost every hour, it seams it’s accelerating, I’m afraid the node will be completely useless at some point.
I see in the log a lot of warning memory usage at (>90%), thus it’s not necessarily bad and not matching the downtime notification.
I see also this: [111] ERR: Electrum error: {}
I’m really getting crazy with this and any help would be welcome.
Hardware Raspberry Pi 4 8GB with 1TB external SSD and long durability SD card.
Umbrel version 0.5.4
Debug log right during a down time (some part removed/replaced for privacy/irrelevance and app log removed because too long to share here)
=====================
= Umbrel debug info =
=====================
Umbrel version
--------------
0.5.4
Flashed OS version
-----------------
v0.5.2
Raspberry Pi Model
------------------
Revision : d03114
Serial : xxxxxxxxx
Model : Raspberry Pi 4 Model B Rev 1.4
Firmware
--------
Dec 1 2021 15:01:54
Copyright (c) 2012 Broadcom
version 71bd3109023a0c8575585ba87cbb374d2eeb038f (clean) (release) (start)
Temperature
-----------
temp=62.3'C
Throttling
----------
throttled=0x0
Memory usage
------------
total used free shared buff/cache available
Mem: 7.8G 5.3G 147M 113M 2.4G 2.3G
Swap: 4.1G 4.1G 1.0M
total: 67.9%
mempool: 14%
samourai-server: 13.2%
nostr-relay: 7.3%
bitcoin: 7.1%
system: 4.7%
btcpay-server: 3.3%
lightning: 3.1%
bluewallet: 3%
electrs: 2.8%
lnbits: 2.7%
uptime-kuma: 1.7%
btc-rpc-explorer: 0.8%
thunderhub: 0.7%
lightning-terminal: 0.7%
specter-desktop: 0.6%
ride-the-lightning: 0.6%
believethehype-wallet-connect: 0.6%
robosats: 0.5%
lightning-shell: 0.5%
Memory monitor logs
-------------------
2024-01-02 19:40:59 Warning memory usage at 93%
2024-01-02 19:43:07 Warning memory usage at 93%
2024-01-02 19:52:05 Warning memory usage at 93%
2024-01-02 19:53:36 Warning memory usage at 93%
2024-01-02 19:56:45 Warning memory usage at 93%
2024-01-02 20:06:03 Warning memory usage at 91%
2024-01-02 20:09:03 Warning memory usage at 92%
2024-01-02 20:10:04 Warning memory usage at 93%
2024-01-02 20:11:20 Warning memory usage at 93%
2024-01-02 20:12:31 Warning memory usage at 93%
Filesystem information
----------------------
Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 3.5G 25G 13% /
/dev/sda1 916G 751G 119G 87% /home/umbrel/umbrel
Startup service logs
--------------------
Oct 16 13:19:16 umbrel umbrel startup[1528]: Creating samourai-server_nginx_1 ...
Oct 16 13:19:16 umbrel umbrel startup[1528]: Creating samourai-server_whirlpool_1 ...
Oct 16 13:19:16 umbrel umbrel startup[1528]: Creating bitcoin_bitcoind_1 ... done
Oct 16 13:19:16 umbrel umbrel startup[1528]: Creating bitcoin_server_1 ...
Oct 16 13:19:19 umbrel umbrel startup[1528]: Creating btcpay-server_web_1 ... done
Oct 16 13:19:20 umbrel umbrel startup[1528]: Creating electrs_app_1 ... done
Oct 16 13:19:23 umbrel umbrel startup[1528]: Creating bitcoin_server_1 ... done
Oct 16 13:19:24 umbrel umbrel startup[1528]: Creating samourai-server_nginx_1 ... done
Oct 16 13:19:24 umbrel umbrel startup[1528]: Creating samourai-server_whirlpool_1 ... done
Oct 16 13:19:25 umbrel umbrel startup[1528]: Umbrel is now accessible at
Oct 16 13:19:25 umbrel umbrel startup[1528]: http://umbrel.local
Oct 16 13:19:25 umbrel umbrel startup[1528]: http://192.168.1.xxx
Oct 16 13:19:25 umbrel systemd[1]: Started Umbrel Startup Service.
Oct 23 19:41:30 umbrel passwd[22664]: pam_unix(passwd:chauthtok): password changed for umbrel
Oct 27 06:29:04 umbrel passwd[20340]: pam_unix(passwd:chauthtok): password changed for umbrel
Nov 09 19:44:00 umbrel passwd[22745]: pam_unix(passwd:chauthtok): password changed for umbrel
Nov 10 21:57:07 umbrel passwd[14850]: pam_unix(passwd:chauthtok): password changed for umbrel
Nov 20 08:23:10 umbrel passwd[13212]: pam_unix(passwd:chauthtok): password changed for umbrel
Nov 29 14:16:52 umbrel passwd[12827]: pam_unix(passwd:chauthtok): password changed for umbrel
Nov 29 14:40:56 umbrel passwd[16133]: pam_unix(passwd:chauthtok): password changed for umbrel
Dec 09 20:56:05 umbrel passwd[30213]: pam_unix(passwd:chauthtok): password changed for umbrel
Dec 09 22:42:24 umbrel passwd[21454]: pam_unix(passwd:chauthtok): password changed for umbrel
Dec 15 09:13:31 umbrel passwd[24849]: pam_unix(passwd:chauthtok): password changed for umbrel
Dec 15 09:31:08 umbrel passwd[9916]: pam_unix(passwd:chauthtok): password changed for umbrel
Dec 15 13:51:10 umbrel passwd[25762]: pam_unix(passwd:chauthtok): password changed for umbrel
Dec 15 14:50:49 umbrel passwd[1464]: pam_unix(passwd:chauthtok): password changed for umbrel
Dec 22 21:29:38 umbrel passwd[26313]: pam_unix(passwd:chauthtok): password changed for umbrel
Dec 26 20:34:43 umbrel passwd[6158]: pam_unix(passwd:chauthtok): password changed for umbrel
Jan 02 20:29:12 umbrel passwd[10528]: pam_unix(passwd:chauthtok): password changed for umbrel
External storage service logs
-----------------------------
Oct 16 13:17:06 umbrel external storage mounter[506]: Rebinding USB drivers...
Oct 16 13:17:07 umbrel external storage mounter[506]: Checking USB devices are back...
Oct 16 13:17:07 umbrel external storage mounter[506]: Waiting for USB devices...
Oct 16 13:17:08 umbrel external storage mounter[506]: Waiting for USB devices...
Oct 16 13:17:09 umbrel external storage mounter[506]: Waiting for USB devices...
Oct 16 13:17:10 umbrel external storage mounter[506]: Checking if the device is ext4...
Oct 16 13:17:10 umbrel external storage mounter[506]: Yes, it is ext4
Oct 16 13:17:10 umbrel external storage mounter[506]: Checking filesystem for corruption...
Oct 16 13:17:10 umbrel external storage mounter[506]: e2fsck 1.44.5 (15-Dec-2018)
Oct 16 13:17:10 umbrel external storage mounter[506]: umbrel: recovering journal
Oct 16 13:17:13 umbrel external storage mounter[506]: Setting free inodes count to 60145255 (was 60257957)
Oct 16 13:17:13 umbrel external storage mounter[506]: Setting free blocks count to 63315164 (was 63937517)
Oct 16 13:17:13 umbrel external storage mounter[506]: umbrel: clean, 909721/61054976 files, 180875044/244190208 blocks
Oct 16 13:17:13 umbrel external storage mounter[506]: Mounting partition...
Oct 16 13:17:13 umbrel external storage mounter[506]: Checking if device contains an Umbrel install...
Oct 16 13:17:13 umbrel external storage mounter[506]: Yes, it contains an Umbrel install
Oct 16 13:17:13 umbrel external storage mounter[506]: Bind mounting external storage over local Umbrel installation...
Oct 16 13:17:13 umbrel external storage mounter[506]: Bind mounting external storage over local Docker data dir...
Oct 16 13:17:13 umbrel external storage mounter[506]: Bind mounting external storage to /swap
Oct 16 13:17:13 umbrel external storage mounter[506]: Bind mounting SD card root at /sd-card...
Oct 16 13:17:13 umbrel external storage mounter[506]: Checking Umbrel root is now on external storage...
Oct 16 13:17:14 umbrel external storage mounter[506]: Checking /var/lib/docker is now on external storage...
Oct 16 13:17:14 umbrel external storage mounter[506]: Checking /swap is now on external storage...
Oct 16 13:17:14 umbrel external storage mounter[506]: Setting up swapfile
Oct 16 13:17:15 umbrel external storage mounter[506]: Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
Oct 16 13:17:15 umbrel external storage mounter[506]: no label, UUID=418785e1-217b-4dd8-a2cd-0c114fa0c6a9
Oct 16 13:17:15 umbrel external storage mounter[506]: Checking SD Card root is bind mounted at /sd-root...
Oct 16 13:17:15 umbrel external storage mounter[506]: Starting external drive mount monitor...
Oct 16 13:17:15 umbrel external storage mounter[506]: Mount script completed successfully!
Oct 16 13:17:15 umbrel systemd[1]: Started External Storage Mounter.
External storage SD card update service logs
--------------------------------------------
-- Logs begin at Mon 2023-10-16 13:17:01 UTC, end at Tue 2024-01-02 20:37:40 UTC. --
Oct 16 13:17:37 umbrel systemd[1]: Starting External Storage SDcard Updater...
Oct 16 13:17:37 umbrel external storage updater[1478]: Checking if SD card Umbrel is newer than external storage...
Oct 16 13:17:37 umbrel external storage updater[1478]: No, SD version is not newer, exiting.
Oct 16 13:17:37 umbrel systemd[1]: Started External Storage SDcard Updater.
Docker containers
-----------------
NAMES STATUS
lnbits_app_proxy_1 Up 5 minutes
lnbits_tor_server_1 Up 5 minutes
lnbits_web_1 Up 5 minutes
btcpay-server_web_1 Up 6 days
btcpay-server_nbxplorer_1 Up 6 days
btcpay-server_postgres_1 Up 6 days
btcpay-server_app_proxy_1 Up 6 days
btcpay-server_tor_server_1 Up 6 days
nostr-relay_relay-proxy_1 Up 7 days
nostr-relay_app_proxy_1 Up 3 hours
nostr-relay_tor_server_1 Up 7 days
nostr-relay_web_1 Up 7 days
nostr-relay_relay_1 Up 7 days
thunderhub_web_1 Up 10 days
thunderhub_app_proxy_1 Up 10 days
thunderhub_tor_server_1 Up 10 days
believethehype-wallet-connect_app_proxy_1 Up 10 days
believethehype-wallet-connect_server_1 Up 6 hours
believethehype-wallet-connect_tor_server_1 Up 10 days
lightning-terminal_tor_server_1 Up 2 weeks
lightning-terminal_web_1 Up 2 weeks
lightning-terminal_app_proxy_1 Up 2 weeks
lightning_lnd_1 Up 15 minutes
lightning_app_proxy_1 Up 2 weeks
lightning_app_1 Up 2 weeks
lightning_tor_server_1 Up 2 weeks
lightning_tor_1 Up 2 weeks
uptime-kuma_tor_server_1 Up 3 weeks
uptime-kuma_server_1 Up 3 weeks (healthy)
uptime-kuma_app_proxy_1 Up 3 weeks
robosats_web_1 Up 3 weeks (unhealthy)
robosats_tor_server_1 Up 3 weeks
robosats_app_proxy_1 Up 3 weeks
ride-the-lightning_web_1 Up 7 weeks
ride-the-lightning_app_proxy_1 Up 7 weeks
ride-the-lightning_tor_server_1 Up 7 weeks
ride-the-lightning_boltz_1 Up 7 weeks
electrs_app_1 Up 7 weeks
electrs_tor_1 Up 7 weeks
electrs_app_proxy_1 Up 7 weeks
electrs_electrs_1 Up 17 seconds
electrs_tor_server_1 Up 7 weeks
bitcoin_server_1 Up 7 weeks
bitcoin_tor_server_1 Up 7 weeks
bitcoin_app_proxy_1 Up 7 weeks
bitcoin_bitcoind_1 Up 20 minutes
bitcoin_i2pd_daemon_1 Up 3 days
bitcoin_tor_1 Up 7 weeks
mempool_api_1 Up 33 minutes
mempool_app_proxy_1 Up 4 weeks
mempool_web_1 Up 7 weeks
mempool_mariadb_1 Up 7 weeks
mempool_tor_server_1 Up 7 weeks
samourai-server_whirlpool_1 Up 2 months
samourai-server_nginx_1 Up 2 months
samourai-server_node_1 Up 2 months
bluewallet_lndhub_1 Up 2 minutes
samourai-server_db_1 Up 2 months
samourai-server_tor_1 Up 2 months
samourai-server_app_proxy_1 Up 2 months
samourai-server_tor_server_1 Up 2 months
bluewallet_tor_server_1 Up 2 months
specter-desktop_app_proxy_1 Up 2 months
specter-desktop_tor_server_1 Up 2 months
bluewallet_redis_1 Up 2 months
specter-desktop_web_1 Up 2 months
bluewallet_app_proxy_1 Up 2 months
lightning-shell_tor_server_1 Up 2 months
lightning-shell_web_1 Up 2 months
lightning-shell_app_proxy_1 Up 2 months
btc-rpc-explorer_web_1 Up 2 months
btc-rpc-explorer_app_proxy_1 Up 2 months
btc-rpc-explorer_tor_server_1 Up 2 months
nginx Up 2 months
auth Up 2 months
manager Up 2 months
tor_proxy Up 2 months
dashboard Up 2 months
tor_server Up 2 months
Umbrel logs
-----------
Attaching to manager
manager | ::ffff:10.21.21.2 - - [Tue, 02 Jan 2024 20:37:57 GMT] "GET /v1/system/temperature HTTP/1.0" 304 - "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/119.0"
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Tue, 02 Jan 2024 20:37:57 GMT] "GET /v1/apps?installed=1 HTTP/1.0" 304 - "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/119.0"
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Tue, 02 Jan 2024 20:37:57 GMT] "GET /v1/apps HTTP/1.0" 304 - "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/119.0"
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Tue, 02 Jan 2024 20:37:57 GMT] "GET /v1/system/storage HTTP/1.0" 304 - "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/119.0"
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Tue, 02 Jan 2024 20:37:57 GMT] "GET /v1/system/memory HTTP/1.0" 304 - "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/119.0"
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Tue, 02 Jan 2024 20:37:57 GMT] "GET /v1/system/get-update HTTP/1.0" 304 - "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/119.0"
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Tue, 02 Jan 2024 20:37:57 GMT] "GET /v1/system/info HTTP/1.0" 304 - "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/119.0"
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Tue, 02 Jan 2024 20:37:57 GMT] "GET /v1/system/debug-result HTTP/1.0" 304 - "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/119.0"
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Tue, 02 Jan 2024 20:37:58 GMT] "GET /v1/system/debug-result HTTP/1.0" 304 - "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/119.0"
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Tue, 02 Jan 2024 20:37:59 GMT] "GET /v1/system/debug-result HTTP/1.0" 304 - "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/119.0"
manager |
manager | umbrel-manager
Tor Proxy logs
--------
Attaching to tor_proxy
tor_proxy | Jan 02 20:20:22.000 [notice] Tried for 120 seconds to get a connection to [scrubbed]:9735. Giving up. (waiting for circuit)
tor_proxy | Jan 02 20:20:22.000 [notice] Tried for 120 seconds to get a connection to [scrubbed]:9735. Giving up. (waiting for circuit)
tor_proxy | Jan 02 20:20:31.000 [notice] Tried for 120 seconds to get a connection to [scrubbed]:9735. Giving up. (waiting for circuit)
tor_proxy | Jan 02 20:22:03.000 [notice] Tried for 120 seconds to get a connection to [scrubbed]:0. Giving up. (waiting for circuit)
tor_proxy | Jan 02 20:35:34.000 [notice] New control connection opened from 10.21.21.9.
================
==== Result ====
================
The debug script did not automatically detect any issues with your Umbrel.