Mempool stopping, making gaps constantly

Hi,

RPi 8G - all apps/OS updated.

because mempool is full of TX and frequently dropping them either because size limit or because TX is old I decided use an option and increase mempool up 2G in size. And dbcache - 150MB to save a bit more.

There should be enough space (only ThunderHub and TriliumNotes using some RAM - together with all except Electrum/LN is barely 500M, also not using it really).

But this still hapenning:
image

Why? What is using, causing the RAM and SWAP usage spikes?

Mempool never reached 2G full - I do get it’s allocated for that in advance but when Umbrel restarted and running again there’s still free space in RAM. Right now “1 GB available” after 17 days but still doing this behavior.

When running the debug looking for what I think is important:

Memory usage
total used free shared buff/cache available
Mem: 7.9G 7.0G 119M 19M 783M 725M
Swap: 4.1G 4.1G 0.0K

total: 88.5%
mempool: 29.1%
bitcoin: 24.3%
electrs: 10.4%
lightning: 9.1%
system: 7.9%
thunderhub: 2.5%
trilium-notes: 1.6%
jam: 1.1%
btc-rpc-explorer: 0.8%
tailscale: 0.5%
lightning-terminal: 0.5%
snort: 0.4%
nostr-relay: 0.3%

Memory monitor logs

2023-12-06 11:10:50 Warning memory usage at 91%
yep - here we go…
2023-12-06 15:57:00 Warning memory usage at 96%

== Result ==
The debug script did not automatically detect any issues with your Umbrel.

Even if remaining few services running uninstalled (except Tails) - that will free nothing.

I did set up also TX to remain in mempool for 6month so it will not flush them all at once - thinking it may cause some CPU overload and freeze node.

Can’t figure it out - it showing system is having some big RAM usage spike - that is just coincidence me having the picture, because try to use it at the time and it somehow allow me load settings page. Otherwise if I’m out of LAN, just won’t load until mempool starts loading TX again fresh from scratch and that’s what I’m trying to avoid.

It doing this before (I did gradually raise size from default 300M until it not hitting the max.) but even like this it crash again.

So frequently not usable…

Sometimes SSH in and upgrade OS is done so stopping Umbrel entirely and start fresh as restart is needed.

LN is 4 rather small chnls - no routing via them - dead.

It’s behind NAT, LAN<>WAN no port forwarding, no incoming conn.

It’s behind UPS, original PWR source. Fans and alu cooling.

Also did set up to save the mempool so it don’t need to load it all from scratch - persistmempool - not sure if it helps.

What can I change in settings or where look more deep?

Thank you all for some info.

And today again - now I see that:

2023-12-07 11:55:31 Warning memory usage at 93%

Memory usage: bitcoin: 1.4%

From Docker containers info in log:

bitcoin_bitcoind_1 stuck at Restarting (137)

also:
lightning_lnd_1 Up About an hour
electrs_electrs_1 Up 8 seconds
mempool_api_1 Up About a minute

I don’t know how to debug what causing this containers to restart/fail - rest of containers running for days or weeks. After 3 hrs from last memory warning the btc node is still restarting.

The debug script did not automatically detect any issues with your Umbrel.

image
And it’s still in this state for hours, even behind with time - it’s now next day 12 o’clock and no change.

From Troubleshoot - SWAP full, RAM available 1.8.

Yet not possible ever see Mempool app working fine.