Problem with Core-lightning not working after reboot

Support and Troubleshooting
1

I wannted to try c-lightning on my umbrel (RPi 4) but it seems like there is an issue with it, I can install it normally an run it, but when i restart my node it doesn’t work anymore, and for it to work i have to remove it completely and install it again.

Luckily i could save the hsm-secretc with funds on it and didn’t had any channels open. However I wasn’t able to restore it for some reason (would appreciate if someone helped me on how to do this.)

So, after many attempts and checking some logs the specific results on core-lightning app are the following:

core-lightning

Attaching to core-lightning_tor_server_1, core-lightning_app_proxy_1, core-lightning_c-lightning-rest_1, core-lightning_app_1, core-lightning_lightningd_1, core-lightning_tor_1
app_1               | 2023/10/11 01:36:03 socat[13977] E connect(5, AF=1 "/root/.lightning/bitcoin/lightning-rpc", 40): Connection refused
app_1               | 
app_1               | Waiting for lightningd
app_1               | 2023/10/11 01:36:04 socat[13983] E connect(5, AF=1 "/root/.lightning/bitcoin/lightning-rpc", 40): Connection refused
app_1               | 
app_1               | Waiting for lightningd
app_1               | 2023/10/11 01:36:05 socat[13989] E connect(5, AF=1 "/root/.lightning/bitcoin/lightning-rpc", 40): Connection refused
app_1               | 
app_1               | Waiting for lightningd
app_1               | 2023/10/11 01:36:05 socat[13994] E connect(5, AF=1 "/root/.lightning/bitcoin/lightning-rpc", 40): Connection refused
lightningd_1        | Core-Lightning starting
lightningd_1        | 2023-10-10T21:24:59.546Z UNUSUAL plugin-clnrest.py: Killing plugin: timed out before replying to getmanifest
lightningd_1        | 2023-10-10T21:25:04.560Z **BROKEN** plugin-clnrest.py: Plugin marked as important, shutting down lightningd!
lightningd_1        | Can't recover from plugin failure, terminating.
tor_1               | Oct 10 21:22:55.000 [notice] Bootstrapped 72% (loading_descriptors): Loading relay descriptors
tor_1               | Oct 10 21:23:02.000 [notice] Bootstrapped 75% (enough_dirinfo): Loaded enough directory info to build circuits
tor_1               | Oct 10 21:23:02.000 [notice] Bootstrapped 80% (ap_conn): Connecting to a relay to build circuits
tor_1               | Oct 10 21:23:02.000 [notice] Bootstrapped 85% (ap_conn_done): Connected to a relay to build circuits
tor_1               | Oct 10 21:23:02.000 [notice] Bootstrapped 89% (ap_handshake): Finishing handshake with a relay to build circuits
tor_1               | Oct 10 21:23:02.000 [notice] Bootstrapped 90% (ap_handshake_done): Handshake finished with a relay to build circuits
tor_1               | Oct 10 21:23:02.000 [notice] Bootstrapped 95% (circuit_create): Establishing a Tor circuit
tor_1               | Oct 10 21:23:03.000 [notice] Bootstrapped 100% (done): Done
tor_1               | Oct 10 21:58:03.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 126 buildtimes.
tor_1               | Oct 11 01:03:11.000 [warn] Received http status code 404 ("Not found") from server 192.158.224.175:443 while fetching "/tor/keys/fp/EFCBE720AB3A82B99F9E953CD5BF50F7EEFC7B97".
c-lightning-rest_1  |     at Socket.<anonymous> (/usr/src/app/lightning-client-js.js:80:23)
c-lightning-rest_1  |     at Socket.emit (node:events:513:28)
c-lightning-rest_1  |     at emitErrorNT (node:internal/streams/destroy:157:8)
c-lightning-rest_1  |     at emitErrorCloseNT (node:internal/streams/destroy:122:3)
c-lightning-rest_1  |     at processTicksAndRejections (node:internal/process/task_queues:83:21) {
c-lightning-rest_1  |   errno: -111,
c-lightning-rest_1  |   code: 'ECONNREFUSED',
c-lightning-rest_1  |   syscall: 'connect',
c-lightning-rest_1  |   address: '/root/.lightning/bitcoin/lightning-rpc'
c-lightning-rest_1  | }
app_proxy_1         | yarn run v1.22.19
app_proxy_1         | $ node ./bin/www
app_proxy_1         | [HPM] Proxy created: /  -> http://10.21.21.94:2103
app_proxy_1         | Waiting for 10.21.21.94:2103 to open...

Also i thought this issue might be due in some way to the bitcoin blockchain not being synced yet, but made some other attepmts with it completely sinchronized and still it didn’t work.

Thanks!

2

Hi @Dionisius

Unfortunately I haven’t come across this issue before.
So you are saying this issue only comes up after you restart your Umbrel?

I would maybe suggest raising this is an issue with the Core LN team too: GitHub - ElementsProject/cln-application: Core lightning application on Umbrel

2 Likes
3

Yep, for some reason something crashes with core lightning on the first restart after installation.

Per se i don’t care about using LND or c-lightning, the thing is haven’t been able to recover my funds in the hsm_secret, because to do so i need lightningd to start in the first place.

EDIT: also there is no guide or command i found to do this recovery using umbrel, as it seems it isn’t as simple as just writing lightning-cli

4

So far this issue hasn’t been solved, got into the c-lightning group on Telegram and the only answers i got were from scammers impersonating group admins using DMs.

5

Ah that’s unfortunate, I’ll raise this with our team too.
Btw, next time you encounter this issue could you please run the following command and share the output here

sudo ~/umbrel/scripts/app compose core-lightning logs --tail 250

I’ll get back to you shortly with a concise set of instructions and commands for managing this recovery from CLN’s docs

1 Like
8

Hi @smolgrrr thanks for your response, the “plugin-clnrest.py” issue seems to have been solved as I switched my hardware for umbrel from a Rapsberry to a Mini PC with a celeron N100 (I am using the same exact files for everything so it was probably a performance issue with the Pi.

Regarding the hsm-secret I’ve tried basically every method and the last command i used was the following one:

sudo umbrel/scripts/app compose core-lightning exec lightningd lightning-hsmtool getcodexsecret umbrel/hsm_secret adi0

I had to move the hsm_secret to umbrel directory as a last resource but in every attempt even using root, as the lightningd/bitcoin directory isn’t accesible as a normal user it just gave as a result that the hsm_secret file or directory doesn’t exist even tried being in the same directory and autocompleting with a tab.

Let me know if there is something i can do in this case.

9

In case it helps, I noticed in the CLN documentation, it says “The id is any 4 character string you can use to identify this secret (e.g. adi0 ): it cannot contain i , o , or b , but can contain digits except 1 .” So maybe try a different four characters.

Maybe a silly question, but did you try changing permissions of the bitcoin folder so you could go in and replace hsm_secret? You should also then check the owner and permission match the hsm_secret file being replaced.

1 Like
10

Appreciate the help here, couldn’t make it work with umbrel, but fortunately with some help I created a tool in order to convert the hsm_secret to an xprv so on-chain funds can be recovered and moved like any other wallet.

For anyone with the same issue you can find the repo HERE

I’m open to check for any issues or suggestions :slight_smile:

1 Like