I noticed the past couple of days that 2/3 of my 36 channels were offline.
I figured this was because of a big software update everyone was doing.
I notice one by one my channels are starting to all close!!
What the hell is going on?!?
I am losing my shit right now, please help.
(Please see screenshots… I scrambled the node alias’s for anonymity)
If those peers were CLN operators, and they did the last update CLN v 0.11.1 then yes, you could have a lot of force closing.
Also this would affect many other nodes, even if are not CLN, because if they got pending HTLCs with those nodes, the spread is going fast to all peers down the line.
I got also 3 force closures in the last 24h (inclusive one with ZFR node) and I think is because of this massive update people are doing.
Wrong, they should wait a bit more.
You can’t do much more, just have patience and wait for the force closures to be reach the maturity and funds back in your onchain wallet.
If you really want to see exactly how many pending HTLCs and at which block maturity funds will be released, read in the troubleshooting guide the section to check closed channels. You will find with that command all details about your channels.
PATIENCE is the key when you have these kind of situations. Wait and don’t do anything stupid.
An interesting analysis from someone on Reddit:
Ok, I looked closer, and found your node on amboss.space.
It looks like you only have 14 channels remaining and most of them reports your node as offline. This is not good.
I also had a look at some of your node’s recent closing transactions and I found that all of the ones I looked at were force closes.
Some of the force closes were initiated by your node, while others were initiated by your peers.
I also noted that most of the closing transactions have HTLCs in them.
Furthermore, all of the force closes looked like they were valid, meaning none were punished by any justice transactions. Therefore, I don’t think this is a case of state db corruption like I thought before.
From all the clues gathered, this is what I think happened:
You need to troubleshoot why your node’s lightning stack is failing to connect while its first layer continues to work.
——-
Come to think of it, I DID install Umbel’s latest update last week. I wonder if this is the initial cause of my problem as the redditor hypothesized in # 2.)
Nice follow up! You are right, these pending HTLCs are a pain and mostly not because your own node or your other peer fault.
Could be also a 3rd peer that got that HTLC and get offline, Then is forcing your node to trigger the force close. I don’t know why the LN code is doing this, is quite idiotic IMHO. This has to be changed, is unacceptable to have these moronic force closed channels.
My feeling is that the main reason for these issues are coming from bad gossip communication.
If the goosip is not announcing the real state of the channels (even if your node is well online), it is considered offline and trigger the force close unfairly.
A quick solution could be that more nodes will run hybrid mode (Tor and clearnet). Helps a lot but is not 100% sure it will fix totally the issue.
Another way is if umbrel devs will look closer into the issue with Tor services.
@mayank @lukechilds this is a serious issue and must be addressed with priority.
I was able to log into my node this morning without issue.
It looks as if everything is fine, but I know it is not.
I literally do not want to make another move until I am advised to do so in here.
I see all sorts of random amounts of Sats being returned to my wallet.
This kills me as I am very OCD.
I have files, and screen shots, and spread sheets trying to track my lightning journey / channel openings this whole time, and this completely just nukes any order I once had.
I feel like this will result in a major loss of Sats that I realistically will never earn back through all my future routing. This is very discouraging for a node operator - especially for a tadpole like myself. I know that a lot of people in my situation would pack up their Sats and go home. It makes me second guess wanting to continue this journey in fear that this can happen again at any moment.
I am in this for the greater good, not to get rich or earn huge amounts of sats.
I want a healthy node, I want to route for dirt cheap competitively, I want the lightning network to prosper. I see its incredibly huge potential.
But I’d be lying if I said I wasn’t totally afraid of opening 50+ channels again in the next year or so now because of this experience. My biggest concern is this issue happening again.
I was able to pull up terminal and run the debug log - (I do not see the link to ‘share to umbrel’ at the bottom as explained in the guide).
Again; I literally do not want to make another move until I am advised to do so in here.
I have no idea what it is I am looking for, I tried skimming through the log to see if I see anything unordinary or seemingly bad.
I did see this, which looks bad:
This is not your fault at all, and yes it stems off the major Tor issue that’s plaguing the Umbrel non-hybrid nodes right now, and I’d update Tor to the latest version, but I’m not sure if it worked because I’ve had 75 channels close out of the blue after the recent Umbrel update, so I’m not entirely sure if this is the best approach. All we can do now is wait for the official update from Umbrel to fix this issue.
@mayank and @lukechilds: Please try to expedite the Tor update into the next update of Umbrel, this issue is getting worse every day.
My channel with you was force closed early on the 21st. I do not recall seeing you down, and I was not down either. I have had a couple of other channels force close as well since updating my Umbrel. I will follow this thread for additional information and hopefully things get resolved. Let me know when you want to open another channel and we can make it happen. Thanks!
I’m one of the peers and I understand the frustration. Maybe the data I have helps diagnosing the problem. My node is c-otto.de (running lnd 0.14.3 at the time), which has IPv4+IPv6+Tor connectivity.
This is my “is connected” data to WJF3 (accurate to about 5 minutes, t meaning the connection was established, f meaning the connection was lost):
2022-04-21 22:48:28+02 | t
2022-05-02 20:28:27+02 | f
2022-05-02 20:33:27+02 | t
2022-05-05 14:58:27+02 | f
2022-05-05 15:03:27+02 | t
2022-05-10 01:04:35+02 | f
2022-05-11 15:29:08+02 | t
2022-05-12 11:00:48+02 | f
2022-05-12 11:05:48+02 | t
2022-05-13 17:00:48+02 | f
2022-05-13 18:30:48+02 | t
2022-05-13 18:35:48+02 | f
2022-05-13 20:45:48+02 | t
2022-05-13 20:55:48+02 | f
2022-05-13 23:45:48+02 | t
2022-05-14 06:05:48+02 | f
2022-05-14 06:10:48+02 | t
2022-05-19 05:09:58+02 | f
The node disappeared on May 19th. The channel was force-closed less than 24h later by WJF3:
2022-05-20 02:22:00.662 [DBG] CNCT: ChannelArbitrator(38a89fee484083faca5dabf7ad16d53413204dce25b92ae126a53f7d1643638e:1): new block (height=737107) examining active HTLC's
2022-05-20 02:22:00.662 [DBG] CNCT: ChannelArbitrator(38a89fee484083faca5dabf7ad16d53413204dce25b92ae126a53f7d1643638e:1): checking commit chain actions at height=737107, in_htlc_count=0, out_htlc_count=0
2022-05-20 02:22:00.668 [DBG] CNCT: ChannelArbitrator(38a89fee484083faca5dabf7ad16d53413204dce25b92ae126a53f7d1643638e:1): no actions for chain trigger, terminating
2022-05-20 02:22:00.673 [DBG] CNCT: ChannelArbitrator(38a89fee484083faca5dabf7ad16d53413204dce25b92ae126a53f7d1643638e:1): terminating at state=StateDefault
2022-05-20 02:35:59.934 [INF] NTFN: Dispatching confirmed spend notification for outpoint=38a89fee484083faca5dabf7ad16d53413204dce25b92ae126a53f7d1643638e:1, script=0 50b50e974fd933cd2adf2575fe42b4bb6f49b706f948a96ec89cf98bd50a2867 at current height=737108: d7dad089065ca77cd69360420f85481e5efdd5710a0dd8912c49e0dabf0494cf[0] spending 38a89fee484083faca5dabf7ad16d53413204dce25b92ae126a53f7d1643638e:1 at height=737108
As you can see, my node didn’t see any pending HTLC. The force close transaction (d7dad089065ca77cd69360420f85481e5efdd5710a0dd8912c49e0dabf0494cf) also doesn’t include any HTLC.
To me it looks like a connectivity issue. I don’t know why WJF3’s node force-closed the channel, though. Sometimes a HTLC may be pending but unknown to the peer (see https://bitcoin.stackexchange.com/questions/113335/unilateral-close-because-of-htlc-in-remote-commitment).
The last action the channel saw was a failed forward on May 19th, 05:03:05 (UTC+02).
Reconnection attempts looked liked this:
2022-05-19 05:43:05.224 [INF] SRVR: Established connection to: 0335f124fefbe54a0a62fc41582949ed808c3d8f35a75a8003e6586444c346b2ec@choe6gii3zwj76zlze2ouon7oopvgqu5aoe2xxwpqeubu6slg4gphmyd.onion:9735
2022-05-19 05:43:05.224 [INF] SRVR: Finalizing connection to 0335f124fefbe54a0a62fc41582949ed808c3d8f35a75a8003e6586444c346b2ec@choe6gii3zwj76zlze2ouon7oopvgqu5aoe2xxwpqeubu6slg4gphmyd.onion:9735, inbound=false
2022-05-19 05:43:06.372 [INF] PEER: disconnecting 0335f124fefbe54a0a62fc41582949ed808c3d8f35a75a8003e6586444c346b2ec@choe6gii3zwj76zlze2ouon7oopvgqu5aoe2xxwpqeubu6slg4gphmyd.onion:9735, reason: unable to start peer: unable to read init msg: EOF
For unknown reasons Orville force closed on WJF3 at txID 7011a1b9d2f1dd5e23a4509b6e69f9b894d86751884387416479f381bb1a7d02. I observed in LNDg that our channel had been inactive for over 2 days at the time of the closing, but I don’t think there were any pending HTLCs between our nodes, so I don’t know why this happened - sorry. I think the sats were distributed fairly between our nodes, and as the (involuntary) closer Orville paid the force close fee 
Looking at WJF3 in Amboss it looks like you still have a lot of inactive channels. I have noticed that many Umbrel nodes have problems with inactive channels and lack of stability right now.
Update:
Bad news: All of my channels have dropped like flies and are mostly offline / closing.
Good news: apparently ‘Telegram’ was nice enough and was able to still open a 1,000,000 sat channel to my node.
Man this really sucks… I was so proud of having a 35+ Channel node with only 2 channel closures.
This is very tarnishing to have to see all of these closed channels - Forever on my record, to be judged before a peer decides to connect with me or not.
The channel initiator pays the close fees, no matter who initiated the close.
Have you ever made a restart? I once had the same problem and after a restart the channels were always available for a while. I then uninstalled all the apps on the Pi. after that my node worked flawlessly.
currently our channel is offline again
Thank you C-Otto for the correction. I can add, that Orville also was the channel opener - so the 2,716 sat force close fee is on me 
Update:
As per @Rene1980 's recommendation;
I uninstalled my few apps and restarted my umbrel 3 hours ago.
Since restarting; Blockchain is no longer syncing. I was able to reinstall RTL, TH, and LNDg. Now Umbrel Interface says all channels are offline while RTL says they’re online, and balances are now shown as $NaN
I know this did not break any new ground - But it just further highlights the major disconnect between my Umbrel Node and it’s surroundings. This is devastating, I really hope a solution is found.
The funds from our channel most likely are locked, this can take up to 14 days. Have a look at https://mempool.space/address/bc1qqkyrgqaey3nzy35s8svtuncg3pjeak2mweuv4r. Once those sats move, you should be able to spend them (your lnd needs to be online and sweep the funds).
Hi @C-Otto
Thank you for providing that information to me!
Hi, you are now Online!!
Update:
I do not know whether to or not to assume that my node is cured or even safe to begin opening channels up again…
Should I wait for a patch / update to be released?
Can you please share your thoughts here when free @mayank and @lukechilds ?
Before you open new channels, you should make sure to understand what went wrong before. Unless you’re OK with it happening over and over again.
