System log flooded with IPv6 warnings

Support and Troubleshooting
1

Umbrel Home running UmbrelOS 1.3.0

The system log of my Umbrel Home is flooded with this kind of IPv6 warnings:

Jan 07 18:33:25 umbrel NetworkManager[565]: [1736274805.0447] platform-linux: do-add-ip6-address[2: fe80::1544:1d89:df73:c4e1]: failure 13 (Permission denied)
Jan 07 18:33:27 umbrel NetworkManager[565]: [1736274807.0473] platform-linux: do-add-ip6-address[2: fe80::7413:7b6b:4d26:9e18]: failure 13 (Permission denied)
Jan 07 18:33:29 umbrel NetworkManager[565]: [1736274809.0498] platform-linux: do-add-ip6-address[2: fe80::8380:72ba:f44e:c3ae]: failure 13 (Permission denied)
Jan 07 18:33:31 umbrel NetworkManager[565]: [1736274811.0500] ipv6ll[03abd298e4cc103b,ifindex=2]: changed: no IPv6 link local address to retry after Duplicate Address Detection failures (back off)
Jan 07 18:33:41 umbrel NetworkManager[565]: [1736274821.0516] platform-linux: do-add-ip6-address[2: fe80::86ef:647:b9d1:b3b9]: failure 13 (Permission denied)
Jan 07 18:33:43 umbrel NetworkManager[565]: [1736274823.0547] platform-linux: do-add-ip6-address[2: fe80::8598:1551:f113:2ffb]: failure 13 (Permission denied)
Jan 07 18:33:45 umbrel NetworkManager[565]: [1736274825.0573] platform-linux: do-add-ip6-address[2: fe80::c8f9:e872:3697:4cc3]: failure 13 (Permission denied)
Jan 07 18:33:47 umbrel NetworkManager[565]: [1736274827.0579] platform-linux: do-add-ip6-address[2: fe80::1544:1d89:df73:c4e1]: failure 13 (Permission denied)
Jan 07 18:33:49 umbrel NetworkManager[565]: [1736274829.0604] platform-linux: do-add-ip6-address[2: fe80::7413:7b6b:4d26:9e18]: failure 13 (Permission denied)
Jan 07 18:33:51 umbrel NetworkManager[565]: [1736274831.0625] platform-linux: do-add-ip6-address[2: fe80::8380:72ba:f44e:c3ae]: failure 13 (Permission denied)
Jan 07 18:33:53 umbrel NetworkManager[565]: [1736274833.0652] ipv6ll[03abd298e4cc103b,ifindex=2]: changed: no IPv6 link local address to retry after Duplicate Address Detection failures (back off)
Jan 07 18:34:03 umbrel NetworkManager[565]: [1736274843.0766] platform-linux: do-add-ip6-address[2: fe80::86ef:647:b9d1:b3b9]: failure 13 (Permission denied)
Jan 07 18:34:05 umbrel NetworkManager[565]: [1736274845.0786] platform-linux: do-add-ip6-address[2: fe80::8598:1551:f113:2ffb]: failure 13 (Permission denied)
Jan 07 18:34:07 umbrel NetworkManager[565]: [1736274847.0807] platform-linux: do-add-ip6-address[2: fe80::c8f9:e872:3697:4cc3]: failure 13 (Permission denied)
Jan 07 18:34:09 umbrel NetworkManager[565]: [1736274849.0819] platform-linux: do-add-ip6-address[2: fe80::1544:1d89:df73:c4e1]: failure 13 (Permission denied)
Jan 07 18:34:11 umbrel NetworkManager[565]: [1736274851.0831] platform-linux: do-add-ip6-address[2: fe80::7413:7b6b:4d26:9e18]: failure 13 (Permission denied)
Jan 07 18:34:13 umbrel NetworkManager[565]: [1736274853.0855] platform-linux: do-add-ip6-address[2: fe80::8380:72ba:f44e:c3ae]: failure 13 (Permission denied)
Jan 07 18:34:15 umbrel NetworkManager[565]: [1736274855.0878] ipv6ll[03abd298e4cc103b,ifindex=2]: changed: no IPv6 link local address to retry after Duplicate Address Detection failures (back off)
Jan 07 18:34:25 umbrel NetworkManager[565]: [1736274865.0910] platform-linux: do-add-ip6-address[2: fe80::86ef:647:b9d1:b3b9]: failure 13 (Permission denied)
Jan 07 18:34:27 umbrel NetworkManager[565]: [1736274867.0942] platform-linux: do-add-ip6-address[2: fe80::8598:1551:f113:2ffb]: failure 13 (Permission denied)
Jan 07 18:34:29 umbrel NetworkManager[565]: [1736274869.0974] platform-linux: do-add-ip6-address[2: fe80::c8f9:e872:3697:4cc3]: failure 13 (Permission denied)
Jan 07 18:34:31 umbrel NetworkManager[565]: [1736274871.0999] platform-linux: do-add-ip6-address[2: fe80::1544:1d89:df73:c4e1]: failure 13 (Permission denied)
Jan 07 18:34:33 umbrel NetworkManager[565]: [1736274873.1025] platform-linux: do-add-ip6-address[2: fe80::7413:7b6b:4d26:9e18]: failure 13 (Permission denied)

It is not only annoying but also causes too many write action on nvme disk.
What can I do about?
Thanks and regards
Thomas

2

Itā€™s hard to know whatā€™s happening to you just from this. The full umbrel log might provide more information on this.
Anyway, ā€œsomethingā€ is conflicting with IPv6. That ā€œsomethingā€ could be many things: some app or VPN service youā€™re using, the way you connect your device to your network, etc.
Itā€™s hard to say anything certain about this since I donā€™t know the context in which your umbrel works.
Anyway, and assuming only theoretically, if you have your umbrel within your local network and you donā€™t have any service/app that requires ipv6, I would disable ipv6 completely and forget about it.
Within your local network you only need ipv4.

3

Thank you for your answer, @jofraprez
I am totally fine with your suggestion but I donā€™t know how to disable IPv6 on Umbrel Home.
Could you please show me the way. Thanks.

Also I would like to hear from other Umbrel Home users whether they also have those errors in their system log. If yes it would mean it is a general problem with Umbrel Home. In this case Umbrel team should take care of it.

4

As far as I can see IPv6 is already disabled on Umbrel Home.
When I run this command:
cat /proc/sys/net/ipv6/conf/*/disable_ipv6
I get back a long list of ā€˜1ā€™ while ā€˜1ā€™ means disabled.
My conclusion is that some App or Tool might try IPv6 nonetheles and hence the error.
Any suggestion how to resolve this and get rid of those warnings?

5

Iā€™m going to give you some advice that you havenā€™t asked for (I hope you take it wisely)ā€¦

As I put in my Good Practices manual:

ā€œā€¦If you already have problems, identify the cause. Donā€™t do what most people do in their desperation to fix their problem and not get an answer from an official technical support: they search the forum and find a post with a problem ā€œsimilarā€ to theirs. Then, they start ā€œtryingā€ the ā€œsolutionsā€ mentioned there, entering commands left and right through the console, without really knowing what they are doing and making the original problem even worse The fact that another userā€™s problem. is very similar to yours does not mean that it is the same problem as yours. In the best of cases, the only real and verifiable thing is that they are similar, but without specifically identifying the problem, you will most likely just end up complicating and screwing everything upā€¦ā€

I told you in my first answer: ā€œā€¦Itā€™s hard to know whatā€™s happening to you just from this. The full umbrel log might provide more information on thisā€¦ā€
You have not used the log and it is information that can help you identify and correctly resolve your problem. There is a lot of information and context missing from your situation to be able to say what is really happening.

An example of this is that you have just discovered that your IPv6 is disabled. . Maybe that is the cause of the problem, that ā€œsomethingā€ needs and tries to connect ipv6.

Neither I nor anyone else will be able to help you without more information on the matter. We are sailing blindly, that is why you need/we need all the information possible. To start with the umbrelā€™s logā€¦
But as we still donā€™t have any more informationā€¦ well, Iā€™ll just repeat what I said a few lines back.

6

Thanks @jofraprez
The Umbrel-log you asked for is all the while 19 MB in size! Just by scrolling through it I can see that the above error messages take up most of the log file.
What I can also see is that the problem obviously started only after upgrading UmbrelOS to v1.3.0.
I would like to ask other Umbrel Home users to check their logs for this kind of warnings.
Excessive logging causes unnecessary write action on nvme disk which is no good for longevity.
Unfortunately upload size is limited to 12 MB and compressed files are not supported to upload.
Do you have an idea how to go about it?
Thanks.

7

I simply splitted the Logfile in two part.
Please have look and let me know.
Thanks.
umbrel-1736448221796_part1.log (9.3 MB)
umbrel-1736448221796_part2.log (9.5 MB)

8

I restarted my Umbrel Home as you can see in the log file part 1 at Jan 07 15:59:10 (line 37935).
shortly after stopping the Apps and docker containers the below error appears. I am not sure if it is of any relevance. The NetworkManager warnings started before restarting Umbrel Home although those warnings have not started at the beginning of the log file.
I hope you or someone else can help.
Thanks

Jan 07 15:59:15 umbrel bos[1769003]: err:
Jan 07 15:59:15 umbrel bos[1769003]: - 503
Jan 07 15:59:15 umbrel bos[1769003]: - ErrorInPaymentsSub
Jan 07 15:59:15 umbrel bos[1769003]: -
Jan 07 15:59:15 umbrel bos[1769003]: err:
Jan 07 15:59:15 umbrel bos[1769003]: - 503
Jan 07 15:59:15 umbrel bos[1769003]: - UnexpectedPastPaymentsErr
Jan 07 15:59:15 umbrel bos[1769003]: -
Jan 07 15:59:15 umbrel bos[1769003]: err:
Jan 07 15:59:15 umbrel bos[1769003]: message: 2 UNKNOWN: htlc event subscription terminated
Jan 07 15:59:15 umbrel bos[1769003]: stack:
Jan 07 15:59:15 umbrel bos[1769003]: - Error: 2 UNKNOWN: htlc event subscription terminated
Jan 07 15:59:15 umbrel bos[1769003]: - at callErrorFromStatus (/home/umbrel/.npm-global/lib/node_modules/balanceofsatoshis/node_modules/@grpc/grpc-js/build/src/call.js:31:19)
Jan 07 15:59:15 umbrel bos[1769003]: - at Object.onReceiveStatus (/home/umbrel/.npm-global/lib/node_modules/balanceofsatoshis/node_modules/@grpc/grpc-js/build/src/client.js:359:73)
Jan 07 15:59:15 umbrel bos[1769003]: - at Object.onReceiveStatus (/home/umbrel/.npm-global/lib/node_modules/balanceofsatoshis/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:323:181)
Jan 07 15:59:15 umbrel bos[1769003]: - at /home/umbrel/.npm-global/lib/node_modules/balanceofsatoshis/node_modules/@grpc/grpc-js/build/src/resolving-call.js:129:78
Jan 07 15:59:15 umbrel bos[1769003]: - at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
Jan 07 15:59:15 umbrel bos[1769003]: - for call at
Jan 07 15:59:15 umbrel bos[1769003]: - at ServiceClientImpl.makeServerStreamRequest (/home/umbrel/.npm-global/lib/node_modules/balanceofsatoshis/node_modules/@grpc/grpc-js/build/src/client.js:342:32)
Jan 07 15:59:15 umbrel bos[1769003]: - at ServiceClientImpl. (/home/umbrel/.npm-global/lib/node_modules/balanceofsatoshis/node_modules/@grpc/grpc-js/build/src/make-client.js:105:19)
Jan 07 15:59:15 umbrel bos[1769003]: - at module.exports (/home/umbrel/.npm-global/lib/node_modules/balanceofsatoshis/node_modules/lightning/lnd_methods/offchain/subscribe_to_past_payments.js:79:32)
Jan 07 15:59:15 umbrel bos[1769003]: - at /home/umbrel/.npm-global/lib/node_modules/balanceofsatoshis/telegram/start_telegram_bot.js:787:23
Jan 07 15:59:15 umbrel bos[1769003]: - at /home/umbrel/.npm-global/lib/node_modules/balanceofsatoshis/node_modules/async/internal/withoutIndex.js:8:40
Jan 07 15:59:15 umbrel bos[1769003]: - at eachOfArrayLike (/home/umbrel/.npm-global/lib/node_modules/balanceofsatoshis/node_modules/async/eachOf.js:61:9)
Jan 07 15:59:15 umbrel kernel: br-2ece48b239b9: port 47(veth7ac238d) entered disabled state
Jan 07 15:59:15 umbrel bos[1769003]: - at eachOf (/home/umbrel/.npm-global/lib/node_modules/balanceofsatoshis/node_modules/async/eachOf.js:181:12)
Jan 07 15:59:15 umbrel bos[1769003]: - at awaitable (/home/umbrel/.npm-global/lib/node_modules/balanceofsatoshis/node_modules/async/internal/awaitify.js:14:28)
Jan 07 15:59:15 umbrel bos[1769003]: - at eachLimit (/home/umbrel/.npm-global/lib/node_modules/balanceofsatoshis/node_modules/async/each.js:125:33)
Jan 07 15:59:15 umbrel bos[1769003]: - at awaitable (/home/umbrel/.npm-global/lib/node_modules/balanceofsatoshis/node_modules/async/internal/awaitify.js:14:28)
Jan 07 15:59:15 umbrel bos[1769003]: - at asyncAuto.payments (/home/umbrel/.npm-global/lib/node_modules/balanceofsatoshis/telegram/start_telegram_bot.js:786:16)
Jan 07 15:59:15 umbrel bos[1769003]: - at runTask (/home/umbrel/.npm-global/lib/node_modules/balanceofsatoshis/node_modules/async/auto.js:293:13)
Jan 07 15:59:15 umbrel bos[1769003]: - at /home/umbrel/.npm-global/lib/node_modules/balanceofsatoshis/node_modules/async/auto.js:233:31
Jan 07 15:59:15 umbrel bos[1769003]: - at processQueue (/home/umbrel/.npm-global/lib/node_modules/balanceofsatoshis/node_modules/async/auto.js:243:13)
Jan 07 15:59:15 umbrel bos[1769003]: - at taskComplete (/home/umbrel/.npm-global/lib/node_modules/balanceofsatoshis/node_modules/async/auto.js:259:9)
Jan 07 15:59:15 umbrel bos[1769003]: - at /home/umbrel/.npm-global/lib/node_modules/balanceofsatoshis/node_modules/async/auto.js:286:17

9

Well, letā€™s see if we can do something for youā€¦

Your log is huge and Iā€™m still reviewing it but there are already a couple of details that caught my attention.

First, Iā€™ve reviewed the logs of my 2 nodes and they donā€™t show those IPv6 errors. My nodes are connected and working through IPv4 and IPv6.

Your umbrel tries to get IPv6 when starting up but it never succeeds. After so many attempts, the same system ā€œdisablesā€ it but attempts to connect appear again. Hence the errors.

I assume that you get IPv4 and IPv6 from your routerā€™s dhcp and, for some reason, your router doesnā€™t assign IPv6 to the device. This is my first theory.
It would be convenient to check if your routerā€™s dhcp is serving both IPv4 and IPv6. Maybe you only have IPv4 active. Also, please confirm how you connect your node to the router. Directly or is there some device (switcher, hub, etc.) between them?

You say you have ā€œconfirmedā€ that ipv6 is completely disabled. Letā€™s do a different test. Log in via SSh and run the following command.

ip addr show

This will give you a list of all active/available connections.
You should look for one of the first 3 or 4 that contains the name ā€œenp1s0ā€. It should show you your ipv4 and ipv6 address. If ipv6 is really disabled it will only show you the ipv4. As an example, I show you the result of one of my nodes:

umbrel@umbrel:~$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc knockout state UNKNOWN group default qlen 1000
 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
 inet 127.0.0.1/8 scope host lo
 valid_lft forever preferred_lft forever
 inet6 ::1/128 scope host noprefixroute
 valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
 link/ether e8:ff:1e:d5:aa:65 brd ff:ff:ff:ff:ff:ff
 inet 192.168.0.10/24 brd 192.168.0.255 scope global noprefixroute enp1s0
 valid_lft forever preferred_lft forever
 inet6 2806:250:42c:c13b:519:6f88:80a6:442b/64 scope global dynamic noprefixroute
 valid_lft 3601sec preferred_lft 3601sec
 inet6 fe80::9333:6caf:bd5:a2ea/64 scope link noprefixroute
 valid_lft forever preferred_lft forever

Well, Iā€™ll wait for you to tell me what youā€™ve discovered and in the meantime Iā€™ll continue reviewing the logs.

10

I was able to solve the problem by following these steps:

  1. stopped LNDg
  2. stopped Lightning Node
  3. shut down Router and turned on again
  4. started Lightning Node
  5. started LNDg

When I checked the next morning no more NetworkManager warnings in the Log, What a relief!
What was the reasoning to try it this way?
Three days ago I run the updates for Elements Core, PeerSwap and RTL from the App Store. All updates installed without issues.
After that I updated npm, bos, node.js to their latest versions. These tools I need for running and managing my Lightning Node with 30+ channels. These updates finished as well without issues.
After finishing the updates I restarted Umbrel from the Umbrel Settings page.
During the restart of my Umbrel Node I also restarted the router (shut of and on again).
I guess this caused the problem with all the NetworkManager warnings regarding IPv6 because there were non those warnings before the restarting process.
I still donā€™t know the exact reason why the restart caused the IPv6 warnings but I am very glad that they are gone for now.

11

Fine. If you are happy with the situation and the problem has been resolved, letā€™s leave it there.
One problem lessā€¦ :slight_smile:

1 Like
12

I agree with your advice although I do sometimes exactly what you warn of.
For sure it would be better to have the technical support to solve the problem but the Umbrel support seems to be overloaded. I opened up a support ticket right after I realized the problem but all I got in answer was

Submitted - Weā€™ll pick up your ticket soon

So what to do when the Log file is growing like crazy?
I still agree with you to treat extremely careful because Linux Systems are all a bit different and the least posted solutions fit to the problem at hand.
I realized this very quickly so I did not change a thing. But what I found during my examination is that IPv6 is obviously disabled on kernel level in UmbrelOS while it is still active/enabled on device level.
I have no clue why that is? Perhaps someone can explain.
Thanks and all the best.

13

Hello jofrapez,

Thank you for your answer.
In the meantime I was able to solve the issue by rebooting my Router.

Best Regards
Thomas