New OOB less than 24 hours online and unreachable after installing Knots

Support and Troubleshooting
1

Well, this is an odd one. I’ve received my 3rd umbrel home and woke up to it being unreachable today, well, by ssh and http. I can ping it… but yah, that’s not much fun.

Last night I installed BTC Knots and asked it to download the BC. Nothing too crazy, right!

I did a power reset after not being able to reach it, but this had no effect. :frowning:

Glad to have this on my 3rd unit, as I would have been really disappointed and my confidence in the product would have been in question.

Thanks for any ideas, D.

2

I hate to wait… so I went ahead and restarted again. Same issue… can ping, can’t ssh, no port 80 web access.

Tried again and kept attempting reconnect via ssh from the moment powered back on. Success this time!!

I remembered that this time I installed Firefox… and hadn’t done that on previous installs. I was able to kill -9 the FF process and BANG! Just like that, the web access lit up. I’ve removed FF and won’t be putting that back on. What a CF.

Anyone else had this happen? What’s the scoop? Maybe I don’t understand the purpose of FF on the Umbrel.

3

That was a short lived success. After removing FF install, I did a reboot… and back to no login issues.

4

Any ideas what the offender might be? SSH was solid until i dropped out of TOP and then tried a ‘last’.

image
image1924×1542 436 KB

5

Start over again as I can’t have flaky stuff… was able to get in via web after trying a reboot again and quickly trying login. Doing a Factory RESET. (I hope)

image
image922×1880 84.6 KB

6

In the interim… Looks like some possible hardware issues here:

umbrel@umbrel:~$ sudo dmesg | grep -i error
[sudo] password for umbrel:
[ 0.169407] ACPI BIOS Error (bug): Could not resolve symbol [_SB.PC00.TXHC.RHUB.SS01], AE_NOT_FOUND (20220331/dswload2-162)
[ 0.169414] ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20220331/psobject-220)
[ 0.169419] ACPI BIOS Error (bug): Could not resolve symbol [_SB.PC00.TXHC.RHUB.SS02], AE_NOT_FOUND (20220331/dswload2-162)
[ 0.169421] ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20220331/psobject-220)
[ 1.547085] usb 3-10: device descriptor read/64, error -71
[ 3.219165] iwlwifi 0000:00:14.3: Start IWL Error Log Dump:
[ 3.219441] iwlwifi 0000:00:14.3: 0x00000B03 | IML/ROM error/state
[ 3.219594] iwlwifi 0000:00:14.3: 0x20000000 | FSEQ_ERROR_CODE
[ 5.279171] iwlwifi 0000:00:14.3: Start IWL Error Log Dump:
[ 5.279446] iwlwifi 0000:00:14.3: 0x00000B03 | IML/ROM error/state
[ 5.279600] iwlwifi 0000:00:14.3: 0x60000000 | FSEQ_ERROR_CODE
**[ 364.893814] nvme0n1: I/O Cmd(0x2) @ LBA 1746617644, 2 blocks, I/O Error (sct 0x3 / sc 0x71) **
[ 364.893834] I/O error, dev nvme0n1, sector 1746617644 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[ 364.893884] nvme0n1: I/O Cmd(0x2) @ LBA 1650148818, 2 blocks, I/O Error (sct 0x3 / sc 0x71)
[ 364.893892] I/O error, dev nvme0n1, sector 1650148818 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
umbrel@umbrel:~$

@Umbrel Support - Thoughts? Perhaps an RMA??

7

Factory reset failed… and drive still complaining:

1431.068428] perf: interrupt took too long (2515 > 2500), lowering kernel.perf_event_max_sample_rate to 79500
[ 1540.969061] nvme0n1: I/O Cmd(0x2) @ LBA 1361045512, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1540.969085] critical medium error, dev nvme0n1, sector 1361045512 op 0x0:(READ) flags 0x80700 phys_seg 32 prio class 2
[ 1541.536848] nvme0n1: I/O Cmd(0x2) @ LBA 1361072136, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1541.536872] critical medium error, dev nvme0n1, sector 1361072136 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 1541.769501] nvme0n1: I/O Cmd(0x2) @ LBA 1361074184, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1541.769526] critical medium error, dev nvme0n1, sector 1361074184 op 0x0:(READ) flags 0x80700 phys_seg 10 prio class 2
[ 1541.891819] nvme0n1: I/O Cmd(0x2) @ LBA 1361076232, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1541.891845] critical medium error, dev nvme0n1, sector 1361076232 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 2
[ 1542.145671] nvme0n1: I/O Cmd(0x2) @ LBA 1361090568, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1542.145695] critical medium error, dev nvme0n1, sector 1361090568 op 0x0:(READ) flags 0x80700 phys_seg 9 prio class 2
[ 1565.929239] nvme0n1: I/O Cmd(0x2) @ LBA 1383834632, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1565.929264] critical medium error, dev nvme0n1, sector 1383834632 op 0x0:(READ) flags 0x80700 phys_seg 9 prio class 2
[ 1595.930140] nvme0n1: I/O Cmd(0x2) @ LBA 1398120456, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1595.930164] critical medium error, dev nvme0n1, sector 1398120456 op 0x0:(READ) flags 0x80700 phys_seg 64 prio class 2
[ 1598.150193] nvme0n1: I/O Cmd(0x2) @ LBA 1398262280, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1598.150220] critical medium error, dev nvme0n1, sector 1398262280 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 2
[ 1648.794965] nvme0n1: I/O Cmd(0x2) @ LBA 1417244168, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1648.794989] critical medium error, dev nvme0n1, sector 1417244168 op 0x0:(READ) flags 0x80700 phys_seg 16 prio class 2
[ 1649.399874] nvme0n1: I/O Cmd(0x2) @ LBA 1417292296, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1649.399898] critical medium error, dev nvme0n1, sector 1417292296 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 1695.744935] nvme0n1: I/O Cmd(0x2) @ LBA 1436833288, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1695.744960] critical medium error, dev nvme0n1, sector 1436833288 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 1698.261732] nvme0n1: I/O Cmd(0x2) @ LBA 1437093896, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1698.261756] critical medium error, dev nvme0n1, sector 1437093896 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 1711.109647] nvme0n1: I/O Cmd(0x2) @ LBA 1443024904, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1711.109671] critical medium error, dev nvme0n1, sector 1443024904 op 0x0:(READ) flags 0x80700 phys_seg 54 prio class 2
[ 1711.408109] nvme0n1: I/O Cmd(0x2) @ LBA 1443061256, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1711.408134] critical medium error, dev nvme0n1, sector 1443061256 op 0x0:(READ) flags 0x80700 phys_seg 64 prio class 2
[ 1717.061321] nvme0n1: I/O Cmd(0x2) @ LBA 1443521544, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1717.061342] critical medium error, dev nvme0n1, sector 1443521544 op 0x0:(READ) flags 0x80700 phys_seg 64 prio class 2
[ 1802.337469] nvme0n1: I/O Cmd(0x2) @ LBA 1490765832, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1802.337493] critical medium error, dev nvme0n1, sector 1490765832 op 0x0:(READ) flags 0x80700 phys_seg 64 prio class 2
[ 1820.421929] nvme0n1: I/O Cmd(0x2) @ LBA 1507597320, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1820.421954] critical medium error, dev nvme0n1, sector 1507597320 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 1820.956056] nvme0n1: I/O Cmd(0x2) @ LBA 1507603464, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1820.956080] critical medium error, dev nvme0n1, sector 1507603464 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 1821.685543] nvme0n1: I/O Cmd(0x2) @ LBA 1507652104, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1821.685567] critical medium error, dev nvme0n1, sector 1507652104 op 0x0:(READ) flags 0x80700 phys_seg 8 prio class 2
[ 1821.878704] nvme0n1: I/O Cmd(0x2) @ LBA 1507654152, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1821.878728] critical medium error, dev nvme0n1, sector 1507654152 op 0x0:(READ) flags 0x80700 phys_seg 8 prio class 2
[ 1880.419162] nvme0n1: I/O Cmd(0x2) @ LBA 1561064456, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1880.419186] critical medium error, dev nvme0n1, sector 1561064456 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 2
[ 1880.583224] nvme0n1: I/O Cmd(0x2) @ LBA 1561072648, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1880.583249] critical medium error, dev nvme0n1, sector 1561072648 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 1880.895330] nvme0n1: I/O Cmd(0x2) @ LBA 1561073056, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1880.895354] critical medium error, dev nvme0n1, sector 1561073056 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
[ 1880.895368] Buffer I/O error on dev nvme0n1, logical block 195134132, async page read
[ 1881.251169] nvme0n1: I/O Cmd(0x2) @ LBA 1561078792, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1881.251193] critical medium error, dev nvme0n1, sector 1561078792 op 0x0:(READ) flags 0x80700 phys_seg 16 prio class 2
[ 1896.589321] nvme0n1: I/O Cmd(0x2) @ LBA 1576735240, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1896.589346] critical medium error, dev nvme0n1, sector 1576735240 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 2
[ 1950.450090] nvme0n1: I/O Cmd(0x2) @ LBA 1610426376, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE[ 1950.450090] nvme0n1: I/O Cmd(0x2) @ LBA 1610426376, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE[ 1950.450090] nvme0n1: I/O Cmd(0x2) @ LBA 1610426376, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1950.450115] critical medium error, dev nvme0n1, sector 1610426376 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 2
[ 1951.026748] nvme0n1: I/O Cmd(0x2) @ LBA 1610442760, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1951.026772] critical medium error, dev nvme0n1, sector 1610442760 op 0x0:(READ) flags 0x80700 phys_seg 16 prio class 2
[ 1967.820644] nvme0n1: I/O Cmd(0x2) @ LBA 1631283720, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1967.820668] critical medium error, dev nvme0n1, sector 1631283720 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 2
[ 1971.131092] nvme0n1: I/O Cmd(0x2) @ LBA 1633366536, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1971.131111] critical medium error, dev nvme0n1, sector 1633366536 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 1971.552249] nvme0n1: I/O Cmd(0x2) @ LBA 1633384968, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 1971.552273] critical medium error, dev nvme0n1, sector 1633384968 op 0x0:(READ) flags 0x80700 phys_seg 16 prio class 2
[ 2023.327749] nvme0n1: I/O Cmd(0x2) @ LBA 1683686920, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2023.327772] critical medium error, dev nvme0n1, sector 1683686920 op 0x0:(READ) flags 0x80700 phys_seg 9 prio class 2
[ 2023.710999] nvme0n1: I/O Cmd(0x2) @ LBA 1683699208, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2023.711023] critical medium error, dev nvme0n1, sector 1683699208 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 2024.219599] nvme0n1: I/O Cmd(0x2) @ LBA 1683715592, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2024.219623] critical medium error, dev nvme0n1, sector 1683715592 op 0x0:(READ) flags 0x80700 phys_seg 8 prio class 2
[ 2026.508174] nvme0n1: I/O Cmd(0x2) @ LBA 1685812232, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2026.508198] critical medium error, dev nvme0n1, sector 1685812232 op 0x0:(READ) flags 0x80700 phys_seg 9 prio class 2
[ 2044.949134] nvme0n1: I/O Cmd(0x2) @ LBA 1702581768, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2044.949158] critical medium error, dev nvme0n1, sector 1702581768 op 0x0:(READ) flags 0x80700 phys_seg 16 prio class 2
[ 2060.487059] nvme0n1: I/O Cmd(0x2) @ LBA 1719361032, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2060.487084] critical medium error, dev nvme0n1, sector 1719361032 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 2060.643018] nvme0n1: I/O Cmd(0x2) @ LBA 1719363080, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2060.643042] critical medium error, dev nvme0n1, sector 1719363080 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 2060.835074] nvme0n1: I/O Cmd(0x2) @ LBA 1719365128, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2060.835099] critical medium error, dev nvme0n1, sector 1719365128 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 2079.037457] nvme0n1: I/O Cmd(0x2) @ LBA 1729833480, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2079.037481] critical medium error, dev nvme0n1, sector 1729833480 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 2163.610057] nvme0n1: I/O Cmd(0x2) @ LBA 1805424648, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2163.610082] critical medium error, dev nvme0n1, sector 1805424648 op 0x0:(READ) flags 0x80700 phys_seg 33 prio class 2
[ 2168.081089] nvme0n1: I/O Cmd(0x2) @ LBA 1807436808, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2168.081113] critical medium error, dev nvme0n1, sector 1807436808 op 0x0:(READ) flags 0x80700 phys_seg 64 prio class 2
[ 2218.738558] nvme0n1: I/O Cmd(0x2) @ LBA 1857689096, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2218.738585] critical medium error, dev nvme0n1, sector 1857689096 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 2
[ 2287.171959] nvme0n1: I/O Cmd(0x2) @ LBA 1917233672, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2287.171983] critical medium error, dev nvme0n1, sector 1917233672 op 0x0:(READ) flags 0x80700 phys_seg 64 prio class 2
[ 2290.657513] nvme0n1: I/O Cmd(0x2) @ LBA 1917547016, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2290.657538] critical medium error, dev nvme0n1, sector 1917547016 op 0x0:(READ) flags 0x80700 phys_seg 10 prio class 2
[ 2291.536488] nvme0n1: I/O Cmd(0x2) @ LBA 1917606920, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2291.536513] critical medium error, dev nvme0n1, sector 1917606920 op 0x0:(READ) flags 0x80700 phys_seg 9 prio class 2
[ 2293.817814] nvme0n1: I/O Cmd(0x2) @ LBA 1917851144, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2293.817839] critical medium error, dev nvme0n1, sector 1917851144 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 2308.087175] nvme0n1: I/O Cmd(0x2) @ LBA 1933258248, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2308.087199] critical medium error, dev nvme0n1, sector 1933258248 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 2
[ 2308.825388] nvme0n1: I/O Cmd(0x2) @ LBA 1933300744, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2308.825412] critical medium error, dev nvme0n1, sector 1933300744 op 0x0:(READ) flags 0x80700 phys_seg 64 prio class 2
[ 2342.807598] nvme0n1: I/O Cmd(0x2) @ LBA 1958128648, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2342.807623] critical medium error, dev nvme0n1, sector 1958128648 op 0x0:(READ) flags 0x80700 phys_seg 7 prio class 2
[ 2343.186875] nvme0n1: I/O Cmd(0x2) @ LBA 1958152712, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2343.186900] critical medium error, dev nvme0n1, sector 1958152712 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 2
[ 2343.409448] nvme0n1: I/O Cmd(0x2) @ LBA 1958152960, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2343.409472] critical medium error, dev nvme0n1, sector 1958152960 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
[ 2343.409486] Buffer I/O error on dev nvme0n1, logical block 244769120, async page read
[ 2363.326066] nvme0n1: I/O Cmd(0x2) @ LBA 1972120072, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2363.326090] critical medium error, dev nvme0n1, sector 1972120072 op 0x0:(READ) flags 0x80700 phys_seg 18 prio class 2
[ 2430.409971] nvme0n1: I/O Cmd(0x2) @ LBA 2033982472, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2430.409990] critical medium error, dev nvme0n1, sector 2033982472 op 0x0:(READ) flags 0x80700 phys_seg 16 prio class 2
[ 2430.908874] nvme0n1: I/O Cmd(0x2) @ LBA 2034021384, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2430.908899] critical medium error, dev nvme0n1, sector 2034021384 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 2433.297346] nvme0n1: I/O Cmd(0x2) @ LBA 2036061192, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2433.297371] critical medium error, dev nvme0n1, sector 2036061192 op 0x0:(READ) flags 0x80700 phys_seg 63 prio class 2
[ 2448.491868] nvme0n1: I/O Cmd(0x2) @ LBA 2044155400, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2448.491892] critical medium error, dev nvme0n1, sector 2044155400 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 2448.598602] nvme0n1: I/O Cmd(0x2) @ LBA 2044157448, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2448.598627] critical medium error, dev nvme0n1, sector 2044157448 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 2483.778566] nvme0n1: I/O Cmd(0x2) @ LBA 2065377288, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2483.778590] critical medium error, dev nvme0n1, sector 2065377288 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 2483.984896] nvme0n1: I/O Cmd(0x2) @ LBA 2065382920, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2483.984920] critical medium error, dev nvme0n1, sector 2065382920 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 2484.440019] nvme0n1: I/O Cmd(0x2) @ LBA 2065399304, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2484.440044] critical medium error, dev nvme0n1, sector 2065399304 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 2
[ 2486.038893] nvme0n1: I/O Cmd(0x2) @ LBA 2067466248, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2486.038918] critical medium error, dev nvme0n1, sector 2067466248 op 0x0:(READ) flags 0x80700 phys_seg 64 prio class 2
[ 2523.999520] nvme0n1: I/O Cmd(0x2) @ LBA 2103144456, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2523.999554] critical medium error, dev nvme0n1, sector 2103144456 op 0x0:(READ) flags 0x80700 phys_seg 33 prio class 2
[ 2573.088566] nvme0n1: I/O Cmd(0x2) @ LBA 2138781192, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2573.088591] critical medium error, dev nvme0n1, sector 2138781192 op 0x0:(READ) flags 0x80700 phys_seg 33 prio class 2
[ 2573.268467] nvme0n1: I/O Cmd(0x2) @ LBA 2138783752, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2573.268491] critical medium error, dev nvme0n1, sector 2138783752 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 2733.728920] nvme0n1: I/O Cmd(0x2) @ LBA 2275113992, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2733.728944] critical medium error, dev nvme0n1, sector 2275113992 op 0x0:(READ) flags 0x80700 phys_seg 33 prio class 2
[ 2734.324315] nvme0n1: I/O Cmd(0x2) @ LBA 2275150856, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2734.324340] critical medium error, dev nvme0n1, sector 2275150856 op 0x0:(READ) flags 0x80700 phys_seg 32 prio class 2
[ 2735.107297] nvme0n1: I/O Cmd(0x2) @ LBA 2275197448, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2735.107321] critical medium error, dev nvme0n1, sector 2275197448 op 0x0:(READ) flags 0x80700 phys_seg 38 prio class 2
[ 2737.031822] nvme0n1: I/O Cmd(0x2) @ LBA 2277202440, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2737.031849] critical medium error, dev nvme0n1, sector 2277202440 op 0x0:(READ) flags 0x80700 phys_seg 32 prio class 2
[ 2774.106567] nvme0n1: I/O Cmd(0x2) @ LBA 2305067528, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2774.106591] critical medium error, dev nvme0n1, sector 2305067528 op 0x0:(READ) flags 0x80700 phys_seg 9 prio class 2
[ 2780.192718] perf: interrupt took too long (3146 > 3143), lowering kernel.perf_event_max_sample_rate to 63500
[ 2790.857469] nvme0n1: I/O Cmd(0x2) @ LBA 24396824, 256 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2790.857494] critical medium error, dev nvme0n1, sector 24396824 op 0x0:(READ) flags 0x80700 phys_seg 32 prio class 2
[ 2812.668784] nvme0n1: I/O Cmd(0x2) @ LBA 2333959688, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2812.668804] critical medium error, dev nvme0n1, sector 2333959688 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 2916.010237] nvme0n1: I/O Cmd(0x2) @ LBA 2401003016, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2916.010261] critical medium error, dev nvme0n1, sector 2401003016 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 2
[ 2932.763292] nvme0n1: I/O Cmd(0x2) @ LBA 2411734536, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2932.763318] critical medium error, dev nvme0n1, sector 2411734536 op 0x0:(READ) flags 0x80700 phys_seg 31 prio class 2
[ 2956.174871] nvme0n1: I/O Cmd(0x2) @ LBA 2417717768, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 2956.174894] critical medium error, dev nvme0n1, sector 2417717768 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 3230.899875] nvme0n1: I/O Cmd(0x2) @ LBA 2591753736, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 3230.899900] critical medium error, dev nvme0n1, sector 2591753736 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 3508.167822] nvme0n1: I/O Cmd(0x2) @ LBA 2805681672, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 3508.167847] critical medium error, dev nvme0n1, sector 2805681672 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2

8

okay, leaving this one here. i think this guy is toast!

[ 3669.956730] critical medium error, dev nvme0n1, sector 2939900936 op 0x0:(READ) flags 0x80700 phys_seg 33 prio class 2
[ 3921.795637] nvme0n1: I/O Cmd(0x2) @ LBA 3116812296, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 3921.795660] critical medium error, dev nvme0n1, sector 3116812296 op 0x0:(READ) flags 0x80700 phys_seg 17 prio class 2
[ 3971.132225] nvme0n1: I/O Cmd(0x2) @ LBA 3157997576, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 3971.132249] critical medium error, dev nvme0n1, sector 3157997576 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 2
[ 3971.310421] nvme0n1: I/O Cmd(0x2) @ LBA 3158003720, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 3971.310445] critical medium error, dev nvme0n1, sector 3158003720 op 0x0:(READ) flags 0x80700 phys_seg 32 prio class 2
[ 4008.730350] nvme0n1: I/O Cmd(0x2) @ LBA 3183179784, 512 blocks, I/O Error (sct 0x2 / sc 0x81) MORE
[ 4008.730375] critical medium error, dev nvme0n1, sector 3183179784 op 0x0:(READ) flags 0x80700 phys_seg 32 prio class 2
[ 4080.478261] nvme nvme0: I/O 327 (I/O Cmd) QID 3 timeout, aborting
[ 4080.478298] nvme nvme0: I/O 328 (I/O Cmd) QID 3 timeout, aborting
[ 4111.195956] nvme nvme0: I/O 327 QID 3 timeout, reset controller
[ 4141.912550] nvme nvme0: I/O 12 QID 0 timeout, reset controller
[ 4229.951428] INFO: task jbd2/nvme0n1p4-:434 blocked for more than 120 seconds.
[ 4229.951443] Not tainted 6.1.0-31-amd64 #1 Debian 6.1.128-1
[ 4229.951448] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[ 4229.951451] task:jbd2/nvme0n1p4- state:D stack:0 pid:434 ppid:2 flags:0x00004000
[ 4229.951464] Call Trace:
[ 4229.951468]
[ 4229.951477] __schedule+0x34d/0x9e0
[ 4229.951498] schedule+0x5a/0xd0
[ 4229.951506] io_schedule+0x42/0x70
[ 4229.951514] bit_wait_io+0xd/0x60
[ 4229.951523] __wait_on_bit+0x48/0x140
[ 4229.951531] ? out_of_line_wait_on_bit_lock+0xb0/0xb0
[ 4229.951541] out_of_line_wait_on_bit+0x91/0xb0
[ 4229.951549] ? sugov_init+0x350/0x350
[ 4229.951563] jbd2_journal_commit_transaction+0x119d/0x1a50 [jbd2]
[ 4229.951603] ? psi_task_switch+0xb3/0x1f0
[ 4229.951614] ? __switch_to_asm+0x3a/0x60
[ 4229.951623] ? finish_task_switch.isra.0+0x90/0x2d0
[ 4229.951633] kjournald2+0xa9/0x280 [jbd2]
[ 4229.951668] ? cpuusage_read+0x10/0x10
[ 4229.951676] ? jbd2_fc_wait_bufs+0xa0/0xa0 [jbd2]
[ 4229.951704] kthread+0xd7/0x100
[ 4229.951714] ? kthread_complete_and_exit+0x20/0x20
[ 4229.951723] ret_from_fork+0x1f/0x30
[ 4229.951735]
[ 4229.951738] INFO: task systemd-timesyn:457 blocked for more than 120 seconds.
[ 4229.951742] Not tainted 6.1.0-31-amd64 #1 Debian 6.1.128-1
[ 4229.951745] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[ 4229.951748] task:systemd-timesyn state:D stack:0 pid:457 ppid:1 flags:0x00000002
[ 4229.951756] Call Trace:
[ 4229.951758]
[ 4229.951761] __schedule+0x34d/0x9e0
[ 4229.951771] schedule+0x5a/0xd0
[ 4229.951778] io_schedule+0x42/0x70
[ 4229.951786] bit_wait_io+0xd/0x60
[ 4229.951793] __wait_on_bit+0x48/0x140
[ 4229.951801] ? out_of_line_wait_on_bit_lock+0xb0/0xb0
[ 4229.951810] out_of_line_wait_on_bit+0x91/0xb0
[ 4229.951818] ? sugov_init+0x350/0x350
[ 4229.951827] do_get_write_access+0x244/0x3b0 [jbd2]
[ 4229.951857] jbd2_journal_get_write_access+0x5b/0x80 [jbd2]
[ 4229.951885] __ext4_journal_get_write_access+0x81/0x180 [ext4]
[ 4229.951964] ext4_reserve_inode_write+0x5d/0xb0 [ext4]
[ 4229.952045] __ext4_mark_inode_dirty+0x74/0x240 [ext4]
[ 4229.952118] ? __ext4_journal_start_sb+0x105/0x120 [ext4]
[ 4229.952183] ext4_dirty_inode+0x56/0x80 [ext4]
[ 4229.952255] __mark_inode_dirty+0x53/0x380
[ 4229.952266] ext4_setattr+0x124/0xb50 [ext4]
[ 4229.952336] ? path_lookupat+0x97/0x190
[ 4229.952344] notify_change+0x2aa/0x5c0
[ 4229.952355] ? vfs_utimes+0x13c/0x270
[ 4229.952363] vfs_utimes+0x13c/0x270
[ 4229.952374] do_utimes+0xe6/0x150
[ 4229.952383] __x64_sys_utimensat+0x9d/0xe0
[ 4229.952394] do_syscall_64+0x55/0xb0
[ 4229.952402] ? do_syscall_64+0x61/0xb0
[ 4229.952408] ? __x64_sys_epoll_ctl+0x6c/0xa0
[ 4229.952416] ? exit_to_user_mode_prepare+0x40/0x1e0
[ 4229.952425] ? syscall_exit_to_user_mode+0x1e/0x40
[ 4229.952434] ? do_syscall_64+0x61/0xb0
[ 4229.952440] ? do_timerfd_settime+0x33d/0x500
[ 4229.952449] ? __x64_sys_timerfd_settime+0x9c/0xf0
[ 4229.952457] ? exit_to_user_mode_prepare+0x40/0x1e0
[ 4229.952464] ? syscall_exit_to_user_mode+0x1e/0x40
[ 4229.952473] ? do_syscall_64+0x61/0xb0
[ 4229.952479] ? do_syscall_64+0x61/0xb0
[ 4229.952484] ? do_syscall_64+0x61/0xb0
[ 4229.952491] ? exit_to_user_mode_prepare+0x40/0x1e0
[ 4229.952498] ? syscall_exit_to_user_mode+0x1e/0x40
[ 4229.952507] ? do_syscall_64+0x61/0xb0
[ 4229.952512] ? do_syscall_64+0x61/0xb0
[ 4229.952518] entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[ 4229.952527] RIP: 0033:0x7f0f9f71b89f
[ 4229.952534] RSP: 002b:00007ffeb0d9f1a8 EFLAGS: 00000202 ORIG_RAX: 0000000000000118
[ 4229.952541] RAX: ffffffffffffffda RBX: 00007ffeb0d9f210 RCX: 00007f0f9f71b89f
[ 4229.952546] RDX: 00007ffeb0d9f210 RSI: 00007ffeb0d9f1b0 RDI: 00000000ffffff9c
[ 4229.952550] RBP: 00007ffeb0d9f1b0 R08: 0000000000000000 R09: 0000000000000069
[ 4229.952554] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000000
[ 4229.952558] R13: 00000000ffffffff R14: 000631f05bbd5c00 R15: 00000000ffffffff
[ 4229.952565]
[ 4229.952638] INFO: task kworker/u8:0:25888 blocked for more than 120 seconds.
[ 4229.952641] Not tainted 6.1.0-31-amd64 #1 Debian 6.1.128-1
[ 4229.952644] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[ 4229.952654] task:kworker/u8:0 state:D stack:0 pid:25888 ppid:2 flags:0x00004000
[ 4229.952662] Workqueue: writeback wb_workfn (flush-259:0)
[ 4229.952673] Call Trace:
[ 4229.952675]
[ 4229.952678] __schedule+0x34d/0x9e0
[ 4229.952685] ? rwb_trace_step+0x90/0x90
[ 4229.952695] ? wbt_done+0xc0/0xc0
[ 4229.952703] schedule+0x5a/0xd0
[ 4229.952710] io_schedule+0x42/0x70
[ 4229.952716] rq_qos_wait+0xbc/0x130
[ 4229.952725] ? karma_partition+0x190/0x190
[ 4229.952731] ? rwb_trace_step+0x90/0x90
[ 4229.952740] wbt_wait+0xa2/0x110
[ 4229.952749] __rq_qos_throttle+0x20/0x40
[ 4229.952757] blk_mq_submit_bio+0x18c/0x610
[ 4229.952767] submit_bio_noacct_nocheck+0x32e/0x370
[ 4229.952776] ? submit_bio_noacct+0x79/0x4b0
[ 4229.952783] ext4_bio_write_page+0x351/0x810 [ext4]
[ 4229.952859] mpage_submit_page+0x5c/0x80 [ext4]
[ 4229.952923] mpage_process_page_bufs+0x135/0x140 [ext4]
[ 4229.952985] mpage_prepare_extent_to_map+0x1f4/0x440 [ext4]
[ 4229.953048] ext4_writepages+0x451/0x11e0 [ext4]
[ 4229.953114] ? krc_this_cpu_lock+0x2e/0x40
[ 4229.953123] ? select_task_rq_fair+0x159/0x1840
[ 4229.953129] do_writepages+0xbd/0x1c0
[ 4229.953137] ? __wb_calc_thresh+0x4b/0x140
[ 4229.953142] __writeback_single_inode+0x3d/0x360
[ 4229.953149] writeback_sb_inodes+0x1f5/0x4d0
[ 4229.953158] __writeback_inodes_wb+0x4c/0xf0
[ 4229.953165] wb_writeback+0x204/0x2f0
[ 4229.953172] wb_workfn+0x2aa/0x4f0
[ 4229.953181] process_one_work+0x1c4/0x380
[ 4229.953188] worker_thread+0x4d/0x380
[ 4229.953193] ? rescuer_thread+0x3a0/0x3a0
[ 4229.953196] kthread+0xd7/0x100
[ 4229.953203] ? kthread_complete_and_exit+0x20/0x20
[ 4229.953211] ret_from_fork+0x1f/0x30
[ 4229.953220]
[ 4257.259793] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 3min)!
[ 4257.259878] systemd[1]: systemd-timesyncd.service: Killing process 457 (systemd-timesyn) with signal SIGABRT.

9

umbrel@umbrel:~$ sudo dd if=/dev/nvme0n1 of=/dev/null bs=4M status=progress
[sudo] password for umbrel:
1643790995456 bytes (1.6 TB, 1.5 TiB) copied, 4435 s, 371 MB/s
391908+3 records in
391908+3 records out
1643790995456 bytes (1.6 TB, 1.5 TiB) copied, 4434.93 s, 371 MB/s
Bus error
umbrel@umbrel:~$