umbrelOS 2.0 + USB SSD: if your box feels unreasonably slow or like the drive is dying, check your readahead before you buy a new one

If you’re on umbrelOS 2.0 with a USB-attached SSD and the machine has become unusable, read this before you replace any hardware. Mine looked exactly like a failing drive for two days. The drive was fine. It’s a storage misconfiguration that 2.0 makes fatal.

Symptoms

  • Dashboard times out whenever a heavy app is running (Bitcoin Core, for me)
  • SSH handshakes fail or hang
  • Boot takes 10+ minutes instead of 1-2
  • Processes sit in D state, disk pegged near 100% busy
  • SMART is clean, benchmarks look fine, dmesg has no errors

Important: this is NOT only a 2.0 problem. On 1.7.4 my box looked fine, dashboard loaded, SSH worked, but the chain had been syncing for MONTHS and was still about a week from catching up. Same bug, quieter symptom. If your node has been grinding away for weeks and never seems to get there, check this even if everything else feels normal.

What 2.0 changed is that the machine stops being usable on top of it. That is the part that makes you think the drive is dying. It isn’t.

The 30-second check

dmesg | grep "Optimal transfer size"
cat /sys/block/sda/queue/read_ahead_kb

If you get this:

sd 0:0:0:0: [sda] Optimal transfer size 33553920 bytes
65520

you have the same problem. 65520 KB is a 64 MiB readahead window. Every other disk on the system is at 128 KB.

Do not buy a new enclosure either

I did. Different brand, and the bridge chip inside is a different ASMedia part (174c:55aa instead of 174c:235c). Same drive, same cable, same Pi. It reports the exact same bogus value and the readahead comes up at 65520 again.

So this is not one bad enclosure, it is how a whole family of ASMedia USB-SATA bridges answers this query. Save your money and set the sysfs value.

Why it wrecks the box

Your USB bridge is reporting an “optimal transfer size” of 65535 blocks. That’s 0xFFFF, the largest number that field can hold, which firmware uses to mean “I don’t know”. Linux believes it and sizes readahead from it.

The drive can’t absorb a read that big:

Setting Value Consequence
read_ahead_kb 65520 one readahead = 64 MiB
max_sectors_kb 512 so that becomes 128 separate requests
nr_requests 60 scheduler queue holds 60
queue_depth 30 device queue holds 30

One readahead burst is more than twice the scheduler queue and more than four times the device queue. Anything else that wants to read 4 KB waits behind the entire burst, about 0.6 seconds. That happens over and over, which is why the dashboard times out and SSH won’t complete a handshake.

The fix

sudo sh -c 'echo 512 > /sys/block/sda/queue/read_ahead_kb'

That’s the whole thing. It takes effect immediately, no reboot, and you can undo it by writing 65520 back.

Why 512 and not 128

128 is the kernel default and it fixes the disaster on its own, which is what I ran first. When I went back and benchmarked the two against each other properly (alternating A/B pairs on an idle box, reading a raw window bigger than RAM so page cache could not fake it), 512 won every single pair:

read_ahead_kb Median throughput p95 latency
128 26.0 MB/s 62.7 ms
512 152.6 MB/s 2.54 ms

The reason is that max_sectors_kb on my device is 512, so at RA=512 one readahead is exactly one maximum-size request and streaming actually engages. At 128 every read is its own round trip.

If you prefer to stick to the documented default, 128 is still a completely fine answer and gets you 99% of the way. 512 is the better number if you want the rest.

Before and after on my box

Same 6 blocks, same checklevel, from Bitcoin Core’s own debug.log:

readahead 65520:  16% done after 8 min 51 s, still going at 17 min
readahead 128:    100% done in 5 seconds
Measure Before After
Boot to usable ~13 minutes 14 seconds
Sync rate 700-2200 blocks per day 1374 blocks per hour
Dashboard while syncing timing out 4 ms median

Worth stressing: in a controlled A/B with identical load, background throughput was 89.9 MiB/s before and 84.1 MiB/s after. The huge readahead was buying no bandwidth at all. It only starved everything else, so there is no trade-off here.

Making it survive a reboot

The value resets on every boot because it’s derived from the drive at probe time. umbrelOS ships a hook that runs as root before any app starts, which is a good place for it:

sudo mkdir -p /opt/umbrel-custom-hooks
sudo tee /opt/umbrel-custom-hooks/run-pre-start >/dev/null <<'EOF'
#!/usr/bin/env bash
for d in /sys/block/sd*; do
    dev="$(basename "$d")"
    if readlink -f "/sys/block/${dev}" | grep -q usb; then
        echo 512 > "/sys/block/${dev}/queue/read_ahead_kb" 2>/dev/null || true
    fi
done
exit 0
EOF
sudo chmod 755 /opt/umbrel-custom-hooks/run-pre-start

It’s picked up by umbrel-custom-pre-start.service, which already exists and runs Before=umbrel.service. Confirmed working across two real reboots, including one where I swapped the enclosure at the same time.

Doing it here rather than after the apps start is also what fixed the slow boots, since the boot itself no longer runs at 64 MiB readahead.

Reported upstream

GitHub issue: umbrelOS 2.0.0-beta.1 on USB storage with a bogus optimal_io_size is unusable, and presents as a failing SSD · Issue #2222 · getumbrel/umbrel · GitHub

I hope this saves someone the time and money it took for me to figure this out.

Umbrel Storage Diagnostics.txt (7.7 KB)