FailSafe Migration Failing, "Dataset Already Exists", Inotify Errors, and Eventually Successful RAIDZ1 Migration

Support and Troubleshooting
1

Hey all

Got a second WD_BLACK SN850X the other day and the Failsafe activation failed. I spent the last 2 days troubleshooting with ChatGPT and I finally managed to initialize Failsafe successfully.

Here’s a summary of what happened, with some suggestions for Umbrel’s devs at the bottom.

Umbrel Home UI Randomly Freezing, FailSafe Migration Failing, “Dataset Already Exists”, Inotify Errors, and Eventually Successful RAIDZ1 Migration

Summary

I wanted to enable FailSafe on my Umbrel Home using two identical WD_BLACK SN850X 8TB SSDs.

What started as a simple storage upgrade turned into a multi-day troubleshooting process involving:

  • Random Umbrel UI lockups

  • Dashboard becoming unreachable while apps remained operational

  • High CPU spikes

  • Suspected SSD overheating

  • Suspected hardware instability

  • FailSafe migration failures

  • ZFS snapshot errors

  • “dataset already exists” errors

  • Inotify watch failures

  • Umbrel dashboard getting stuck at 52%

  • RAID resilver monitoring

In the end:

:white_check_mark: FailSafe successfully enabled

:white_check_mark: RAIDZ1 created

:white_check_mark: Both 8TB SSDs healthy

:white_check_mark: ZFS pool healthy

:white_check_mark: No data loss

However, I believe there may still be an underlying Umbrel issue involving file watchers, Nextcloud previews, and inotify limits.

Environment

Hardware

  • Umbrel Home

  • 16GB RAM

  • WD_BLACK SN850X 8TB (existing drive)

  • WD_BLACK SN850X 8TB (new drive)

Installed Apps

  • Bitcoin

  • Lightning

  • Electrs

  • BTC RPC Explorer

  • Nextcloud

  • OnlyOffice

  • AdGuard Home

  • Ollama

  • OpenClaw

  • Passky

  • Jellyfin

  • Other miscellaneous apps

Original Problem (Before Installing Second SSD)

The first symptom was not related to FailSafe at all.

Umbrel had become unstable.

Symptoms:

  • Dashboard frequently becoming unresponsive

  • Clicking buttons did nothing

  • Refreshing the page often resulted in a blank page

  • Sometimes the dashboard would not load at all

  • Apps remained functional

  • Nextcloud continued working

  • Bitcoin node continued working

This made the issue very strange because Umbrel itself appeared partially alive.

Initial Investigation

Checked:

free -h


top


sudo docker ps


sudo docker stats --no-stream

Findings:

  • CPU occasionally spiking above 80%

  • Memory usage elevated but not exhausted

  • Containers appeared healthy

Suspicion Falls on the Second SSD

At this point I had previously experimented with a second SN850X.

Observations:

  • The second SSD became extremely hot to the touch

  • Considerably hotter than the primary SSD

  • UI instability appeared worse with the second SSD installed

This led me to suspect:

  • NVMe overheating

  • PCIe issues

  • Hardware instability

First Major Discovery

Umbrel logs repeatedly contained:

No space left on device

along with:

inotify_add_watch failed

and

Failed to watch directory

This was the first indication that the issue might not be hardware-related at all.

Removing Unused Apps

As a test, I removed:

  • Canary

  • Several unused applications

Result:

  • CPU usage dropped

  • Umbrel became dramatically more stable

  • Dashboard remained responsive much longer

This was the first sign that resource exhaustion might be involved.

Reinstalling Second SSD

After stability improved, I reinstalled the second WD_BLACK SN850X.

Verification:

lsblk

Both drives detected.

SMART:

sudo smartctl -a /dev/nvme0
sudo smartctl -a /dev/nvme1

Results:

  • Critical Warning = 0x00

  • Percentage Used = 0%

  • No SMART errors

Temperatures:

  • ~32°C

  • ~38°C

Much cooler than originally suspected.

This largely eliminated hardware concerns.

Attempting FailSafe

FailSafe failed immediately.

Error:

cannot create snapshot 'umbrelos-d16679a1@migration':
dataset already exists

ZFS Investigation

Checked:

zpool status


sudo zfs list


sudo zfs list -t snapshot

Results:

  • Pool healthy

  • No migration snapshot

  • No migration dataset

Yet FailSafe insisted:

dataset already exists

Verifying ZFS Functionality

Manual snapshot test:

sudo zfs snapshot umbrelos-d16679a1@testsnapshot


sudo zfs list -t snapshot


sudo zfs destroy umbrelos-d16679a1@testsnapshot

All worked perfectly.

This proved:

  • ZFS healthy

  • Snapshot functionality healthy

  • Storage healthy

Discovering the Inotify Problem

Logs repeatedly showed:

inotify_add_watch ... failed:
No space left on device

Example:

/home/umbrel/umbrel/app-data/nextcloud/data/nextcloud/data/appdata_xxxxx/preview/...

Notice this was inside:

Nextcloud preview cache

Increasing Inotify Limits

Current values:

cat /proc/sys/fs/inotify/max_user_watches
cat /proc/sys/fs/inotify/max_user_instances

Returned:

122404
256

We increased them:

echo "fs.inotify.max_user_watches=1048576" | sudo tee /etc/sysctl.d/99-inotify.conf

echo "fs.inotify.max_user_instances=1024" | sudo tee -a /etc/sysctl.d/99-inotify.conf

sudo sysctl --system

New values:

1048576
1024

FailSafe Suddenly Starts Working

After increasing the limits:

FailSafe progressed.

Dashboard showed:

Enabling FailSafe...
1%

Migration Progress

Migration advanced normally.

At approximately 50%:

Umbrel rebooted automatically.

This was expected behavior.

Dashboard Gets Stuck at 52%

After reboot:

Dashboard remained at:

52%

for a very long time.

Initially appeared stuck.

ZFS Reveals Reality

Running:

zpool status

showed:

scan: resilver in progress

and:

1.3 GB/s reads
1.3 GB/s writes

with:

36% done

The migration was continuing normally.

The UI simply wasn’t updating.

Migration Completes Successfully

Eventually:

zpool status

reported:

scan: resilvered 4.19T in 01:07:46 with 0 errors

Pool:

state: ONLINE

Configuration:

raidz1-0
  SSD1 ONLINE
  SSD2 ONLINE

Dashboard Still Broken

Even after migration completed:

  • Dashboard remained stuck

  • UI became unresponsive again

However:

  • Bitcoin worked

  • Nextcloud worked

  • Apps worked

This strongly suggested the problem was isolated to Umbrel itself.

Restarting Umbrel Service

Checked:

sudo systemctl status umbrel

Umbrel daemon healthy.

Restarted:

sudo systemctl restart umbrel

Dashboard returned.

Final Verification

ZFS

zpool status

Result:

pool: umbrelos-d16679a1

state: ONLINE

scan: resilvered 4.19T in 01:07:46 with 0 errors

errors: No known data errors

SMART

sudo smartctl -a /dev/nvme0
sudo smartctl -a /dev/nvme1

Results:

Drive 1:

Critical Warning: 0x00
Temperature: 37°C
Percentage Used: 0%

Drive 2:

Critical Warning: 0x00
Temperature: 39°C
Percentage Used: 0%

Umbrel Dashboard

Storage Manager now shows:

16 TB Total Storage
8 TB Protected Storage
FailSafe Enabled

Remaining Concern

After reboot, the inotify values reverted:

cat /proc/sys/fs/inotify/max_user_watches
cat /proc/sys/fs/inotify/max_user_instances

returned:

122404
256

again.

Umbrel logs showed:

[files:watcher] Setting system inotify limits

which suggests Umbrel is overriding custom values.

The same error still appeared:

inotify_add_watch ... failed:
No space left on device

inside:

nextcloud/data/appdata_xxxxx/preview

My Theory

I no longer believe the second SSD was ever the root cause.

Instead I suspect:

  1. Umbrel Files is attempting to watch a massive Nextcloud preview tree.

  2. Inotify limits are being exceeded.

  3. This causes intermittent dashboard lockups.

  4. The FailSafe migration process amplified the problem because it repeatedly restarted indexing and file scanning.

Questions for Umbrel Team

  1. Why does FailSafe sometimes fail with:
dataset already exists

when no migration snapshot exists?

  1. Why does Umbrel override manually configured inotify limits?

  2. Is there a known issue with Umbrel Files indexing large Nextcloud preview caches?

  3. Could inotify exhaustion explain:

    • dashboard freezes

    • UI lockups

    • partial dashboard loading

  4. Is there a supported way to permanently increase:

fs.inotify.max_user_watches
fs.inotify.max_user_instances

on Umbrel Home?

Hopefully this helps someone else experiencing random dashboard lockups, FailSafe migration failures, or mysterious “No space left on device” errors despite having plenty of disk space available.