If the disk filled up earlier (e.g., during a sync attempt before this log snippet), it might’ve corrupted the database, and now bitcoind can’t proceed past loading it. To test this, you’d need to check disk usage with df -h on the /data/.bitcoin directory. If it’s near capacity (say, <1 GB free), that’s your smoking gun.
What’s likely happening: Bitcoin Core is crashing due to a corrupted txindex database, possibly from a past disk space exhaustion or an interrupted write. It’s not syncing further because it can’t get past the startup phase. To fix this, you could:
- Check disk space: Run df -h to see if /data/.bitcoin is full.
- Disable txindex temporarily: Edit bitcoin.conf, set txindex=0, and restart. This skips loading the corrupt index, letting you assess if the core blockchain data is intact.
- Rebuild txindex: If space is available, delete /data/.bitcoin/indexes/txindex, set txindex=1, and run with -reindex. This rebuilds the index but needs ~150 GB free and takes hours or days.
- Add storage: If the disk is full, attach more storage, move /data/.bitcoin, and symlink it back.
FYI:
Let’s break down these two options in more detail so you can understand exactly what they do, why they might help, and what to expect when you try them. Both are strategies to get your Bitcoin node running again after the txindex corruption issue we identified, and they tackle it from different angles.
Option 1: Disable txindex Temporarily
What it does:
The txindex setting in Bitcoin Core tells the node to maintain a full transaction index—a database that lets you look up any transaction by its ID (txid), not just those in your wallet or unspent outputs. It’s optional and off by default, but your config has txindex=1, meaning it’s enabled. The corruption is in this index (/data/.bitcoin/indexes/txindex/MANIFEST-000002), not the core blockchain data (blocks and chainstate). By setting txindex=0 in bitcoin.conf, you’re telling Bitcoin Core to skip loading or using the transaction index entirely. This bypasses the corrupt files, letting the node start up using only the essential blockchain data.
How to do it:
- Open your bitcoin.conf file, likely at /data/.bitcoin/bitcoin.conf.
- Find the line txindex=1 and change it to txindex=0. If it’s not there, add txindex=0.
- Save the file and restart Bitcoin Core (e.g., bitcoin-cli stop then bitcoind, or reboot your Umbrel node).
Why it helps:
- The crash happens when Bitcoin Core tries to open the corrupted txindex database during startup (AppInit). Disabling it means the node won’t even attempt to read those files, avoiding the fatal error.
- It lets you test if the core blockchain data—stored in /data/.bitcoin/blocks (block files) and /data/.bitcoin/chainstate (UTXO set)—is still intact. The log showed it loaded up to height 788,758 and verified the last 6 blocks fine, so the blockchain itself might be okay.
What to expect:
- If successful, Bitcoin Core will start and sync from height 788,758 to the current block (around 840,000+ as of April 2025). You’ll see it downloading new blocks in the logs (e.g., “UpdateTip: new best=…”).
- Downside: You lose the ability to query historical transactions by txid via RPC calls (e.g., getrawtransaction) unless they’re in your wallet. For a basic node syncing the chain, this might not matter.
- It’s quick—no extra disk space or long reprocessing needed. Syncing the ~50,000 blocks since May 2023 could take hours to a day, depending on your network and hardware (e.g., Raspberry Pi 4).
When to use it:
- If you just want the node running ASAP and don’t need txindex right now.
- To confirm the blockchain data is safe before deciding on a bigger fix.
Option 2: Rebuild txindex
What it does:
This option fixes the corrupted transaction index by deleting it and rebuilding it from scratch. The -reindex flag tells Bitcoin Core to reprocess all blocks it already has (up to 788,758 in your case), reconstructing the txindex database along the way. It keeps txindex=1 functionality, so you’ll still have full transaction lookup capability once it’s done.
How to do it:
- Stop Bitcoin Core (bitcoin-cli stop or stop the Umbrel service).
- Delete the corrupted txindex directory: rm -r /data/.bitcoin/indexes/txindex. This removes the broken files but leaves blocks and chainstate untouched.
- Ensure txindex=1 is still in bitcoin.conf (it already is in your setup).
- Start Bitcoin Core with the -reindex flag: bitcoind -reindex. On Umbrel, you might need to edit the startup script or use a manual command.
- Let it run—it’ll recreate /data/.bitcoin/indexes/txindex as it reindexes.
Why it helps:
- The corruption is isolated to the txindex database. Deleting it and reindexing regenerates it from the blockchain data, which seems intact based on the log (successful verification of blocks up to 788,758).
- It restores full functionality without losing the ability to query any transaction, unlike disabling txindex.
What to expect:
- Time: Reindexing 788,758 blocks (about 550 GB of blockchain data) is slow. On a Raspberry Pi 4 with an SSD, expect 1-3 days; on an SD card, maybe a week. It’s CPU- and disk-intensive because it reprocesses every transaction.
- Space: The txindex database grows to ~100-150 GB for 788,758 blocks (roughly 20-25% of the blockchain size). You need this much free space, plus room for new blocks as it syncs to 2025 (another 50-70 GB). Total free space needed: ~150-200 GB minimum.
- After reindexing, it’ll sync the remaining blocks to the current height, adding a few more hours or a day.
- Logs will show “Reindexing block file blkXXXXX.dat” and progress updates. Once done, the “Fatal LevelDB error” should disappear.
When to use it:
- If you need txindex=1 for your use case (e.g., running a block explorer or querying old transactions).
- If you have enough disk space and can wait out the process.
Key Differences and Considerations
- Speed: Disabling txindex is fast—you’re back online in minutes, just syncing new blocks. Rebuilding takes days.
- Space: Disabling requires no extra space beyond what’s needed for new blocks (~50-70 GB to catch up). Rebuilding needs 150-200 GB free upfront.
- Functionality: Disabling sacrifices txindex features; rebuilding keeps them.
- Risk: Disabling is low-risk—just a config tweak. Rebuilding assumes the blockchain data is uncorrupted; if it’s not (e.g., blocks or chainstate are also damaged), you might need a full resync from scratch, which is even slower.
Disk Issues
Checking the health of your SSD is a great idea, especially since you’re troubleshooting potential disk-related issues with your Bitcoin node. A failing or worn-out SSD could explain the txindex corruption, either through write errors or data loss. Since you’re likely running this on an Umbrel node (probably a Raspberry Pi), I’ll focus on practical methods using Linux tools available in that environment. Here’s how you can assess your SSD’s health:
1. Use smartctl (SMART Data)
The most reliable way to check SSD health is with the smartctl tool, part of the smartmontools package. SMART (Self-Monitoring, Analysis, and Reporting Technology) is built into most SSDs and provides detailed stats like wear level, bad sectors, and error rates.
Steps:
-
Install smartmontools:
Umbrel’s OS might not have it pre-installed. Connect to your node via SSH (e.g., ssh umbrel@umbrel.local) and run:
sudo apt update
sudo apt install smartmontools
If Umbrel’s package manager is locked down, you might need to check their docs or forums for enabling extra tools.
-
Identify your SSD:
Find the device name of your SSD. Run:
lsblk
Look for your storage device—on a Pi, it’s often /dev/sda (USB SSD) or /dev/mmcblk0 (SD card if no SSD). The Bitcoin data at /data/.bitcoin is likely mounted there. For example:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 960G 0 disk
└─sda1 8:1 0 960G 0 part /data
-
Run a SMART health check:
Use smartctl on the device (replace /dev/sda with your device):
sudo smartctl -a /dev/sda
This dumps a full report. Key sections to check:
- SMART overall-health self-assessment: Look for “PASSED” or “FAIL”. Example:
SMART overall-health self-assessment test result: PASSED
If it says “FAIL”, your SSD is in bad shape.
- Wear_Leveling_Count or Percentage Used: Shows how much life is left. For SSDs, this might appear as:
Wear_Leveling_Count 0x0032 098 098 000 Old_age Always - 2
Here, 98 means 2% worn (100 is new, 0 is end-of-life). Some drives report “Percentage Used” (e.g., 10% used = 90% life left).
- Reallocated_Sector_Ct: Bad sectors moved to spares. Non-zero values (e.g., 5) indicate wear or issues.
Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
- Uncorrectable_Error_Count: Errors that couldn’t be fixed. Non-zero is a red flag.
-
Run a short self-test (optional):
To actively test the SSD:
sudo smartctl -t short /dev/sda
Wait a few minutes, then check results:
sudo smartctl -l selftest /dev/sda
Look for “Completed without error” vs. failures.
What to look for:
- “PASSED” health, low wear (e.g., <50% used), and zero or few errors = healthy SSD.
- “FAIL”, high wear (e.g., >90% used), or many errors = dying SSD, likely causing your corruption.
2. Check Filesystem Integrity
If the SSD has bad blocks or corruption, the filesystem might show issues. This doesn’t directly measure SSD health but can reveal symptoms of failure.
Steps:
-
Identify the filesystem:
Run:
df -h
Find the mount point for /data (e.g., /dev/sda1). Note the filesystem type (e.g., ext4).
-
Run a filesystem check:
Unmount the drive first (stop Bitcoin Core with bitcoin-cli stop):
sudo umount /data
Then check it (replace /dev/sda1 with your device):
sudo fsck /dev/sda1
If it’s ext4, use:
sudo fsck.ext4 -f /dev/sda1
-f forces a check even if it seems clean.- Answer “yes” to repair prompts if errors are found.
-
Remount:
sudo mount /data
What to look for:
- “Filesystem is clean” = no issues.
- Errors like “bad blocks” or “inodes corrupted” = SSD might be failing, especially if recurring after repairs.
3. Check I/O Errors in System Logs
The Linux kernel logs disk errors, which can hint at SSD health.
Steps:
-
View dmesg logs:
dmesg | grep -i error
Look for lines mentioning your device (e.g., sda):
[1234.567] sda: error: uncorrectable read error at sector 123456
These indicate hardware issues.
-
Check system logs:
sudo cat /var/log/syslog | grep -i disk
Or for recent errors:
sudo journalctl -xe | grep -i disk
Watch for I/O errors, timeouts, or “medium error”.
What to look for:
- No errors = likely healthy.
- Repeated I/O or read/write errors = SSD degradation.
4. Monitor Disk Usage (Tie to Your Bitcoin Issue)
Since you asked about disk space exhaustion, combine health checks with capacity:
df -h /data
Example output:
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 960G 900G 60G 94% /data
- If
Avail is near zero, space exhaustion could’ve corrupted txindex.
- Even if not full now, past exhaustion (e.g., before a reboot) could’ve caused the issue.
Interpreting Results
Practical Notes for Umbrel
- Access: SSH into your node (default user/password might be
umbrel/umbrel unless changed).
- Stopping services: Use
docker stop bitcoin_bitcoind_1 if bitcoin-cli isn’t accessible via Umbrel’s setup.
- Backup: Before major changes (e.g.,
fsck repairs), back up /data/.bitcoin if possible—corruption could worsen.
- SSD type: If it’s a cheap USB SSD or SD card, wear is common on Bitcoin nodes due to constant writes. High-endurance drives (e.g., Samsung T7) last longer.
Run smartctl -a first—it’s the gold standard. Let me know the output or if you hit snags installing it, and I’ll guide you further!