Hello everyone. I have recently installed an umbrel full node and have synced Bitcoin Node 100%. Everything seemed to be going well but I have had it disconnected for a few days and when I restart it again it does not start and remains permanently in the “Starting…” state. I don’t know what it is and how I can fix it. I would appreciate advice and help from you. Greetings
Hey @Bosco can you navigate to System > Troubleshoot > Start and generate a debug log file? You can share the file here or you can paste it to www.pastebin.com and share a link to review easily
If for any reason it gives you an error when attempting to generate the log you can reference How To SSH and the Generating Troubleshooting Logs steps in the FAQ here by SSHing and using the command listed there: sudo ~/umbrel/scripts/debug to output the log and you can copy it from there to pastebin as well
Likely the log will give us a few indications of what might be the cause of the problem (maybe a corrupt chainstate or peers.dat error) and we can go from there
I had a similar issue when I rebooted my Umbrel one time.
For me it was a cache/cookie issue and every time I tried to access the login page via my bookmark it would just say “Starting”.
I was able to access the Umbrel successfully when trying a different browser, and then clearing my cookies and cache on the main browser, allowed it to let me login again.
Hi I have the same problem on my initial Bitcoin node sync, it gets to 80% and then stays on starting with no information about peers etc
Logs attached, any help much appreciated.
umbrel_bitcoin_2025-02-13_09-56.log (72.9 KB)
I’ve looked at your log and, to start, you have a problem with the bitcoin-core database, as your log indicates:
bitcoin_bitcoind_1 | Error: Error reading from database, shutting down.
bitcoin_bitcoind_1 | 2025-02-13T09:55:45Z LevelDB read failure: Corruption: block checksum mismatch: /data/.bitcoin/chainstate/1007517.ldb
bitcoin_bitcoind_1 | 2025-02-13T09:55:45Z Fatal LevelDB error: Corruption: block checksum mismatch: /data/.bitcoin/chainstate/1007517.ldb
bitcoin_bitcoind_1 | 2025-02-13T09:55:45Z You can use -debug=leveldb to get more complete diagnostic messages
bitcoin_bitcoind_1 | 2025-02-13T09:55:45Z [error] Error reading from database, shutting down.
bitcoin_bitcoind_1 | 2025-02-13T09:55:45Z [error] Error reading from database: Fatal LevelDB error: Corruption: block checksum mismatch: /data/.bitcoin/chainstate/1007517.ldb
To help you understand better, your bitcoin-core starts and when it gets this error, it restarts the “starting correctly” process but it gets this error again and enters a loop from which it cannot get out.
Something has happened. Maybe a “bad shutdown or reboot” of the node but usually it has to do with hardware issues.
Bitcoin Core makes heavy use of CPU, RAM, and disk IO. Hardware defects might only become visible when running Bitcoin Core.
Another possible cause is that the user installs bitcoin-core and then immediately installs other apps without bitcoin-core having finished the process of synchronizing the blockchain completely. For example, he installs Electrs, block-explorer or mempool and wants to see everything working in the next second, straining the hardware to no avail because these apps will not work properly until the blockchain has been 100% synchronized but while they are taking resources away from the hardware.
Anyway, let’s get to the solution.
First, let’s try the most practical and least invasive option: reindexing the chainstate.
Process:
1.- Connect via SSH to your umbrelos.
2.- Edit the config file of your bitcoin node using the following command:
sudo nano /home/umbrel/umbrel/app-data/bitcoin/data/bitcoin/umbrel-bitcoin.conf
3.- The file to be edited will open and you add the following to the end of everything you have written there: reindex-chainstate=1
4.- Then save and close using Ctrl-X, Y. Restart the RPI and when it starts up again it will start indexing the chainstate. It will seem like it is downloading the blockchain again but in reality it is checking and indexing what you have downloaded.
5.- While bitcoin-core is doing this, open the file again (as you did in point 2) and make sure that the line you added ( reindex-chainstate=1 ) is no longer present, so that the next time you restart the node in the future it doesn’t redo the whole process.
In theory that command/line you added ( reindex-chainstate=1 ) should disappear automatically but it’s better to be sure that it did.
Let it finish the reindexing and when the process is finished it will continue to sync what’s missing from the blockchain.
Well, let us know if it worked.
Thank you for your reply, you are right in what you’re saying. I had Core installed and fully synced, then uninstalled and tried to move to Knots to do my part to filter out the spam. That’s when this issue started on Knots. So uninstalled and moved back to Core but here we are again.
I’ve done as you’ve said and can see it’s reindexing now from genesis, the reindex-chainstate=1 string has gone as well from the .conf file.
I’ll keep you posted, thanks for taking the time to have a look at this for me.
@jofraprez it’s started to get very slow around block number 850,000 - I’m very nearly fully synced. It’s now just sitting at “starting” again. Attached updated logs.
umbrel_bitcoin_2025-02-15_07-43.log (159.1 KB)
Without a doubt, one of the best tools to test your hardware is bitcoin-core.
Well, it’s back to the same loop. This last log shows the same error: corrupted chainstate.
bitcoin_bitcoind_1 | 2025-02-15T07:41:30Z LevelDB read failure: Corruption: block checksum mismatch: /data/.bitcoin/chainstate/1061103.ldb
bitcoin_bitcoind_1 | 2025-02-15T07:41:30Z Fatal LevelDB error: Corruption: block checksum mismatch: /data/.bitcoin/chainstate/1061103.ldb
bitcoin_bitcoind_1 | 2025-02-15T07:41:30Z You can use -debug=leveldb to get more complete diagnostic messages
bitcoin_bitcoind_1 | 2025-02-15T07:41:30Z [error] Error reading from database, shutting down.
bitcoin_bitcoind_1 | 2025-02-15T07:41:30Z [error] Error reading from database: Fatal LevelDB error: Corruption: block checksum mismatch: /data/.bitcoin/chainstate/1061103.ldb
bitcoin_bitcoind_1 | Error: Error reading from database, shutting down.
There is a slight difference. In your first log the corrupted file was 1007517.ldb. If you notice, now it has gone beyond that file but has found another one which is 1061103.ldb.
Now, several questions arise. One of them is how many more chainstate files are corrupted?
Here’s another one: Is it worth all this time, work, and effort to try to save the downloaded blockchain?
As I explained to you previously, this corruption can be due to hardware issues, such as overheating or disk problems, or software bugs… and many times caused by the user themselves not knowing what they are doing.
We have several options to continue trying to save your copy of the blockchain.
-
(We have already tried this one) Restart with -reindex-chainstate: Restart your Bitcoin Core node with the -reindex-chainstate command. This will reindex the chainstate database without re-downloading the entire blockchain.
-
Check Disk Health: Run disk diagnostics to ensure there are no hardware issues with your storage device. Tools like smartctl or CrystalDiskInfo can help check the health of your SSD or HDD.
-
Remove Corrupted Files: If the corruption is isolated to specific files, you might be able to remove the corrupted LevelDB files and restart Bitcoin Core. However, this approach is risky and should be done with caution.
-
Reindex or Re-download Blockchain: If the corruption is widespread, you may need to reindex the entire blockchain or download it again. This is a more time-consuming process but ensures a clean database.
I don’t know a lot of information about your context, such as the hardware you use, the state it’s in, if you have more apps installed on your umbrel, etc. For this reason, I don’t feel capable enough to tell you what you have to do next.
But I’m going to give you a guide based on my personal situation and my context as if I were having this problem. These are the options I would consider if I had these problems…
1.- I would delete all the content of the /data/.bitcoin/chainstate/ folder and reindex the blockchain again. This implies that you would have to wait while it reindexes everything from scratch. Same as the process you’ve been doing but with the difference that, it is supposed to create the chainstate from scratch. This carries an implicit risk that it won’t succeed because there may be problems with the disk, processor overheating, etc.
2.- I would delete the entire blockchain and download it again, with the assurance that my copy does not contain any possible errors, but it implies the cost of waiting several days for the blockchain to synchronize. Here I have an advantage because one of my nodes saves a copy of the blockchain in real time. In my case, I only have to copy the updated copy of the blockchain from one node to another. Something that will take me 3 to 4 hours and then whatever it takes for the node to index the new copy. On average, in less than 1 day everything is back to normal as if nothing had happened.
Well, these are the most practical options I see. It is up to you to decide which path you want to take and there is no problem. I will help you as much as possible to try to achieve your goal.
It would of been helpful to explain that I’m on a RPi5 8GB with a NVME 2TB drive attached to a Pimoroni hat. Like I say, I have had a fully synced node before so when I’ve uninstalled Core, downloaded the chain via Knots and gone back to Core would this imply that maybe not all of the Bitcoin files were deleted? Ideally I’d be on Knots so maybe I stop Core - delete all the files and folders that you’ve described above and start again on Knots?
Since your first message about reindexing I right-clicked and stopped all other apps (Immich, Navidrome, Plex & Transmission) before going any further.
Yes, this clarifies the situation more.
Since you have already done a series of steps going from bitcoin-core to Knots, which is where the problem started, there are many “unknowns” as to where the problem may be.
Then, I assume that you do not want to “risk” all other apps (Immich, Navidrome, Plex & Transmission).
And if you have already decided that what you want to use is Knots… Well, why waste time trying to solve the problem of your current bitcoin-core?
Am I right? Well, now you clarify if I have misinterpreted it.
Anyway, if what you want is Knots and make sure to start with a “clean” scenario because the current one, even if it is solved, has already presented problems… I would do the following.
Make sure to completely uninstall Bitcoin-Core and then install Knots and have it sync the blockchain from scratch and correctly.
If this is what you want, I’ll detail the process for you.
1.- Uninstall Bitcoin-core. Preferably, Stop all the apps you have installed so that your RPI runs more comfortably during the process.
2.- Check that there is no trace of anything related to bitcoin. Log in via SSH to your node and list the content of the installed apps using the following command
ls /home/umbrel/umbrel/app-data
It should only show you the folders of the installed apps. There should be nothing from bitcoin-core or Knots.
3.- (optional) If there are still traces, delete that folder with the traces. Right there, via SSH, enter the following command.
sudo rm -r /home/umbrel/umbrel/app-data/name of the folder in question (Example, Bitcoin)
It would look like this:
sudo rm -r /home/umbrel/umbrel/app-data/Bitcoin
4.- Once this is done, you disconnect via SSH and reboot your umbrelos.
5.- When you reboot, there should be no trace of bitcoin. You can safely install Knots and set it to sync.
Yes, I know it will take a few days until you have the entire blockchain. But this way you make sure you have something clean and problem-free. Also, you are not going to do it but your poor RPI and your drive that will suffer doing it. You can go watch a movie, a series, come back later, check that everything is going well, go back to do something else and so on until it finishes.
Everything described should end well. If not, then the problem is deeper. For example, part of your hardware would have been affected by everything you were doing and that caused the problem. But I have faith that it is not because your hardware is recent.
Anyway, you will tell me what you do…
Found something really strange… so I do the command ls /home/umbrel/umbrel/app-data and it shows the bitcoin folder, so do sudo rm -r /home/umbrel/umbrel/app-data/bitcoin to remove the Bitcoin folder. Do a restart of Umbrel and when running the ls command again it shows the Bitcoin folder again - have done this a few times now and it’s still there… any ideas?
Wow… this is getting more and more interesting…
ok. Let’s check that there are no more remnants…
Via SSH apply the command:
sudo nano /home/umbrel/umbrel/umbrel.yaml
This will open a window showing the boot configuration for the installed apps. I’ll show you an example with one of my nodes. You should see something like this:
version: 1.3.0
apps:
- bitcoin
- qbittorrent
- adguard-home
appRepositories:
- https://github.com/getumbrel/umbrel-apps.git
torEnabled: false
recentlyOpenedApps:
- qbittorrent
- bitcoin
- adguard-home
user:
name: XXXXXX
hashedPassword: XXXXXXXXXXXX
wallpaper: '8'
language: en
settings:
externalDns: true
widgets: []
If you notice, in this example there are two calls to Bitcoin. One in the “Apps” section and the other in “recentlyOpenedApps”.
You should not have any calls to Bitcoin because you are supposed to have uninstalled it. If there are calls to Bitcoin, edit the file by deleting the call. That is, deleting “Bitcoin”. Only what you have installed should appear in this file.
Delete the calls if there are any, Save (Ctrl+X, Y, Enter)
Delete the Bitcoin folder again as I indicated above and reboot.
I’ll wait for you to tell me what you discovered…
Done all of that, and the Bitcoin folder is still there under ls /home/umbrel/umbrel/app-data.
If it helps there is a single file called settings.yml in the Bitcoin folder - only about 3 bytes.
How strange!
ok. While I write you what you need to do, please confirm this for me.
When you checked the umbrel,yaml file via SSH with this:
sudo nano /home/umbrel/umbrel/umbrel.yaml
Were there any Bitcoin calls?
Yes in apps and the recently open bit.
OK. This is super weird. If you like, let’s do it in a more surgical way with the idea of finding the file that may be “stuck” preventing the total deletion of the folder.
Via SSH, list the contents of the bitcoin folder with the following command.
ls /home/umbrel/umbrel/app-data/bitcoin
I’ll show you one of my nodes as an example of what you’ll see:
umbrel@umbrel2:~$ ls /home/umbrel/umbrel/app-data/bitcoin/
data exports.sh scripts torrc umbrel-app.yml
docker-compose.yml hooks settings.yml torrc.template
In your case, it shouldn’t look like anything since you’ve already uninstalled it, but you say that you’ve confirmed that there’s at least one file there named: settings.yml
Take note of everything you find there (the names) and proceed to delete one by one what you find there. If there’s only one (settings.yml), use the following command:
rm /home/umbrel/umbrel/app-data/bitcoin/settings.yml
If there are more, the same command changing only the end; the name of the file.
Once you’ve deleted all the contents of the folder, check the contents again with the command at the beginning:
ls /home/umbrel/umbrel/app-data/bitcoin
Now, there should be nothing there. It should be empty.
If it is empty, proceed to delete the bitcoin folder with the command:
rm -r /home/umbrel/umbrel/app-data/bitcoin
Then, confirm that it has been deleted with the following command:
ls -l /home/umbrel/umbrel/app-data/
If it has been deleted, only the folders for the installed apps should appear.
If everything has gone well up to this point, check that there are no calls to bitcoin in the configuration for the installed apps. Only the names of the installed apps should appear. If there is something about bitcoin, delete it. Use the following command:
sudo nano /home/umbrel/umbrel/umbrel.yaml
If there is nothing, exit without saving.
Reboot your RPI and check everything:
sudo nano /home/umbrel/umbrel/umbrel.yaml
ls -l /home/umbrel/umbrel/app-data/
Let’s see if this works or if an error pops up somewhere in the process that tells us “who” is the culprit.
I have a fully synced node again!!
Thanks so much for your help - I think what you said about only running Core/Knots and closinf everything else is what helped. Even now Knots is running at ~2GB.