Umbrel 1.2 Beta 2 - Do not sync the blockchain - Help!

Support and Troubleshooting
1

I recently migrate from Pi4 to Umbrel Home 1.1.2 and everything was working fine than suddenly, my node stopped synchronizing the blockchain.

Logs bellow of bitcoin node and mempool, showing they cannot connect to peer, but I can ping the internet normally with for example sudo ping www.google.com

I upgraded to 1.2 Beta 2 thinking that could be a network problem, but didnt changed after the update.

Maybe some data corruption??? Please, help!

umbrel_bitcoin_2024-06-15_12-25.log (100.0 KB)
umbrel_mempool_2024-06-15_12-16.log (67.2 KB)
umbrel-1718465321284.log (5.5 MB)

2

I just updated as well but I had the Umbrel Home and updated. It seems you migrated from the RPi4 to the Umbrel Home, is that correct? If so, did you make sure that port 8333 is still open?

If not, off of the top of my head, I’d check storage space, see if there’s anything taking up high memory or CPU usage and in the event of data corruption you may need to reindex the blockchain.

I’d start with a system reboot and manually stop any other apps that you don’t currently need to see if that frees up some CPU or memory.

3

Yes, I migrate from a RPi4, but the it was working for 2 days than it stopped with no change at all.

Surely not a CPU, memory or space problem, maybe except for a possible corruption of some data.
umbrel_screen

My umbrel-bitcoin.conf has these lines:

Required to configure Tor control port properly

[main]
bind=0.0.0.0:8333
bind=10.21.21.8:8334=onion

But if I do a telnet 192.168.0.46 8333 that is my umbrel local ip that appears in the display, it says “connection refused”.

How do I open that port?

4

That depends on what router you have but usually you can just go to your router settings and then open it.

Check out this site.

5

@TheDecentralizedVet

My router is surely forwarding that port because I used the node for many months and it was working yesterday, after the migration.

I just discovery that the bitcoin docker keeps restarting… look this sequence of commands, they were sent immediately after one another.

umbrel@umbrel:~$ sudo docker ps | grep bitcoind
da5826addaa4   lncm/bitcoind:v27.0                      "bitcoind -port=8333…"     2 hours ago   Up 4 seconds                    8080/tcp, 18332-18333/tcp, 18443-18444/tcp, 28332-28333/tcp, 0.0.0.0:8332-8333->8332-8333/tcp, :::8332-8333->8332-8333/tcp            bitcoin_bitcoind_1

umbrel@umbrel:~$ sudo docker ps | grep bitcoind
da5826addaa4   lncm/bitcoind:v27.0                      "bitcoind -port=8333…"     2 hours ago   Up 5 seconds                    8080/tcp, 18332-18333/tcp, 18443-18444/tcp, 28332-28333/tcp, 0.0.0.0:8332-8333->8332-8333/tcp, :::8332-8333->8332-8333/tcp            bitcoin_bitcoind_1

umbrel@umbrel:~$ sudo docker ps | grep bitcoind
da5826addaa4   lncm/bitcoind:v27.0                      "bitcoind -port=8333…"     2 hours ago   Up 8 seconds                    8080/tcp, 18332-18333/tcp, 18443-18444/tcp, 28332-28333/tcp, 0.0.0.0:8332-8333->8332-8333/tcp, :::8332-8333->8332-8333/tcp            bitcoin_bitcoind_1

umbrel@umbrel:~$ sudo docker ps | grep bitcoind
da5826addaa4   lncm/bitcoind:v27.0                      "bitcoind -port=8333…"     2 hours ago   Restarting (139) Less than a second ago                                                                                                                                         bitcoin_bitcoind_1

umbrel@umbrel:~$ sudo docker ps | grep bitcoind
da5826addaa4   lncm/bitcoind:v27.0                      "bitcoind -port=8333…"     2 hours ago   Up 3 seconds                    8080/tcp, 18332-18333/tcp, 18443-18444/tcp, 28332-28333/tcp, 0.0.0.0:8332-8333->8332-8333/tcp, :::8332-8333->8332-8333/tcp            bitcoin_bitcoind_1

That is bitcoin node does not start because it keeps restarting, but what could be causing that???

6

The frequent restarting of your Bitcoin node’s Docker container suggests that there’s a persistent issue causing it to crash. To diagnose and resolve this, we need to examine the logs of the Docker container to understand what’s causing the restarts. Here’s how you can proceed:

  1. Check Docker Container Logs: Start by viewing the logs for the Bitcoin container to get more insight into why it’s restarting.
sudo docker logs bitcoin_bitcoind_1

Look for any error messages or warnings that could provide clues about the issue.

  1. Resource Constraints: Ensure that your device has enough resources (CPU, RAM, and disk space) to run the Bitcoin node. If resources are constrained, the node might crash repeatedly.
free -h
df -h
  1. Check Disk Usage: Verify that you have sufficient disk space available. Bitcoin nodes require a significant amount of disk space.
df -h
  1. Configuration Issues: Ensure that there are no issues with your bitcoin.conf file. You might want to check if the configuration file was correctly migrated and if there are any syntax errors.
sudo nano /home/umbrel/umbrel/bitcoin/bitcoin.conf

Compare your configuration with the Bitcoin documentation to ensure it’s correctly set up.

  1. Reindex Blockchain (if needed): If you suspect data corruption, you might need to reindex the blockchain. Stop the Docker container and start it with the -reindex flag.
sudo docker stop bitcoin_bitcoind_1
sudo docker run -it --rm -v /path/to/bitcoin/data:/root/.bitcoin lncm/bitcoind:v27.0 bitcoind -reindex
  1. Inspect Docker Container: Inspect the Docker container for any additional information about its state.
sudo docker inspect bitcoin_bitcoind_1
  1. Restart Docker Service: Sometimes, restarting the Docker service can help resolve underlying issues.
sudo systemctl restart docker
  1. Update Docker Image: Make sure you are using the latest version of the Docker image for the Bitcoin node.
sudo docker pull lncm/bitcoind:v27.0
sudo docker-compose up -d
  1. Check System Logs: Check the system logs for any relevant messages that might indicate why the container is failing.
sudo journalctl -xe

By following these steps, you should be able to gather more information about the cause of the restarts and take appropriate action to resolve the issue. If you find specific error messages in the logs, those can be further investigated to identify the root cause.

1 Like
7

@TheDecentralizedVet

After checking the logs, looks like a data corruption was causing the docker to shutdown and restarting…

I follow the guide to set reindex=1 on bitcoin.conf than it started reindexing, but then after it ran a few percentage, it started again… then I look on the logs and found this:

2024-06-16T00:18:20Z *** Corrupt block found indicating potential hardware failure; shutting down
2024-06-16T00:18:20Z Error: A fatal internal error occurred, see debug.log for details
2024-06-16T00:18:20Z ERROR: ConnectTip: ConnectBlock 00000000000007f2c79652faa7d5052ade49ce4a423b848a0bf1ac698c417bd7 failed, bad-txnmrklroot, hashMerkleRoot mismatch
2024-06-16T00:18:20Z *** Failed to connect best block (bad-txnmrklroot, hashMerkleRoot mismatch)
2024-06-16T00:18:20Z Error: A fatal internal error occurred, see debug.log for details
Error: A fatal internal error occurred, see debug.log for details
Error: A fatal internal error occurred, see debug.log for details
2024-06-16T00:18:20Z txindex thread start
2024-06-16T00:18:20Z txindex is enabled at height 188459
2024-06-16T00:18:20Z txindex thread exit
2024-06-16T00:18:20Z basic block filter index thread start
2024-06-16T00:18:20Z basic block filter index is enabled at height 188459
2024-06-16T00:18:20Z basic block filter index thread exit
2024-06-16T00:18:20Z Loading 17 mempool transactions from disk...
2024-06-16T00:18:20Z initload thread exit
2024-06-16T00:18:20Z Shutdown: In progress...
2024-06-16T00:18:20Z addcon thread exit
2024-06-16T00:18:20Z opencon thread exit
2024-06-16T00:18:20Z net thread exit
2024-06-16T00:18:20Z msghand thread exit
2024-06-16T00:18:20Z DumpAnchors: Flush 2 outbound block-relay-only peer addresses to anchors.dat started
2024-06-16T00:18:20Z DumpAnchors: Flush 2 outbound block-relay-only peer addresses to anchors.dat completed (0.00s)
2024-06-16T00:18:20Z scheduler thread exit
2024-06-16T00:18:20Z Flushed fee estimates to fee_estimates.dat.
2024-06-16T00:18:25Z Shutdown: done
2024-06-16T00:18:26Z Warning: reindex=1 is set in the configuration file, which will significantly slow down startup. Consider removing or commenting out this option for better performance, unless there is currently a condition which makes rebuilding the indexes necessary

That is a hardware failure on the re-indexing…
So the reindexing probably will never end since there is some problem on my 2TB SSD.

Can you please give me directions of what are my options now ???

8

Have you tried uninstalling the Bitcoin node app and then reinstalling to see if it will redownload the blockchain?

I would also see if you can get any additional debug logs to get more info. To view the debug.log file on UmbrelOS:

  1. Connect to Umbrel via SSH:
  • Use terminal to connect to your Umbrel node and replace umbrel.local with your Umbrel’s IP address:
ssh umbrel@umbrel.local
  • Enter your password when prompted.
  1. Navigate to Bitcoin Core’s Data Directory:
  • Move to the Bitcoin data directory where debug.log is stored:
cd /home/umbrel/umbrel/data/bitcoin
  1. View the debug.log File:
  • Use cat to display the contents of debug.log:
cat debug.log
9

@TheDecentralizedVet

At the moment it is trying to re-indexing again (210,000 blocks of 848,099)… I am watching to see if it will fail again and restart. I removed the reindex=1 of bitcoin.conf to see if will finish continue from there instead of begin the reindexing from zero again.

If it fails, I think I cannot trust this ssd to this task and will have to replace it.

My debug.log is under ~/umbrel/app-data/bitcoin/data/bitcoin/ right… it has 59MB, too long and keep adding new lines.

Can you tell me if it is possible to turn my old RPi4 running umbrel 0.54? I have not cleaned it either the sdcard neither the ssd on it. My lightning node has made only a few transactions on the new Umbrel Home. What can happen if I shutdown Umbrel Home e turn on my old RPi4?

10

I was going to suggest that as well but if not we can possibly ask (@)smolgrrr to give some input on this as well as your last question as I’m not sure what can happen.

11

@TheDecentralizedVet

It just crashed again, here is the lines on debug.log

2024-06-16T01:22:07Z UpdateTip: new best=00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048 height=1 version=0x00000001 log2_work=33.000022 tx=2 date='2009-01-09T02:54:25Z' progress=0.000000 cache=0.3MiB(1txo)
2024-06-16T01:22:07Z Fatal LevelDB error: Corruption: block checksum mismatch: /data/.bitcoin/indexes/txindex/002400.ldb
2024-06-16T01:22:07Z You can use -debug=leveldb to get more complete diagnostic messages
2024-06-16T01:22:07Z 

************************
EXCEPTION: 15dbwrapper_error       
Fatal LevelDB error: Corruption: block checksum mismatch: /data/.bitcoin/indexes/txindex/002400.ldb       
bitcoin in scheduler       

2024-06-16T01:22:07Z UpdateTip: new best=000000006a625f06636b8bb6ac7b960a8d03705d1ace08b1a19da3fdcc99ddbd height=2 version=0x00000001 log2_work=33.584985 tx=3 date='2009-01-09T02:55:44Z' progress=0.000000 cache=0.3MiB(2txo)

Then it starting to re-synching / re-indexing from 0% again.

I think I have no other option but to shutdown this Umbrel Home and try to bring my RPi4 back alive. It would be awesome @smolgrrr to give some input on this maneuver.

Could be interpreted by the network as a hack? … since a few transactions ocurred on the new Umbrel after the migration, or the maybe the RPi4 is supposed to resync and update these transactions from the lightning chain?

Or maybe I should consider the option of re-download the entire chain.
Do you know the commands to force this?

12

Or maybe I should consider the option of re-download the entire chain.
Do you know the commands to force this?

Resyncing the blockchain is always an option if all else fails. It seems like there could be something wrong withing the chain locally.

Please note, these steps will remove all the bitcoin blockchain data from your umbrel and you will need to redownload the entire chain again.

Steps:

  1. Stop the Bitcoin Core app, including any associating apps that use the blockchain (e.g. electrs, mempool, etc)
  2. Open terminal or dial into your umbrel box as you did above
  3. Go to /home/umbrel/umbrel/app-data/bitcoin/data/bitcoin folder and delete the blocks, chainstate, and indexes folder.
  4. Go back to the umbrel webpage and start the Bitcoin Core app
  5. Once synced, start your other apps.

If after all of this it still doesn’t work, consider removing the core app, delete everything in the bitcoin core app data folder (/home/umbrel/umbrel/app-data/bitcoin/), and reinstall the app and start from there. This will remove any backend config data including config files, host and banlists for your local instance.

Another option you can do is to seed your blockchain from another computer or device. See my other post here for details.

13

After 2 long days battling this, I could make my umbrel home live again.

Reindex did not work and deleting folders blocks, chainstate and indexes also did not work.

What worked for me was to delete the entire folder bitcoin under /home/umbrel/umbrel/app-data/bitcoin/data/ and copy it back entire from another node that was 100% synchronized.

Here are the steps in case someone need:

  1. On new Umbrel Home delete the folder bitcoin under /home/umbrel/umbrel/app-data/bitcoin/data/

  2. I put my old RPi4 to run, stopped all apps except bitcoin node, using

docker stop [CONTAINER ID]

I waited it to sync 100%, then stopped all it with:

~umbrel/scripts/stop

  1. Back to the new Umbrel home, I stopped all apps including bitcoin node by right clicking the app icon and STOP, then used the command scp to download the bitcoin folder:

scp -r umbrel@[RPi4 Node Local IP]:/home/umbrel/umbrel/app-data/bitcoin/data/bitcoin ~/umbrel/app-data/bitcoin/data

  1. After some hours copying all files, I just right click the bitcoin and START. It just have to sync a few new blocks, no reindex, no huge download. Then I started the lightning node and LND and my channels were there.

@TheDecentralizedVet and @Cali_Fornia: Thank you very much for the support, it helped a lot.

Now I need to figure out how to make the system more robust because the way it is currently, it is too fragile… A blink of a eye, a file corrupt and you lose all.

2 Likes
14

This is amazing to hear and glad I could be of some help! This is what makes Umbrel great. It’s the community and support. :raised_hands:t4::pray:t4:

1 Like