No connection to Umbrel

Support and Troubleshooting
1

Hi, I have to write another post.
A few month ago, my umbrel told me that the hdd is full, so I replaced the 1 TB with a 2 TB only to find out, that the first one wasn’t full at all (couple of 100 MB still free). I thought it was ok to update and futureproof anyway.

Yesterday, when logging in, umbrel tells me that the 2 TB SSD is full (which is ofc bs). I restart the device and cannot access it since. Curiously, I can “see” the device in my routers firmware, so its active on the network. It’s connected via wifi and neither umbrel.local nor the iIP lets me get to the login page.

Can anyone help here?

2

It’s hard to give you a more concrete answer because more information about your context is needed. Any detail, no matter how small, can help find a solution.

Anyway, and based on a lot of ignorance about your situation, I would connect the SSD to my computer and confirm the status of the disk.

Since I don’t know what you have installed, I’ll give you an example. Let’s say you have Jellyfin installed and some movies or series stored there on the SSD. I would delete some of those movies/series to get enough space to allow me to access them again via (umbrell.local)

I don’t want to tell you more because, as I said before, more information about your context is needed.

3

Thanks, i get it. Sorry for the little detail. I only run btc node, tor node and lightning node. the 2 tb are only for the btc blockchain.

Even with the ssd full (which it certainly isn’t), the system should at least boot up and let me log in, right? I’m not sure how I can gain browser access to the node by ssh’ing into it.

4

OK. I’m going to answer again from ignorance because I still don’t know all your context.
If you run out of disk space, what is happening to you now can easily happen. The system needs space to “write” the tasks it performs.

You say:

the 2 tb are only for the btc blockchain

I don’t fully understand this.

Anyway, the blockchain only weighs 729Gb at the time I write this.

If you have a 2Tb blockchain, it means you have a previous problem that caused the one you report. Something has happened and either you have duplicated the blockchain, or some blocks were corrupted and it has downloaded them again without deleting the failed blocks, etc.
It’s hard to say more without the logs available and with so little information.

5

Hi, I am very certain that the ssd is not full as the former 1tb ssd was shown as full by umbrel, but not by another pc that I connected it to. Umbrel is giving out a false error here. I copied the blockchain from the old ssd to the new using a guide from the forum and all worked fine with the new ssd showing the correct occupied space for several weeks. But there is no way to tackle this now, given that I can’t even gain access to the browser frontend.

6

Can you confirm if you have access via ssh?

7

yes, I just logged into umbrel@umbrel.local

8

Confirm the storage status of your disks. Via SSH run the following command.

df -h

It will show you all the disks and how much space they actually occupy.

If your disk is full it will be a matter of freeing up some space so you can access it normally again.

9

This is most irritating. The command sends back sda1 with size 1.9 T and used 1.9 T mounted on /mnt/data

But how can that be, when all I have in there is the Bitcoin Node, Ride the Lightning, Electrs and Tor Snowflake?

10

Stay calm. We’re on the right track.
Could you please paste the results here so I can see exactly what it shows you?

11

I was waiting for you to show me what you were seeing, but oh well…

Anyway, via ssh run the following command:

sudo du -h --max-depth=1 /home/umbrel/umbrel/app-data

This will show you all the folders of the apps you have installed and the size of each one. This way you can know exactly where the problem is and proceed from there to free up space.

12

I’m sorry for the late reply! Here’s the output:

510M    /home/umbrel/umbrel/app-data/snowflake
752G    /home/umbrel/umbrel/app-data/bitcoin
50G     /home/umbrel/umbrel/app-data/electrs
425M    /home/umbrel/umbrel/app-data/lightning
14M     /home/umbrel/umbrel/app-data/ride-the-lightning
802G    /home/umbrel/umbrel/app-data
13

This just got more interesting.
The output you get is correct. You don’t have the problem there. The folder sizes shown are the correct ones.
You have the problem in one or more other folders or umbrelos has duplicated the installation.

Please perform the following tests via ssh by running the following commands and paste each result here to see if we can find the location that is eating up storage.

test1:
df -h

test2:
sudo du -h --max-depth=1 /home/umbrel/umbrel

test3:
sudo du -h --max-depth=1 /home/umbrel

test4:
sudo du -h --max-depth=1 /home

14

Okay, here we go:
Test 1

Filesystem      Size  Used Avail Use% Mounted on
overlay         4.2G  429M  3.6G  11% /
tmp             1.9G  1.2M  1.9G   1% /run
/dev/mmcblk0p7  4.2G  429M  3.6G  11% /run/rugpi/mounts/data
/dev/mmcblk0p5  3.3G  2.9G  230M  93% /run/rugpi/mounts/system
/dev/mmcblk0p1  256M  720K  256M   1% /run/rugpi/mounts/config
/dev/mmcblk0p2  128M   89M   40M  70% /boot
devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sda1       1.9T  1.9T     0 100% /mnt/data
tmpfs           380M     0  380M   0% /run/user/1000

Test 2

20K     /home/umbrel/umbrel/db
404K    /home/umbrel/umbrel/scripts
8.0K    /home/umbrel/umbrel/nginx
216K    /home/umbrel/umbrel/statuses
5.0M    /home/umbrel/umbrel/.backup
108K    /home/umbrel/umbrel/events
802G    /home/umbrel/umbrel/app-data
12K     /home/umbrel/umbrel/jwt
16K     /home/umbrel/umbrel/.github
28M     /home/umbrel/umbrel/logs
15M     /home/umbrel/umbrel/packages
37M     /home/umbrel/umbrel/bin
1.3M    /home/umbrel/umbrel/tor
1.3M    /home/umbrel/umbrel/deps
40M     /home/umbrel/umbrel/app-stores
37M     /home/umbrel/umbrel/repos
20K     /home/umbrel/umbrel/templates
16K     /home/umbrel/umbrel/build
8.0K    /home/umbrel/umbrel/secrets
802G    /home/umbrel/umbrel

Test 3

802G    /home/umbrel/umbrel
802G    /home/umbrel

Test 4

802G    /home/umbrel
802G    /home

Thanks again for your help, by the way!!!

15

Okay, I’ll explain in detail what I see.

Test 1 shows us your SD and your disk. Your SD is all the partitions shown (They are correct) and your disk ( /dev/sda1 ) shows us that it has a capacity of 1.9Tb and that it is 100% occupied. Full, it can’t store any more. For that reason, the rest of the tests.

Test 2 shows us that everything is fine including the folder /home/umbrel/umbrel/app-data where each app is located and where everything related to each app is stored. In your case and as an example, there you have the bitcoin-core folder and the blockchain stored there. The data is correct and consistent. The folder /home/umbrel/umbrel/app-data weighs 802Gb which is correct at this time.

Test 3 confirms that the previous folder has the corresponding storage and is correct: 802Gb.

Test 4 confirms all of the above. Corresponding storage is correct: 802Gb.

In conclusion, your system is seeing and storing what it should correctly.

With the information available, everything leads me to think that the problem is with your disk. I remember you mentioned the following:

I copied the blockchain from the old ssd to the new one using a guide from the forum and everything worked fine with the new ssd showing the correct occupied space for several weeks.

I think that, maybe you did not copy the blockchain correctly and/or the system has duplicated the content on your disk making it full and without being able to detect it.

Well, let’s resolve that doubt. Let’s go directly and strictly to your disk and confirm. Via SSH execute the following commands and paste the results.

Test A:
sudo du -h --max-depth=1 /mnt/data

Test B:
sudo du -h --max-depth=1 /mnt

I’ll tell you right away that if the results are correct and match the results of the previous tests, then all that remains is to think that your disk is a fake disk or is damaged.

But, well… let’s see what the test shows…

16

What is most irritating to me is that the setup worked for about 2 to 3 month without any problem, giving me the correct size of the SSD and the occupied space (> 1 TB) inside umbrel settings.
anyway…

Test A

802G    /mnt/data/umbrel
4.0G    /mnt/data/swap
16K     /mnt/data/lost+found

Test B

1.9T    /mnt/data
17

Well, things and everything in life usually go well… until they start to go wrong.

Forget the irritating… the incredible mystery we are wrapped in. Let me explain.

umbrel mounts your disk on the /mnt/ partition with the “name” data.

In test 2 we see that the disk is mounted correctly and has 1.9Tb occupied. It is full because we know that its maximum capacity is 1.9Tb, according to what you have said.

In test 1 we see the content of your disk (data) and that it has only 3 folders (umbrel, swap and lost+found) that weigh in total: 802 Gb. Inside your disk there is nothing else, no other files, only those 3 folders.

All the tests we have done give the same results. umbrel only occupies 802 Gb which is correct. But your disk is full, occupying 1.9 Tb (100% of its capacity)

Where is that more than 1 Gb that is being occupied and by what or by whom?

Personally, I would turn off the node, disconnect the disk and connect it to a PC/laptop. When it is detected, look at the disk properties to see its capacity and how much it is occupied. Even browse within the disk. Maybe there is a file that umbrel does not detect and is occupying the lost 1 Gb.

I would even go a little further. I would check the structure of the disk with a program like gparted to be sure of the capacity and see if we can find that lost 1 Gb.

If you have any questions, ask before proceeding.

18

What happened in the end?
I’m curious to know if it was some file/folder that Umbrelos can’t see or if it was your disk…

19

Let’s hope that it’s not the end…
I just booted up foxclone and the readings in there match what umbrel told us via ssh.
I upload 2 pictures I took with my phone (since it was a live boot of foxclone). One is from gparted, showing 1 full partition, the other one is from the disc space analyzer.

I am not sure how to proceed from here. Do you have any suggestions? I greatly appreciate your help!

IMG20250311165306
IMG202503111653061920×856 110 KB

IMG20250311165024
IMG202503111650241920×856 69.1 KB

20

Well… we are seeing the same thing as with all the tests done on umbrelos.

gparted shows you that your 1.86 Tb (2Gb) disk is full.

Foxclone correctly shows the content of umbrelos and it is the correct one.

We are still with more than 1Gb lost. For me, it is your disk. What could have happened?

Common Causes:

Hard Drive Driver Not Updated: In this case, I don’t think so.

Incomplete Formatting: A format that is interrupted or not completed correctly can cause the disk to show an incorrect capacity. This is possible.

Incorrect Partition Table: If the hard drive has been initialized with an incorrect partition table, especially if it is larger than 2TB and has been initialized as MBR, the space that exceeds 2TB will appear as “Not allocated”. This could also be the case.

Bad Sector Errors: Too many bad sectors can cause the disk to not be able to read or write properly and show the wrong capacity. This could also be the case.

Hidden Files and Viruses: Hidden files or a virus attack can take up space without you noticing, which can cause the disk to show the wrong capacity. This could also be the case.

Fake Disk: This could also be the case.

In anticipation of your questions, I’ll answer what I would do.

  • I would backup the app-tada folder ( /home/umbrel/umbrel/app-data ) which is where all your apps are installed and stored.

  • I would format the disk again in etx4, GPT Partition Table.

  • I would reinstall umbrel with the most recent version.

  • I would install apps one by one. Example, bitcoin core. I install, stop, transfer blockchain from the existing copy and let it index until it is syncing correctly. Once this is confirmed, I go to the next app and the same thing.

Yeah, I know. It’s a hell of a job. Well, it’s not that bad either but we like to complain for fun. :smiley:

The other option. Change to a disk that you know is safe and start from scratch… and reinstall all your apps from your backup.

And I can’t think of anything else…