Umbrel unexpectedly stops working after the last update

I’ve had Umbrel for almost three years on the same HW and so far it has run without any problems. After the last update, however, Umbrel stops working and I can’t connect to it via the browser, nor can I ping it via the console or connect to it via SSH. I don’t want to link it to the update though, it’s possible it’s unrelated, but I’ve been having this problem ever since.

I tried reflashing the SD card yesterday but it didn’t help. The scenario is always the same:

  • Umbrel boots up
  • It starts to sync missing blocks, because it always reverts to the state of about 2 days ago after this failure and it has no blocks saved from the last time
  • It runs for a few hours and stops working
  • I can usually get it working without having to hard disconnect it from power. I don’t know how I get it to work, but when I try to connect to it in various ways, it sometimes wakes up and starts working. It’s completely unavailable for a while, then it’s possible to ping it via IP for a while, but it doesn’t respond via browser and umbrel.local address.
  • Then it runs again for a few hours and the scenario repeats.

I’ve looked at the logs and there is a strange transition, with the last logs being Jun 16 and then immediately followed by today Aug 29, as if the logs have been lost as well. I also see a lot of errors in the logs, but I don’t know how to interpret them.

Can someone help me solve this problem please?

Logs attached, just need to remove the last *.log extension and then unzip.

umbrel-1724930871013.log.gz.log (379.5 KB)

It just happened again. The log again shows when Umbrel rebooted itself. Before the reboot a number of errors are visible. When I accessed it now, it was no longer inaccessible, but when I refreshed the homepage, I saw that it again didn’t have all the blocks and showed that it was 99.88% synchronized and synchronizing blocks from the last 2 days or so.

Logs attached, just need to remove the last *.log extension and then unzip.

umbrel-1724949104366.log.gz.log (607.4 KB)

I tried to restart Umbrel after a week, but it crashed again. Despite the reboot I got this in the log. Can someone please help? My Umbrel has been down for a week now.

Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=info msg="State transition: check-wait [Idle] -> inventory-update [Sync]"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=warning msg="Returning artifact name from /etc/mender/artifact_info file. This is a fallback, in case the information can not be retrieved from the database, and is only expected when an update has never been installed before."
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=info msg="Device unauthorized; attempting reauthorization"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=error msg="Failed to authorize with \"\": failed to build authorization request: failed to obtain authorization message data: failed to obtain identity data: failed to call /usr/share/mender/identity/mender-device-identity: fork/exec /usr/share/mender/identity/mender-device-identity: no such file or directory"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=warning msg="Reauthorization failed with error: transient error: authorization request failed"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=error msg="Failed to submit inventory data: transient error: authorization request failed"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=error msg="inventory submit failed: transient error: authorization request failed"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=warning msg="Failed to refresh inventory: failed to submit inventory data: inventory submit failed: transient error: authorization request failed"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=info msg="State transition: inventory-update [Sync] -> inventory-update-retry-wait [Sync]"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=info msg="Handle update inventory retry state try: 12"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=warning msg="Returning artifact name from /etc/mender/artifact_info file. This is a fallback, in case the information can not be retrieved from the database, and is only expected when an update has never been installed before."
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=info msg="Device unauthorized; attempting reauthorization"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=error msg="Failed to authorize with \"\": failed to build authorization request: failed to obtain authorization message data: failed to obtain identity data: failed to call /usr/share/mender/identity/mender-device-identity: fork/exec /usr/share/mender/identity/mender-device-identity: no such file or directory"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=warning msg="Reauthorization failed with error: transient error: authorization request failed"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=error msg="Failed to submit inventory data: transient error: authorization request failed"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=error msg="inventory submit failed: transient error: authorization request failed"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=warning msg="Failed to refresh inventory: failed to submit inventory data: inventory submit failed: transient error: authorization request failed"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=warning msg="RetryPollIntervalSeconds is not defined"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=info msg="Handle update inventory retry state: failed to send inventory: Tried maximum amount of times"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=info msg="State transition: inventory-update-retry-wait [Sync] -> check-wait [Idle]"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=warning msg="UpdatePollIntervalSeconds is not defined"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=warning msg="InventoryPollIntervalSeconds is not defined"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=info msg="State transition: check-wait [Idle] -> update-check [Sync]"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=warning msg="Returning artifact name from /etc/mender/artifact_info file. This is a fallback, in case the information can not be retrieved from the database, and is only expected when an update has never been installed before."
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=info msg="Device unauthorized; attempting reauthorization"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=error msg="Failed to authorize with \"\": failed to build authorization request: failed to obtain authorization message data: failed to obtain identity data: failed to call /usr/share/mender/identity/mender-device-identity: fork/exec /usr/share/mender/identity/mender-device-identity: no such file or directory"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=warning msg="Reauthorization failed with error: transient error: authorization request failed"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=error msg="Error receiving scheduled update data: update check request failed: transient error: authorization request failed"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=error msg="Update check failed: transient error: update check request failed: transient error: authorization request failed"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=info msg="State transition: update-check [Sync] -> error [Error]"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=info msg="Handling error state, current error: transient error: update check request failed: transient error: authorization request failed"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=info msg="State transition: error [Error] -> idle [Idle]"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=info msg="State transition: idle [Idle] -> check-wait [Idle]"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=warning msg="UpdatePollIntervalSeconds is not defined"
Sep 04 18:46:41 umbrel mender[362]: time="2024-09-04T18:46:41Z" level=warning msg="InventoryPollIntervalSeconds is not defined"