24.4.0 Gateway crashes

[jgleigh]

Anyone else experiencing a lot more Gateway 2 crashes with 24.4.0? My system is almost three years old, but recently I've been seeing a lot of Gateway reboots. Normally this only happens when there's a new firmware install (or the rare lockup), but now I'm seeing them every few days. Very strange and not totally noticeable since it auto-recovers.

Posting dates and times in case some pattern emerges.

2024-04-21 17:24:50
2024-04-30 12:39:51
2024-05-01 06:38:30
2024-05-02 13:53:05

 

[offandonagain]

Mine restarted 2 days ago, but before that it was up for 30 days (on version 24.4.0 0fe780c9 throughout).

 

[scdoubleu]

Glad it's not just me. I've had to hit the reset button on mine twice now since 23.44.0 (predecessor to 24.4.0) due to it going unresponsive, both over the local API and via the Tesla App.

There have been a few other occasions where it's gone offline but came back by itself after 10-20 minutes or so.

 

[EVRider-FL]

Excuse my ignorance on this topic, but can I determine when the gateway restarts from the Tesla app, or do I have to access the gateway for that?

I’ve had my system for 2.5 years and have never had to restart anything manually. I wouldn’t even know how to do it without looking it up first. Is the need to manually restart the gateway a common occurrence, and I’ve just been lucky? My current gateway version is 24.4.0.

 

[scdoubleu]

Excuse my ignorance on this topic, but can I determine when the gateway restarts from the Tesla app, or do I have to access the gateway for that?

Not that I'm aware of. I run some code which queries the Gateway's local API every 10 seconds or so to get status info which is used for some graphing and home automation. It is when these queries fail that I can see (in my log files) that the Gateway is occasionally offline.

Is the need to manually restart the gateway a common occurrence, and I’ve just been lucky? My current gateway version is 24.4.0.

I suspect the Gateway has some form of watchdog feature which will restart it eventually in case of a crash. I manually reset it on two occasions because it stopped responding to changes I was making in the App (lowering standby reserve to avoid pulling from the grid about 30 minutes before the off-peak period kicked in).

 

[jgleigh]

Excuse my ignorance on this topic, but can I determine when the gateway restarts from the Tesla app, or do I have to access the gateway for that?

I'm running Powerwall Dashboard. It's quite handy if you're really into the low level data and monitoring your system. There are also some phone apps that report similar information.

GitHub - jasonacox/Powerwall-Dashboard: Grafana Monitoring Dashboard for Tesla Solar and Powerwall Systems

Grafana Monitoring Dashboard for Tesla Solar and Powerwall Systems - jasonacox/Powerwall-Dashboard

github.com

You can also access some of the API locally if you have a Gateway 2.

http://192.168.1.6/api/status (replace with IP of your Gateway)

{
  "start_time": "2024-05-02 21:18:07 +0800",
  "up_time_seconds": "9h1m16.354123845s",
  "is_new": false,
  "version": "24.4.0 0fe780c9",
  "git_hash": "0fe780c991e052023349431b3c539429d1db036c",
  "commission_count": 0,
  "device_type": "teg",
  "teg_type": "unknown",
  "sync_type": "v2.1",
  "cellular_disabled": false,
  "can_reboot": true
}

I suspect the Gateway has some form of watchdog feature which will restart it eventually in case of a crash. I manually reset it on two occasions because it stopped responding to changes I was making in the App (lowering standby reserve to avoid pulling from the grid about 30 minutes before the off-peak period kicked in).

The watchdog usually reboots the Gateway successfully. Very rarely I've had to manually intervene and restart it myself. When that happens the whole system has shutdown and solar/Powerwalls are offline. The current batch of reboots only seem to affect the Gateway and the solar/Powerwalls keep operating through the short period while the Gateway resets. Saw another one late last night.

2024-04-21 17:24:50
2024-04-30 12:39:51
2024-05-01 06:38:30
2024-05-02 13:53:05
2024-05-02 21:22:31

 

[jgleigh]

2024-04-21 17:24:50
2024-04-30 12:39:51
2024-05-01 06:38:30
2024-05-02 13:53:05
2024-05-02 21:22:31
2024-05-04 15:11:25

 

[CrazyRabbit]

"start_time":"2024-04-28 11:56:49 +0800","up_time_seconds":"178h13m39.346408843s","is_new":false,"version":"23.44.0 eb113390"

 

[adspguy]

Mine is running 24.4.0 0fe780c9 and has been up for 47 days

 

[jgleigh]

"start_time":"2024-04-28 11:56:49 +0800","up_time_seconds":"178h13m39.346408843s","is_new":false,"version":"23.44.0 eb113390"

Interesting that you're still on 23.44.0. As usual they never say what changed or when there's a rollback.

Did you get 23.44.0 on 4/28 or was that just a reboot cycle?

 

[offandonagain]

Interesting that you're still on 23.44.0.

I figured this might be useful:

https://www.netzeroapp.io/firmware_versions

 

[jgleigh]

The mystery continues. Today I had two longer than usual dropouts of the Gateway connection. 16 minutes and 14 minutes. And it only rebooted during the longer first outage and not the second outage.

 

[zƬesla]

I figured this might be useful:

https://www.netzeroapp.io/firmware_versions

I'd be curious whether the version numbers correlate to PW2 or 3 and region/utility. For example, I'm on 23.28.2 and have PW2s in NH

 

[Matt-FL]

I'm also on 23.28.2, PW2s in FL.

 

[offandonagain]

I'd be curious whether the version numbers correlate to PW2 or 3 and region/utility. For example, I'm on 23.28.2 and have PW2s in NH

I can't find any pattern, no. I can think of two reasons why one might be on an older version: a deliberate holdout by Tesla (either to keep a baseline for comparing metrics, or because of a particular circumstance of the system), or an issue updating to the new firmware.

The Tesla One app should show you if a newer firmware is available, which might offer some clues -- if no new version is available, it's more likely a holdout. If you use Netzero to access the gateway or know how to access APIs directly, you can also look at the /system/update/status and /sitemaster APIs for clues (the latter has a can_reboot property which seems relevant).

Note that more recent firmware versions remove functionality (e.g. the vitals API), so as long as everything is working well you might be better off on an older one

 

[offandonagain]

The mystery continues. Today I had two longer than usual dropouts of the Gateway connection. 16 minutes and 14 minutes. And it only rebooted during the longer first outage and not the second outage.

A few things to try if you haven't already:

• The steps above to see if these are unsuccessful attempts to upgrade to a newer firmware.
• Stop the local gateway API traffic, to see if that might be causing it.
• Power-cycle the gateway. In my case this requires turning the disconnect, Powerwall switches, and relevant breakers off, and then additionally resetting the gateway. Otherwise the Powerwalls and built-in battery interfere with the process. Don't do this if there's any risk of grid outage. Details here.

 

[Matt-FL]

I am in no rush to have my gateway upgraded.

 

[jhn_]

I'd be curious whether the version numbers correlate to PW2 or 3 and region/utility. For example, I'm on 23.28.2 and have PW2s in NH

24.4.0 on PW2s

 

[jgleigh]

Reboot today even took down the inverters. Now I'm worried my Gateway is starting to die.