Wall Connector network connections to AWS servers

[ATechGuy]

My Wall connector started showing "Offline" when I changed my firewall from IPFire to a Unifi gateway (UXG-Lite). I went back and forth with the Wall Connector support group and eventually they decided to send me a replacement charger even though I was 99.44% certain it was simply a "network" thing. Problem was, Tesla could SEE my TWC connected to their server(s) but it was still reporting "offline" in the app. (And that means no logging of charging history, etc.) The one question I never could get answered was what host FQDNs are requested by the TWC in making that connection to the AWS servers. Since Tesla thought it was connected, I'm beginning to believe the problem lies on HOW the connections are made the the servers.

After the new unit arrived and was installed, I continued to have the same "Offline" status so I decided to revert back to IPFire for a few days to see if the problem would go away and of course, it did. So, for now, I see some IP addresses that have been used by the TWC but I'm sure there must be a large POOL of IPs from AWS that are used. One IP/circuit seems to be possibly used for "Command and Control" while the other is used for exchanging much more data -- after a week, it's about 20MB on the data circuit and less than 1MB on the "C/C" circuit. I don't believe my IPFire provides an easy way for me to log DNS requests (and ideally, filter it from the TWC only) but I was hoping someone else out there has this ability and can monitoring these DNS requests to see what is being asked. Mostly, I think you'll see the requests within the first five minutes after it's powered on and it's established its connection to wireless. Once that happens, it seems to make three connections but terminates one of them very quickly--perhaps to first connect with an authentication server to find the owner's Tesla account and establish the necessary info to make the Command/Control circuit. After that the "data exchange" circuit seems to get established. But I'm seeing different IP almost each time.

Ultimately I am trying to figure out how to get it working under my Unifi Gateway, so perhaps another question is for anyone using a Unifi infrastructure at home that also has a TWC and has figured out how to make it appear "online" in the app.

Thanks!

 

[ewoodrick]

Why are you blocking outbound access that much?

 

[el-guapin]

does your UXG-Lite uses WPA2 0r WPA3?

 

[ATechGuy]

Why are you blocking outbound access that much?

Funny you ask that because initially, I was taking most of the "auto" defaults on the gateway and hadn't thought about it until I started trying to troubleshoot the issue for Tesla support. At that point, I *tried* to remove ALL outbound filtering (except multi-casting) so that's why I was drawing a blank on why the TWC wasn't able to connect to AWS.

 

[ATechGuy]

does your UXG-Lite uses WPA2 0r WPA3?

Well, I want to be polite, but the TWC IS connected to MY network just fine. The UXG-Lite is only a gateway. I don't believe in "wireless routers". I have an extensive Unifi network of access points and other equipment. Each device performs a single function. As for the TWC, I can run all the API calls on it, ping it, view version info, etc. The "offline" problem doesn't have anything to do with wifi, it has everything to do the gateway.

 

[dirty_park]

I also recently switched to a ugx-lite and have the exact same offline issue. It prevents my charger schedule from working so my charger doesn’t turn on and charge. This just started happening today by the way.

When I view the settings for the charger page it shows online and connected to Tesla. The only other thing I charged when I got the new gateway was DNS, I used to use 1.1.1.1

 

[swaltner]

If your firewall isn’t logging the blocked connections and the vendor (Tesla) isn’t able to tell you what’s required, it’s time to break out a packet capture tool like Wireshark (tcpdump) to capture the relevant details. The problem with that is getting the relevant traffic to the device running the packet capture. Wireless networks and Ethernet switches make that more difficult.

A DNS service like nextdns.io might be able to log your DNS queries without the need to do packet captures. That’s only half the data you need since it wouldn’t give you the destination port.

 

[ATechGuy]

I also recently switched to a ugx-lite and have the exact same offline issue. It prevents my charger schedule from working so my charger doesn’t turn on and charge. This just started happening today by the way.

When I view the settings for the charger page it shows online and connected to Tesla. The only other thing I charged when I got the new gateway was DNS, I used to use 1.1.1.1

No surprise you're seeing the issue as well.

As for the next entry, the thought of logging DNS would help me find the FQDN being requested so I could do more analysis on that connection. I would never want to blindly open ALL connections to AWS as AWS is sometimes used by bad actors for temporary servers and having some real-time block lists block those connections is desirable. The sad thing is that the Wall Connector folks really don't know much, and whenever my CSR would reply with "I have forwarded your comments (meaning my question on the AWS hosts) to my team..." is essentially saying, "Ha, good luck with getting an answer for that one!"

As for @dirty_park, I'm hoping you can try a few settings while I try the same. I can typically take my internet down during "Maintenance hours" which means my wife is out-of-the-house as my kids are long gone and I'm recently retired so I don't need it for business any more. I typically run with a country block list containing 88 countries (currently blank) and all the usual blocking of either ads, rogue, known to RBL, etc. but at the moment, I have all that cleared. My IPFire has similar blocking capabilities, but for some reason, is allowing the TWC to connect properly.

For now, I have two suspicions: 1) it's a multi-cast initial connection that's not being allowed (I have yet to unblock that on the UXG) and 2) I may not have my UDP or TCP timers set long enough.My IPFire uses 120 seconds for the UDP timeout and 120 minutes for TCP.

As for my DNS, my IPFire (in DHCP) hands out 1.1.1.3 and 1.0.0.3. If you wanted to experiment, try changing your DNS to these two and let me know if you see a change, but I'm guessing this isn't a DNS issue.

 

[ATechGuy]

If your firewall isn’t logging the blocked connections and the vendor (Tesla) isn’t able to tell you what’s required, it’s time to break out a packet capture tool like Wireshark (tcpdump) to capture the relevant details. The problem with that is getting the relevant traffic to the device running the packet capture. Wireless networks and Ethernet switches make that more difficult.

A DNS service like nextdns.io might be able to log your DNS queries without the need to do packet captures. That’s only half the data you need since it wouldn’t give you the destination port.

Yep. That's exactly what I did and provided the info to Tesla. I think it ended up being info that was WAY over their pay-grade.

 

[dirty_park]

DNS did not fix anything when I tested it. I am not blocking anything other than the default firewall settings. I do have IDS and IPS enabled but I tried disabling that with no luck. Here are my current settings. What’s interesting is when I reconnect the TWC to my WiFi it comes online for a few minutes then drops off.

I connected it to my neighbors WiFi and as expected it’s working just fine. It’s definitely a unifi issue.

 

[dirty_park]

@ATechGuy I figured it out, setting PFM to optional fixes the issue. Not sure if that's a long-time solution though as I am not familiar with this setting and what this might break.

5 minutes later....
Okay, it got weirder. I turned PMF back to disabled and the TWC disconnected as expected. I then created a new WiFi specifically for the TWC with the PMF setting set to optional. I did not connect the TWC to this new WiFi but as soon as I saved the new WiFi the TWC came online. Maybe bouncing WiFi fixed it for some odd reason. I have no idea. Let's see how long it stays online.


And 10 minutes later it’s offline. I think the solution is to create a new network with PMF as optional and connect it to that and call it a day.

 

[dirty_park]

Follow up to my last post. I tried my last solution with a new network with PMF optional and it stayed online for about an hour and then dropped.

 

[dirty_park]

Okay it’s happening to lots of people for months https://community.ui.com/questions/...n-Issues/7b5835a1-cab4-4fc3-8bc6-02fb368a0b6c

 

[brkaus]

PMF would allow the wifi connection to happen or not happen.

Does the new device have any additional helpers enabled by default? (Tftp, sip, etc)

 

[ATechGuy]

Okay it’s happening to lots of people for months https://community.ui.com/questions/...n-Issues/7b5835a1-cab4-4fc3-8bc6-02fb368a0b6c

Misery loves company, right?

I too had noticed that bouncing the TWC off the wifi would result in it "appearing online" for a short period of time. And, once it's off, one thing of interest is to hit the TWC with an API call such as: http://twc-ip-address/api/1/wifi_status and you'll most likely see "internet" = true for the status, even hours after it's been off line.

Again, back to my theory of TCP time outs...

I think I am getting closer to the problem. Still investigating, but I DEFINITELY believe it's a timeout issue. Perhaps I can provide a little bit of info here and the Unifi Gurus can chime in on suggestions.

Anyone who wants the full TCPdump file, it's attached. I didn't obfuscate my IPs as I really don't care if you know my TWC LAN IP (this hour) or my gateway IP. I stopped the capture once I could confirm the App started showing "Offline". OK here goes the summary:

1. TWC connects to wireless. It uses DHCP and requests an address, then ARPs for the gateway. All standard stuff. UXG is the DHCP server and I have no additional DHCP options enabled. I DO, however set the DHCP DNS to use my domain controller but I do this with any other (working) firewall/DHCP server.
2. What I don't show is the DNS translation (going to be my next tcpdump task -- I need to read up on how to filter a bit more as DNS requests could be a very busy log!
3. Anyway, the first thing the TWC does is connect to a Cloudfront.net ip, 18.161.34.52 on port 80. It makes an HTTP request from that server and gets a nice-sized return packet, sometimes two. The TWC then sends a disconnect packet to that server and it acknowledges it and disconnects. I think of this connection as the "OK, who do I need to connect with?" connection.
4. The very next packet is to an AWS server for unknown reasons. The file shows it connecting to 34.210.174.200.443
5. Immediately after closing this unknown connection, it opens what I refer to as the "data" connection. For me, this is always on 52.26.165.19:443
6. For 14 seconds, only the "Data" connection is active/open. Then the TWC makes a connection to what I've been calling the "Command and Control" connection. In my case, that's usually 35.161.98.75 on port 443.
7. The command and control (C/C) connection REMAIN connected when I'm using a firewall that keeps the TWC "online". However, under the UXG-lite, it seems to stop communicating after about 7 minutes. There is no attempt from the TWC to revive this connection, as it may not know it could be closed. This is expected behavior with TCP/IP. This is the part that's confusing and unknown.
8. The last communication to the C/C server happens on this line:
   12:38:48.548758 IP 192.168.10.79.65002 > 35.161.98.75.443: Flags [P.], seq 300:1233, ack 3446, win 14000, length 933. It's from the TWC, but the C/C server makes NO (apparent) acknowledgement to this packet. The "ack 3446" is the TWC acknowledgement of receiving the previous packet from the C/C server, which prevents the C/C/ server from re-transmitting sequence 3446.
9. For about ten minutes, only the "data" connection remains open/active. The TWC appears to send a possible keep-alive to the "data" server, but makes no effort to keep the "C/C" connection alive. Furthermore, the TWC makes NO effort to re-transmit sequence 300:1233 when it doesn't receive any acknowledgement! My hunch is the TWC reports the "offline" App condition once it hasn't received a packet from the C/C server and considers a timeout condition and closes that connection.
10. The "data" connection does not seem to affect the App status and while clearly connected, the App shows the TWC offline.
11. For full disclosure, the point for me when the App reported "offline" was immediately after the TWC made a new ARP request for finding the local DNS server. (and coincidentally, my Sense device as well). It seems that it was wanting to re-initialize the C/C connection and once again, it made the HTTP request to the Cloudfront server and received the same sized packet it received from it when it made the first set of requests after coming on-line.
12. And, as you might imagine this second time around, the TWC failed to make ANY connection requests to the C/C server. None. So, either the CloudFront server told it to piss off (while continuing to remain connected to the "data" server) or it became confused as to why it would be making that HTTP request while the "data" server was connected. While this logic will most likely never be known, it still remains a question why the C/C connection was terminated in the first place. I'm sure anyone with a "working" firewall that can display active connections will see that they have these two TCP connections to the two servers and they remain active (they are refreshed about once a minute) , keeping the App status as "online".

If anyone has additional suggestions on what I should be capturing, please chime in. I feel like we're closer and most likely it's going to get solved not by Tesla or Ubiquiti but with our combined knowledge. My next step is to create a DMZ for the TWC and examine that results.

Keep in touch!

 

[Ostrichsak]

I've got an all-UniFi network myself although I'm far from a guru so I'm afraid I won't be able to offer too much help with your issue specifically. I'm following this thread though hoping that maybe you'll uncover something that may help an issue I've been having also related to Tesla as a WiFi client and timeouts. My guess is that the hardware/software stack in the gen 3 HPWC is similar if not the same or else at least engineered by the same team on Tesla's end. There may be some consistencies with these issues.

I've got an issue where my cars sitting in the garage will suddenly drop connection to my network and not re-establish connectivity. This can be after several hours or a day or more randomly. It seems as though our older Model S cars are more prone to it than our newer Model Y but I'm still not entirely sure if it's something in the local hardware/software stack that is the culprit or just how Tesla handles things server-side. This makes it so the app is unresponsive and I'm unable to view or control anything when it happens. I'm left going out to the garage and waking the car or even rebooting even though all of the user settings are set to keep the car awake. It's not a signal strength issue as AP U6Pro coverage is more than adequate to reach the garage and they both show excellent tx/rx when they're actually connected. It's not a typical consumer grade WiFi router connectivity situation but most of what I've seen online is approaching the issue from that standpoint which is more commonly the issue, probably.

I did stumble upon something on a Reddit where someone was having similar issues as me. They claimed that adjusting the timeout helped. UniFi doesn't offer that ability as the end-user that I'm aware of. He also said something about Teslas in general not doing well when DHS was present. It was suggested that maybe disabling this might help. I debated creating a separate VLAN just for my Teslas w/that disabled but never got that far down the road. I WFH and rarely drive my Model S and my wife hasn't complained of any issues w/the 2023 Model Y she's been driving about a year now so I haven't really had the motivation. That and we change cars pretty frequently so I don't feel like creating a new VLAN each time we acquire another Tesla.

 

[ATechGuy]

I've got an all-UniFi network myself although I'm far from a guru so I'm afraid I won't be able to offer too much help with your issue specifically. I'm following this thread though hoping that maybe you'll uncover something that may help an issue I've been having also related to Tesla as a WiFi client and timeouts. My guess is that the hardware/software stack in the gen 3 HPWC is similar if not the same or else at least engineered by the same team on Tesla's end. There may be some consistencies with these issues.

I've got an issue where my cars sitting in the garage will suddenly drop connection to my network and not re-establish connectivity. This can be after several hours or a day or more randomly. It seems as though our older Model S cars are more prone to it than our newer Model Y but I'm still not entirely sure if it's something in the local hardware/software stack that is the culprit or just how Tesla handles things server-side. This makes it so the app is unresponsive and I'm unable to view or control anything when it happens. I'm left going out to the garage and waking the car or even rebooting even though all of the user settings are set to keep the car awake. It's not a signal strength issue as AP U6Pro coverage is more than adequate to reach the garage and they both show excellent tx/rx when they're actually connected. It's not a typical consumer grade WiFi router connectivity situation but most of what I've seen online is approaching the issue from that standpoint which is more commonly the issue, probably.

I did stumble upon something on a Reddit where someone was having similar issues as me. They claimed that adjusting the timeout helped. UniFi doesn't offer that ability as the end-user that I'm aware of. He also said something about Teslas in general not doing well when DHS was present. It was suggested that maybe disabling this might help. I debated creating a separate VLAN just for my Teslas w/that disabled but never got that far down the road. I WFH and rarely drive my Model S and my wife hasn't complained of any issues w/the 2023 Model Y she's been driving about a year now so I haven't really had the motivation. That and we change cars pretty frequently so I don't feel like creating a new VLAN each time we acquire another Tesla.

Since your issue isn't related to "Supercharging and Charging infrastructure" per se, you might want to try another sub-forum. But since you're here, what version is your firmware on the U6Pro? As for me, I used to have an Open Mesh AP40 for my "garage area" but recently (when I started seeing the issue being discussed in this thread) changed to a Unifi AP as the rest of my house is fully managed and on Unifi gear. My Model S (2020) has been connected for over a week but it sounds a little newer than yours. Not sure what DHS is. If you enable device logging, you might see something that would give a hint to when it drops the connection. I know in the earlier days, there was something like this and it was related to the car sleeping. Do you have premium connectivity for the Model S, and is your cell reception ok in the garage? Also, you might want to review your issue with someone in the motor club who knows more about "Service Mode" as there might be something in there that can give you the car's feeling of your wireless. Lastly, from a networking guy WAG, any chance you have the U6Pro very near the car and the transmission power is set very high. Unifi gear often works better when you have the power levels set down a tad.

 

[ATechGuy]

OK, so I did a dump on the WAN interface to see if we might find more useful information and this time I captured the C/C server retrying to send its packets as I had guessed (but couldn't see) above. This is the ROOT of our problem. Since the capture is small, I am including it here within the dialog so you don't need to download it just to see what's going on. For starters, we also now know the NAME the TWC is requesting for the C/C server, and it appears the first one being returned from DNS is the one I am always using: 35.161.98.75. With just a few entries now known, it might be worth writing a few rules around these IPs... Anyway let's have a look below:

# tcpdump -npi eth1 | grep 35.161.98.75
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:41:18.632171 IP 64.222.165.243.53 > 70.16.74.83.36582: 40044 4/0/1 CNAME ad81273b4123d4259ab6ebf516f3bca4-2fa6a1634a1eb935.elb.us-west-2.amazonaws.com., A 35.161.98.75, A 54.148.15.24, A 44.233.12.65 (199)
14:41:18.642503 IP 1.1.1.1.53 > 70.16.74.83.36582: 40044 4/0/1 CNAME ad81273b4123d4259ab6ebf516f3bca4-2fa6a1634a1eb935.elb.us-west-2.amazonaws.com., A 44.233.12.65, A 54.148.15.24, A 35.161.98.75 (199)
14:41:18.642740 IP 64.222.84.243.53 > 70.16.74.83.36582: 40044 4/0/1 CNAME ad81273b4123d4259ab6ebf516f3bca4-2fa6a1634a1eb935.elb.us-west-2.amazonaws.com., A 54.148.15.24, A 35.161.98.75, A 44.233.12.65 (199)
14:41:18.643200 IP 70.16.74.83.65025 > 35.161.98.75.443: Flags [Start] - test-, seq 2667518317, win 14000, options [mss 1400], length 0
14:41:18.734723 IP 35.161.98.75.443 > 70.16.74.83.65025: Flags [S.], seq 1098028064, ack 2667518318, win 62377, options [mss 1460], length 0
14:41:18.751182 IP 70.16.74.83.65025 > 35.161.98.75.443: Flags [.], ack 1, win 14000, length 0
14:41:18.753527 IP 70.16.74.83.65025 > 35.161.98.75.443: Flags [P.], seq 1:300, ack 1, win 14000, length 299
14:41:18.844664 IP 35.161.98.75.443 > 70.16.74.83.65025: Flags [.], ack 300, win 61978, length 0
14:41:18.846075 IP 35.161.98.75.443 > 70.16.74.83.65025: Flags [P.], seq 1:2801, ack 300, win 61978, length 2800
14:41:18.847196 IP 35.161.98.75.443 > 70.16.74.83.65025: Flags [P.], seq 2801:3446, ack 300, win 61978, length 645
14:41:18.850175 IP 70.16.74.83.65025 > 35.161.98.75.443: Flags [.], ack 2801, win 11200, length 0
14:41:18.950396 IP 70.16.74.83.65025 > 35.161.98.75.443: Flags [.], ack 3446, win 12578, length 0
14:41:21.313237 IP 70.16.74.83.65025 > 35.161.98.75.443: Flags [P.], seq 300:1233, ack 3446, win 14000, length 933
14:42:12.267752 IP 35.161.98.75.443 > 70.16.74.83.65025: Flags [P.], seq 7027:7058, ack 3563, win 58715, length 31
14:42:12.614555 IP 35.161.98.75.443 > 70.16.74.83.65025: Flags [P.], seq 7027:7058, ack 3563, win 58715, length 31
14:42:13.286413 IP 35.161.98.75.443 > 70.16.74.83.65025: Flags [P.], seq 7027:7058, ack 3563, win 58715, length 31
14:42:14.602392 IP 35.161.98.75.443 > 70.16.74.83.65025: Flags [P.], seq 7027:7058, ack 3563, win 58715, length 31
14:42:17.226438 IP 35.161.98.75.443 > 70.16.74.83.65025: Flags [P.], seq 7027:7058, ack 3563, win 58715, length 31
14:42:22.602547 IP 35.161.98.75.443 > 70.16.74.83.65025: Flags [P.], seq 7027:7058, ack 3563, win 58715, length 31
14:42:33.094629 IP 35.161.98.75.443 > 70.16.74.83.65025: Flags [P.], seq 7027:7058, ack 3563, win 58715, length 31
14:42:54.854589 IP 35.161.98.75.443 > 70.16.74.83.65025: Flags [P.], seq 7027:7058, ack 3563, win 58715, length 31
14:43:37.866542 IP 35.161.98.75.443 > 70.16.74.83.65025: Flags [P.], seq 7027:7058, ack 3563, win 58715, length 31

Two things to note here:

1. The fully bolded line is the one I noted in my earlier note and it never gets acknowledged by the TWC.
2. The Following line shows a new sequence (7027:7058) while also acknowledging packet "3563" but we never see the packet 3563 going outbound from the TWC. So, in true TCP/IP fashion, the C/C server re-transmits the same packet in hopes it will not "get lost" like the original packet. This continues for nine more retries and then gives up.
3. Meanwhile... The TWC keeps waiting for the ACK and it never arrives. So, it considers itself to be "Offline".

Looks like we have a classic broken NAT issue here with the UXG. Anyone still have an open ticket with Ubiquiti who could update the ticket with the info above (both log files might be of some help)?

 

[Rocky_H]

Since your issue isn't related to "Supercharging and Charging infrastructure" per se, you might want to try another sub-forum.

Why isn't it? I would think wall connectors is part of "& charging infrastructure". It certainly shouldn't go under any of the specific car models, so this does seem like the correct category for wall connector discussions. I don't see anywhere else it would fit better.

 

[Ostrichsak]

Since your issue isn't related to "Supercharging and Charging infrastructure" per se, you might want to try another sub-forum. But since you're here, what version is your firmware on the U6Pro? As for me, I used to have an Open Mesh AP40 for my "garage area" but recently (when I started seeing the issue being discussed in this thread) changed to a Unifi AP as the rest of my house is fully managed and on Unifi gear. My Model S (2020) has been connected for over a week but it sounds a little newer than yours. Not sure what DHS is. If you enable device logging, you might see something that would give a hint to when it drops the connection. I know in the earlier days, there was something like this and it was related to the car sleeping. Do you have premium connectivity for the Model S, and is your cell reception ok in the garage? Also, you might want to review your issue with someone in the motor club who knows more about "Service Mode" as there might be something in there that can give you the car's feeling of your wireless. Lastly, from a networking guy WAG, any chance you have the U6Pro very near the car and the transmission power is set very high. Unifi gear often works better when you have the power levels set down a tad.

Latest firmware across all of my UniFi gear and I did pull back the transmit power on the AP side upon installation years back. I appreciate the suggestion to try in other subforums. Like I said, I've all but given up on my issue at this point (it seems Tesla has too... lol) Especially since I know that a new(er) car may likely solve this issue. We'll likely be replacing the 2017 Model S P100D w/a Plaid sooner than later once I find the right deal so this issue has become far less pressing for me personally.

I was more following along with your situation just in case whatever you found out with the outbound/inbound comms from the HPWC came into play w/my situation as well. Doubtful but if I learn something along the way a bonus. Again, I wasn't trying to derail the thread so much as let you know that others were following your journey even if we weren't actively posting.

If you haven't already, I'd suggest posting over in the Unifi user forum too. There's quite a few things similar to this (non-Tesla mostly but traffic-related) being discussed as well as outstanding issues on the UniFi side of things. Beyond that, I'm afraid I'm not of much help other than to offer some encouragement on your troubleshooting.

BTW, my home s/u is also fully UniFi managed as well. Here's an unsolicited picture of my rack as it currently sits (until I get a bug to change something again):

I really need to expand to a 48pt PoE switch but can't bring myself to shell out the 2x price tag for the new one with the purdier lights.