FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[kabin]

We're starting to see videos of V11 in traffic. His camera seems to be overexposed as the sun doesn't seem to be low enough to cause terrible glare. There was a quick transition from brightness to shadow but is it enough to fail to negotiate a turn? Maybe they need to add more turn offset magic pixie dust. More likely it's an indication of system latency.

 

[bd7349]

We're starting to see videos of V11 in traffic. His camera seems to be overexposed as the sun doesn't seem to be low enough to cause terrible glare. There was a quick transition from brightness to shadow but is it enough to fail to negotiate a turn? Maybe they need to add more turn offset magic pixie dust. More likely it's an indication of system latency.

Where did you see it fail to negotiate a turn there? It looked like it handled everything very well in that video.

 

[jj3884]

Where did you see it fail to negotiate a turn there? It looked like it handled everything very well in that video.

I think he's saying it kinda weaved in lane by the shadow of the overpassed. Like it wasn't finding a good angle for the exit/curve

 

[Mardak]

Like it wasn't finding a good angle for the exit/curve

Just to be clear, this part where the blue predicted path touches the yellow line? Seems like it might be an intentional path to minimize lateral jerk while taking the exit at a higher speed.

The width of the blue path looks be sized to match the vehicle to give a better sense of where the wheels will be in the future. This could help with situations where people disengage thinking FSD Beta won't steer enough to avoid an obstacle or curb. The blue path here did indeed reflect getting close to the yellow line, but unfortunately, it did not account for the raised yellow reflectors that you can hear the car going over in the video.

 

[kabin]

Where did you see it fail to negotiate a turn there? It looked like it handled everything very well in that video.

The curve under the overpass is slow to negotiate. Listen to the thump, thump, thump as the tires run over the edge bumper/reflectors. That's a good warning for non-attentive drivers but not something anyone should expect for FSD.

 

[kabin]

Just to be clear, this part where the blue predicted path touches the yellow line? Seems like it might be an intentional path to minimize lateral jerk while taking the exit at a higher speed.

View attachment 910488

The width of the blue path looks be sized to match the vehicle to give a better sense of where the wheels will be in the future. This could help with situations where people disengage thinking FSD Beta won't steer enough to avoid an obstacle or curb. The blue path here did indeed reflect getting close to the yellow line, but unfortunately, it did not account for the raised yellow reflectors that you can hear the car going over in the video.

Per the release notes the team has been chasing the issue for sharp turns at fast speeds. This roadway looked pretty simple for FSD to be so far behind the curve.

 

[JB47394]

Per the release notes the team has been chasing the issue for sharp turns at fast speeds..

"Improved handling through high speed and high curvature scenarios by offsetting towards inner lane lines."

I read this and immediately thought of a pair of back-to-back low speed sharp curves that I negotiate weekly. Unfortunately, FSD ends up riding the outside lane lines - a double yellow. It's another scenario where I know I have to disengage FSD. I hope this release note doesn't describe a solution as simpleminded as it reads where both conditions have to be true.

I'd rather that the car predict lateral Gs in order to know how aggressively to hug the inside of the turn - meaning lower speeds with a sharper turn requires the same offset as a higher speed with a flatter turn. If the calculation says that the offset would be at its maximum and the lateral Gs still exceed some limit then it's time to slow down to get the offset in the correct range. Allow the driver to set the lateral G limit. I hate to think what FSD is doing to my tires.

Being a software engineer is so frustrating at times like this. Lemme see the code.

I wonder how the driving behavior will change once a neural net is controlling things. Will it put the C++ code to shame or will such a network be so difficult to train that Tesla abandons it?

 

[kabin]

I think he's saying it kinda weaved in lane by the shadow of the overpassed. Like it wasn't finding a good angle for the exit/curve

No noticeable left/right steering slop leading into this turn. And in the turn FSD appears confident hitting the road edge as there's no sign of FSD over or under correcting the steering. Very odd to see. It's almost as if FSD doesn't know where it is in the lane. More bake time needed.

 

[jj3884]

No noticeable left/right steering slop leading into this turn. And in the turn FSD appears confident hitting the road edge as there's no sign of FSD over/under correcting the steering. Very odd response almost as if FSD doesn't know where it is in the lane. More bake time please.

That's what I saw too, I meant weaving in the sense it didn't follow the curve of the road, it hit the left edge let the car pull out of the curve by decreasing steering/not increasing steering input then increased steering input to follow the road once.

That's what I meant by weaving.

 

[EVNow]

That link is dead for me.

I think they deleted the tweet. Now they have a confirmation that the HW4 car they tweeted about was definitely wrong.

 

[jj3884]

That's what I saw too, I meant weaving in the sense it didn't follow the curve of the road, it hit the left edge let the car pull out of the curve by decreasing steering/not increasing steering input then increased steering input to follow the road once.

That's what I meant by weaving.

By the way how a human would drive would be dependent on if taking the fastest most efficient route through a curve or centering. I'd prefer it stop freaking centering which would result in maybe a similar path as videoed depending on what you're trying to do when exiting the apex/curve, but it was sloppy and definitely corrected itself.

And no this doesn't mean speeding or racing through a curve/turn, it's about lowering the magnitude of a directional change... Especially when Lanes are pretty wide.

 

[JB47394]

To me, that sloppiness looks exactly like any FSD city street turn. It always swings wide when entering a lane, which is exactly what it did there.

 

[AlanSubie4Life]

Seems like it might be an intentional path to minimize lateral jerk while taking the exit at a higher speed.

But there was minimal turning required here and the speeds were not high, only 55-60mph or so.

Per the release notes the team has been chasing the issue for sharp turns at fast speeds.

This didn’t seem like one of these cases though.

I wonder if they are using the NN planner as advertised or whether this is still the cost function approach.

It always swings wide when entering a lane, which is exactly what it did there.

I haven’t really noticed this but I should pay closer attention to how it paths in these situations. Of course, I rarely let it take turns unless speeds are very low, since it is so annoying, so this may be why I don’t have this impression.

I know AP loves to plunge into new lanes aggressively but also likes to stay in the lane, but that is not applicable here.

 

[sleepydoc]

All I read is "we made assumptions without fact-checking" = fake news.

Poor reporting perhaps, and a story that needs correction but it is clearly not fake news.

 

[mtndrew1]

With limited videos it's hard to say but V11's lane shift looks pretty extreme. At the least, a feature as silly as this should have a customer enable/disable switch.

They did something like this about three years ago and it made me seasick. The car wobbled all over the lane, back and forth, as I drove past trucks. It was awful. Chief Twit said it was a feature and then they dialed it back because there were a lot of complaints.

 

[D Good]

I have very little coding experience (just some python) but I feel like even I (with considerable time ofc) could write code that predicts lateral gs and then adjusts speed accordingly. It’s just math.
With that being said, why has it taken so long for the engineers at Tesla to fix this issue? It’s just such a simple issue. Really makes you think if these people actually know what they are doing or if they are just flying by the seat of their pants… idk.
Then again, I realized from my little software experience that the simplest seeming problems are often the hardest.

 

[powertoold]

There's a very simple explanation: yellow lines don't matter unless there's a safety risk. If fsdb simply touches a yellow line with no cars around, that's a non-issue. Humans do it all the time in all sorts of situations (turns, sharp curves, passing, avoiding debris, etc. etc.).

If we're going to critique fsdb's every nuance, we'll never get anywhere. We should be pointing out real safety issues.

Tesla even mentions V11's new ability to bias all over the lane for all sorts of reasons: turns, debris, road blockage, etc. You can definitely expect V11 to be less "centered" and more free-flowing, which is a welcome change for some and a regression for others.

 

[thewishmaster]

Today it’ll touch that yellow line on some maneuver, and tomorrow it won’t. On the same stretch of road sometimes it’ll take the very slight curve a little too close to the curb and concrete median on the left or conversely swing too far into the bike lane on the right side and I end up taking over; other days it drives through perfectly. The unpredictability is the real safety issue IMO. I doubt v11 will improve much on that and we’ll all need HW4 with higher-res cameras and faster processing for improved predictability

 

[Supcom]

I have very little coding experience (just some python) but I feel like even I (with considerable time ofc) could write code that predicts lateral gs and then adjusts speed accordingly. It’s just math.
With that being said, why has it taken so long for the engineers at Tesla to fix this issue? It’s just such a simple issue. Really makes you think if these people actually know what they are doing or if they are just flying by the seat of their pants… idk.
Then again, I realized from my little software experience that the simplest seeming problems are often the hardest.

Perhaps it's not as simple as you think.

 

[Dewg]

This obviously isn't for Tesla, but ChatGPT. But it gives you a glance at how neural networks are created and trained. Some of theses things we want our cars to do, and think it should be easy with hard coded algorithms, aren't as easy when you think about what it takes to train a NN to handle it.

If anything, it's an interesting watch: