FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[jj3884]

I can *assure* you there won't be a big improvement.

Which is fair, and also fair on not continuing the subscription.

I'd hope at least on very simple things, like jerkiness of the wheel on a turn (at a stop even with clear views in all directions) would be smoother if not completely solved. That would vastly improve me confidence after the huge disappointment of experiencing Beta for the first time when it went wide.

If not, ok, well then I'll wait until it is. Hopefully, at the very least, they'll continue the wide public betas

 

[EVNow]

Which is fair, and also fair on not continuing the subscription.

Ofcourse.

I'd hope at least on very simple things, like jerkiness of the wheel on a turn (at a stop even with clear views in all directions) would be smoother if not completely solved. That would vastly improve me confidence after the huge disappointment of experiencing Beta for the first time when it went wide.

I agree - the jerkiness on a turn is very disconcerting. It took me almost a year to get used to it.

One of the things I had thought an AV would be able to do is to stop, start, turn extremely smoothly (after all they can calculate and find the best way !). Turns out since perception and prediction are not perfect, the control is not perfect either.

Though I should say, some of the turns surprise me. Very smooth - and majority are jerky.

 

[ddeflyer]

I've been assuming the jerkiness is due to lack of object permanence in the perception model (for example, the way a car disappears from the visualization when it goes behind (from our perspective) a larger vehicle). I think that the leaked V11 release notes mentioned temporal permanence improvements so hopefully pathing will be less spastic if the considered environment is more stable frame to frame.

 

[jabloomf1230]

I think thats an exaggeration. Definitely seen a lot of improvements over the last 6 months - including ULT. After CULT, all ULTs are way better, for eg.

6 months back I used to have disengagements on a ULT out of my neighborhood on a 2 lane street (one each way) @ 25 mph

I think that some people may not notice the improvements, because other things are still broken, such as:

1. Roundabouts
2. Right turn on red
3. Speed bumps
4. School zones
5. Running red lights
6. Not following the navigation correctly
7. Auto high beams
8. Auto wipers
9. Improper lane changes
10. Taking curves too fast
11. Too long a hesitation at 4 way stop intersections
12. Slow commit time for ULTs and URTs.

These discrete failures tend to be more obvious to the casual FSDb driver.

 

[Dewg]

Keep in mind that when a new NN is created, or expanded, it requires training. It's not uncommon for the first couple releases to be a bit rough as the new NN is dialed in. Most people call this a "regression", but it's just a training issue for the NN.

Think of it like learning to juggle. You start with a ball, then add a second one, and then third one. It takes time for you to get trained for that. Then I ask you to hop on one foot while juggling - you're not going to be as good at juggling until you get used (trained) to hopping on one foot. Then I ask you to spin while hopping... you get the idea. Each time a NN is created or expanded, there has to be some training from the fleet to get it dialed in.

 

[EVNow]

I think that some people may not notice the improvements, because other things are still broken,

Lots of "broken" or not yet implemented things - but that doesn't negate improvements.

 

[Joboo7777]

I think that some people may not notice the improvements, because other things are still broken, such as:

1. Roundabouts
2. Right turn on red
3. Speed bumps
4. School zones
5. Running red lights
6. Not following the navigation correctly
7. Auto high beams
8. Auto wipers
9. Improper lane changes
10. Taking curves too fast
11. Too long a hesitation at 4 way stop intersections
12. Slow commit time for ULTs and URTs.

These discrete failures tend to be more obvious to the casual FSDb driver.

I think if you were to show this list to a driving instructor they would say the person you are referring to must have failed their driving test.

 

[jabloomf1230]

I think if you were to show this list to a driving instructor they would say the person you are referring to must have failed their driving test.

Or they have already been in a large number of crashes.

 

[Supcom]

I've been assuming the jerkiness is due to lack of object permanence in the perception model (for example, the way a car disappears from the visualization when it goes behind (from our perspective) a larger vehicle). I think that the leaked V11 release notes mentioned temporal permanence improvements so hopefully pathing will be less spastic if the considered environment is more stable frame to frame.

Although the jerkiness on turns has improved greatly over the last year, it is somewhat surprising that it still can be a problem. My suspicion is that there is too much latency in the system. It's hard to execute a smooth maneuver when you are working with data that is 0.5+ seconds old. It may be that much of the improvement is related to processing time improvements made over the last year. After all, There's only a few car configurations, so you would think that Tesla would have worked out how to do basic maneuvers by now. But, if you are working with old data, you have to be very careful.

I think this is also the reason why the car does not accelerate smoothly from a stop when it's behind another vehicle as well as why the car often has hard stops - especially when trailing another car.

Ever notice how FSDb tends to slow down too much when a leading car slows to make a turn? Even after the car has cleared the lane, FSDb still slows the car for about a second before it starts accelerating again. That, I believe, is the processing lag. Since FSDb has not yet seen the car clear the lane, it has to keep braking with the result that the car slows too much and for too long.

Perhaps HW4 will improve this (assuming we get upgraded to HW4). I suspect that Tesla has squeezed about as much as they can from the processing lag with the current hardware.

 

[heapmaster]

Although the jerkiness on turns has improved greatly over the last year, it is somewhat surprising that it still can be a problem. My suspicion is that there is too much latency in the system. It's hard to execute a smooth maneuver when you are working with data that is 0.5+ seconds old. It may be that much of the improvement is related to processing time improvements made over the last year. After all, There's only a few car configurations, so you would think that Tesla would have worked out how to do basic maneuvers by now. But, if you are working with old data, you have to be very careful.

I think this is also the reason why the car does not accelerate smoothly from a stop when it's behind another vehicle as well as why the car often has hard stops - especially when trailing another car.

Ever notice how FSDb tends to slow down too much when a leading car slows to make a turn? Even after the car has cleared the lane, FSDb still slows the car for about a second before it starts accelerating again. That, I believe, is the processing lag. Since FSDb has not yet seen the car clear the lane, it has to keep braking with the result that the car slows too much and for too long.

Perhaps HW4 will improve this (assuming we get upgraded to HW4). I suspect that Tesla has squeezed about as much as they can from the processing lag with the current hardware.

The lag might be fixed when they start using raw image data instead of using processed image data in v11

FSD AP improvements in upcoming v11 from Lex Fridman interview

The recent Lex Fridman interview of Elon had some interesting tidbits about the next major FSD beta version (https://www.youtube.com/watch?v=DxREm3s1scA). First, something simple. Tesla’s AI inference chip (each car has two of them) has an image signal processor that processes each image frame...

teslamotorsclub.com

 

[Supcom]

The lag might be fixed when they start using raw image data instead of using processed image data in v11

If you are referring to the use of photon counts, I believe that is already in 10.x. And Musk has stated that much of what makes up 11.x is already in 10.x, so I don't expect any groundbreaking changes anytime soon.

It's also curious that there hasn't been any addition to things like sign interpretation. Perhaps adding new object identification adds too much to the processing time and pushes it beyond the maximum permissible. So, the car does not read lane assignment signs, school zones signs or any other signs other than the required speed limit signs.

 

[Zane Liu]

If you are referring to the use of photon counts, I believe that is already in 10.x. And Musk has stated that much of what makes up 11.x is already in 10.x, so I don't expect any groundbreaking changes anytime soon.

It's also curious that there hasn't been any addition to things like sign interpretation. Perhaps adding new object identification adds too much to the processing time and pushes it beyond the maximum permissible. So, the car does not read lane assignment signs, school zones signs or any other signs other than the required speed limit signs.

I think eventually they will need to train more signs that are critical to driving. No turn on red, dynamic signs, and closures would all be necessary for FSD

 

[powertoold]

Looks like another two weeks folks

 

[jeremymc7]

Looks like another two weeks folks

Did Musk post an update ?

 

[powertoold]

Did Musk post an update ?

Doesn't sound promising:

https://twitter.com/x/status/1616635070156255232

 

[jeremymc7]

Yep. That sounds like two weeks, best case.

 

[EVNow]

The lag might be fixed when they start using raw image data instead of using processed image data in v11

They already made those changes.

 

[JulienW]

Doesn't sound promising:

https://twitter.com/x/status/1616635070156255232

Unless you consider "V11 wide release not happening in January" a promise.

Hate to conjecture it but highway Beta must be really bad since they are struggling so much and have been for sooooooooo long with it.

 

[willow_hiller]

From what we've heard from those who have spoken to employees, the pace of releases will slow down because they're fine tuning new features before releasing them.

The current state of V11 may not be bad. But it may still contain some bugs they want to fully work out before release.

We're not early access testers anymore, we're just regular customers.

 

[JulienW]

.....The current state of V11 may not be bad. But it may still contain some bugs they want to fully work out before release......

Hopefully but obviously there have been major problems (again hopefully they have been mostly addressed) since it was supposed to be released a full year ago. Hell this thread has already celebrated its anniversary and now into the 2ed year.