FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[JHCCAZ]

Not impressive…

To me, impressive but imperfect.

 

[Mardak]

there is at least one car on 2022.45.15 queued up for 2023.7.10 so... there's hope?

New push of 11.4.2 now a total of 10% rollout. The increase of 7% rollout seems to be almost all on 11.3.1 / 2022.45.15, so new additions to 11.4.x. 2023.7.10 Official Tesla Software Update | Teslascope

 

[j0shm1lls]

ooo ahh push it

 

[kabin]

I didn’t think I would be saying, “Junk” so early in this release cycle….

I hear ya. For the sake of progress the expectation bar might need to be lowered for now. It's almost like FSD ran out of fingers to plug the leaks in the dike. Fill a new leak only to have an old leak return.

 

[FSDtester#1]

ooo ahh push it

View attachment 941509

"Push it reallll Good!"

 

[MarcG]

I got a push

 

[Ramphex]

What in the *sugar* is this? Why isn’t the new junk on the same 2023.12.X branch as the polished turd?!

 

[JHCCAZ]

We can't lean farther forward than the windshield, where the camera is. The front-back position isn't the problem.

I'm not sure we're discussing the same issue here. The front facing cameras are at the top of the windshield, but the primary camera used four looking at cross-traffic is the B pillar camera, which is definitely behind the driver's head.

The wide front camera is not wide enough to be very helpful for high-speed cross traffic, even less so as the cars pose angle changes when it starts to creep into the turn

if and when they add the forward-angle repeater camera, that will be located somewhat further forward than the driver's angle of view.

If anything, it is sometimes a disadvantage, because our eyes' vision doesn't get completely obscured by a single drop of rain, thanks to the drop of rain blocking 0.2 degrees of arc relative to our eyes instead of 30 degrees of arc.

Here I partly agree with you; I've mentioned this in some of the auto-wiper debates where I've pointed out that the Tesla isn't seeing things the same way we do for the reasons you mentioned. A similar point can be made regarding the effects of sun glare.

On the other hand, particularly for the pillar and repeater cameras, I think the droplets are close enough to the lens that they're somewhat out of focus and don't exactly "block" a well-defined angle, though hey certainly reduce the contrast and acuity. This is probably even more true with the newer cameras that appear to have a larger entrance aperture.

The main difference is that because we are far from the window, we can get a useful approximation of an image at a steep angle through the glass, looking to both sides, whereas with a single front camera, or even with a normal and zoomed-in front camera, you can't do that.

What they need to do is replace the cluster of three cameras with three cameras that are zoomed in somewhat more than they currently are, and with the outer two cameras angled outwards at about 45 degrees. That would greatly improve their ability to pull out of parking places, make unprotected turns, etc. This should be an easy change, even for existing cars.

Agreed that this could be a possible approach, as an alternative to front corner cameras (which have never even been rumored) or the forward-angle repeater camera (which have been rumored but haven't appeated). However, I haven't seen any evidence of symmetrical left/right canted windshield cameras either.

The other thing they need is two corner cameras at the rear, for parking purposes. By the time the side repeater cameras or B-pillar cameras have a view of anything, you're already out in the lane, and it is too late for them to matter. However, you can work around that by always backing into every non-slanted parking place.

Completely agree. Rear corner cameras for parking assistance including very good rear cross-traffic detection would be outstandingly helpful, but again no evidence so far.

 

[FSDtester#1]

View attachment 941511

What in the *sugar* is this? Why isn’t the new junk on the same 2023.12.X branch as the polished turd?!

So you can get it and then take up the next 100 pages complaining ...
At least with this release you will get your AM radio you've been wanting

 

[JHCCAZ]

ooo ahh push it

"Push it reallll Good!"

I got a push

My wife asked me if I wanted to go shopping with her, but I said I was waiting for a push. She gave me a very strange look and wient out the door

 

[powertoold]

Haven't watch this yet, but must be good:

Great assertive merge into traffic here:

 

[Bet TSLA]

I just got 11.4.2 (2023.7.10) coming from 11.3.6 (2022.45.15). Keeping my expectations very low.

 

[powertoold]

Here's one with refresh S:

 

[PACEMD]

lysergic acid diethylamide....safe and non toxic

Tune in, turn on, and drop out........Steve Jobs with his barefoot apples........

 

[loquitur]

I'm not sure we're discussing the same issue here. The front facing cameras are at the top of the windshield, but the primary camera used four looking at cross-traffic is the B pillar camera, which is definitely behind the driver's head.

The wide front camera is not wide enough to be very helpful for high-speed cross traffic, even less so as the cars pose angle changes when it starts to creep into the turn

if and when they add the forward-angle repeater camera, that will be located somewhat further forward than the driver's angle of view.

Here I partly agree with you; I've mentioned this in some of the auto-wiper debates where I've pointed out that the Tesla isn't seeing things the same way we do for the reasons you mentioned. A similar point can be made regarding the effects of sun glare.

On the other hand, particularly for the pillar and repeater cameras, I think the droplets are close enough to the lens that they're somewhat out of focus and don't exactly "block" a well-defined angle, though hey certainly reduce the contrast and acuity. This is probably even more true with the newer cameras that appear to have a larger entrance aperture.

Agreed that this could be a possible approach, as an alternative to front corner cameras (which have never even been rumored) or the forward-angle repeater camera (which have been rumored but haven't appeated). However, I haven't seen any evidence of symmetrical left/right canted windshield cameras either.

Completely agree. Rear corner cameras for parking assistance including very good rear cross-traffic detection would be outstandingly helpful, but again no evidence so far.

Why is there a continuing undercurrent of folks who claim that their two eyes can see better
than eight cameras? This reminds me of "audiophiles" who claim they can hear things better
than 16-bit audio, while never doing better in A-B comparison tests. If humans need to lean forward
to see better at blind intersections, then Tesla cars can creep forward to do the same thing.

 

[PACEMD]

You guys are pushing it.............
Now if I were the president of this land
You know, I'd declare total war on The Pusher man......

 

[powertoold]

All hail the almighty Zero Disengagement Gods!

It's incredible to me that the visualization stayed rock solid during the dippity dip

 

[mlambert890]

Why is there a continuing undercurrent of folks who claim that their two eyes can see better
than eight cameras? This reminds me of "audiophiles" who claim they can hear things better
than 16-bit audio, while never doing better in A-B comparison tests. If humans need to lean forward
to see better at blind intersections, then Tesla cars can creep forward to do the same thing.

This analogy is terrible. The two eyes are wired to a human brain and attached to a head that swivels around. They are also wired up to hearing and to "feel" (multi-sensor). The car has questionable local processing capacity and a software stack so unsettled that Elon has now decided it "needs AI end to end", connected to a set of fixed low quality cameras.

It's impressive that it works at all, but every drive I intervene and save FSD Beta. Conversely it has never once managed to see something I didn't see and save me. So demonstrably it is not that unreasonable to claim that yeah, a human with two eyes can "see better" than FSD Beta.

What I'm genuinely curious about is how folks who constantly point out how "FSD is a Beta and an assistance system. You must always be paying full attention and ready to intervene. In the end it is always the drivers responsibility!", every time someone points out a screw up, as if they were part of the Tesla legal team, feel so comfortable making these claims about how obviously "superhuman" it is as long as the discussion isn't about one of its infinite screw ups...

 

[loquitur]

This analogy is terrible. The two eyes are wired to a human brain and attached to a head that swivels around. They are also wired up to hearing and to "feel" (multi-sensor). The car has questionable local processing capacity and a software stack so unsettled that Elon has now decided it "needs AI end to end", connected to a set of fixed low quality cameras.

It's impressive that it works at all, but every drive I intervene and save FSD Beta. Conversely it has never once managed to see something I didn't see and save me. So demonstrably it is not that unreasonable to claim that yeah, a human with two eyes can "see better" than FSD Beta.

Brains are great, perhaps even currently better than FSDb (sensors+software) at playing
the driving game, although the statistics don't show that. (FSD stats actually show that the hybrid
combo of humans + FSD > humans alone for accident rates.)

Now, the software may be relatively stupid at planning tasks, etc. but what shows that eyes vs.
cameras are inherently superior?

P.S BTW I drive with FSDb in San Francisco, CA, and it's quite serviceable on most city streets,
once you trust it for the most common situations, don't mind driving slow, and learn where its
weaknesses are, which change over time (used to be things like curvy roads, narrow streets,
pedestrians in crosswalks, construction zones, etc. but all gets incrementally better over time.)
But here in a busy city, it's never about eyes vs. cameras. I do believe that that newer 4K cams can
likely read road signs better, and that's cool, but it's not a showstopper for providing great driver assistance.

 

[JHCCAZ]

Why is there a continuing undercurrent of folks who claim that their two eyes can see better than eight cameras?

???
There may be those folks, but I'm not one of them and that's not at all what I said.
Eight (simultaneous) cameras obviously have major potential advantages. Several, but not all, have been realized. But the fact remains that point-of-view selection matters. There are many common situations that would be much better served with better camera placement, it's a matter of geometry and obstructions , not of misplaced bias or fuzzy thinking.

Regarding creeping, I think that most rational users can easily see that it's being pushed too far for comfort, and still doesn't solve as many cases as would be possible with a different arrangement. Not hopeless but could be better.

If there were no issues with the legacy camera setup, Tesla would not be working on it at all.