FSD v12.x (end to end AI)

[Mardak]

The levels describe what we can EXPECT from a driving system and doesn't convey anything about its performance.

Here's California 2023 Autonomous Mileage Report including Autonomous Vehicle Permit Holders who reported no miles:

---- APEX.AI
---- BEEP
---- BLACK SESAME TECHNOLOGIES INC
---- BLUESPACE.AI, INC
---- HELM.AI INC
---- MAY MOBILITY
---- MOBILEYE
---- NIO USA INC.
---- NVIDIA CORPORATION
---- PLUSAI, INC
---- PONY.AI
---- RIDECELL INC
---- TELENAV, INC.
---- TESLA
---- VUERON TECHNOLOGY USA, INC
---- XMOTORS.AI, INC
0.1K VALEO NORTH AMERICA
0.6K GATIK AI INC
0.7K AURORA OPERATIONS, INC.
0.7K IMAGRY INC
0.8K BOSCH
  4K QUALCOMM TECHNOLOGIES, INC.
  4K DIDI RESEARCH AMERICA
  4K WOVEN BY TOYOTA, U.S., INC.
  5K NISSAN
  8K AUTOX
 15K AIMOTIVE INC.
 15K APOLLO
 16K MOTIONAL AD, INC.
 42K WERIDE CORP
 53K GHOST AUTONOMY INC
 58K MERCEDES
0.1M NURO INC
0.5M APPLE INC.
0.6M CRUISE LLC
0.7M ZOOX, INC
3.7M WAYMO LLC

Presumably all of these companies have some desire to eventually test a vehicle that does not require a human. One possible reason for no miles is the performance of the autonomous vehicle is not good enough yet to test on California roads. Tesla is in a special set that can make progress towards more autonomous technology by improving their driver assistance feature, and I would guess even the current state of FSD Beta 12.x is much more useful and even safer than most of that list especially towards the top. E.g., "Test car not allowed to go over the double yellow lane marking" or "Driver took over for ego vehicle attempting lane change without enough space."

Potentially that list has a startup with design intent of full driving automation but their vehicle could fail to move (i.e., not useful) or would crash (i.e., not safe) or generally having performance deficiencies. I would much rather have the performance of Tesla's current end-to-end even as driver assistance, and there's potential for 12.x performance to exceed all on that list even while the driver is responsible.

Another aspect of performance is comfort, and what we've seen from 12.x so far is that it can be much more comfortable than not only 11.x but also probably almost all on that list. Unclear how explicitly Tesla selected training data for comfort as opposed to making sure the vehicle drives correctly and safely, so there could still be significant comfort improvements to come in later 12.x.

 

[Mardak]

I think that Tesla said that recall was fixed in 2023.44.30.13

Yeah, I also noticed that in the recall report, but TeslaInfo pulling release notes from the small number of its vehicles getting 2023.44.30.13 did not include the recall fix. Now with 2023.44.30.14 going wider (TeslaFi shows only HW3 so far), it still seems like preparation for updating vehicles to a later 2023.44.30.x presumably including both FSD Beta 12.x and Telltale Text Size fix as opposed to just pushing people to 2024.x with the recall fix.

 

[Dewg]

Where is the confirmed list of deaths while using FSDb? (Not while using AP or NoA.) I haven't seen anything, and probably certainly not many, if any. (There have been a lot of people claim it, but most of those appear to be false claims.)

The only list I found is www.tesladeaths.com
But it does not decern between AP and FSD Beta.

 

[AlanSubie4Life]

what we've seen from 12.x so far is that it can be much more comfortable than not only 11.x but also probably almost all on that list.

Have we seen that? Has someone done a detailed analysis of the speedometer in the provided videos and generated acceleration and jerk profiles over time (obviously limited to generally one axis, and does not cover all perceived jerk or effect of hills)?

I don’t have the patience to watch hours of video to figure out that “actually it is not as smooth as everyone losing their mind on the Internets said.”

But anyway, when I receive it, I do not anticipate that I will describe v12 as “smooth as a good human driver.” That would be really amazing though, and it would be awesome, and I would accept many failings and limitations to ODD, as long as everything it did was smooth (except to adapt to unanticipated errors by other drivers), and any errors it made could be smoothly corrected from.

For an example of jerkiness (the only one I have seen since I have not looked), see the latest video kindly posted by @powertoold! That was pretty rough!

Good human drivers anticipate, and only very rarely make reversals in their steering angle mid turn (and when they do, it is usually due to surprise events)!

 

[drtimhill]

What the word means has not just major engineering but also legal significance.

Hence why I remain baffled by the folks insisting "bah, the actual law and engineering are just semantics!"

Semantics: The study of meanings in a language. (from the dictionary.)

You (above): What the word means has not just major engineering but also legal significance.

You (to me): It's NOT about semantics!

 

[drtimhill]

But anyway, when I receive it, I do not anticipate that I will describe v12 as “smooth as a good human driver.” That would be really amazing though, and it would be awesome, and I would accept many failings and limitations to ODD, as long as everything it did was smooth (except to adapt to unanticipated errors by other drivers), and any errors it made could be smoothly corrected from.

I would expect it to be, at most, comparable to V11. This is the "sawtooth" effect on projects like this. When you do a major technology shift you create the framework for an advance, but the initial iterations are a (hopefully) minor step backwards. The C++ code was becoming a bottleneck to innovation, but had set a bar (how high we can debate endlessly) against which we will all judge V12. My guess is that the advance initially WONT be that V12 is breathtakingly better then V11, it will be that they have the groundwork laid for significant subsequent improvement.

 

[drtimhill]

Nope.

If the system is L2, it means it cannot perform entire DDT. Removing driver supervision, does not make it L4. It is still L2 since it cannot perform entire DDT. Removing driver supervision for L2, just means that you still have L2, just L2 with no driver supervision (not a good idea).

You seem to be under the illusion that a system can functionally be L4 (it can do entire DDT) but the developer just designates it as "L2" because it might need driver supervision. This is a common misconception by some Tesla fans who think that FSD beta is basically L4, it is just designated "L2" because it needs driver supervision. And once it is reliable enough to not need driver supervision, then Tesla can simply declare it is now L4.

That is not how it works. If the system can perform entire DDT and DDT-fallback in limited ODD, then it is L4, whether there is driver supervision or not. And if the system cannot perform entire DDT, then it is L2, whether there is driver supervision or not (of course, no driver supervision for L2 does not really make sense).

I think Tesla fans need to pick a lane. If FSD beta is performing entire DDT and DDT-fallback, then it is L4. It would be L4 with driver supervision but it is L4. If FSD beta is not performing entire DDT then it is L2. You can't say it is both L2 and L4.

Overall totally agree, though I would make two comments. It seems to me there is a difference between a system being "L4 capable" with one that is "declared L4". I can certainly see a manufacturer reaching L4 technically, but not wishing to declare this for liability reasons. The SAE documents discuss the technical requirements necessary for a system to be capable of a certain level, but I dont think that automatically triggers (e.g.) liability until such a time as an actual claim is made by a manufacturer.

Second, there are a number of posts here that seem to claim that an L2 system cannot evolve into an L4 system over time, as if L4 was fundamentally different rather than a superset/progression from L2 (albeit a large one). These arguments are then (dubiously, imho) used to claim that Tesla FSDb is essentially a dead-end. I think you might make a case that getting to L4 via L2 is not an optimal path, but to say its not possible seems flawed logic to me.

 

[Mardak]

Has someone done a detailed analysis of the speedometer in the provided videos and generated acceleration and jerk profiles over time (obviously limited to generally one axis, and does not cover all perceived jerk or effect of hills)?

As you've pointed out before, it's much easier to feel the jerkiness than to see it from a video, but some adjustments of 11.x are easier to see such as steering wheel changing directions (clockwise <-> counterclockwise). For approaching and completing this left turn, 11.x seems to change directions 12 times (although it [unnecessarily?] adjusted for cross traffic approaching the stop sign) while 12.1.2 changed directions 5 times (and in much smaller intensity approaching the stop that it was hard to count). I would think 12.x could still do better as it does pause then increase the amount of steering (without changing directions) as opposed to a continuous motion.

Also probably not the best example but easiest to find when going through AI DRIVR's V12 First Impressions:

It feels like the robot that used to drive the car has been replaced by a competent human, and all the little micro adjustments and odd steering inputs I've gotten so used to while testing the beta over the years are either drastically reduced or completely eliminated. It no longer feels like it's calculating what to do all the time, and instead it just does it. Even when the car was dealing with something unexpected that would definitely trip up current versions like this light turning yellow while we were in the crosswalk, it behaved naturally and confidently. The full self-driving beta has been getting more and more smooth over the years but this thing's on a completely different level. I kept waiting for the moment it was going to screw up and do a hard break like it usually does when pedestrians walk out in front of you, but that moment never came. Here you can see it's not even stopping as these pedestrians cross the street just a smooth slowdown and a smooth speed up. I can't overstate how much better the interactions with pedestrians are. It still treats them with additional caution but doesn't feel scared of them anymore. This is something pretty hard to get across when you're just watching a video like this, but when you're actually in the car it will become immediately apparent.​

 

[AlanSubie4Life]

As you've pointed out before, it's much easier to feel the jerkiness than to see it from a video, but some adjustments of 11.x are easier to see such as steering wheel changing directions (clockwise <-> counterclockwise). For approaching and completing this left turn, 11.x seems to change directions 12 times (although it [unnecessarily?] adjusted for cross traffic approaching the stop sign) while 12.1.2 changed directions 5 times (and in much smaller intensity approaching the stop that it was hard to count). I would think 12.x could still do better as it does pause then increase the amount of steering (without changing directions) as opposed to a continuous motion.

Also probably not the best example but easiest to find when going through AI DRIVR's V12 First Impressions:

It feels like the robot that used to drive the car has been replaced by a competent human, and all the little micro adjustments and odd steering inputs I've gotten so used to while testing the beta over the years are either drastically reduced or completely eliminated. It no longer feels like it's calculating what to do all the time, and instead it just does it. Even when the car was dealing with something unexpected that would definitely trip up current versions like this light turning yellow while we were in the crosswalk, it behaved naturally and confidently. The full self-driving beta has been getting more and more smooth over the years but this thing's on a completely different level. I kept waiting for the moment it was going to screw up and do a hard break like it usually does when pedestrians walk out in front of you, but that moment never came. Here you can see it's not even stopping as these pedestrians cross the street just a smooth slowdown and a smooth speed up. I can't overstate how much better the interactions with pedestrians are. It still treats them with additional caution but doesn't feel scared of them anymore. This is something pretty hard to get across when you're just watching a video like this, but when you're actually in the car it will become immediately apparent.​

Yeah we’ll see how it goes when it releases. The lack of skittishness is certainly welcome, but it also needs to reliably respond to VRU’s which are actually a concern. Not sure where it will end up on balance.

 

[diplomat33]

Overall totally agree, though I would make two comments. It seems to me there is a difference between a system being "L4 capable" with one that is "declared L4". I can certainly see a manufacturer reaching L4 technically, but not wishing to declare this for liability reasons. The SAE documents discuss the technical requirements necessary for a system to be capable of a certain level, but I dont think that automatically triggers (e.g.) liability until such a time as an actual claim is made by a manufacturer.

Companies have to tell the CA DMV that their system is L4 when they are just seeking a testing permit with safety drivers. So as early as testing with safety drivers, the manufacturer needs to declare it L4 if they want to test on public roads. And the manufacturer would be liable for accidents then. That's why safety drivers are so important because they are responsible for ensuring the safety of the L4 while it is being tested on public roads. But certainly, the system might technically be L4 sooner but the manufacturer could just do testing on closed tracks, and not do public road testing, if they wanted to avoid declaring it L4 and thus avoid liability.

Second, there are a number of posts here that seem to claim that an L2 system cannot evolve into an L4 system over time, as if L4 was fundamentally different rather than a superset/progression from L2 (albeit a large one). These arguments are then (dubiously, imho) used to claim that Tesla FSDb is essentially a dead-end. I think you might make a case that getting to L4 via L2 is not an optimal path, but to say its not possible seems flawed logic to me.

I would clarify that L2 cannot become L4 by simply becoming more reliable or safer. So when Elon or Tesla fans say things like "FSD beta is L2 now because it requires driver supervision. Eventually, it will be reliable enough to not need driver supervision anymore and then Tesla will tell regulators that it is L4". That is false. That is misusing the terms L2 and L4.

But L2 can become L4 or even L5 if the manufacturer adds new driving tasks to the L2 until the system can do the entire DDT and OEDR in a defined ODD. That's the key part. The system can only go from L2 to L4 by adding new driving tasks capabilities. If the system cannot do the entire DDT, simply becoming more reliable will not make it L4.

 

[powertoold]

Waymo apparently can't do the entire complete perfect DDT because it hit the same truck twice! Imagine that!

https://www.cnn.com/2024/02/14/business/waymo-recalls-software-after-two-self-driving-cars-hit-the-same-truck/index.html

 

[AlanSubie4Life]

Waymo apparently can't do the entire complete perfect DDT because it hit the same truck twice! Imagine that!

https://www.cnn.com/2024/02/14/business/waymo-recalls-software-after-two-self-driving-cars-hit-the-same-truck/index.html

It’s a testament to the human spirit that people keep posting here about the possible imminent widespread arrival of > Very Limited L3 vehicles, and about how somehow v12 is a game changer in this regard. As though they had not seen the horribly flawed (but still quite useful in many applications!) performance of extremely high powered LLMs.

 

[Daniel in SD]

Eventually, it will be reliable enough to not need driver supervision anymore and then Tesla will tell regulators that it is L4". That is false. That is misusing the terms L2 and L4.

Except Tesla told the DMV that they use miles per intervention to determine SAE level.

This is why I discount the letter to the DMV as evidence that the design intent of the system is L2. The design intent is obviously L5.
(It is funny how upset people got about the “does not match engineering reality” quote. Seems accurate in hindsight!)

 

[powertoold]

As though they had not seen the horribly flawed (but still quite useful in many applications!) performance of extremely high powered LLMs.

I intuited this prior. V12 involves a very high bitstream input and comparatively small output. The input to output data ratio is on the order of 1,000,000:1 or more.

V12 is therefore very different than the way LLMs work, where the input is a single data packet at a single point in time, followed by a single larger output.

 

[diplomat33]

Waymo apparently can't do the entire complete perfect DDT because it hit the same truck twice! Imagine that!

https://www.cnn.com/2024/02/14/business/waymo-recalls-software-after-two-self-driving-cars-hit-the-same-truck/index.html

You don't understand DDT. DDT means it is capable of doing driving tasks (lane keeping, making turns, detecting objects, reading signs, obeying stop signs and traffic lights, going around double parked vehicles etc...) Waymo can do the complete DDT since it can do all driving tasks in its ODD. But perfection is not a requirement for L4. Are you trolling or just clueless?

 

[drtimhill]

But L2 can become L4 or even L5 if the manufacturer adds new driving tasks to the L2 until the system can do the entire DDT and OEDR in a defined ODD. That's the key part. The system can only go from L2 to L4 by adding new driving tasks capabilities. If the system cannot do the entire DDT, simply becoming more reliable will not make it L4.

Yep, again agreed.

 

[diplomat33]

Except Tesla told the DMV that they use miles per intervention to determine SAE level.
View attachment 1018409
This is why I discount the letter to the DMV as evidence that the design intent of the system is L2. The design intent is obviously L5.
(It is funny how upset people got about the “does not match engineering reality” quote. Seems accurate in hindsight!)

This is why I say that Tesla fans need to pick a lane. Either say that the FSD beta cannot do the entire DDT and therefore is L2. Or say that it can do the entire DDT but still needs driver supervision and therefore is supervised L4. But it can't be both. It can't do the entire DDT (L4) but still be L2 because it requires driver supervision. That is not correct.

 

[AlanSubie4Life]

I intuited this prior. V12 involves a very high bitstream input and comparatively small output. The input to output data ratio is on the order of 1,000,000:1 or more.

V12 is therefore very different than the way LLMs work, where the input is a single data packet at a single point in time, followed by a single larger output.

I don’t see how that architectural difference would automatically lead to much more reliable outputs. I think you underestimate just how precise the output has to be (it’s not just a coarse hash of inputs - seems to me timing of the small set of outputs has to be super precise and smooth and follow certain rules and trajectories, nearly regardless of the inputs). This leaves completely unaddressed the issue of the massive set of inputs.

Yeah it is a different problem, but the approach being used to solve it has similarities to the LLMs, so I expect similar problems to arise. But I am extremely unsophisticated in my understanding of these things, so perhaps I am wrong.

To me, it just seems there’s no reason to think (at least not with the certainty and optimism expressed here) the primitive on-board compute will ever be capable of performing driving tasks, except as a limited L2. I personally have doubts about L3 but of course in a limited ODD, maybe.

But anyway, that is the indomitable human spirit.

 

[stopcrazypp]

It is a L2 ADAS system that people have misused to their own peril and in many cases has led their death.

Sure, you can make that point, but the OP that started the whole discussion was simply saying that the Tesla can move itself around the block with no user intervention when dropped into a random neighborhood in the US, while the Waymo presumably can't.

That then exploded into a whole argument about what "drive" means, and why he was "wrong", but that is largely irrelevant to his point. He even came back to clarify what he meant, but people instead continued to focus on the word "drive".

 

[powertoold]

I don’t see how that architectural difference would automatically lead to much more reliable outputs. I think you underestimate just how precise the output has to be (it’s not just a coarse hash of inputs - seems to me timing of the small set of outputs has to be super precise and smooth and follow certain rules and trajectories, nearly regardless of the inputs). This leaves completely unaddressed the issue of the massive set of inputs.

Yeah it is a different problem, but the approach being used to solve it has similarities to the LLMs, so I expect similar problems to arise. But I am extremely unsophisticated in my understanding of these things, so perhaps I am wrong.

I don't have a good technical understanding of it either, but we've seen ~3 hours of driving from V12, and in none of those videos did it freak out or do something nonsensical. Perhaps you have a different view.

Obviously 3 hours is nothing when it needs to drive thousands without intervention.