This is a great discussion, and probably headed a bit off topic. I would love to pursue it!I think you might be overestimating the kind of intelligence that an AI system will need in order to safety drive a car. There's a computer scientist named Ajeya Cotra that does some interesting research on forecasting when AI will become transformative by comparing it to biological anchors. She was interviewed by Freakonomics, and the episode is definitely worth a listen: New Technologies Always Scare Us. Is A.I. Any Different? - Freakonomics
By her team's estimates, GPT2, with 1.5 billion parameters, had intelligence equivalent to a honey bee. And GPT4, with an estimated 1.8 trillion parameters, has the intelligence of a squirrel.
The last estimate of the model size of FSD we had came pre-V12, but it was 1 billion parameters at the time. So maybe moving planning and control has increased the model size, but we are currently working with something that's likely closer to honey bee intelligence than squirrel or even human.
All that being said, think about what bees can do with their limited brains. They can navigate a hive, with a complex series of chambers. They can fly at high speed, avoiding obstacles and seeking certain targets. They might not know what a bear is, but they know not to fly into one, or to sting it if it attacks their hive for honey.
That said, I would not give the task of driving my car to a honeybee. It is wonderful, but it will never be able to anticipate the hard stuff: the likelihood of a kid running out between two cars, or navigate the best path through 100 potholes, even.
It's the last 10% that it can't do that's critical, and FSD as it exists isn't close. And that's where it's stuck, and that's why it's just an algorithm and why that's all it will ever be. Not intelligent.