Opinion: what I think Tesla should do to "solve" FSD

First, I want to acknowledge all the hard work of the Tesla FSD team. Tesla has spent years building a sophisticated vision-only system. And the perception part is very advanced. I am not saying that Tesla Vision is perfect. There are still gaps in the perception system. But I feel like Tesla has build a pretty good foundation for FSD. I am not suggesting Tesla start from scratch. On the contrary, I think Tesla should continue to build on that vision-only foundation.

But here are 3 things that I think Tesla should do in order to deploy a more reliable and more robust FSD system.

TLDR Tesla should copy Mobileye.

1) Crowdsourced maps
Tesla has a big fleet of vehicles on the road. They could leverage the vision system in every car on the road to crowdsource detailed maps, similar to what Mobileye is doing. With such a large fleet of vehicles, Tesla could map large areas pretty quickly. Tesla could probably map every road in the US in a relatively short time. And with such a large fleet of vehicles on the road, Tesla could also update the maps pretty easily too since there would always be a Tesla somewhere checking the maps. A lot of the errors that FSD beta makes seem to be due to poor map data. Crowdsourcing could really help solve those issues since there would be a Tesla likely checking that spot pretty regularly. And detailed maps could help make FSD more robust. With crowdsourcing, only the first car would need to drive the road mapless, other cars that encounter the road later, would have the benefit of a map as a prior. And detailed maps can provide useful non-visual info like slowing down for a bend in the road that you can't see because of obstacles or preferred traffic speed different from actual speed limit.

2) Driving Policy
Tesla has done a lot of work with perception but one area where FSD Beta is very weak, is driving policy IMO. For example, FSD beta is poor at knowing when to change lanes when traffic is dense to avoid not missing an exit. FSD beta can wait too long and then miss its chance to merge because of dense traffic. Also, FSD beta can be overly cautious at intersections when there is no traffic at all. FSD beta can also be too shy when going from a stop sign or too aggressive when making unprotected left turns. These are issues that better driving policy would help with. It would improve the driving decisions of the car and make for a safer and smoother ride. Mobileye has a very good RSS policy that helps the car drive safely. So I think Tesla needs to focus more on driving policy. I think FSD Beta would benefit greatly from a driving policy.

3) Sensor redundancy
I think Tesla is smart to focus on vision-only. This is important as a foundation for perception. And I think vision-only will work great for L2 "door to door". So what I am proposing is that Tesla continue to do vision-only for L2 but work on a lidar-radar subsystem that could be added on top of the existing vision-only FSD system to provide extra reliability and redundancy, that could help get the system to "eyes off". This is essentially what Mobileye is doing and I think it is smart. I think vision-only is fine for L2 but having radar-lidar as a back-up is crucial for "eyes off". This because in order to do "eyes off", you really need to be able to trust the system to be super reliable in all conditions. Vision-only cannot do that. With vision-only, if the cameras fail, the entire system will fail or need to pull over. But with cameras, radar and lidar, your system is less likely to fail if the cameras fails. I think having extra sensors as back-up will really help to reach that extra reliability.

"Full Self Driving Tesla" by rulenumberone2 is licensed under CC BY 2.0.
Admin note: Image added for Blog Feed thumbnail

 

[dgatwood]

If you took an average 16 year old child, gave them two days of instruction, and told them to drive around NYC until they have a crash, and FSD to do the same, the kid would last longer.

Out of my interventions that seemed like safety problems initially, nearly all of them actually weren't safety problems with the benefit of hindsight, though I have had to intervene to keep it from trying to pass a long line of stopped cars at a stop sign a few times, and that's kind of bad.

While I agree that the kid would be two orders of magnitude more likely to reach any given destination, that's not because of crashes. Rather, FSD would just keep making wrong turns, getting in the wrong lane, and driving around aimlessly, or else would just plain stop for no reason and be too scared to start again.

FSD's lane planning is seriously problematic, and it drives like a 16-year-old who has never been behind the wheel before and is scared to make a mistake, but driving in SF yesterday, there were times when the path through the street was so narrow that I was scared to drive it without FSD beta being in charge. I was confident that it was less likely to cause a wreck than I was. When it got to one that was so narrow that it had to retract the mirrors, I was just about crapping my pants, but there was no way in h*** that I was going to even think about intervening.

So it's very much a mixed bag, and I'd hesitate to bet on which one would last longer.

 

[diplomat33]

RADAR can potentially handle heavy rain better than vision, of course, but there are fundamental limitations that preclude driving without usable vision, including being unable to read signs, determine traffic light colors, etc., so it is an open question whether adding high-definition RADAR would make enough difference to matter, except perhaps in terms of slightly improving safety in the first few seconds after rain suddenly starts pouring hard enough that the vehicle has to pull over to the side of the road.

The benefit of radar is that it will give you very precise distance and velocity measurements of objects, even when visibility is reduced like in rain or snow. This will greatly help in collision avoidance. Having an extra sensor, like radar, is a good idea so you are not dependent on just vision alone for collision avoidance. Even on a clear day, there can be cases where vision alone might not properly detect an object. And of course, in rain or snow where visibility is reduced, radar will help with collision avoidance even more. So radar can help in all conditions. And on highway driving, there are no traffic lights or stop signs so that is a moot point. Radar can help a lot with collision avoidance on highways.

The point of radar is to supplement vision, to make the driving safer in all conditions. If vision fails, then you will need to pull over no matter what. But even when vision works, it is useful to supplement it with radar and/or lidar, to make it MORE reliable.

The whole point of radar (and lidar) is not to replace vision but to make vision MORE reliable because in order to do safe driverless, you need higher reliability than what vision-alone can achieve.

Notice that Mobileye includes imaging radar for all their L4 products (Chauffeur, Drive).

Even Wayve, which is doing E2E autonomy, includes radar is their autonomous driving:

Wayve AI Driver uses camera and radar data from the car’s sensors to detect objects and obstacles around the vehicle, predict how other road users will move, and then perform a safety-verified motion plan for driving.

Source: Wayve Driver – AI software & fleet learning platform

Tesla is probably the only company claiming they can do L4 without radar. Everyone else uses radar. The notion that radar isn't really useful is just nonsense. Nobody in the AV industry agrees with that.

 

[dgatwood]

The benefit of radar is that it will give you very precise distance and velocity measurements of objects, even when visibility is reduced like in rain or snow. This will greatly help in collision avoidance. Having an extra sensor, like radar, is a good idea so you are not dependent on just vision alone for collision avoidance. Even on a clear day, there can be cases where vision alone might not properly detect an object. And of course, in rain or snow where visibility is reduced, radar will help with collision avoidance even more.

I'll grant you rain, but in frozen weather, your bumper quickly ends up with such a thick layer of ice that RADAR becomes useless, and if not properly ignored, can actually do more harm than good.

And what do you do when (not if) RADAR fails? Do you refuse to drive until someone clears the ice from the bumper? Because if so, you've now made the system as a whole less reliable overall. Does the improvement in safety make up for the rather significant loss of reliability in winter months in more than half of the U.S.? Is there actual data proving that? If not, then the best thing we can do is have a bunch of companies all doing their own thing, and see what approach ends up working best.

Tesla is probably the only company claiming they can do L4 without radar. Everyone else uses radar. The notion that radar isn't really useful is just nonsense. Nobody in the AV industry agrees with that.

Every other company used CCS until a few weeks ago, too. That's not a good indication that it is a superior system. Everyone uses RADAR in part because they have to have it anyway for cars' automatic emergency braking system, which is likely separate from the ADAS and bought off-the-shelf. If Tesla doesn't get buried in lawsuits over AEB with vision, you can safely assume that at least a few other companies will try a vision-only approach in the coming years.

BTW, Toyota agrees with Tesla, at least as a long-term direction.

 

[diplomat33]

I'll grant you rain, but in frozen weather, your bumper quickly ends up with such a thick layer of ice that RADAR becomes useless, and if not properly ignored, can actually do more harm than good.

And what do you do when (not if) RADAR fails? Do you refuse to drive until someone clears the ice from the bumper? Because if so, you've now made the system as a whole less reliable overall. Does the improvement in safety make up for the rather significant loss of reliability in winter months in more than half of the U.S.? Is there actual data proving that? If not, then the best thing we can do is have a bunch of companies all doing their own thing, and see what approach ends up working best.

Uh!?! AVs have heating elements to melt the ice and prevent it from accumulating on the radar. Easy fix. with a heating element to melt the ice off, there won't be a significant loss of reliability in winter.

And ice is not a problem unique to radar. Ice can also accumulate on cameras. And ice on cameras will prevent a vision-only system from working during winter weather. So ice is not a problem unique to radar. That is why all sensors need heating elements to solve this problem. So the argument that we should not use radar because of ice does not make sense, since ice would also affect cameras, and there is an easy solution. Why would we forego the benefits of radar when there is an easy solution to keep radar working in winter, the same solution that is required anyway to keep cameras working in winter?

 

[dgatwood]

Uh!?! AVs have heating elements to melt the ice and prevent it from accumulating on the radar. Easy fix. with a heating element to melt the ice off, there won't be a significant loss of reliability in winter.

I have my doubts that heaters are a complete solution. There's only so much that a heating element can do when the outside of the bumper is being super-cooled by traveling through 5-degree Fahrenheit air at 65 MPH, and then gets hit by a hunk of packed snow thrown up by the wheel of a passing semi. That's almost certainly going to stick and stay stuck for a while before it melts enough to blow off. And the closer it is to the road, the more debris will stick in general, whether it is snow or mud or deer fur or whatever.

And ice is not a problem unique to radar. Ice can also accumulate on cameras.

Ice accumulating on cameras mostly occurs while the car is parked, with the exception of the windshield, which has both heat and mechanical wipers to clear away any accumulation. Put mechanical wipers on the RADAR units and we'll talk.

Maybe having high-res RADAR with vision is a clear win over vision alone. Maybe it isn't. In a decade or so, we'll have enough data from enough systems in both configurations to have a good idea.

 

[Fairchild]

Continue to copy mobileye. @Bladerskb

Wow. I had been away for a while and genuinely expected to get blasted for my comments. I am so happy to see so many thumbs up, supporting my brand of sanity. One user told me, that from his perspective, the acronym FSD actually stood for Forget Safe Driving.

 

[D Good]

I’ll never forget about 2 years ago I was driving at 40 mph on an extremely foggy road at night with autopilot on. I was staring at the road ready to brake for cars at any moment. All of a sudden, autopilot slammed on the brakes. I had no idea what it was stopping for but I let it do its thing.

It turns out a black SUV was broken down right in the middle of my lane with their lights off.
I very much believe that autopilot saved my life that night. There is no way a vision only system would have seen the car in time.

 

[aset]

I’ll never forget about 2 years ago I was driving at 40 mph on an extremely foggy road at night with autopilot on. I was staring at the road ready to brake for cars at any moment. All of a sudden, autopilot slammed on the brakes. I had no idea what it was stopping for but I let it do its thing.

It turns out a black SUV was broken down right in the middle of my lane with their lights off.
I very much believe that autopilot saved my life that night. There is no way a vision only system would have seen the car in time.

I don’t see a problem with vision-only L5 reducing speed to 5-10mph. Slowing down is less costly than radar/LiDAR.

 

[ninefiveone]

If you think you’re so much smarter than the Tesla FSD engineers, please go work there. I guarantee that they will compensate you far better than your current job to illuminate them with your intelligence.

No? Then stop second guessing the engineers who are and have been working on this full time and let them do their thing the way they think best.

Ignore what Elon says and thinks. He’s not writing a line of code here.

 

[Kognos]

Elon said the reason they dropped radar was because it was much less accurate and it kept confusing the vision system.

What Elon says isnt often correct.

Also, it's funny to me that some problems are like "we can totally fix that; programmers are idiots if they can't figure it out!" and others are "its so confusing..." I mean, it's code. They can figure it out. From a simple project management standpoint, they had it right at the start, with two separate systems that worked in tandem in case one of them ever got out of sync, Total reliance on one system is a bad idea.

 

[nickkovach]

If you think you’re so much smarter than the Tesla FSD engineers, please go work there. I guarantee that they will compensate you far better than your current job to illuminate them with your intelligence.

No? Then stop second guessing the engineers who are and have been working on this full time and let them do their thing the way they think best.

Ignore what Elon says and thinks. He’s not writing a line of code here.

I personally give the engineers a break on this because if you really look at how inconsistent the road signs are, the lines are sometimes painted on the roads and other times just a bunch of dots or metal lines when they haven’t painted lines yet. My point is that it’s confusing as hell for a human sometimes to understand what is required as a driver.. a computer can’t always figure it out either.

The only thing I agree in this discussion is an HD radar could be very helpful if the software can utilize the data effectively. Wish mine had that HD radar (since I don’t even have ultrasonics.. feel like I’m putting way too much faith in vision only)

 

[kabin]

If you think you’re so much smarter than the Tesla FSD engineers, please go work there. I guarantee that they will compensate you far better than your current job to illuminate them with your intelligence.

No? Then stop second guessing the engineers who are and have been working on this full time and let them do their thing the way they think best.

Ignore what Elon says and thinks. He’s not writing a line of code here.

Of course this is an enthusiast forum intended for tesla owners etc to present their own opinions. Heck you can even give your opinion to stifle other's opinions. It's all good.

 

[kabin]

What Elon says isnt often correct.

Also, it's funny to me that some problems are like "we can totally fix that; programmers are idiots if they can't figure it out!" and others are "its so confusing..." I mean, it's code. They can figure it out. From a simple project management standpoint, they had it right at the start, with two separate systems that worked in tandem in case one of them ever got out of sync, Total reliance on one system is a bad idea.

Tesla seems to suffer from a 'not designed here' mentality so much so they ignore reality and need to create fantasies like mindblowing, something happening by the end of the year, and OEMs are interested in FSD licensing. I wouldn't be surprised if Tesla falls so far behind the competition they need to license competitor technology.

 

[dgatwood]

What Elon says isnt often correct.

In this case, it was, though. It wasn't just Musk saying it. Their RADAR was low-resolution hardware designed for emergency braking and not much else. It had imprecision that was severe enough to make guard rails appear to be within the lane bounds under a lot of circumstances, and removing the input substantially improved overall precision.

Also, it's funny to me that some problems are like "we can totally fix that; programmers are idiots if they can't figure it out!" and others are "its so confusing..." I mean, it's code. They can figure it out.

It's not code. It's data. Code can be reasoned through. Data is coming at you continuously, and must be dealt with or ignored.

When you have a high-precision data source and blend it with a low-precision data source, the only possible outcome is a reduction in the overall precision compared with using the high-precision source by itself. Moving to two high-precision sources is useful for redundancy, particularly in a few very specific edge cases (e.g. heavy rain), but adding garbage data to a good signal is almost never a good idea unless you're intentionally trying to reduce the quality for some reason (e.g. dithering).

Sensor fusion is well understood at this point, so if they can't fuse good data with vision, then we should question whether they know what they are doing. But that doesn't mean they should continue trying to fuse borderline noise with actual data.

 

[Bitdepth]

In this case, it was, though. It wasn't just Musk saying it. Their RADAR was low-resolution hardware designed for emergency braking and not much else. It had imprecision that was severe enough to make guard rails appear to be within the lane bounds under a lot of circumstances, and removing the input substantially improved overall precision.

He wasn't. You can't use a chainsaw to perform surgery then conclude well a chainsaw is not meant for surgery and expect to be path on the back for realizing that when everyone was saying a chainsaw is not meant for surgery. Mercedes, Nissan, Audi and others were using those same radars as intended for ACC, Emergency Braking. It is literally on the product page and description that those radars were suitable for only those functions, but Tesla insisted on using those radars for autonomous driving and then concluded maybe they weren't meant for autonomous driving after all.

AP 1.0 - 2.0

Bosh MRREVO14F

Predictive emergency braking system
Adaptive cruise control
Heading distance indicator

AP 2.5 - 3.0

Continental AR410

Advanced Radar Sensor 410
The Advanced Radar Sensor 410 realized a broad field of view by two independent scans in conjunction with the high range functions like Adaptive Cruise Control, Forward Collision Warning and Emergency Brake Assist can be easily implemented.

 

[SalisburySam]

I wish our much-weakened consumer protection regulators had enacted regulations prohibiting the sale of vaporware. Would have saved me $3000 back in 2018. Oh well, that whole caveat emptor thingie I guess. How many millions have now fallen for that?

 

[aset]

In this case, it was, though. It wasn't just Musk saying it. Their RADAR was low-resolution hardware designed for emergency braking and not much else. It had imprecision that was severe enough to make guard rails appear to be within the lane bounds under a lot of circumstances, and removing the input substantially improved overall precision.




It's not code. It's data. Code can be reasoned through. Data is coming at you continuously, and must be dealt with or ignored.

When you have a high-precision data source and blend it with a low-precision data source, the only possible outcome is a reduction in the overall precision compared with using the high-precision source by itself. Moving to two high-precision sources is useful for redundancy, particularly in a few very specific edge cases (e.g. heavy rain), but adding garbage data to a good signal is almost never a good idea unless you're intentionally trying to reduce the quality for some reason (e.g. dithering).

Sensor fusion is well understood at this point, so if they can't fuse good data with vision, then we should question whether they know what they are doing. But that doesn't mean they should continue trying to fuse borderline noise with actual data.

As far as I know, doppler radar is not designed to detect stationary objects at all.

Where do people get that confidence saying hi-res radar data is good?

How about fusion being "well understood"? "Solved"?

I tried googling for vision/radar fusion paper and the first one says radar manufacturers are still fighting noise by assigning thousands of low-paid workers for labeling.

 

[diplomat33]

If anyone is interested in sensor fusion, here is a nice interview with Ghost Autonomy on how they do late fusion of cameras and radar:

 

[dgatwood]

As far as I know, doppler radar is not designed to detect stationary objects at all.

That's not really true. RADAR detects reflections without regard to whether the reflection comes from something moving or stationary. The "doppler" part just refers to the fact that it measures the frequency of the reflection and can use that to determine the speed of the object relative to the transmitter.

Navigation RADAR typically throws away reflections that aren't moving, because the ground under you is considered clutter (and to some degree, that's true for weather RADAR as well). You don't care about things that aren't moving if they are part of the ground, even if the ground goes up ahead of you, because you don't want to panic brake every time the road has a dip.

Of course, without high enough resolution to tell the difference between the ground sloping up a hundred feet away and a parked firetruck a hundred feet away, that can be a problem.

Where do people get that confidence saying hi-res radar data is good?

It's unclear how good high-def RADAR is, and the answer will vary from manufacturer to manufacturer, but it is clearly better than what we have on our cars.

How about fusion being "well understood"? "Solved"?

Nothing is ever "solved", but there's decades of research on sensor fusion, and although I'm sure there will always be room for improvement, multiple companies are doing a good enough job of it to pull off L4 autonomy. I would consider that to be a well-understood problem.

I tried googling for vision/radar fusion paper and the first one says radar manufacturers are still fighting noise by assigning thousands of low-paid workers for labeling.

Not sure what paper you're talking to, but paying people to do data labeling is a necessary first step in supervised learning, which is pretty much required for sensor fusion. That's not evidence of bad data. It's just the way the process works. And I suspect that at least some of that has to be repeated when the RADAR hardware changes enough for the data to look significantly different.

 

[aset]

It's unclear how good high-def RADAR is, and the answer will vary from manufacturer to manufacturer, but it is clearly better than what we have on our cars.


Not sure what paper you're talking to, but paying people to do data labeling is a necessary first step in supervised learning, which is pretty much required for sensor fusion. That's not evidence of bad data. It's just the way the process works. And I suspect that at least some of that has to be repeated when the RADAR hardware changes enough for the data to look significantly different.

Guess where all the data and results are crystal clear? Mobileye powerpoints

I briefly went through this paper: [2304.10410] Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review
Hope the quote below gives the taste how challenging radar de-noising currently is, and how high margin of error could potentially be introduced, causing fused data to have more uncertainty:

b) Difficulties in Labeling: The process of labeling data is labor-intensive and time-consuming, especially when dealing with multi-modal data. This is particularly true for radar camera fusion, where the physical shapes of objects cannot be discerned directly from the radar data representation. InRadarScene [191], all the radar points are labeled by human experts. However, only moving objects are annotated, while the static points are ignored. Auto-labeling radar data is a potential research direction to address the challenge of laborious data labeling. Actually, labels for radar data can be calculated based on the corresponding ground truth from camera images and the extrinsic matrix between the radar sensor and the camera sensor. But the problem is that applying this labeling approach for radar data is not perfect, as radar targets may not always be located in the ground truth from images. Sengupta et al. [281] proposed a camera-aided method for automatically labeling radar point clouds, leveraging a pretrained YOLOv3 [55] network and the Hungarian algorithm for enhanced accuracy and efficiency. However, despite the potential advantages of auto-labeling radar data, filtering out noisy data around the object of interest is still challenging.

It also has some points on temporal drift, which is also a big fusion challenge, introducing more latency and errors.

My opinion is unchanged: adding more sources of data, denoising and fusing it in isolation simply reduces quality. Even if radar/lidar data is near perfect, fusion will lose precision.