Andrej Karpathy - AI for Full-Self Driving (2020)

[diplomat33]

I feel like folks are arguing two points that converge, but are different. Yes HD Maps are nice to have, but someone has to do the mapping first and it has to be done often, otherwise you will have to rely on vision for ground truth, which makes people argue why HD Maps are needed at all.

HD maps and lidar are needed because we cannot do autonomous driving with 99.99999% reliability with camera vision alone.

 

[Ludalicious]

All these arguments back and forth yet no video's of any Tesla's driving on city streets. Funny how Tesla had no problem releasing video's in 2017 showing their progress but yet has showed not much progress since then. Im here just to see videos of their progress and a little disappointed I haven't seen one yet.

 

[willow_hiller]

So can you drive with Lidar ALONE? the answer is an absolute yes.

Random cheap off the shelf automotive grade Lidar ($800)

I don't disagree that LIDAR can see a lot, but surely that first image you posted is a rendering and not real. How can LIDAR alone see painted arrows on the road and a painted crosswalk?

 

[Bladerskb]

I don't disagree that LIDAR can see a lot, but surely that first image you posted is a rendering and not real. How can LIDAR alone see painted arrows on the road and a painted crosswalk?

The lidar data of that picture is the video below. Its not fake. Lane lines, road markings and crosswalk are painted and are reflective so lidar sees it easily. No camera needed.

However Waymo's and Ouster's more advanced Lidar can go even further. They can read road signs and traffic signs but they are not using a camera. They are not hooking up a camera on top of the Lidar and then trying to combine them. But rather as Ouster puts it, "the Lidar IS the Camera!". Because its using the active Lidar sensor it means it works in pitch darkness. Meaning it can read road/traffic signs in pitch darkness. So it sees everything like below as though its broad daylight even if there was absolutely no light.

 

[diplomat33]

The lidar data of that picture is the video below. Its not fake. Lane lines, road markings and crosswalk are painted and are reflective so lidar sees it easily. No camera needed.

However Waymo's and Ouster's more advanced Lidar can go even further. They can read road signs and traffic signs but they are not using a camera. They are not hooking up a camera on top of the Lidar and then trying to combine them. But rather as Ouster puts it, "the Lidar IS the Camera!". Because its using the active Lidar sensor it means it works in pitch darkness. Meaning it can read road/traffic signs in pitch darkness. So it sees everything like below as though its broad daylight even if there was absolutely no light.

Very cool. Thanks for sharing. Honestly, I don't understand why anyone would not include lidar in their sensor package considering how good it is for autonomous driving.

 

[Bitdepth]

A surgeon cannot cut tissue well without a scalpel, the scalpel is required. They cannot perform a heart transplant without a heart lung machine, the HLM is required. It the car cannot maneuver without maps, the map are required, not additional.

Your analogy is illogical and incongruous with the topic. Its like saying a car cannot drive without a tire, well yea? A more apt analogy would be;

Self-driving is akin to the task of Medical diagnosis/Surgery

In medical diagnosis you need lots of tools to aid in the diagnosis just like self driving cars needs multimodal sensors

A medical doctor needs their eyes just like a self-driving car needs cameras

In addition, when a doctor needs to look under a tissue, they need x-rays or an mri which are akin to lidars which see further than cameras and can track objects more accurately.

In addition a medical doctor needs an ultrasound which again is akin to an ultrasonic sensor in cars

A medical doctor would have been trained in the anatomy and physiology of a human body which is akin to a self-driving car needing a trained neural network

Also doctors often use human anatomical models which is akin to a self driving car using a map.

Before every surgical procedure, you get an MRI, X-ray, etc, which the doctor references during the procedure depending on how extensive the procedure is.

 

[willow_hiller]

Very cool. Thanks for sharing. Honestly, I don't understand why anyone would not include lidar in their sensor package considering how good it is for autonomous driving.

Cost has got to be the primary reason. Before Waymo started selling their cheaper LIDAR sensor, how much was each beacon? Some quick searches say the most commonly used module was $75,000. Let's say you can get away with one good sensor right on top of the car, and 80% of every Tesla ever sold included one of those sensors. $75,000 worth of margins out of 800,000 vehicles is $60 billion worth of lost revenue, when Tesla's total market value is $135 billion or so.

Now that LIDAR is cheaper, it does become a "why not" unless there's still some factor we're not accounting for.

 

[nepenthe]

The lidar data of that picture is the video below. Its not fake. Lane lines, road markings and crosswalk are painted and are reflective so lidar sees it easily. No camera needed.

However Waymo's and Ouster's more advanced Lidar can go even further. They can read road signs and traffic signs but they are not using a camera. They are not hooking up a camera on top of the Lidar and then trying to combine them. But rather as Ouster puts it, "the Lidar IS the Camera!". Because its using the active Lidar sensor it means it works in pitch darkness. Meaning it can read road/traffic signs in pitch darkness. So it sees everything like below as though its broad daylight even if there was absolutely no light.

I'm not buying it... Lidar by definition, is a surveying method to measure distance of the light it is projecting when it bounces back. Even if we want to generously refer to a Lidar sensor as 2 channel camera, which can capture monochrome luminescence via the albedo effect along with distance; I am not seeing how this could be used to reliably read a sign or traffic light. Perhaps it could tell you which light is lite up, but not which is green. As well, if a sign is painted the same, or coated after painting, there's no way it's making out anything written on it.

There is no doubt Lidar can provide a rich amount of data, at cm level accuracy, but saying it can read signs or traffic lights is a false equivalency. As what reason is there to not include a extremely cheap vision system to pull in that wealth of data? Perception has to occur either way and you don't need to know how many cm away you are from a traffic light to determine if the light is green or red.

 

[Joel]

The lidar data of that picture is the video below. Its not fake. Lane lines, road markings and crosswalk are painted and are reflective so lidar sees it easily. No camera needed.

However Waymo's and Ouster's more advanced Lidar can go even further. They can read road signs and traffic signs but they are not using a camera. They are not hooking up a camera on top of the Lidar and then trying to combine them. But rather as Ouster puts it, "the Lidar IS the Camera!". Because its using the active Lidar sensor it means it works in pitch darkness. Meaning it can read road/traffic signs in pitch darkness. So it sees everything like below as though its broad daylight even if there was absolutely no light.

Do you have any citations that Waymo is not or does not have to rely on cameras to read traffic lights?

Waymo is stating otherwise. It would be nice for verification.

Learn how Waymo drives - Waymo Help

 

[mtndrew1]

Cost has got to be the primary reason. Before Waymo started selling their cheaper LIDAR sensor, how much was each beacon? Some quick searches say the most commonly used module was $75,000. Let's say you can get away with one good sensor right on top of the car, and 80% of every Tesla ever sold included one of those sensors. $75,000 worth of margins out of 800,000 vehicles is $60 billion worth of lost revenue, when Tesla's total market value is $135 billion or so.

Now that LIDAR is cheaper, it does become a "why not" unless there's still some factor we're not accounting for.

Well it’s also ugly which matters for private vehicle purchases. If you dump the LIDAR unit low in the grille or bumper it’s going to have a tremendous amount of occlusion and will be less useful than vision. It will also add to insurance rates for fender benders. Cameras can be placed almost anywhere on the car with very little impact to styling or small collision costs.

For LIDAR to be meaningfully useful it needs to be high up and/or perched on either side of the vehicle to reduce occlusion from the vehicles ahead of it . This is why that’s exactly how Waymo, Apple, Cruise and others perch this stuff all over the roof of high profile vans and SUVs and that’s a non starter for privately purchased cars, cost aside. Such an arrangement is also not ideal for range, wind noise, and so forth.

Vision must be solved for L5, LIDAR or not, the system must have multiple nines with vision. LIDAR needs placement that isn’t in line with customer preferences if it’s to remain minimally unaffected by occlusion to the degree that it adds ability to vision in urban environments.

 

[diplomat33]

Cost has got to be the primary reason. Before Waymo started selling their cheaper LIDAR sensor, how much was each beacon? Some quick searches say the most commonly used module was $75,000. Let's say you can get away with one good sensor right on top of the car, and 80% of every Tesla ever sold included one of those sensors. $75,000 worth of margins out of 800,000 vehicles is $60 billion worth of lost revenue, when Tesla's total market value is $135 billion or so.

Now that LIDAR is cheaper, it does become a "why not" unless there's still some factor we're not accounting for.

I agree. I think cost was the primary reason. Certainly, at the time when Tesla launched AP2, LIDAR was too expensive for Tesla to consider. So I do understand why cost prevented Tesla from including LIDAR. Tesla needs to sell cars at a profit to stay in business.

Cost is coming down. LIDAR does not ruin the aesthetics of the car as I show below. And LIDAR offers huge benefits in terms of reliable FSD. I see few reasons not to include LIDAR at this point and a lot of reasons to include it. Every serious FSD company is using LlDAR.

Well it’s also ugly which matters for private vehicle purchases. If you dump the LIDAR unit low in the grille or bumper it’s going to have a tremendous amount of occlusion and will be less useful than vision. It will also add to insurance rates for fender benders. Cameras can be placed almost anywhere on the car with very little impact to styling or small collision costs.

For LIDAR to be meaningfully useful it needs to be high up and/or perched on either side of the vehicle to reduce occlusion from the vehicles ahead of it . This is why that’s exactly where Waymo, Apple, Cruise and others perch this stuff all over the roof of high profile vans and SUVs and that’s a non starter for privately purchased cars, cost aside. Such an arrangement is also not ideal for range, wind noise, and so forth.

Lucid Air has LIDAR in the bumpers. It is not ugly or ruin range. LIDAR is still effective.

Byton K-Byte has LIDAR on the sides and in the roof and it is still aerodynamic and stylish. It does not ruin range or look ugly. And the LIDAR is still effective.

So it certainly possible to make a good looking car with LIDAR. So the argument that LIDAR ruins the car is weak IMO.

Also, this is the new Waymo Jaguar i-Pace. Waymo has worked to make the sensors less intrusive and more aerodynamic. Are you telling me that consumers would not hail a robotaxi ride from this car or even buy this just because of the LIDAR pod? Maybe. But aesthetics is subjective. I am sure there are plenty of consumers who would still buy it if it had great range and real L4/L5 autonomous driving. I would certainly buy it if the range was excellent and it had safe, proven, L4/L5 autonomous driving.

 

[NHK X]

Do you guys not know that lidar can detect and classify lane lines, road markings, crosswalks, road edges, curbs, road signs and traffic signs, and possibly light status, etc?

Lidars has always been able to see lane lines, road markings, crosswalks, etc. Because it checks the light reflectivity. Additionally it always has been able to get the geometric shape which includes stop signs, yield signs, anything as you see below.

That alone you can drive ANYWHERE without having to go through a traffic light control. Why? because there is this thing called turn right on red. This allows you to turn right on any intersection with a traffic light. In addition, because you can also turn left away from an intersection into another road or side streets. There is not a destination that i couldn't get to if i avoided going through a traffic light. For example, my work which is 20 miles away i can get through without going through a traffic light. The same is the case for every spot i usually visit.

So can you drive with Lidar ALONE? the answer is an absolute yes.

Random cheap off the shelf automotive grade Lidar ($800)

Now you say, what about construction. A stop sign that is the same as a slow sign or a circular stop sign, etc.
Well this is 2020 not 2010. Some Lidars can read the ambient lighting which allows them to get imaging information. Waymo lidar can do that, Ouster aswell, etc. So it allows them to read road signs and traffic signs. Including traffic light status although that one is the only question mark. But since light status isn't determined by the color of the light but by the position of the illuminance.

So can you drive on Lidar only? again the answer is yes.

Waymo's Lidar which can read road and traffic signs not just get its geometric shape.

Ouster Lidar which can also read road and traffic signs and not just get its geometric shape.

LIDAR can see a lot, in certain conditions. Totally agree but you haven’t really answered the question. Can LIDAR accurately detect stoplight colors, to a degree to make it reliable for self driving with LIDAR alone as you argue?

And the answer that you can make right turns at every intersection is a totally non practical answer for this argument. You want to spin circles all around down town SF?

 

[mtndrew1]

I agree. I think cost was the primary reason. Certainly, at the time when Tesla launched AP2, LIDAR was too expensive for Tesla to consider. So I do understand why cost prevented Tesla from including LIDAR. Tesla needs to sell cars at a profit to stay in business.

Cost is coming down. LIDAR does not ruin the aesthetics of the car as I show below. And LIDAR offers huge benefits in terms of reliable FSD. I see few reasons not to include LIDAR at this point and a lot of reasons to include it. Every serious FSD company is using LlDAR.



Lucid Air has LIDAR in the bumpers. It is not ugly or ruin range. LIDAR is still effective.

Byton K-Byte has LIDAR on the sides and in the roof and it is still aerodynamic and stylish. It does not ruin range or look ugly. And the LIDAR is still effective.

So it certainly possible to make a good looking car with LIDAR. So the argument that LIDAR ruins the car is weak IMO.

Also, this is the new Waymo Jaguar i-Pace. Waymo has worked to make the sensors less intrusive and more aerodynamic. Are you telling me that consumers would not hail a robotaxi ride from this car or even buy this just because of the LIDAR pod? Maybe. But aesthetics is subjective. I am sure there are plenty of consumers who would still buy it if it had great range and real L4/L5 autonomous driving. I would certainly buy it if the range was excellent and it had safe, proven, L4/L5 autonomous driving.

Wow you’re really hooked on LIDAR lol. Goodness.

My thesis is:

1) Vision MUST be solved to multiple nines for any L5 solution, regardless of what other hardware is on the car. LIDAR + sonar + radar alone cannot provide L5. LIDAR alone cannot solve for L5. Nothing can replace vision, only add to it.

2) LIDAR is most useful for a dot cloud in highly complex spaces such as urban environments.

3) For LIDAR to be most effective in complex urban environments it must be perched on top of and/or on the sides of an automobile as in the Waymo examples. Any other LIDAR placement results in severe occlusion when surrounded by other automobiles. This technology relies on light emission and mapping and if it’s one meter behind the bumper of an SUV it will create literally zero useful information beyond what the cheap radar puck already does. Low-mounted LIDAR would be exactly as useful as a camera in the center of the front bumper trying to see the world around it.

4) People buying their own private automobiles will largely be put off by a giant LIDAR puck on the roof and two more hanging off the sides of their car, even if they don’t have to pay extra for the units as compared to cameras (they will). If this solution reduces range and adds wind noise that’s even less desirable.

5) For vehicles sold strictly for commercially generating revenue the above issues won’t matter much and the extra marginal increase in capability might be worth the cost and unsightliness.

Therefore, for people buying cars that they’ll drive every day, if vision is solved to multiple nines it will be the ultimate L5 solution for privately-purchased cars, full stop. Customers will shun unsightly LIDAR units, insurance companies will hate them, and the Monroney will reflect such a solution both in price and fuel economy.

I am not convinced true L5 will ever happen with any technology but if it does I think Tesla is best poised to solve the vision component. Vision, again, MUST be entirely solved for L5 to work. If Tesla does indeed solve vision to multiple nines it will make LIDAR superfluous on privately purchased passenger cars. It will be a crutch for other automakers trying to race towards any solution with solved vision.

 

[diplomat33]

Wow you’re really hooked on LIDAR lol. Goodness.

That's because everything I've seen suggests that lidar is a powerful sensor if you want to achieve autonomous driving to 99.99999% reliability.

Remember, I am not advocating lidar-only for FSD. I am advocating cameras + lidar + radar. I acknowledge that camera vision must be solved. But I believe that camera + radar + lidar is the only way to achieve 99.99999% reliability.

 

[mtndrew1]

That's because everything I've seen suggests that lidar is a powerful sensor if you want to achieve autonomous driving to 99.99999% reliability.

Remember, I am not advocating lidar-only for FSD. I am advocating cameras + lidar + radar. I acknowledge that camera vision must be solved. But I believe that camera + radar + lidar is the only way to achieve 99.99999% reliability.

Noted. Moving on...

 

[diplomat33]

To sum up both sides of the debate:

One side believes that camera vision must be solved for L5 and believes that camera vision can be solved to 99.99999% and when it does, lidar and HD maps become superfluous. So they conclude a camera-only approach is the best approach and the focus should be on data and machine learning to solve camera vision to 99.99999%. This is the "Tesla approach". This side is fervently believes that the Tesla approach is the winning approach.

The other side argues that camera vision cannot be solved to 99.99999%. Therefore a combined sensor approach of cameras + lidar + radar + HD maps must be used. They believe that this multi sensor approach is required to achieve 99.99999%. This is the "Waymo + everybody else" approach. And this side fervently believes that their approach is the winning approach.

 

[NHK X]

To sum up both sides of the debate:

One side believes that camera vision must be solved for L5 and believes that camera vision can be solved to 99.99999% and when it does, lidar and HD maps become superfluous. So they conclude a camera-only approach is the best approach and the focus should be on data and machine learning to solve camera vision to 99.99999%. This is the "Tesla approach". This side is fervently believes that the Tesla approach is the winning approach.

The other side argues that camera vision cannot be solved to 99.99999%. Therefore a combined sensor approach of cameras + lidar + radar + HD maps must be used. They believe that this multi sensor approach is required to achieve 99.99999%. This is the "Waymo + everybody else" approach. And this side fervently believes that their approach is the winning approach.

Pretty much the same dead horse we like to keep kicking.

Question to you all. Suppose either party is wrong, how hard is it retrain a neutral net to either include more detectors vs subtracting and relying on just one? What’s the implications of being wrong (not from a cost or safety perspective) but time spent redeveloping machine learning algorithms. Does one side have an advantage in the case they are wrong?

 

[willow_hiller]

Question to you all. Suppose either party is wrong, how hard is it retrain a neutral net to either include more detectors vs subtracting and relying on just one? What’s the implications of being wrong (not from a cost or safety perspective) but time spent redeveloping machine learning algorithms. Does one side have an advantage in the case they are wrong?

This is a fun new direction to take this debate in.

I would argue that Waymo probably has the driving policy advantage, but Tesla has the edge-case training data advantage. Waymo has this advantage because they've used HD maps to bypass the neural net training stage that Tesla is in right now and focus on the nitty gritty "What do you do in X situation" programming. But Tesla either has, or will have, the training data advantage. You can't just show a neural net a few examples of odd edge cases and expect it to correctly identify them. If you give your network 99,000 stop signs and 100 "except right turn" signs, that bias in the training data will make it difficult for the network to correctly identify that edge case.

 

[mtndrew1]

To sum up both sides of the debate:

One side believes that camera vision must be solved for L5 and believes that camera vision can be solved to 99.99999% and when it does, lidar and HD maps become superfluous. So they conclude a camera-only approach is the best approach and the focus should be on data and machine learning to solve camera vision to 99.99999%. This is the "Tesla approach". This side is fervently believes that the Tesla approach is the winning approach.

The other side argues that camera vision cannot be solved to 99.99999%. Therefore a combined sensor approach of cameras + lidar + radar + HD maps must be used. They believe that this multi sensor approach is required to achieve 99.99999%. This is the "Waymo + everybody else" approach. And this side fervently believes that their approach is the winning approach.

Perfectly summarized.

First one with a car that I can buy which allows me to sleep from my garage at home to my office, wins.

 

[DanCar]

To sum up both sides of the debate:

One side believes that camera vision must be solved for L5 and believes that camera vision can be solved to 99.99999% and when it does, lidar and HD maps become superfluous. So they conclude a camera-only approach is the best approach and the focus should be on data and machine learning to solve camera vision to 99.99999%. This is the "Tesla approach". This side is fervently believes that the Tesla approach is the winning approach.

The other side argues that camera vision cannot be solved to 99.99999%. Therefore a combined sensor approach of cameras + lidar + radar + HD maps must be used. They believe that this multi sensor approach is required to achieve 99.99999%. This is the "Waymo + everybody else" approach. And this side fervently believes that their approach is the winning approach.

Although I believe Waymo is correct, I believe Tesla will win because:

1. Tesla will take more blood risk than Waymo. Tesla needs fewer nine(s).
2. Tesla will be the perceived winner if they introduce level 3 for freeways before Waymo has a level 4 winner.

Waymo isn't done adding sensors. If they haven't done already, I'll bet they will add thermal sensors and more. Also Waymo has some sort of remote pilot capability. Will one of them add depth sensing cameras using PDAF array? Will Tesla add full RGB camera(s) at some point? Do both have microphones?