HW2.5 capabilities

[AnxietyRanger]

You can keep piling on sensors. Add thermal imaging. Add microphones. The question is: how many people will die because you waited three years for your sensor suite to be cheap enough to include in a $35,000 car? If Tesla’s Hardware 2 suite enables full self-driving at 10x average human safety, isn’t that good enough to deploy?

Well, first of all, as a customer of Model S/X, I naturally have a higher price-point in mind than $35,000. Tesla certainly could afford to add more redundancy in a higher-priced car. If they want to do everything by the lowest common denominator, they will hurt for it eventually.

The other question is, how much can autonomous progress be delayed if regulation or public perception is hurt by some spectacular crash in an autonomous car that wasn't redundant enough? I do get the benefit of Silicon Valley's rapid iteration and quick/dirty agility, but it has its downsides.

And even on its upsides I can see them totally iterating on this. So I expect even Tesla will keep piling on sensors, probably even to Model 3. What we got as an AP2 was the cheapest possible bare minimum for some kind of self-driving at the time. Not even Tesla is aiming at 10x for AP2/2.5. The figure they have been using is at least 2x as safe. (I don't blame someone for being confused as Tesla has been using the 10x figure carelessly, perhaps intentionally muddying the picture, but really when it comes to AP2 specifically, they have been more carefully to say twice as...)

That said, I think there is real, genuine concern that AP2 will not be anywhere near 2x as safe as humans, nor get regulatory FSD approval, in many conditions around the world. Perhaps it will in some fair-weather states. Last year we were told AP2 could be summoned from the other side of the country, drive without a driver to pick you up. I have a seriously hard time seeing that happening if the trip included, say, winter weather without a person in the car to clean it up.

AP2 has no long-distance or high-speed redundancy towards the sides or in the rear. It has no mechanism to clean (other than heating) five of its cameras, including the serious dirt-magnet that is the rear-camera.

So, consider me sceptical that this suite would somehow be optimal (or perhaps even sufficient) for Level 5 capability (as it was advertised by Elon on launch) which is basically driverless driving in most conditions. For Level 5, we will likely need much more redundancy.

To summarize, the choice is not between e.g. lidar or no lidar. It’s between deploying self-driving hardware in 2016 or in 2019 (or later). You can always add lidar later. But you can’t deploy in 2016 with lidar.

But that's not the argument. The argument is, could you have added rear and side long-distance redundancy in 2016. Yes, totally. By adding corner radars. Lidar is a red-herring.

 

[Saghost]

But that's not the argument. The argument is, could you have added rear and side long-distance redundancy in 2016. Yes, totally. By adding corner radars. Lidar is a red-herring.

360 degree LIDAR was your argument...

 

[BigD0g]

Take the lidar conversation elsewhere, clearly HW 2.5 doesn't have Lidar, so maybe start a HW3.0 speculation Lidar thread..... Come on guys.

 

[AnxietyRanger]

360 degree LIDAR was your argument...

Well, no, to be clear it has always been about sensor technology redundancy in general for me. I think I've been consistent about it. Anything else is a misunderstanding.

 

[AnxietyRanger]

Take the lidar conversation elsewhere, clearly HW 2.5 doesn't have Lidar, so maybe start a HW3.0 speculation Lidar thread..... Come on guys.

Could Model 3 HW 2.5 have rear corner radars? Probably not.

 

[gearchruncher]

Sorry if I missed it, but can anyone explain what all the NN's in the system are there for? As far as I know, AP has the same features as earlier software, so why put a NN in shipping software if it isn't hooked to anything?

 

[cronosx]

For now, they "simply" replace mobileye, what they are tring to figure out if it does something 'extra' that mobileye didn't do and how good are the system in general

 

[BigD0g]

Right now, the NN's are basically acting as a emulation for MobileEye hardware. However as they grow/learn and get better that emulation layer if done correctly will morph into the FSD vision solution. The emulation is a stop gap I suspect, until the Vision really takes flight sometime next year.

 

[BeesKnees]

Or, just for fun, let’s take the extreme version of the question. Imagine there is code really not used (yet). Getting the deployment pipeline all in place from unit test through staging and field deploy is real work. Getting into the mainline deploy is great.

I know of a project that chose a separate test stack approach. After a year they had so many versions of the system in test the velocity of the overall project was in trouble.

On a modern project I’m doing we just deployed an ecommerce engine that the customer can not see. Deploys great. Check. When the rest of the functionality comes on line, we’re already to go.

Integrate early. Integrate often. It could be just that.

 

[Reciprocity]

Well, no, to be clear it has always been about sensor technology redundancy in general for me. I think I've been consistent about it. Anything else is a misunderstanding.

More is not better. Better is better. To much sensor data can actually cause problems. Which is is right? You have to decide which means there will be times when when false positives occur which can be as bad as false negatives. I am not some kind of sensor engineer, but my guess is that Tesla decided that Radar and Lidar where to similar and would often miss the same things. The key is Vision, which is actually a good proxy for lasers because both are based on light bouncing off a target. If you could make a perfect vision system, it would be far superior to anything that Lidar or Radar could do, but its extremely hard because there are an infinite number of objects to identify and determine their intent is even more difficult. Even peoples eyes/brain play tricks on them. Making a great sensor mesh that is more complimentary then conflicting is key. I will agree that in the end, we probably need V2V communications and V2R - Road or environment to Vehicle communications. The mapping and tracking of whats going on in an area could be communicated to vehicles as they approach so they actually know whats going on and what to watch before they get there.

 

[Cirrus MS100D]

A brief anecdotal note from a 300 mile drive yesterday in the unrelenting east coast rain in an AP2 car with 2017.42

First and foremost, I believe this is the best version yet. On the trip up (in the clear, day and night) it performed wonderfully on the highway and even on rural-ish roads.

But more interesting to me was how it performed yesterday in the rain that oscillated between “this rain is insane?!” to “OMG I think we’re going to die.” AP2 on the highway did a phenomenal job, even during times when I wasn 100% sure *I* could see the road markings. This was mostly 95 through RI, CT, NY, NJ.

I will note that auto lane change did NOT work all in heavy rain. (Yet I had no issues on the way up in the clear.) The dash did indicate AP2 “saw” the other lanes, but it makes me wonder what it may be using to determine whether or not it feels confident enough to execute a lane change? I realize @verygreen indicated the side cameras still aren’t in use in 2017.42, but clearly it didn’t feel confident enough when the rain was heavy (maybe it simply couldn’t see the other lanes well enough with the front cameras?)

I will also add that the rear camera was largely unusable in heavy rain and above (as we already knew), but as I mentioned my thoughts in another thread on the topic, I wouldn’t have faulted an “L5 designated” car from pulling over in some instances yesterday that my stupid human brain (and thousands of others like mine...) persisted and pushed on through with such low visibility.

I was just shocked at how it simply shrugged off such poor visibility - I even drafted behind a tractor trailer for a bit (thanks radar) but the way it held the lanes was remarkable IMO.

 

[ABC2D]

@Reciprocity , @AnxietyRanger ,et al

could you PLEASE debate your theorizing in another thread so we have a slight chance for @verygreen @jimmy_d to continue ...

 

[J1mbo]

I will note that auto lane change did NOT work all in heavy rain. (Yet I had no issues on the way up in the clear.) The dash did indicate AP2 “saw” the other lanes, but it makes me wonder what it may be using to determine whether or not it feels confident enough to execute a lane change?

I think it uses the ultrasonics for autolane change, that rain sounds like it would have been more than enough to defeat them...

 

[thomasmb]

I had quite a different experience the other day. Drove at night time through horrible weather, wishing for another step up for the wipers since the highest interval it wasn't enough to keep up. Autopilot worked way beyond what I expected – especially since AP1 often phoned it in when raining heavily. But anyway, I drove for about 100km in that kind of weather and did numerous lane-changes. Only once did I have to intervene halfway through a lane change, as it became unsure. But, .42 was a little less stable than .40 which I had used a couple of days earlier in similar weather. Maybe random? I also kept an eye out for the ultrasonics, as our AP1 car often registered something behind the car when driving in wet conditions, but as far as I can tell it wasn't really bothered by it – at least not consistently.

 

[stopcrazypp]

Shutter speed related motion blur is entirely different from the rolling shutter effect.

Rolling shutter - Wikipedia

Essentially with a rolling shutter not all parts of the image are recorded at the same time.

Yes, I know that, which is why I didn't say rolling shutter. I was going to do another post on rolling shutter looking at the AR0132 datasheet but was busy over the weekend. I post in the camera thread instead, since it's kind of off topic for this thread.

 

[jimmy_d]

This may at first seem like a digression, but bear with me a moment.

An issue that comes up in the topic of HW2.5 capabilities is the notion that extracting depth information using a camera is not possible, and that without depth information driving cannot be done well. This reflects on HW2.5's *ultimate* capabilities because, aside from low resolution radar and limited range sonar, HW2.5 appears to not be equipped with sensors that can provide essential depth information at the scale needed for general vehicle operations.

It's an understandable concern. Nonetheless, extracting depth information from a camera image is doable and the technology is getting better at a rapid pace. As recently as 3 years ago this was not possible, but every year since the capability has grown by leaps and bounds. Much of it is so new that it has not yet made it's way into commercial products. I'm looking for evidence that it has been included in recent HW2.5 networks but as yet I haven't got anything firm. The basic technique is probably already good enough to function as a substitute for the essential services provided by other depth sensors (lidar and high resolution radar).

So that leads to this interesting piece of work, which demonstrates using an NN to extract depth information from a single image. In this case they are use faces. Of course the details of how you do streets differs somewhat from how you do faces, but the general principles are quite similar and a demonstration of one can function as a rough guide to what is possible in the other.

The authors include detailed technical info and code that allow replication of the work, but I wanted to draw attention to the demo they have that works with a photograph. Try uploading a photo of your own.

3D Face Reconstruction

In general terms, a neural network is trained to be able to perform the translation of image data to depth data by learning the correlation of depth with all the contextual information in an scene. Lighting, occulsion, scale, and perspective are used by such a network to predict depth information.

 

[lunitiks]

Wouldn't structure from motion also give you depth?

 

[verygreen]

My worry then is - if you show the camera a picture (a flat object) would it detect depth of the scene on the picture instead of the depth of the actual object ?

 

[lunitiks]

This one's informative:
Structure from Motion - MATLAB & Simulink - MathWorks Nordic

 

[jimmy_d]

My worry then is - if you show the camera a picture (a flat object) would it detect depth of the scene on the picture instead of the depth of the actual object ?

That depends on the network of course, and also whether the image in question fills the whole frame or just a portion, and if it fills a portion to what extent do the lighting and perspective make sense with regard to the external environment. In the extreme case it will be possible to fool a monocular depth-from-context system in the same way humans can easily be fooled in the same situation:

Trompe-l'œil - Wikipedia

Of course, doing that in the real world in such a manner that it creates a real problem for self driving cars is challenging, though it does represent an interesting corner-case failure. Unlike the roadrunner coyote cartoons there aren't many highways that terminate in a cliff wall waiting for someone to paint a tunnel on them.

Fast and Furry-ous - YouTube

But any situation that can fool a carefully design depth-from-context system would also fool a human being. I guess the car ends up winning by having forward looking radar.