Blog NTSB Releases Preliminary Report on Autopilot Crash

The U.S. National Transportation Safety Board issued a preliminary report on a fatal March 23 crash involving a Tesla Model X using Autopilot near Mountain View, Calif.

Investigators leveraged data pulled from the car’s computer that shows the driver’s hands were on the steering wheel for just 34 seconds during the minute before impact.

Data also showed that the Model X sped up to 71 miles per hour just before hitting a highway barrier. Tesla issued a release in March that included most of the info in the report. Tesla said “the driver had received several visual and one audible hands-on warning earlier in the drive” and the driver had about five seconds and 150 meters of unobstructed view of the concrete…but the vehicle logs show that no action was taken.”

The NTSB report said the crash remains under investigation, with the intent of issuing safety recommendations to prevent similar crashes. No pre-crash braking or evasive steering movement was detected, according to the report.

“Tesla Autopilot does not prevent all accidents — such a standard would be impossible — but it makes them much less likely to occur,” Tesla wrote in its March post. “It unequivocally makes the world safer for the vehicle occupants, pedestrians and cyclists.”

Read the full report here.

 

[SO16]

How can the driver who supposedly knows AP has an issue with that section of road, still get in an accident at that location? If I knew something bad could happen there and is considered a known issue, I sure would be paying attention. Something still doesn’t add up.

 

[SlicedBr3ad]

In fact, all of the crashes in which AP was involved likely would not have occurred had the driver not been using AP. That makes it less safe, not more.

Everyone should realize this statement is not fact! The guy that died in model x median crash actually died precisely because someone without autopilot crashed in that same exact spot.

 

[jkennebeck]

And my software update this weekend (AP1 now on SW version 2018.2.9 75bdbc11) DRASTICALLY increased the frequency of AP nags. Lovely. I figured this was coming. I'm going to put in a customer complaint/request that they roll that back. Probably fall on deaf ears though.

 

[Barklikeadog]

And my software update this weekend (AP1 now on SW version 2018.2.9 75bdbc11) DRASTICALLY increased the frequency of AP nags. Lovely. I figured this was coming. I'm going to put in a customer complaint/request that they roll that back. Probably fall on deaf ears though.

Is this one of those updates where they change something that critics say is a problem, and then claim that this change was planned all along?
Yes yes we increased the notification frequency... no no it wasn't in response to critics, the NTSB, victims, etc... it wasn't our fault before, so we fixed it, and it's not our fault now.
Oh, and beta. beta!

 

[Jlwine]

What the NTSB report ACTUALLY says:

"During the 60 seconds prior to the crash, the driver’s hands were detected on the steering wheel on three separate occasions, for a total of 34 seconds; for the last 6 seconds prior to the crash, the vehicle did not detect the driver’s hands on the steering wheel."

It is VERY precise in its wording. It does NOT say the driver was not holding the steering wheel. It says the vehicle did not detect the driver's hands.

It is disingenuous, misleading and disheartening that this has been reported right here in the blog post as "investigators leveraged data pulled from the car’s computer that shows the driver’s hands were on the steering wheel for just 34 seconds during the minute before impact."

I agree 100%< it is a very important distinction.

Also, I would like to know why the car took no action in the last five seconds that both it and the driver apparently had an unobstructed view. Ultimately, the driver is 100% responsible, but it would have been nice if the car had saved the driver from his/her inaction.

 

[SO16]

Everyone should realize this statement is not fact! The guy that died in model x median crash actually died precisely because someone without autopilot crashed in that same exact spot.

Exactly!!!!!! The barrier was not replaced.

 

[benfar]

I don't have evidence about higher-speed Tesla AEB performance yet, but did find encouraging actions that the European New Car Assessment Program ("NCAP") recently started to test AEB, FCW, and the combination for speeds up to 48 miles per hour (80 kilometers per hour). From here http://cdn.euroncap.com/media/32278/euro-ncap-aeb-c2c-test-protocol-v201.pdf is this diagram showing tests for a stationary target designed to appear like a vehicle to radar, lidar, and camera sensors. Tests will be run straight-on and with lateral offset left and right up to 50% of the width of the target. (There are also tests for slower-moving vehicles and braking vehicles.)

View attachment 308383

I just saw a partial answer to whether current Teslas engage AEB for stationary vehicles here Insurers warning on 'autonomous' cars . The researchers in the video are the same ones who developed the fake car used for Euro NCAP testing of forward collision warning ("FCW") and automatic emergency braking ("AEB"), so I believe their test in the video that showed the Tesla did not respond to the fake stationary car was valid, with one caveat.

My caveat is that they were testing at a special facility, not a normal highway, which is appropriate for safety reasons. However, it means that Tesla's "fleet learning for radar" would not have had the opportunity to create their maps that support emergency braking for stationary objects, based on radar detection. A good follow-on test would be for them to block off a section of a highway and conduct the same test in a normally-traveled lane. I suspect in that case that both FCW and AEB would trigger, with FCW first and then AEB afterwards if needed in cases when the time-to-collision is long enough for the driver to take evasive action, but with FCW and AEB at the same time when closer.

I don't recommend that anyone but professionals in controlled conditions try this, of course, for obvious safety reasons.

 

[mongo]

I just saw a partial answer to whether current Teslas engage AEB for stationary vehicles here Insurers warning on 'autonomous' cars . The researchers in the video are the same ones who developed the fake car used for Euro NCAP testing of forward collision warning ("FCW") and automatic emergency braking ("AEB"), so I believe their test in the video that showed the Tesla did not respond to the fake stationary car was valid, with one caveat.

My caveat is that they were testing at a special facility, not a normal highway, which is appropriate for safety reasons. However, it means that Tesla's "fleet learning for radar" would not have had the opportunity to create their maps that support emergency braking for stationary objects, based on radar detection. A good follow-on test would be for them to block off a section of a highway and conduct the same test in a normally-traveled lane. I suspect in that case that both FCW and AEB would trigger, with FCW first and then AEB afterwards if needed in cases when the time-to-collision is long enough for the driver to take evasive action, but with FCW and AEB at the same time when closer.

I don't recommend that anyone but professionals in controlled conditions try this, of course, for obvious safety reasons.

Regarding that test video, do you know if the dummy car is treated with radar reflective paint, or otherwise visible in the 77GHz spectrum?

Also wonder if the stationary truck to the side has any effect...

 

[benfar]

Regarding that test video, do you know if the dummy car is treated with radar reflective paint, or otherwise visible in the 77GHz spectrum?

Also wonder if the stationary truck to the side has any effect...

Yes, since this test was done by Thatcham Research, the same company that designed the dummy car used in the Euro NCAP tests, I assume they made sure the dummy car in the filmed tests would have been recognized by the Tesla radar as an obstacle. I believe the dummy cars used for the NCAP tests are filled with air and bounce away, so since the one filmed did not, I doubt it was the identical design. The "balloon car" used in the NCAP tests was developed to be recognized by 77 GHz radar, vision, and lidar sensors as an actual car.

That's a good question about the stationary truck. The Tesla's radar, whether the original Bosch MRR or the more recent Continental ARS410, has more than good-enough performance to identify them as separate objects. Both use FMCW ("frequency modulated continuous wave") modulation and sophisticated antennae which together detect objects according to range, azimuth, and Doppler (relative speed). Radars using less-sophisticated modulation like FSK ("frequency shift key") detect mainly on relative speed, so in many cases they group stationary objects detected within the radar beam together into one detected "object". After that modulation along was successfully used for early truck-radar designs, the industry transitioned to more-sophisticated radar modulation schemes, including the FMCW versions in the radars used by Tesla.

 

[benfar]

Yes, since this test was done by Thatcham Research, the same company that designed the dummy car used in the Euro NCAP tests, I assume they made sure the dummy car in the filmed tests would have been recognized by the Tesla radar as an obstacle. I believe the dummy cars used for the NCAP tests are filled with air and bounce away, so since the one filmed did not, I doubt it was the identical design. The "balloon car" used in the NCAP tests was developed to be recognized by 77 GHz radar, vision, and lidar sensors as an actual car.

That's a good question about the stationary truck. The Tesla's radar, whether the original Bosch MRR or the more recent Continental ARS410, has more than good-enough performance to identify them as separate objects. Both use FMCW ("frequency modulated continuous wave") modulation and sophisticated antennae which together detect objects according to range, azimuth, and Doppler (relative speed). Radars using less-sophisticated modulation like FSK ("frequency shift key") detect mainly on relative speed, so in many cases they group stationary objects detected within the radar beam together into one detected "object". After that modulation along was successfully used for early truck-radar designs, the industry transitioned to more-sophisticated radar modulation schemes, including the FMCW versions in the radars used by Tesla.

I visited @TrafficEng today and rode in his company Tesla while he talked about his experience and insights into Tesla system behavior. He pointed out something that hadn't occurred to me, which is that there is a delay after Autopilot detects the previous lead vehicle for TACC has left the lane, before it accepts another vehicle/object as being in the lane. That could account for the Tesla systems not braking quickly for stopped vehicles in the lane ahead, right after a vehicle they had identified as the closest in-path vehicle changes to another lane.

Thanks again to him for taking the time to demo (non-dangerous) situations he has experienced, and discussing possible reasons for some of the unexpected &/or undesired vehicle behavior I saw.

 

[chillaban]

I visited @TrafficEng today and rode in his company Tesla while he talked about his experience and insights into Tesla system behavior. He pointed out something that hadn't occurred to me, which is that there is a delay after Autopilot detects the previous lead vehicle for TACC has left the lane, before it accepts another vehicle/object as being in the lane. That could account for the Tesla systems not braking quickly for stopped vehicles in the lane ahead, right after a vehicle they had identified as the closest in-path vehicle changes to another lane.

I've seen this too. The delay is about 1-2 seconds. If you flip a turn signal that seems to immediately cause the car to re-determine the lead car.

I presume this is all to improve comfort for TACC/AP being a driver-assist feature right now. It would rather err on the side of driving smoothly (AND ASSUMING THAT THE DRIVER WILL INTERVENE) rather than freaking out and unnecessarily slowing down or abruptly hitting the brakes all the time.

 

[TrafficEng]

Thanks for the visit @benfar .Interesting conversation and drive, with a few edge cases that proved quite interesting:
1) a missed detection of a weird trailer carrying a car, stopped at a traffic light as we approached from behind. The system just missed this one completely. It is 50/50 detecting stopped regular vehicles, but poor on weird vehicles like the one we encountered.
2) A missed detection of a stopped black SUV that was revealed when the 'in path object" changed lanes just before the traffic light. Seems to me that black cars against black asphalt sometimes gives the AI trouble. This was the Thatcham test - just happened to occur on our drive.
3) I braked when we were cut off by a careless merge by another driver. AP probably would not have had an accident, but to close for comfort.

All of these are known shortcomings of the current version of AP, so as an experienced AP user you are ready for them.

However, I can say that having three events like this on one drive is unusual. Just happen to be an 'edge case' day!

 

[Seesaw]

Everyone should realize this statement is not fact! The guy that died in model x median crash actually died precisely because someone without autopilot crashed in that same exact spot.

That’s a pretty low threshold... a drunk person had crashed there.

 

[SlicedBr3ad]

That’s a pretty low threshold... a drunk person had crashed there.

For drunk person or person not paying attention while driving?

I was thinking about each time I get nag to hold the wheel on 21.9 how quickly I respond to these nags. Would it be telling if take releases how quickly person rained to nags in each instance of the deadly crash? I bet it was flashing for a while. If not, what does that mean, the person that died WAS paying attention?!

 

[Az_Rael]

I bet it was flashing for a while. If not, what does that mean, the person that died WAS paying attention?!

The NTSB report states the Model X driver had last received a nag 15 minutes prior to the crash. He was not getting an active alert at the time of the crash since the torque sensor detected his hands for 34 seconds of the last minute prior to the crash.

 

[Peteski]

My caveat is that they were testing at a special facility, not a normal highway, which is appropriate for safety reasons. However, it means that Tesla's "fleet learning for radar" would not have had the opportunity to create their maps that support emergency braking for stationary objects, based on radar detection. A good follow-on test would be for them to block off a section of a highway and conduct the same test in a normally-traveled lane. I suspect in that case that both FCW and AEB would trigger, with FCW first and then AEB afterwards if needed in cases when the time-to-collision is long enough for the driver to take evasive action, but with FCW and AEB at the same time when closer.

In that particular test, FCW and AEB did actually both trigger, but AEB only slowed by 5 mph before impact I believe. The FCW warning was very clear in the in-car video I saw (not shown in the BBC coverage).

 

[benfar]

In that particular test, FCW and AEB did actually both trigger, but AEB only slowed by 5 mph before impact I believe. The FCW warning was very clear in the in-car video I saw (not shown in the BBC coverage).

I'm interested in that video, to learn how long in advance of the crash the FCW triggered. If it's online somewhere do you have a link?

 

[S4WRXTTCS]

For drunk person or person not paying attention while driving?

I was thinking about each time I get nag to hold the wheel on 21.9 how quickly I respond to these nags. Would it be telling if take releases how quickly person rained to nags in each instance of the deadly crash? I bet it was flashing for a while. If not, what does that mean, the person that died WAS paying attention?!

It doesn't really tell us anything.

Tesla makes it seem like a torque sensor can actively detect hands at the wheel on a per second sampling rate. But, the reality is likely much different.

We don't know what the driver was doing.

Maybe he was distracted by his phone
Maybe it happened so quickly that he had a bit of a panic paralysis.
Maybe he was sneezing.

It's pretty scary how much of life comes down to stacking of momentary failures. Perfectly timed bad luck that removes a person from existence.

 

[mongo]

Tesla makes it seem like a torque sensor can actively detect hands at the wheel on a per second sampling rate. But, the reality is likely much different.

The torque sensor is the basis of electric assist power steering. The sample rate is much faster than once a second.

 

[S4WRXTTCS]

The torque sensor is the basis of electric assist power steering. The sample rate is much faster than once a second.

The way I worded it makes it seem like the sampling rate is somehow relevant. It doesn't matter how many times a second it's sampled because the sensor itself isn't really even designed to detect hands at the wheel.

It's designed to detect torque. As you said it's meant for electrical power steering where it's necessary to determine where the driver wants to go.

Most of the time it senses my hand not because I'm applying torque, but I'm resting my hand at the wheel while it's doing minor correction.

What's funny is Tesla is changing the hold steering wheel message to something else to help educate drivers that it's actually detecting torque. So the drivers apply torque versus just holding.

But, what they report still reads as if the sensor somehow knows if someone is holding the wheel on a per second basis.