2017.50.2 Torture Test: ails, but fails more consistenly

[Padelford]

After the snow had melted back a bit, I ran my MX with 2017.50.2 on my torture route yesterday. This route has revealed some serious lane control and guidance issues with past firmware versions. The route has two exits and a "rise plus turn" feature in the middle. The rise + turn has led to oscillatory lane control behavior and complete lane control failure in past versions. I've also included a segment in which the roadway drops and turns to the left, and this has also produced lane control instabilities in past firmware versions. The maximum vehicle speed was set to 60 MPH, and I tried to create a buffer between the vehicle and any other vehicles ahead. This wasn't always possible.

2017.50.2 failed at passing the second exit in this test, but in three sequential trials it failed the same way each time. The vehicle tries to leave the highway on the exit rather than staying on the main road. 2017.44 failed three different ways at the same exit in three successive runs (including taking the exit). There's something to be said for consistency as a positive attribute, even if it's a performance failure.

One could speculate that in the second exit situation, this version of AP loses track of the lane markers twice in a row (cresting the hill then turning to the right to go down), re-acquires the lane markers but puts more weight on the right line marks than the left lane marks when it recovers. The vehicle thus takes the exit rather than staying on the road, and it doesn't oscillate side-to-side beforehand. 2017.44 seemed to have more "indecision" (vehicle oscillating side-to-side) about which line marks to believe at the second exit, so the vehicle acted differently each time I passed that exit.

I've only posted the first of the three runs here, since the other two showed the same behavior but in different traffic density situations. For those that can't wait, the failure appears just after 2:34:

I posted a discussion of the 2017.44 test in the Model X User Interface section.

I also noted that in this firmware version, the vehicle generally tends to take turns on the outside of the lane rather than staying centered. I don't recall that 2017.44 did this, but in earlier firmware versions AP definitely exhibited this behavior, leading to some cases of getting too close to adjacent vehicles and driver take-over of the vehicle.

Edit: My cat jumped on me just as I tried to edit the title and submit. Hence a goofy title!

 

[TexasTeslaRacing]

Too much lane ping ponging with this update. We drive 1,200+ miles this week on 50.2 and at times I turned AP2 off. Also the lane changing is more aggressive and should be dialed down a bit.

 

[Padelford]

Too much lane ping ponging with this update. We drive 1,200+ miles this week on 50.2 and at times I turned AP2 off. Also the lane changing is more aggressive and should be dialed down a bit.

I will agree that the lane change maneuver has become a bit more aggressive. It was really too aggressive around 2017.34, then it smoothed out at or after .36, now it’s swinging the other way.

 

[ChrML]

For me 2017.50.2 first seemed like a major regression for Autosteer on AP2.5. I activated in on a pretty straight forward road. Not very good lane markings, but it would've worked on 2017.50. It pingponged a lot, then it abruptly tried to steer right into oncoming traffic. But as the good Autopilot user I am, I keep my hands on the wheel, so no dangerous situation occurred. It has never done that before on 2017.48 or 2017.50.

Tried it again after driving for 30 min and it seemed better. Maybe calibrated itself? Not sure if better or worse now, not tested enough.

ACC seems same as before. Pretty good, but sometimes a bit abrupt (ie suddenly changes speed, to slow back down shortly after when something changes).

 

[daktari]

After another 200 kms today, mostly rainy roads, I am shocked how much more comfortable it is when I steer and not 50.2. Way to much pingpong, too sudden speed changes. Bad update.

 

[conman]

For me 2017.50.2 first seemed like a major regression for Autosteer on AP2.5. I activated in on a pretty straight forward road. Not very good lane markings, but it would've worked on 2017.50. It pingponged a lot, then it abruptly tried to steer right into oncoming traffic. But as the good Autopilot user I am, I keep my hands on the wheel, so no dangerous situation occurred. It has never done that before on 2017.48 or 2017.50.

Tried it again after driving for 30 min and it seemed better. Maybe calibrated itself? Not sure if better or worse now, not tested enough.

50 and 50.2 are the same autopilot... There is no ongoing calibration with autopilot, what calibrates over time is your expectation. We get used to the feel of something and its quibbles start becoming familiar and don't feel so bad over time.

 

[ChrML]

50 and 50.2 are the same autopilot... There is no ongoing calibration with autopilot, what calibrates over time is your expectation. We get used to the feel of something and its quibbles start becoming familiar and don't feel so bad over time.

There was definitely a change in the beginning.

The car is new, so my only experiences are with 2017.48, 2017.50 and 2017.50.2. With the first two I've driven 800 km mostly with AP and it has worked surprisingly well. Then all of a sudden in 50.2 it's swerving a lot in much better conditions.

 

[Kurt'sX]

On a good note.....Highway design is changing nation wide, and now requires a dotted line to separate on and off ramps because I believe the reality of autonomous driving is upon us. In California nearly all of our freeways have now updated this process and have painted thru most of the on and off ramps. Use of the AP has improved because the car now recognizes the lane marking to the right.

 

[ChrML]

On a good note.....Highway design is changing nation wide, and now requires a dotted line to separate on and off ramps because I believe the reality of autonomous driving is upon us. In California nearly all of our freeways have now updated this process and have painted thru most of the on and off ramps. Use of the AP has improved because the car now recognizes the lane marking to the right.

Makes sense that the autopilot takes these off-ramps. When there are two paths to choose from, it's always almost correct to choose the one to the right. Because when you enter a two-lane road, you drive in the right lane unless you're passing someone.

Except these off-ramps when suddenly left is the correct choice. The only way to program that at this moment is reading the signs, or by fleet learning which road most people with AP on chooses (by overriding or by luck). I don't believe logic at this level exists in currently released code and that this situation is simply not handled.

Although I've seen it trying to choose both lanes and going for the center divider. Now that should not happen...

Although painted lines all the way for on-ramps may have different meaning in different countries. Around here a painted line all the way for on-ramps means that the car joining the road must give way for the cars on the road. When the on-ramp is not painted all the way it means that the car joining the road has priority and cars already on the road should adapt to join him in.

 

[Padelford]

Today on I-5 & using AP, the vehicle dived for a left exit as we passed it in the left lane. We were doing 65 at the time. The left exit was on a flat stretch.

OTOH, we experienced no other lane control instability while on AP in more than an hour of driving.

 

[banned-66611]

This is a really good demonstration of how AP is severely weakened by not having much of a memory or predictive capability. For example, when cresting a hill or temporarily losing road markings a human will expect them to be in a certain place, based on experience of how roads are laid out. Road designers generally don't hide sudden turns after crests, for example.

AP is more like a cat/dog level intelligence. It doesn't do much prediction or remember how things were before it lost sight of the road markings, it just reacts as soon as it picks up some markings and as such often gets it wrong.

It also seems to be easily confused by inclines, especially on curves. That's probably due to it's interpretation of what it sees being too simplistic to understand the change in elevation.