I just gave unmodified, unhacked, fully stock 2018.42.3 Navigate on Autopilot a good try with an open mind on a ~300 mile drive this weekend.
I've seen some videos, tweets, posts, etc praising the feature. I hadn't really had a chance to try it on a longer drive myself as of yet, though, until this weekend.
TLDR version: This is the most useless thing I've ever seen. I've seen some whoppers, but this takes the cake.
Let's do a rundown of what I think was improved:
- Autosteer in highway interchanges and off-ramps was improved. It would stay in the ramp without too much trouble, while prior it would freak out and demand the driver intervene for sharper curves. (We'll ignore that it was taking the turns at ~15 MPH lower than the suggested speeds, but baby steps I suppose).
- Visual indication of what travel lane was needed for upcoming interchanges was reasonable and a good addition to normal navigation.
- I do like the path visualization when lane changes are initiated.
- It does usually try to take exits without intervention (more on this later) which is a step in the right direction for on-ramp to off-ramp autopilot.
So, some improvements I suppose.
Now for the bad.
- I now fully understand why Tesla makes it require confirmation. If it had been allowed to make the suggested lane changes on its own without confirmation, I'd likely have died 10-20x if I didn't take control every time.
- AP1 and AP2 previously did *okay* when following a lane that ended and gradually merged into a single lane. While using NavOnAP this weekend, the car just wanted to make its own lane instead every time instead of merging... usually trying to run into a barrier or median, requiring intervention every time.
- The car regularly suggested lane changes directly into objects it clearly detected. It would even show the proposed path on the visualization as going directly through the other vehicle. In one instance I wondered if it really was going to let me change lanes into a semi truck, or if it would wait until it was clear. Nope, it started to move right towards it after confirmation. No red lane, nothing, while directly along side a semi. *shakes head*
- NavOnAP has no concept of "Keep Right, Pass Left". It never suggests lane changes back to the right in any of the available modes.
- Further, it randomly suggests lane changes to the left for no reason whatsoever. No traffic, no interchanges, nothing.
- I found the car randomly decelerating at least 10x during the trip with no obvious cause. More common when driving in the right lane vs left. It would also set a seemingly random max speed at times, with no speed limit changes or interchanges.
- AP2 still doesn't read speed limit signs, so the noted speed limit doesn't always match the real highway speed limit in areas where it was recently upped or lowered (happens a lot around here with places bumping to 70).
- At least once the car detected a construction zone with a popup about it (kudos on that) and then immediately proceeded to try and suggest a lane change into construction cones..... which negates this from making the "improvements" list above.
- Overtake suggestions are useless. On two lanes, driving in the right lane, I would approach a vehicle ahead that was traveling more slowly. No other traffic. The car would decelerate... 5.... 10.... 15 MPH.... as it sees the vehicle. Then, after matching its speed at my set following distance, a few seconds later it'd popup "Confirm lane change" to overtake. Seriously, wtf. And not just once in a while. Every single time I waited for the suggested change, it behaved this way. In every mode setting, including "Mad Max".
- The car detects the other vehicle way in advance, even when just using the in-car visualization for reference, and could easily make the suggested lane change early enough so that no deceleration at all would be needed, even with the delay of requiring confirmation.
- On multiple occasions the car would start doing a lane change (either a confirmed one, a manually initiated one, or an automatic one for an exit), get part way through, and quickly veer back into the starting lane for no reason. About half of those times it would popup with "Lane change cancelled". In one instance I actually missed an exit because it was 2/3's into the exit ramp lane, stayed there a moment, then just jumped back to the left for no reason.... ugh.
- Even features that were usable before, like manually initiated auto lane changes, are no longer reliable.
Overall, using "Navigate on Autopilot" did not improve the experience of using Autopilot at all, with the limited exception of autosteer's new ability to mostly keep in lane on a tight interchange... with that being negated by the fact that it tries to kill you any time a lane ends. Also, it seems that the ability to take tight interchanges is mostly thanks to nav fusion, as the vision model does not appear to be properly detecting lanes in some of these situations, yet the system presses onward.
The suggested lane changes were completely useless on every mode. It would either suggest changes that weren't necessary, weren't safe, or weren't useful. It was even suggesting lane changes for an interchange upwards of 8 miles away at one point, then refusing to suggest overtake lane changes until after that interchange.
Some more notes:
- Vehicle detection to the sides and behind your vehicle is complete garbage.
- This is super obvious when sitting still with other still vehicles all around. You'll seem them "swimming" around the visualization, colliding with each other, with you, etc.
- Also obvious when overtaking large vehicles. Almost every single semi truck, bus, or RV I passed ended up with a twin ghost visual on the screen.
- Finally, vehicles to the side are regularly shown overlapping my own vehicle visual, despite them being firmly in their own lane.
- Vehicles behind your vehicle are actually detected only part of the time, apparently due to some issue with the rear cam setup in the hardware (@verygreen I believe has documented this).
- It seems very obvious that Tesla has no real data fusion whatsoever between the cameras. This results in both huge gaps in the usable data as well duplicate data (like the ghost trucks). This is like computer vision 101 stuff that I don't understand why Tesla hasn't overcome this, especially in something shipped to thousands of customers.
- Radar/vision fusion on AP2 appears to be significantly worse than AP1, with AP1 easily accurate for a few cm... AP2 easily worse than +/- 1m... very obvious when looking at the lead vehicle visualization.
- Some of the failings of NavOnAP don't even make sense. If it clearly "sees" a vehicle, it seems like a basic sanity check in the higher level code would prevent it from suggesting a lane change into it.... but this isn't what happens.
Could probably go on for quite a while, but suffice it to say I won't be using the feature any further... not at least until it's actually useful.
It doesn't improve the experience of using autopilot for me one bit. In fact, it makes it even more frustrating. This is ignoring the super frequent nags that plague the more recent firmwares, too.
I'll be sticking to my AP1 vehicles for longer trips from now on I think. In fact, I'm probably going to try and make time to make some videos/posts about AP1/AP2 modifications that are actually useful.
For example, my modded AP1 vehicle would handle the situation I noted above (overtaking a vehicle) smoothly with zero deceleration. AP1 (and AP2) can detect a vehicle ahead of you over 100m away... no excuse for the behavior of NavOnAP.
I'm just super disappointed in Tesla. Their spat with Mobileye has cost Tesla customers a huge amount of progress on the autopilot front. AP1 owners are completely screwed because they will get zero improvements. (Despite promises of ongoing improvements, AP1 hasn't had a single improvement in about two years). Meanwhile, AP1 is running on Mobileye hardware that was released nearly 5 years ago and still handles many situations better than AP2. And it's not like Mobileye has stopped. They're positioned to blow Tesla out of the water with their current hardware (EyeQ4), and off the face of the Earth with their upcoming hardware (EyeQ5). Had Tesla not screwed us all over in that regard, it's likely AP1 would still be improving and that AP2 would be running the next gen of Mobileye hardware with features well beyond what Tesla is capable of doing today. Again, just disappointing that they've decided to forsake early adopters yet again, and also give current adopters less value for their $ in the meantime.
I'm sure people will come out in force to defend Tesla, say how great NavOnAP is, etc... and by all means, do what you must. I personally own both types of vehicles (AP1 and AP2) and drive both regularly... pretty simple to tell the deficiencies of AP2. You're not going to convince me that somehow my extensive first-hand experience is somehow flawed and that things are way better than I claim. lol.
I find it hard to believe that as a one-man free-time dev crew I've been able to do better than Tesla's entire multi-million-dollar-funded AP dev team has been able to do in more than two years. I have maybe 40 hours of total work into my AP1 mods, and they've been more usable than NavOnAP for nearly two years. I just don't get it. I'm pretty good at what I do, but I can't believe I'm better-than-full-teams-with-millions-in-funding good.
To that end, when/if I get the time, I'm going to set a goal for myself of ~25 hours of work to make a hardware/software modification to an AP2 vehicle that actually does what NavOnAP is supposed to do. Start to finish, from scratch. It's been suggested that I get some basics prepped (hardware I need in-hand, for example), then keep a GoPro on a chesty running until the modifications are done and working to document the entire thing.
If doable, then I think it would be even more obvious that we have to reevaluate Tesla's progress on the driver assistance front.
Anyway, enough of that for now.
Disclaimer: I have no positions with any of the companies mentioned nor do I intend to initiate one at any point in the future.
