Welcome to Tesla Motors Club
Discuss Tesla's Model S, Model 3, Model X, Model Y, Cybertruck, Roadster and More.
Register

Can someone explain the Autopilot "Learning" algorithm?

This site may earn commission on affiliate links.
I've read many times that Autopilot learns as we drive, that Tesla is creating an ultra-precision map based on the cars' experiences on the road. Sounds very convincing and logical.

But I still have no clue how that learning process is supposed to work in detail, in particular what kind of user input is used to improve the car's behavior.

I read that users report that cars had learned with time to avoid taking certain highway exits, or similar improvements.

But it what user action does Autopilot use to "learn"? I think it must be one of these two:


A) User steering that causes Autopilot to disengage (like: car wants to take an exit, user brings it back into the straight lane, canceling Autopilot in the process)

B) User steering that corrects the heading of the car slightly, Autopilot stays on (like: car is too near the median, I steer it slightly to the right to better center it in the lane)


I have problems with either option (and can't think of others):

Everybody has to cancel Autopilot occasionally, usually at the same spots, where the car wants to follow the road but we must take a turn. How can the algorithm distinguish between that situation (where there is nothing to learn, the car is doing the right thing but I must take my own route, while most others will want to continue following the road) from the "correction" that keeps the car in straight at an exit ramp?

And the slight corrections, they are really difficult (the wheel stiffly wants the car to keep going too far left, in my example), and if I use too much force Autopilot disengages.

In any case, my car doesn't show any signs of learning on my daily route, it still wants to hit the traffic island that I must steer around each time (turning off Autopilot in the process ), and it keeps almost hitting the median where in one stretch of road it only recognizes the center line ...

How does Auot-Learning really work, does anybody know?
 
Paul, thanks for the link, an excellent article.

But it doesn't mention how Tesla is supposedly creating super precise maps based on Autopilot plus human corrections.

There have been numerous references to this, and how the Tesla supposedly improved how it masters certain routes based on that learning (which is said to be based on uploads by the many vehicles running on Autopilot).

I hope someone will share their knowledge or insights ...
 
The car is sending back data (cancellation/correction/avoidance events along w/GPS and front camera stills) back to Tesla HQ. Gnomes are busy 'curating' and integrating this into the database; differential updates (non-firmware) are downloaded to the car (without notification). Metro areas and the frequency/pattern of localized events are given corrective priority. <- All of this is simply my 'speculation'.
 
Last edited:
I doubted the learn on the fly thing when I first heard it. Now that we have had a full update and some problems with the AP have been addressed, it is perfectly clear to me that the only learning that is being done is by the engineers at Tesla doing the programming.
 
  • Like
Reactions: davidc18
Paul, thanks for the link, an excellent article.

But it doesn't mention how Tesla is supposedly creating super precise maps based on Autopilot plus human corrections.

This is probably something Tesla wants to keep as proprietary information to maintain an advantage over competitors. Nobody here is going to be able to answer for you, unless somebody on the autopilot team decides they want to risk a serious lawsuit. Or maybe if Elon gets an account here and finds an hour a day to browse the forums and post. (The latter is...unlikely).
 
My understanding is fleet-learning has never been proven to take place.

There are lots of people who monitor the data going to the car (over wifi) and they've never seen it happen, or at least no one has noticed anything yet.

By bet is it's something Elon said prematurely and it hasn't been fully rolled out yet. Or they're still in the process of collecting data.
 
  • Like
Reactions: davidc18
Mobileye's fleet learning uses all of 10K bytes per mile (or was it kilometer), so it isn't much bandwidth. As per the CES panel talk by Mobileye CEO. That little amount of data could easily get lost unless you knew what you were looking for.
 
You bet they are collecting data, as they should. Makes total sense. And I see that collection of data as fleet learning. Not cars teaching other cars. But cars teaching humans by collecting massive amounts of data. Fortunate we mortal being are still needed to correlate it. Yay. Job security for the smart ones! But just wait. Cars will be teaching cars directly before we know it.

This stuff is so complex that it will take time to see any large scale improvements. All of us expect immediate results and improvements. I certainly want them. But I think it is wise for them to roll things out very carefully. 7.1 added some improvements. But according to comments on this forum, some things actually got worse. Typical unintended consequences. I think that underscores just how hard this stuff is. Even a simple thing like not being able to mute the hifi when the front distance sensor is activated (now fixed) shows just how many oops' there are when developing any software. Heck, look at Apple. Sometimes they get it right. But they mess things up more than not with their updates and then need to scramble to fix them. I am not a SW developer, though I was in the business for years a long time ago, so I do have a sensitivity to what these guys go through. They are making highly educated guesses much of the time as they push the limits. I think something as complex as AP probably pushes those limits to the very edge.

They will get there. But just not as quickly as any of us want.
 
The goal of fleet learning is to create a 1D map (a line) and place the car accurately on that 1D map by finding visual reference points and store those points on a 3D map.

The 1D map places the car accurately in the lane. AP has apparently started using these maps on interstates, as exits and faded road markings are causing fewer problems. While mobileye is almost certainly providing the raw data, Tesla is apparently creating their own proprietary mapping. Mobileye will start building their own maps, at first with GM.

- - - Updated - - -

Mobileye's fleet learning uses all of 10K bytes per mile (or was it kilometer), so it isn't much bandwidth. As per the CES panel talk by Mobileye CEO. That little amount of data could easily get lost unless you knew what you were looking for.

Part of mobileye innovations is providing low bandwidth data collection.
 
I've chimed in on this before, as have others, but if indeed the fleet learning is an actual "learning" algorithm, then the OP's initial assumptions are close. The learning would take place mostly when AP is disengaged, as that's when Elon's "expert trainers" (the drivers) were training the network. I could see, however, augmenting that data with failure feedback when AP breaks boundaries (disengage, cross a line, etc). As the model is trained with real data, it would be rolled out to the vehicles OTA. However, I haven't seen any evidence that these new models are being delivered except as potentially bundled with firmware packages.

This is a fun old video of what was being done in 1992 (!) using neural networks and a driver to train the model. This was a 3-layer network, compared to our much deeper networks today, but it learned pretty quickly how to generalize. Despite being incredibly basic and outdated, it gives an idea of how a learning algorithm would aid in self-driving.

 
Last edited by a moderator:
Here you can see how mobileye is doing their croud-sourcing algorithm. I would speculate that Tesla is using it the same way or is using directly that technology.

Professor Amnon Shashua CES2016 PressConference - YouTube
From watching the video it's pretty clear that each Tesla car will generate about 10K bytes per kilometer of mapping data, sent back to the Tesla mothership via cellular or WiFi links. This can then be used by Tesla to generate higher-definition maps (i.e., "Roadbooks"), that if used by our cars, can lead to improved autodriving performance on each road previously driven by a Tesla. Roadbooks are crowdsourced, in that all Tesla cars with Autopilot hardware help generate them without the driver having to do anything special.

What I don't understand is how our cars access the Roadbooks; this isn't explained well in the video. Do each of our cars download the appropriate Roadblock data from the Tesla mothership as we drive?
 
The car is sending back data (cancellation/correction/avoidance events along w/GPS and front camera stills) back to Tesla HQ. Gnomes are busy 'curating' and integrating this into the database; differential updates (non-firmware) are downloaded to the car (without notification). Metro areas and the frequency/pattern of localized events are given corrective priority. <- All of this is simply my 'speculation'.

This is what I was told as well.
 
AP has apparently started using these maps on interstates, as exits and faded road markings are causing fewer problems.

While I hope you're right about the map use, it could also be the case that the better behavior is due to a simple rule update, right? I.E. - "When a right hand line begins to diverge from a left hand line do not automatically follow the right hand line" and "If the road markings fade do not freak out, just drive straight ahead."

It would be more impressive if precision maps are the explanation for the improved behavior - I hope you're right.

- - - Updated - - -

This is what I was told as well.

By whom? Inquiring minds want to know. :)
 
Raising this thread from the dead to see if there are new thoughts / theories / data on how the learning algorithms work.
Based on my limited experience with it in the X, I think it's something like this:

Steering wheel determines age of driver. If age is less than or equal to 16 years old, learn from the child. Emulate this behavior on the road.