Autopilot is already improving.

[Todd Burch]

I've been pondering whether AP is improving as well. From evidence of how the car has performed taking me 35 miles each way to and from work the last few days, I've seen improvement. Today about 90-95% of my drive (almost everything except turns) was done on autopilot, and the route is 60% busy suburban streets and 40% highway. I touched the wheel very little.

Part of me wants to believe that it's getting better, but it seems like too short a timeframe for something like that to be happening. (After all, autopilot's been in testing for a year, and beta testers have been using it for the last 2 months--why wouldn't it have learned some of this stuff during that time?)

So that had me thinking about more likely reasons why the car has been doing better:

1. AP does better when following a car vs. when there's no car in front. Some improvement might be a result of that.
2. Position of the sun (low on the horizon in front of the car vs. high noon vs. behind you vs. nighttime) has a big impact on contrast and what the camera can make out.
3. Sky conditions affect it as well. Clear skies result in harsh lighting and glare that can reduce contrast and make it hard for the camera to pick out lane markings, whereas overcast skies can help with clarity.
4. Wet conditions can result in reflections and increased glare that might interfere with finding lane markings that dry weather would not impact.
5. Random differences in specific lane position can change what the camera can make out.
6. Proximity to a vehicle in front can change the camera's view of the lane markings and lower its confidence in what it sees.

I'm not including "I can better anticipate how the car will behave" here as a reason we might be sensing improvement, since I'm referring to undeniable improvement--for example, if the car tried to go for a turn-lane yesterday but held its lane today, that's a clear improvement that can't be argued and couldn't be a result of our perception of its behavior.

So think about these things when you notice an improvement and see if any of them could be a factor as well.

 

[rjcbox]

From TM site: "...This combined suite of features represents the only fully integrated autopilot system involving four different feedback modules: camera, radar, ultrasonics, and GPS. These mutually reinforcing systems offer realtime data feedback from the Tesla fleet, ensuring that the system is continually learning and improving upon itself..."

 

[bmah]

From TM site: "...This combined suite of features represents the only fully integrated autopilot system involving four different feedback modules: camera, radar, ultrasonics, and GPS. These mutually reinforcing systems offer realtime data feedback from the Tesla fleet, ensuring that the system is continually learning and improving upon itself..."

Yes but even assuming what's on the Web site is true, that doesn't necessarily imply improvements over the span of a few days ("continually" is not the same as "continuously"). Short of an official announcement with more details, we don't know one way or the other. We're all just guessing.

 

[Discoducky]

A couple of people suggested something that would make it definitive.

Just take the exact same route, same lanes, same times of day, same days of the week and compare interventions (instances when the driver needed/wanted to take control of steering)

If the system is truly learning, outside of anomalous environment issues, then you'd see those that trend across a wide set of drivers. In fact, you should *never* see it trend worse.

I think this calls for a spreadsheet, but since I'm starting a new job Monday I don't have the time to manage. Would be good to track nearest big city in the data so it would be easy to compare weather as a hint

 

[EarlyAdopter]

At the Autopilot press conference, Elon provided a great deal of technical detail and answered most of the questions present in this thread. No need to speculate - the fleet learns and the updates come down daily.

Here are some of the most interesting quotes with direct links to those time stamps:

1. "Depending on where the car is in the world [...] it will know to use the left lane marking, the right lane marking, follow other vehicles, holistic path prediction, or GPS." 5:50
2. "When one car learns something, the whole fleet learns it." 27:37
3. "People should see the car improve with each passing week, even without a new software update because the data is continually improving. It should get better with each passing day, but you probably won't notice it until after a week. The car wouldn't steer quite right past a freeway off-ramp one week, but the next week it does." 38:06
4. "It is an automatic machine learning system." It's not even remotely like general AI, which is what Elon is concerned about. "Cars aren't going to take over the world." 28:36

 

[Coiled]

I did see a significant download come over my network last night in the middle of the night. I would estimate it very roughly at 30-35 MB. I thought maybe it was a bugfix update to 7.0, but the car didn't ask me to install an update. Could this be a stealth GPS and/or AP algorithm update? Who knows? I would be interested to know if others are seeing the same thing on a nightly basis.

What worries me about this is that Tesla can (and clearly does) make significant changes to AP without our consent or knowledge. These is no option to decline the update. Meanwhile, Tesla has taken a big risk by releasing AP and trusting their customers not to abuse it. If lawyers and politicians get involved, Tesla could end up nerfing AP with geo-restrictions (it won't work in some states/countries) or stricter hands-on-wheel requirements. And if they do, you might not have an option to decline the changes.

 

[GlmnAlyAirCar]

We're all just guessing.

Isn't that what we do best on this forum?

 

[garygid]

Is there a transcript of the Autopilot Press Conference available somewhere?
It is difficult for me to understand a lot of what was being said. Thanks.

 

[Vger]

The press conference included a slide show which Elon annotated. The images showed clearly what they are talking about, and clearly support those who say the system is learning on a daily basis (on the same route). Look specifically at the "high precision maps" which are built BY THE FLEET:

Tesla reveals all the details of its and its software v7.0 [slide presentation and audio conference] | Electrek

 

[ecarfan]

Thanks to @EarlyAdopter for those Elon quotes from the AP press conference. The audio quality of that recording was so bad I stopped listening after a few minutes so didn't know everything that was said.
Tesla is doing amazing things, and is far ahead of the competition. Oh wait, there isn't any competition....

 

[ohmman]

I continue to assert that the raw training data is uploaded continuously, and models are built daily (or less frequently). This is completely in line with the statements made in the press conference, and it's just how machine learning works.

The data point that there was a 30-35MB transfer overnight is interesting. Again, if someone with Autopilot would like to run a capture overnight, it would be pretty cool to see file sizes, source location, etc., and to confirm that indeed there are background updates for the model. I think the press conference certainly implies that's the case.

As for Coiled's worries about the updates being pushed without intervention; I'm not sure exactly how this would be better managed. It's in their best interest to make sure that the fleet is running the most up-to-date model, and that it's consistent. This is especially true from a safety perspective. My assumption is that they do some kind of model sandboxing prior to pushing it, which is a typical way to stress test a model before release. I certainly hope regulators don't get in the way at some point, or it will slow AP progress tremendously.

 

[Vger]

The press conference included a slide show which Elon annotated. The images showed clearly what they are talking about, and clearly support those who say the system is learning on a daily basis (on the same route). Look specifically at the "high precision maps" which are built BY THE FLEET:

Tesla reveals all the details of its and its software v7.0 [slide presentation and audio conference] | Electrek

I also would specifically point out this image from the slide show:

The display on the right side of the IC clearly shows a debugging/testing mode of the car, with the stream of log messages scrolling by. One can learn/infer quite a lot about the system by pondering that little snippet!

 

[larmor]

I also would specifically point out this image from the slide show:
View attachment 98685
The display on the right side of the IC clearly shows a debugging/testing mode of the car, with the stream of log messages scrolling by. One can learn/infer quite a lot about the system by pondering that little snippet!

On the other sied of the screen is a suggestion that once enough live data is available, merged with online maps, the AP software model can be run virtually....

 

[mspohr]

I continue to assert that the raw training data is uploaded continuously, and models are built daily (or less frequently). This is completely in line with the statements made in the press conference, and it's just how machine learning works.

The data point that there was a 30-35MB transfer overnight is interesting. Again, if someone with Autopilot would like to run a capture overnight, it would be pretty cool to see file sizes, source location, etc., and to confirm that indeed there are background updates for the model. I think the press conference certainly implies that's the case.

As for Coiled's worries about the updates being pushed without intervention; I'm not sure exactly how this would be better managed. It's in their best interest to make sure that the fleet is running the most up-to-date model, and that it's consistent. This is especially true from a safety perspective. My assumption is that they do some kind of model sandboxing prior to pushing it, which is a typical way to stress test a model before release. I certainly hope regulators don't get in the way at some point, or it will slow AP progress tremendously.

I also believe that the car continuously uploads data from its driving experience. Even when AP is not activated, the display shows grey lane markings so the system is active and collecting data.
I don't know that they would need to do a push update beyond regular version updates (which I expect to be fairly frequent as they fine tune their algorithms). I think that as the car drives, it can download the latest model for the road ahead.

 

[ecarfan]

Interesting! How were you able to produce that data display?

I also would specifically point out this image from the slide show:
View attachment 98685
The display on the right side of the IC clearly shows a debugging/testing mode of the car, with the stream of log messages scrolling by. One can learn/infer quite a lot about the system by pondering that little snippet!

 

[Vger]

Interesting! How were you able to produce that data display?

It is just a slide from the press conference. See link upthread. I sure wish I knew how to turn it on!

 

[wk057]

Paging wk057

I haven't noticed any significant increase in data on the Wifi connection. That's not to say the car isn't doing anything on LTE, though, while driving. I'll dig through my traffic logs in a bit.

 

[brec]

I did see a significant download come over my network last night in the middle of the night. I would estimate it very roughly at 30-35 MB. ...

How does one see this, or estimate its size? Something in your router?

 

[wk057]

Double checked my car's Wifi data captures. Nothing new really. The car has periodically been downloading a 4 to 5MB data burst from Tesla every few days for months, which I've been guessing is the charger list and associated data.

Nothing new since the autopilot update.

 

[Cosmacelf]

After finally listening to the Autopilot press conference (really Auto Steer), I finally understand more about how it works.

Mobileye is the technology that is used to find lane boundaries, pedestrian and bicycle detection, car detection, and traffic sign decoding. At some point, it will also do stop sign detection and signal light decoding. Tesla then layers on top of this their own algorithms, and most importantly, high precision GPS maps.

It is the GPS maps that allows the system to get better on a daily basis. People have reported that initially their cars would try to follow the right hand freeway lane marker and exit the freeway prematurely. After a week of driving, the car doesn't do that anymore. What happened is that over the hundreds of trips made that week by the entire Tesla fleet (including non autopilot enabled cars, do note), they updated their model to say that in this lane on this part of the freeway, use the left lane marker, not the right as a guide. This system would be impervious to jokers trying to trick the system since such bad driving would be rare and ignored. This system seems to be smart enough to be used in areas that have either no lane markers, or contradictory lane markers (Elon gave an example of a horror show of an area on the 405 freeway).

So Tesla uses Mobileye lane detection, but augments it with their own high precision GPS maps which tells it what inputs to use in a particular location. And this information is gathered in real time, and new model updates seem to be downloaded to the cars on a daily basis.

This is very, very powerful. No other competitor, including Google, has anything approaching the ability for 80,000 cars (today) to be continually providing input to make the model smarter.