Sure about that?
Karpathy mentions shadow mode in the fleet during the autonomy investor presentation, 2:16:35 mark:
Actually he talks about shadow mode in detail at 2:10:00 mark.
Basically he says that shadow mode is them deploying a NN that they are working on and having it run in the car.
For example a NN that detects and predicts that a car is gonna cut in in-front of us. So the NN is always making that predictions.
So lets say the NN makes a prediction that a car to the right is gonna cut in. Then there is a program that takes that predicition and then checks for example three second later to see if the car actually ended up in-front of us/the lane that we are in. If the car did end up infront of it then that counts as a successful prediction.
If it didn't then that's a failed prediction.
So if they deployed a NN for something like this for a week and there were 10 million car cut in predictions for example and then only 85% were successfully predicted as a cut-in by the NN, it would mean they need to improve the NN more.
That's shadow mode.
This is quite different from what Elon Musk said shadow mode was
New Tesla Model S and Model X automobiles will run Autopilot in “shadow mode” and collect driving data that pits a human versus computer. Autopilot vehicles running in shadow mode will not take any driving-assist or self-driving actions. Rather they will only log instances when Autopilot would have taken action and compare those results to the real life actions taken by human drivers. Musk told the press.
So, the car will run Autopilot in “shadow mode” in order for Tesla to gather statistical data to show false positives and false negatives of the software. In shadow mode, the car isn’t taking any action, but it registers when it would have taken action. Then, if the Tesla is in an accident, the company can see if the autonomous mode would have avoided the accident (or the other way around, with the self-driving system potentially causing an accident).
It will record how the car would have acted if the computer was in control, including information about how the car might have avoided an accident (or caused one).
So, the car will run Autopilot in “shadow mode” in order for Tesla to gather statistical data to show false positives and false negatives of the software. In shadow mode, the car isn’t taking any action, but it registers when it would have taken action. Then, if the Tesla is in an accident, the company can see if the autonomous mode would have avoided the accident (or the other way around, with the self-driving system potentially causing an accident).
It will record how the car would have acted if the computer was in control, including information about how the car might have avoided an accident (or caused one).