jebinc
Well-Known Member
While "Hope" may be a strategy for some, consider it may not be the best one to rely on.
-
Want to remove ads? Register an account and login to see fewer ads, and become a Supporting Member to remove almost all ads.
Musk: V10 wide release "hopefully end of august" after early access
- Thread starter diplomat33
- Start date
While "Hope" may be a strategy for some, consider it may not be the best one to rely on.
boonedocks
MS LR Blk/Blk 19”
With past performance being what it is with @ElonMusk tweets believe me that I don’t rely on any of them.
When we actually receive something of “value”....then and only then do I believe it ¯\_(ツ)_/¯
They are still using branches. We use a modern devops approach at work and we also use branches. Each patch get a branch until they are sufficiently good to be merged into the master branch. Number of concurrent branches depends on the size of the team.
The big difference is that our branches rarely live for more than 2-3 days, maximum 7 days. If they do, then the mistake was made in the scheduling by making the patch too complex (could probably be split up into more dependencies).
The fact that Tesla has diverted from the main branch for about 3 months suggests that they are still doing some very significant rewriting and restructuring, or maybe they just don't structure their software department very well. Because even significant restructuring or replacing central infrastructure can usually be scheduled for weekly patching by planning correct (usually by 1. partial use, 2. replace 1:1, 3. introduce the feature, 4. activate the feature).
Earlier speech by Karpathy suggest they also iterate their neural-network pretty much the same way as traditional code. By committing changes to the datasets in a repo that is built (trained) by the servers, having their NN layout descriptors by code and unit-testing through their similators and the "shadow mode" we hear so much about.
Agree with all that. If a branch exists for a long time the difficulty in merging it back rises pretty quick. So keeping branches alive for a few days is ideal. Merge main out (it's probably not changed much) and then merge the branch in. Pretty standard stuff.
We'll never really know how Tesla does their management of branches. But it would appear that it's not as you describe (i.e. best practice SW development). Just having releases in the field that overlap in weird ways would say they don't have a stable main. It should be possible at any time to release main. It's supposed to be stable. Instead, we have perhaps 8 current releases. Early access appears to be the only one that has a single release label.
Everyone uses branches. The question is how difficult and extensive the new branch is. I disagree that incremental changes can be integrated in every few days or a week for the most difficult problems.
Projects with known solutions are easy, but problems for which a solution is unknown require a lot of time for experimentation. The research can take far longer than a few days or weeks, especially for open ended behaviors that react to the real world.
However, we all seem to be saying the same thing - that there is either brand new code or a significant rewrite afoot.
At work we do things that has never been done before too, although not quite self-driving, it is considered highly innovative and disruptive. I lead the software development crew for this company.
I disagree that complex and unknown problems cannot be solved incrementally with rapid branching. It's all about how you structure the changes and how the code is structured. Making sure iterative development is easy is one of the main design criteria when architecting the boundaries between microservices, libraries and in-app services. Each patch does not need to be perfect, but it must add an improvement or reduce complexity or both.
We've had problems/area-tasks that has taken us 6 months to find a very good solution for, that we had no idea how to solve or even begin. Yet we were able to incrementally steer patches to address some part of that problem and improve related aspects. About 150 patches later the solution to the problem was obvious, because the problem was broken down so much and suddenly we realized it was almost solved.
All this was done to industrial production systems that require 24/7 stability with zero tolerance for errors.
Rewrites can be really bad, they destroy the flow the development. Should only be required if there's a fundamental change of technology or the project has been poorly managed. Experimental branches happens outside the devops cycle and they are usually done to prove a concept and get the experience required to develop something. Once experiment is completed, it's usually evaluated, split into workloads and redeveloped into the master branch by small patches. Redevelopment is significantly faster than the initial experiment, but it ensures that the changes are introduced in a planned production-safe manner and that necessary refactoring gets done (rather than a classic "merge" from a distant branch which is hard to control). Experiments can take months, but they are never merged back.
But yeah, seems there are some major things going on at Tesla.
Last edited:
EVNow
Well-Known Member
What do you mean by core of FSD ? NN or procedural code or both ?
There are basically 2 schools of thought around Tesla FSD.
- What we see now in production cars + EAP is quite current and Tesla doesn't have any separate HW3.0 FSD NN/software.
- Tesla has separate HW3.0 FSD NN/software that we saw in the demo.
Didn't Karpathy talk about software branches in his most recent talk about multitasking? He mentioned how tricky it can be because one team may have developed say recognizing traffic lights and added it to software version A while another team has developed say recognizing road debris but they were working off of software version B. So now you have to combine A and B.
With Neural Networks, I'm not sure how you would combine them after the fact.
Off the top of my head, I'd think that the network's variables can't be mixed or operated on arithmetically to produce a network that does both.
You can of course run the two NNs in sequence, but that doubles the processor workload.
Actually, deciding which pieces you can combine effectively into one network and which should be separated for a given camera image analysis might be important to FSD, I'd think.
But most of what I know about modern complex neural networks I learned from TMC...
Yepp exactly.
It's the reason software development requires tight management. So that the tasks can be split properly:
1. Both teams might have common dependencies. These can be rolled into production through the master branch early. If nobody keeps track, they do it their seperate ways complicating the overall solution.
2. Separate proof of concepts (experiments).
3. In parallel introduce as many "safe" patches as possible into master from both teams. Patches that make the code cleaner, better, or less complex are in general a good idea to take out of the experiments so lessen the diff with master.
4. Hopefully the experiments now are not too big. They should not be merged, but redeveloped by key persons from both teams as manageable patches against master that can be made production ready fast and rolled out with little risk.
And there's a lot of details to this. Key is a very competent team and an even more competent software management. Cannot go a single day without keeping close track.
You can source control the training material as well as the neural network description and the software v1 code around. Add cases which the system should be able to solve for each new feature.
Merging feature A + feature B should be roughly feature A and B. Truth with some modifications, but hopefully the unit test cases should iron most of the issues. Real life tests def required too.
Looks like Tesla is adding pick up trucks to the driver display in an upcoming software update. Check out the little pick up truck in this pic:
I'm referring to the navigational / motion planning component, and its ability to handle exceptions.
Whether it's procedural, NN, or a combination doesn't really matter. NNs are good at recognizing objects and feeding their interpretations into the next component, but deciding what to do in the face of imperfect detection and context is much harder.
My feeling is that Tesla's FSD using eAP can only handle the simpler, expected cases right now, which results in high intervention rates for local traffic (e.g. optimistically 1-5 miles local traffic with turns). True FSD requires 1M+ miles or 5-6 orders of magnitude improvement. That level of improvement doesn't happen with current code unless there are lots of huge bugs.
AP has been around for 2-3 years now, so I don't think it's filled with huge bugs. I think it's just hitting its limits.
Examples
--------------
1. I drove from LA to SF last weekend on a portion of HWY 5 with faded road markings. There were a few cars around, but none in front, and the road was patched like a quilt so it was difficult to tell which were lane markings and which were patches in the sunset light. The AP got confused and began driving into the wrong lane before we intervened.
Although it was hard for anyone to locate the lanes, human drivers pay extra attention and drive more cautiously until they can spot the lane. However, a NN operates however it was trained and doesn't try harder. It either can or it can't. When it can't, that's very bad for FSD.
2. In cities and near colleges, there are frequently crowds of people blocking the road. How aggressive the car needs to be and where it drives depends less on lane markings than what the people are doing and how they're behaving.
Is it a demonstration, a bunch of people walking across the road, or a protest? This is similar to how a car needs to behave in parking lots filled with pedestrians and other cars moving randomly about, where the situation is more fluid.
--------------
In the general FSD case, Tesla can use AP for the easy stuff, but advanced summon behavior (aka navigating in a more fluid and less structured environment) needs to take over when AP fails. When lane markings are incomplete or absent, the car needs to use alternate methods to determine where to go.
As a human driver, I barely care about lanes when people are around, an accident is nearby, or something is blocking my way. I don't think AP comes close to handling all the exception cases.
Last edited:
motocoder
M3 LR AWD w/FSD and premium interior
I think the biggest challenge with NN is that when it doesn't give the desired output, your ability to directly troubleshoot it is zero. It's not like fixing a bug in a traditional software algorithm, where you identify the root cause and then fix it. With NN the only solution is to add additional labeled data for the failing cases (and hopefully other samples of those same cases) to the training set and retrain the net. With limited training sets, it is really easy to over-train and end up with something that gives great results on the training set, but has major issues once you go beyond that. Tesla's ability to mine training data from the "fleet" is IMHO a fantastic asset that gives them an advantage over competitors.
However, I think that the "long tail" in this space will make a NN solution at today's level of technology inadequate. The human brain is way more complicated than even the most elaborate NN we have created with silicon. The convolutional NN that are being used are modeled after the visual cortex, but that is only one portion of the brain that's involved with driving.
As an example, do you imagine you could fool an autonomous car by driving in front of it and presenting some visual cues designed to fake it out? You might have to be quite elaborate with this, but I think you could. It would be extremely difficult to do that with a human driver, because the driver would pick up on other cues and eventually figure out what was going on.
To maintain adequate levels of safety, I think Tesla and the other teams working on an autonomous car will have to have a two-part solution. One part will be the machine-learned models, like NN , the other part will be some sort of fail-safe logic based on more traditional programming techniques. My concern is that Tesla's current hardware is inadequate to pull this off. I think it can be a very good Level 2 solution, but not beyond that. I laughed when I watched Elon dismiss lidar in the autonomy day discussion with the analysts. Lidar is exactly the sort of additional sensor input that can fill in the gaps when a camera-based neural net would make a mistake. To dismiss this as "fools errand" suggests either ignorance or duplicity of a high order.
So my hope is that the Tesla engineering team is focused on an incremental approach with a great L2 system as the next milestone, and not putting that aside while the bulk of their team shoots for the moon (L4/5).
Darn. I want that and the visualization that you can spin around with your fingers
I just want stationery vehicles.
Looks like Tesla is adding pick up trucks to the driver display in an upcoming software update. Check out the little pick up truck in this pic:
Checked on my commute this morning. On my AP3 X running 2019.32.1, pickups are still shown as either SUVs or Vans depending on how big/square they are.
So this must be something newer, maybe from an early access car.
Similar threads
- Article
