Well then I don't see why Isidro JR's explanation of how the tech Musk claimed about "4d" works is controversial?
I don't think there's anything here that really rise to a state of "controversial", but I don't think Isidro JR really did any explaining about what is really different between the previous "2.5D" system and the current "4D" system in Tesla Vision. His statement seems to imply the only difference is it's fusing the different views (from 8 images) and other than that it is no difference (still only doing everything frame by frame).
I can't say I know all the details either (like everyone here, we are only able to go by things Tesla is willing to make public), but from Elon's statement and from watching the latest video in full, my understanding is different. For example, Andrej talks about the last few months they've been working on this, they were able to determine it was possible for their synthetic vision NNs to get depth,
velocity, acceleration sensing with vision alone that is just as good if not better than radar. That implies their previous solution was
not doing that (at least not in a very reliable way).
It makes more sense if you consider the NNs as treating the data more like "video", instead of isolated "frames" or "images". It's possible to still do monocular depth sensing from a single frame or image, but getting velocity/acceleration is much harder (although not impossible: there are ways to determine velocity for example based on motion blur in a single frame). However, once you consider multiple frames, by comparing differences between two or more frames, determining velocity is obviously much easier.
And from the diagram shown in this post (around 19 minutes in the video), the part I would be interested in is what is being spit out in the line that goes to each "head". Note that the line is taking input from multiple points in time in the tensor video queue and I note it goes to a "video module" in each head (not an "image module"). Andrej says they first fuse all the images across all the cameras, then they fuse the information across all of time (temporal), saying it's done either by recurrent neural network, transformer, or 3D convolutions.
Tesla.com - "Transitioning to Tesla Vision"
I also found the older discussion from ScaledML in February 2020 before Tesla Vision. You can see they were already fusing the different cameras using a Fusion layer. There was talk about a temporal module with the role of "smoothing out" predictions (discussed at around 20 minutes in the linked video). This ties in with Elon's explanation of how the "2.5D" worked during the July 2020 earnings call: "Well, the actual major milestone that's happening right now is really a transition of the autonomy system of the cars like AI, if you will, from thinking about things in, I call it like, 2.5D. It's like -- think of taking like isolated pictures and doing image recognition on pictures that are harshly correlated in time but not very well and transitioning to kind of a 4D where it's like -- which is video essentially." So they weren't completely ignoring time even in the 2.5D approach, but it's only taking into account of it in a very rough way.
And in this video, it's all talking about recognition objects/road features in a static context, and there is no mention of detecting the velocity and acceleration of such objects using vision.
How Tesla uses neural network at scale in production
Edited Transcript of TSLA.OQ earnings conference call or presentation 22-Jul-20 9:30pm GMT
I don't think people really care about the mechanism of how it splits the video stream into individual frames before feeding it into a NN, only that the NN now takes into account the data from more than one frame at once (not just the current frame it is processing), and allows detection of velocity and acceleration of objects (especially now in the absence of radar, which previously could have at least done some of this).
Edit: I wrote my comment before reading
@powertoold and he said basically the same thing in a more succinct way.
