AP2.0 Cameras: Capabilities and Limitations?

[lunitiks]

 

[Bladerskb]

Its been over 40 mins. Those tesla ninjas must have gotten to him. Probably remotely overrided his car and is driving him out to sea as we speak with him locked in it screaming..


Rest in peace my brave warrior!

 

[stopcrazypp]

I noticed a new thread here:
Inside the NVIDIA PX2 board on my HW2 AP2.0 Model S (with Pics!)

 

[Bladerskb]

I noticed a new thread here:
Inside the NVIDIA PX2 board on my HW2 AP2.0 Model S (with Pics!)

He betrayed us!
I thought we had the exclusive?

 

[KyleDay]

Here is the thread - I did the leg work. I figured a new thread was appropriate.

Inside the NVIDIA PX2 board on my HW2 AP2.0 Model S (with Pics!)

 

[KyleDay]

Here's a pic for everyone's enjoyment.

 

[lunitiks]

Yeah it really deserves its own thread. This one should continue the camera vision route!

Big day

 

[verygreen]

Looking into blinding sun, but can you see the difference between red, orange and green on traffic light warning signs:

View attachment 227858

Red sign seen in the back facing camera (there is no stop sign in the set):

View attachment 227859

Combined intersection and parking lot scenes, just like the ones made by @bjornb:

View attachment 227860 View attachment 227861 View attachment 227862

Thanks, this is very impressive processing!

 

[verygreen]

Ok, to tie some loose ends from before.

The Tuning parameters screen requested:

The GPS screen (wow, 12 sats in my garage)

btw on ape there's an "inertializer" binary that I guess also might be doing stuff related to positioning?

And this is ape on a parked car (in park), happily blasting away analyzing my garage. Still not sure how much it really consumes.

 

[verygreen]

I figured out the snapshot trigger system to some degree I think I now can make them pretty much arbitrary, so if there's a particular scene desired with all cameras engaged, request away and I'll see if I can do it.

The triggers pushed from the mothership basically look like this:

{
    "version": "0.3",
    "triggers": [
        {
            "query": {"$nop": true},
            "topics": ["git-hash"],
            "requester": "ack-triggers-0001",
            "max-requests": 1,
            "timeout": 1494633000,
        },
        {
            "query": {"$nop": true},
            "topics": ["git-hash"],
            "requester": "wifi-connected-0001",
            "max-requests": 1,
            "timeout": 1494633000,
            "request-time-to-live": 259200,
            "request-wifi-hold-time": 259200
        },
        {
            "query": {"$gt": ["@MiscTriggerFields.uptime_s", 300]},
            "topics": ["dense", "can"],
            "requester": "random-replay-0001",
            "min-period": 86400,
            "max-requests": 1,
            "timeout": 1494633000,
            "sample-likelihood": 0.000016666,
            "request-time-to-live": 259200,
            "request-wifi-hold-time": 259200
        }
    ]
}

"topic" field lists possible data to get, options are: logs git-hash calibration-main calibration-narrow main main-replay narrow-replay dense sparse default

They started to request sparse in random snapshots eventually too.
Where sparse is 1fps video from all cams, and dense is 30fps from main and narrow (+radar).
timeout is unix time when this trigger becomes invalid.

Query language is json queries apparently, possible operators are: $not $neg $gt $lt $gte $lte $eq $strict_eq $feq $neq $minus $div $pow $and $or $sum $prod $conv

Initial 300 seconds since boot eventually got replaced with 360, no idea what it's now since I've not been getting those requests for a few days now.

Possible arguments:

MiscTriggerFields.driver_side
MiscTriggerFields.git_hash
MiscTriggerFields.local_time.tm_gmtoff
MiscTriggerFields.local_time.tm_hour
MiscTriggerFields.local_time.tm_isdst
MiscTriggerFields.local_time.tm_mday
MiscTriggerFields.local_time.tm_min
MiscTriggerFields.local_time.tm_mon
MiscTriggerFields.local_time.tm_sec
MiscTriggerFields.local_time.tm_wday
MiscTriggerFields.local_time.tm_yday
MiscTriggerFields.local_time.tm_year
MiscTriggerFields.local_time.tm_zone[0]
...
MiscTriggerFields.local_time.tm_zone[49]
MiscTriggerFields.local_time.valid
MiscTriggerFields.mono_time_ms
MiscTriggerFields.uptime_s
MiscTriggerFields.vehicle_model
MiscTriggerFields.wall_time_ms

min-period is how frequent to evaluate the trigger, in ms

As you can see the request actually allows Tesla quite a bit of flexibility about when to take those snapshots, how often, on what kind of cars, what firmwares and so on.

 

[mitchellh3]

The triggers pushed from the mothership basically look like this:

Interesting. It looks like they don't have the ability [yet, seems trivial to push in an update] to trigger a snapshot at specific locations or types of locations. For example, if they were going to use this for mapping, I would assume instead of spray-and-pray there'd be a trigger field for intersection based on GPS.

Still, very interesting stuff!

 

[lunitiks]

if there's a particular scene desired with all cameras engaged, request away and I'll see if I can do it

How about some in twilight conditions? (I remember Musk tweeted something about day/night being easy, but dusk/dawn the hard part.)

Also perhaps some pics in heavy, super busy city traffic, with pedestrians and cars everywhere around the car.

Third, how do the cameras perform when exiting a dark tunnel entering sunlight? Any problems seeing oncoming traffic?

Finally: I'm curious to if heavy rain impairs the vision (or even more relevant, snowfall, but that's probably going to have to wait)

 

[AnxietyRanger]

Much thanks @kdday!

Anyone feeling this theory?

 

[lunitiks]

So I made a quick overview of the ECU connector layout (subject to possible errors). Note that the USB-C port is not referenced. However as @verygreen has pointed out, that should be for clip logging

 

[lunitiks]

btw on ape there's an "inertializer" binary that I guess also might be doing stuff related to positioning?

Yeah I bet this is info from the IMU (Inertial Measurment Unit), i.e. gyro/accelerometer data that perhaps works in conjuction with the dead reckoning stuff in the gps module. That is, figuring out where you're at even with no sat signal like in tunnels.

Anyway I noticed in your screenshot that there's a value for "Wheels/Aligned/IMU/INS"... I wonder what "2022" means

 

[J1mbo]

So I made a quick overview of the ECU connector layout (subject to possible errors). Note that the USB-C port is not referenced. However as @verygreen has pointed out, that should be for clip logging

Also possible that the USB-C port could be used as a display port?

 

[Jonas_man]

Looking into blinding sun, but can you see the difference between red, orange and green on traffic light warning signs:

View attachment 227858

Red sign seen in the back facing camera (there is no stop sign in the set):

View attachment 227859

Combined intersection and parking lot scenes, just like the ones made by @bjornb:

View attachment 227860 View attachment 227861 View attachment 227862

can we combine the side images into a kinda of a 270º image (in the bottom image set)?
I have no photoshop skill for that.

 

[lunitiks]

Yeah that would be really great

 

[Jonas_man]

Ok i took some time to do it. I really have no skill but it should give a good idea.
Maybe with more images from the fenders we can do a near 360.

 

[J1mbo]

Put this together with @bjornb's example, before @DamianXVI's excellent work with the colours.

The problem with these images is that they are all taken at a slightly different time, so moving objects are misaligned. Also, I don't have the tools to correct for the lens distortion, but the overlap between the fisheye and the B-pillar cams hides the worse of it.

EDIT: @Jonas_man beat me to it *kudos*