Technical: How does Autosteer work?

[lunitiks]

I'd like to get a deeper understanding of what actually happens from the moment your forward cameras (can) "see" that your lane lines start to curve ahead of you, until your steering wheel physically starts to turn, keeping you centered inside the lane. Both software and hardware-wise.

My problem is that I don’t know jack about the software. OTOH, I think I can contribute with some hardware and wiring information, and from there leave the SW questions for others to answer.

So allow me to begin...

• Your Daimler/Mercedes steering wheel is attached – through various shafts, rods and intermediate links – to a steering rack that sits between your front drive wheels. Basic theory of operation here.

• On the steering rack, Tesla has mounted a ZF/Hella Power Steering Motor. It looks like this:

• The electronically commutated, brushless Power Steering Motor is what physically forces your car to turn when no hands on the wheel (in Autosteer). The Power Steering Motor works by turning a drive nut via a rubber belt, which - via a series of ball bearings - translates into linear motion of the rack. Linear movement of the rack is transferred to the tie rod ball joints, steering arms and road wheels. Voilà, your car turns.

• The Power Steering Motor is controlled by a sealed Power Steering ECU, which is mounted right next to the Motor. Looks like this:

• The Power Steering ECU houses microprocessors and the output stage for the Motor. This thing is (of course) subject to interrogation for diagnostic data and firmware updates by Tesla, which happens through 500 kbd Chassis CANBUS. (BTW, autopilot hardware 2.5 sports a brand new power steering ECU. I don’t have many details about it other than it’s now fused though a new autopilot 2.5 e-fuse. More on that here and here.)

• The CAN-wires from the Power Steering ECU make very little sense just from looking at the wiring schematics. They are spliced and diced all over the place, with connections to the iBooster (power braking), the ABS / Stability ECU, Air Suspension Controller, OBDII, Instrument Cluster, Gateway / Center Display and CAN Diagnostics. In AP2-vehicles, it also have a link to the AP2ECU).

So what I would really, really like to know is exactly *how* the visual input (to the cameras) translate into steering commands in this Power Steering ECU. What happens in between? What kind of messages does the ECU get – does it get decisions, does it get “suggestions”, does it get “confidence levels”, or something else? Can anyone help me answer this?

@verygreen you’re of course on the top of my list. But anyone, please chime in!

 

[J1mbo]

Here's a datasheet for you (pdf) - pretty sure it's the paraxal server unit variant on page 10.

Seems clear that AP sends explicit instructions to the ECU. It's probable that AP also reads back from the ECU.

 

[BigD0g]

AP definitely sends explicit instructions to the gateway, the gateway then confirms the signature on the instructions to make sure they are valid and then the gateway tells the steering to move based on those instructions. The APE tells the software what and where it sees it being lane lines and other cars and obstacles and then the QTCar app picks the algorithm to use.

 

[buttershrimp]

This is how autosteer works

 

[lunitiks]

AP definitely sends explicit instructions to the gateway, the gateway then confirms the signature on the instructions to make sure they are valid and then the gateway tells the steering to move based on those instructions. The APE tells the software what and where it sees it being lane lines and other cars and obstacles and then the QTCar app picks the algorithm to use.

Anyone got an example (readout) of CANBUS-signals sent to the Power Steering ECU?

(For it to be meaningful, I guess we need to translate it too)

 

[buttershrimp]

Anyone got an example (readout) of CANBUS-signals sent to the Power Steering ECU?

(For it to be meaningful, I guess we need to translate it too)

Sorry I crashed your thread @lunitiks .... please continue

 

[BigD0g]

Anyone got an example (readout) of CANBUS-signals sent to the Power Steering ECU?

(For it to be meaningful, I guess we need to translate it too)

https://skie.net/uploads/TeslaCAN/Tesla Model S CAN Deciphering - v0.1 - by wk057.pdf

 

[BinaryField]

I'm not really a controls guy, but my expectation is that the neural network feeds centreline deviation information to a classical control scheme, e.g. PID controller. That would seem like the most conservative way to approach the problem. I could be wrong though.

Here's a cool little briefing of the concept:

.

 

[lunitiks]

https://skie.net/uploads/TeslaCAN/Tesla Model S CAN Deciphering - v0.1 - by wk057.pdf

Thx, but seems wk hasn't got readouts for the Power Steering ECU there.

After doing a bit of googling, I'm pretty sure power steering runs on channel 6 (wk's only laid out can3)

 

[BigD0g]

I suspect someone has it mapped, now if they are sharing that map is a completely different story....

 

[secondstage]

Interesting topic here. Here are my thougs:
Maybe for lane detection there isn't a neuronal network. There could be a Hough-Transformation (

).
Next step would calculate your offset from the middle of the line.
How you translate that offset into a steering input? For that you need a control loop. Otherwise you would constantly bounce between your lanes.
So this would be the chain (just an overview. There is plenty other things that AP also does):

• streaming the video from the cameras with Video for Linux
  
• Analyze the Image with Hough-Transformation and neural net.
• calculate your offset position from the middle of the lane
• feed the offset into your control loop
• the control loop calculate the steering angle