MCU fails for the second time

[verygreen]

Ha. So much for the s/x being the flagship models and more technologically advanced.

I am sure updated cid (and ic?) are coming to S/X sometime soonish too. Updated CPU and memory, more space, what's not to like?

I see how they can get rid of the IC computer and just replace that with a cid unit that drives two displays, this would simplify a bunch of things and would close some security problems they've been having due to current design too.
Combine that with autopilot board colocated with the CID in the same block so there's no externally visible unprotected network to plug into for attackers and the deal starts to look very compelling for them I bet.

 

[apacheguy]

Wait, the APE is initegrated with the CID on the Model 3? What are they going to use the 64 gb emmc for? Hopefully not dashcam recording.

 

[verygreen]

Wait, the APE is initegrated with the CID on the Model 3? What are they going to use the 64 gb emmc for? Hopefully not dashcam recording.

no idea what they are going to use 64G for, they don't need to, but the more you have and less you use it the better is for the write levelling, the write amplification is lower and everything is faster. And indeed I verified they only use 32G of that emmc, so pretty much 50% reservation, very likely a stab at greatly reducing write ampification for now.
APE has its own separate 64G emmc, I guess it's harder and harder to procure smaller ones now? Also it helps with longevity (ape logging is on a whole different level compare to cid, considering all the camera snapshots and the like). Similarly only 32G only in use

 

[verygreen]

Ah, scratch that, apparently only 32G, not 64G, so only ~4G of unused buffer space

 

[apacheguy]

So 32 gb for cid and 64 gb for the ape. How do you know the ape and cid will be in the same head unit?

 

[verygreen]

So 32 gb for cid and 64 gb for the ape. How do you know the ape and cid will be in the same head unit?

32 for ape, and 32 or 64 for cid (ice as they call it on model 3)

 

[RyanT]

Tesla just pushed out a software update and I got the notification on my phone so I went out to the car a little later and my 17" MCU is dead. It was working just fine before that. I'm at 155k miles. There's no way I'm paying for this out of pocket. They bricked their own CPU firmware so there's just no way I'm shelling out $3k for the repair. Absolutely no way. They want me to drop the car off at SC...

I had the exact same thing happen yesterday : /

 

[bjornb]

32 for ape, and 32 or 64 for cid (ice as they call it on model 3)

I am a bit curious about the storage capacity in our cars..
What is the storage on the IC on S/X?
Is there any storage on the gateway?

Is the recap below correct?

Tesla Model S/X:
IC: ??
CID/MCU: 8GB emmc + sdcard for maps (size?)
APE: 64GB emmc (32GB in use)

Model 3:
ICE/CID: 32GB emmc + sdcard??
APE: 64GB emmc (32GB in use)

 

[verygreen]

I am a bit curious about the storage capacity in our cars..
What is the storage on the IC on S/X?
Is there any storage on the gateway?

Is the recap below correct?

Tesla Model S/X:
IC: ??
CID/MCU: 8GB emmc + sdcard for maps (size?)
APE: 64GB emmc (32GB in use)

Model 3:
ICE/CID: 32GB emmc + sdcard??
APE: 64GB emmc (32GB in use)

IC is also 8G, same as cid.
micro-sd for maps: 16G
ic and cid also have 64M flash that's accessible as MTD device.

gateway has sdcard (full size) for storage of logs and some other stuff that's 4G in size.

APE 2.0 is actually 32G, but lvm partitioning ensures that 4G of those are never used (at least for now). The partitioning info was in sectors, not in kilobytes which is what got me confused at first.

The lvm volumes are:

    lvc        var   1GB
    lvc  autopilot   1GB
    lvc    factory  32MB
    lvc        map  10GB
    lvc       home  10GB

model 3 is 64G emmc, but not all of it is in use similar to ape. It does not appear to have external maps sdcard. + APE same as S/X (hw2.5 storage seems to be the same as hw2.0, but I am not sure if it has additional emmc for the other node or not at this point)

 

[Dborn]

My MCU died last week at 42000 km. Was doing a local trip to a restaurant and nav worked, but trip showed zero km and zero kWh. So, I rebooted instrument console, no benefit. On parking, did a reboot with brake depressed, screen went black and stayed that way. Only way to start and drive the car was to do a "reboot" every single time, screen remained black however. MCU replaced under warranty. Diagnosis was a very helpful " internal error".
So no actual diagnosis. Worrying though. It is winter here right now so temperature was not an issue.

 

[appleguru]

My MCU died last week at 42000 km. Was doing a local trip to a restaurant and nav worked, but trip showed zero km and zero kWh. So, I rebooted instrument console, no benefit. On parking, did a reboot with brake depressed, screen went black and stayed that way. Only way to start and drive the car was to do a "reboot" every single time, screen remained black however. MCU replaced under warranty. Diagnosis was a very helpful " internal error".
So no actual diagnosis. Worrying though. It is winter here right now so temperature was not an issue.

This sounds like filesystem corruption caused by the forced reboot, and not necessarily hardware failure. Most of the critical OS files are on partitions mounted Read Only, which helps here, but doesn't make corruption impossible. I suspect a "reimage" of the MCU would have brought it back to life, but no idea if tesla has a way of doing this or not.

@verygreen what filesystem is being used? Is it journaled?

 

[MP3Mike]

This sounds like filesystem corruption caused by the forced reboot, and not necessarily hardware failure. Most of the critical OS files are on partitions mounted Read Only, which helps here, but doesn't make corruption impossible. I suspect a "reimage" of the MCU would have brought it back to life, but no idea if tesla has a way of doing this or not.

Nope this is the exact failure mode of when the emmc is worn out.

 

[verygreen]

This sounds like filesystem corruption caused by the forced reboot, and not necessarily hardware failure. Most of the critical OS files are on partitions mounted Read Only, which helps here, but doesn't make corruption impossible. I suspect a "reimage" of the MCU would have brought it back to life, but no idea if tesla has a way of doing this or not.

@verygreen what filesystem is being used? Is it journaled?

yes, it's journaled ext3. Yes, there's a way for tesla to reimage the CID, but without seeing actual logs we cannot really tell if it's emmc failure or something else.
emmc failure is pretty likely in my view (Which reminds me I need to get smartctl or some such to be able to more accurately gauge mine)

 

[verygreen]

alas, apparently the main emmc does not report wear level (or I cannot find the right datasheet).
main emmc reports that it is "H8G2d" which I think is a hynix part?

The maps sdcard on the other hand does report it. You can get it by doing this:

cat /sys/kernel/debug/mmc2/mmc2\:0001/ext_csd | python -c 'import binascii, sys; print "~%d%% wear" % (ord(binascii.unhexlify(sys.stdin.read().strip())[0x5e])*10)'

Naturally wear level is very low on my car

 

[RyanT]

I wonder if it's a coincidence that three of us had the failures last week. I'm also out of warranty, but it's ok. It's a car and things break. It has 60k miles on it. Same symptoms as above except I could drive without a reboot. It just took about a minute while the car said "waiting for systems to power up". I guess it would fail to start up the mcu and at some point give up.

 

[yobigd20]

I wonder if it's a coincidence that three of us had the failures last week. I'm also out of warranty, but it's ok. It's a car and things break. It has 60k miles on it. Same symptoms as above except I could drive without a reboot. It just took about a minute while the car said "waiting for systems to power up". I guess it would fail to start up the mcu and at some point give up.

i'm convinced it was a bad software patch they pushed out that bricked a lot of MCU's...

 

[ImperialG]

Had the same issue with the bricked MCU and now I'm faced with the issue of having to pay for it out of warranty. Seems like something that is a widespread issue for early model S's

 

[apacheguy]

IC is also 8G, same as cid.
micro-sd for maps: 16G
ic and cid also have 64M flash that's accessible as MTD device.

1st gen IC is 4 gb

 

[robapodaca]

2013 (Oct) P85. My MCU locked up on me last week... while driving, the navigation map wasn't updating. I tried the scroll button reset and the screen went dark and never came back. Called Tesla and they had me try the reset also. Ended up back home and my Air Conditioning wouldn't shut off, also couldn't charge the car. Spent an hour on the phone with Tesla, multiple tries to reset, no luck. AC ran most of the night, but at one point stopped in the middle of the night - can't turn it back on, can't access anything anymore, even thumbwheels can't access stuff like open sunroof, turn on AC, etc. App can't access the car.

Took it into the Service Center (no Tesla loaners available, using a Hertz rental - they never have loaners available any more) and they notified me they have backordered a MCU for me. Just rolled over 50k miles last month.

I have had various problems with screens freezing, both the instrument cluster and the center screen. Both have been replaced in the past.

Interesting note - Tesla mentioned over the phone to disconnect any USB devices, as they can cause an issue at boot-up, if any of my music has corrupt files.

Very disappointed with Tesla quality lately.

 

[DFibRL8R]

To add to the list, my 2013 experienced the delamination of the touchscreen (bubbles) after about 3 years and 75K miles, I paid to replace out of warranty in January this year for $983 plus tax (parts only, labor goodwilled). The MCU was not replaced, just the screen (early on I think they were replacing both, now do it separately).

Just 3 months later I had a defect in the screen (thin red line constantly), I decided to wait to fix until it was closer to the 12 months that the darned thing is warranteed for post-repair.

A few months after that the MCU failed while I was driving on a road trip in the heat a long way from home, my destination or the nearest service center. Nav was frozen, went to reboot and screen went dark never to return again, no climate control, no odometer readings. Fortunately it would still Supercharge but would not charge off the mobile connector/UMC (in fact my UMC coincidentally crapped out when I attempted to use it to charge while the MCU was in a crippled state). I was charged $2370 for the REMANUFACTURED MCU, $165 labor (labor discounted due to the fact that they needed to also replace the defective touch screen I had previously paid for). No problems since other than the fact that my lifetime stats for Wh/mile were wiped. They did send me a new mobile connector/UMC for free which was awfully nice though. No hard feelings, somewhat expected some of this as an early adopter. Hopefully my $2500 remanufactured MCU lasts longer than the original...