Think I need a new PEM Fan... Anything else to check?

[gregd]

Hi folks,

This is weird. Not making sense. I need some sanity...

To my comment about PWM on the output from the PEM turns out to not be correct. The meter is seeing a glitchy AC component to the output, but it looks more like noise, not PWM. Here's what it looks like on the oscilloscope with and without a 6w bulb attached. The voltage is in the few mv range, moving around a bit, and about 13.something open circuit when the car wakes up.



Now, here's the puzzle. I disconnect the bulb, leaving the scope attached. It, and a volt meter in parallel, should be the only things between the connector under the car, and the connector at the PEM. But when I disconnect the bulb, the voltage RISES SLOWLY, in a typical R-C fashion. After several seconds (5+ maybe?) the voltage crosses about 10v I get an 1144 alert in the car. Repeat. Attach the bulb for a few seconds, disconnect, wait, 10 volts, beep-beep-beep. I repeated the experiment on the Motor Fan connector, and the identical R-C behavior repeats there too. Since the motor fan is is associated with the 1146 alerts, and these don't show in the cabin (it boggles the mind why not), I download the logs after. Yep, 1146s after the bulb is disconnected.

SO for the budding EE's in the group, here's the problem. If the source of the high resistance is a presume-failed connector under the PEM, that means that the capacitor MUST be on the blower motor side of that connector. R, then C. But there's supposed to only be wires between the two. Certainly no sizeable capacitor. And given that the 1144/1146 alerts are delayed, the voltage sensing must be after that capacitor. WHAT IS GOING ON?

The only thing I can think of is that the high resistance is internal to the PEM. Is there any other connector up-stream that could account for this, or any other fuse which is in line with BOTH of the fan connections, yet allows the PEM to otherwise work normally? (Note that both of the 20A blower fuses under the PEM cover check out good.)

Help?

Greg

 

[ML Auto]

It's been over 40+ years since I've had an electonics class, so i'm a bit confused at what you are attempting to ascertain. If you dont want to get at the connector from the bottom, remove the PEM cover and you can get at the terminals from inside the PEM.

 

[gregd]

It's been over 40+ years since I've had an electonics class, so i'm a bit confused at what you are attempting to ascertain. If you dont want to get at the connector from the bottom, remove the PEM cover and you can get at the terminals from inside the PEM.

The point, basically, isn't that of not wanting to dig into things, it's trying to understand where to dig. I think what I have determined that the infamous PEM connector, and everything downstream of it cannot be where the fault is.

I'm hoping somebody knows of some other place farther upstream where the two power sources come together in such a way that it could explain the symptoms. If I open the PEM up, further troubleshooting will be hampered by not being able to power anything up.

 

[hcsharp]

The scope traces don't necessarily mean there's a C in the circuit. Among other things, we haven't seen the shape of the curve when it changes, up or down.

But what baffles me the most is why you don't just pull the PEM and inspect the connectors. Unplugging it once is not going to ruin your car. It would save you from a lot of speculation and other time-consuming guess work.

 

[gregd]

Pulling the PEM, which I have never done, is a multi-person operation, and there's only me and the cat in the household. So there is risk involved to do this, which I need to take seriously.

Nobody's responded to the question of whether the PEM can be flipped (rotated) up on the back bolts if I simply remove the 3 small screws at each end and the 3 power cables going to the motor. That might make it reasonable for a single person to accomplish with some wooden wedges and such. But...

To the "RC" part of the circuitry, I can see the discharge into the bulb when I connect a small (ma) analog meter directly in line. Nice little "bump", then it settles down to a bit under 1ma. No warmth at that current, so it's not the temp coefficient of the bulb. I've been tinkering with electronics since before I can remember, and I'd be extremely surprised that it is anything else. 15k resistance, charging a maybe 1,000 mf cap feels about right. That's not a bad PEM Fan connector.

 

[PV-EV]

I’ve always pulled mine solo. Except the first time when Carl Medlock was here. A helper for 5 minutes would do it. Don’t expect much from the cat, they don’t appreciate sports cars. If you are sure it’s not the motor or the wires or the connectors that pretty much leaves only one thing.

 

[KJA2001]

I’ve always pulled mine solo. Except the first time when Carl Medlock was here.

That really could be very badly misconstrued

 

[ML Auto]

You can keep the PEM powered once the cover is off by just depressing the safety switch. Since you have been "tinkering" for awhile, I shouldn't have to advise you what not to touch without proper protection.

 

[gregd]

You can keep the PEM powered once the cover is off by just depressing the safety switch. Since you have been "tinkering" for awhile, I shouldn't have to advise you what not to touch without proper protection.

Rubber soled shoes, one hand in your pocket at all times. Static discharge wrist strap. Assume everything is live, even if powered off. It's amazing how much energy can be stored in the HV "flyback" section of an old TV set, or how easy it is to fry an IC if you touch a pin without being grounded. There's a keyboard encoder chip somewhere in a Santa Clara County landfill with my fingerprints on it...

I'm going to be studying the various pictures folks have posted on the internals, before going any further. A schematic for this thing would be reeeeally handy about now. There's got to be something really simple that I'm missing.

Are there any records in the logs related to the PEM/Motor fan that the TeslaGLOP program doesn't happen to decode and graph?

 

[ML Auto]

Schematics for the PEM internals are not available, it would be really nice if someone could convince Tesla to release them. I've read some posts stating Tesla is not even repairing them now, so maybe they will.

To answer your other question reguarding the PEM-no, you can't just rotate it upward. But you can remove all the holdown bolts and 3 motor cables and lift it just enough to get at the connector.

 

[gregd]

Ok, another update.

My higher-power meter arrived, along with a fresh stock of PowerPole connectors. Saddled up a Big A** Battery (94AH Deep Cycle Marine / RV) with short(er) wires, looking for a ruling on the max Blower Motor current draw. Still a bit of voltage sag with the wires, but it looks fine to me. Let me know if you disagree (i.e. that it needs replacing). Final chance.



I also took a look at the output of the PEM, right as the car wakes up, to see if that R-C time constant is there. It should be, if my earlier theory is correct. It does not appear to be, at least, not quite.

Three experiments.
1. Watching the 'scope as the car first wakes up, no load on the PEM. Expecting to see the RC-induced voltage rise. Result: Voltage immediately spiked to 13.4 and stayed there. Hmmpf.

2. Hooked up a resistor in series with the 6w bulb. 500-ish ohms, so the bulb probably wouldn't light, but I should see a larger voltage out of the PEM. Attached that to the output, which was at 13.4v, and watched the output drop. Result: nice smooth drop to about 0.8v over the course of a few seconds, and it stayed there. Expected.

3. Left the resistor + bulb connected, waited for the car to go to sleep, then woke it up again. Result: Voltage spike and slow drop back to 0.8v, as in test #2.

Not sure what to make of this. The first test suggests that there is some intelligence behind the output, and that it's not as simple as, well, it could have been. The second experiment, however, is consistent with the original RC theory. The third kind of feels like a circuit inside the PEM is (incorrectly) detecting an overcurrent (a few MA!) and shutting down the output. I need to find something "loose" that can explain this. I presume that if the 12V input to the PEM (36 amp connector on the passenger side) were loose (I tried wiggling it - it is tight - but have not tried removing it), there'd be a bunch of other alerts popping off as other circuits went undervoltage. What remains, perhaps, is the 12v going to the board that drives the PEM Fan connector. That would be a common point upstream that might explain stuff. Or maybe a loose ground (though the fan motor is isolated)... Any thoughts on this?

I did find some wiring diagrams in the service manual, but only the external connections. Nothing that goes inside of the PEM. My own diagram has been updated a bit. Based on the earlier comment about the two positive lines being tied to the APS+ line, I've removed the external shunt that I postulated must exist. Given that I'm measuring less than 1 ohm between the positive Motor and PEM lines, that also means that at least the positive sides of the under-PEM connector are in good shape.

 

[PV-EV]

Look at photo #4 in post 39 for a hint about how the pins are connected to the board inside the PEM.

 

[gregd]

Look at photo #4 in post 39 for a hint about how the pins are connected to the board inside the PEM.

Exactly. That big fat wire harness at the top, which I'm guessing goes to the two fuses under the cover. Is there any history of that connector going bad?

Need to figure out now how to open the top without removing the entire PEM. The security torx along the front will be tricky with the tub in the back.

 

[DeedWest]

Greg,

Sorry this happened, though you are lucky to be sitting at a full Standard Charge. A similar thing happened to my 1462 two months ago. Mine was faulty pins inside the PEM connector itself. I would strongly recommend taking a look at the connector first, and rebuilding the pins. If that doesn't work, check the connector soldered to the PEM and the CIC board as @MLAUTO and @jfischer have pointed out. If you take it to the SC, they're just going to order a new CIC board "to fix inoperative PEM fan". Lead times on that are unknown, and it's likely around $1,798, as I have seen this repair cited on an old vehicle of mine's service records.

Any local Roadster owners near you, or any friends that could come by and help you remove the PEM? If I weren't in Texas, I'd help you myself. The other thing I'd recommend, is seeing if @CM_007 would be able to fly out to you and help fix it. Guy is a wizard, and serviced these cars in their heyday. I'd trust him with my life.

 

[gregd]

Greg,

Sorry this happened, though you are lucky to be sitting at a full Standard Charge. A similar thing happened to my 1462 two months ago. Mine was faulty pins inside the PEM connector itself. I would strongly recommend taking a look at the connector first, and rebuilding the pins. If that doesn't work, check the connector soldered to the PEM and the CIC board as @MLAUTO and @jfischer have pointed out. If you take it to the SC, they're just going to order a new CIC board "to fix inoperative PEM fan". Lead times on that are unknown, and it's likely around $1,798, as I have seen this repair cited on an old vehicle of mine's service records.

Any local Roadster owners near you, or any friends that could come by and help you remove the PEM? If I weren't in Texas, I'd help you myself. The other thing I'd recommend, is seeing if @CM_007 would be able to fly out to you and help fix it. Guy is a wizard, and serviced these cars in their heyday. I'd trust him with my life.

Thanks, @DeedWest. I've been discussing the latest measurements with Peter at Gruber, and looking back in the logs with his tools, we see some evidence that the fan connector was having troubles back in December. A bunch of 1146 errors that went away on their own, but his log parser reveals that they were under-current events, all on the "motor" side of the connector. (This, mind you, within months of the SC declaring the connector was fine.) Then the weather got cold, and no more errors until June, when everything failed at once. The only way this makes sense, if it even does, is if the motor side actually failed last December, and the blower was only running from the PEM side. Then the PEM side failed, and we're done?

He also did some tracing of the wiring, and has confirmed what @MLAUTO said about the positive side. Both positive pins go through the 20 amp fuses under the cover, and are then wired together to the APS + side, which comes in through the connector on the side. The minus side is where the regulation occurs, and there is a voltage measurement point, immediately followed by a hall-effect current sensor, and then the connector pins. So with both sides being driven together (the firmware configuration flag, that's associated with the dual fan upgrade), it's still inconsistent that that if the motor side had failed, that it wouldn't continue to have 1146 alerts, since both sides should share the load more or less evenly. There were also errors related to the traction motor's temperature (1115, 1116), a few weeks before the complete failure. We don't know if they were related. So I'm going to dig in and see about that bottom connector, but I'm still skeptical.

Thinking out loud as I write this, if the power going to the whole board were flaky, that might cause other readings to be flagged as bad, too... The motor temperature sensors connect to the same PC board as the PEM Fan. First connector on the left.

 

[gregd]

Another update. Progress, I suppose, but no solution.

I opened up the PEM's top (after inhibiting APS first), and looked for anything amiss. The idea of doing this before going after the PEM connector on the bottom is that I can do it without disconnecting any wires, especially any HV ones. I found two issues.

1. One of the pins on the connector next to the one that goes to the blower had one pin raised, apparently when the mating end was re-installed. I can't find any information on what this connector goes to, or what that pin specifically does. The connector is a 6-pin, and the errant pin was the one closest to the back of the car, on the passenger side. It pushed back into place with an angle bracket wrapped with tape. Seemed to be about right in terms of the push required.

I'm hoping the pin goes to the motor's temperature sensor (related to the 1115 / 1116 errors). If so, I can't test it until after the car is back to running. None of the PEM Fan pins appeared to be similarly displaced. Darn.



2. The top of the board was sitting loose, about a mm separated from the pair of standoffs that clearly were there to support it. One of the standoffs was clearly intended to also serve as a ground, since there is copper around and through the hole in the PC board. The other was just bare fiberglass. I added screws to both, and cinched them down. My fear is that the vibration caused by the missing screws might have cracked a trace, or damaged a component.

The back story on this PEM is that it's a replacement for one where the PEM Fan connector had failed in 2016-ish. The SC manager (Randy) worked some magic and worked out a swap with the factory for this unit; I only paid the labor for the replacement. I suspect now that someone replaced the board, but forgot the screws.




3. I also removed and re-inserted a couple of the cables, just in case they had dirty contacts, and swapped the 20 amp PEM and Motor fuses position. Re-assembled everything and powered it up. Unfortunately, no change in the resulting behavior, as seen from the cable harness at the blower motor.

{sigh}

Next step is probably to attack the bottom of the PEM and see if the fan cable is ok, unless someone has a better idea.

 

[gregd]

Huh. I just noticed that the image in post #39 has the same connector pin raised. @jfischer, did you notice this on yours?

Is it supposed to be that way?

The base of the pin on mine certainly looked like it had a shoulder on both sides just above the connector body, and none of the other pins had that. It also pushed in without a fight.

What does that pin go to?

 

[PV-EV]

the fan connector is a two stack 90 degree. The ESS heater is the next one over and has six pins, 3x3. Five pins are used. Do you need the car schematics? Kind of looks like someone did a poor job of installing the CIC board in your PEM.

 

[gregd]

the fan connector is a two stack 90 degree. The ESS heater is the next one over and has six pins, 3x3. Five pins are used. Do you need the car schematics? Kind of looks like someone did a poor job of installing the CIC board in your PEM.

I'd LOVE to have the schematics for the PEM, or at least the CIC board. I have the service manual, but it doesn't seem to include anything inside the PEM, presumably since Tesla considered (past tense?) it an FRU.

Since I don't recall I've ever had the battery cold enough to need heat, and the first owner lived in the SF Bay Area, I suspect the battery heater may have never been used. For completeness sake, do you know which is the pin that's not used?

Agreed on the poor assembly. I wonder what else is missing? As I said, the PEM itself has an unknown pedigree. It came with Roadster Sport badging (changed to non-sport at my request), and I understand that it was a "used" unit, as opposed to "remanufactured". As I noted, my car seems to have been the (or one of the) ones that traced the 1144 / 1146 errors to the burned connector pins and the inadequate specs on that connector, and the SC manager at the time "worked a deal" with the factory to get me the unit as a trade, with me only paying the labor. $1k vs $10k, round numbers. Unfortunately, that service manager is no longer there.

Thinking about the vibration-induced failure theory. The loss of the fan occurred while the car was sitting in my garage, charging, and there were no 1144 / 1146 errors recorded since December even though the car was being driven during that time. I'm thinking (hoping) that I don't have a cracked trace or lifted chip pad somewhere, as that would be very unlikely given the lack of vibration at the time. There was a smell of burnt-something, however, when I opened the lid. Not sure if that is normal or not, given the voltages and currents managed therein. It seemed pretty clean (no dust/dirt), and I did not see anything obviously fried. I may open it up again and take another look, before trying to mess with the connector on the bottom.

 

[PV-EV]

I have car schematics, not PEM. I’ve never seen PEM schematics nor hear of anyone who has. The ESS heater/connector would not have anything to do with your symptoms. I think the ESS heater is active when the charge port is flashing blue but never really confirmed that. I get lots of that here! Not to sound like a broken record, but you need to get to the fan connector...the high voltage stuff is not a big deal if you follow the procedures and shut it all down.