Third, and this is the big one, they instruct participants to report "only problems you considered serious." If people really like the car, possibly in part because of what they've read in CR, they are less likely to report a problem. If they expected some problems because the car and the company making it are both new, they are less likely to report a problem. If they feel they received good service, they are less likely to report a problem. Conversely, if people don't like the car, the dealer, or the manufacturer they can report virtually anything as a serious problem. Many people report rattles. Some do not report transmission replacements because they got a loaner car and the warranty covered everything. Combine all of these factors, and Tesla owners can HONESTLY report far fewer problems on the CR survey than they do on ours, which instructs people to report ALL repairs.
My issue with CR's approach is that it introduces a large amount of subjectivity into what is being measured, such that in the end it's unclear how much they're measuring reliability and how much they're measuring how people FEEL about the car, about dealer service, about the manufacturer, and so forth. A key advantage of their approach is that they don't have to thoroughly screen responses, since they have no clear, objective criteria of what should and should not count in their analysis. With a subjective question, you can assume that if someone reports something as a problem, then it counts as a problem. And if they don't report it then it should not count. With objective criteria like we use, every response must be thoroughly checked and adjusted as necessary to correct errors. This is far more practical with sample sizes like ours rather than sample sizes like theirs.
The results are not contradictory if you know what they actually represent. Based on CR's results, owners felt that the Tesla Model S had an about-average number of serious problems during the year ending in April 2014. Based on our results, the cars required far more repairs than the average car during the year ending June 2015.