Being an automotive journalist is a dream job, but it does come with a few sacrifices. Your inbox will quickly become completely unusable, because it’ll be inundated with nonsense studies and irrelevant cold-pitched media releases that nobody with even half a brain would spin into content. Case in point? Here’s a study claiming that the Mazda CX-5 is the “most disappointing” car to drive. Beg your pardon?
I like the way the Mazda CX-5 drives, and so does everyone I’ve recommended it to. They like the taut body control, the well-weighted steering, and the normalcy of a conventional automatic transmission, even if they aren’t car enthusiasts. So what gives? Do people just want cars that aren’t great to drive? I mean, that would explain why many Subarus sell well, but no. It turns out that the methodology used in this study is incredibly bad.
Let’s start with what Compare The Market AU was looking for with this study, which turns out to be surprisingly vague. Instead of individual negative experiences pertaining to a vehicle’s powertrain, steering, suspension, or braking, it just looked for a bunch of keywords. As per Compare The Market:
We analysed user reviews across a range of models, looking for negative mentions of “difficult”, “hard”, “confusing”, “uncomfortable”, and “disappoint”.
Notice how there’s no mention of negative keywords having to directly pertain to the driving experience, a decision that draws the entire title of the study into question. In addition, counting keywords rather than individual negative reviews could skew data, as it wouldn’t be unusual for a negative review to feature more than one keyword.
Also, all of the user reviews analyzed were posted on Edmunds, which is limiting in a few ways. Firstly, Edmunds is only one platform. Secondly, online reviews suffer from several forms of bias. As researchers at the Stevens Institute of Technology, Temple University, and the University of Texas found in a 2017 paper:
…two self-selection biases, acquisition bias (mostly consumers with a favorable predisposition acquire a product and hence write a product review) and underreporting bias (consumers with extreme, either positive or negative, ratings are more likely to write reviews than consumers with moderate product ratings), render the mean rating a biased estimator of product quality, and they result in the well-known J-shaped (positively skewed, asymmetric, bimodal) distribution of online product reviews.
The bottom line? From espresso machines (seriously, will I be happy with a modded De’Longhi ECP3630?) to cars, everything subject to online user reviews will have a bunch of exceptionally positive reviews and a handful of extremely negative reviews. Any non-polarized consumers are essentially disengaged, which means input data from online user reviews alone will be flawed.
Oh, and don’t think the flaws in this study stop there. The terrible data was then used in one of the most egregious, moronic normalization methods I’ve ever seen. To quote:
Once the data for the factors was collected, the factors were normalised, to provide each factor with a score of between 0 and 1. If data was not available, a score of 0 was given. The normalised values were then summed and multiplied by 20, to give each of the cars a total score out of 100. The cars were then ranked from highest to lowest, based on their total scores.
I beg your pardon? Look, this isn’t a good way of using the data because it doesn’t account for total scale. Negative keywords, or really reviews, would need to be expressed as a ratio to the number of positive reviews to normalize for production numbers and reporting frequency.
So, the sample size and diversity isn’t great, the keyword methodology is flawed, and the data normalization methods used here pump numbers to preposterously inflated levels. This is a bad study, and it’s one that some outlets, in their desperation for content, will probably still use. Anyone who takes this seriously should be embarassed, particularly as there are experts and entire analytic firms with a much closer, substantially more careful eye on the automotive industry.
For example, Consumer Reports surveys actual vehicle owners, collecting extensive primary data from owners, normalizing it, and using it to build a great reputation as an expert source.
Cox Automotive collects data on the state of the car market, using a high degree of expertly-sourced primary data and some expertly-sourced secondary data from other studies to build insightful car market reports. S&P Global Mobility is arguably the leading global institution for data on the automotive industry, collecting and interpreting an immense amount of primary and secondary data. Expertise and data sourcing matters.
So, don’t be fooled by bullshit studies. Look for a trusted source, look for good methodology, and understand that a massive number of studies out there are complete and utter horseshit.
(Photo credits: Mazda, Compare The Market AU)
Support our mission of championing car culture by becoming an Official Autopian Member.
-
MIT Study Finds Something No One Mentions About Self-Driving Cars: They’re Lousy For The Environment
-
These Are The 20 Most Durable Cars On The Road According To A New Study
-
Mercedes Loses Big In Consumer Reports’ Latest Reliability Rankings
-
New IIHS Study Confirms What We Suspected About Tesla’s Autopilot And Other Level 2 Driver Assist Systems: People Are Dangerously Confused
-
Tesla Has The Most Crashes In NHTSA’s First Advanced Driver Assist Study But What That Means Isn’t Clear
Got a hot tip? Send it to us here. Or check out the stories on our homepage.
It would have been cheaper and more accurate to have chatgpt do that.
Great takedown of the methodology, though I will say I had a chance to drive the sibling CX-50 turbo a few months ago and I can tell you firsthand it definitely was very disappointing
What was disappointing about it? My brother got one roughly a year ago and I can’t think of anything in the same class that is remotely as fun. That’s not to say the CX-50 is an extremely fun car, but simply it’s the most fun option in the segment. Plus, the interior shoots well above the segment.
Interior was just ok, the knob they force you to use felt super cheap, which is an odd choice for a regularly used control. Most of my disappointment came from driving it around town and on the highway. It was slooow, both from the engine making lots of noise but little power in any drive mode to the overly conservative transmission – it was a little better by shifting into manual mode most of the time, but I came away from a week with it very much turned off. I was sad because it’s definitely the best looking in the segment. I enjoyed a Ford Edge more.
I certainly wouldn’t call it the fastest car around, but the few times I’ve driven it, it felt damn peppy to me, and a hell of a lot faster than my Outback. It’s got like 260hp and 320 lbft at a pretty reasonably 3,800 lbs (had to look it all up). The main reason it felt somewhat slowish was that it was very isolating.
I know, it still doesn’t make sense to me when the stats should suggest it’s capable.
And what about if those words were preceded by “not”
“is not difficult”, “does not disappoint”
these are not negative comments, but they would be picked up as if they were.
Yeah, that’s…
Not hard.
We have a 2016 CX-5 and I love it. It is a lot more sprightly than its competition of comparable years/trims. I feel like everyone calling the CX-5 “boring” or a “snooze-fest” would be rendered comatose by driving a Toyota RAV-4 or a Honda CR-V.
The only thing my sister was disappointed with her CX-5 was the number of minor recalls she was subjected to when she bought it. She had good luck with her NA MX-5 before she bought the CX-5, so she was inclined to buy Mazda again.
I wouldn’t look to Consumer Reports as some bastion of neutrality. Their methodology invites major confirmation bias, and they rank things like infotainment into the overall reliability score.
While I would of course be disappointed if my car’s infotainment were unreliable, I still think it should be ranked in an entirely separate category from mechanical reliability. But CR’s ratings give the mistaken impression that any car with a low reliability rating is less mechanically reliable, which is not necessarily the case according to their methodology.
You have to look at the detailed scores in the annual buyer’s guide to see the breakdown on which areas have the most and least problems. With infotainment however, it is going to be a drivability issue for many cars that use the screens for things like HVAC. If a screwed up screen means you can’t turn on the defroster on a day when it is below freezing, you may not be able to safely drive the vehicle because you can’t keep the windshield clear.
I think I can easily see why this is the case. The CX-5, Tucson, X5, and CX-30 are among the most recommended cars by Aussie motoring sites like Drive, CarAdvice.au (RIP), and CarExpert which sets them up for high expectations that aren’t met in practice. I mean, they’re mid-range CUVs, what do you expect? Luxury car ride and comfort? Hardly.
In contrast, the ‘least disappointing’ are mostly cars that few journalists would recommend to the average buyer for fear of unreliability (Cherokee, A-Class), impracticality (Camaro, CTS-V), or simply not being flashy enough (Insight). Thus when consumers buy those models they probably get a much better product than anticipated and therefore have fewer qualms even if the driving experience isn’t perfect.
It’s the same problem I have with Consumer Reports and their surveys. Somehow it’s the ‘least reliable’ cars which have glowing ‘owner satisfaction’ ratings and vice versa. They’re so obsessed with recommending the most bulletproof models that they’ll tell you to buy a CX-3 over a Chevy Trailblazer even though the Mazda has ‘very poor’ satisfaction to the Chevy’s ‘very good’ satisfaction just because the Mazda is so reliable.
Lets also not ignore this “study” that wants to list the 5 most and least disapointing cars on sale today (in Australia) only looked at 40 TOTAL models. Think about that, 40 total models. I’m pretty sure GM alone has over 40 models in their portfolio, and they don’t list the other 30 not ranked here. The only sources they list for car models is a half dozen other worthless Lists, most of which are generic “what cars are hard to drive” nonsense. What nonsense.
More praise to the Autopian for continuing to uphold journalistic integrity in an industry that seems to be sliding downhill.
Try owning one and you’ll have the same conclusion. We had a loaded 2018, every option – and ditched it after 20k miles. It was more than disappointing: in the real world, away from a journalists keyboard, the CX-5 sucks.
Your article reeks of desperation – perhaps desperate to keep the free Mazda test vehicles coming? Because you are dead wrong.
The CX-5 is a nice-looking vehicle… at first look it looks great. What’s that famous line from the “Gumball Rally” – “It’s a handsome design”?
The white interior looks gorgeous – but then you find the seats are flat. Awful for long trips. And in our 2k miles we did a lot of long trips across the state. And on those long trips you find that the cross-wind aero is very poor – such as lane-changing poor. And here in TX with an 85 MPH speed limit on many roads (and people driving that anyway), you find out the vehicle is not stable at that speed. Bordering on dangerously unstable. Try the 90 that many people cruise the big highways on and you are not long for this world. And you can watch the fuel economy go off the cliff at about 75 anyway. Oh, yea, only 6 gears – antiquated engineering.
Ok, let’s take it into the city. Now you try the safety systems. If somebody 10 cars lengths ahead of you brakes to turn into a driveway, say, at your speed of 30 MPH – your brakes slam on hard and with a loud red warning flashing. You have calculated closing speed and you are plenty safe because of your experience – but the car acts as your nanny. This is the worst safety braking I’ve ever seen… and as an HPDE instructor for 40 years I’ve seen it all. And no I am not driving it at HPDE speeds, nor requiring it to be more than 1% HPDE capable. It’s just outright dangerous.
Braking? Small brakes and with lots of brake dive. Goes right with the typical-for-Mazda body roll. Yes, this is not a handling type vehicle; it is just a tippy box raised very high. And I’ve driven a lot of SUVs and CUVs that are far better than this – which also means safer.
Now for the infortainment disaster… it’s the worst in the industry, slow, stupid, with mis-routing despite all the dial twirling. We couldn’t rely on it at all, and it took too long twirling thru the poorly designed UX of the menus anyway. And where is the touch screen capability?
So who is this vehicle for? Certainly not for Autopian readers. Maybe for Consumer Digest readers… but even then I’d suggest the safety systems so very poor I wouldn’t recommend this vehicle to anybody. So we put a lot of miles on our because we were stuck with it, and then in a short period of time we ditched it.
There seems to be a lot of defense of Mazda in certain press elements… why? As a former Miata owner, too, I’d say their best days are well behind them. Their engines are boring, their excuses for the cheap beam axles on the Mazda3 and CX-3 are inexcusable (as C&D said, it ruined the driving dynamics), handling is tailored for what – secretaries (as the old joke goes)? CX-50? An off-road poseur with no capability at all. Just for urban poseurs. And now everybody gets excited by the fake “return of the rotary” – which if you’ve ever been saddled with an RX-7 you’ll remember was just about the least reliable engine even if it was fun to drive (while eating oil and until one of the 50 vacuum lines pops off and you find it’s one of the rats nest of lines under the intake manifold). Yes, a great car on track, but also one that I have noticed never lasts a season for anybody in stock form without one big issue or another.
So let’s get real about Mazda. That haven’t had it in years, and for those of us with high expectations they can’t afford to engineer a well-executed product line anyway.
Definitely not the point of this article but what modes would/have you make to the De’Longhi ECP3630? I got my SO a DeLonghi Stilosa for her birthday last month and we’ve been pretty happy with it.
Likely a bottomless portafilter to diagnose any instances of channeling and take better advantage of what a burr grinder can do, a Rancilio Silvia steam wand, and that’s really it. Glad you’re happy with the Stilosa, it’s one of the best entry-level machines out there.
We analysed user reviews across a range of models, looking for negative mentions of “difficult”, “hard”, “confusing”, “uncomfortable”, and “disappoint”.
So, they put it into a sort of AI that looks for specific words and then were too lazy to do any follow up. Maybe we should let the robot overlords rule us.
It’s not AI, it’s just Big Data analytics, which used to be the top buzz tech term before AI.
It’s not big data analytics, it’s a word counter. Takes 5 minutes to write.
“It’s difficult to believe how taut the handling is in this car. The ride is firm, but no too hard. It avoids the trap of a confusing infotainment system. The judicious application of tech will satisfy even those Luddites who are uncomfortable with today’s automotive frippery. Buy one, it won’t disappoint.”
I guess according to the study this might count as a bad review.
“Suspension rewards hard cornering and provides a smooth ride, even on uncomfortable pavement. Controls are comfortable and never confusing. Guaranteed not to disappoint!”
The CX-5 does not Disappoint! It’s Hard to find anything to complain about. I have never dealt with an Infotainment system that has been less Confusing. The leather in this vehicle makes it Difficult to find an Uncomfortable spot to sit. All this together means the car does not Disappoint.
Apparently that would be the most negative comment ever.
I’m disappointed I couldn’t buy 10!
Oops, looks like I independently did the same thing…
Good work Thomas!
Now let’s flip the crap back to them.
Most worthless article to read -2023
“After analyzing car articles published in 2023 using negative descriptive words in various reviews, we then ranked them on a scale of 1 (very informative) to 100 (complete horseshit).”
Compare the Market ranked 99.9 with near universal negative words “terrible”, “not helpful”, “misleading”, “confusing” as most commonly read. 0.1 favorable was because one person clicked the wrong site and is not fluent in English.
Ah the power of the worlds information in the hands of stupid people.
Haha, I’m glad this study exists if for no other reason than to get to read this article of Thomas absolutely destroying it.
Compare the market AU is an Australian site. The CX-5 comes with a different choice of engines compared to those offered in the US (including a diesel and an smaller 2.0 liter). Aside from the poor method of pulling the data it could be that the CX-5 is disappointing in the context of the Australian market? Certainly the CX-5 is not as well received in the UK as it is in the US.
Also my wife has a CX-5 it’s a perfectly fine car for its type but I would never say it has great steering and the sluggish gearbox is definitely the weakest point on the car.
That was a thought I had too, but the CX-5 has been a top 10 seller in Australia for a few years now. Surely if the buying public found it disappointing, sales would have tumbled. Hyundai Tucson is another on the list that has sold well.
Been a while since I’ve driven a Mazda, but those I have driven I have liked. I was really impressed by the Mazda 6 of 5 or 6 years ago. I thought it compared very favorably against the Acura TSX I owned at the time with far more interior space.
Wouldn’t this get confused by autowriters ragging on their favorite punching bag, “hard plastics”???
Using “hard” as a keyword would also loop in people who are fans of the car to an uncomfortable degree.
Figures don’t lie, but liars can figure.
It’s even more egregious when you see the X5 is ranked 4th most disappointing to drive lol. Say what you will about SUVs in general…but I’ve driven multiple X5s and know multiple people who’ve owned them (including my dad) and “disappointing to drive” is not a description that’s even in the same universe as the X5, which is built on a dedicated RWD sedan platform.
It drives like it too. There’s a reason why it’s been near the top of its class for seemingly forever. It drives like a big sedan. BMW got it right 20 years ago and the same formula still works today. My dad’s 50e PHEV is a pleasure to drive with the additional electric powaaaaa it offers but I’ve driven a base 40i with just the B58 and was thoroughly impressed with it as well.
I mean sure, if you’re getting into one from a 911 or Miata or something then yeah you’ll probably think it feels less connected. But compared to other luxury barges and family haulers? It’s night and day. I’ve ripped my old man’s on roads that were definitely not public before with a full charge and that damn thing corners like a 3 series and will hit triple digits in sport mode before you can blink.
If money was no object and I wanted a one car solution an X5M would be near the top of my list.
Yep, my mom had a 2013 X5 and currently drives a 2019 model. An SUV like that isn’t my type of car, but “disappointing” to drive they are not.
God dammit, now I want an X5.
Disappointing to insure, maybe; disappointing to maintain after the warranty expires, perhaps; disappointing to look at if you got the X6 instead of the X5, for damn sure; but not disappointing to drive unless your expectations for a fairly large and heavy “Sports Activity Vehicle” are incredibly unrealistic.
All those negative mention terms could easily apply to a Ford GT.
Esp. if no other context was solicited/aggregated from the reviewers other than “did you in fact drive this car?”
“hard”, “difficult”, “disappoint”
One drive and it’s not hard to see why the CX-5 is seen as of the best SUVs on the market.
It’s difficult to compare the CX-5 to other SUVs of a similar price, it is far better appointed and will not disappoint.
I hope this isn’t the sort of keyword harvesting they are doing. We would need to see quotes, not just random words.
All of the above being said, I was never enthused to drive my ex wife’s CX-5. For multiple reasons.
thanks, you just bumped up the score 4 points!
Some of these probably come from the infotainment controls, some from dealership experiences that often get included when people submit product reviews.
Terrible study.
You beat me to it…I should have read all the comments more carefully before replying!
But totally. “I found it difficult that I was expected to pay attention and drive it; I swear I saw in the news that the car does that now for us, right?!”
Mazda also started trying to go ‘premium’ and raise their prices, which also doesn’t help.
Their products aren’t really any more expensive than the other Japanese cars. I mean, the CX-5 starts at pretty much the same price as a CRV or about $1k more than a Rav4, but comes with AWD standard. You go down the list of every model they offer, it’s about as close as you can get. Sometimes Mazda has better standard features, sometimes not, sometimes worse powertrains, sometimes not; but regardless, it’s usually pretty dang close to the same prices as the competition when you add in major features like similar power or AWD.
Hondas and Toyotas are still selling at over MSRP as well because of their stupid allocation system and artificial product scarcity. In my area most new Mazdas are listed at or under MSRP.
The CX-5 base trim starts at under $30K. For an SUV!