Is Tesla Miscounting Crashes To Make Autopilot Seem Safer? New Study Suggests It Might Be

We’re in an interesting stage when it comes to automated driving; we’ve left the honeymoon period, and lost a bit of the hopeful optimism that was spurred on by the rapid successes and developments of the 2010s, which saw development of advanced Level 2 driver assist systems, like Tesla’s Autopilot, which has been predicted to be fully self-driving “next year” since 2014. Of course, that hasn’t happened, and Tesla’s Autopilot, like all Level 2 systems, requires constant driver supervision.

The industry seems to have realized the substantial amount of work still required to get past Level 2 and into more unsupervised levels of driving autonomy, and getting there requires good data about how current systems are performing. To get this data, the National Highway Traffic Safety Administration (NHTSA) has a General Standing Order that requires carmakers that sell cars with Advanced Driver Assist Systems (ADAS) to self-report crashes while using these systems.

Tesla’s information has been very important because of the sheer number of ADAS-assisted miles their cars have driven: over three billion miles of on-road driving since 2015. A new safety report from Tesla just released , shows improvement in Autopilot crashes, but a peer-reviewed study from the journal Traffic Safety Research suggests that the revisions Tesla has made regarding how they report their data do not make statistical sense, and researchers speculate that Tesla may be miscounting crashes in order to make Autopilot seem safer than it actually is.

Here’s what Tesla notes in their Q2 2024 Safety Report:

In the 2nd quarter, we recorded one crash for every 6.88 million miles driven in which drivers were using Autopilot technology. For drivers who were not using Autopilot technology, we recorded one crash for every 1.45 million miles driven. By comparison, the most recent data available from NHTSA and FHWA (from 2022) shows that in the United States there was an automobile crash approximately every 670,000 miles.

…and, to compare, here was the statement from the Q2 2023 report:

In the 2nd quarter, we recorded one crash for every 6.18 million miles driven in which drivers were using Autopilot technology. For drivers who were not using Autopilot technology, we recorded one crash for every 1.46 million miles driven. By comparison, the most recent data available from NHTSA and FHWA (from 2022) shows that in the United States there was an automobile crash approximately every 670,000 miles.

So, over the course of a year, Autopilot-enabled crashes went from one every 6.18 million miles to one every 6.88 million miles, a notable improvement, while crashes for cars not using Autopilot held pretty much the same, one per 1.45 million miles in 2024, and one per 1.46 million miles in 2023.

Here’s the abstract from the study about these results from the  Traffic Safety Research  journal:

Abstract: Between June 2018 and December 2023, Tesla released quarterly safety reports citing average miles between crashes for Tesla vehicles. Prior to March 2021, crash rates were categorized as (1) with their SAE Level 2 automated driving system Autopilot engaged, (2) without Autopilot but with active safety features such as automatic emergency braking, and (3) without Autopilot and without active safety features. In January 2023, Tesla revised past reports to reflect their new categories of with and without Autopilot engaged, in addition to making small adjustments based on recently discovered double counting of reports and excluding previously recorded crashes that did not meet their thresholds of airbag or active safety restraint activation. The revisions are heavily biased towards no-active-safety-features—a surprising result given prior research showing that drivers predominantly keep most active safety features enabled. As Tesla’s safety reports represent the only national source of Level 2 advanced driver assistance system crash rates, clarification of their methods is essential for researchers and regulators. This note describes the changes and considers possible explanations for the discrepancies.

Noah Goodall, the author of the paper, is a Senior Research Scientist with the Virginia Transportation Research Council, and has published a large number of other papers about vehicle automation, traffic operations, and vehicular safety.

The author of the study also tweeted about the study and the suspected results:

Back when NHSTA SGO required ADAS crash reporting, Tesla reclassified and revised their self-reported Autopilot crash rates. The revisions don't make statistical sense, and I suspect Tesla is miscounting crashes to make Autopilot look safer.

New peer-reviewed pub out, ???? 1/ https://t.co/6nNHYeKcBi pic.twitter.com/rJfopXF3oX

— Noah Goodall (@NoahGoodall) August 7, 2024

Essentially, what seems to have happened, according to Goodall, is that Tesla changed the way they classified the categories of the recorded crashes. At first Tesla grouped crashes into three categories:

• ‘Autopilot engaged’

•‘without Autopilot but with our active safety features

• ‘without Autopilot and without our active safety features’

So, the differences here are crashes with the full Autopilot suite active, then no autopilot but safety features like emergency automatic braking that operate in the background, and then driving with everything turned off, effectively like the old deathtraps I tend to drive.

Later, again, according to Goodall, Tesla changed the categories to be:

• ‘Autopilot technology (Autosteer and active safety features)’

•‘not using Autopilot technology (no Autosteer and active safety features)’

…and then finally simplified them again, to be pretty much a simple binary:

• using Autopilot technology’

•‘not using Autopilot technology’.

Here’s a more detailed breakdown of what the non-autopilot active safety features are:

"Active safety features" refer to things like automatic emergency braking – standalone safety systems that can run even when AP is off. 3/ pic.twitter.com/XTk6FF6RAX

— Noah Goodall (@NoahGoodall) August 7, 2024

Essentially what happened is that Tesla combined the non-Autopilot categories together; one would think the resultant crash numbers of these combined categories would be, essentially, an average of the two categories. But, as Noah Goodall points out, they’re not, they seem to be closer to the crash results found from having none of the safety features engaged. This would only make sense if people routinely de-activated safety features like automatic braking, which are enabled by default when the car is turned on and would require about seven button presses to disable at the beginning of each drive. According to Goodall, drivers that willingly turn off all the safety features would need to be driving over six times more than other drivers to get the new crash rates to makes sense.

The result here seems to be that these new numbers appear to inflate the number of non-Autopilot crashes in relationship to crashes with Autopilot enabled, making the Autopilot record appear better.

From the conclusions of the paper:

The analysis presented as part of this note revealed counter intuitive crash rates, particularly in the ‘notusing Autopilot technology’ category where the data suggested that crashes without active safety features were heavily weighted in the post revision reports. This finding is surprising, as prior research suggests that drivers usually keep active safety features enabled.

Finally, the possibility of unreported data or exclusions in the revised crash rates was considered. Some evidence suggests that certain data might not havebeen included in the revised rates, potentially affecting the overall results. In order to produce the revised crash rates when combining with and without active safety features crashes, vehicles would need to have traveled at least 6.3 times further with no active safety features active. This is unlikely, as default active safety features are rarely deactivated by individual drivers. Furthermore, the supposed smaller sample sizes of ‘with active safety features’ do not exhibit smaller variances as would be expected.

In conclusion, Tesla’s revisions to its safety reports raise questions and warrant further investigation. The discrepancies observed in the crash rate data, especially in the ‘not using Autopilot technology’ category, highlight the need for greater transparency and clarity in reporting methodologies. Given the limited data on automated driving system crash rates, a clearer understanding of the data sources and classifications used by manufacturers like Tesla is essential for researchers, regulators, and consumers.

Now, I’m not qualified to give an opinion either way, I’m just reporting on what this particular report says. I think I can say that I agree that it is extremely unlikely for drivers – of any kind of car, not just Teslas – to go out of their way to disable default active safety systems, unless they’re getting ready for track driving or something very specific like that. It’s just not something people normally do, because why would they? Why go through all that confusing effort just to make yourself less safe?

So, was this deliberate on Tesla’s part, to improve the stats for Autopilot? I can’t say and Tesla doesn’t like answering reporter questions. I can say that this report brings up some important questions that I agree merit further investigation. Plus, the paper is short– you should read it for yourself!

These Tesla FSD Hands-Free Mod Devices Are Such A Bad Idea And Promoting Them Is Dumb

It’s Reasonable To Conclude Musk Knew About Autopilot ‘Failing To Detect Cross Traffic’ Says Florida Judge

Tesla’s Faked 2016 Autopilot Video Was The Start Of Where Things Went Wrong

New IIHS Study Confirms What We Suspected About Tesla’s Autopilot And Other Level 2 Driver Assist Systems: People Are Dangerously Confused