Once upon a time, firmware updates on a car were a fraught process, involving expensive proprietary programming tools or literally ripping out and replacing chips. These days, by virtue of wireless connectivity, it’s possible to remotely update cars over the air. It’s convenient for customers and automakers, right up until it goes wrong, as Rivian has found out.
The latest Rivian update was pushed out on Monday for R1T and R1S vehicles, labelled “2023.42.0” It was intended to improve proximity locking and unlocking and fix a variety of other bugs. Customers unlucky enough to be first in line for the update found that the update would stall at 90% completion. From there, the infotainment screen would go completely black and become unusable, with the instrument display affected as well according to some reports.
Affected users can rest assured that their vehicle is not bricked; indeed, it’s still possible to drive the vehicle in this condition. However, there’s no way to control the HVAC system with the infotainment screen dead, and in cases where the main cluster is similarly dead, there’s no speedometer available either. Owners stuck in a bind in this situation would be best advised to buy a thick jacket and use a mobile phone GPS speedometer app. Rivian has advised customers that vehicle resets or sleep cycles will not solve the problem.
Once 2023.42 fails to install (which it will at 90%), you are left with 2 black screens. Vehicle is drivable but with no infotainment. Even though the screens are black, the vehicle is not bricked. pic.twitter.com/lMrCWEIOTf
— RivianTrackr (@RivianSoftware) November 14, 2023
One notable point is that the reversing camera still works under these condtions, popping up on the infotainment screen and operating otherwise as normal. The Rivian mobile app may be of use, too, reportedly retaining functionality for some users with regards to ancillary features.
In a statement posted to Reddit, Rivian put the issue down to human error—” a fat finger where the wrong build with the wrong security certificates was sent out.” These certificates are used to digitally “sign” a firmware update so that the vehicle can verify it is a trusted update from the automaker. It’s a routine security measure to stop random hackers flashing a vehicle with malicious software. However, in this case, it suggests that Rivian’s vehicles weren’t set up to gracefully reject a build with improper certificates, which resulted in the screen failures.
The update campaign was cancelled when the fault was realized, with Rivian planning to restart it in due course with the appropriate build with the proper certificates. Rivian has reported that it has developed a solution and will be notifying impacted owners of the next steps by email. As yet, it’s unclear whether affected vehicles will be able to receive an over-the-air update, or whether they will require an update by physical connection. It’s not uncommon for a bad over-the-air update to cripple an affected system to the point it loses connectivity, at which point it becomes impossible to update remotely.
The Autopian reached out to Rivian for comment regarding the fix, with the automaker confirming that remote update will be used to fix the affected vehicles. “Our customers’ experience matters tremendously to us. We’re making an over-the-air update available for an R1T and R1S infotainment issue that will restore full functionality to the 3% of Rivian vehicles that were impacted,” said Wassym Bensaid, Senior Vice President of Software Development at Rivian. “From here, we’re deeply evaluating our process and quality checks to prevent this from happening again.” The automaker also noted that the issue was identified within 90 minutes and further rollout was halted, with only 3% of customers affected. It credits its vehicle’s compartmentalized design for safety critical systems for allowing its vehicles to still function despite the faulty update.
Overall, it’s a rough lesson for a nascent company. Send out a bad update over the airwaves, and it’s possible to annoy a lot of people really quickly. The same thing wouldn’t happen with a physical roll-out—any dealer tech would stop installing an update that killed a screen immediately and report back to head office. Operating in today’s fast-paced wireless world requires more care to avoid causing headaches at scale.
I feel lucky to have a vehicle with CarPlay yet unable to receive over the air updates so everything that’s working can’t get blown up.
Minor peeve but my god I’m tired of resetting my radio presets every time Ford does an update on my Mach E. I finally took a picture of the screen so I’m not trying to remember where my favorite industrial/grindcore channel went.
One major reason I refuse to buy newer cars.
someone trying to get an update out on a Friday to be fresh over the weekend by Monday it would seem.
This just doubles down my commitment to driving stripped models. Bottom-tier cars with just the basics: go, stop, turn and reverse, and maybe air conditioning, have a lot less to go wrong. And with so little added equipment shoved into the vehicle, you can get by with the most basic of controls: knobs. Buttons. Ooh, a slider for those of you who need speeds in between speeds.
Just saying.
I don’t know that “Oopsie, butterfingers! My bad!” is necessarily the image that a forward-thinking builder of luxury vehicles wants to project when it remotely disables its customers’ expensive vehicles due to a pretty basic mistake.
Cars have become cell phones. When they have an update it’s best to wait a few days and see what happens to the other lab rats.
Aaaaand another reason not to buy an electric vehicle
ICE cars get over-the-air updates as well, but nice try.
Only new ones. I wouldn’t buy any of those, either.
Shit, Chevy bricked the infotainment unit in my all-gas Sonic more than once with an update.
This is most definitely not an EV specific issue. Programmers derp all the time.
Personally I don’t like OTA updates because if something like this were to happen, I’d want it to happen, say, at the dealership while I am specifically there to get an update, so they can fix their own FUBAR. Not when I’m, say, trying to get home after a rough day at work.
The happy medium is downloadable updates that can be loaded from an SD card which a couple of my cars can do.
I’m guessing Rivian is the same as Tesla in that updates only download over wifi. In general, that means you’re at home. Still not as ideal as having the dealer fix it or give you a loaner while you’re there, but it shouldn’t happen in the middle of a trip.
Chevy did it via 4G, so it could trigger at any time. Once it tried updating when I turned it on while parked 4 levels below ground in a parking garage. Probably the safest place to be when that kind of update is attempted.
Stuff like this is why I prefer not over the air updates.
Agreed. I really hope that the idea of screens control everything is dead by the time I have to buy a new car.
At the very damn least, put climate controls back to their own separate controller/buttons. This is the 2nd day in a row I haven’t been able to adjust the climate controls in my Santa Fe because the stereo would only partially boot, and though the climate controls have physically separate buttons, they’re still partially controlled through the head unit. I’ve gotta disconnect my battery for a bit to see if that resets the HU, and if not, it may be time to do some… creative engineering.
As a Canadian, I could not deal with non-functional HVAC. It would be downright dangerous to drive with a frosted up windscreen.
And the HU reset fixed it… until I let it sit overnight. So the new (cheap) fix is to reset the HU when I turn on the car every morning (takes about 15 seconds and requires me to keep a ball point pen in the car). My USB stick will never again be able to remember where it left off, but that’s a small price to pay for not having to completely reengineer what was installed in the car. Maybe I’ll replace the USB stick with one of my backup iPods.
Just get the damn thing right and then stop fiddling with it.
This is one of the _many_ reasons it’s a stupid idea to get rid of physical controls.
Sadly a lot of physical controls just route through the computer so you would be no better off, except you could push a button and swear
*nods sadly in Santa Fe*
Doesn’t the FMVSS require all cars to have speedometers? A NHTSA with guts would immediately issue a recall, and fine Rivian for thousands of violations of vehicle safety standards.
After that, then you would see Rivian and others actually implement software quality control.
Happened to my friend’s R1S yesterday….
Michael: “I always do that. I always mess up some mundane detail.”
Peter: “THIS IS NOT A MUNDANE DETAIL, MICHAEL!”
Lumbergh: “Yeaaaaaahhhhhh….”
Rivian’s aren’t cheap. Why don’t they have a way to revert to the previous version built in? Especially if they are doing over-the-air updates that could fail just due to a communication error. And as already mentioned here, another reason for physical HVAC controls.
Why on earth would they not test this on an internal unit before sending it to customers?
Also, what a world when corporations are posting official statements to Reddit.
Funny we just talked about OTAs yesterday. I didn’t even consider this scenario
Yeah, this honestly wouldn’t be *awful* if the speedometer and climate controls weren’t affected.
We still haven’t seen the true nightmare scenario where the car itself is bricked.
Scenarios like this are exactly why I don’t want an OTA car. This is one of the lower risk examples, but if mistakes made by some buffonn months after I’ve departed the dealer keep me from driving, I may end up getting it towed back to the dealer and leaving it there.
With Twitter on the wane, everyone is wondering how to communicate with the masses easily. For the price of entry, Rivan should have a user messaging service they could reach out with.
They could send a message to show on the infotainment screens of affected veh…oh, wait.
What’s wrong with text messages?
Apparently Rivian is using the same QA processes as Microsoft before shipping updates #fullsend
I think that the statement on Reddit was the most surprising part of the story for me. Doesn’t every company in the world have their own website? Why not post it there? Are they trying to be cool and hip like Tesla where the CEO
tweetsXes every thought that enters his mind?Maybe someone also fat-fingered a website update and crashed it.
First thing I thought out. At least test this on a few internal center stacks before sending out. Don’t sacrifice reliability for speed.
The world has gone stupid. If you’ll excuse me, I am going to tie an onion on my belt and go yell at a cloud.
Don’t forget to string a potato and wear it around your neck!
#youllbefine #donttakechances
Why? Why should they?
Right now software on all EVs seem to allow the programmer to directly issue an update, and if something goes wrong, the company issues a half-hearted Reddit “My Bad”.
Until either would-be customers (as in, haven’t paid yet) or government fines force them to, why should they take on the cost of rigorous software quality control?
Hasn’t this happened to Tesla a bunch of times already?
You know what never went black after software updates or, for that matter, never needed software updates in the first place? Actual physical buttons and knobs. What was ever wrong with those?
You know what else never went black after software updates or, for that matter, never needed software updates in the first place?
Wheelbarrow of shrimp.
And that’s why the Autopian is the leader in automotive journalism.
There’s still not a membership level which guarantees an actual wheelbarrow of shrimp. Not even a toolbox of cocktail sauce! I have to personally hold off on such accolades until such membership perks are offered.
I get where you’re coming from, but in this decade it’s absolutely absurd to expect to buy a $90k electric vehicle and be presented with buttons and sliders straight out of 1992. We can whine and express our displeasure all we want, but the features and creature comforts we expect out of our cars today just won’t play with old school stuff. Even if it had redundant controls, it wouldn’t surprise me if they were routed through the infotainment and would become inoperable in this scenario anyway.
Having to choose between risking taking your eyes off the road to look at a touchscreen or finding a place to pull over just to change the temperature in your $90,000 vehicle doesn’t seem like much of an advancement to me
That argument is tired and doesn’t really hold water anymore. My car has buttons, but to operate the majority of them you need to look. To change the drive mode, turn on/off parking aids, auto stop, seat heating/ventilation etc you have to look at a row of featureless, identical buttons and figure out which picture means what. So many things are automated now, who cares if they’re buried in a touchscreen? Why are you messing with the headlights when they’re automatic? The amount of cars I see driving sans headlights at night tells me many drivers are turning knobs they shouldn’t be. Leave it in auto. Media controls, and usually lots of other stuff, are available on the steering wheel.
I’m not firmly in one camp or another, I’m just over this “old man yells at cloud” stance on screens. Nobody is getting their 1991 Accord dashboard back and it’s for reasons both good and bad. In the end as long as the UI is well designed it doesn’t make a difference whether the button is on a screen or clumped together with a bunch of similar buttons at the bottom of the dashboard.
Oh it very much matters. When you have actual physical buttons, your fingers can tell where one ends and the next begins. After a couple weeks of driving any given car, you should not need to look at your climate controls if you have physical buttons; muscle memory will take over and you can do it while keeping your eyes on the road. But if those controls are on a touchscreen, because there is no physical difference along the entire screen’s surface and the car can’t figure out if you’re just groping for something or you really do want to fire your photon torpedoes at the Klingons, you do actually need to look at what you’re doing.
2019 GTI owner here:
Drive mode – button next to shifter
Parking aids – steering wheel button
Auto stop – button next to shifter
Seat heater – center stack button
Still a touchscreen in the center stack for radio related items. This is the major reason I bought a Mk7.5 over a Mk8 earlier this year.
Baking every function into a touchscreen is neither necessary nor effective. Screens die, lack physical gaps between buttons, get greasy/washed out in the sun, and are generally a pain in the ass to muddle through while driving. I can find all the above functions while still trying to figure out what that vanity plate ahead of me in traffic is supposed to mean.
“in this decade it’s absolutely absurd to expect to buy a $90k electric vehicle and be presented with buttons and sliders straight out of 1992.”
What’s absurd is I’m still rolling on rubber like its 1892, where’s my flying car? And my Mr Fusion to power it?
Huge infotainment screens that control very basic functionality aren’t what I’d consider “creature comforts”. They are harder to use, more distracting, and fail way more easily and in much more dramatic fashion than physical buttons and knobs do. They solve a problem that nobody had. “Oh gee, I sure wish it was more complicated to adjust the temperature in my car”. Buttons and knobs work. Big screens in lieu of buttons are purely a cost-cutting measure.
The same can be said of almost any technological advancement. Dynamic chassis control? VVT? Forced induction? Fuel injection? Electronically controlled autos? It all sounds like my dad saying “You can’t work on these dang new computer cars! Back in the 70’s I could rebuild a carb with my eyes closed!”. Everyone just wants to be a stick in the mud and pine for the olden days instead of learning or adapting. The “SCREEN BAD” rhetoric is like watching a bad infomercial. I picture a bunch of people randomly jabbing at a screen like they’ve never seen one before while pulling their hair out.
It’s not that bad, and if you’re tech-averse, nobody is telling you that you can’t keep driving your 90’s Whatever SE (at least in the US).
I mean, look at this thing. There is no argument that a screen is better suited to controlling the functions of any modern car than 75 knobs and buttons all around the dash and console.
https://en.wheelsage.org/pontiac/bonneville/1987-91/gallery/y08kb3
The difference between forced induction, VVT, etc. and screens to control basic functions is that they solved problems that existed. They improve efficiency, power, pretty much everything besides reliability. Screens have their place in the automotive world, just not to control climate, headlights, windshield wipers, etc. Buttons do everything that screens do in this context, except they do them better. What benefit does a screen have to buttons in the context of climate control?
Also, the Pontiac is not and would not be the norm. That setup is way too complex, hard to use, confusing, and just isn’t as effective as a more reasonable number of buttons would be. Sort of like a big screen to control your climate. Basic functions should be controllable via easy to use, tactile buttons. The rest of it can be hidden in menus.
“…nobody is telling you that you can’t keep driving your 90’s Whatever…”
Well, I certainly can’t. I’ve never owned a car that new.
See, the thing is I’m not tech averse. I like having a touchscreen for things like tweaking the way my heated seats work and telling me who is calling when my phone rings (which, I will note, I answer using a physical button on my steering wheel so I don’t have to look at the screen any longer than it takes to read a name).
I do not like having to use a touchscreen for things I need to interact with while I’m driving. Touchscreens are an objectively bad interface for that and it needs to stop.
As usual, taking either design philosophy to its most extreme form is bad. 75 buttons is a bad user interface. That same functionality entirely on a touchscreen is also a bad user interface. There’s a right way and a wrong way to integrate a screen in a car and far too many manufacturers are doing the latter.