Export thread

In today's Security Now (June 24,2025) the show notes say

As we’re going to see at the end of today’s podcast, the flaws that were once present in Cisco’s Internet OS XE network devices were leveraged to admit the attackers into these networks. But Cisco had found and patched those vulnerabilities long before – as in years before – those flaws were used to gain illicit entry into these companies’ networks. ...
I’m at a loss to know how we can ever get this behavior to change. ... In a sprawling organization with thousands of routers and switches spread across a continent, where every device is receiving periodic updates, keeping everything updated – with the risk that an update might cause more trouble than the “potential trouble” it’s presumed to prevent ...

If large companies do not have the manpower or the will to update their assorted network devices, then I agree with Steve that things will not change. And, has he says, the more competent companies have to weigh the risk or updating with patches vs. not updating. In some/many cases an outage can not be tolerated.

The risk of installing updates can be greatly minimized on a device that has two copies of its firmware. Install version 8.4 and if it seems to cause a problem, reboot and fall back to firmware version 8.3. If the fallback is simple, fast and easy, then there is little reason to avoid updates (assuming a company has the manpower to make the updates at all).

On the router front Peplink devices have had two firmwares for at least 12 years, probably longer. Anyone know about pfsense or opnsense? There were a handful of consumer routers that also did this, but the feature was not front and center. I once ran across a Linksys router that could do this. In contrast, some devices NEVER let you fall back. Aruba may fall into that bucket (not sure).

Here is a screen shot of rebooting a Peplink router showing the two available copies of the firmware.

SN show notes:

I'm at a loss to know how we can ever get this behavior to change. ... with the risk that an update might cause more trouble than the "potential trouble" it's presumed to prevent

Click to expand...

Speaking not just about routers but more broadly about tech devices in general, I'm firmly in the "if it ain't broke..." camp.

I don't subscribe to the view that consumers who forego updates are the ones primarily at fault. I feel it's just as much, if not more, the fault of manufacturers who let "mission creep" contaminate the update channel.

In my case, it's not that I don't welcome security updates, it's that manufacturers often bundle in, willy nilly, all sorts of non-security updates such as changing the UI, moving buttons around, or removing functions I was using.

How many times have we seen UI changes just because some twenty-something engineer had too much time on his hands or felt the need to justify his salary? Like, why does the Windows taskbar now need to be centered? (Oh yeah, I forgot ... because that's the way MacOS does it.)

I have a smartphone with a lot of apps, three quarters of which are technically not "up-to-date". But when I dig into the change logs, it's sometimes something as mundane as "We added support for the Khazakastani language!" That's benign enough, but worse it's often to "improve" the app from small banner ads to full-screen takeover ads that can't be dismissed. Why would I want those updates?

I've grown just as leery of PC updates over the years. Like the Win7 updates that nagged you to upgrade to a newer OS. Don't update and you could happily go on using Win7, but update and you were nagged to death.

Or like the printer updates that don't fix anything wrong, but are merely so they can now block the use of third-party ink/toner. Don't update and you could happily continue using less expensive ink and toner.

I used to have more stomach for fixing problems, but I'm old and retired now, and I no longer want to waste any of my remaining time on this earth fixing problems some "update" broke, or relearning how to do things that I already knew how to do before they moved it.


MichaelRSorg:

"The risk of installing updates can be greatly minimized on a device that has two copies of its firmware. Install version 8.4 and if it seems to cause a problem, reboot and fall back to firmware version 8.3. If the fallback is simple, fast and easy, then there is little reason to avoid updates"

Click to expand...

I like that idea, and it looks well implemented in the Peplink. All my machines multiboot more than one Windows partition, and that's essentially how I've long approached Windows updates, but it would be nice if the average user could do the same without resorting to a full-blown multiboot environment.

I don't know if most routers have the hardware to support that strategy, though. Back in an earlier age, I used to only buy routers I could reflash with DD-WRT. I recall the developer had a particular challenge on some devices, though, trying to shoe-horn his enhanced firmware into some devices that had just enough ROM space for the OEM firmware and nothing more. So your proposal would need manufacturers to be willing to spent a few extra cents per device to include more space.

Ultimately you're right, though -- the dread with trying to revert is a major impediment, and a simple fallback mechanism would go a long way to fixing that.

dg1261 said:

bundle in, willy nilly, all sorts of non-security updates

Click to expand...

Unfortunately it's not that simple. (I speak from past professional experience on an embedded device that shipped to thousands of customers affecting many millions of customers and had a version every year or two for more than three decades. I can't say more because of my NDA.)

Developers want to work on cool new things. Doing bugfix is NOT fun, although necessary. So there is one current stream of the software where new things are slowly getting added and old things updated/tweaked. These are the feature requests from product management that everyone wants to work on. At the same time when bugs (security or otherwise) are isolated, they're fixed first in the main stream because that is ultimately closest to release and users. If the expense is important enough, then the fix may get back-ported to older software releases. Unfortunately, this is harder than it might sound for many reasons. Some of them are:
- the tools we use to build the software also get updated over time, and the ability to keep the old versions around and still work only for older builds is not as easy as you'd hope, especially if those tools themselves are made by a company who is doing all of these same things with their software.
- the people who worked on that software version are no longer available, or no longer with the company. The documentation needed to build that specific version may not have been created or properly verified or even properly entered into any tracking system.
- the proper fix relies on new software that is in the main stream, but was not part of the release at the time the bug was added. This means the port back may be a tremendous amount of work, or might rely on licenses that were not purchased for that point time (were added after and are not valid for the old software.)
- the proper fix would need to change the system configuration database in a way that would make the future upgrade incompatible or very risky to apply.

So in the end, product management may simply make the decision to ship whatever is in the main stream as soon as it can pass testing, as a means to save money, time or because it's the only reasonable option (if the old software just can't be patched or even built for some reason).

PHolder said:

So in the end, product management may simply make the decision to ship whatever is in the main stream as soon as it can pass testing, as a means to save money, time or because it's the only reasonable option (if the old software just can't be patched or even built for some reason).

Click to expand...

I find this reflected in ubiquiti's products. Their unifi line has an option for self-updating. While their uisp line doesn't. Guess how much more damage can be done by a problem with an update, they will or won't