Here’s a quick thought experiment: We have a car and we want to update its firmware, which is a fancy term for software embedded in hardware. Updating the firmware might improve the car’s performance, reduce harmful emissions, extend the car’s lifespan and elevate its resale value. But we’re also worried that updating the firmware will allow hackers to inject malware, disable safety systems and make it easier for thieves to steal the car.
Our choices include updating the car’s firmware and living with the risks, selling the car to an unsuspecting buyer or locking it in the garage and never driving it again. Maybe there’s another choice: What if we had a detailed view into the car’s firmware versions and we could compare the potential vulnerabilities of each one? Then we would know which firmware could be updated safely and which firmware should be replaced.
Now let’s shift our thought experiment from cars to the Internet of Things (IoT), which includes mobile phones, digital cameras, routers, sensors, scanners, printers and television remotes. Billions of IoT devices, ranging from smart refrigerators to medical implants, depend on firmware to run effectively and reliably. There’s firmware in nuclear power reactors, hydroelectric generators, municipal water plants, aviation safety systems and telecommunications networks. Firmware is all around us and it plays an increasingly critical role in our lives.
“Safety is the number one concern for any machinery that uses embedded software,” says Charlie Hart of Hitachi America Ltd. “Almost every machine in the world has firmware controlling it – cars, planes, elevators, building systems, appliances, you name it. If any of that firmware is hacked, there is no way to ensure safety.”
Firmware vulnerabilities are neither imaginary nor hypothetical – they are very real. Hackers have attempted to exploit firmware vulnerabilities at a water treatment facility in Florida and at wastewater treatment plants in Israel. Cisco recently disclosed “multiple vulnerabilities” in the web UI feature in its IOS-XE software used in routers and switches. In a collaborative effort aimed at reducing hardware, software and firmware security risks, the National Security Agency (NSA), the Federal Bureau of Investigation (FBI), the Cybersecurity and Infrastructure Security Agency (CISA) and the U.S. Treasury Department have released guidance for securing operational technology (OT) and industrial control systems (ICS) environments.
Yet as billions of new devices join billions of existing devices in the Extended Internet of Things (XIoT), the risks multiply exponentially. “This is not a trivial problem,” says Michael Scott, co-founder, CTO and chief scientist at NetRise, a company that specializes in device firmware security.
Many organizations don’t have a precise inventory of devices running on their network and don't know exactly which of those devices have embedded code.
Moreover, it’s not uncommon for the firmware in devices to be customized or altered by resellers. Even if the reasons for altering the firmware are benign or advantageous, the changes can make it harder to track and identify issues that can affect the performance or security of a device.
“You may think you’re getting the device from a specific vendor, but the firmware in the device may have been changed or altered by someone else, either for good reasons or bad reasons,” Scott explains. “As a result, you’re missing the big picture. If there’s an infection somewhere in your network, it’s going to be more difficult to trace it back and figure out exactly where it came from.”
Scott and his colleagues at NetRise have developed a platform for bringing automated, scalable firmware analysis to companies of all sizes, essentially democratizing a process that previously had been available to only a handful of large organizations with the resources and willpower required to rip open hardware devices and carefully reverse engineer their firmware.
“The first step is getting a handle on what’s inside these black boxes,” Scott says. “Then if it turns out there’s firmware in there that hasn’t been updated in 10, 15 or 20 years, you can go back to the supplier and tell them to fix it or you’ll find another supplier.”
When Scott talks about firmware that hasn’t been updated for decades, he isn’t exaggerating. As consumers, we’ve grown accustomed to regular automatic software updates on our phones, tablets and laptops. But in the world of operational technology – which includes machines and devices used routinely in factories, warehouses, retail stores, hospitals, power plants, water treatment facilities and transportation hubs – updating firmware can be a cumbersome task.
In some industrial settings, firmware updates involve crawling or maneuvering under heavy industrial equipment with a USB key. In situations where critical services are delivered, installing updates may require shutting down complex systems – and hoping the systems can be restarted without unexpected problems surfacing after the updates are completed.
Fortunately, there’s light at the end of the tunnel. The emergence of projects such as sigstore and the SLSA framework, a collaboration between Google and OpenSSF, are sure signs of a trend towards greater accountability and transparency in the global software supply chain.
From Scott’s perspective, coming to grips with the challenges posed by firmware security is an important step forward in the continuing evolution of digital technology. “Seeing into the black box means everyone will have to tighten up their development processes,” he says. “And that will set the stage for a brighter future.”