On a clear day this summer, security researcher Ang Cui boarded a boat headed to a government biosafety facility off the northeastern tip of Long Island. Cui's security company, Red Balloon, will spend the next year studying how its Internet of Things threat-scanning tool performs on the building control systems of Plum Island Animal Disease Center. If successful, the project could provide a critical tool in the fight against vulnerabilities in embedded industrial systems and critical infrastructure.
"The island is only accessible via a ferry. The dock is protected by armed guards and I presume patrolled by the Coast Guard," Cui says. Those protections, though, mean nothing to potential hackers. So Cui's goal is to "help make the island’s cybersecurity as resilient as its physical security."
The sorry state of IoT security is widely known at this point. Your television, your router, and your electric toothbrush all use microprocessors to crunch data, and more and more of these devices gain internet connectivity all the time. But many aren't built with any plan for how to patch vulnerabilities if—more often when—they're discovered. That lack of investment has already led to real security crises, most recently Krack, which left basically every connected device exposed.
Complicating the issue: The vast majority of embedded devices are black boxes full of unknown hardware components and proprietary software implementations. Many are architected off of popular platforms like Linux, but tweaked and manipulated in countless ways for any given product. That makes tracking down what bugs affect which devices a serious challenge, one that's too often simply ignored.
But at the S4 security conference in Miami, Florida on Thursday, Cui and Red Balloon research scientist Joseph Pantoga are presenting an automated strategy for determining whether software vulnerabilities found in certain embedded devices persist in other IoT gadgets.
"The reactive 'patch each vulnerability that comes along' approach is not a tenable strategy moving forward, especially for sectors like industrial control," says Cui. "You can’t depend on the vendor to fix every single problem, and you can't depend on the world to magically apply each patch. So that’s the real purpose here, we’re showing how easy it is to do this type of analysis in all sorts of embedded devices."
Red Balloon's approach could reveal exponentially more vulnerable devices in an already bug-ridden population; Cui and Pantoga emphasize that it's crucial for defenders to develop this type of vulnerability "miner" now, before attackers do. If they haven't already.
Cui and Pantoga's miner doesn't hunt for previously unknown bugs, or "zero-day vulnerabilities," in embedded devices. Other research, like DARPA's Cyber Grand Challenge, has worked to automate the process of finding novel zero days. Instead, the Red Balloon work focuses on finding "n-days" in IoT devices—vulnerabilities that have been publicly disclosed for any number of days, but haven't necessarily been discovered in specific products, much less patched.
Anyone with the skills to reverse-engineer a product's fundamental code (known as "firmware reversing") can manually determine whether a particular device contains a particular vulnerability. But Cui and Pantoga's research automates that process, and even automatically develops the code that would reliably exploit the vulnerability. They aim to show that an autonomous system can develop and test tailored, working exploits for each new vulnerable device it finds, as evidence that motivated attackers might use these techniques as well.
"We’re not just going out and identifying a version of the operating system, the analysis is identifying specific structures of the software and analyzing that structure to create a workable exploit as quickly as possible," Cui says. "If you’re an attacker you can build this capability for much cheaper than what you need to spend to find zero days, so if you're looking to exploit as many, say, industrial control installations as possible you’re going to do something like this."
The threat of automated IoT vulnerability finders is a genuine concern. "Absolutely it is coming," says Anders Fogh, a malware analyst for the German security firm GData. "We are waiting for the vendors to realize that security is relevant. They need a dose of bitter medicine." Other researchers are beginning to work on large scale IoT firmware analysis and automatic n-day mining projects as well, acknowledging a future in which attackers can fully exploit IoT vulnerability.
Some qualify, though, that it will still take time for attackers to focus time and resources on developing these techniques, given the wealth of vulnerable embedded devices that are already known and exploitable. Besides, there are often easier ways to crack an IoT device than complicated malware. "Right now we haven't seen much of it because there are so many IoT systems already out there with even more trivially exploitable problems like default passwords," says Brendan Dolan-Gavitt, a software analysis and embedded device researcher at New York University. "So until those become more scarce, I wouldn't expect attackers to expend effort."
Using a firmware evaluation and unpacking tool Cui developed during previous research, he and Pantoga honed their vulnerability-identifying process and exploit creator. They tested their n-day miner on a group of vulnerabilities first disclosed in 2016 in the popular VxWorks embedded device and industrial control operating system—used in devices like temperature or positive air-pressure controllers, industrial networking devices, and communication modules. The bugs exist in multiple versions of the operating system, and the initial 2016 disclosure looked at vulnerable VxWorks software running on a type of processor architecture called MIPS. For their n-day mining tests, Cui and Pantoga also targeted ARM and PowerPC processors to look for the vulnerabilities in an even larger swath of embedded devices.
The results were concerning. Though Cui and Pantoga readily admit that the process still isn't completely automated, the n-day miner did surface multiple industrial control devices that are exposed by the VxWorks vulnerabilities. Cui and Pantoga are working with the manufacturers who make the newly found vulnerable devices to make sure they get patched and say that they are convinced it would be too dangerous to reveal the models until fixes become available. VxWorks maker Wind River said in a statement to WIRED that, "Wind River worked closely with the researcher at the time [in 2016] and released updates to all affected versions of VxWorks before the vulnerabilities were published." The n-day miner did also find the vulnerabilities in the Cisco SPA 303 IP phone, a standard office phone model, after Cisco had already released a patch.
"Usually I'm the one who wants to disclose things no matter what," Cui says. "But here's today’s reality. People are disclosing small numbers of vulnerabilities inside embedded devices, the vendor fixes them, and that’s barely a sustainable proposition. If we have a capability for an automated system to find vulnerabilities inside firmware suddenly we’re up to our eyeballs in vulnerabilities and there’s no way yet to address all of them at one time."
For the embedded device research community, the ultimate goal is a practical and feasible way for manufacturers to start building security into their IoT products. Even without finding new vulnerabilities, automated bug discovery tools could easily overwhelm the flimsy patching structure that's currently in place. And that's worrying for Red Balloon as it works to secure the door controllers, biocontainment and decontamination units, pressurization modules, and other embedded systems at Plum Island Lab. Being on an island simply isn't protection enough.
