Triple Meltdown: How So Many Researchers Found a 20-Year-Old Chip Flaw at the Same Time

The uncanny coincidences among the Meltdown and Spectre discoveries raise questions about "bug collisions"—and the safety of the NSA's hidden vulnerability collection.
The vulnerabilities behind the devastating Meltdown and Spectre attacks have existed for decades. Four groups of...
The vulnerabilities behind the devastating Meltdown and Spectre attacks have existed for decades. Four groups of researchers independently found them within mere months of each other.HOTLITTLEPOTATO

On a cold Sunday early last month in the small Austrian city of Graz, three young researchers sat down in front of the computers in their homes and tried to break their most fundamental security protections.

Two days earlier, in their lab at Graz's University of Technology, Moritz Lipp, Daniel Gruss, and Michael Schwarz had determined to tease out an idea that had nagged at them for weeks, a loose thread in the safeguards underpinning how processors defend the most sensitive memory of billions of computers. After a Saturday night drinking with friends, they got to work the next day, each independently writing code to test a theoretical attack on the suspected vulnerability, sharing their progress via instant message.

That evening, Gruss informed the other two researchers that he'd succeeded. His code, designed to steal information from the deepest, most protected part of a computer's operating system, known as the kernel, no longer spat out random characters but what appeared to be real data siphoned from the sensitive guts of his machine: snippets from his web browsing history, text from private email conversations. More than a sense of achievement, he felt shock and dismay.

"It was really, really scary," Gruss says. "You don’t expect your private conversations to come out of a program with no permissions at all to access that data."

From their computers across the city, Lipp and Schwarz soon tested proof-of-concept code they'd written themselves, and could see the same results: Lipp remembers seeing URLs and file names materializing out of the digital noise. "Suddenly I could see strings that shouldn't belong there," he says. "I thought, 'Oh God, this is really working.'"

Graz University of Technology researchers (from left) Daniel Grüss, Moritz Lipp, and Michael Schwarz represent just one team of four that independently discovered the same two-decade-old critical security flaw in processors within months of one another.Graz University of Technology

That night, none of the three Graz researchers slept more than a few hours. The next day, they sent a message to Intel warning them of a potentially industry-shaking flaw in their chips. They'd found a gap in one of the most basic security defenses computers offer: that they isolate untrusted programs from accessing other processes on the computer or the deepest layers of the computer's operating system where its most sensitive secrets are kept. With their attack, any hacker who could run code on a target computer could break the isolation around that low-privilege program to access secrets buried in the computer's kernel like private files, passwords, or cryptographic keys.

On cloud computing services like Amazon Web Services, where multiple virtual machines coexist in the same physical server, one malicious virtual machine could peer deeply into the secrets of its neighbors. The Graz team's discovery, an attack that would come to be known as Meltdown, proved a critical crack in one of computing's most basic safeguards. And perhaps most troubling of all, the feature they had exploited was introduced into Intel chips in the mid-1990s. The attack had somehow remained possible, without any apparent public discovery, for decades.

Yet when Intel responded to the trio's warning—after a long week of silence—the company gave them a surprising response. Though Intel was indeed working on a fix, the Graz team wasn't the first to tell the chip giant about the vulnerability. In fact, two other research teams had beaten them to it. Counting another, related technique that would come to be known as Spectre, Intel told the researchers they were actually the fourth to report the new class of attack, all within a period of just months.

"As far as I can tell it’s a crazy coincidence," says Paul Kocher, a well-known security researcher and one of the two people who independently reported the distinct but related Spectre attack to chipmakers. "The two threads have no commonality," he adds. "There’s no reason someone couldn’t have found this years ago instead of today."

Quadruple Collision

In fact, the bizarre confluence of so many disparate researchers making the same discovery of two-decade-old vulnerabilities raises the question of who else might have found the attacks before them—and who might have secretly used them for spying, potentially for years, before this week's revelations and the flood of software fixes from practically every major tech firm that have rushed to contain the threat.

The synchronicity of those processor attack findings, argues security researcher and Harvard Belfer Center fellow Bruce Schneier, represents not just an isolated mystery but a policy lesson: When intelligence agencies like the NSA discover hackable vulnerabilities and exploit them in secret, they can't assume those bugs won't be rediscovered by other hackers in what the security industry calls a "bug collision."

The Meltdown and Spectre incident isn't, after all, the first time major bugs have been found concurrently. Something—and even Schneier admits it's not clear what—leads the world's best security researchers to make near-simultaneous discoveries, just as Leibniz and Newton simultaneously invented calculus in the late 17th century, and five different engineers independently invented the television within years of one another in the 1920s.

"It's weird, right? It’s like there’s something in the water," says Schneier, who last summer co-authored a paper on vulnerability discovery. "Something happens in the community and it leads people to think, let’s look over here. And then they do. And it definitely occurs way more often than chance."

So when the NSA finds a so-called zero-day vulnerability—a previously unknown hackable flaw in software or hardware—Schneier argues that tendency for rediscovery needs to factor into whether the agency stealthily exploits the bug for espionage, or instead reports it to whatever party can fix it. Schneier argues bug collisions like Spectre and Meltdown mean they should err on the side of disclosure: According to rough estimates in the Harvard study he co-authored , as many as one third of all zero-days used in a given year may have first been discovered by the NSA.

"If I discover something lying dormant for 10 years, something made me discover it, and something more than randomly will make someone else discover it too," Schneier says. "If the NSA discovered it, it’s likely some other intelligence agency likely discovered it, too—or at least more likely than random chance."

Speculative Speculation

While some elements of Meltdown and Spectre's four-way bug collision---a bug pile-up may be a better description---remain inexplicable, some of the researchers followed the same public breadcrumbs to their discoveries. Most prominently, security researcher Anders Fogh, a malware analyst for German firm GData, in July wrote on his blog that he had been exploring a curious feature of modern microprocessors called speculative execution. In their insatiable hunger for faster performance, chipmakers have long designed processors to skip ahead in their execution of code, computing results out of order to save time rather than wait at a certain bottleneck in a process.

Perhaps, Fogh suggested, that out-of-order flexibility could allow malicious code to manipulate a processor to access a portion of memory it shouldn't have access to—like the kernel—before the chip actually checked whether the code should have permission. And even after the processor realized its mistake and erased the results of that illicit access, the malicious code could trick the processor again into checking its cache, the small part of memory allotted to the processor to keep recently used data easily accessible. By watching the timing of those checks, the program could find traces of the kernel's secrets.

Fogh failed to build a working attack, due to what other researchers now say were quirks of his testing setup. But Fogh nonetheless warned that speculative execution was likely a "Pandora's box" for future security research.

Still, Fogh's post hardly sounded alarms for the broader hardware security research community. It was only months later that the researchers at the Graz University of Technology started to closely consider his warnings. Their first real clue came instead from the Linux kernel mailing list: In October, they noticed that developers from major companies including Intel, Amazon, and Google were all suddenly interested in a new defensive redesign of operating systems, called KAISER, that the Graz researchers had created, with the goal of better isolating the memory of programs from the memory of the operating system.

The Graz researchers had intended KAISER to solve a far less serious issue than Meltdown or Spectre; their focus was on hiding the location of a computer's memory from malicious, not necessarily blocking access to it. "We felt happy," Lipp remembers. "People were interested in deploying our countermeasures."

Soon, however, developers on the mailing list began to note that the KAISER patch could slow down some Intel chips by as much as five to 30 percent for some processes—a far more serious side effect than the Graz researchers had found. And yet, Intel and other tech giants were still pushing for the fix.

"There must be something bigger here," Lipp remembers thinking. Were the tech firms using KAISER to patch a secret, more severe chip-level flaw? Only then did he and the other Graz researchers think back to Fogh's failed speculative execution attack. When they decided to try it themselves, they were shocked when their slightly tweaked implementation of Fogh's technique worked.

They also weren't alone. Just weeks earlier, by chance, researcher Thomas Prescher at Dresden, Germany security firm Cyberus had finally gotten around to testing Fogh's method. "I had looked at it half a year ago and found the ideas very interesting, but at some point I just forgot about it." Prescher says. "In November, I came across it again by chance and just decided to try it. I got it to work very, very quickly."

In the end, the Cyberus and Graz researchers reported their work to Intel within days of each other in early December. Only after Intel responded to each of the researchers' bug reports in the middle of that month did they learn that someone had independently discovered and reported their Meltdown attack months prior—as well as the distinct speculative execution attack known as Spectre. That warning came from Project Zero, Google's elite team of bug-hunting hackers. In fact, Project Zero researcher Jann Horn had found the attack in June—weeks before Anders Fogh's blog post.

Starting From Zero

How did Horn independently stumble on the notion of attacking speculative execution in Intel's chips? As he tells it, by reading the manual.

In late April of last year, the 22-year-old hacker—whose job at Project Zero was his first out of college—was working in Zurich, Switzerland, alongside a coworker, to write a piece of processor-intensive software, one whose behavior they knew would be very sensitive to the performance of Intel's chips. So Horn dived into Intel's documentation to understand how much of the program Intel's processors could run out-of-order to speed it up.

He soon saw that for one spot in the code he was working on, the speculative execution quirks Intel used to supercharge its chip speed could lead to what Horn describes as a "secret" value being accidentally accessed, and then stored in the processor's cache. "In other words, [it would] make it possible for an attacker to figure out the secret," Horn writes in an email to WIRED. "I then realized that this could—at least in theory—affect more than just the code snippet we were working on, and decided to look into it."

By early May, Horn had developed that technique into the attack that would come to be known as Spectre. Unlike Meltdown's more straightforward abuse of the processor, Spectre leverages speculative execution to trick innocent programs or system processes on a computer into planting their secrets in the processor's cache, where they could then be leaked out to a hacker performing a Meltdown-like timing attack. A web browser, for instance, could be manipulated into leaking a user's browsing history or passwords.

Spectre is harder for attackers to exploit than Meltdown, but also far more complex to fix. It also works not only in Intel chips, but across ARM and AMD chips too, an even thornier and longer-term problem for the industry. Horn reported his findings to the chipmakers on June 1. And as he continued to explore speculative execution's other possibilities, he found and reported the Meltdown attack to Intel three weeks later.

Finally, there would be one more coincidence in the storm of bug collisions around Meltdown and Spectre. Just around the time that Horn was beginning to test his attacks, Paul Kocher was starting a sabbatical from the San Francisco-based company he'd founded, Cryptography Research. He wanted time, in part, to explore a broad issue he saw in computer security: the increasingly desperate drive to squeeze ever-greater performance out of microchips at all costs—including, perhaps, the cost of their fundamental security.

At a cryptography and hardware conference in Taipei last September, Kocher's former colleague Mike Hamburg raised suspicions about speculative execution. Kocher was immediately determined to prove the problem. "It wasn't so much of an 'aha' moment as an an 'eww' moment," Kocher says of the realization that led him to the same attack method. "As soon as I started to look at speculative execution, it was pretty clear to me as a security person that this as a really bad idea."

Not long after he'd returned from Taipei, Kocher had coded a working exploit of his own—with no knowledge that Google's Horn had found exactly the same decades-old issue just months earlier.

Outlier or Telling Anecdote?

For Kocher, the key question isn't how so many researchers stumbled onto the same class of attack at roughly the same time. It's how the attacks remained undiscovered for so long—or whether they were in fact discovered, and used to hack unwitting targets in secret.

"If you asked me whether intelligence agencies found this years ago, I would guess certainly yes," Kocher says. "They have some of the world’s best efforts at these sorts of things. It would be quite likely they would have noticed. And if they found something like this, as long it's yielding good intelligence, they don’t tell anyone."

"It's not just the NSA," he adds. Other state-sponsored hackers likely have the skills—and had the time—to have potentially found the Spectre and Meltdown attacks, too.

On Friday, White House cybersecurity coordinator Rob Joyce, a former senior NSA official, told The Washington Post that the NSA didn't know about Spectre and Meltdown and had never exploited the flaws. Joyce has also touted a move to reveal more about the NSA's rules for disclosing vulnerabilities it finds, a policy known known as the Vulnerabilities Equities Process.

Despite the almost uncanny anecdotal evidence for bug rediscovery that Spectre and Meltdown represent, it's far from clear just how common that phenomenon has become. The Harvard Study co-authored by Bruce Schneier, for one, examined a trove of bug report data containing 4,300 vulnerabilities. Fourteen percent of Android vulnerabilities were reported again within just 60 days of their initial discovery, and around 13 percent of Chrome bugs. "For the NSA, holding onto vulnerabilities is way more dangerous than you’d think, given the raw numbers," Schneier says.

But another study released last year by the RAND corporation, which looked at bugs from an unnamed research organization, found only a 5.7 percent chance that a given bug would be found again and reported within a year—although the study didn't account for other, secret bug discoveries.

Lillian Ablon, one of the RAND study's authors, sees the Spectre and Meltdown rediscoveries not as a broad sign that all bugs are found several times over, but that trends in computer security can suddenly focus many eyes on a single, narrow field. "There may be bug collisions in one area, but we can’t make the grand statement that bug collisions happen all the time," she says. "There will be codebases and classes of bugs where no attention exists."

Paul Kocher argues the real lesson, then, is for the security research community not to follow in each others' footsteps but to find and fix bugs in the obscure code that rarely attracts widespread attention.

"Throughout my career, whenever I've looked somewhere there isn’t a security person looking, I find something nasty and unpleasant there," Kocher says. "The shocker for me is that these attacks weren't discovered long ago. And the question that I struggle with and fear is, how many other things like this have been sitting around for 10 or 15 years?"

More Meltdown