Even today, a hacker with a command of artificial intelligence may be able to force a self-driving car to miss a stop sign, or a facial recognition system to believe it’s seeing a completely different person in a security setting. Researchers have shown that virtual personal assistants like Siri or Google Now can be tricked into visiting potentially malicious websites by audio that sounds like white noise to humans.
To thwart such hackers, Elon Musk’s OpenAI and Pennsylvania State University released a new tool this week called “cleverhans,” that lets artificial intelligence researchers test how vulnerable their AI is to adversarial examples, or purposefully malicious data meant to confuse the algorithms. Once the vulnerability has been found, a defense to the attack can automatically be applied.
The tool is meant to be a “collection of attacks and defenses, along with tutorials on how to use them,” according to Nicolas Papernot, co-creator and security researcher at Pennsylvania State University, in an email to Quartz. He hopes it will serve as a tool for the industry to understand the vulnerabilities, while academics can use it to test and benchmark other new attacks and defenses.
The name Clever Hans comes from a 20th century horse, whose trainer claimed could do arithmetic. The trainer would ask a simple math question, and Clever Hans would tap out his hoof the correct number of times. It was later found that the horse was just reading social cues to know when to stop tapping. In other words, it got the right answer, but had no idea what it was doing.
“The story of Clever Hans is a metaphor for machine learning systems that may achieve very high accuracy…but that do not actually understand the underlying task and perform poorly on other inputs,” the researchers write on the project’s GitHub page.
In December 2015, Tesla CEO Musk and Y Combinator president Sam Altman announced OpenAI, a privately-funded nonprofit to make sure evildoers couldn’t use artificial intelligence to the detriment of humanity. They’ve been busy since then, snatching up some of the industry’s leading minds, moving into a new office, and setting up a “gym” to create standard AI benchmarks.
While OpenAI has published work before, this tool shows how the organization can use open-source software to protect vulnerable AI from attacks. The tool is free, editable, and can show vulnerabilities that might otherwise have gone unnoticed. Cleverhans is flexible enough to handle still images or audio, according to co-creator Ian Goodfellow of OpenAI.