One person unsurprised by the unfolding data scandals surrounding Cambridge Analytica and Facebook is Cathy O’Neil. In 2016 Cathy published her book Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. In the book O’Neil reveals how a silent bureaucracy governed by algorithms and big data is emerging across every corner of society.
This new bureaucracy is increasingly deciding who gets a job, who gets credit (and at what rate), who goes to prison and what information people read. Some of these systems may be making accurate decisions. However, Cathy argues that accuracy and efficiency alone are not sufficient metrics for success. Fairness, equity and other social considerations need to be built into the algorithms. Until this is done, these secretive and unaudited algorithms will continue to make unfair, bias and discriminatory decisions, on a systemic level.
Today, there is mounting evidence of this silent, discriminatory decision-making system in action. In the UK, one police force edited its algorithm after receiving complaints that it was unfairly targeting people from specific neighborhoods. And new research from Princeton on AI natural language applications, confirms that automated systems are not only making discriminatory decisions, they are also learning the language of human prejudice as well
One of the most influential ideas to emerge from O’Neil's book is the idea of creating a 'Hippocratic oath' – an ethical code of conduct – for data scientists to follow. The idea is to imbue data scientists with a moral conscience which would guide their thinking when designing systems and force them to consider the wider societal impact of their designs.
Since O’Neil proposed the idea, a host of initiatives have sprung up, offering their own version. Within the fields of AI and machine learning, the biggest tech companies have formed the Partnership on AI and Google's DeepMind has its own AI ethics unit. However few of these efforts have yet had much influence. We caught up with Cathy to explore how to give a data scientist Hippocratic oath some teeth.
Tom Upchurch: Cathy, remind us why a Hippocratic oath for data scientists is so important?
Cathy O’Neil: First, I just want to make it clear that a Hippocratic oath alone is insufficient for the task that lies ahead, because at the end of the day data scientists are not corporations. They work within corporations, and they get fired if they don’t do what the corporations tell them to do. So, I don’t want to make it seem like once this ethical framework has been set up we’re going to be good, because it’s just not true.
Having said that, so many of the data scientists that are in work right now think of themselves as technicians and think that they can blithely follow textbook definitions of optimisation, without considering the wider consequences of their work. So, when they choose to optimise to some kind of ratio of false positives or false negatives, for example, they are not required by their bosses or their educational history to actually work out what that will mean to the people affected by the algorithms they’re optimising. Which means that they don’t really have any kind of direct connection to the worldly consequences of their work.
Just as an example, I talked to a guy who has a PhD in statistics who built a recidivism risk algorithm for a state prison system and I asked him, “do you ever use race as an attribute to determine recidivism risk?” and he said, “Oh, no. I would never do that.” Then I asked, “Well, do you ever use Zipcode?” and he was like, “Oh, I sometimes do, because it makes it so much more accurate.”
What he’s exposing there is that what he actually cares about is accuracy, rather than some kind of concept of fairness or equity. I then asked, “Is there anybody at the state prison system who talks to you about which attributes are acceptable versus unacceptable?” He said, “No, they just trust me because I have a PhD.” Then, I said, “Well, in that case do you feel personally responsible for the consequences of your algorithm?” and he said, “No, I just built the algorithm. They decide how to use it.”
I feel like that’s a perfect story because it’s also the paradigm that we’re working with. We have a total disconnect between the people building the algorithms and the people who are actually affected by them. That’s true on a scientific level, between those who understand big data techniques versus those who see it as voodoo or magic. But it’s also true in a socioeconomic sense.
Read more: DeepMind's Mustafa Suleyman: In 2018, AI will gain a moral compass
TU: What should be the core principles lying behind the oath?
CO: First, it needs to stop a specific mindset where we think that data people are magical and that they have any kind of wisdom. What they actually have is a technical ability without wisdom.
Then, I think the biggest thrust of a Hippocratic oath would be to realise that we have the ability and the potential to have an enormous amount of influence on society but without the wisdom to understand the true impact of this influence. So, we have to ask the relevant experts for that wisdom and for that understanding and expertise.
The more I’ve thought about algorithms, the more I’ve realised that it’s not even fair to ask about the ethics of an algorithm, because you really have to ask about the ethics of an algorithm in a given context.
TU: How would this be implemented practically?
CO: To make it more practical, I’m introducing a concept called the ethical matrix, which is a concept that I’ve borrowed from a moral philosopher, called Ben Mepham, who originally used it in the context of bioethics.
The ethical matrix is a framework to force people to think more broadly. The rows of the matrix includes all the stakeholders that may be impacted by a given algorithm. For example, in the case of the recidivism risk algorithm it will include the judge, it will include the prosecutor, it will include the defendant and it will include the general public.
Then, the columns of the matrix are the things that each of these stakeholders actually worry about. For example, the judge is going to worry about freeing someone who is dangerous to the public (a false negative.) Whereas the defendant’s worrying about being punished unnecessarily for something they didn’t or weren’t going to do (a false positive).
The ethical matrix helps lay out all of these competing implications, motivations and considerations and allows data scientists to consider the bigger impact of their designs.
TU: In your book you talk about the importance of building transparent systems that are continually improved by positive feed-back loops of data. Do these practical steps feature in your vision of an ethical code of conduct?
CO: Yes, they do. Essentially, I want to put the science into data science. I feel like we’ve called data science a science prematurely, because we as the public don’t demand evidence that it works. So, we need to start demanding scientific evidence and testing to make sure this stuff is working.
How do we define “working”? I have three different questions to ask.
The first question is, are the algorithms that we deploy going to improve the human processes that they are replacing? Far too often we have algorithms that are thrown in with the assumptions that they’re going to work perfectly, because after all they’re algorithms, but they actually end up working much worse than the system that they’re replacing. For example in Australia they implemented an algorithm that sent a bunch of ridiculously threatening letters to people saying that they had defrauded the Australian Government. That’s a great example where they actually just never tested it to make sure it worked.
The second question is to ask, for whom is the algorithm failing? We need to be asking, “Does it fail more often for women than for men? Does it fail more often for minorities than for whites? Does it fail more often for old people than for young people?” Every single class should get a question and an answer. The big example I have for this one is the facial recognition software that the MIT Media Lab found worked muchbetter for white men than black women. That is a no-brainer test that every single facial recognition software company should have done and it’s embarrassing that they didn’t do it.
The third category of question is simply, is this working for society? Are we tracking the mistakes of the system? Are we inputting these mistakes back into the algorithm so that it’ll work better? Is it causing some other third unintended consequence? Is it destroying democracy? Is it making people worse off?
Read more: May's Davos speech exposed the emptiness in the UK's AI strategy
TU: How do you think your system will mix with an industry guided by the mantra “move fast and break things”? Can you see this being taken up?
CO: I can, because first of all, they can be standardised, so they can be part of a corporate checklist that every data science group has to abide by. It wouldn’t be that hard to make them a standard approach.
The other thing is that the ethical matrix and the questions will only act as third party support. They will not threaten the ‘secret sauce’ or the hidden IP of an algorithm. One of the most common complaints you’ll hear from companies is that they don’t want to give away their secret sauce. They have this amazing algorithm that they want to protect. But nothing I just mentioned actually exposes those secrets.
TU: Although in your book you do mention that a key part of being transparent is allowing third party, independent auditing of algorithms. Wouldn’t this expose the secret sauce?
CO: Just to be clear, I think the people that should be doing these tests are not the companies themselves – it should be third party auditors. I should also, in full disclosure mention, that I have started a company that does audits. So, I have been developing this framework with a business goal.
Just to be clear, if Facebook hired me to audit their algorithm, I could do that, but I wouldn’t need their source code. I would need access to their black box and I would need them to perform tests with data. So, I would only need indirect access to their data, but I wouldn’t need direct access to the source code of their data.
**TU: Ever since you came out with this idea of the Hippocratic oath, there have been many separate initiatives set-up. But in medicine the Hippocratic oath has persisted because it is the Hippocratic oath. It is regarded globally as the definitive ethical standard for doctors and has become an important part of the ritual in making a doctor.
Is there room for more than one Hippocratic oath for data scientists?**
CO: I think it would be better if there were one but obviously only if it was a good one!
I have been asked to sign a bunch of different letters of intention for this kind of thing and I have not signed any of them because I haven’t found one that I think is sufficient.
The ones I have seen so far, are still dodging responsibility. At the end of the day, what we should be worried about isn’t that this algorithm might be inefficient or might not maximise profits, but that somebody’s rights have been violated.
I have, as of yet, not seen anything in any of those other codes of conduct or Hippocratic oath, that are this specific.
TU: The medical Hippocratic oath formed over thousands of years. How long do you expect this new kind of moral conscience to take hold amongst data scientists?
CO: Well, I don’t think we have thousands of years, but I am willing to wait ten years! I don’t think it is going to happen overnight. I think it is going to require us to see the damage that algorithms can do, and we’ve started to see that. But we’re still focusing on the wrong things. We’re not measuring the actual damage being done to democracy, because it is hard to measure democracy. Instead, we’ll focus on things like pedestrian deaths with self-driving cars. That is of course a tragedy when it happens, but it is not actually the biggest, systemic problem. The biggest problem is all the invisible failures of algorithms that we’re not keeping track of at all.
TU: I got a sense from Weapons of Math Destruction that there is a conflict between profit seeking companies and a code of ethical standards. Is there a conflict emerging between the two?
CO: Yes, there is a conflict. But I would like to add, that at this point, Zuckerberg is begging for someone else to deal with the unintended consequences of Facebook’s business model. He has basically said yes, you guys should regulate us, go ahead, because I can’t solve this problem. It is getting kind of obvious that something needs to be done.
But the way I look at it is if you want to stick with the analogy of doctors, medicine was practiced for a long time by mystics and snake oil salesman that would look at someone, figure out what they’re insecure about and sell them the thing that would make them the most profit at the smallest cost, whether it was actually helpful or not. The point is that when you don’t optimise to ethics, you end up optimising away from ethics.
So, when Facebook optimises everything around keeping you on Facebook, they are optimising you to their own profit. As a result, they are optimising you away from things that we actually care about, like truth and civic engagement and civil disagreement. Because civil disagreement doesn’t keep you on Facebook. Outrage keeps you on Facebook.
So, they end up optimising directly away from what we care about and so the answer is yes, absolutely there is a conflict.
Read more: AI has no place in the NHS if patient privacy isn’t assured
TU: On the subject of changing corporate behaviour, in your book you talk about how IBM started to take a more progressive position over same sex marriages in the 1990s. You argue this was largely driven by the desire to attract the best and brightest talent, which included lots of gay and lesbian people.
This is an example where a company sought competitive advantage in taking a progressive and ethical position. Do you see any evidence to suggest that taking an ethical stance on data standards may soon become a source of competitive advantage for companies?
CO: That is a great question, and I think the answer is that it definitely depends on whether the company that is building and deploying the algorithm is consumer facing or not. In the case of Facebook, they are definitely consumer facing. To be sure, we are not the customers of Facebook, we are the product of Facebook, but we still have to go to Facebook for them to get our data.
So, in that case, the PR element of it forces them to behave better, at least it is supposed to. We will see if it actually happens. Zuckerberg has made promises in the past that he hasn’t kept. But theoretically it is possible that public outrage will force Facebook to change.
In many other cases, they are not consumer facing organisations. Companies that build the algorithm to decide who to hire, do not care and don’t have to respond to public outcry. There really probably isn’t even an outcry because the effects are happening silently and the people that suffer from these algorithms are poorer, less powerful people.
TU: On the subject of regulation, who needs to be enforcing and backing up the ethical standards? Is there a particular institution out there at the moment, or does it need to be created?
CO: It needs to be created, or alternatively, every single regulator needs to have a division that audits algorithms. A lot of people have floated the idea of an FDA for algorithms and I think that is the kind of thing we should be thinking about. I think the FDA for algorithms would enforce those three kinds of tests, and the ethical matrix.
TU: How important is it that these new kinds of regulators have recourse to financial sanctions and penalties?
CO: I witnessed the pathetic response to the illegal goings-on of the banks during, before and even after the financial crisis and I saw the numerous settlements where penalty fees were charged that were less than the profits they made on their fraudulent dealing. I think it is fair to say that this is not a very strong incentive to keep them from doing it in the future.
So, I think the point is that you have to threaten these organisations with a meaningful and existential threat.
TU: Looking to the future, what do you see as the biggest challenges in getting this code of conduct and this framework widely adopted over the next ten years?
CO: The biggest obstacle in the next three years is Trump because he can’t focus or concentrate on anything, never mind abstract threats to democracy. Not to mention that every political party benefits from the way politics are promoted online. So, they also have a conflict of interest.
I see this as the larger problem as well. Like the lobbyists for the healthcare industry, the lobbyists from insurance, the lobbyists from real estate. All these industries are benefitting from this. They are benefiting directly from being able to choose the winners and losers of their industries. It is an arms race. They always think they are winning and they’re caught up in it.
It is definitely a test of democracy, because whilst this big data thing might be working for individual actors in a competitive marketplace, in a given industry, it is not necessarily working for the public. So, how is that going to play out? It is a real test and I don’t know if America is going to lead, to be honest.
This article was originally published by WIRED UK