Fairness is Not A Fact

Once we get Artificial General Intelligence, we'll see an explosion in innovation; we need more data, bigger models and better benchmarks. The more intelligent the machine, the better for everyone. This seems like the obvious conclusion, but it is wrong. It is similar to saying humanity's achievements can be attributed to greater intelligence; it's the same intuition, and both statements are incorrect for the same reasons.

Value of Fairness

Capuchin monkeys are very intelligent. They understand transactions, and they'll happily swap a token for a slice of cucumber, just like any of us would. But as soon as a capuchin sees another monkey getting a grape in exchange for their token, they suddenly refuse to trade. Getting cucumber for a token was a good deal until someone else got a different deal. This is believed to be proof that capuchins have a sense of fairness. They may lose all interest in cucumbers as soon as they see a grape, but the same has been observed in humans as well. In fact, it was a central premise in the book Never Split the Difference: people don't want a good deal; they want to feel like they've been treated fairly.

We will destroy our own lives if we think the alternative is that someone gets a better deal than we do. My life would be better if it had a slice of cucumber in it, provided someone else who contributed equally didn't get given a grape. Then I'd go hungry until I get treated fairly. The issue is that I, just like you, can't define fairness. And when I try, it's unlikely my definition will be like your definition, the 'grapes' definition, or anyone else's. That is why defining fairness matters more than any increase in intelligence.

Humans aren't even uniquely particularly intelligent. Young children perform about as well as chimpanzees or orangutans of the same age on spatial, quantitative, and causal problems. But human advantages come later, with communication, learning from others, and cooperation. Henrich calls this our collective brain. And we do not collaborate if we think the collaboration isn't fair.

Knack and Keefer's cross-country studies found that a ten-point increase in interpersonal trust was associated with roughly 0.8 percentage points of additional annual growth, roughly equivalent to a primary school education. Later researchers tested whether trust actually drives growth. Using instrumental-variable methods to rule out reverse causation, Algan and Cahuc still found "a decisive determinant of growth".

Cooperation in society requires people to both follow and believe others will follow a set of stated and unstated rules. Trust exists on the intersubjective layer, alongside things like money and nations; it exists because enough people believe it does. And once enough people don't think trust exists, it doesn't. So we're not just talking about laws that are written down and enforced, but about social or cultural norms that we all know others around us will live by.

A thousand chimpanzees will always be a thousand chimpanzees. But a thousand people can be a company, a religion, a nation, or even a scientific institution or a research and development team.

A business in Japan can agree to sell goods to a company in Indonesia, and both sides can get to work knowing the agreement will be successful. They feel this way because an arbitrator in London backs their agreement. Is that because the English are considered more intelligent or even more reliable? Probably not; the arbitrator probably wasn't even born in England. They're simply working within a legal framework with a 400-year history and reputation of trust to uphold. English common law accounts for 40% of governing law in global corporate arbitrations. That reputation lets people in two completely different countries work together.

In 17th- and 18th-century England, society underwent huge transformations—civil wars, the Glorious Revolution, and the gradual evolution of English common law. Over time, this legal system helped establish more predictable rights and protections, contributing to a climate where enterprising individuals could thrive. The growing stability and cooperation led to advances in innovation across banking, trade, the Industrial Revolution, and the abolition of slavery.

Measuring Fairness

You may be familiar with the COMPAS debate from 2016. If not, COMPAS was a risk-assessment tool used in parts of the US criminal-justice system to predict who might reoffend. ProPublica obtained 2 years of COMPAS data and painstakingly combined it with data on who actually reoffended. They concluded the algorithm was racist. This is even more shocking when you consider that COMPAS didn't know the race of any of the defendants. But this can happen because of correlated variables: certain postcodes, the sports someone plays, and career choices could all signal someone's race to the algorithm.

Black defendants who did not reoffend were much more likely to have been predicted as high risk than White defendants who did not reoffend. The false-positive rate was about 44.9% for Black defendants and 23.5% for White defendants. As this confusion matrix shows, the numbers are relative to the outcome (did reoffend / did not reoffend).

This seems pretty indefensible and a clear indicator of racism. If you're in a group that is twice as likely to be labelled as dangerous, you get harsher bail, stricter supervision and other less favourable treatments. So this needs to be fixed for us to live in a fairer society, which we already know is a society that is better for everyone.

So, should we drastically change this algorithm or get rid of this algorithm? Well, Northpointe (the team behind COMPAS) said that it isn't unfair. They used a different definition of fairness, so they calculated it differently. Northpointe had been asking: Of all the people who did not reoffend, how many were incorrectly predicted to reoffend? They considered this a better measure of fairness because it has the fewest people incorrectly assigned and actually has the same level of accuracy across groups. If we look at the confusion matrix but consider the numbers relative to the prediction rather than the outcome for each ethnic group, we see something drastically different.

Simply put, ProPublica defines fairness as classification parity, while Northpointe defines fairness as predictive parity. Both sound good on their own, but sadly you can't have both together, because you cannot guarantee that a high-risk score means the same thing across groups while also ensuring that groups experience the same rates of false positives and false negatives. If Northpointe changed COMPAS to use classification parity as its definition of fairness, then the overall number of people incorrectly classified as high risk would increase, which doesn't sound very fair.

Northpointe stuck with predictive parity, but they left us with a problem: There isn't a simple answer to whether a model is fair. Because it matters what fair means. In 2021, when the Toeslagenaffaire scandal in the Netherlands falsely accused thousands of committing benefit fraud, most of those accusations targeted dual nationals; the Dutch government assessed the algorithm's fairness using classification parity.

And it doesn't stop here; there are far more than two ways to measure fairness. Different groups have identified anything from ten to twenty definitions, many of them incompatible. The EU AI Act is well aware of the fairness paradox and thus allows individual algorithms and AIs to define and defend fairness. COMPAS would not be allowed in the EU because of criminal profiling, but using predictive parity as a measure of fairness would be acceptable, as long as Northpointe tracked the data and could defend its choice.

There isn't a perfect measure of fairness or a perfectly fair action; this is unpalatable to a utopianist. For the rest of us, agreeing on what we mean by fairness in any specific context is a huge challenge. Still, we also have a potential advantage here, one far greater than the bankers and business people of London in the 17th century.

Our New Fairness Advantage

In 2016, the same year as the COMPAS debate, Cathy O'Neil wrote the book Weapons of Math Destruction. Across ten chapters, she showed how models can and have harmed people in education, employment, policing, insurance, credit, and healthcare. The book was a wake-up call for many; however, she made the unsubstantiated claim that algorithms were "increasing" inequality, without comparing how well we made decisions before algorithms. That weakens her argument, because the real question is not whether algorithms can cause harm, but whether they are better or worse than the alternatives. For example, a policing algorithm might send more police to a part of town where they previously made the most arrests, which can create a perpetual loop because they simply have more data for that area; it's a simple availability bias. Cathy O'Neil has data on why and how this algorithm is a problem, but she doesn't have information on whether the way police patrols were assigned in that town before was better or worse; she's criticising the thing she has data about too, which is the same availability bias.

The algorithms have some nice, clean input and output data to analyse. The algorithms she critiqued are clearly far from perfect, but they could still be the best decision makers we've ever had. After reading Cathy's book, I became like one of those people who reads the first chapter of a data analytics book (or even worse one of the millions of book about how data can lie) and then puts it down and proudly states that correlation isn't causation to any problem that doesn't have a cast-iron causation (basically anything other than a chemistry or physics experiment in a controlled environment); the statement is true and simultaneously useless if you aren't helping to prove or disprove the causation. Yes, all algorithms are imperfect whether they run in a brain or on a computer, but if you aren't working to improve them, then what use is this revolution?

AI gives us one of the most unique opportunities in human history: for the first time, we have a way to measure fairness and a need to talk about it. We've always been able to accuse a judge or a teacher of racism, but it was never more than an accusation; we never had the quantity of comparable data to ever say with any certainty. We could never take the judge's brain out of the skull and run a million simulations through it while subtly adjusting input variables. If we disagreed with the judge's definition of fairness afterwards, we could never reliably update it and put the brain back. Further to this, we could have a fully functional judge brain where everyone agreed with its definition and application of fairness, and that judge could wake up one day and decide to be a racist. The possibility to continually monitor the fairness of a model as it continues to learn gets us over one of the largest problems we've historically had with fairness: power corrupts, and absolute power corrupts absolutely; many good people in a position of power can quickly become less good over time; the longer it's been since they had to pass any actual exams to prove their judgement, the fewer reasons they have to continue having a good judgement. A model can be in a state of continuous testing even as new tests are created.

Working towards more intelligent AI will be the easy part and have a limited impact on humanity. Working towards a more transparent and ethical AI could potentially have the biggest positive impact on humanity yet. We need to trust AI like we trust a teacher, judge, coworker, or boss, and we need to feel it will treat us as equally as it treats everyone else.