Fairness is Not A Fact

Once we get Artificial General Intelligence, we'll see an explosion in innovation; we need more data, bigger models and better benchmarks. The more intelligent the machine, the better for everyone. This seems obvious, but it's like saying humanity's achievements can be attributed to greater intelligence. It seems intuitive, but maybe something else matters far more.

Value of Fairness

Capuchin monkeys are also very intelligent. They understand transactions, and they'll happily swap a token for a slice of cucumber, just like any of us would. But as soon as a capuchin sees another monkey getting a grape in exchange for their token, they suddenly refuse to trade. Getting cucumber for a token was a good deal until someone else got a different deal. This is believed to be proof that capuchins have a sense of fairness. They may lose all interest in cucumbers as soon as they see a grape, but the same has been observed in humans as well. In fact, it was a central premise in the book Never Split the Difference: people don't want a good deal; they want to feel like they've been treated fairly.

We will destroy our own lives if we think the alternative is that someone gets a better deal than we do. My life would be better if it had a slice of cucumber in it, provided someone else who contributed equally didn't get given a grape. Then I'd go hungry until I get treated fairly. The issue is that I, just like you, can't define fairness. And when I try, it's unlikely my definition will match yours, the guy who gives out the grapes' definition, or anyone else's. That is why defining fairness matters more than any increase in intelligence.

Humans aren't even uniquely particularly intelligent. Young children perform about as well as chimpanzees or orangutans of the same age on spatial, quantitative, and causal problems. Human advantages come later, with communication, learning from others, and cooperation. Henrich calls this our collective brain. And we do not collaborate if we think the collaboration isn't fair.

Knack and Keefer's cross-country studies found that a ten-point increase in interpersonal trust was associated with roughly 0.8 percentage points of additional annual growth, roughly equivalent to a primary school education. Later researchers tested whether trust actually drives growth. Using instrumental-variable methods to rule out reverse causation, Algan and Cahuc found "a decisive determinant of growth".

Cooperation in society requires people to both follow and believe others will follow a set of stated and unstated rules. Trust exists on the intersubjective layer, like money and nations; it exists because enough people believe it does. And once enough people don't think trust exists, it doesn't. So we're not just talking about laws that are written down and enforced, but about social or cultural norms that we all know others around us will live by.

A thousand chimpanzees will always be a thousand chimpanzees. A thousand people can be a company, a religion, a nation, or even a scientific institution or a research and development team.

A business in Japan can agree to sell goods to a company in Indonesia, and both sides can get to work knowing the agreement will be successful. They feel this way because an arbitrator in London backs their agreement. Is that because the English are considered more intelligent or even more reliable? Probably not; the arbitrator probably wasn't even born in England. They're simply working within a legal framework with a 400-year history and reputation of trust to uphold. English common law accounts for 40% of governing law in global corporate arbitrations. That reputation lets people in two completely different countries work together.

In 17th- and 18th-century England, society underwent huge transformations—civil wars, the Glorious Revolution, and the gradual evolution of English common law. Over time, this legal system created more predictable rights and protections, allowing inventors and entrepreneurs to do their thing. This stability and cooperation led to huge improvements in innovation, banking, trade, the Industrial Revolution, and the abolition of slavery.

Measuring Fairness

You may be familiar with the COMPAS debate from 2016. If not, COMPAS was a risk-assessment tool used in parts of the US criminal-justice system to predict who might reoffend. ProPublica got their hands on two years of COMPAS data and painstakingly combined it with data on who actually reoffended. And then they concluded that the algorithm was racist. This is even more shocking when you consider that COMPAS didn't know the race of any of the defendants. But this can happen because of correlated variables: certain postcodes, the sports someone plays, and career choices could all signal someone's race to the algorithm.

Black defendants who did not reoffend were much more likely to have been predicted as high risk than White defendants who did not reoffend. The false-positive rate was about 44.9% for Black defendants and 23.5% for White defendants. As this confusion matrix shows, the numbers are relative to the outcome (did reoffend / did not reoffend).

This seems pretty indefensible and a clear indicator of racism. If you're in a group that is twice as likely to be labelled as dangerous, you get harsher bail, stricter supervision and other generally less favourable treatments. So this needs to be fixed for us to live in a fair society, which, as we already know, is a society that is better for everyone.

So, should we drastically change this algorithm or get rid of this algorithm? Well, Northpointe (the team behind COMPAS) said that it isn't unfair. They used a different definition of fairness, so they calculated it differently. Northpointe had been asking: Of all the people who did not reoffend, how many were incorrectly predicted to reoffend? They considered this a better measure of fairness because it has the fewest people incorrectly assigned and actually has the same level of accuracy across groups. If we look at the confusion matrix but consider the numbers relative to the prediction rather than the outcome for each ethnic group, we see something drastically different.

ProPublica defines fairness as classification parity, while Northpointe defines fairness as predictive parity. Both sound good on their own, but sadly you can't have both; you cannot guarantee a score means the same thing across groups while also ensuring groups experience the same rates of false positives and false negatives. If Northpointe changed COMPAS to use classification parity as its definition of fairness, then the overall number of people incorrectly classified as high risk would increase, which doesn't sound very fair either.

Northpointe stuck with predictive parity, but they left us with a problem: There isn't a simple answer to whether a model is fair. Because we don't have a single definition of what fair means. In 2021, when the Toeslagenaffaire scandal in the Netherlands falsely accused thousands of committing benefit fraud, most of those accusations targeted dual nationals; the Dutch government assessed the algorithm's fairness using classification parity.

There are far more than two ways to measure fairness. Different groups have identified anything from ten to twenty definitions, many of them incompatible. The EU AI Act is well aware of the fairness paradox and thus allows individual algorithms and AIs to define and defend fairness. COMPAS would not be allowed in the EU because of criminal profiling, but using predictive parity as a measure of fairness would be acceptable, as long as Northpointe tracked the data and could defend its choice.

There isn't a perfect measure of fairness or a perfectly fair action; this isn't a very satisfying conclusion to a utopianist. For the rest of us, agreeing on what we mean by fairness in any specific context is a huge challenge. But it's actually a huge potential opportunity too, one far greater than the bankers and business people of London in the 17th century.

(It's worth noting that the COMPAS debate brought up many other discussions about algorithms and predictions, including how reoffence was calculated based on rearrest, which includes their own biases; I'm consciously focusing specifically on the measure of fairness paradox here)

Our New Fairness Advantage

In 2016, the same year as the COMPAS debate, Cathy O'Neil wrote the book Weapons of Math Destruction. Across ten chapters, she showed how models can and have harmed people in education, employment, policing, insurance, credit, and healthcare. The book was a wake-up call for many. One example was a police force using data on the locations of past crimes to decide where to patrol. This leads to a perpetual cycle of arrests: more arrests happen where police patrol, so more police end up patrolling that area. The algorithm suffers from an availability bias where it assumes a district has higher crime simply because it has more available data on it.

It took me many years to realise that a central premise of Cathy's book suffered from the exact same availability bias. The book claimed "Big Data Increases Inequality" as its subheading, yet that "increases" remained entirely unsubstantiated throughout (I understand enough about how this works to know the publisher probably pressured her to use a sensational premise in the subheading. I still consider it a fantastic book). If you want to make a claim that inequality is increasing, you need to have a measurement of how non-algorithmic decisions affected inequality to benchmark against. Including the decisions made by racist police chiefs prior to the use of algorithms. Assuming one is worse because the data is available is the same error in judgement the book criticised.

The algorithms have some nice, clean input and output data to analyse. The algorithms are and always will be far from perfect, but they could still be the best decision makers we've ever had.

AI gives us one of the most unique opportunities in human history: for the first time, we have a way to measure fairness and a need to talk about it. We've always been able to accuse a judge or a teacher of racism, but it was never more than an accusation; we never had the quantity of comparable data to ever say with any certainty. We could never take the judge's brain out of the skull and run a million simulations through it while subtly adjusting input variables. If we disagreed with the judge's definition of fairness, we could never reliably update it and put the brain back. Further to this, we could have a fully functional judge brain where everyone agreed with its definition and application of fairness, and that judge could wake up one day and decide to be a massive racist. The possibility to continually monitor the fairness of a model as it continues to learn gets us over one of the largest problems we've historically had with fairness: power corrupts, and absolute power corrupts absolutely; many good people in a position of power can quickly become less good over time; the longer it's been since they had to pass any actual exams to prove their judgement, the fewer reasons they have to continue having a good judgement. A model can be in a state of continuous testing even as new tests are created.

Summary

AI or algorithms achieving more intelligence than humans will be the easy part; humans were never particularly intelligent to begin with. Call me a utopianist if you like, but I believe working toward a more transparent and ethical AI could have a far larger positive impact on humanity than any improvements to its intelligence. Maybe the biggest we've ever had. First, we need to be able to trust AI like we trust a teacher, judge, coworker, or boss, and we need to feel it will treat us as equally as it treats everyone else.

Right now, we're moving away from this target, though, and the argument about who will win the AI race between the USA and China becomes laughable when we consider that its true value is likely a new era of transparency and ethics. This is the first in a series I intend to write about AI ethics to complement my studies and writing in AI interpretability and observability. Follow me if you find this interesting.