By Marlene Lutz
We live in an age in which intelligent computer systems have become an indispensable part of our lives. Algorithms decide which advertisements we are exposed to, which movies are suggested to us, and which search engine results we see, tailored to our individual needs. Banks use algorithms to assess the creditworthiness of clients, large companies let computer systems filter and analyze the resumes of job applicants, and in criminal law, statistical scores are used to assess the likelihood of recidivism of offenders. We are at a point where so-called machine learning systems are used to make decisions that directly impact what opportunities we can seize and whether we can achieve our goals.
While one of the driving reasons for using machine learning systems was to replace subjective human decision-making with “objective” algorithmic ones, it has become apparent that these systems are vulnerable to biases and have the potential to discriminate. Just in 2018, Amazon revealed that the hiring algorithm they have been developing for years showed a bias against women. Female applicants were systematically rated lower for technical positions, even though their resumes had comparable qualifications to those of male applicants. Two years later, the tech giants IBM, Microsoft, and Amazon suspended sales of facial recognition software to the police after realizing that their facial recognition technology performed significantly worse on people of color and especially darker-skinned women, putting them at particularly high risk for false accusations of crimes.
To understand how a machine learning algorithm is capable of discrimination, we need to take a closer look at the steps leading to a decision. A typical decision-making process begins by gathering information and identifying those that are relevant to the problem. For many people, this process begins with intuition. Perhaps an employer has observed that individuals who major in math perform particularly well in the financial sector. By looking at the historical evidence, this observation can be tested and a pattern can be identified to what degree the major of an individual correlates with their success on the job. This is the work of traditional statistics and often these data-driven decisions are more consistent and accurate than human judgment. The discipline of machine learning now goes one step further and discovers relevant patterns and relationships on its own. Instead of starting with an intuition about a relationship and testing it statistically using historical data, the decision about which factors are important for a task and to what extent is transferred to the data itself.
But let’s take a step back. What is Machine Learning, one of the buzzwords these days, in the first place? Machine learning is a subfield of Artificial Intelligence “that gives computers the ability to learn without being explicitly programmed” (Arthur Samuel, 1959). It can be used to solve all kinds of tasks, for example, the classification of e-mails into spam and no spam, the decision whether a bank grants a loan, or the detection of tumors in medical imaging. Let’s say we want to develop a machine learning model that tells us whether a job applicant should be hired or not. We have certain information about the applicants, called features, such as their major, grades, age, and address. These features are obviously not equally important to the decision process. However, we leave it up to the model to decide which features are relevant for hiring an applicant. For this purpose, we show our model a bunch of examples of candidates who have applied in the past and whether or not they were ultimately hired. They form the so-called training data. In the learning step, the features are then combined in various ways until the model can reproduce the hiring decisions for the applicants from the training data with high accuracy. The more examples the model sees, the better it gets and the more general the rules and patterns it extracts from the data become. After training, a good model will eventually be able to predict whether or not applicants not previously seen by the model should be hired.
The fact that machine learning is able to autonomously discover patterns and rules in historical evidence data is a great achievement. Connections can be discovered that we humans would have overlooked due to their inconspicuousness or complexity. In clinical applications, for example, cutting-edge machine learning systems now outperform radiologists in diagnosing lung cancer as they can detect even the smallest abnormalities.
More than that, machine learning can overcome an interesting weakness of humans. While we are easily able to identify objects in images and videos, we are not able to determine the full set of rules with which we have achieved this. Thus, we cannot write a computer program that would, for example, recognize a dog against every possible background and from every angle. In contrast, a machine learning model learns such rules and patterns itself by exposing it to a large set of images with dogs in different environments and from different perspectives.
But how do machine learning models come to discriminate? It is a common misconception that data-driven systems are fair and objective by virtue of the fact that their decisions are based on evidence. In fact, one of the main sources of unfairness in data-driven systems lies in the data itself. By exposing machine learning models to large amounts of data, they learn correlations and patterns hidden within them. Yet, the world we live in is full of disparities, prejudices, and stereotypes, which are very likely to be reflected in the data that a machine learning algorithm uses for training. As an example, let’s take a look at the machine learning-based translator DeepL and enter the sentences “She is a doctor. He is a nurse.”. When these sentences are translated from English into Estonian, a language without grammatical gender, and back into English, we get “He is a doctor. She is a nurse.”. Thus, DeepL has reversed the genders of the pronouns. The translation reflects actual statistics about the gender distribution in the respective professions, yet they correspond to stereotypes that are not desired to be included in the translation model. In general, a machine learning algorithm cannot distinguish between patterns in the data that represent useful knowledge and should be learned by the model and those that are potentially harmful. Let’s return to Amazon’s biased hiring algorithm mentioned in the beginning. The training data consisted of resumes of applicants from the last 10 years. Most applicants and employees during this period were men – a reflection of the overall male dominance in the tech industry. This pattern was detected by the machine learning model and adopted accordingly. An obvious solution to this problem seems to simply withhold information about the applicant’s gender from the model. This idea sounds tempting, but unfortunately removing explicit information about a person’s gender does not make the model oblivious to it. After all, the idea of machine learning is based on inferring patterns and absent information from those that are present. Amazon’s hiring algorithm, for example, scored applicants lower who went to all-women colleges or were captains of the “women’s chess club”.
But even if our training data were free of disparities, such may enter the model by other means. Suppose we want to design a machine learning model for a population that contains minority groups that differ in some way from the general population with respect to the task at hand. By definition, there is fewer data available for minorities. A machine learning model, however, improves its accuracy with the number of data points used for training. Conversely, this means that the model will make worse predictions for minorities than for the general population. And indeed, a landmark project called “Gender Shades” has revealed that various facial recognition technologies, including those from IBM and Microsoft, perform worse on people of color, and especially on women. One reason for this is that the standard training databases predominantly contain images of white males.
These findings paint a rather bleak picture and raise the question of whether it would not be better to banish machine learning systems from our lives. But despite the challenges and risks for discrimination that machine learning presents us with, it also offers great opportunities and potential in many domains such as medicine, speech recognition, or computer vision.
The good news is that the existing shortcomings led to a thriving research community dedicated to the fairness and transparency of machine learning. Computer scientists need to work hand-in-hand with governments and policymakers whose responsibility is to establish regulations and ethical standards for these systems. The development of fairness-aware machine learning algorithms forces us to look at our moral standards and ask the honest question of whether humans would have made fairer decisions in the first place. After all, the data that data-driven systems learn from is merely a mirror of human behavior. In the end, it seems like the problem is not machine learning itself, but the things the machines learn from us.
Cover: ThisIsEngineering
