AI systems increasingly help make decisions about people: who gets a loan, which job applicants are shortlisted, what content people see and how medical cases are prioritized. When those systems are biased, they can disadvantage people at scale, often invisibly. Bias in AI is rarely the result of anyone’s intent. It usually comes from the data and choices behind the system.
Key takeaways
- Bias in AI usually comes from data and design choices, not from anyone intending it.
- The main sources are unfair historical data, unrepresentative data, proxies for sensitive attributes and the choice of what a system optimizes for.
- There is no single definition of fairness, and some definitions conflict with each other.
- Organizations can reduce bias by testing performance across groups, keeping humans responsible and offering explanations and appeals.
- Under the EU AI Act, data governance duties aimed at bias in high-risk systems apply from December 2027.
Where bias comes from
The first source is historical data. Models learn from the past, so if past decisions were unfair, a model trained to imitate them will reproduce that unfairness. In 2018 Reuters reported that Amazon had abandoned an experimental recruiting tool after finding it penalized résumés containing the word “women’s” and downgraded graduates of all-women’s colleges, reflecting a male-dominated hiring history (AI Incident Database summary). Amazon said recruiters never relied solely on the tool. We discuss hiring in our job hunting article.
Second, data can be unrepresentative. If some groups are underrepresented in training data, the system may work worse for them. The Gender Shades study by Joy Buolamwini and Timnit Gebru found that commercial gender classification systems misclassified darker-skinned women up to 34.7% of the time, against a maximum of 0.8% for lighter-skinned men. Similar concerns come up in medical AI, as noted in our healthcare analysis.
Third, removing sensitive attributes doesn’t remove the problem. Even when race is excluded, other data, like postal codes or shopping patterns, can act as a proxy for it.
Fourth, design choices matter, especially what the system optimizes for. A 2019 study in Science found that a widely used US healthcare algorithm predicted future healthcare costs rather than illness. Because less money had historically been spent on Black patients’ care, “at a given risk score, Black patients are considerably sicker than White patients.” The researchers estimated that fixing the problem would raise the share of Black patients flagged for extra care from 17.7% to 46.5% (abstract).
Finally, there’s language and culture. Large language models absorb stereotypes present in text from the internet and can reflect them in what they generate, though developers work to reduce this.
How bias is measured
There is no single definition of fairness, and some definitions mathematically conflict with each other. Common approaches compare outcomes between groups:
- Do different groups receive positive decisions at similar rates?
- Are error rates, such as false rejections, similar across groups?
- Does the system perform equally well for everyone it serves?
Choosing the right measure depends on context and values, which is why fairness is not purely a technical problem.
What organizations can do
Start by examining training data for gaps and historical patterns. Test how the system performs across groups before and after deployment, as the Gender Shades researchers did. Check what you’re actually predicting; the healthcare case shows how a reasonable-looking target, cost, can stand in for something else.
For high-stakes decisions, keep humans responsible and able to override the system. Give the people affected an explanation and a way to appeal. And keep monitoring, because data and populations change.
A useful question For any AI system that affects people, ask: who might this work worse for, and how would we know?
What regulators are doing
The EU AI Act requires high-risk systems, including many used in hiring, credit and public services, to meet data governance standards aimed at detecting and reducing bias (Article 10). After 2026 amendments, those obligations apply from December 2027. In the US, existing anti-discrimination laws apply to automated decisions.
New York City’s Local Law 144 has, since July 2023, required employers using automated hiring tools to commission a bias audit within the previous year, publish a summary of the results and notify candidates. Enforcement has been weak: a December 2025 audit by the New York State Comptroller found that the city’s review of 32 companies turned up one case of non-compliance, while auditors identified at least 17 potential violations in the same sample. State rules elsewhere are changing quickly, so check the current position where you operate.
Looking for bias on purpose
AI bias is a predictable result of learning from an imperfect world. It can be reduced, but only when organizations look for it deliberately, measure it honestly and keep people accountable for the decisions their systems make.
Sources
- Incident 37: Amazon’s experimental hiring tool displayed gender bias, AI Incident Database (summarizing Reuters, October 2018)
- Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification, Proceedings of Machine Learning Research, 2018
- Dissecting racial bias in an algorithm used to manage the health of populations, Science, October 2019
- Automated Employment Decision Tools, NYC Department of Consumer and Worker Protection
- Enforcement of Local Law 144, Office of the New York State Comptroller, December 2025
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), EUR-Lex
Related reading: why AI decisions need explanations.
