Bias in AI Systems: Where It Comes From and What Can Be Done

AI systems increasingly help make decisions about people: who gets a loan, which job applicants are shortlisted, what content people see and how medical cases are prioritized. When those systems are biased, they can disadvantage people at scale, often invisibly. Bias in AI is rarely the result of anyone’s intent. It usually comes from the data and choices behind the system.

Updated September 2026: we added links to the original studies and reporting behind each example, and to the laws and audits mentioned.

Where bias comes from

1. Historical data. Models learn from the past. If past decisions were unfair, a model trained to imitate them will reproduce that unfairness. In 2018 Reuters reported that Amazon had abandoned an experimental recruiting tool after finding it penalized résumés containing the word “women’s” and downgraded graduates of all-women’s colleges, reflecting a male-dominated hiring history (AI Incident Database summary). Amazon said recruiters never relied solely on the tool. We discuss hiring in our job hunting article.

2. Unrepresentative data. If some groups are underrepresented in training data, the system may work worse for them. The Gender Shades study by Joy Buolamwini and Timnit Gebru found that commercial gender classification systems misclassified darker-skinned women up to 34.7% of the time, against a maximum of 0.8% for lighter-skinned men. Similar concerns exist in medical AI, as noted in our healthcare analysis.

3. Proxy variables. Even when sensitive attributes such as race are excluded, other data, like postal codes or shopping patterns, can act as proxies.

4. Design choices. What the system optimizes for matters. A 2019 study in Science found that a widely used US healthcare algorithm predicted future healthcare costs rather than illness. Because less money had historically been spent on Black patients’ care, “at a given risk score, Black patients are considerably sicker than White patients.” The researchers estimated that fixing the problem would raise the share of Black patients flagged for extra care from 17.7% to 46.5% (abstract).

5. Language and cultural bias. Large language models absorb stereotypes present in text from the internet and can reflect them in generated content, though developers work to reduce this.

How bias is measured

There is no single definition of fairness, and some definitions mathematically conflict with each other. Common approaches compare outcomes between groups:

• Do different groups receive positive decisions at similar rates?
• Are error rates, such as false rejections, similar across groups?
• Does the system perform equally well for everyone it serves?

Choosing the right measure depends on context and values, which is why fairness is not purely a technical problem.

What organizations can do

1. Examine training data for gaps and historical patterns.
2. Test performance across groups before and after deployment, as the Gender Shades researchers did.
3. Check what you are really predicting. The healthcare case shows how a reasonable-looking target, cost, can stand in for something else.
4. Keep humans responsible for high-stakes decisions, with the ability to override.
5. Provide explanations and appeal routes to people affected.
6. Monitor continuously, because data and populations change.

What regulators are doing

The EU AI Act requires high-risk systems, including many used in hiring, credit and public services, to meet data governance standards aimed at detecting and reducing bias (Article 10). After 2026 amendments, those obligations apply from December 2027. In the US, existing anti-discrimination laws apply to automated decisions.

New York City’s Local Law 144 has, since July 2023, required employers using automated hiring tools to commission a bias audit within the previous year, publish a summary of the results and notify candidates. Enforcement has been weak: a December 2025 audit by the New York State Comptroller found that the city’s review of 32 companies turned up one case of non-compliance, while auditors identified at least 17 potential violations in the same sample. State rules elsewhere are changing quickly, so check the current position where you operate.

Looking for bias on purpose

AI bias is a predictable result of learning from an imperfect world. It can be reduced, but only when organizations look for it deliberately, measure it honestly and keep people accountable for the decisions their systems make.

Sources

1. Incident 37: Amazon’s experimental hiring tool displayed gender bias, AI Incident Database (summarizing Reuters, October 2018)
2. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification, Proceedings of Machine Learning Research, 2018
3. Dissecting racial bias in an algorithm used to manage the health of populations, Science, October 2019
4. Automated Employment Decision Tools, NYC Department of Consumer and Worker Protection
5. Enforcement of Local Law 144, Office of the New York State Comptroller, December 2025
6. Regulation (EU) 2024/1689 (Artificial Intelligence Act), EUR-Lex