If a bank turns down your loan application, you can usually ask why and get an answer: income too low, too much existing debt, a missed payment. If the decision was made by a complex AI model, that answer is much harder to produce. The model may weigh hundreds of factors in ways even its builders cannot easily describe.
Key takeaways
- Explainable AI aims to make AI decisions understandable to the people who build, use and are affected by them, especially when stakes are high.
- Some models are simple enough to read directly, while complex models need after-the-fact tools such as feature importance and counterfactuals.
- Explanations are approximations: they can differ between methods, be gamed, and a chatbot's account of its own answer may not be faithful.
- If you face an AI decision, ask whether AI was involved, what the main factors were, whether a person reviewed it and how to appeal.
Explainable AI, often shortened to XAI, is the effort to make AI decisions understandable to the people who build, use and are affected by them. It matters most where decisions carry real consequences.
Why explanations matter
Someone denied a job interview, insurance or benefits has a strong interest in knowing why, and in being able to challenge a mistake. The professionals using the AI need explanations too. A doctor, loan officer or recruiter has to know when to trust a recommendation and when to question it.
Explanations help developers find bugs, such as a model leaning on an irrelevant feature that happened to line up with the right answer in the training data. And for regulators and courts, laws against discrimination and unfair treatment are hard to enforce if nobody can say how a decision was made.
A well-known example of why this matters comes from medical AI research, where a model that appeared accurate at detecting pneumonia from chest X-rays turned out to be relying partly on markers identifying which hospital, and which kind of scanner, produced the image, rather than only on signs of disease. Explanation tools help catch problems like that before they reach patients.
Two kinds of explainability
Models that are simple enough to understand
Some models are interpretable by design: a short set of rules, a decision tree, or a scoring system where each factor adds or subtracts a known number of points. You can read the model and see exactly why it made each decision. Some researchers argue that for high-stakes decisions, these models should be preferred whenever they perform nearly as well as complex ones.
Explaining complex models after the fact
For deep neural networks and other complex models, researchers use techniques that approximate an explanation. Feature importance methods such as SHAP and LIME estimate how much each input pushed the result up or down. Counterfactuals ask what the smallest change would have been that flipped the outcome: “If your income had been $4,000 higher, the loan would have been approved.” These are often the most useful explanations for the people affected.
For images, saliency maps highlight the parts of the picture that most influenced the model. Another approach is to show similar past cases and how they were decided.
For large language models, a research field called mechanistic interpretability tries to understand what individual parts of the network represent, which we touch on in our explainer on AI alignment.
The limits of explanations
Explanations can create false confidence if they’re treated as the full truth. Post-hoc methods estimate what a model did; they don’t reveal its full reasoning, and different methods can give different answers. They can also be gamed. Research has shown it’s possible to build models whose explanations look reasonable while the underlying decisions rely on problematic factors.
When a chatbot explains its own answer, the explanation is generated text, not a readout of its internal process, and studies have found such explanations can leave out the factors that drove the answer. Detail has limits too. An explanation listing 200 weighted factors is technically transparent and practically useless.
A good explanation for the person affected answers three questions: What were the main reasons? What could I change? How do I challenge the decision if it is wrong?
What the law says
Rules vary by country, but several already touch on explanations:
- In the EU, the GDPR gives people rights around decisions based solely on automated processing that significantly affect them, including meaningful information about the logic involved. The EU AI Act adds transparency and human oversight requirements for high-risk AI systems, and a right to an explanation of certain decisions.
- In the US, lenders must give applicants the specific principal reasons for adverse credit decisions under the Equal Credit Opportunity Act and its Regulation B, and that duty does not disappear because a complex model made the call. The Consumer Financial Protection Bureau spelled this out in a 2022 circular, which it withdrew in 2025 along with dozens of other guidance documents; the requirement itself comes from the statute and still applies.
Questions to ask about any AI decision
- Was AI involved in this decision, and how much?
- What were the main factors?
- Did a person review it?
- How can I correct wrong information or appeal?
If an organization cannot answer these, that tells you something about how carefully it is using AI. Our explainer on bias in AI systems covers the related problem of unfair outcomes.
Frequently asked questions
What is explainable AI?
Explainable AI, or XAI, is the effort to make AI decisions understandable to the people who build, use and are affected by them. It matters most where decisions have real consequences, such as loans, jobs, insurance and healthcare.
What is a counterfactual explanation?
It describes the smallest change that would have changed the outcome, for example that a loan would have been approved if income had been $4,000 higher. These are often the most useful explanations for the people affected.
Can a chatbot explain how it reached its answer?
Not reliably. A chatbot's explanation of its own answer is generated text, not a readout of its internal process, and studies have found such explanations can leave out the factors that actually drove the answer.
Do I have a right to know why an AI turned me down for a loan?
In the US, lenders must give the specific principal reasons for adverse credit decisions under the Equal Credit Opportunity Act, even when a complex model made the call. In the EU, the GDPR and the AI Act give people rights to meaningful information about certain automated decisions.
Sources
- Four Principles of Explainable Artificial Intelligence, NIST, 2021
- Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead, Cynthia Rudin, 2019
- Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs, Zech et al., PLOS Medicine, 2018
- 12 CFR 1002.9: Notifications (Regulation B), Electronic Code of Federal Regulations
- Interpretive Rules, Policy Statements, and Advisory Opinions; Withdrawal, Consumer Financial Protection Bureau (Federal Register), 2025
- Language Models Don’t Always Say What They Think, Turpin et al., 2023
