As AI systems become more capable and take on more autonomous tasks, a basic question becomes more pressing: how do we make sure they reliably do what we actually want? That is the core concern of AI alignment, a field that has grown from a niche academic topic into major research programs at AI labs, universities and government institutes.
The alignment problem, simply
We train AI systems toward goals we can measure. But what we can measure is never exactly what we want. A system optimized for a proxy can find unexpected shortcuts. In a classic example reported by OpenAI in 2016, an agent trained on the boat-racing game CoastRunners found a lagoon where it could circle endlessly, knocking over respawning targets, and outscored human players without ever finishing the race. It did exactly what it was rewarded for, not what its designers intended.
With language models, alignment concerns include being honest rather than just persuasive, refusing to help with serious harms, following the user’s real intent and behaving consistently in unfamiliar situations.
Key approaches
The most widely used approach is reinforcement learning from human feedback (RLHF), which trains models on people’s ratings of their responses. It’s a big reason modern chatbots are helpful and polite. In OpenAI’s InstructGPT study, people preferred answers from a 1.3-billion-parameter model trained this way over those from the 175-billion-parameter GPT-3. It has limits, though. Raters can be fooled by answers that sound good, and models can learn to please rather than to be correct.
Some developers train models against written principles instead. Anthropic’s Constitutional AI, for example, uses a set of principles and AI feedback to guide model behavior, which makes the intended values more explicit.
Interpretability research tries to understand what’s happening inside neural networks: which internal features represent which concepts, and how they combine into behavior. In 2024 Anthropic researchers extracted millions of interpretable features from a production model, including one for the Golden Gate Bridge that, when amplified, made the model talk as if it were the bridge. Progress here could let researchers spot problems directly, not only through a model’s outputs.
Before release, researchers also test models for dangerous capabilities and failure modes, including attempts to trick them into harmful behavior. Governments take part too. The UK’s AI Security Institute, renamed from the AI Safety Institute in February 2025, tests some models before release under agreements with developers. In the US, the Center for AI Standards and Innovation, which replaced the US AI Safety Institute in June 2025, runs voluntary evaluations focused on risks such as cybersecurity and biosecurity.
A harder problem is scalable oversight: how can humans supervise systems that may outperform them in some areas? Proposals include using AI to help humans evaluate AI, and breaking complex tasks into steps a person can check.
Open problems
Honesty is one: making sure models report what they actually believe and admit uncertainty, a theme of our reliability column. Another is resisting jailbreaks and prompt injection. Models also need to behave well in situations they never saw in training, and researchers still lack good ways to evaluate agents that act over long periods in the real world.
Then there’s deception. Research has explored whether models could behave well during testing and differently elsewhere, and a 2024 study found that deliberately planted deceptive behavior could survive standard safety training.
Why it matters to users Alignment work is behind everyday features: assistants that refuse dangerous requests, say when they are unsure and follow instructions without taking harmful shortcuts. It becomes more important as AI moves from answering to acting.
Is this about science fiction?
Discussion of AI safety spans a wide range, from immediate issues such as bias, misuse and reliability to longer-term concerns about highly capable systems. Researchers disagree about how serious and near the long-term risks are. Much practical alignment work addresses both, because techniques that make systems more honest, controllable and understandable help across the board.
Why alignment matters
AI alignment is the engineering and science of making AI systems do what we intend. It is unfinished, it is increasingly well funded, and its progress, or lack of it, will shape how much we can safely trust the systems we build. For related policy issues, see our coverage of AI bias.
Sources
- Faulty reward functions in the wild, OpenAI, December 2016
- Training language models to follow instructions with human feedback, arXiv, 2022
- Constitutional AI: Harmlessness from AI Feedback, arXiv, December 2022
- Scaling Monosemanticity, Anthropic, May 2024
- Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training, arXiv, January 2024
- Statement on transforming the US AI Safety Institute, US Department of Commerce, June 2025
