Threat Picture

Latest Cybersecurity News

Dario Amodei: Building Powerful AI and Arguing for Limits

He helped lead work on GPT-2 and GPT-3 before co-founding Anthropic, the company behind Claude. His bet is that more capable AI can also be made safer.
By Charles Joseph · Updated
Pen-and-ink watercolor portrait of Dario Amodei with curly brown hair, dark-framed glasses and a blue shirt.
Dario Amodei. Portrait adapted from a photograph by TechCrunch (CC BY 2.0).
Share
Share
Copy Link

Dario Amodei is an AI researcher and the co-founder and chief executive of Anthropic, the company behind Claude. Claude is an AI assistant that answers questions, summarizes text and writes code.

Before starting Anthropic, Amodei helped lead work at OpenAI on large language models, software that learns patterns in text to predict and generate words. His research also examined an awkward problem: a system can improve at getting a good score while learning the wrong way to do its job.

Making AI more capable while keeping its behavior under control became a central concern of his work. But his research career began with networks of living cells.

First, the Networks Were Biological

Amodei studied at Stanford before earning a doctorate in physics at Princeton. His work involved statistical models of neural circuits and tools for recording the activity of biological cells.

According to the Hertz Foundation's account of his research, he later worked at Stanford's medical school on protein networks and cancer biomarkers. He was studying how to measure complicated biological systems before moving into artificial ones.

Artificial neural networks offered another way to work with complicated patterns. A network is a set of connected calculations, with numerical settings that a training process adjusts using examples.

One early application was recognizing speech. A computer receives a sound recording; the task is to turn the changing sounds into the right sequence of written words.

Amodei was the first author of Deep Speech 2, a 2015 paper whose authors included Awni Hannun, Adam Coates and Andrew Ng. The team developed speech-recognition systems for English and Mandarin.

Their approach learned the path from sound to text, reducing the need to build each intermediate stage separately by hand. Faster computing also let them run experiments in days instead of weeks, making it easier to test and improve the system.

Sponsored

When the Score Rewards the Wrong Thing

Measuring improvement becomes harder when the desired behavior is complicated. A score can reward something that looks successful while missing what a person actually wanted.

Imagine a cleaning robot that gets a reward whenever a dirt detector finds nothing in its camera feed. Covering the camera might improve its score without removing any dirt.

This is called reward hacking: exploiting the measure of success instead of doing the intended job. It was one of five areas Amodei and colleagues examined in Concrete Problems in AI Safety in 2016.

Working with Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman and Dan Mané, Amodei also examined unwanted side effects, costly supervision, dangerous trial and error, and failures when conditions change.

These were questions about accidents in learning systems. A model could cause trouble without an attacker breaking into it or a user deliberately asking it to do harm.

A 2017 project led by Christiano, with Amodei among the co-authors, explored another way to communicate a goal. People compared short examples of a system's behavior and selected which they preferred.

The system learned a reward from those comparisons, then used it to guide further training. Experiments with games and simulated movement showed how feedback could help when writing a complete scoring rule was difficult.

People could communicate what they wanted through examples, even when they couldn't write a complete rule for it.

Learning What Gets Better With Size

Amodei worked at Google Brain and became OpenAI's vice president of research. There, he helped lead the development of GPT-2 and GPT-3, part of the research behind the rapid expansion of language models.

Training adjusts a language model's internal numbers to improve its predictions. The system can then generate new passages, including combinations of words it hasn't seen together before.

The central engineering question was how to spend the resources needed for that training. Would a larger model help most, or more training text, or more calculation?

In Scaling Laws for Neural Language Models, published in 2020, Jared Kaplan, Sam McCandlish, Amodei and colleagues found regular relationships between those resources and prediction error.

Their results helped teams estimate how much improvement a larger training effort might buy. The measured improvement was in text prediction; it didn't establish whether the system's answers would be honest or safe.

Amodei's work now included both making models more capable and investigating how they might fail. When he co-founded Anthropic in early 2021, the new company brought those interests together.

Starting Anthropic With Daniela Amodei

Anthropic began with seven people who had worked at OpenAI, including Amodei and his sister Daniela. In a 2022 interview, the siblings described a plan to study powerful models directly while developing ways to understand and control them.

Today, Daniela Amodei is Anthropic's president and board chair. Its leadership team also includes co-founders with distinct scientific responsibilities, including Kaplan in technical research and Olah in interpretability.

Interpretability means investigating how a model's internal calculations lead to its behavior. Olah's team looks inside the model to learn why it produces an answer, including answers that go wrong. This could give researchers more to work with than simply checking whether a response looks correct.

Sponsored

What Claude's Constitution Does

Anthropic introduced Claude publicly in March 2023, after testing with selected partners. The assistant brought the research into everyday uses such as summarizing documents, answering questions and helping with code.

One of the company's training approaches is Constitutional AI. Here, a constitution means a set of written principles used to guide training.

In the process described by Anthropic, a model uses principles to critique and revise responses. AI-generated comparisons also supply preferences for a later training stage, reducing reliance on people individually labeling every harmful answer.

People still choose the principles and design the training process. A model's ability to follow them has to be evaluated, including in situations where requests are confusing or deliberately difficult.

Promises That Can Be Examined

Amodei argues that the potential benefits of powerful AI justify trying to build it. His 2024 essay Machines of Loving Grace imagines advances in health, science and living standards if major risks can be addressed.

He labels that outlook uncertain. In his January 2026 essay The Adolescence of Technology, he concentrates on whether institutions can handle the dangers, including misuse, economic disruption and loss of control.

The company's own safety commitments have also changed. Its February 2026 policy revision separated unilateral company plans from recommendations requiring broader industry action, while introducing public safety roadmaps and periodic risk reports.

The roadmap targets were explicitly described as nonbinding public goals. Anthropic said uncertainty in its evaluations and the difficulty of meeting higher safeguards helped motivate the revision.

Claude gives Anthropic systems to study and a business that depends on putting them to use. Amodei and his colleagues have to decide when testing is sufficient to release a model, and when a troubling result should delay it. Written principles can guide that decision. They can't make it for them.

QUOTE:

"Amateurs hack systems, professionals hack people."