Nicholas Carlini Finds the Cracks in AI
Nicholas Carlini is a computer-security researcher who studies how AI systems can be attacked and what they reveal about the data they learned from. His work has exposed ways to fool image-recognition systems and recover information from models trained to generate text.
One of those models was GPT-2, which learns patterns from text and uses them to predict what comes next. In a study published in 2021, Carlini and his collaborators recovered hundreds of verbatim passages, including chat conversations, code and personal details from the public web material used to train it.
Those results exposed a privacy problem within the learning process. Years earlier, as a student at Berkeley, Carlini had found a very different security problem in the university's own software.
The Puzzle Beneath the Promise
Carlini studied computer science and mathematics at the University of California, Berkeley, earning his bachelor's degree in 2013. He stayed for a doctorate under security researcher David Wagner, completing it in 2018.
In his essay “Why I Attack,” Carlini recalled discovering a serious flaw in Berkeley's Telebears enrollment system as a freshman. It could let someone change another student's enrollment.
He wrote that he kept the details private for years because students depended on the service and couldn't protect themselves by choosing another system. Later, he reported web flaws to developers and worked with Wagner on Chrome-extension security.
Carlini describes the appeal of this work partly as the pleasure of solving puzzles. His personal projects show a similar taste for unusual constraints. He has built a chess engine out of regular expressions, rules normally used to find patterns in text.
A security puzzle comes with other people attached. With Telebears, finding the flaw meant deciding how to handle information that could hurt students if it became public.
When a Defense Passes the Wrong Test
Carlini's doctoral research focused on neural networks, systems that learn through layers of calculations. Training adjusts their numerical values to improve at a task, such as assigning labels to images.
A good score on ordinary pictures can hide a weakness. Carefully chosen changes may leave an image looking much the same to a person while causing the model to give a wrong answer with confidence. Researchers call these altered inputs adversarial examples.
In work first released in 2016, Carlini and Wagner developed attacks that defeated a proposed defense called defensive distillation in their tests. They used the model's calculations to search for damaging changes.
Stopping one attack could make a defense look reassuring. But a tester who understood the defense could adapt the attack and find a route the first method missed.
Carlini, Anish Athalye and Wagner examined that problem further in their 2018 paper, “Obfuscated Gradients Give a False Sense of Security.”
A gradient is a mathematical signal showing how a small change affects a result. Some defenses disrupted the signals an attack relied on, making the attack appear to fail. When the researchers changed their methods to account for that disruption, they could get around several of the tested defenses.
The weakness was still there. The original test had lost its way of finding it.
Carlini later applied the same critical eye to his own work. In a 2022 retrospective, he described the Carlini–Wagner attack as too complicated and slow. He also acknowledged that the paper had missed earlier research on attacks against machine learning.
He valued the careful comparisons and the approach to searching for failures. He was quite willing to criticize the method that carried his name.
What a Model Can Remember
After his doctorate, Carlini joined Google Brain in 2018. His work there increasingly examined what trained models could reveal about the information they had absorbed.
Learning patterns in text can also leave a model able to reproduce particular passages. Training-data extraction means making those passages reappear and verifying where they came from.
For the GPT-2 study, Carlini worked with collaborators including Florian Tramèr and Eric Wallace. They generated candidate passages, then investigated which matched training material. Recovering a confirmed example required both getting the model to produce it and checking the match.
This wasn't a search engine for the entire training collection. The researchers were finding particular records that a text-generating model could repeat. Some contained personal information, despite having come from publicly available material.
The result raised a concern for developers using sensitive records to train models: memorization could provide another route for information to escape. Protecting the original files would address only part of the problem if a trained model could reproduce their contents.
Pictures Can Reappear Too
Carlini and colleagues including Jamie Hayes and Milad Nasr later tested diffusion models, image generators that learn to turn noise into pictures. Their 2023 study recovered more than a thousand closely matching training examples across the systems they examined, including images of people and logos.
They generated images and filtered the results to find matches. The work extended the memorization question beyond text to a different kind of learning system.
In a 2025 explanation of his privacy research, Carlini described these studies as establishing lower bounds: confirmed examples of what researchers could extract under deliberately challenging tests.
They weren't a tally of everything a model had memorized or a measure of how often an ordinary user's output would repeat training material. Checking possible matches took substantial work. Each confirmed match supplied evidence of a failure, while its frequency remained a separate question.
Testing From Inside the Lab
Carlini worked at Google DeepMind from 2023 to 2025, then moved to Anthropic. In his March 2025 announcement, he said he wanted more freedom to publish security and privacy findings, along with an opportunity to affect powerful models before their release.
At Anthropic, he has continued testing defenses with colleagues. He was a co-author of the team's November 2026 research on Anvil, an internal system that automates attacks to test model safeguards.
He has also explored what AI can help people build. In a February 2026 account of a programming experiment, Carlini described using 16 AI assistants to build a C compiler, software that translates C programs into instructions a computer can run.
His account emphasized human-designed tests and coordination. Feedback exposed mistakes and caught changes that broke previously working code. The assistants could keep producing code; the experiment depended on the tests being able to tell when it was wrong.
Ian Goodfellow: The AI Breakthrough Built on a Contest

Data Leakage: How Private Information Escapes Without Disappearing

A Threat Model Follows the Request Past the Login Screen

QUOTE:
"Amateurs hack systems, professionals hack people."