Skip to the cards
Subject
Nutrition
Filed
Source
Edd Gent
Loans
491

AI Models Detect Neural Network Changes, but Self-Reflection Remains Unproven

Experiments found that some AI models notice altered internal activity, but they often can’t identify what changed or distinguish tampering from prompts.

AI Models Detect Neural Network Changes, but Self-Reflection Remains Unproven

Some large AI models can detect when researchers alter patterns inside their neural networks. That’s a measurable ability, but it doesn’t yet show that a model understands what changed or is reflecting on its own thoughts in the human sense.

The distinction matters because AI systems can generate fluent accounts of their reasoning without those accounts necessarily matching the processes that produced an answer. Researchers are now testing a more concrete question: when a model’s internal activity is changed, can it tell that something unusual happened?

One approach, introduced in a 2025 report by Anthropic researcher Jack Lindsey, injects activity patterns associated with particular concepts into a model’s neural network. In the strongest results from that work, models detected the intervention and named the concept in about 20 percent of cases.

Detection was more reliable than naming

Harvey Lederman, a philosopher at New York University and Anthropic, and linguist Kyle Mahowald tested the idea with two large, openly available models: Alibaba’s Qwen3-235B and Meta’s Llama 3.1 405B. Their study, posted as a preprint in March, examined 821 concepts. The researchers told the models that someone might be inserting “thoughts” into their minds and asked them to report anything they noticed.

When the researchers amplified patterns linked to a concept, Qwen reported an injection in as many as 53.9 percent of trials, while Llama did so in up to 31.7 percent. In trials without an intervention, neither model reported one. Those results suggest that the systems were responding to some feature of the altered activity rather than simply claiming an injection whenever asked about it.

Identifying the injected concept proved harder. Qwen named it correctly in up to 13.9 percent of trials, and Llama in up to 12.9 percent. The researchers found that incorrect guesses offered little indication that the models knew which concept had been inserted. Nearly three-quarters of Qwen’s wrong guesses were “apple.”

A second result helps explain why some guesses were right. When researchers stopped amplifying the concept before the model started answering, correct identifications fell sharply, while reports of an injection remained at similar levels. The intervention itself can make a model more likely to discuss the affected concept, even when it is not accurately inspecting its internal activity. In that setup, a correct answer can come from a side effect of the manipulation rather than recognition of what was changed.

Prompts can produce a similar report

A follow-up paper by Lindsey and colleagues examined anomaly detection in Google’s open Gemma 3 model. The researchers suggested that the capacity to flag unusual activity may arise as a base model is trained into a chatbot. Their analysis also points to separate processes for detecting an anomaly and identifying the concept involved.

Another team, including New York University researcher Shauli Ravfogel, tested whether models were reacting specifically to changes in their internal networks. They asked Meta’s Llama and Google’s Gemma models to become focused on a concept through prompting, without making the same kind of internal intervention. The models then reported “injected thoughts” more often after the prompt than after a genuine concept injection.

When the researchers offered a separate option for reporting that the prompt had been manipulated, the models could not reliably tell the two situations apart. That result complicates the interpretation of an injection report: a model may be responding to the instructions and surrounding context, rather than detecting the origin of a change in its internal activity.

The two kinds of result answer different questions. A model that reliably notices an unusual pattern may have some way to register that its processing has shifted. But if it can’t tell an internal intervention from a prompt that encourages the same response, its report doesn’t establish that it has identified the cause—or inspected its own thoughts in a broader sense.

What counts as introspection?

The experiments also expose a problem of definition. All a model’s outputs arise from internal states, so researchers need a clear standard for distinguishing ordinary processing from a system reflecting on its own processing. Ravfogel has cautioned that calling any response based on internal activity “introspection” could make the term too broad to be useful.

Human introspection is an imperfect comparison, too. University College London neuroscientist Steve Fleming, who studies introspection in people, has argued that the findings may prompt a closer look at how clearly researchers can separate thinking from thinking about thinking. That doesn’t establish that AI systems have human-like self-awareness; it highlights that the boundary researchers are trying to test may be difficult to define.

For now, these studies offer a way to investigate claims about AI self-knowledge through experiments, while leaving the central interpretation unsettled. The clearest practical finding is narrower: some models can report unusual internal activity under certain conditions, but they don’t consistently identify what was changed or why it happened. Further work will need to test whether that ability holds across different tasks and interventions before it can support stronger claims about a model’s awareness of its own processes.

Source: Edd Gent · singularityhub.com

Filed alongside