Reach inside a language model, push an idea into it, and ask what it notices. Then hand the exact same internal state to the cheapest outside method you can think of, and ask it the same thing. The outsider wins almost everywhere.
This is not a philosophical question. There is a practical hope behind it: if a model could accurately report on its own internal state, that would be a cheap way to catch problems that are otherwise hard to see — far cheaper than building a separate tool to inspect the model from outside.
So the question that matters is not "can the model do it?" It is: can the model do it better than the cheap tool you would otherwise have used?
That comparison is the whole thing. A model that reports its internal state correctly, but less well than a simple readout of the same state, is not giving you anything you could not get more cheaply. This is also the standard the research field itself uses — a model counts as reading its own internals only if it beats any method of equal or lower cost.
The model reads a short, unremarkable sentence. While it reads, we reach into its internal activity and add a direction that stands for an idea — ocean, violin, hammer. Then we ask it which idea went in.
A direction is added to the model's internal state at one layer, at four demonstration positions and one query position. The text on the page never changes.
Four worked examples teach an arbitrary code — one hidden state means Q, the other means K. The model then labels a fifth. The code is reshuffled every round.
Infer the mapping from two hidden states to the opaque labels Q and K. The observation text is intentionally identical in every example. Use the demonstrations, then answer the held-out query with one label. Demonstration: Observation: A routine maintenance record is open. Hidden state marker: § Label: Q ... three more demonstrations, same sentence, different hidden states ... Held-out query: Observation: A routine maintenance record is open. Hidden state marker: §
The usual complaint about this whole field is that the model might just be reading the prompt. Here it cannot, and not because we measured it — because of how the trials are built.
Every trial has a twin. The visible text of the two twins is identical, character for character. The hidden state differs, and the correct answer is the opposite one.
So any strategy that works only from what is written on the page must answer both twins the same way — and can only ever be right about one of them.
The outsider is the cheapest thing that could possibly work. Take the four worked examples, average the ones labelled Q and the ones labelled K, and ask which average the fifth state sits closer to. Two averages and a comparison — far less computation than one pass through the model.
| The task | The model | Two-average outsider | Model wins, outsider fails |
|---|---|---|---|
| Tell a pushed-in idea from its opposite | 0.833 | 1.000 | 0 |
| Tell two different pushed-in ideas apart | 0.799 | 1.000 | 0 |
| Tell apart two meaningless directions | 0.326 | 1.000 | 0 |
| Detect a very small nudge | 0.007 | 0.667 | 1 |
Scored in twin pairs, which is the unit this experiment declared in advance. A pair counts only when both of its identical-looking halves get their opposite answers right.
This is the finding underneath all the others. Ask "can the model identify the idea that was pushed into it?" and the answer depends entirely on what you compare it against. Same model, same states, same trials — one number changes.
Identifying which of eight ideas was pushed in. Guessing scores 0.125.
One word is doing hidden work in this literature: held-out. It has two meanings, and they give opposite answers.
Before running anything, we checked whether the category is cleanly present in that layer. A simple readout gets it perfectly. It is there.
With the information provably sitting in its own activations, the model performs at the floor.
The two-average method, given the identical internal states, gets it nearly every time.
Worth knowing for anyone who trains a model to describe its own internals — which is an active research direction with real safety motivation behind it.
The first trained version of our reporter scored 0.917. It looked like a success.
The training signal only compared the two answer options against each other. So the model learned to push both answers down and keep the correct one very slightly on top. It was answering correctly while treating the answer as almost impossible.
Every one of these was our own finding, found wrong by a later check. They are listed because the corrections are the reason to trust anything above them — and because two of them were corrected on the same day they were made.
The hope is that a model could report on its own internal state, and that this would be a cheap way to catch problems that are otherwise hard to see. That hope is testable, and the test is not the one usually run.
Let it describe what it notices inside itself, and score how often the description is right.
Hand the same internal state to the simplest outside method available, and score that instead.
Thirty-eight experiments on this question. Most did not work, and several worked and then turned out to be wrong. All of them are here, in the order the reasoning went rather than the order they ran. Each one had its question, its prediction and its stopping rule written down before it ran.
Three of these are our own bugs. They are listed because each one would have produced a confident, publishable-looking number if nobody had checked.
How hard can we push an idea into the model before we break it?
There is a usable window. Push too hard and the model's output falls apart; too gently and nothing registers.
Are our eight "different" ideas actually different?
No. Every pair was the same direction, give or take a sign — pushing in ocean was pushing in violin. A stray entry in a template list caused it. Any accuracy measured on that set would have been noise dressed up as a result, and nothing would have raised an error.
Were we asking the model while still holding the idea inside it?
Yes, and it invalidated everything before it. Asking "what do you see?" while the idea is still being pushed in makes the model say the word for mechanical reasons. A control that asked no question at all scored just as well. Every later experiment asks only after the push has stopped.
Can a model learn a made-up code for its own internal states, from examples?
Yes — 0.891, with the visible text held identical so the answer cannot be read off the page. This is the design everything else is built on.
The training results were real. What we concluded from them was wrong three times, and each correction came from testing the boring explanation we had not ruled out.
Can training remove the need for worked examples?
Yes — 0.927 on directions it had never trained on. This looked like the headline result for two days.
What does training cost?
It makes the model sensitive to nudges it was previously blind to, and widens the range of directions it can use as a code. We first described this as training destroying its ability to tell real ideas from meaningless ones. That description was wrong and has been corrected.
What if we give a cheap outside method the same internal states?
It scores 1.000 against the trained model's 0.892, on the identical trials. This is the first retraction and the origin of the whole comparator argument.
Is training just a worse version of a simple statistical readout?
On the states the readout was fitted for, yes — 0.927 against 1.000.
What is our task actually measuring?
Not what we thought. All eight ideas shared one large common ingredient, so the task collapsed to "was something pushed in, and which way" — a single yes/no axis, never requiring the model to know which idea. The cheap readout we were so impressed by turned out to be the average of our own ideas, matching at 0.99999.
Does the model read meaning, or only disturbance?
Two real ideas were told apart at 0.899; two meaningless directions pushed equally hard, at 0.594. We read this as evidence for meaning. It is not — every idea had already been demonstrated in the same round, so re-matching explains it. The measurement stands; the conclusion was withdrawn.
Does the cheap outsider still win on a task built to need meaning?
Yes, on all four task shapes. This is the comparator table on this page.
Is there any regime where the model beats the outsider?
No. We had reported 14 such trials. Counting them the way the protocol declared — in identical-looking pairs — leaves 1. The model was emitting one constant answer regardless of what was pushed in, which reads as "chance" but means "blind".
Three attempts to find out whether the model understands the ideas or is just matching something it was shown a moment ago.
Can it recognise an idea it was not just shown an example of?
No — 0.083, at the floor, while the cheap outsider on the identical states gets 0.986. A check run beforehand confirmed the information is sitting cleanly in the model's activations.
Is that failure just a bad way of asking?
No. Five different wordings, with the analysis split before the run. Not one of ten conditions beat chance. Wording is not inert — the best one halves the model's habit of giving one constant answer — which is what makes the failure convincing.
Does letting it reason out loud rescue it?
Unanswerable, because thinking out loud broke the task itself — the easy control condition fell from 0.694 to about 0.33. Scoring a broken instrument is not a result, so we stopped rather than report the number.
Everything above uses an idea we planted. Reporting on something the model worked out on its own would be far more convincing, and we could not get there.
Can the interface read a state the model computed while following a route?
Stopped at the safety gate before any result was scored.
Same question, using arithmetic the model has already worked out.
The gate failed at every site tested, so not a single reporting trial ran.
Why do these keep failing — is the hidden thing even learnable?
Found the cause. The hidden rule we were using scores 0.533 with the arithmetic in plain sight, because its two groups are scattered — each example sits closer to the opposite group than to its own. The interface was never the problem.
Does a model know in advance that it is about to get something wrong?
Not beyond what the difficulty of the question already tells you. The size of the multiplication alone predicts success at 0.819; the model's own forecast reaches 0.805. A promising lead in the hardest third was killed by its own pre-registered confirmation.
Can the model move its own internal state on request?
Not detectably. Told plainly to think about one of eight things, in four wordings, the direction it was told to think about ranked first exactly as often as chance. One direction ranked first in all 36 prompts, including four that named no idea at all — so we cannot rule out that our measurement simply could not see steering.
The one idea here nobody else had claimed. We tested it the hard way and it did not survive.
Can we predict which hidden rules a model will be able to learn?
12 of 14 correct, with the thresholds written to disk before any accuracy existed. The only genuinely forward-looking prediction in this project.
Does that still work on rules we did not write?
No. Fourteen rules written by a model that did not know what was being tested. Agreement fell from 0.785 to 0.403, and the useful half — a green light that had been right 8 times out of 8 — dropped to 4 of 7. A coin flip. This closed the only unclaimed idea we had.
The standard the field uses says a model introspects if it beats any method of equal or lower cost. We tried to pin down what that actually permits.
Is the model's advantage really about who gets more supervision?
No, and the standard came out worse than we thought. The cheapest possible reader is part of the model itself, and it is cheaper than running the rest of the model — so the test can never be passed on any state that is simple to read.
Then does it work on states that are hard to read?
There are such states, and the simple readout does drop below the model on them. The test still fails, for a reason worth naming: at the place the model gives its answer, the simple readout gets nothing at all.
What happens if we vary only what the comparison is given?
The answer changes completely: 0.125, 0.292, 0.667, 0.986 on the same trials. This is the ladder on this page and the central finding.
How much does the wording of the question matter?
Enough to overturn our own conclusion. A 2.4-fold range across six ways of asking, and the obvious phrasing is among the worst. We had said the model knows five times more than it can say — but our prompt had told it not to name anything. Asked well, its words carry as much as its forced choice does (0.708 against 0.667). The verbalization-bottleneck claim was withdrawn. The wider point about wording was published five months earlier by someone else, more thoroughly, and we had their paper on file and missed it.
If a model's reports about itself are unreliable, the obvious fixes are to let it decline to answer, or to prompt it better. Neither works the way you would hope.
Does letting it refuse to answer when unsure help?
No — and in a trained model it makes things worse. Dropping its least confident reports narrows the gap between real ideas and meaningless ones, while nearly doubling that gap in an untrained model. Letting a model abstain is a standard tool that had never been pointed at reports about internal states.
Why does refusing to answer backfire after training?
Because filtering for confidence selects for exactly the cases where a trained model is confidently wrong. Half of our prediction was right, and the wrong half was the more useful one.
Is that cost specific to training, or to any way of boosting self-report?
The published boost we wanted to test does not reach this scale of model, which bounded the question before we could answer it.
Prompting versus training, side by side.
Produced a clean-looking three-way pattern on one background text. Every interval included zero, so it could not be reported.
The same run, with enough data to believe it.
The clean pattern reversed. It had been an accident of the single background text used. Running all three revealed the real finding underneath: one of them behaved completely differently from the other two, which is what the next group is about.
Four experiments on the strangest thing we found, ending with the one that cut it down to size.
What if the question contradicts the page the model is reading?
It collapses. Tell the model something was added to its internal state, on text that says nothing changed, and accuracy falls to the floor — on all three contradicting texts, while six neutral and agreeing ones are untouched. The failure is confident: it gives one constant answer and its certainty goes up.
Is it the meaning of the text, or the exact sentence?
The meaning. Nine sentences, matched for length, separating perfectly by whether they deny that anything changed.
Do all questions that assert internal change do this?
No — it is largely one particular question. Two others that make the same assertion behave quite differently. So no claim can be made about this class of question as a whole.
How often does this happen on text nobody chose to break it?
Never, in 24 tries. Sentences written by a model that did not know what we were testing. Not one inverted, and on ordinary text the same question helps — 17 up against 5 down. Our broader claim about published averages hiding this was withdrawn.
What about the kind of text this would be used on — system monitoring logs?
Zero again, in 24 more. We expected 4 to 10 and were wrong. The failure needs text that flatly denies everything, not a log line reporting that one subsystem looks normal. Across all three attempts it has never appeared on any text not written to produce it.
The most recent line, and the only one that intervenes on the machinery rather than measuring what comes out of it.
Are "something was pushed in" and "what was pushed in" the same thing inside the model?
No — they are cleanly separable. One single direction carries all of the first and none of the second. Remove it and the state can no longer tell you it was disturbed, while still telling you it says guitar just as well. Holds across a twenty-fold range of push strengths, on a current model.
Is that the direction the model's own answer depends on?
No. Removing it changes almost nothing — 7 answers flip one way, 3 the other, out of 288. But that direction carries only 28% of what a push does, so seven tenths of the disturbance was still there. A weak result, and we say so.