This started with a toy. I built the smallest useful AI agent I could: a single language model, running on my own desk, given exactly one tool and exactly one rule. The tool was a calculator. The rule was a single line of instruction: don't do the arithmetic yourself, use the tool.
I gave it a sum. Here is the entire conversation:
User: 12 + 7
Model: CALCULATE: 12 + 7
Tool: 19
Model: 19The whole agent is just a loop: ask the model, look for a CALCULATE or ANSWER command in its reply, run the calculator if asked, and stop when it answers.
for step in range(MAX_STEPS):
reply = ask_model(messages)
calc_match = re.search(r"CALCULATE:\s*(.+)", reply)
answer_match = re.search(r"ANSWER:\s*(.+)", reply)
if answer_match:
break
elif calc_match:
result = run_calculator(calc_match.group(1).strip())(Full code: bare_react_loop_v6.py)
It worked. The model saw the sum, decided to call the calculator, wrote out the tool command, got the answer back, and reported it. Textbook. And it would have stayed a boring success story except for one small, nagging question I couldn't let go of: when the model wrote CALCULATE instead of just answering 19, a sum any model this size can do in its sleep, how sure was it, really? Was it weighing the instruction against its own confidence, the way a person might think "well, I know this one, but I was told to use the tool"? Or was it simply obeying, with no inner debate at all?
You cannot answer that from the transcript. The transcript only shows what the model did. To find out what it was thinking as it decided, I had to open it up. This is the story of what I found when I did, including the part where the most obvious answer turned out to be wrong, and the evidence I trusted most turned out to be lying to me.
The bug that made it interesting
Before any of the serious work, there was a bug, and the bug is worth a moment because it's the reason I stopped trusting the transcript in the first place.
An earlier version of my agent read the model's output with a naive bit of pattern-matching: scan the text, find the word CALCULATE, grab whatever comes after it, feed that to the calculator. Reasonable enough. Except this model, like most modern ones, thinks out loud before it acts. And in its thinking, it quoted my own instructions back at itself, something like "the rules say I should write CALCULATE: followed by the expression." My pattern-matcher pounced on that quoted example instead of the model's actual command, fed the calculator a fragment of the instructions, and got back a nonsense error. The model then had to sheepishly notice the error and work around it.
It was funny. It was also the whole lesson in miniature: what a model appears to be doing on the surface, and what is actually driving its behaviour underneath, are two different things, and the gap between them is exactly where the interesting stuff hides. I fixed the parser. Then I went looking underneath.
def strip_thinking(reply: str) -> str:
"""Drop everything up to and including a </think> block."""
return re.split(r"</think>", reply, maxsplit=1)[-1](Full code: bare_react_loop_v6.py)
Total, unwavering certainty
The first thing to measure was that nagging question about confidence. To do it, I need to introduce one idea, and I'll keep it plain.
When a language model picks its next word, it doesn't just pick, it scores every possible option. Those raw scores are called logits, and once you squash them into percentages that add up to 100%, they become probabilities. So at the exact instant my agent was deciding what to write, I could read off the odds it assigned to each choice. The two that mattered were the token that starts a tool call (CALCULATE) and the token that starts a direct answer (ANSWER).
I expected a lean, not a landslide. Something like 80/20, confident, but with a flicker of "I could just answer this." What I got was this:
Reading those odds is a few lines: run the model, turn its raw scores into probabilities, and pick out the two tokens that matter.
output = model(input_ids=input_ids, attention_mask=attention_mask)
probs = torch.softmax(output.logits[0, -1, :], dim=-1)
calc_p = probs[CALCULATE_ID].item()
answer_p = probs[ANSWER_ID].item()(Full code: bare_react_loop_v6.py)
P(CALCULATE) = 1.0000. P(ANSWER) = 0.0000. Not 99%. Not "overwhelmingly likely." As close to a mathematical certainty as the numbers can represent, and it stayed that way even when I fed it sums so trivial a child would roll their eyes. The instruction I'd written wasn't being treated as a strong preference. It was being treated as an absolute law. There was no inner debate to find, because from the model's point of view there was nothing to debate.
Which only sharpened the question. A decision that certain has to come from somewhere. Where inside the model does an instruction get turned into a law?
Where in the "brain" does the certainty live?
A model of this kind is built as a stack of layers, 36 of them, in this case, and information flows upward through them, getting refined at each step, until the top layer produces the final answer. If the decision is that absolute, I wanted to know which floor of the building it actually gets made on.
There's a lovely technique for this called the logit lens. The idea: instead of only reading the model's mind at the very top, you stop it at each layer along the way and ask, "if you had to commit right now, from this floor, what would you say?" Do that at all 36 layers and you get a trace of the decision forming, a play-by-play of the model making up its mind.
My first plot of that trace looked like a simple light switch: nothing, nothing, nothing, then CALCULATE snaps to certain near the very top.
The logit lens is exactly that idea in code: take each layer's hidden state and push it through the model's own output head, as if that layer had to speak now.
for layer_hidden in output.hidden_states:
last_token_hidden = layer_hidden[0, -1, :]
normed = model.model.norm(last_token_hidden)
layer_logits = model.lm_head(normed)(Full code: inspect_early_layers.py)
But a linear scale flattens everything small into an indistinguishable zero, and small early signals are exactly what I was after. So I replotted it on a logarithmic scale, which stretches out the tiny values so you can see their shape. And the switch turned into a story.
For the first thirty-two layers or so, there's only a faint, directionless murmur, the low-level hum of a model processing text, with no clear commitment either way. Then, around layer 33, something happens: for a brief moment both CALCULATE and ANSWER rise together, as if the model is genuinely entertaining both options at once. And then, in the final few layers, it resolves, hard, onto CALCULATE.
That was the first real surprise. The certainty I'd measured at the top isn't slowly accumulated across the whole network. It's manufactured late, in a short, specific window, after a brief flash of something that looks a lot like deliberation. Layer 33 was where the decision actually happened. So that's where I pointed the next instrument.
Eavesdropping on the argument
To see what layer 33 was doing, I turned to attention. In plain terms, attention is the mechanism a model uses to look back at earlier words while it works on the current one, it's how the model keeps track of context. At every layer, the model has a set of independent "attention heads," and each head can be thought of as a separate spotlight, each free to shine back on whatever earlier words it finds relevant. If I could see where layer 33's spotlights were pointing at the moment of decision, I'd know what the model was consulting to make its choice.
One quick, funny aside first, because it nearly threw me. A lot of the attention pooled onto the very first token of the input, a spot that carries almost no meaning. This is a well-known quirk; researchers call it an "attention sink." It's essentially the model's resting gaze, the place its spotlights idle when they've got nothing better to do. Once you know to ignore it, the real pattern comes through.
And the real pattern was striking. A handful of specific heads in layer 33 were reaching back, at the exact instant of the decision, to reread particular words in my instructions, the literal words CALCULATE and ANSWER. Not the sum. Not the surrounding chit-chat. The two words naming the choice itself.
Getting the attention weights out means loading the model with eager attention and asking it to return them, then pulling the layer we care about.
model = AutoModelForCausalLM.from_pretrained(
MODEL_PATH, dtype=torch.bfloat16, attn_implementation="eager"
)
output = model(input_ids=input_ids, attention_mask=attention_mask, output_attentions=True)
layer_attn = output.attentions[TARGET_LAYER][0] # [heads, seq, seq](Full code: inspect_layer33_attention.py)
This is the kind of moment that makes the whole exercise worth it. The decision wasn't some vague, smeared-out computation with no locatable cause. At the crucial layer, identifiable components were caught in the act, going back to the source text and reading the rules. But a single line of evidence, however satisfying, is just a single line of evidence. I wanted a second opinion, ideally from a method that had nothing to do with attention.
A second, independent witness
The second method was gradient attribution. The plain-language version: a gradient answers the question "if I nudged this input a little, how much would it change the model's decision?" Run that backward from the decision to every word in the prompt, and you get a ranking of which words the decision was most sensitive to, which ones, if you tweaked them, would most move the model's mind. It's a completely different route through the mathematics than attention, so if it pointed at the same words, that would be real corroboration rather than the same finding wearing a different hat.
Mechanically, that means differentiating the decision, the gap between the two options, with respect to the input words, and reading off how sensitive it is to each.
inputs_embeds = embed_layer(input_ids).detach().clone().requires_grad_(True)
output = model(inputs_embeds=inputs_embeds, attention_mask=attention_mask)
decision_score = last_logits[CALCULATE_ID] - last_logits[ANSWER_ID]
decision_score.backward()
saliency = (inputs_embeds.grad[0] * inputs_embeds[0]).sum(dim=-1)(Full code: inspect_gradient_attribution.py)
It pointed at the same words. Two witnesses, two unrelated techniques, one story: at the moment of decision, the model leans on the specific instruction words that name the choice. At this point I felt I had it more or less solved. I was wrong, and the next instrument is where the case cracked open.
The twist: the "most important" neuron wasn't
Attention is only half of what a layer does. The other half happens in a component called the MLP, think of it as the layer's bank of small computational units, its "neurons," each one able to activate in response to particular patterns and push the result one way or another. Layer 33 has thousands of them. If specific neurons were driving this decision, I wanted their names.
So I ranked them. For every neuron in layer 33's MLP, I measured how strongly its activity correlated with the CALCULATE decision, how reliably it lit up when the model chose the tool. One neuron towered over the rest. Call it neuron 941. By every indirect measure I had, it looked like the single most important unit in the whole decision.
Ranking the neurons means asking, for each one, how much its activity pushes the final scores toward the tool, its own direction times how strongly it fired.
neuron_direction = down_proj_weight[:, neuron_id]
per_unit_logits = model.lm_head(model.model.norm(neuron_direction))
contribution = intermediate_activation[neuron_id] * per_unit_logits(Full code: inspect_layer33_mlp.py)
The obvious next move is to declare victory and write up "neuron 941, the tool-use neuron." But there's a test you can run that goes beyond looking, and it's the single most important idea in this whole piece. It's called ablation: instead of merely observing that a neuron correlates with the outcome, you reach in and switch it off, physically silence it, then rerun the exact same decision and see what changes. Correlation tells you what travels together. Ablation tells you what actually causes. They are not the same question.
I silenced neuron 941, the crowned champion, the unit that looked more important than any other, and reran the decision.
Silencing a neuron is a hook that subtracts its contribution from the layer's output before the computation continues, so the rest of the model runs as if it never fired.
def zero_neuron_hook(module, inputs, output):
contribution = inputs[0][0, -1, neuron_idx] * down_proj_weight[:, neuron_idx]
patched = output.clone()
patched[0, -1, :] -= contribution
return patched(Full code: inspect_causal_patching.py)
Almost nothing changed. The model chose CALCULATE just the same, with essentially the same crushing certainty. The neuron that looked, by every correlational measure available, like the cause of the decision turned out to be a bystander. The evidence I'd trusted most had lied to me, not maliciously, but in the ordinary way correlation always can: neuron 941 was reliably present at the scene of the decision without being what made it happen.
Finding the real culprit
If the loudest neuron wasn't the cause, which one was? Ablating all of them one by one and rerunning is expensive, so I did it in two passes: first a cheaper estimate of each neuron's likely causal effect, then direct silence-and-rerun verification on the top candidates the estimate flagged. Estimate to narrow the field; verify to be sure.
The cheap estimate uses gradients to guess each neuron's causal effect, then only the top handful get the slow, exact silence-and-rerun treatment.
estimated_ablation_effect = -(intermediate_activation * intermediate_activation.grad)
top_estimated = torch.argsort(estimated_ablation_effect.abs(), descending=True)[:15](Full code: inspect_all_neurons_causal.py)
The real driver was a different unit entirely, neuron 36. It hadn't topped the correlational ranking; it just happened to be the one that, when actually removed, moved the decision. And here's the honest coda, the part a tidier story would leave out: even neuron 36 isn't the whole answer. Silencing it moves the needle more than anything else does, but it doesn't collapse the decision on its own. The choice is a shared effort spread across several units, and neuron 36 is the biggest single contributor to that effort, not a lone dictator. The truth was messier than "one neuron decides," and the mess is the point.
What it means
Strip away the layers and the neurons and here is what the investigation actually taught me, and it's a lesson that reaches well past this one toy agent:
Looking like the cause of something and being the cause of it are two different things. Neuron 941 looked like the cause by every available indirect measure, and it wasn't. The only way to tell the difference was to stop observing and intervene, to reach in, change one thing, and watch what happened. That's as true inside an artificial neural network as it is anywhere else in life, where the most conspicuous suspect and the actual culprit are so often not the same. Correlation points; only intervention proves.
And the original question, was my little agent weighing the instruction against its own confidence, or just obeying?, has a clear answer now. It obeys, utterly and without debate, and that obedience is assembled late in the network, in a brief window where the model rereads the words of the rule and commits. The calculator loop that opened this piece wasn't a model deciding it needed help with arithmetic. It was a model following a law, and I got to watch the law being enforced.
Where to go deeper
Everything here came out of a single, small, deliberately minimal experiment, and it left one big thread hanging. Is what I found a stable, reusable piece of machinery, a genuine "circuit" the model relies on whenever it faces this kind of choice, or was it a one-off, particular to this exact prompt? Answering that means running the same instruments across many different prompts and seeing whether the same layer, the same heads, the same handful of neurons keep showing up. That test is the natural next entry, and it's where this investigation goes from a single striking result to something you can actually rely on.
A fuller companion piece, with the complete code behind every chart above and the raw numbers in full, will follow this one. In the meantime, all the code and every chart from this investigation are public: github.com/pau1a/Kelstone. For now, the takeaway stands on its own: when you want to know why a model did something, don't settle for what looks responsible. Reach in and test it.