When an agent misbehaves, the easy fix is a line in the prompt. It’s tempting because it’s easy, and the next run usually looks fixed.
But present LLMs are quite good at working around things. If the prompt is clean, the wrong behaviour is probably coming from somewhere else, and the patch only hides the symptom.
Worse, the patch doesn’t stay where you put it. It adds a step the model has to think about on every request, even the ones that never needed it, and the agent turns over-cautious in places that were fine before.
A prompt patch after one bad run is a bet twice over: that the failure will happen again, and that the prompt is where it came from. Both bets usually lose.
One failure is a sample
With deterministic code, a failing test is a fact. Run it again and it fails again. You fix the code, the test passes, and that’s the end of it.
An agent doesn’t work like that. Run the same prompt twice and you get two different programs, not two phrasings of the same one. One run adds a rate limit and the other doesn’t. Both look reasonable, and both pass a quick read.
So a test asserts, but an eval estimates. One run is one sample from a distribution, and the question isn’t “did it fail?” but “how often, and how badly?”
That changes what a fix means. Before you touch anything, run the failing case again, several times. A surprising number of failures don’t come back. You’d have been patching a one-off.
It also means the run after your patch proves nothing. It was going to pass most of the time anyway. Whether the fix worked is a question about a batch of runs, not the next one.
The cause is rarely in the prompt
When a failure does repeat, it’s tempting to assume the model misunderstood the instructions. Usually it understood them fine. It was working with something else that was wrong.
Some of the usual sources:
- A tool that invites misuse. A vague parameter, a confusing name, or an error message that suggests the wrong next step. The model does what the tool seems to want.
- Missing context. The agent guessed because nothing told it. The guess was reasonable; it was just wrong.
- A bad default. A template, a scaffold or a config value it inherited and had no reason to question.
- The harness. Stale state, a truncated tool output, or instructions that changed mid-session without the model being told.
None of these is fixed by telling the model to behave. The patch makes this symptom go away while the cause keeps producing others.
The quickest way to find the cause is to look at what the agent saw, not what it did. Read the trace around the failure.
Then ask the agent one follow-up question about why it made that choice. Agents are often honest one question deep; the answer tends to point at the real source.
Patches leak
A prompt line doesn’t apply to the case you wrote it for. It applies to every request, forever.
I once watched a fix go wrong exactly this way.
An agent was supposed to ask the user a few questions through a dedicated question tool. Half the time it asked them in plain text instead, or announced them and stopped.
So a line went in: every question goes through the tool, and never announce one you don’t ask. The plain-text questions disappeared. In four of the next ten runs, the agent asked nothing at all, though the task required it.
The patch worked on the symptom and taught the wrong lesson. That’s the usual pattern. Rules get applied too broadly or quietly ignored; reasons generalize.
Each patch also costs attention. The model has to consider it on every turn, even when it’s irrelevant. A prompt that has grown one patch at a time ends up full of special cases the model is always thinking about.
When the prompt is the right fix
None of this means never touch the prompt. Sometimes the prompt really is the root cause: the model was missing information it couldn’t have known.
The test is what you’re adding. Context and reasons are fine. If the agent didn’t know a constraint existed, tell it, and say why. A model that understands the reason applies it sensibly.
Rules that force behaviour are the ones that leak. And if a behaviour must always hold, a prompt is the wrong place for it anyway. Put it in code, as a guard or a check the agent can’t talk its way around.
What to do instead
When an agent fails, the order matters more than the fix:
- Reproduce it. Run the case several times. If it doesn’t come back, it may not be worth fixing.
- Find what the agent saw. Read the trace and ask it why. Look at tools, context, defaults and harness before the prompt.
- Sort it. A wrong outcome with real cost gets a mechanical guard. A suboptimal but valid choice gets left alone. A harness problem gets fixed in the harness. Missing information goes in the prompt, with the reason.
- Check it on runs you didn’t tune on. A fix that only works on the case that prompted it is a patch.
The same rules apply to your evaluator. LLM judges are non-deterministic too: they score the same output differently across runs and miss failures. Check them against cases where you already know the answer before trusting their numbers.
A prompt patch makes the failure you saw go away. The job is to make its cause go away.