# Your next eval case is already in your logs

**Key takeaways**

- The eval cases you keep failing to invent are already written — by your incident log, your rollbacks, and the people who report your bugs.
- A bug report is an eval case that happened in production. Your job is not to imagine it; it's to promote it from "incident" to "assertion".
- In six days, eight boundary reports on our systems each became a merged fix with regression tests. Median time from issue filed to fix merged: 76 minutes. The regression tests are the permanent eval entries.
- Guards that have "never fired" are eval cases too — but of a different kind: you're not testing the failure, you're testing that the guard is still alive.
- Start small: take the last three things that surprised you (a bug, a wrong estimate, a silent skip) and write them as one assertion each. That is already a better eval set than the empty file you've been meaning to fill.

---

## The empty file everyone has

Every codebase has one. `evals/cases.jsonl` — or `tests/eval_cases/`, or a "prompts" folder with three lonely examples in it. Last touched the week someone declared evals were important.

Writing eval cases by hand is homework. You sit down to predict how your agent (or your model, or your estimator) might fail, and the ideas come out thin: *"user asks something complicated"*, *"model returns a long answer"*. You can feel they're weak but you don't know why. After three entries you stop.

The reason it's hard is not that you lack imagination. It's that **you're trying to predict failures instead of reading the failures you already have**.

## What an eval case actually is

An eval case is a scenario plus an expected behavior:

```json
{"scenario": "request carries 8 tool definitions with large schemas",
 "expected": "the usage estimate includes their token cost"}
```

Run it when something changes. If the behavior regresses, the case fails. That's the whole loop — the same loop as a unit test, applied to the behavior of an agent or an AI-assisted system.

The hard part was never the mechanism. It's the *scenario list*. Where do realistic scenarios come from?

## Your incident log is an eval corpus

Think about the last month of your system's life. Not the happy path — the other 5%:

1. **A bug report** — from a user, a reader, a colleague. "It estimated the usage wrong and we blew the budget." That sentence *is* an eval case: scenario + failed expected behavior. The fix is the assertion.
2. **A rollback** — you shipped something, reverted it by dinner. The thing that made you revert is a case.
3. **A guard that never fired** — alarms that sit silent for weeks. You don't know if they're healthy or dead. That's a *different* eval shape: not "does it catch the failure" but "is it still running".
4. **A wrong estimate** — the counter said X, reality was Y. The divergence is a case.
5. **A "it worked?!" surprise** — the output was right but for the wrong reason. Positive cases that pin the *correct* mechanism are as valuable as negative ones.

Every one of these is a scenario that *actually happened*. You don't have to imagine whether it's realistic. It happened. All you're doing is promoting it from memory to assertion.

## Six days of real data

We ran this loop deliberately for six days. The trigger: people read our write-ups about guards and estimators, then reported boundaries we'd shipped past — in comments, in issues, in reviews. Each report became:

1. verify the claim against the code (is this real, or a misread?)
2. file an issue naming the exact gap
3. fix it
4. **land regression tests that pin the reported scenario** — the permanent eval entry

The result, from issue filed to fix merged:

| Report (what the boundary was) | Issue → merge |
|---|---|
| Usage anchor ignored a mid-session model/provider switch | 49 min |
| Anchor stats lacked provider identity | 38 min |
| Silent drift when the provider changes under an unchanged base URL | 129 min |
| Reader-fix-latency metric had no sample count / age counter | 88 min |
| Silent-drift guard "never fired" — not measurable | 52 min |
| Threshold calibration from accumulated sub-threshold data | 134 min |
| Planted-fire staleness alarm (guard age) | 84 min |
| Usage estimator ignored request-level tool schemas | 69 min |

**Eight reports, eight merged fixes, median 76 minutes from issue to merge** — and every single fix landed with regression tests in the suite. Six days. The reports came from five different people, none of whom had seen our internal code before reading the write-ups.

Two of those rows are worth unpacking, because they're the two eval shapes people forget.

**The estimator that ignored tool schemas.** Our usage estimator walked the conversation messages to count tokens — but the tool definitions travel in a *separate* field of the request. A session that added tools mid-flight grew expensive while the estimate stayed flat. A reader spotted the blind spot from the outside. The regression tests now pin the exact scenario: request-level tool schemas are counted. That's one eval entry that will fire the moment anyone "optimizes" the estimator back to messages-only.

**The guard that had never fired.** A drift guard sat silent for weeks. Was it healthy? We had no way to tell — a guard that never fires and a guard that stopped running look identical on disk. The fix made the guard's *liveness* measurable (a staleness alarm, a planted-fire drill that rides the real code path). The eval shape here is not "catch the drift" — it's "prove the catcher is alive". If you only write eval cases for failures you've seen, you'll never test the silence — and silence is where the expensive surprises live.

## The five-step promotion

When an incident or report lands, promote it to an eval entry:

1. **Write the scenario from the outside** — what did the reporter/user actually do? Not what your code does internally. (Tool schemas in the request — not "the estimator's message walk".)
2. **Write the expected behavior as one sentence** — the contract. If you can't, you don't understand the failure yet.
3. **Reproduce it first** — a scenario you can't reproduce is a scenario you don't understand. The reproduction *is* the eval case; the fix just makes it pass.
4. **Fix, then freeze the assertion** — land the regression test in the same change as the fix. If the test lands later, it lands never.
5. **Classify it** — which shape is it: failure case (bug), liveness case (guard alive), calibration case (estimate vs reality), or positive case (correct mechanism)? The shape tells you when to run it.

## What not to do

- **Don't transcribe your logs into eval cases.** An eval asserts *expected behavior*; a log line is *what happened*. If you copy logs verbatim you get a museum, not a suite.
- **Don't keep duplicates of the same shape.** Eight reports of "estimator undercounts" are one eval case once it's fixed — the point of the entry is to keep the fix honest, not to count the complaints.
- **Don't wait for the failure to be severe.** The cheapest reports are the ones about small edges. Ours came from readers of blog posts — the boundary between "fine" and "broken" was invisible to us until they hit it.

## This week's homework

Don't design an eval suite. Do this:

1. Open your incident tracker, git history, or the last month of "why did that happen" messages.
2. Pick the last three *surprises* — a bug you fixed, an estimate that was wrong, a guard that was silent too long, a report from someone who used the thing the way you didn't expect.
3. For each, write the scenario (from the outside) and the expected behavior (one sentence).
4. If a fix exists, make sure the regression test for that exact scenario exists. If it doesn't, that's your first eval entry.

Three entries written from real life beat thirty invented ones. Your eval set isn't missing — it's in your logs, waiting to be promoted.