The eval theater problem
Every team writes evals. Most of them measure the wrong thing — usually 'does the output contain the expected string?' — which passes 100% of the time until a prompt change silently breaks a downstream tool call in production.
The failure mode: evals give a green light because they're too narrow to catch what actually broke. The team ships, users hit the regression, everyone loses trust in the eval process. The fix is not more evals, it's better ones.
Designing evals that catch things
Start with production failures, not synthetic examples. Every real bug becomes a permanent eval case. This is the single practice that separates useful eval suites from theater.
Every case gets three grades: exact-match where possible, structured-output-schema-match for tool calls, and an LLM grader for the fuzzy stuff. Track them separately so you can see which axis regressed.
Cover the boring cases: empty input, extremely long input, adversarial input, non-English input, ambiguous input. These break more workflows than clever attacks.
LLM graders without the drift
LLM graders are the only way to score subjective outputs at scale — and they drift as underlying models change. Version your grader prompt, pin its model, and re-baseline monthly.
Structure grader prompts to output a numeric score + rationale, not pass/fail. This gives you drift signal: if average score creeps down 0.3 points across a stable set, something changed in either the grader or the target.
Sanity check: have a human grade 30 random cases each quarter. Correlate to LLM grader scores. If correlation drops below 0.8, your grader has drifted and needs work.
Wiring into CI
Run the full eval suite on every PR that touches a prompt file. GitHub Actions with a self-hosted runner (or Modal / Baseten) keeps costs bounded.
Fail the build only on regressions vs main, not on absolute thresholds. Absolute thresholds create pressure to game the metric; regression-only keeps focus on 'don't make it worse'.
Post the eval delta as a PR comment. Reviewers see 'this change moved 3 cases from pass to fail' and can decide whether it's acceptable.
Production monitoring is half the job
Offline evals catch known regressions; production monitoring catches unknown ones. Log every LLM call with input, output, tool calls, latency, and cost.
Sample 1-5% of production traffic through your LLM grader nightly. Alert if scores drop more than 5% over a 7-day rolling window.
The most valuable production data is user thumbs-down. Route those directly into your eval suite as new test cases within 48 hours.