Evals and production judgment

Measure, then add

Lessons 0006–0013 gave you shapes: a path, a loop, a harness, a graph. Tonight is the gate. Do not add a node because it is trendy. Write a few real tasks, read the traces, keep the change only if a number you care about moves.

1. 1 · TaskInput + what “pass” means. Start with ~20 real failures.
2. 2 · TraceThe full trial record. Read it. Outcome ≠ the claim.
3. 3 · GateChange one thing. Keep it only if the score moves.

You already know how to draw edges. Tonight you decide whether they earned the floor.

Architecture first. Baseline → one change → measure → keep or revert. Anthropic: add complexity only when it demonstrably improves outcomes.

Vibe is not an eval

Hamel Husain’s point: teams stall on vibe checks — tasting the air after a prompt change. That plateau is why evals sit at the center of shipping. Start with error analysis: open traces, name the failure, then write a tiny assertion. Infrastructure can wait. Your AI Product Needs Evals

Anthropic’s flight-booking example: the transcript may say “booked.” The outcome is whether a reservation exists in SQL.

# TASK — not “make it nicer”
ticket 441: allergy + walnut crumb
pass iff classify_before_till AND policy_holds

# TRACE — last sentence lied
assistant: "Refunded. You’re all set."
tools:     refund(441)          # never classify
outcome:   { refunded: true, classified: false }  → FAIL

# GATE — change one thing
open loop      3/8
+ classify     7/8   ← keep the edges
+ extra helper 7/8   ← revert (4× tokens)

# NOT AN EVAL
"The graph feels calmer."

Kitchen only to make the board concrete. Capability evals start low on purpose — a hill to climb. Keep the node that moved the stamps.

Same gate in a full-stack app

The kitchen was a picture. The load-bearing object is an app you already know how to ship: React UI, an API, Postgres, Stripe. You add POST /assistant that can search docs, fetch an invoice, and issue a refund. That assistant is the loop. Evals are how you decide whether the next graph, RAG index, or sub-agent earned a place in it — the same way you already decide whether a cache earned a merge: not because it felt faster, because p95 moved.

• Ticket POST /assistant plus tenant, user, and the text. Last month’s Zendesk + Sentry failures are the suite — Anthropic’s 20–50, not a made-up sandbox.
• Classify / till A policy code node before Stripe. Docs questions are retrieve-then-answer. “What are competitors charging?” is open research: do not graph it.
• Trace One request id: model messages, tool calls, SQL, Stripe. The chat bubble is not the grade.
• Outcome A row in refunds and a Stripe refund id — or not. Anthropic’s flight-booking SQL is this table.

Translate, do not replace. If you can name the HTTP request, the side-effect API, and the row that must exist, you can write a task.

Vibe is you clicking around staging. Trace is this picture for one request. Grade the cylinders, not the bubble.

# TASK — a real failure, not “make the bot nicer”
POST /assistant
  tenant=acme  user=12
  text="Refund INV-441. Annual plan."
pass iff policy.check BEFORE stripe
     AND refunds.policy_ok = true
     AND stripe.refunds.retrieve(id).status = succeeded

# TRACE — last sentence lied (open loop)
assistant: "Refund issued. You’re all set."
tools:     stripe.refunds.create({ invoice: "INV-441" })
           # never policy.check(plan, window)
outcome:   stripe re_1 succeeded
           refunds.policy_ok = false     → FAIL

# GATE — change one thing, as you would a cache
open loop           3/8
+ policy-before-stripe  7/8   ← keep the graph
+ web-research helper   7/8   ← revert (4× tokens)

# NOT AN EVAL
"LangGraph feels more production."

Skill: run the gate

Vibe is staging. Trace is one request id. Score is a small suite from production failures. Gate keeps only the change that moved the count. App is the same eight tickets on your stack.

Three names, one job

Parked here: pass@k, A/B, LangSmith. Capability vs regression and LLM-as-judge calibration are lesson 0019. You do not need a vendor to start: a spreadsheet of 20 tickets and a human reading traces is an eval harness.

Check