Cheat sheet

Evals

From lesson 0014. Measure, then add a node only when a number you care about moves.

1. TaskInput + success criteria. ~20 real failures to start.
2. TraceFull trial record. Outcome ≠ last sentence.
3. GateChange one thing. Keep it only if the score moves.

Not

• Not a vibe check. Feeling “the graph is calmer” is not a grader.
• Not “wait for hundreds of tasks / a vendor platform.” Anthropic: 20–50 from real failures.
• Not the claim in the transcript. Outcome is environment state after the trial.
• A graph, helper, or skill is a change. The eval decides if it stays. Graph card.
• CI grades the branch. In production, join the trial to HTTP and Stripe with one id. Production traces.
• Two boards once you have trees: capability vs regression. Eval suites.

In a full-stack app

• Task = POST /assistant + pass rule (policy before Stripe). Source: Zendesk / Sentry, ~20 failures.
• Trace = one request id (messages, tools, SQL, Stripe). Not the chat bubble.
• Outcome = refunds row + Stripe status. Same idea as Anthropic’s reservation SQL.
• Gate = keep policy-before-Stripe if pass rate moves; revert a helper that only burns tokens.

Anthropic: simplest solution; add complexity only when it demonstrably improves outcomes. Hamel: error analysis before infrastructure — read traces first.