Cheat sheet

Eval suites

From lesson 0019. Two boards on the same POST /assistant traces. Not a re-teach of task · trace · gate.

1. SplitCapability starts low. Regression stays full.
2. GraduateA green hard task moves to CI.
3. CalibrateCode on the row. Judge after labels.

Not

• Not one mixed score (4/8 hides the fire). Anthropic: two questions.
• Not an uncalibrated LLM-as-judge dashboard. Hamel: TPR/TNR on held-out human labels.
• Not a vendor SKU. The boards are the architecture. Trees are how you sample.
• Not pass@k tonight. Saturation: 100% on the hill means graduate and add harder work.

CI = regression + code assertions. Nightly = the hill. Production failures become new hill tasks. Stripe amount is a code grader; tone is a judge only after ~50 labels.