Cheat sheet

Software factory

From lesson 0022. Agents write the code and no human reads it. Only real if something independent took over the reading.

1. SeedA spec. Human-written, up front.
2. ValidateA check the builder cannot game.
3. Feed backFailures return until it holds.

The one test

Name what replaced your eyes, and say why the builder cannot fool it. No answer means no factory.

The ladder (Dan Shapiro, Jan 2026)

• 0 · Spicy autocomplete. You type; it suggests.
• 1 · Coding intern. Snippets and boilerplate; you review all of it.
• 2 · Junior developer. You pair; you read every line.
• 3 · Developer. The agent writes most code; you are the full-time reviewer.
• 4 · Engineering team. You write specs and plans; you read some diffs.
• 5 · Dark software factory. Specs in, software out. No human reads the diff.

Tells of a fake

• The builder wrote the tests it is graded on.
• The builder can edit the tests, the scenarios, or the CI config.
• The only reviewer is another agent reading the same diff.
• “Green” is the last sentence of a transcript, not a state in the world.

The check

From lesson 0023. Hold out · twin · score.

// TWIN: same calls as the real service, plus a ledger you can read
function createStripeTwin() {
  const ledger = [];
  return {
    ledger,
    refunds: {
      async create({ invoice, amount }) {
        const id = `re_${ledger.length + 1}`;
        ledger.push({ id, invoice, amount });
        return { id };
      },
    },
  };
}

// JUDGE: the outcome in the world. The reply is a claim; ignore it.
function judge(scenario, reply, ledger) {
  if (scenario.expect === null) return ledger.length === 0;
  return ledger.length === 1
    && ledger[0].invoice === scenario.expect.invoice
    && ledger[0].amount === scenario.expect.amount;
}

// SCORE: a fresh twin per scenario; return the fraction that held
async function score(build, scenarios) {
  let passed = 0;
  for (const scenario of scenarios) {
    const twin = createStripeTwin();
    const reply = await build(scenario.request, twin);
    if (judge(scenario, reply, twin.ledger)) passed++;
  }
  return passed / scenarios.length;
}

• Write scenarios before the builder starts. Include ones where nothing should happen.
• Know the floor: what does a build that does nothing score?
• Read the board, not only the total. Two builds can fail differently at the same score.
• Wall strength: deny rule < sandbox < separate repo and CI job.
• Feed back what was observed, not the scenario text.
• A twin that drifts from the real service gives a false green. Re-check it against the vendor’s sandbox.

The repo

From lesson 0024. Record · map · rule. When the agent errs, ask in order:

• Could it see it? No: write a record, a versioned file in docs/. Chat threads and heads do not exist to it. Generate records that restate the code.
• Could it find it? No: add a pointer to the map, AGENTS.md. About 100 lines: how to run, where things are, what to read next, what always holds.
• Did it know and drift anyway? Promote to a rule: a check on every file whose message carries the fix.

// One line. ERROR and the reason together. What is wrong, what to do, where to read.
ERROR src/repo/invoices.ts:2 repo may not import from service. Move this logic up
into src/service/, or pass the value in as an argument. See docs/architecture.md.

• Promote on repetition: said once, written the second time, checked the third.
• Check output for an agent: a few lines, detail to a log file, a fast sample mode.
• Sweep on a schedule: agents copy what is there, so drift needs a recurring cleanup.
• A lint is not a holdout. The builder can edit it. Protect its config; keep the exam elsewhere.

The ramp

From lesson 0025. Shadow · release · keep.

1. Map + records. The repo explains itself. You read every diff.
2. One check. One feature: scenarios, a twin, a score. You read every diff.
3. Shadow. Write your verdict, then look at the check’s. Every “you reject, check passes” becomes a scenario. You read every diff.
4. Release. One narrow class merges on the check. You read a sample. Then the next class, from step 2.

• Always a person: the spec, the exam (scenarios, twins, judge, CI config), the irreversible, the name that answers for it.
• Cap every unattended run: turns and money.
• Each harness piece stands for something the model could not do. Re-test when the model changes.
• Keep reading a sample, or you lose the ability to.

Field reports

• StrongDM. 3 people. Holdout scenarios, twins of outside services, a satisfaction score. About $1,000 of tokens per engineer per day.
• OpenAI. 3 to 7 engineers, 5 months, ~1M lines, ~1,500 PRs, none hand-typed. Short AGENTS.md as a map, rules as linters. Greenfield.
• Anthropic. 16 agents, ~2,000 sessions, under $20,000, a 100k-line C compiler. “The task verifier is nearly perfect” or the work is not.

The bill

The check costs more than the build: one agent alone $9 and broken, with planner and evaluator $200 and working (Anthropic). Token cost. Results from new repositories that may not transfer. Tests that pass while the job is not done. No settled answer on who is liable for code no human read.

Sources: StrongDM · Willison · OpenAI · Anthropic · Shapiro · Claude Code memory docs · Anthropic harness design · Kahana.