Cheat sheet

Tokens

Compressed facts from lesson 0001. Print this. Do not use it to learn the idea for the first time.

What a token is

• A token is an integer ID from a fixed vocabulary. The model’s input and output are sequences of those IDs.
• A tokenizer maps text ↔ token IDs. The neural net never sees raw characters.
• Modern LLM tokenizers use byte-pair encoding (BPE) and similar subword schemes — not “one word = one token.”

Rules of thumb

• English: often ~4 characters per token, or ~100 tokens per 75 words — rough, encoding-specific. OpenAI
• Case, leading spaces, punctuation, and non-English text change the split. “hello” ≠ “Hello” ≠ “ hello”.
• Context windows and API bills are measured in tokens, not pages or words.

Same skeleton, different flavour

• Every modern LLM: text → tokenizer → IDs → next-token loop. Subword algorithms (BPE, WordPiece, Unigram / SentencePiece). HF tokenizer summary
• Each family trains its own vocabulary. Do not reuse token IDs or token counts across models.
• OpenAI encodings differ even inside one vendor: r50k_base, p50k_base, cl100k_base, o200k_base. tiktoken cookbook
• Llama 2 = SentencePiece; Llama 3 = tiktoken-style BPE. Claude 4.7+ uses a newer tokenizer (~30% more tokens on the same text). Llama 3; Claude counting

Why later patterns care

• A tool result, a skill file, a retrieved doc, a chat history — all spend the same budget.
• Context engineering = choosing the smallest high-signal token set. Anthropic

Check with a real tokenizer

tiktokenizer.vercel.app — pick an encoding, paste text, read the IDs. The lesson’s chip lab is a toy. Next unit: context window.