Atoms
The model does not see words
Your first engineering skill: look at a chat and see a sequence of token IDs, not English. Every later pattern — context, tools, skills, loops, graphs — is a way of spending that sequence on purpose.
The trick ChatGPT hides
When you type a sentence, it feels like the assistant “reads” it. Under the hood the pipeline is mechanical — and this skeleton is shared by GPT, Claude, Llama, Gemini, and the rest:
Watch the highlight move: every chat turn is this loop. The flavour is which tokenizer sits in step 2.
Karpathy’s one-line picture: the network is a next-token predictor. Chat products are that predictor, wrapped in a loop, with extra training so it answers as an assistant rather than continuing a webpage. Intro to Large Language Models, first third of the talk.
OpenAI: tokens “can be as short as a single character or as long as a full word.” Billing and context limits are token counts, not word counts. What are tokens and how to count them?
Tokens are a learned alphabet
If we gave the model one ID per Unicode character, sequences would be long and expensive. If we gave it one ID per English word, it would choke on typos, code, and other languages. The compromise used by GPT-style models is byte-pair encoding (BPE): start from bytes, repeatedly merge the most common adjacent pairs into a new vocabulary entry. Frequent chunks (“the”, “ing”, a leading space plus a common word) become single tokens. Rare strings stay split.
Two consequences you will keep meeting:
- Case and spaces are real.
Hello,hello, andhelloare often different IDs. - Odd failures are sometimes tokenizer failures. Arithmetic, Unicode, and “the model can’t see that space” bugs often start here — Karpathy calls tokenization gnarly for exactly this reason. LLM Tokenization lecture notes
Skill: watch the split
Type in the box. This page uses a toy BPE — a handful of merges so you can see chips appear and split. It is not GPT-4 or Claude. After you get the feel, you will check a real encoding.
Toy token lab
Try: extra spaces, ALL CAPS, a number like 398, then a non-English word. Leading spaces often glue to the next chunk — that is a feature of real tokenizers, not a bug in the chips.
Now the real thing: open tiktokenizer.vercel.app, paste the same strings, and switch the encoding in the dropdown. Karpathy uses this app because the live split is the ground truth for a given encoding — not for “language” in general. minbpe lecture
Same skeleton, different flavour
The dropdown changing the split is not a bug. The job is the same for every modern LLM: map text to a sequence of integer IDs from a fixed vocabulary, then predict the next ID. Hugging Face groups the algorithms as three subword cousins — BPE, WordPiece, and Unigram — plus SentencePiece as a library that can run BPE or Unigram on raw text (spaces included). Tokenization algorithms
The dictionary is not shared. Each model family trains its own vocabulary on its own data, with its own size and merge rules. Token ID 1234 is a different chunk in GPT-4 than in Llama. OpenAI says this outright: “Different models use different encodings,” and ships several (r50k_base, p50k_base, cl100k_base, o200k_base). Their cookbook even compares the same English word across encodings and gets different splits. How to count tokens with tiktoken
Shared
Text → tokenizer → IDs → next-token loop. Subword chunks, not whole words.
Per family
Vocabulary, merge table, special tokens, even how spaces are stored. Count with that model’s tokenizer.
It changes inside a vendor too. Llama 2 used SentencePiece; Llama 3 switched to a tiktoken-style BPE. Anthropic documents that Claude 4.7 and later use a newer tokenizer: the same input is about 30% more tokens than on earlier Claude models — so you recount against the model you will actually call. Llama 3 tokenizer note; Claude token counting
Feel it with two toy “families.” Same string. Different merge tables. Different chip counts — the same phenomenon as Tiktokenizer’s model dropdown.
Two flavours, one sentence
The wide table has learned more English chunks, so the same text is shorter. A real model trained on more code or more Japanese would look like a third flavour. Never mix IDs across flavours.
Engineering rule for later (RAG, agents, cost caps): count tokens with the tokenizer that belongs to the model you will run. A GPT encoding will mis-budget a Claude or Llama prompt.
Why this is lesson one for an AI engineer
This course ends at harnesses, skills, plugins, context engineering, loop engineering, and graph engineering. Those names all sit on one fact:
Context is a finite attention budget, measured in tokens. The job is the smallest set of high-signal tokens that make the next action likely. Anthropic, Effective context engineering
A tool dump, a retrieved PDF, a SKILL.md, a looping agent’s history — each is more tokens in the same window. If you cannot see the window, you cannot choose an architecture. That is the whole course in miniature: judgment starts with the budget.
Check
Answer from memory. Options are the same length on purpose.
Primary source (do this next). Watch Karpathy’s Intro to Large Language Models through the tokenization / next-token section (~first 25 minutes). Pause and replay; do not treat it as background TV. Then spend five minutes on Tiktokenizer with strings you care about (your name, a JSON blob, a bit of code).
On Tiktokenizer, switch encodings on one string and write down the two counts. That is the flavour difference in the wild.
Keep this lesson’s token cheat sheet for later. The map of the era is the rest of the path.