Atoms
The window is working memory
Lesson 1: the model sees tokens. This lesson: those tokens share one finite slot. Karpathy’s picture is RAM. Anthropic’s picture is an attention budget. Both mean you cannot dump everything in and hope.
One box, everything in it
A context window is the maximum number of tokens the model can condition on in a single call — system instructions, the chat so far, retrieved docs, tool results, and the reply it is about to write. OpenAI’s overflow error is blunt: input plus reserved output exceeded the maximum. How to count tokens with tiktoken
Karpathy’s LLM-as-OS sketch puts the model in the CPU seat and the context window in the RAM seat: fast, limited, wiped when the process ends. Disk (files, RAG, notes) is how you keep what will not fit. Intro to Large Language Models, LLM OS section
One window. Four tenants. Chat apps hide the fight by dropping or summarising old turns — that is already a tiny harness.
Skill: spend the bar on purpose
Add layers until the toy window overflows. Real windows are 32k–200k+ tokens depending on the model; the shape does not change. Count with that model’s tokenizer (lesson 1).
Toy context RAM
Worked examples (OpenAI’s style)
OpenAI’s tokenizer articles don’t stop at the definition. They pin a real string, show the chips and IDs, then say what you just saw. Same move here — these numbers are from their cookbook, not the toy bar above.
Example · one English sentence
“tiktoken is great!”
cl100k_base · 6 tokens · spaces glued to the next word as “·is” / “·great”
What this shows. The model never sees the sentence. It sees six IDs. OpenAI’s cookbook encodes this exact string and lists those integers. Rule of thumb on English: roughly 4 characters per token — here 18 characters / 6 tokens = 3.0, in the same zip code. tiktoken cookbook; What are tokens?
Example · same word, two encodings
“antidisestablishmentarianism”
r50k_base / p50k_base → 5 tokens · cl100k_base / o200k_base → 6 tokens
Older: ant | idis | establishment | arian | ism
Newer: ant | idis | establish | ment | arian | ism (cl100k)
What this shows. Flavour is not a UI quirk. OpenAI compares encodings on this word and the splits disagree. A “token count” without naming the encoding is not a count. §5 Comparing encodings
Example · a chat is more than the words
{
"model": "gpt-4o-mini",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Hello!"}
],
"max_tokens": 64
}
Window math (shape, not a live API count): words in the two contents + wrapping tokens per message + 64 reserved for the reply. All of that must fit.
What this shows. Chat Completions wrap each message (OpenAI’s cookbook uses about 3 extra tokens per message, plus a few to prime the assistant). max_tokens is output headroom inside the same window. That is why “my prompt is short” can still overflow: you reserved a huge completion. Full anatomy is lesson 0003. §6 Counting tokens for chat
Full is not the same as useful
Anthropic: context is a finite resource with diminishing returns. They name context rot — as token count rises, recall from that context gets worse. Transformers let every token attend to every other token, so n tokens make n² relationships; the attention budget thins out. Effective context engineering
Liu et al. measured a sharper positional effect: in long prompts, models use facts at the start or end more reliably than facts in the middle — a U-shaped curve, even for “long context” models. Bigger RAM did not mean even access. Lost in the Middle
Where you park the fact
Sketch of Liu et al.’s U-curve, not their raw numbers. Drag the fact through a long prompt. Start and end stay usable; the middle dips. That is why later “context engineering” puts the needle where attention still works — or refuses to stuff the haystack at all.
Hard limit
Overflow → the call is rejected. Chat UIs truncate or compact so you rarely see the error.
Soft limit
Still under the cap, but rot + the U-curve. The model “had it in context” and still missed it.
This is why it matters for everything that follows. An agent loop that appends every tool result is filling RAM with hay. RAG that stuffs 50 chunks is buying a bigger middle. Skills and compaction (later lessons) exist to keep the high-signal set small. Anthropic: “the smallest possible set of high-signal tokens.”
Check
Answer from memory. Sampling (how the next token is picked) waits for the API lesson.
Primary source (do this next). Read Anthropic’s Effective context engineering from the opening through “Why context engineering is important” (finite budget, n² attention, context rot). Then skim the abstract and Figure 1 of Lost in the Middle — you already felt that U-curve in the lab.
Keep the context-window cheat sheet. Tokens recap: tokens. Path: map of the era.