Waves & Position · the order problem
P(i)
A Transformer reads your whole sentence in one instant — every word lands at the same time, like a deck of flashcards dumped on a table. So how does it know "the cat sat" isn't "sat the cat"? Get this wrong and every chatbot, translator, and autocomplete on your phone turns your sentences into word salad — meaning would come unglued from order. The fix: we tattoo every position with its own set of waves — a barcode made of sines and cosines that never repeats.
Step 1: the world's fastest, most forgetful reader
A Transformer doesn't creep left-to-right like you do. It looks at every word at once — that's the whole trick that makes it fast. But drop "the," "cat," "sat" on a table all at once and it's just... a pile. Rearrange the same three tiles into "sat," "the," "cat" and it's still the same pile. Same words, opposite sentence, and the model can't tell them apart — unless something extra marks which seat each word is sitting in.
Step 2: last rung's trick, remembered
Remember the dot spinning around the circle, tracing a wave as it went? Spin it fast, you get a tight, twitchy wave. Spin it slow, a long lazy one. Same tool — sine and cosine — just a different speed knob. Keep that knob in your hand. You're about to turn six of them at once.
Step 3: stamp the seat, not the word
Here's the fix. Before the model even asks "what does this word mean," every seat number — 0, 1, 2, 3… — gets its own fingerprint: the height of several waves, spinning at different speeds, all read at that exact seat. Seat 0 gets one exact combo of wave-heights. Seat 1 gets a different combo. Seat 205 gets yet another. Glue that fingerprint onto the word sitting there, and now the model can feel where a word is, not just what it is.
Step 4: why waves, and not just... counting?
Why not staple the plain number 0, 1, 2, 3… onto each word and call it done? Two reasons it falls apart. First, the numbers explode — word #4,000 needs a huge tag, word #2 a tiny one, and the same math has to handle both gracefully. Waves never leave −1 to 1, no matter how long the sentence runs. Second, plain counting gives you nothing about "how close" two seats feel. Waves do: seat 5 and seat 6 land as close cousins, while seat 5 and seat 500 look nothing alike.
Step 5: this is a real box on the real machine
This isn't a toy trick invented for this lesson — it's the literal name of a box in the actual Transformer diagram: Positional Encoding. It fires before a single word is judged for meaning, on every real model doing translation, chat, or code. That's the whole de-fanging: no mysticism, just wave-arithmetic stapled onto meaning, so "the cat sat" and "sat the cat" never get confused again.
Prove it · get one wrong and I’ll walk you through it
Two quick ones and the rung is yours.
A Transformer looks at "the cat sat" and "sat the cat" all at once, with no built-in sense of order. What problem does that create?
Why stamp a position with several sine/cosine waves at different speeds, instead of just writing the plain number 0, 1, 2, 3…?
Answer both to unlock the next rung ↓
Positional encoding — mastered.
You watched a Transformer patch its own biggest blind spot: it sees everything at once, so waves hand it a wristwatch. Every seat gets a fingerprint no other seat shares — anywhere this shows up again, you skip it.
Next rung → The Function Machine