it → cat

Why this quietly runs your chatbot

This glance-back-and-blend trick isn’t a classroom curiosity — it’s the actual mechanism from the 2017 paper literally titled “Attention Is All You Need,” the one that kicked off the modern AI boom. Every pronoun a chatbot untangles, every “it” it doesn’t get confused by, every long paragraph it somehow keeps straight — this is the whole trick underneath, not a metaphor for it. You’re about to build the real thing, from math you already own.

Step 1: you just did the hard part for free

“It” could grammatically mean the cat, the sitting, or nothing at all. Your brain didn’t guess — it glanced back over every earlier word and gave each one a weight: “cat” got almost all of it, “sat” got a little, “the” got basically none. That silent weighing-up has a name now: attention.

Step 2: watch a machine do the same glance-back

Same sentence, live. Click any word and watch it glance back — bars light up over the other words showing how much weight each one gets. Start with “it.”

Step 3: how it scores relevance — dot product, again

The machine doesn’t “just know” cat matters. It compares “it” against every earlier word using the exact dot product similarity meter you built back in Module 2 — how aligned are these two arrows? “It” and “cat” point a similar way (both are “the thing that’s tired”), so their score is high. “It” and “the” barely align, so their score is near zero.

Step 4: turning scores into weights — softmax, again

Raw dot-product scores are just numbers — some could even be negative, and they don’t add up to anything tidy. Two moves fix that. First, shrink every score: divide it by √d, where d is how many numbers are in each word’s arrow. Long arrows make dot products come out big just by having more terms to add up, and big scores would make the next step hand one word nearly the whole vote. Dividing by √d keeps the election fair. Then feed the shrunken scores into softmax, the exact machine from Module 1: eˣ makes every score positive and stretches the leader out ahead, then divide by the total.

That’s exactly what set the height of every bar you just clicked through.

Step 5: don’t pick a winner — blend

Here’s the part that isn’t obvious: attention doesn’t just crown “cat” the winner and throw the rest away. It builds a blend — mostly cat, plus a pinch of everything else, in exactly the proportions the weights say:

That blended mix becomes the new, sharper meaning of “it” the model carries forward.

Step 6: this has a name, and you just built it

Score with the dot product, shrink by √d, weigh with softmax, blend by the weights — that combo is attention, full stop. Next rung, you stack this together with everything else you’ve climbed and assemble The Transformer itself.