The world's smallest transformer.
Running in your browser. Four words, two sentences, sixteen numbers. Every one of them is on this page, and you can change them.
The whole training set: Mary greeted Bob. Bob ignored Mary. That is all this model has ever read. From it, it learned two things nobody told it: that there are people and verbs, and that there is friendly and unfriendly. Those two facts are the square below.
The understander 8 numbers
Each word gets two coordinates. Left–right turned out to mean verb–person; up–down turned out to mean unfriendly–friendly. Nobody chose that. Edit a number and watch the word move.
The talker 8 numbers
For each word, an arrow in that word's colour. The further a word sits along the orange arrow, the more likely it is to come before Mary; the green arrow is for greeted, and so on. Scores are dot products; nothing else.
Ask it what comes next
Scores go through a softmax: e^(score / temperature), then divided by the sum. Low temperature is a lookup table. High temperature is a poet.
Let it talk
With the original two sentences the model is nearly a lookup table: Mary always greets, Bob always ignores. The third button retrains on a world where, once in a blue moon, Mary ignores and Bob greets. Watch the arrows stretch.
This page is a playable version of the idea in Luis Serrano's video The world's smallest transformer: the 4-2-4 network, the square, the arrows, the point system. He built it; this just lets you touch it. No attention yet, so the model can still say "Bob saw Bob". That is his third video. Part of The Worth Building Organization.