Start with your sentence.
Type any sentence above or click a token to focus it. In “River water flooded the bank”, “bank” is ambiguous until it connects to the words around it.
Slice words into token IDs.
Silicon only computes on numbers. A tokenizer splits your sentence into 5 atomic tokens and looks up each one's integer ID in a fixed vocabulary.
Turn each ID into a 3D vector.
Each token ID becomes an arrow in high-dimensional space. Because lookup is static, “bank” starts without any context from its neighbors.
Rotate vectors by word order.
Processing all words in parallel erases sequence order. Rotary Position Embedding (RoPE) twists each vector along a 3D helix according to its slot 0 → 4.
Split one token into three roles.
Three learned matrices project “bank” into a Query (what it seeks), a Key (what it advertises), and a Value (the meaning it carries).
Score relevance by 3D angle.
“bank” compares its Query against every token's Key using a dot product. Aligned arrows score high; dividing by √dₖ keeps large dimensions stable.
Focus a 100% spotlight.
Softmax turns raw scores into weights that sum to 100%, while the Causal Mask hides future words (−∞) so tokens only look backward.
Pull meaning into focus.
Each token adds up the Value vectors weighted by its attention. Upstream words physically pull “bank” into its contextualized meaning.
Run parallel heads on a highway.
Multiple attention heads track nouns, verbs, and syntax simultaneously, adding their updates onto the Residual Stream before a 4d MLP extracts facts.
Stack layers to predict the next word.
Stacking Transformer blocks refines your sequence from raw bytes into deep context. The final layer projects onto the 50,257-token vocabulary, predicts “river”, and loops it back in.