Explain how self attention works in a transformer, out loud, without drawing anything.

Each token is projected into a query, a key, and a value vector. You score every query against every key with a dot product, scale by the square root of the head dimension, softmax the scores into weights, and take the weighted sum of the values. Each token ends up holding a representation mixed from whichever other tokens it found relevant. Multiple heads do this in parallel with different projections.

Every token embedding gets multiplied by three learned matrices to produce a query, a key, and a value. The query is what this token is looking for, the key is what it advertises, the value is what it passes along if selected. You take the dot product of each query with every key, which gives a compatibility score for every pair of positions. Divide by the square root of the head dimension, because otherwise the dot products grow with dimensionality and push the softmax into a region where it is effectively a hard argmax with vanishing gradients. Softmax across the row turns those scores into weights that sum to one, then you take the weighted sum of the value vectors. So each token walks away holding a blend of the tokens it cared about. Multiple heads run that whole operation in parallel with separate projections, so one head can track syntax while another tracks coreference, and the outputs get concatenated and projected. The cost worth naming is that the score matrix is sequence length squared, which is exactly why long context serving is expensive and why flash attention and KV caching matter so much.