Transformer attention playground

Sentence

Text to tokenize

Examples

Up to 10 tokens. Words are lowercased, and punctuation and contractions become tokens of their own.

Tokens0 / 10

Loading the example sentence.

Attention pipeline

Simulated

Ribbon thickness encodes link strength: fixed, illustrative values of 2 into each projection, 3 from query to attention and 4 from attention to output. Each lane follows one token. Keys and values stop at the projection column because every token's query reads them, and that mixing is what the matrix shows.

Attention matrix

0%attention weight–

Rows are the attending token and columns the attended token. Every row sums to 100%, which is the job row-wise softmax does in a real attention layer. Here each simulated row is rescaled by its own total.

Head readout

Tokens–

Even split–

Self-attention–

Peak weight–

Even split is what every cell would hold if a token spread its attention evenly. Self-attention is the average of the diagonal. Switch heads and watch both move.

Next token

Illustrative

A fixed example of the distribution a model would produce after the last layer. It is not predicted from your text.

Reading guide

Attention(Q, K, V) = softmax(QK^T / √d_k) · V

Queries, keys, values. Every token is projected three ways. Its query is compared with every key, and the dot products are scaled by the square root of the key size to give one row of scores.

Row-wise softmax. Softmax turns each row of scores into weights that sum to 100%, so a row reads as where that token looks. The output mixes the values by those weights.

Heads. A layer runs several heads side by side, and each can pick up a different relationship. The four heads here are hand-made patterns with random noise, not learned weights.