Transformer Architecture Explained (2026)

RNNs process tokens one at a time — slow on GPUs, hard to scale. Transformers process all tokens in parallel via self-attention. Combined with residual connections and layer normalization, this made it possible to stack much deeper models and train them faster.

For every token, learn three projections: Query (Q), Key (K), Value (V). Compute attention scores as (Q · Kᵀ) / √d_k. Apply softmax to get weights. Sum the V vectors weighted by those scores. The result for token i is a mix of all tokens' V vectors, weighted by how relevant they are to token i.

Instead of one big attention, split Q/K/V into h smaller subspaces (heads) and do attention in each. Concatenate the outputs and project back. Different heads learn different relationship patterns — syntactic dependencies, coreference, positional patterns.

Self-attention has no idea of word order. Fix this by adding a position vector to each input embedding. Original paper used sinusoidal encodings; modern models use RoPE (Rotary Positional Embeddings, in Llama) or ALiBi (which extrapolates to longer sequences).

Three flavors of Transformer:

• →Encoder-only (BERT) — bidirectional context, masked language modeling, great for classification/embedding
• →Decoder-only (GPT, Llama, Claude) — causal masking, next-token prediction, the chat-model family
• →Encoder-decoder (T5, original 2017 paper) — separate encoder reads input, decoder generates output

The core block is unchanged, but production transformers have evolved:

• →Attention variants — MHA → MQA → GQA (Llama 3) for cheaper inference
• →FlashAttention v3 — memory-efficient attention; required for long context
• →RoPE positional encoding — better extrapolation than sinusoidal
• →Mixture of Experts (MoE) — sparse activation in the FFN; DeepSeek-V3 popularized it
• →Multi-head Latent Attention (MLA) — compresses KV cache, used in DeepSeek-V3