Transformer

Transformers replaced recurrent networks (RNNs, LSTMs) by processing all tokens in parallel instead of sequentially. The core idea is self-attention: every token computes relationships with every other token, then aggregates information weighted by those relationships. Stacking many such layers produces a model that can capture long-range dependencies and train efficiently on modern GPUs.

A transformer block has: multi-head self-attention, layer normalization, a feedforward MLP, and residual connections. "Attention" computes query-key-value projections of each token, takes dot products of queries with all keys, softmaxes them into weights, and outputs a weighted sum of values. Multi-head attention runs several of these in parallel at different subspaces, capturing different relational patterns. Positional embeddings (absolute, RoPE, ALiBi) encode token position since attention itself is order-agnostic. Decoder-only transformers (GPT family, Llama, Claude) are the dominant architecture for generative LLMs. MoE transformers replace the feedforward with a bank of experts plus a router (see MoE entry).

Attention: Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) V. Multi-head: h parallel attention heads with output = Concat(head_1, ..., head_h) W^O, enabling different representational subspaces. RoPE (Rotary Position Embedding) rotates query and key vectors by position-dependent angles, preserving relative position in dot products. ALiBi adds a linear bias to attention scores based on position distance. FlashAttention reformulates attention to use tiling + online softmax, reducing memory from O(n²) to O(n). Grouped-Query Attention (GQA) shares K/V across multiple query heads, reducing KV cache by 4-8×. Multi-Query Attention (MQA) takes this to the limit with a single K/V pair. Mixture-of-Experts scales the FFN component by routing each token to a subset of expert MLPs.