Mamba

Mamba (Albert Gu + Tri Dao, 2023) is a selective state-space model · an alternative to transformer attention. Key property: O(N) compute vs transformer's O(N²). Uses a fixed-size hidden state that "compresses" past tokens, reducing memory. Training throughput similar to transformers; inference throughput much better at long context. Mamba-2 (2024) improved the design further; hybrid models (Jamba, Zamba) combine Mamba + attention layers.

Mamba's core innovation is making state-space models content-aware · the state update depends on the input token, so the model can "selectively forget" or retain. This solves earlier SSM limitations (previous SSMs like S4 were linear, lost expressiveness). Mamba models perform competitively with transformers at small scales but quality gap at 70B+ remains unclear. Hybrid models like Jamba (AI21) mix Mamba + transformer layers to get best of both.

Mamba kernels require custom CUDA implementations (Selective Scan). Tri Dao's FlashAttention-level engineering made Mamba practical · without it, the theoretical speed advantage didn't materialize. Jamba-1.5 demonstrates Mamba+MoE+transformer hybrid at 398B (52B active) with strong long-context performance. Mamba-2 adds better structured state-space duality with attention, enabling cleaner theoretical analysis and faster implementations. Open research question: do pure Mamba models scale to frontier quality? Current evidence suggests hybrids win.