Hyena

Hyena (Stanford + Together AI, 2023) replaces attention with implicit long convolutions generated from a small MLP. Like Mamba, it aims for sub-quadratic scaling. Key application: Evo · a long-context DNA foundation model (~131K context, genome modeling) that uses a Hyena-based architecture. Evo demonstrated Hyena's viability on real-world sequence-modeling tasks.

Hyena operator: learn a function f(t) that generates convolution kernels on-the-fly based on relative position. This implicit convolution is computed in frequency domain via FFT, giving O(N log N) complexity. Together AI's Evo (131K context, biology) showed Hyena works on long-sequence biology tasks. For text, Hyena has seen less adoption than Mamba · the architecture is harder to tune and lacks Mamba's selective-state flexibility.

Hyena scales have been shown up to ~7B parameters for language; Evo ships at 7B with biology-specific training. FFT-based computation is efficient on GPU but requires careful implementation for long contexts. Comparison to Mamba: Mamba's selective update is more expressive; Hyena's convolution is simpler but less content-aware. Hyena's main legacy may be biology · Evo's demonstration on DNA modeling is a distinct win from frontier text modeling.