Latency

Latency (Time to First Token, TTFT) is the delay between hitting send and seeing the first character of the response. Good TTFT under 500ms · feels instant. 1-2s is acceptable for chat. Above 5s feels broken. Latency is distinct from tokens-per-second (how fast the response streams after the first token).

TTFT depends on: queue wait (how many requests are ahead), prefill time (proportional to input length), model size (bigger = slower), and geographic distance (speed of light from your server to theirs). Reasoning models add 30-120s thinking latency before the first visible token. Groq and Cerebras hit sub-100ms TTFT by using exotic hardware (LPU, wafer-scale). Most frontier APIs: 400-2000ms TTFT at p50.

Prefill cost is O(n) in input tokens for full attention, O(n) for sliding-window. Grouped-Query Attention reduces the multiplicative constant. KV cache reuse (prefix caching) eliminates prefill for repeated prompts · massive TTFT win for multi-turn chat with a stable system prompt. Claude's prompt caching and OpenAI's cached input pricing reflect this optimization. Geographic routing (regional endpoints) reduces network latency from 100ms+ to under 20ms.