Chain of Thought

Chain-of-thought (CoT) was first shown in a 2022 Google paper: adding "Let's think step by step" to a prompt makes models solve problems they previously failed. Modern models trained on CoT data produce step-by-step reasoning by default. Reasoning models (o1, Claude Extended Thinking, DeepSeek R1) extend this further by generating internal CoT traces that are hidden from the user but billed as output tokens.

CoT exploits the transformer's auto-regressive generation: each token conditions on all previous tokens. By generating intermediate reasoning, the model has more compute per final-answer token and can catch errors in earlier steps. Zero-shot CoT ("Let's think step by step") was the original trigger. Few-shot CoT provides worked examples in the prompt. Models trained on CoT data (Chinchilla and later) produce CoT naturally. Self-consistency improves CoT by sampling multiple reasoning paths and voting on the final answer. Tree-of-Thought extends CoT to branching exploration. For tasks with verifiable answers, sample-and-verify schemes dramatically outperform single-shot CoT.

CoT's mechanism: each step token effectively adds a forward pass of compute. For an n-step reasoning problem, the total FLOPs are n times a direct-answer pass. This matches theoretical results that transformers need either depth or sequence length to solve certain problem classes. Self-Consistency samples k traces and picks the most common answer; works well when the reasoning distribution is unimodal around correct. Tree-of-Thought (2023) searches a tree of partial reasoning states with an explicit value function · more expensive but handles multi-modal reasoning distributions. Chain-of-Verification and Reflexion add self-critique loops · diminishing returns past 3-5 iterations.