Fine-tuning

Fine-tuning takes a pre-trained foundation model and continues training it on task-specific data. Use cases: customer-support tone, domain knowledge (legal, medical), structured output formats, specific language dialects. Methods range from full fine-tuning (update all parameters, expensive) to LoRA or QLoRA (update only a small adapter, cheap). Most production fine-tuning uses LoRA in 2026.

Fine-tuning recipes: Supervised Fine-Tuning (SFT) on labeled demonstrations · the most common. Direct Preference Optimization (DPO) on preference pairs · preferred over RLHF for alignment because it skips reward modeling. RLHF still used for safety-critical alignment. LoRA (Low-Rank Adaptation) trains a small adapter matrix while freezing base weights · 10-100× cheaper than full fine-tune, recoverable to base. QLoRA adds 4-bit quantization to LoRA for consumer-GPU training. Cost: a 7B model full fine-tune ~$5-20K on rented H100s; same workload with LoRA ~$50-500. Fine-tuning is reversible by discarding the adapter.

LoRA approximates the fine-tune delta ΔW as BA where B ∈ R^{d×r} and A ∈ R^{r×k} with r << min(d, k). Typical rank r = 8-64. Adapter parameters count at LoRA rank 16: <0.1% of full weight count. QLoRA adds NF4 quantization of the frozen base to run fine-tuning on single-GPU setups · degrades quality ~1-2 points. DPO loss: -log σ(β * (log π_θ(y_w|x) - log π_θ(y_l|x) - log π_ref(y_w|x) + log π_ref(y_l|x))). Catastrophic forgetting is the main risk · mitigate with low learning rates (1e-5), limited epochs (1-3), and mixing general-purpose data into the training set.