Embedding

An embedding turns text into a fixed-size vector of numbers. Similar meanings produce nearby vectors · "dog" and "puppy" map to similar points, "dog" and "car" to distant points. Embeddings are the foundation of RAG, semantic search, clustering, and recommendations. Production embedding models: text-embedding-3-large (OpenAI), Voyage 3, Cohere embed, NV-Embed (NVIDIA).

Modern embedding models are trained with contrastive learning: pairs of semantically related text (e.g., query and relevant doc) are pulled close in embedding space while random pairs are pushed apart. Dimensions range from 256 (fast) to 8192 (highest quality). Cosine similarity is the dominant distance metric. Good embedding models hit 70%+ MTEB leaderboard scores. Costs: $0.05-0.20/M tokens on the major APIs. Embedding a large corpus is a one-time cost; serving queries is fast and cheap.

Training objective: contrastive NT-Xent loss or InfoNCE. Hard negative mining (sampling confusingly-similar negatives) improves retrieval quality. Dimensionality trade-offs: 1536D gives strong retrieval on MTEB but 8GB per million items in memory. Matryoshka embeddings (truncate to shorter dims at query time) give flexibility. Domain adaptation via in-domain fine-tuning improves retrieval 10-30% on specialized corpora. Evaluated with MTEB (Massive Text Embedding Benchmark) across 56 tasks. Top 2026: NV-Embed-v2 (73.7), Voyage-3 (~73), text-embedding-3-large (~70).