Vector search represents a paradigm shift from keyword matching to semantic understanding. By converting text into dense vector representations using models like BERT, E5, or BGE-m3, search systems can find conceptually similar content even when exact keywords differ. This article traces the evolution from early word2vec embeddings through transformer-based models to modern production systems. We examine approximate nearest neighbor (ANN) algorithms including HNSW, IVF, and product quantization that make billion-scale vector search practical. Integration patterns with traditional lexical search (hybrid search) combine the precision of keyword matching with the recall of semantic search. Practical considerations include embedding model selection, vector dimensions vs accuracy tradeoffs, index update strategies, and monitoring embedding drift over time.
Tag: vector-search
-
Understanding Hybrid Search: Combining Vector and Lexical Approaches
Hybrid search represents a paradigm shift in information retrieval. By combining traditional lexical (keyword-based) search with modern vector (semantic) search, we can achieve results that are both precise and contextually relevant.
Lexical search excels at exact matches — when a user searches for “PHP 8.3 migration guide”, lexical search finds documents containing those exact terms. However, it fails at understanding intent. A search for “how to upgrade my scripting language” won’t match documents about PHP migration.
Vector search solves this by encoding queries and documents into high-dimensional vector spaces using embedding models like E5-large-instruct. Semantically similar content clusters together, so “upgrade scripting language” lands near “PHP migration” in vector space.
The {!bool} query parser in Apache Solr combines both approaches in a single request. Lexical scores from edismax and KNN vector scores are summed, with configurable weights controlling the balance. Union mode surfaces hits from either signal; intersection mode requires both.
Key tuning parameters include: lexical_weight (0.1 = semantic-dominant, 1.0 = full lexical), vector_topk (candidate pool size), mm (minimum match), and quality_boost (content richness scoring).