Why Vectors Need a Dedicated Database
Embeddings are high-dimensional float arrays (384–1536 dims). A SQL database can store them but cannot search them: “find the 5 most similar rows” means computing distance to every row — a full scan. Vector databases specialize in this with approximate nearest-neighbor (ANN) indexes that answer in milliseconds over millions of vectors.
| Property | SQL | Vector DB |
|---|---|---|
| Search | WHERE, indexes on keys | Similarity in embedding space |
| “Nearest” query | Not native | Native, milliseconds |
| Filters | First-class | Applied around ANN |
| Metadata + vectors | Awkward | Designed together |
Similarity Metrics
| Metric | When to use |
|---|---|
| Cosine similarity | Default for normalized text embeddings |
| Euclidean distance | When magnitude carries meaning |
| Dot product | Optimized, works with normalized vectors |
For normalized embeddings, cosine and dot product agree — and the metric choice is a speed-vs-interpretability decision.
Exact vs Approximate Search
Exact k-NN scans everything (O(n)) — fine for thousands of vectors, hopeless for millions. ANN indexes trade a small amount of recall for orders of magnitude speed: instead of guaranteeing the true top-k, they return the almost top-k. Two dominant designs:
HNSW: A Multi-Layer Graph
HNSW builds layers of neighbor graphs. Search starts at a sparse top layer, descends greedily, and zooms into the dense bottom layer containing the real nearest neighbors. The result: logarithmic-ish search time with excellent recall.
- M (connections per node) — higher M = better recall, more memory.
- efSearch — more candidates examined = better recall, slower.
IVF: Inverted File Index
IVF clusters the vectors (k-means) and stores each vector with its cluster. Search probes only the clusters nearest the query, pruning the search space. Add PQ (product quantization) to compress vectors for memory savings. IVF is memory-lean and well understood; HNSW is usually faster with a higher memory cost.
Hybrid Search and Filtering
Real queries mix similarity and structure: “docs about refunds from the last 30 days”. Hybrid search combines vector similarity with keyword (BM25) and/or metadata filters. A vector database that can’t filter pre-ANN — or re-rank correctly afterward — either returns wrong results or scans too much.
Tuning Recall vs Latency
| Knob | Effect |
|---|---|
| More candidates (efSearch / probes) | Recall ↑, latency ↑ |
| Larger M / more clusters | Recall ↑, memory ↑ |
| Fewer clusters probed | Latency ↓, recall ↓ |
There is no free lunch: pick the operating point that meets your recall target at the latency budget, then measure — never tune blind.
When It’s the Right Tool
A vector database is the right tool when you need similarity search over millions of embeddings with low latency: RAG retrieval, semantic dedup, recommendation by similarity, anomaly detection. For a few thousand vectors, an in-memory search over a NumPy array is simpler and cheaper. Choose by scale, not by default.
Practice Trajectory
- Explain why a full vector scan fails at million-scale.
- Describe HNSW search in three sentences.
- Choose a recall/latency operating point for a chatbot with a 200 ms budget.
- Design a hybrid query that combines similarity with a time filter.
- Decide between in-memory search and a vector DB for a 5k-doc wiki.