trendingAI & ML — INTERMEDIATE
Vector Databases for Engineers: Index Types & Metadata Filtering
“Find the closest vectors” hides two real engineering decisions: which index structure to build, and how to combine that search with the ordinary metadata filters every production query actually needs.
The Quick Answer
Vector databases pick from a small set of approximate nearest neighbor (ANN) index types — most commonly HNSW, IVF, or LSH — each trading off speed, recall, and memory differently. Production queries almost always also carry a metadata filter (“only in-stock items,” “only documents from this user”), and whether that filter runs before or after the similarity search meaningfully changes both correctness and performance.
The Three Common Index Types
| Index | How it narrows the search | Trade-off |
|---|---|---|
| HNSW | Multi-layer graph — jump via long-range links at the top, refine locally at the bottom. | High recall, low latency; heavier memory footprint. |
| IVF | Pre-clusters vectors (e.g. via k-means), searches only the closest clusters. | Smaller memory footprint, tunable via how many clusters get scanned; somewhat lower recall than HNSW at the same speed. |
| LSH | Hashes vectors so similar ones fall in the same bucket; only compares within a query’s bucket. | Simple, no training step; needs many hash tables for strong recall, and degrades as dimensions grow. |
| Late interaction (ColBERT-style) | Skips single-vector pooling entirely — keeps one vector per token and compares token-to-token at query time. | Finer-grained matching; far more storage and compute per document than HNSW, IVF, or LSH. |
A practical default: reach for HNSW when memory is available and recall matters most, IVF (often paired with quantization) when the corpus is huge and memory is the binding constraint, and LSH mainly when you need a simple index with no training phase.
A Fourth Approach: Late-Interaction (ColBERT-Style) Retrieval
HNSW, IVF, and LSH all assume the same thing going in: every document (or chunk) gets reduced to a single pooled embedding before it’s indexed, and search is one vector-to-vector comparison. Late-interaction retrieval, introduced under the name ColBERT, changes that starting assumption instead of just indexing differently. Rather than pooling a whole passage into one vector, it keeps a separate embedding for every token in the document. A query is embedded the same way — one vector per query token — and at search time each query-token vector is compared against every document-token vector, keeping only the best match per query token (an operation called MaxSim), then summing those best-matches into a final score.
The trade-off is mechanical, not magic: storing one vector per token instead of one vector per document means dramatically more storage and more compute per document — a 200-token passage now needs roughly 200 stored vectors instead of 1. What that buys back is precision a single pooled vector can’t offer — a query token can match whichever specific token in the document is actually relevant, instead of the whole passage being judged by one averaged-out representation that can dilute a sharp, narrow match. That makes it most attractive as a reranking step over a small candidate set (say, the top 50 – 100 results an HNSW or IVF pass already narrowed down), rather than as the first-pass index over an entire multi-million-document collection.
This isn’t just an academic idea anymore — Qdrant added native multivector support built specifically to run late-interaction models like ColBERT, storing per-token vectors and scoring them with MaxSim as a built-in query stage rather than something you bolt on yourself.
Source: Qdrant, “Qdrant 1.10 - Universal Query, Built-in IDF & ColBERT Support”
Metadata Filtering: Pre- vs. Post-Filter
Real queries are rarely just “find similar vectors” — they’re “find similar vectors where category = electronics” or where user_id = this user. There are two ways to combine that filter with the similarity search:
- Pre-filtering — narrow to matching documents first, then run ANN search only on that smaller set. Cheap when the filter is highly selective.
- Post-filtering — run ANN search first to get the closest candidates, then discard the ones that fail the filter. Simple and works with any index, but risks returning too few results if the filter is narrow and most of the initial candidates get discarded.
Try both strategies below against the same 1-million-document collection, under a narrow filter and a broad one.
Hybrid Search
Vector search is excellent at meaning but can miss an exact term that genuinely matters — a product SKU, a specific error code, a proper noun the embedding model never learned to weight heavily. Hybrid search runs a vector search and a traditional keyword search side by side, then merges both result sets — combining the meaning-matching strength from embeddings with the precision of exact keyword matching, rather than picking one at the expense of the other.
Fun Fact
IVF’s clustering step usually runs k-means once, up front, to decide the cluster boundaries — meaning the quality of every future search depends on how well that initial clustering pass captured the actual shape of your data. A poorly-clustered index can search fast and still miss the right answer, illustrating why speed and correctness are genuinely separate concerns here, not the same thing measured two ways.
Test Yourself
What is the main trade-off HNSW makes compared to IVF?
How does IVF (Inverted File Index) narrow down a search?
When does pre-filtering (filter, then search) work best?
What risk does post-filtering (search, then filter) carry with a narrow filter?
What does hybrid search combine?
What makes late-interaction (ColBERT-style) retrieval mechanically different from HNSW, IVF, and LSH?
Ready for a worked compression example with real numbers, plus actual metadata filtering code and why getting it wrong can silently drop the best match? Continue to the Advanced level →