Embeddings in Production: Matryoshka Compression & ANN Search

A Worked Example, With Code

Recall the formula from the Intermediate level:

Take two sentences that mean nearly the same thing, and one that doesn’t. Below are fixed, hand-authored 8-dimensional toy embeddings standing in for what a real embedding model would output (which would use hundreds or thousands of dimensions instead of 8):

import numpy as np

# Toy 8-dimensional embeddings for three sentences — illustrative fixed
# numbers standing in for what a real embedding model would output.
A = np.array([0.72, 0.61, 0.15, 0.08, 0.30, 0.05, 0.02, 0.10])  # "The dog barked at the mailman."
B = np.array([0.68, 0.58, 0.20, 0.12, 0.25, 0.08, 0.03, 0.09])  # "The puppy growled at the visitor."
C = np.array([0.05, 0.10, 0.85, 0.72, 0.02, 0.60, 0.55, 0.15])  # "The stock market crashed today."

def cosine_similarity(a, b):
    dot_product = np.dot(a, b)
    magnitude_a = np.linalg.norm(a)   # sqrt(sum of squares) — the vector's length
    magnitude_b = np.linalg.norm(b)
    return dot_product / (magnitude_a * magnitude_b)

print("cos(A, B) =", round(cosine_similarity(A, B), 4))
print("cos(A, C) =", round(cosine_similarity(A, C), 4))
# cos(A, B) = 0.9959
# cos(A, C) = 0.2444

“The dog barked at the mailman” and “The puppy growled at the visitor” score 0.9959— both sentences are about a small animal reacting to someone at the door, and the embedding places them almost on top of each other despite sharing zero words in common. “The stock market crashed today” scores only 0.2444 against the same reference sentence — different topic entirely, and the geometry reflects that.

Matryoshka Representation Learning

A full-size embedding costs real money to store and real compute to search — OpenAI’s text-embedding-3-large returns 3,072 numbers per piece of text by default. Matryoshka Representation Learning is a training technique that packs the most important signal into the leading dimensions on purpose, coarse-to-fine, so that truncating a vector down to its first N dimensions still yields a useful, meaningful embedding — unlike an ordinary embedding, where cutting it down usually destroys it.

The result OpenAI reported when they shipped this: text-embedding-3-large, truncated all the way down to just 256 dimensions, still outperformed the older text-embedding-ada-002 model running at its full 1,536 dimensions, on the MTEB benchmark. Fewer dimensions means less storage and faster comparisons — a direct, measurable cost saving at query time.

Source: OpenAI, “New Embedding Models and API Updates”

Try it below: the same two sentences from the worked example above, truncated to fewer and fewer dimensions.

Why Production Search Doesn’t Compare Against Every Vector

Computing cosine similarity between a query and one stored embedding is cheap. Doing it against every embedding in a database of millions, for every single search, is not — at the sub-50-millisecond latency real search systems need, brute-force exact comparison stops being viable well before you reach a million stored vectors.

Production vector databases instead use approximate nearest neighbor (ANN) algorithms — most commonly HNSW (Hierarchical Navigable Small World), which organizes vectors into a multi-layer graph so a search can jump straight to a promising neighborhood instead of checking everything. The trade-off is explicit: ANN search accepts a small, controlled chance of missing the true best match — typically still returning 95–99% of what exact search would — in exchange for running roughly 50–200× faster. Building and querying that index is its own topic, covered in the Vector Database article.

Source: Malkov & Yashunin, “Hierarchical Navigable Small World” (HNSW)

Fun Fact

Matryoshka Representation Learning takes its name from Russian nesting dolls — a smaller, complete version of the embedding sits inside the larger one at every truncation point, the same way each smaller doll is a complete doll in its own right, not a broken half of a bigger one.

Test Yourself