What is vector projection and how does it relate to the dot product?

Vector projection is the "shadow" one vector casts onto the direction of another. The scalar projection of A onto B equals A dotted with the unit vector in B's direction — the same dot product operation, just applied after normalizing one side, which gives a length rather than an abstract number.

How exactly does the dot product get used in transformer attention?

In scaled dot-product attention, every query vector is dotted against every key vector to produce a matrix of raw relevance scores (QKᵀ). Those scores are divided by the square root of the key dimension (√d_k) to keep them in a numerically stable range, then passed through softmax to turn them into attention weights that sum to 1 per query.

Why does scaled dot-product attention divide by √d_k?

As vector dimension grows, dot products between random vectors tend to grow larger in magnitude simply because there are more terms being summed. Without scaling, these larger values push softmax into regions where its gradient is nearly zero, which stalls learning. Dividing by √d_k keeps the scores in a range where softmax still has a usable gradient.

Why don't vector databases compare a query against every stored vector with a brute-force dot product at large scale?

Because the cost scales directly with both the vector dimension and the number of stored vectors — a brute-force search over a billion vectors at 1,536 dimensions each requires well over a trillion multiply-add operations per query. Approximate nearest neighbor indexing (HNSW, IVF, and similar) exists specifically to avoid computing the overwhelming majority of these comparisons.

← Back to the Intermediate level

AI & ML — ADVANCED

Dot-Product Attention & Vector Projection: The Full Mechanics

The Intermediate level covered the dot product’s geometric meaning and cosine similarity. This level goes further in two directions: the dot product’s role as a projection (a literal “shadow” one vector casts on another), and its role as the core operation inside a transformer’s attention mechanism — plus the real computational cost of comparing vectors at scale.

The Quick Answer

Beyond measuring similarity, the dot product has a second geometric reading: A · unit(B)is exactly how far vector A extends along vector B’s direction — its projection. That same operation, scaled and passed through softmax, is the core of scaled dot-product attention: every query vector is dotted against every key vector, producing a score matrix that determines which tokens a transformer pays the most attention to. And because a dot product costs roughly one multiply-add per dimension, comparing a query against millions or billions of stored vectors by brute force gets expensive fast — which is exactly why approximate nearest neighbor search exists.

Vector Projection: The Dot Product as a Shadow

Take vector A and vector B. The scalar projection of A onto B — how far A reaches along B’s direction — is computed as A dotted with the unit vector in B’s direction:

\text{proj}_B(A) = A \cdot \frac{B}{|B|}

Geometrically, this is the length of the perpendicular shadow A would cast if a light shone straight down onto the line running through B. It’s the same dot product operation covered throughout this topic — just interpreted as a literal distance instead of an abstract similarity score.

Scaled Dot-Product Attention

Inside a transformer, every token position produces a Query, a Key, and a Value vector. Attention scores how relevant every other token’s Key is to the current token’s Query using exactly the dot product covered on this page:

\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^{T}}{\sqrt{d_k}}\right)V

The √d_k scaling factor matters because dot products between random vectors tend to grow larger in magnitude as dimension grows, simply from summing more terms — left unscaled, this pushes softmax toward near-zero-gradient regions and stalls learning. The full derivation of this formula, its Q/K/V shapes, and a worked NumPy implementation are covered in How LLMs Work — Advanced; this page focuses specifically on the dot-product mechanics that make it work at all.

Try a minimal version below: one query, a few keys, real dot products computed live, and the resulting attention weights after scaling and softmax:

Source: Vaswani et al., “Attention Is All You Need,” NeurIPS 2017

Every Query Against Every Key at Once

In practice, a transformer doesn’t score one query against one key at a time — it computes Q K^T in a single matrix multiplication, producing every pairwise dot product at once. Hover over a cell below to see exactly which query and key produced that score:

The Real Cost of Comparing Vectors at Scale

A single dot product costs roughly one multiply-add operation per dimension. That’s trivial for one comparison — but a similarity search that brute-force compares a query against every vector in a large collection multiplies that cost by the size of the collection. At real production scale, this adds up fast enough to matter:

This is exactly the problem approximate nearest neighbor indexing solves, covered in detail in Vector Database — Intermediate. The dot product itself is never the bottleneck in isolation — doing it billions of times per query is.

Fun Fact

The Transformer architecture’s original paper explicitly chose dot-product attention over an alternative, additive attention mechanism, specifically because dot products can be computed with highly optimized matrix multiplication code — making it faster in practice despite the two approaches having similar theoretical complexity.

Source: Vaswani et al., “Attention Is All You Need,” NeurIPS 2017

Test Yourself

What does the scalar projection of A onto B (computed as A · unit(B)) represent geometrically?

In scaled dot-product attention, what does the QKᵀ matrix contain before scaling and softmax are applied?

Why does scaled dot-product attention divide the QKᵀ scores by √d_k before applying softmax?

Why does brute-force nearest-neighbor search become impractical at large scale (e.g. a billion stored vectors)?

What is the practical fix for the cost problem in the previous question?

Brainstorm

Open-ended questions worth pausing on before expanding the model answer.

Vector projection and the dot product formula A·B = |A||B|cos(θ) are two different-looking ideas that turn out to be deeply connected. Why do you think the same underlying operation (the dot product) supports two seemingly different interpretations — "alignment" and "shadow length"?

The QKᵀ matrix in attention scores every token against every other token, including itself. Why do you think a token is allowed to attend to itself, rather than only to other tokens?

If you removed the √d_k scaling factor from attention entirely, what specifically do you think would go wrong during training, based on how softmax behaves?

The brute-force search cost calculator shows trillions of operations for a billion vectors. Given that scale, why do you think approximate (not exact) nearest-neighbor search is an acceptable trade-off for most real search systems?

Multi-head attention runs several of these dot-product attention computations in parallel with different learned projections. Why might one single dot-product attention pass not be enough to capture everything a model needs from a sequence?