How is a neural network layer’s computation a matrix multiplication?

A layer’s output is computed by multiplying its input vector (or a batch of input vectors stacked into a matrix) by a matrix of the layer’s learned weights, then adding a bias vector. Every neuron in the layer corresponds to one column of that weight matrix, so a layer with many neurons computing many outputs at once is exactly one matrix multiply, not many separate calculations.

Is a transformer’s forward pass really just matrix multiplications?

Yes, almost entirely. Vaswani et al.’s "Attention Is All You Need" defines the query, key, and value projections as linear (matrix) transformations, the attention scores as a matrix multiply between queries and keys, the attention output as a matrix multiply between attention weights and values, and the feed-forward block as two more linear (matrix) layers. Softmax normalization is the one common non-matrix-multiply step in that chain.

Why are GPUs so much better than CPUs at running AI models?

Every output cell in a matrix multiply is computed independently of every other cell, which means the work can be split across many processors at once with no cell waiting on another. GPUs pack thousands of simpler cores built for exactly this kind of parallel arithmetic — NVIDIA’s first Tensor Core GPU (Tesla V100) added 640 cores purpose-built to accelerate matrix multiply-accumulate operations, reaching a theoretical peak of 125 teraflops/s in mixed precision.

← Back to the Introduction level

AI & ML — INTERMEDIATE

Matrix Multiplication in Transformers: Layers, Batching, and GPUs

The Introduction level covered how matrix multiplication works and how one neuron is a tiny version of it. This level scales that up: a full neural network layer computed at once, a real transformer layer traced step by step, batching’s real operation-count math, and exactly why matrix multiplication is the operation GPUs were built to accelerate.

The Quick Answer

A neural network layer computes input × weights + bias, one matrix multiply per layer. A transformer stacks many such layers, and — as the architecture paper “Attention Is All You Need” lays out — even its distinctive attention mechanism is built from matrix multiplies: projecting the input into queries, keys, and values is a matrix multiply; comparing every query against every key is a matrix multiply; combining values by their attention weights is a matrix multiply. Softmax is the one non-matmul step in that chain. Batching multiple inputs together turns many small matrix multiplies into one larger one with identical total arithmetic but far better hardware utilization — and GPUs, with thousands of cores built for exactly this kind of independent, parallel arithmetic, are what make running this at scale practical at all.

A Full Layer, Computed at Once

The Introduction level showed one neuron as a single row times a single column. A real layer has many neurons reading the same input, each with its own weights — which means the single column of weights becomes a full weight matrix, one column per neuron, and the layer’s entire output becomes one matrix multiply instead of many separate weighted sums. Step through a 2-output layer computed for real below:

This exact shape — input times weights, plus bias — is what every dense layer in a neural network computes, and it is also, as covered further below, what each of a transformer’s query, key, value, and feed-forward projections computes. For the broader architecture this fits into, see How LLMs Work — Advanced.

Tracing a Real Transformer Layer’s Matrix Multiplies

Click through each step of one transformer layer below to see which parts are matrix multiplies, and the real shapes involved for an illustrative model size:

Self-Attention, Traced as Matrix Multiplies

Attention gets its own name and its own diagrams, but the arithmetic underneath is the same row-times-column operation from the Introduction level, run several times in a row. Step through a tiny, fully worked 2-token example below — every number is computed for real from the fixed toy input and weights shown:

Batching: Same Arithmetic, Fewer Operations

The Introduction level showed batching conceptually. Here’s the real operation count: batching B inputs of size d_in through a layer producing d_out outputs does the exact same total number of multiply-adds whether you run it as B separate small multiplies or 1 larger one. Adjust the sliders below and watch the totals stay locked together while the operation count changes:

Why GPUs Are Built For This

Every output cell in a matrix multiply only depends on one row of the first matrix and one column of the second — never on any other output cell. That independence is what makes the work splittable across many processors at once, with no cell waiting on another’s result. A CPU has a small number of powerful cores built for doing many different kinds of tasks well; a GPU packs thousands of simpler cores built to do exactly this one kind of independent, repetitive arithmetic in parallel. Try the comparison below:

Fun Fact

NVIDIA’s Tesla V100, the first GPU with dedicated Tensor Cores, added 640 of them specifically to accelerate matrix multiply-accumulate operations, reaching a theoretical peak of 125 teraflops/s in mixed FP16/FP32 precision — up to 12x higher peak throughput for training than the prior Pascal-generation GPUs that came before Tensor Cores existed.

Source: NVIDIA Developer Blog, “Volta Tensor Core GPU Achieves New AI Performance Milestones”

Test Yourself

A neural network layer has 4 input features and 6 output neurons. What shape is its weight matrix?

You process 16 inputs one at a time versus stacking them into one matrix multiply. What is true about the total arithmetic in both cases?

Why does a matrix multiply parallelize well across many processor cores?

In self-attention, which of these steps is NOT a matrix multiply?

Brainstorm

No single correct answer here — think it through, then expand each one to see one way to reason about it.

A layer with 512 inputs and 512 outputs, and a layer with 512 inputs and 2,048 outputs, both process the same batch of tokens. Which one does more arithmetic, and by roughly how much can you estimate that without a calculator?

If softmax isn't a matrix multiply, why does it still matter for how fast a transformer runs?

You have a batch of 32 requests, but they arrive one at a time over several seconds instead of all at once. What tension does that create for a system trying to batch them into one matrix multiply?

Multi-head attention runs several attention "heads" with different weights on the same input. Why might it be better to think of this as one bigger batched matrix multiply rather than several small separate ones?

If you doubled the number of layers in a transformer but kept every layer the same size, what would you expect to happen to the total number of matrix multiplies per forward pass, and why?

Ready for real FLOP counts, GPU tiling, batching throughput backed by a real serving benchmark, and DeepMind’s AlphaTensor finding a genuinely faster way to multiply matrices? Continue to the Advanced level →