What is matrix multiplication, in plain terms?

Matrix multiplication combines two grids of numbers into a new grid, where each entry in the result comes from multiplying a row from the first matrix against a column from the second matrix, term by term, and adding those products up. It is not the same as multiplying two matrices entry by entry — each output entry mixes an entire row with an entire column.

Why does AI use matrix multiplication so much?

A neural network layer computes its output by multiplying an input vector (or a batch of input vectors, stacked into a matrix) by a matrix of learned weights, then adding a bias. Every layer in a neural network, including every layer of a transformer, repeats this same operation. That is why matrix multiplication is not just one topic in AI math among many, but the single operation that dominates how much computing an AI model actually needs.

Do two matrices always have to be a certain shape to be multiplied?

Yes. To multiply matrix A by matrix B, the number of columns in A must equal the number of rows in B. If A is m rows by k columns and B is k rows by n columns, the result is an m by n matrix. If those inner numbers do not match, the multiplication is undefined and cannot be performed.

AI & ML — INTRODUCTION

Matrix Multiplication: The Math Behind Every AI Model

Every time an AI model turns your question into an answer, underneath all the talk about “neural networks” and “transformers,” the actual computer chip is doing one thing, over and over, billions of times: multiplying grids of numbers together. That operation is called matrix multiplication, and this page teaches you exactly how it works and why it is the operation AI runs on.

The Quick Answer

Matrix multiplicationcombines two grids of numbers (matrices) into a new grid. Each number in the result comes from taking one full row of the first matrix and one full column of the second matrix, multiplying them position by position, and adding those products up. A neural network layer’s entire job — turning an input into an output — is computed as exactly this operation: an input multiplied by a matrix of the model’s learned weights. Stack enough of these layers together and you have a full AI model, which is why matrix multiplication isn’t a side topic in AI math — it is the operation an AI model spends nearly all of its computing time on.

What Matrix Multiplication Actually Is

A matrix is just a grid of numbers arranged in rows and columns. Multiplying two matrices together is not like multiplying two spreadsheets cell by cell — it’s a specific rule: to get one number in the result, you take a full row from the first matrix and a full column from the second matrix, multiply each pair of numbers that line up, and add all of those products together. Do that for every row/column pair, and you get the full result matrix.

The best way to see this is to compute it yourself, one cell at a time:

Why Shapes Have to Match

You can’t multiply just any two matrices together. For A × B to work, the number of columns in A has to equal the number of rows in B — because that shared number is exactly how many terms get multiplied and added for each output cell, as you just saw above. If A is 2 rows by 3 columns, B needs exactly 3 rows (any number of columns) for the multiplication to be defined at all. Try a few combinations below and see which ones work:

One Neuron Is Just a Row Times a Column

Here’s the connection to AI: a single artificial neuron takes a list of input numbers, multiplies each one by its own learned weight, adds those products up, and adds one more number called a bias. That is a 1-row matrix (the input) multiplied by a 1-column matrix (the weights) — the exact same row-times-column arithmetic from above, just with a fixed name in AI: a weighted sum. Try it with real numbers:

A real neural network layer has many neurons like this side by side, all reading the same input but each with its own weights — which turns the single column of weights above into a full matrix, and the single neuron above into one full matrix multiply. This idea is covered in more depth, with a full layer computed at once, at the Intermediate level.

Why Many Inputs Get Stacked Into One Matrix

When an AI model has several inputs to process — several sentences, several images, several requests from different users at once — it doesn’t have to process them one at a time. Instead, the inputs get stacked as rows of a single matrix and multiplied against the weights in one shot. This is called batching, and it is a large part of why AI systems can serve many users at once without slowing to a crawl. Try comparing the two approaches:

Fun Fact

Multiplying two n×n matrices the ordinary, textbook way takes roughly n³ individual multiplications — for two 1,000×1,000 matrices, that’s about a billion multiplications for a single matrix multiply. Real AI models perform matrix multiplies far larger than that, many times per layer, for every layer, for every response they generate — which is exactly why specialized hardware for this one operation turned out to matter so much, covered further at the Intermediate and Advanced levels.

Test Yourself

When you multiply matrix A by matrix B, how is each entry of the result computed?

A is a 2×3 matrix and B is a 3×2 matrix. Can you multiply A × B, and if so, what shape is the result?

Why does matrix multiplication show up in almost every layer of a neural network?

Brainstorm

These don’t have a single correct answer — they’re here so you pause and think before checking one way to reason about each one.

If matrix multiplication requires a row from A and a column from B to have the same length, what does that tell you about what "shape" really represents in a neural network layer?

Why might it matter that matrix multiplication mixes a whole row with a whole column, rather than just multiplying matching positions together?

You stack 8 separate sentences into one matrix instead of processing them one at a time. What do you expect to change, and what do you expect to stay the same?

If you had to guess, why do you think GPUs (originally built for rendering video game graphics) turned out to be so useful for AI, given what you now know about what AI models actually compute?

A single output number in a matrix multiply result never depends on any other output number in that same result. Why might that specific property be more important than the multiplication itself?

Ready to see a full neural network layer computed at once, real operation counts for batching, and why GPUs specifically are built for this kind of math? Continue to the Intermediate level →