Why Memory Decides Everything

In the last lesson you learned the Spark has one big pool of memory. This lesson is about how fast that pool can be read, because that single number decides almost everything about how the machine feels to use.

Link to The fact that explains everythingThe fact that explains everything

To produce one token, a model reads every number it needs for that token.

For most models it needs the entire file. Then to produce the next token, it reads the entire file again.

A 27 billion parameter model squeezed down to about 16 GB is reading all 16 GB to give you one token. Then all 16 GB again for the next one. (Lesson four explains how 27 billion numbers become 16 GB.)

Writing a sentence of twenty tokens means reading the whole model twenty times.

That is how these models work rather than a flaw in them, and once you accept it the machine's behavior stops being mysterious.

Link to So the speed question is a reading questionSo the speed question is a reading question

If producing a token means reading the whole file, then how fast the machine writes depends on how fast it can read from memory.

That reading speed has a name: memory bandwidth. It is measured in gigabytes per second, and it means what it sounds like, how much data can move out of memory each second.

The DGX Spark has about 273 GB/s.

Now the arithmetic writes itself:

tokens per second  ≈  memory bandwidth ÷ bytes read per token
 
273 GB/s ÷ 16 GB  ≈  17 tokens per second

Link to What that misses at firstWhat that misses at first

The model reads one more thing per token, and it grows.

Alongside the weights it re-reads its notes on the conversation so far, which lesson four covers under the name KV cache. Early in a chat those notes are tiny and the sum above holds. Deep into a long document they can rival the weights themselves.

The same 70B model, same machine, at two conversation lengths:

Same model, half the speed, and nothing broke.

This is worth knowing before it happens to you, because the machine appears to get slower the longer you talk to it, and people go hunting for a fault that is not there.

Seventeen is the ceiling. Measured speed on a real machine is around 12.6 tokens per second, about 74% of the theoretical number. The gap is overhead the arithmetic ignores. That ratio is not a law, and small models sit further from it than big ones, so treat it as a sanity check rather than a target.

You can run this for any model on any machine, which makes it the most useful calculation in the course.

Link to One important exceptionOne important exception

Some models do not read the whole file per token. They are called mixture-of-experts models, and instead of one big network they hold many smaller sub-networks. For each token a router picks a few and ignores the rest.

gpt-oss-120b holds 117 billion parameters and uses about 5 billion of them for any given token. Its file is around 63 GB, but it is only reading a slice of that each time, and it measures at roughly 59 tokens per second on a Spark. Judging it by its file size would have you expecting 4.

Do not try to derive 59 from the arithmetic above. Routing a token to the right experts costs time that the simple sum ignores, so these models land well under what their active size alone suggests. The number to take away is that they are far faster than their file size implies, and slower than a pure calculation would promise.

So the honest version of the rule is: divide bandwidth by the bytes read per token. Ordinary models read the whole file, which is why the shortcut works. For a mixture-of-experts model, look for "active parameters" on its page, and treat any figure you calculate as an optimistic ceiling.

Link to The trade you madeThe trade you made

Here is where the Spark's design shows its cost.

A high-end gaming card reads at roughly 1,800 GB/s, nearly seven times faster. On a model that fits in its 32 GB, it will bury the Spark.

But a 70 billion parameter model needs about 42 GB, and 42 will never go into 32. You can force it by keeping part of the model on the slower system memory, and then you are looking at one or two tokens per second, which nobody would sit through.

The Spark buys you room, and pays for it in speed.

Say that to anyone who asks whether it beats a gaming PC. The question has no single answer until you say which model.

Link to Why reading is fast and writing is slowWhy reading is fast and writing is slow

Now the strangest behavior you will notice, and the reason it happens.

Paste in a long document and the machine chews through it quickly. Then the answer comes out at a crawl.

Measured on a real Spark:

It reads about 66 times faster than it writes. That is expected, and here is the reason.

Reading can happen all at once. When you hand the model a thousand tokens, it can look at all of them at once. Those thousands of tiny GPU workers each take a piece. One pass over the model handles the entire prompt.

Writing cannot. The model has to produce token one before it can produce token two, because token two depends on how token one turned out. Each token needs its own full pass over the model. A single answer cannot be split up, though a server answering ten people at once can carry all ten along the same pass, which is why busy servers are so much more efficient than idle ones.

Link to What this means for how you use itWhat this means for how you use it

This asymmetry is genuinely useful once you plan around it.

The machine is good at jobs that read a lot and write a little. Summarizing a pile of documents, pulling structured data out of messy text, answering questions about your own files, classifying things. In all of these the input is huge and the output is small.

It is poor at generating volume. Asking it to write fifty long articles will take all night, and you will watch every word appear.

A concrete example. Feeding it two hundred video transcripts, roughly a million tokens, takes a bit under half an hour of reading. Asking it to write a detailed essay about each one would run all night.

Same machine. Same model. Completely different experience, decided entirely by which direction the words are flowing.

Link to Check your understandingCheck your understanding

You have an ordinary 8 GB model and a machine with 273 GB/s of bandwidth. Roughly how many tokens per second should you expect?

273 divided by 8 is about 34, so the ceiling is around 34 tokens per second, and something near 25 in practice.

Notice what that tells you: a smaller model is a faster model on this hardware. Not because it is simpler, but because there are fewer bytes to read for every single word. That fact drives the whole next lesson.