Getting a Model Running

This is where the course stops explaining and starts doing.

If you do not own a Spark, read it anyway. The three things that break here explain more about how this machine works than any spec sheet, and none of it needs the hardware to follow.

Every step below was run on a real Spark. Where a step exists because something went wrong, that is said plainly.

Link to First, what a terminal isFirst, what a terminal is

If you have never used one, start here. Nothing later makes sense without it.

A terminal is a window where you type instructions instead of clicking things. You type a line, press enter, the machine does it and prints something back. Linux, which is what the Spark runs, is normally driven this way.

Two ways to get one.

Plug a monitor, keyboard and mouse into the Spark. It boots to a desktop like any computer, and there is an app called Terminal. Simple, and it means walking to the machine every time.

Or reach it from your own laptop, which is what most people settle on. Open Terminal on a Mac or PowerShell on Windows and type:

ssh yourname@spark-1234.local

ssh stands for secure shell. It gives you a terminal on the other machine, encrypted, as though you were sitting at it. Your laptop becomes a keyboard and a screen. The Spark does the work.

yourname is the account you made during setup. spark-1234.local is the name your Spark announces on your home network, which NVIDIA's setup app shows you. Its network address works too.

Link to How to read the command blocksHow to read the command blocks

Worth stating once:

• A block without a $ is a command to type.
• A block with a $ shows a command and what the machine printed back. Do not type the $ itself.
• A \ at the end of a line means the command continues on the next one. Copy the whole block.
• Anything in <angle brackets> is a placeholder. Replace it, brackets included.

Link to Three things that breakThree things that break

Knowing them in advance turns each from a lost afternoon into a minute.

Link to The GPU looks broken on first bootThe GPU looks broken on first boot

Run the standard graphics command and you may get this:

$ nvidia-smi
Failed to initialize NVML: Driver/library version mismatch

This looks alarming and is almost always harmless. It happens on any Linux machine after a graphics driver update: the new driver files are installed, the running system still has the old one loaded, and the two disagree, so the tool refuses to answer.

sudo reboot

sudo means "do this as an administrator," and it will ask for your password. After the reboot:

$ nvidia-smi --query-gpu=name,driver_version,compute_cap --format=csv,noheader
NVIDIA GB10, 580.173.02, 12.1

Note that last number, 12.1. It matters shortly.

Link to This is not an Intel machineThis is not an Intel machine

The Spark's processor is ARM, the same family as your phone or a modern Mac. Written down it appears as aarch64 or arm64.

Almost every server and desktop in the world is a different family called x86_64. Software has to be built for one or the other, and something built only for x86 will not run here.

You meet this when installing things. If a tool refuses to install, or tries to build itself from scratch and fails, this is usually why.

Link to Your chip is newer than some software expectsYour chip is newer than some software expects

That 12.1 from earlier is the chip's compute capability, a version number NVIDIA gives each GPU generation. In code it is written sm_121.

Software that runs on GPUs is often compiled ahead of time for particular versions. sm_121 is new enough that some projects have not caught up, so this one number decides which tools work well on your machine.

The short version: vLLM is a restaurant kitchen built for a dinner rush. llama.cpp is a home kitchen. If you are one person asking one question at a time, a home kitchen is what you want.

Link to Install the one missing pieceInstall the one missing piece

sudo apt install -y libssl-dev

apt is Ubuntu's software installer. -y answers yes to its confirmation question so it does not stop and wait for you.

Why this package: the Spark already ships with the tools for building software, so cmake, gcc and git are there. What it lacks are the files a program you compile yourself needs in order to make encrypted web connections.

Skip it and llama.cpp still builds. It loses the ability to download models over an encrypted connection without saying so, and the download step later fails with a confusing message about HTTPS not being supported. This step exists because I skipped it the first time and lost an evening to it.

Link to Build llama.cppBuild llama.cpp

llama.cpp is the program that loads a model and runs it. You compile it yourself, which sounds intimidating and is three commands.

Compiling means turning human-readable source code into a program your specific machine can run. Most software you install was compiled by somebody else for a common machine. Yours is not a common machine, which is why you do it here.

git clone --depth 1 https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

git clone downloads the source code. --depth 1 takes only the current version rather than the entire history, which is far smaller. cd moves you into the folder it created.

cmake -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=121 \
  -DLLAMA_OPENSSL=ON

cmake works out how to build the software for your machine. Each option:

• -B build puts everything it generates in a folder called build, keeping it away from the source.
• -DCMAKE_BUILD_TYPE=Release builds the fast version rather than the one meant for debugging.
• -DGGML_CUDA=ON is the important one. CUDA is NVIDIA's system for running ordinary software on a GPU. Leave this out and everything runs on the CPU, roughly twenty times slower.
• -DCMAKE_CUDA_ARCHITECTURES=121 targets your exact chip, the 12.1 from earlier. llama.cpp detects this on its own when you build on the machine itself, so this is belt and braces.
• -DLLAMA_OPENSSL=ON switches on the encrypted downloads that libssl-dev made possible.

You will see it print Replacing 121 with 121a. That is correct and not an error. The a version carries some Blackwell-specific instructions.

cmake --build build --config Release -j 20

This does the actual compiling. -j 20 uses twenty processor cores at once, because the Spark has twenty. On a different machine you would use however many it has.

Roughly twenty minutes. When it finishes, check the GPU was found:

$ ./build/bin/llama-server --list-devices
Available devices:
  CUDA0: NVIDIA GB10 (124610 MiB, 78190 MiB free)

There is the single shared memory pool from lesson two, seen from the software's side. If nothing about CUDA appears, the build did not pick up the GPU and running a model will be painfully slow.

Link to Get a modelGet a model

Models live on Hugging Face, at huggingface.co. It works like a code-sharing site, except the things being shared are those giant files of numbers.

You want a packaging format called GGUF, which puts a model into a single file that llama.cpp reads directly. Very large models are sometimes published split across several numbered files instead, and the download command below handles that for you.

One command downloads and runs it:

./build/bin/llama-server -hf OBLITERATUS/Qwen3.8-27B-OBLITERATED:Q4_K_M

-hf means "fetch this from Hugging Face." The part before the slash is who published it, the part after is the model, and the bit after the colon is the quantization level from lesson four. Q4_K_M is the sensible default.

Two things that will confuse you:

It prints nothing while downloading if you have sent the output to a file. It looks frozen and it is not. Watch the file growing in ~/.cache/huggingface/hub/ if you want reassurance.

Some models are gated, meaning the publisher wants you to accept terms before downloading. Google's Gemma models are gated. If a download fails with a permissions error, that is why, and you accept the terms on the model's page first. The model above is not gated.

Link to Serve itServe it

Once the file is on disk, start the server properly:

./build/bin/llama-server \
  --model <path to your .gguf file> \
  --host 0.0.0.0 \
  --port 8080 \
  --ctx-size 32768 \
  --spec-type draft-mtp

What each part does:

• --model is the file you just downloaded.
• --host 0.0.0.0 lets other devices on your network reach it. Without it, only the Spark itself can. It also means anyone on your network can use it, which lesson six fixes with a password.
• --port 8080 is the numbered door it listens at. A port is how one machine offers several services at once without them colliding.
• --ctx-size 32768 is the conversation length from lesson four. Its memory cost depends on the model, so check yours.
• --spec-type draft-mtp is covered below and is worth having.

Guides online often add --n-gpu-layers 999, --jinja and --flash-attn auto. Those were all necessary once and are now the defaults, so they change nothing.

Link to Check that it workedCheck that it worked

The most common silent failure here is a model that loaded onto the CPU instead of the GPU. It works. It is twenty times slower, and nothing tells you.

The obvious command gives you nothing useful:

$ nvidia-smi --query-gpu=memory.total --format=csv
memory.total [MiB]
[N/A]

Lesson two explained why: there is no separate graphics memory to report on. But the same tool can tell you what each program is using:

$ nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv
931202, llama-server, 32660 MiB

If your server appears there holding gigabytes, the model is on the GPU. If the list is empty, it is not, and the thing to check is that -DGGML_CUDA=ON was in your cmake command.

Speed is the other confirmation. The server logs every request:

eval time = 1806.33 ms / 24 tokens (12.73 tokens per second)

Double figures means the GPU. One or two tokens per second means the CPU.

Link to Open it in a browserOpen it in a browser

Go to http://<your machine name>:8080 from any device on your network.

You get a chat interface, and a fair question is where it came from, since you never installed one.

It is built into the server. llama.cpp ships a small web app inside its source code, and compiling the server compiles that in too. There is no separate install and no web server to configure. Ask the server for its home page and it hands you the interface.

That is your first conversation with a model running on hardware you own.

Link to A free speedup already in your model fileA free speedup already in your model file

Look back at the startup log for lines like this:

W model has unused tensor blk.64.nextn.eh_proj.weight (29 MB) -- ignoring

That is a multi-token prediction head, usually shortened to MTP. Some models ship with one and most tools ignore it by default.

Lesson three explained the hard limit: to write one token the model reads itself end to end, so twenty tokens means twenty full reads. MTP works around it. A small extra part guesses the next few tokens, and the main model checks all of those guesses in a single read instead of one read each.

Picture a fast assistant scribbling down the next three words while an expert glances at them and asks one question: is that what I was going to write? Guesses the expert agrees with are kept. Guesses it disagrees with go in the bin, and the expert writes their own word.

The assistant never gets a vote. Nothing it suggests survives unless the main model would have produced it anyway, which is what makes this a speed trick rather than a change to what your model says.

Measured on a Spark with a 27B model:

Between 1.6 and 2.2 times faster for one flag and no extra memory, because the part doing the work was already in the file you downloaded.

Link to Why the two numbers differWhy the two numbers differ

Everything depends on how often the guesses are right, which is called the acceptance rate. Measured over 100 guesses on that machine:

Guessing one word ahead is easy and three is hard, and that decay is why you get a useful speedup rather than a threefold one. It also explains code beating prose: code is predictable, so more guesses land.

Link to If something goes wrongIf something goes wrong

Three checks, in order:

1. nvidia-smi shows the GPU, and if it reports a driver mismatch, reboot.
2. nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv lists your server. If it does not, the model is on the CPU.
3. free -h shows memory available. If very little is free, something else is still loaded.

Those three cover most of what goes wrong here.

Lesson six turns this into something that survives a reboot and that you can reach from anywhere.