michael's post
← All field notes
Field note

Running a 27B model across two desktops with llama.cpp RPC

10 mins readllmgpudocker
I have two machines with a 16 GB consumer GPU in each, and a model I wanted to run that does not fit in either one on its own. llama.cpp's RPC backend lets you treat the two cards as one pool, so I put Qwen3.8-27B (Q4_K_M, 15.93 GiB of weights) across both and now serve it on an OpenAI-compatible endpoint at 192k context, at roughly 23 t/s on prose and 35 t/s on code. This is how it is put together, including the parts that cost me the most time.

Credit where it is due: I got the idea, and a good chunk of the initial setup, from this video. Their reference numbers were 16.39 t/s tuned before MTP and 22 t/s with it, on a different pair of machines. What follows is where I ended up after adapting it to my two cards.

The equipment

Two ordinary Fedora 44 desktops on the same LAN. Nothing rack mounted, nothing exotic.
  • gpu-main, which runs llama-server: RTX 5060 Ti 16 GB, Blackwell, sm_120, ~448 GB/s. 30 GB RAM, 16 cores. This one drives a desktop, so around 3 GB of its VRAM is always spoken for by the compositor and browser.
  • gpu-worker, which runs the RPC worker: RTX 4060 Ti 16 GB, Ada, sm_89, ~288 GB/s. 27 GB RAM, 12 cores. Headless as far as this workload is concerned.
  • Interconnect: plain 1 GbE, 0.29 ms RTT. gpu-main has a 2.5 GbE port but gpu-worker does not, so the shared path is 1 Gb.
1 GbE sounds like it should be the bottleneck and it is not. The RPC backend ships activations between the two halves of the model, not weights, so the per-token payload is on the order of 10 KB. The weights are transferred once at load, and even that only happens on a cold worker (more on the cache below).

Why this model

Qwen3.8-27B is a 27.78 B dense model, Apache 2.0, 262,144 native context. The reason it fits on consumer cards at all is its attention layout: only about 4 of its 64 layers use full attention, and the rest are Gated DeltaNet linear attention carrying a fixed-size recurrent state that does not grow with context. That works out to roughly 16 KB of KV per token instead of the ~512 KB a conventional dense 32B would want.

The practical consequence is that long context is nearly free. Measured on one identical prompt, 96k ran 23.8/24.3 t/s and 192k ran 22.7/23.0 t/s. A conventional model would sag badly over that range.

Build both ends from the same binary

This is the single most important thing in the whole setup. Both hosts must run a byte-identical llama.cpp build, because a mismatch does not announce itself as a version problem. It presents as a network failure, which is a miserable thing to debug.

So build once, in Docker, and ship the image:

docker save llamacpp-rpc:84e908c6 | zstd -3 -T0 \
  | ssh user@gpu-worker 'zstd -d | docker load'
Same image ID on both ends, guaranteed. Two separate docker build runs do not give you that.

The flags that matter in the build:
  • -DCMAKE_CUDA_ARCHITECTURES="89;120" for one fat binary covering both cards.
  • -DGGML_NATIVE=OFF. It defaults to ON and compiles for the build host's CPU. The two hosts have different CPUs.
  • -DGGML_RPC=ON, obviously.
  • -DGGML_CUDA_NO_VMM=ON, without which the build does not link at all.

The link failure

Worth spelling out because the obvious fix is wrong. Without GGML_CUDA_NO_VMM you get:

/usr/bin/ld: libggml-cuda.so: undefined reference to `cuMemCreate'
                              undefined reference to `cuMemMap'
                              undefined reference to `cuMemAddressReserve'
Those are CUDA driver API symbols, used by ggml's VMM allocator. In an nvidia/cuda:*-devel image the real libcuda.so only exists at runtime, injected by the nvidia container runtime, so linking has to go through the stub.

Pointing LIBRARY_PATH at the stubs directory does not fix it. The stub is present and findable; the problem is one layer earlier, in that -lcuda never reaches libggml-cuda.so's own link line, so the shared object records no dependency and everything linking against it inherits the unresolved symbols. -DGGML_CUDA_NO_VMM=ON removes the driver-API path entirely. The cost is coarser VRAM pool growth, which is irrelevant for a fixed-size resident model.

One more container detail: use LIBRARY_PATH, not LD_LIBRARY_PATH. The latter would shadow the real driver at runtime.

Wiring the two hosts together

gpu-worker runs ggml-rpc-server bound to its own loopback, and gpu-main reaches it through an SSH tunnel. Nothing binds a routable address on either host.

docker run -d --name llamacpp-rpc-worker \
  --runtime=nvidia --gpus all --network host \
  -e CUDA_VISIBLE_DEVICES=0 \
  -v $HOME/.cache/llama.cpp/rpc:/cache:z \
  llamacpp-rpc:84e908c6 \
  ggml-rpc-server -H 127.0.0.1 -p 50052 -c -d CUDA0
ssh -f -N -M -S ~/.ssh/cm-llamarpc.sock \
    -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 \
    -L 127.0.0.1:50052:127.0.0.1:50052 user@gpu-worker
Two things in there earn their keep. -c on the worker caches its share of the weights on gpu-worker's own disk, so restarts skip the transfer entirely; first load is slow, later ones are not. The :z on the bind mount is required on Fedora or SELinux blocks the container from writing that cache.

The SSH ControlMaster socket is there so teardown can be ssh -O exit. The obvious alternative, pkill -f on the tunnel's command line, matches its own command line and kills the calling shell.

The tunnel is not paranoia

ggml-rpc-server has no authentication of any kind. Anyone who can reach the port can load code onto your GPU. My llama-server also runs with no API key and CORS open, so the same applies there.

There are two ways to deal with that. Either open port 50052 on the worker to the one address that needs it and nothing else, and satisfy yourself that your firewall really is enforcing that, or skip the firewall question entirely and forward the port over SSH. I went with the tunnel: the worker binds 127.0.0.1, so there is no listener on a routable address to get the rules wrong about.

If you do test firewall rules, test from a host on the same subnet. A machine that cannot reach the subnet at all looks exactly like a machine that was blocked, which is how you end up trusting a rule that is not doing anything.

One container caveat either way: published Docker ports (-p) are DNAT'd and traverse FORWARD through Docker's own chains, so host firewall rules on INPUT do not apply to them. Hence --network host on both containers, so the bind lands on the host stack where you expect it.

Bind to loopback and tunnel. It is the only part of this that is actually a boundary.

Serving it

The interesting arguments on the gpu-main side:

llama-server -m /models/Qwen3.8-27B-Q4_K_M.gguf \
  --rpc 127.0.0.1:50052 \
  -dev RPC0,CUDA0 -ts 6,4 -ngl 99 -sm layer \
  -c 196608 --flash-attn 1 -ub 256 \
  -np 1 -ctxcp 4 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --spec-type draft-mtp \
  --cache-reuse 256 --jinja \
  --host 127.0.0.1 --port 8080

Things that cost me time

Always pass -ngl 99 explicitly. Omit it and common_fit_params auto-fits to free VRAM and silently spills layers to CPU. Measured 10.4 t/s versus 16.9 t/s for otherwise identical config, and it comes up healthy, so it looks like success. Confusingly, when you do set it the log says:

W common_fit_params: failed to fit params to free device memory:
  n_gpu_layers already set by user to 99, abort
That reads like an error. It is auto-fit standing down as instructed, which is exactly what you want.

--spec-type draft-mtp is worth 2.1x and I nearly missed it. This model has a built-in multi-token-prediction head, shipped in the stock GGUF as blk.64.nextn.* tensors, which llama.cpp loads and logs as "unused tensor" until you enable it. The flag is not named for MTP; it is a value of --spec-type, so grepping --help for "mtp" or "nextn" finds nothing and leaves you concluding it is unsupported. With it on, 16.9 t/s becomes ~35 t/s.

Draft acceptance is prompt-dependent, not config-dependent. Agentic and code prompts accept 0.72 to 0.90 of drafted tokens and run ~35 t/s. Free-form prose accepts 0.40 to 0.50 and runs ~23 t/s. A low number on prose is not a regression, so check acceptance before you go chasing it.

Slot defaults quietly cost 1.9 GiB. -np defaults to auto and picked 4 slots; -ctxcp defaults to 32 context checkpoints per slot. Because this is a hybrid model, the recurrent state is allocated per slot at fixed size regardless of -c, and every checkpoint copies it. That is VRAM scaling with neither model size nor context length. Setting -np 1 -ctxcp 4 and changing nothing else took total llama.cpp footprint from 21614 to 19686 MiB with no measurable speed cost. That saving is what paid for the jump from 64k to 96k context. The tradeoff is that concurrent requests serialize instead of batching, which is fine for one agent client.

The tensor split cannot be sized by card speed. The binding constraint is free VRAM minus the compute buffer, not bandwidth. It also moves: when I relocated a resident 8B model off gpu-worker and freed 5.5 GB there, the roomy card and the constrained card swapped places, and the split I had tuned earlier became clearly wrong. -ts 6,4 now gives gpu-worker the larger share, keeps ~3.0 GiB free on gpu-main and ~1.9 GiB on gpu-worker, and costs nothing in throughput.

192k fits, 256k does not. I tried three ways. -ts 4,6 dies on a 5440 MiB KV allocation; both 5,5 and 6,4 get past KV and then die on a ~1192 MiB compute buffer; -ub 128 does not rescue it, which confirms that -ub is not the lever at long context because that buffer is attention scratch rather than the logits term. The logits term is genuinely large here, mind: vocab is 248,320, so logits are ubatch x 248320 x 4B, around 508 MB at -ub 512 before any activations. That is why -ub 256.

llama-bench and llama-server disagree on syntax. Splits are -ts 5,5 against -ts 5/5, devices -dev RPC0,CUDA0 against -dev RPC0/CUDA0, RPC --rpc against -rpc, flash attention --flash-attn 1 against -fa on, and bench has no -c at all (use -d for depth). Also, bench uses a tiny context, so a config that benches happily can still OOM when served.

-sm layer is pinned deliberately. It is already the default, but upstream reports reproducible CUDA lockups with this model under --split-mode tensor, and pinning it means a future default change cannot quietly move me onto the broken path.

--jinja is required for correct tool-call formatting with agentic clients.

One script to drive it

Three moving pieces with a strict ordering (worker, then tunnel, then server, because llama-server dials RPC at startup) is exactly the sort of thing you get wrong at 11pm. So it is one script:

./llm-cluster.sh up | down | status | restart | logs | bench
up preflights that the model and the image exist on both ends, brings the three pieces up in order, polls /health, and tears everything back down if the server fails to come up so nothing is left half-started. status prints VRAM on both hosts and distinguishes a tunnel the script owns from one opened by hand, which otherwise reports as down and makes a working system look broken.

Clients

This is where I lost the most time for the least reason. The endpoint is OpenAI-compatible, so in principle every coding agent works. In practice the deciding factor is not generation speed at all, it is how much prompt the client resends every turn.

Qwen Code sends 16,000 to 26,000 tokens per turn of system prompt plus tool definitions, which at ~600 t/s is roughly 40 seconds of prefill before a single token comes back. --cache-reuse helps only sometimes, because the prompt varies in size between turns and keeps invalidating the cached prefix. That is client behaviour and no amount of server tuning fixes it.

The pi coding agent sends about 1,900 tokens on the first turn and then adds 20 to 320 new tokens per turn on top of a stable cached prefix. A full read-a-file tool round trip takes about 9 seconds end to end. Same server, same model, completely different experience. If you are pointing a local model at an agent, measure the per-turn prompt size first.

Where it landed

Two mid-range consumer cards on a 1 GbE LAN, running a 27B model at 192k context, ~23 t/s on prose and ~35 t/s on code, ~600 t/s prefill, entirely on loopback and an SSH tunnel. The remaining known issue is a GPU watchdog crash on gpu-main's card, which drives the desktop; a video call mid-generation once killed a 1,700-token completion with CUDA error: the launch timed out and was terminated. The mitigation, if it becomes a habit, is to push the long kernels onto the headless card, which has no watchdog.