Running a 27B model across two desktops with llama.cpp RPC

I have two machines with a 16 GB consumer GPU in each, and a model I wanted to run that does not fit in either one on its own. llama.cpp's RPC backend lets you treat the two cards as one pool, so I put Qwen3.8-27B (Q4_K_M, 15.93 GiB of weights) across both and now serve it on an OpenAI-compatible endpoint at 192k context, at roughly 23 t/s on prose and 35 t/s on code. This is how it is put together, including the parts that cost me the most time.

Credit where it is due: I got the idea, and a good chunk of the initial setup, from this video. Their reference numbers were 16.39 t/s tuned before MTP and 22 t/s with it, on a different pair of machines. What follows is where I ended up after adapting it to my two cards.

The equipment

Two ordinary Fedora 44 desktops on the same LAN. Nothing rack mounted, nothing exotic.

gpu-main, which runs llama-server: RTX 5060 Ti 16 GB, Blackwell, sm_120, ~448 GB/s. 30 GB RAM, 16 cores. This one drives a desktop, so around 3 GB of its VRAM is always spoken for by the compositor and browser.
gpu-worker, which runs the RPC worker: RTX 4060 Ti 16 GB, Ada, sm_89, ~288 GB/s. 27 GB RAM, 12 cores. Headless as far as this workload is concerned.
Interconnect: plain 1 GbE, 0.29 ms RTT. gpu-main has a 2.5 GbE port but gpu-worker does not, so the shared path is 1 Gb.

1 GbE sounds like it should be the bottleneck and it is not. The RPC backend ships activations between the two halves of the model, not weights, so the per-token payload is on the order of 10 KB. The weights are transferred once at load, and even that only happens on a cold worker (more on the cache below).

Why this model

Qwen3.8-27B is a 27.78 B dense model, Apache 2.0, 262,144 native context. The reason it fits on consumer cards at all is its attention layout: only about 4 of its 64 layers use full attention, and the rest are Gated DeltaNet linear attention carrying a fixed-size recurrent state that does not grow with context. That works out to roughly 16 KB of KV per token instead of the ~512 KB a conventional dense 32B would want.

The practical consequence is that long context is nearly free. Measured on one identical prompt, 96k ran 23.8/24.3 t/s and 192k ran 22.7/23.0 t/s. A conventional model would sag badly over that range.

Build both ends from the same binary

This is the single most important thing in the whole setup. Both hosts must run a byte-identical llama.cpp build, because a mismatch does not announce itself as a version problem. It presents as a network failure, which is a miserable thing to debug.

So build once, in Docker, and ship the image:

docker save llamacpp-rpc:84e908c6 | zstd -3 -T0 \
  | ssh user@gpu-worker 'zstd -d | docker load'

Same image ID on both ends, guaranteed. Two separate docker build runs do not give you that.

The flags that matter in the build:

-DCMAKE_CUDA_ARCHITECTURES="89;120" for one fat binary covering both cards.
-DGGML_NATIVE=OFF. It defaults to ON and compiles for the build host's CPU. The two hosts have different CPUs.
-DGGML_RPC=ON, obviously.
-DGGML_CUDA_NO_VMM=ON, without which the build does not link at all.

The link failure

Worth spelling out because the obvious fix is wrong. Without GGML_CUDA_NO_VMM you get:

/usr/bin/ld: libggml-cuda.so: undefined reference to `cuMemCreate'
                              undefined reference to `cuMemMap'
                              undefined reference to `cuMemAddressReserve'

Those are CUDA driver API symbols, used by ggml's VMM allocator. In an nvidia/cuda:*-devel image the real libcuda.so only exists at runtime, injected by the nvidia container runtime, so linking has to go through the stub.

Pointing LIBRARY_PATH at the stubs directory does not fix it. The stub is present and findable; the problem is one layer earlier, in that -lcuda never reaches libggml-cuda.so's own link line, so the shared object records no dependency and everything linking against it inherits the unresolved symbols. -DGGML_CUDA_NO_VMM=ON removes the driver-API path entirely. The cost is coarser VRAM pool growth, which is irrelevant for a fixed-size resident model.

One more container detail: use LIBRARY_PATH, not LD_LIBRARY_PATH. The latter would shadow the real driver at runtime.

Wiring the two hosts together

gpu-worker runs ggml-rpc-server bound to its own loopback, and gpu-main reaches it through an SSH tunnel. Nothing binds a routable address on either host.

docker run -d --name llamacpp-rpc-worker \
  --runtime=nvidia --gpus all --network host \
  -e CUDA_VISIBLE_DEVICES=0 \
  -v $HOME/.cache/llama.cpp/rpc:/cache:z \
  llamacpp-rpc:84e908c6 \
  ggml-rpc-server -H 127.0.0.1 -p 50052 -c -d CUDA0

ssh -f -N -M -S ~/.ssh/cm-llamarpc.sock \
    -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 \
    -L 127.0.0.1:50052:127.0.0.1:50052 user@gpu-worker

Two things in there earn their keep. -c on the worker caches its share of the weights on gpu-worker's own disk, so restarts skip the transfer entirely; first load is slow, later ones are not. The :z on the bind mount is required on Fedora or SELinux blocks the container from writing that cache.

The SSH ControlMaster socket is there so teardown can be ssh -O exit. The obvious alternative, pkill -f on the tunnel's command line, matches its own command line and kills the calling shell.

The tunnel is not paranoia

ggml-rpc-server has no authentication of any kind. Anyone who can reach the port can load code onto your GPU. My llama-server also runs with no API key and CORS open, so the same applies there.

There are two ways to deal with that. Either open port 50052 on the worker to the one address that needs it and nothing else, and satisfy yourself that your firewall really is enforcing that, or skip the firewall question entirely and forward the port over SSH. I went with the tunnel: the worker binds 127.0.0.1, so there is no listener on a routable address to get the rules wrong about.

If you do test firewall rules, test from a host on the same subnet. A machine that cannot reach the subnet at all looks exactly like a machine that was blocked, which is how you end up trusting a rule that is not doing anything.

One container caveat either way: published Docker ports (-p) are DNAT'd and traverse FORWARD through Docker's own chains, so host firewall rules on INPUT do not apply to them. Hence --network host on both containers, so the bind lands on the host stack where you expect it.

Bind to loopback and tunnel. It is the only part of this that is actually a boundary.

Serving it

The interesting arguments on the gpu-main side:

llama-server -m /models/Qwen3.8-27B-Q4_K_M.gguf \
  --rpc 127.0.0.1:50052 \
  -dev RPC0,CUDA0 -ts 6,4 -ngl 99 -sm layer \
  -c 196608 --flash-attn 1 -ub 256 \
  -np 1 -ctxcp 4 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --spec-type draft-mtp \
  --cache-reuse 256 --jinja \
  --host 127.0.0.1 --port 8080

Adding TurboQuant without disturbing the working build

I wanted to see whether TurboQuant could buy enough KV-cache space to reach the model's native 256k context. I did not replace the known-good llama.cpp image. I built TheTom's llama-cpp-turboquant fork, release tqp-v0.3.0 at commit 30d6881eb97be0844b77ff7bc93175e15972d689, as a second image called llamacpp-turboquant:tqp-v0.3.0. As before, I built it once and copied that image to gpu-worker; both hosts have image ID sha256:106d78f81d67b6c7223403cd52b9e9aeea07e3269e2078b821b2e4ce33e6b5fa.

The build is almost the same as upstream llama.cpp:

FROM nvidia/cuda:12.8.1-devel-ubuntu24.04 AS build
RUN apt-get update && apt-get install -y \
    git cmake ninja-build build-essential libcurl4-openssl-dev libgomp1
RUN git clone https://github.com/TheTom/llama-cpp-turboquant.git /src \
 && cd /src \
 && git checkout 30d6881eb97be0844b77ff7bc93175e15972d689
RUN cmake -S /src -B /src/build -G Ninja \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_CUDA=ON -DGGML_RPC=ON -DGGML_NATIVE=OFF \
    -DGGML_CUDA_NO_VMM=ON \
    -DCMAKE_CUDA_ARCHITECTURES="89;120" \
    -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF \
    -DLLAMA_BUILD_TOOLS=ON -DLLAMA_BUILD_SERVER=ON \
    -DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI=OFF \
 && cmake --build /src/build -j12

The last two UI flags are not cosmetic. This fork's embedded UI build expected a missing loading.html asset and failed without them. The RPC executable is also named rpc-server, not ggml-rpc-server. Its startup is unusually long and quiet: the TCP socket opens before CUDA and RPC initialization are actually ready, so a successful connect is not a sufficient readiness check. I use a local llama-cli --rpc localhost:50053 --list-devices probe on gpu-worker and wait for it to succeed before starting the main server.

At runtime the useful change is the V-cache type:

--cache-type-k q8_0 --cache-type-v turbo4

TurboQuant is principally a memory win here, not a magic faster-model switch. On Qwen3.8-27B at 192k it reduced usage to 12,059 MiB on gpu-main and 13,102 MiB on gpu-worker, roughly 1.2 to 1.3 GiB saved per card. More importantly, native 256k now fits at 14,439 and 14,728 MiB. One 256k prose run made 24.16 t/s and a comparable 192k run made 26.44 t/s: useful, but not evidence that TurboQuant itself makes generation faster. What it bought was the context that q8/q8 could not fit.

There is one real regression for agent use: this fork disables --cache-reuse for these hybrid contexts. That can cost more wall time on repeated large prompts than TurboQuant saves anywhere else, so I have kept this as a separate experimental image rather than silently replacing the normal server.

Trying Qwen3.5-35B-A3B

The more interesting speed experiment was Qwen3.5-35B-A3B. It has 35B total parameters but only about 3B active per token, and the ordinary Unsloth Q4_K_M ran immediately on the same RPC setup. At 128k with q8/q8 KV it used 11,465 MiB locally and 13,914 MiB remotely. At 192k with q8/Turbo4 it used 13,946 and 12,018 MiB. Single prompt runs produced 51.69 t/s on prose and 62.23 t/s on Go code, and structured tool calling emitted the correct function and JSON arguments.

But --spec-type draft-mtp failed with a useful, unambiguous message:

context type MTP requested but model doesn't contain MTP layers

Qwen trained the model with multi-token prediction, but that does not mean every converted GGUF contains the MTP tensors. The working file is the separate Unsloth MTP release, specifically Qwen3.5-35B-A3B-UD-Q4_K_M.gguf. It is a complete 22,627,735,712-byte replacement model, not a small sidecar draft file. Its SHA-256 is a20c5a9f0cee94d24af3553e7828df1337c7d24213789911c74e1f4936894936.

With that file, llama.cpp logs an embedded MTP draft context with up to three speculative tokens. Same 5:5 split, 192k context, -ub 256 -np 1 -ctxcp 4, q8/Turbo4 cache:

Prose: 91.75 t/s, 166 of 264 draft tokens accepted (62.9%).
Go coding: 96.40 t/s, 248 of 402 accepted (61.7%).
Structured tool call: 114.70 t/s, 64 of 75 accepted (85.3%), with the correct function name and JSON.
VRAM: 14,983 MiB on gpu-main and 12,156 MiB on gpu-worker.

Those are spot checks, not a statistically respectable benchmark suite, and speculative performance is unusually prompt-dependent. Still, against the ordinary GGUF's corresponding 51.69 and 62.23 t/s runs, embedded MTP improved observed generation by about 78% on prose and 55% on code. This is the first alternative I tried that was not merely smaller or faster, but felt like a materially faster high-quality model on this hardware.

The model shortlist after testing

Model	What happened	Where I would use it
Qwen3.8-27B Q4_K_M	~23 t/s prose, ~35 t/s code with MTP; excellent 192k long-context behaviour.	Dense quality and repository-scale context.
Qwen3.8-27B + Turbo4 V-cache	24.16 t/s at native 256k; saved roughly 1.2–1.3 GiB per GPU at 192k.	When 256k context matters more than prompt-cache reuse.
Qwen3.5-35B-A3B ordinary Q4_K_M	51.69–62.23 t/s, but the file has no MTP layers.	Fast fallback if the MTP conversion is unavailable.
Qwen3.5-35B-A3B MTP UD-Q4_K_M	91.75–114.70 t/s in these tests; correct tool call; fits at 192k.	Current speed/quality winner for this two-card setup.

For gpu-worker by itself, the next model I would test is Qwen3-Coder-30B-A3B-Instruct at Q3_K_M. Its GGUF is 14.7 GB and only 3.3B parameters are active, so it should fit a useful 16k–32k quantized cache on the 16 GB 4060 Ti while retaining its agentic-coding and tool-use training. The faster, less ambitious option is Qwen3.5-9B at Q8_0 (9.53 GB). I have not benchmarked either locally yet, so those are candidates, not results.

Things that cost me time

Always pass -ngl 99 explicitly. Omit it and common_fit_params auto-fits to free VRAM and silently spills layers to CPU. Measured 10.4 t/s versus 16.9 t/s for otherwise identical config, and it comes up healthy, so it looks like success. Confusingly, when you do set it the log says:

W common_fit_params: failed to fit params to free device memory:
  n_gpu_layers already set by user to 99, abort

That reads like an error. It is auto-fit standing down as instructed, which is exactly what you want.

--spec-type draft-mtp is worth 2.1x and I nearly missed it. This model has a built-in multi-token-prediction head, shipped in the stock GGUF as blk.64.nextn.* tensors, which llama.cpp loads and logs as "unused tensor" until you enable it. The flag is not named for MTP; it is a value of --spec-type, so grepping --help for "mtp" or "nextn" finds nothing and leaves you concluding it is unsupported. With it on, 16.9 t/s becomes ~35 t/s.

Draft acceptance is prompt-dependent, not config-dependent. Agentic and code prompts accept 0.72 to 0.90 of drafted tokens and run ~35 t/s. Free-form prose accepts 0.40 to 0.50 and runs ~23 t/s. A low number on prose is not a regression, so check acceptance before you go chasing it.

Slot defaults quietly cost 1.9 GiB. -np defaults to auto and picked 4 slots; -ctxcp defaults to 32 context checkpoints per slot. Because this is a hybrid model, the recurrent state is allocated per slot at fixed size regardless of -c, and every checkpoint copies it. That is VRAM scaling with neither model size nor context length. Setting -np 1 -ctxcp 4 and changing nothing else took total llama.cpp footprint from 21614 to 19686 MiB with no measurable speed cost. That saving is what paid for the jump from 64k to 96k context. The tradeoff is that concurrent requests serialize instead of batching, which is fine for one agent client.

The tensor split cannot be sized by card speed. The binding constraint is free VRAM minus the compute buffer, not bandwidth. It also moves: when I relocated a resident 8B model off gpu-worker and freed 5.5 GB there, the roomy card and the constrained card swapped places, and the split I had tuned earlier became clearly wrong. -ts 6,4 now gives gpu-worker the larger share, keeps ~3.0 GiB free on gpu-main and ~1.9 GiB on gpu-worker, and costs nothing in throughput.

192k fits, 256k does not. I tried three ways. -ts 4,6 dies on a 5440 MiB KV allocation; both 5,5 and 6,4 get past KV and then die on a ~1192 MiB compute buffer; -ub 128 does not rescue it, which confirms that -ub is not the lever at long context because that buffer is attention scratch rather than the logits term. The logits term is genuinely large here, mind: vocab is 248,320, so logits are ubatch x 248320 x 4B, around 508 MB at -ub 512 before any activations. That is why -ub 256.

llama-bench and llama-server disagree on syntax. Splits are -ts 5,5 against -ts 5/5, devices -dev RPC0,CUDA0 against -dev RPC0/CUDA0, RPC --rpc against -rpc, flash attention --flash-attn 1 against -fa on, and bench has no -c at all (use -d for depth). Also, bench uses a tiny context, so a config that benches happily can still OOM when served.

-sm layer is pinned deliberately. It is already the default, but upstream reports reproducible CUDA lockups with this model under --split-mode tensor, and pinning it means a future default change cannot quietly move me onto the broken path.

--jinja is required for correct tool-call formatting with agentic clients.

One script to drive it

Three moving pieces with a strict ordering (worker, then tunnel, then server, because llama-server dials RPC at startup) is exactly the sort of thing you get wrong at 11pm. So it is one script:

./llm-cluster.sh up | down | status | restart | logs | bench

up preflights that the model and the image exist on both ends, brings the three pieces up in order, polls /health, and tears everything back down if the server fails to come up so nothing is left half-started. status prints VRAM on both hosts and distinguishes a tunnel the script owns from one opened by hand, which otherwise reports as down and makes a working system look broken.

Clients

This is where I lost the most time for the least reason. The endpoint is OpenAI-compatible, so in principle every coding agent works. In practice the deciding factor is not generation speed at all, it is how much prompt the client resends every turn.

Qwen Code sends 16,000 to 26,000 tokens per turn of system prompt plus tool definitions, which at ~600 t/s is roughly 40 seconds of prefill before a single token comes back. --cache-reuse helps only sometimes, because the prompt varies in size between turns and keeps invalidating the cached prefix. That is client behaviour and no amount of server tuning fixes it.

The pi coding agent sends about 1,900 tokens on the first turn and then adds 20 to 320 new tokens per turn on top of a stable cached prefix. A full read-a-file tool round trip takes about 9 seconds end to end. Same server, same model, completely different experience. If you are pointing a local model at an agent, measure the per-turn prompt size first.

Where it landed

Two mid-range consumer cards on a 1 GbE LAN, running a 27B model at 192k context, ~23 t/s on prose and ~35 t/s on code, ~600 t/s prefill, entirely on loopback and an SSH tunnel. The remaining known issue is a GPU watchdog crash on gpu-main's card, which drives the desktop; a video call mid-generation once killed a 1,700-token completion with CUDA error: the launch timed out and was terminated. The mitigation, if it becomes a habit, is to push the long kernels onto the headless card, which has no watchdog.