Local LLM Deployment Notes

Running local LLMs on consumer-grade GPUs boils down to a budgeting problem: model weights, KV cache, and the desktop environment compete for the same VRAM. How you allocate it determines the maximum model size and context length you can run. This article uses an RX 7900 XTX (24GB) as a case study to provide a selection methodology and a fully reproducible configuration, extending to 16GB and 12GB cards. All commands are compatible with Linux, and WSL2 follows the same logic.

1. Constraints: An Inequality, Independent of the Card

All decisions for local deployment—which quantization to choose, what precision to set for KV cache, and how long to make the context—are solving the same problem:

Model Weights + KV Cache + Runtime Buffer + Desktop Headroom ≤ VRAM Capacity

This inequality holds for 24GB, 16GB, and 12GB cards; only the right-hand side differs. Treat it as a budget to allocate, and it becomes the universal starting point for deployment. My machine has a 7900 XTX (24GB), which serves as the example below.

2. Step 1: Choose the Backend Based on Your GPU

llama.cpp supports multiple GPU backends. You do not need to install any vendor-specific AI training stacks (CUDA Toolkit / ROCm / oneAPI); it relies solely on the runtime provided by the GPU driver. Choose based on your card:

Your GPU	Backend	Compile Flag
NVIDIA (GTX 10 series and above)	CUDA	-DGGML_CUDA=ON
AMD (RX series / Radeon VII)	Vulkan	-DGGML_VULKAN=ON
Intel Arc	Vulkan	-DGGML_VULKAN=ON
Apple Silicon (M series)	Metal	-DGGML_METAL=ON

Common questions:

"Do I need to install the CUDA Toolkit for NVIDIA cards?" No. llama.cpp only depends on the CUDA runtime provided by the GPU driver; you don't need to install the multi-gigabyte CUDA Toolkit.

"Why Vulkan for AMD instead of ROCm?" ROCm is for training with PyTorch/vLLM; it's cumbersome to install (especially on Windows) and unnecessary for GGUF inference. Vulkan uses the runtime provided by the GPU driver and works out-of-the-box on both Linux and Windows.

"What if I have both integrated and discrete GPUs?" Vulkan users can lock to the discrete GPU using GGML_VK_VISIBLE_DEVICES=0 (0 = first card); CUDA users use CUDA_VISIBLE_DEVICES=0. Metal users usually don't need to worry about this.

Windows Users: Run all commands below in WSL2. WSL2 can directly access GPUs (NVIDIA cards work with official drivers; AMD cards require kernel 5.15+ and the official AMD WSL driver). The compilation and installation process is identical to Linux. First, install a distribution with wsl --install, then follow the subsequent steps.

Now, let's address the core issue: how to allocate VRAM.

3. How to Choose: Spend VRAM Where It Matters

There are only four knobs to adjust: parameter count and weight quantization are fixed when downloading the file; context length and KV precision are startup parameters that can be adjusted each time. First, let's look at the solution for this machine, then explain how to make the same decisions on your own card.

The Solution for This Machine: Invest Saved VRAM in Context, Not KV Precision

Choose Qwen3.6-35B-A3B (MoE, 35B total parameters but only 3B activated per step, offering the speed of a small model with the knowledge base of a large one). The real trade-off is where to spend the saved VRAM—on "how long the conversation can be remembered" rather than "how precisely the KV cache stores it":

Breaking down this 24G:

• Model Weights: 16.96 GiB (IQ4_XS). MoE makes it fast; IQ4_XS provides sufficient quality.
• KV Cache uses q4_0 + flash-attn, compressing 192K context into about 5G. If you switch to f16, the same context would require four times the VRAM and overflow directly; switching to Q8 would only allow half the context. The impact of KV precision on generation quality is far less than its impact on context length — so invest this budget entirely in length.
• Actual resident usage is 21.8G, with 2.2G remaining, dropping to low-power mode when idle, with no pressure on the desktop.

I previously ran a conservative configuration with Q3_K_XL + short context; after switching to IQ4_XS + q4_0 KV, the balance between quality, speed, and context actually improved. When the desktop occasionally reloads and becomes tight, you can switch to a Q3 tier as a fallback.

Where Are the Knobs, and When to Set Them

When applying this to your own card, the four knobs are divided into two groups:

Step 1: Choose Parameter Count + Weight Quantization (Fixed at Download)

Choosing Parameter Count relies on the "Tier Table" below — 30–35B on 24G, 14B on 16G, 8B on 12G.

Choosing Weight Quantization: The same model will have a row of GGUF files on Hugging Face; the filename suffix indicates the quantization level:

• Numbers ≈ bit count: Q8_0 (8-bit, near-lossless, largest) > Q6_K > Q5_K_M > Q4_K_M (4-bit, most common balance point) > Q3_K_M (saves VRAM, quality drops).
• IQ4_XS / IQ3_XXS are i-quant: smaller at the same bit count, with similar quality, requiring imatrix (provided by the publisher). Use these if VRAM is tight.
• _S / _M / _L = Small / Medium / Large variants; _M is most commonly used within the same bit count.

How to Choose: GGUF file size ≈ VRAM usage. Compare the weight budget against the file size and pick the largest that fits:

Weight Budget = VRAM − KV Cache − Desktop Headroom (leave ~2G)
Example: 24 − 5 (192K q4_0) − 2 ≈ 17G → 35B using IQ4_XS (actual 16.96G) fits perfectly;
      Switching to Q5_K_M (~24G) won't fit.

Experience priority: Aim for Q4_K_M first; if it doesn't fit, go down to IQ4_XS → IQ3; if you have surplus, go up to Q5_K_M / Q6_K. Don't use below Q3 unless you have no choice.

Step 2: Set KV Precision (Adjusted at Startup)

KV precision and context length are both startup parameters, adjustable each time. KV relies on three switches:

Switch	Function	Common Value
--flash-attn / -fa	Prerequisite for quantized KV, must be on	On / Off
--cache-type-k / -ctk	K cache precision	f16 (default) / q8_0 / q4_0
--cache-type-v / -ctv	V cache precision	Same as above

K and V are set separately, with precision tiers f16 → q8_0 → q4_0, consuming roughly 1 → 1/2 → 1/4 of VRAM — which is why q4_0 can achieve about 4 times the context of f16. Three typical tiers:

# Aggressive: Long context priority
llama-server ... -fa -ctk q4_0 -ctv q4_0 -c 196608
# Conservative: Quality priority, half context
llama-server ... -fa -ctk q8_0 -ctv q8_0 -c 98304
# Asymmetric: K is more sensitive to quantization; preserve K, compress V for max VRAM savings
llama-server ... -fa -ctk q8_0 -ctv q4_0

How to Verify How Much Was Adjusted: The startup log prints KV self size = ...; just check how much it takes up. In terms of quality, q8_0 is basically lossless, and q4_0 is hard to perceive in daily conversation and coding, only occasionally failing in precise long-text retrieval. Commands to check VRAM usage vary by platform: nvidia-smi (NVIDIA), cat /sys/class/drm/card0/device/mem_info_vram_used (AMD Linux), Task Manager → Performance → GPU (Windows).

Pitfall: Setting -ctv q4_0 without -fa will cause an error or be ignored — quantized V cache depends on flash attention.

Starting Points for Different VRAM Sizes

Pushing the same budget formula further (these are estimates; calibrate based on your own card):

VRAM	Realistic Choice
24G	30–35B MoE (IQ4/Q4) or 32B dense IQ4; long context relies on q4_0 KV
16G	14B dense Q4–Q5, or 30B MoE Q3 + moderate layer offload; medium context
12G	8–9B Q4–Q5 comfortable (e.g., Qwen3-8B GGUF ~5.4G), or 14B Q4 compact
8G	7–8B Q4, short context

How to Cut When VRAM Is Insufficient, in Priority Order: First lower KV precision (f16→q8_0→q4_0) to preserve context → then reduce context length → finally switch to a smaller model or more aggressive weight quantization. Don't forget that the desktop and games are also competing for this card; leave a few GB for them.

Once the decision is made, let's pull the model and run it.

4. Running It: From Installation to First Response

Install llama.cpp (must include server + corresponding backend, in priority order):

1. Use official release binaries — universal across platforms, download and extract to use, already includes all backends.
2. Linux distributions have ready-made packages: apt install llama-cpp (Debian/Ubuntu), pacman -S llama.cpp-vulkan (Arch).
3. If neither is available, compile from source (replace <YOUR_BACKEND> with the switch selected in Section 2, e.g., -DGGML_CUDA=ON):

git clone https://github.com/ggerganov/llama.cpp --depth 1
cd llama.cpp
cmake -B build <YOUR_BACKEND> -DLLAMA_CURL=ON
cmake --build build --config Release -j
# Artifacts are in build/bin/

Download the model to the local model directory:

pip install -U "huggingface_hub[cli]"
# Search HF for "<Model Name> GGUF", find quantized repos from unsloth / bartowski
hf download unsloth/Qwen3.6-35B-A3B-MTP-GGUF \
  --include "*IQ4_XS*.gguf" --local-dir ~/models

Large model GGUFs are often sharded (…-00001-of-0000N.gguf); --include "*IQ4_XS*" downloads all at once; llama.cpp points to the first shard and automatically loads the rest.

Start (single model, suitable for initial trial):

llama-server \
  -m ~/models/Qwen3.6-35B-A3B-MTP-UD-IQ4_XS.gguf \
  -ngl 99 -c 196608 \
  --flash-attn --cache-type-k q4_0 --cache-type-v q4_0 \
  --host 0.0.0.0 --port 18080

Vulkan users with multiple GPUs should add GGML_VK_VISIBLE_DEVICES=0 in front to lock to the discrete GPU; CUDA users skip this.

http://localhost:18080 is the OpenAI-compatible endpoint. Verify it's running:

curl http://localhost:18080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-3.5-turbo","messages":[{"role":"user","content":"hi"}]}'

What is MTP in the path, why use unsloth's GGUF, and is it optional — Section 5 covers this specifically.

Single-model commands are suitable for trials; for daily persistent use, the router mode is recommended (models.ini centrally manages multiple models, LRU eviction as needed). MTP acceleration and tidal scheduling are also based on router mode, detailed below.

5. Speedup: MTP Speculative Decoding

What Is It

MTP (Multi-Token Prediction) is a speculative decoding technique: predicting multiple tokens in one forward pass instead of generating them sequentially token by token. On this machine, using Qwen3.6-35B-A3B (IQ4_XS), the actual measurement shows: without MTP, ~75 t/s; with MTP, ~111 t/s, a speedup of about 50%. Your model and quantization may differ, so actual improvements will vary.

Standard llama.cpp models do not have this capability — it comes from unsloth (a team specializing in GGUF quantization and acceleration) which embeds a draft head into the GGUF during quantization. The draft head first "guesses" the next 2 tokens, and the main model verifies them in parallel in one forward pass; correct guesses are earned, incorrect guesses fall back. No independent draft model is needed, and it doesn't consume extra VRAM:

How to Find It, and What If You Don't Use It

MTP is not a native model feature; it's an addition by unsloth during quantization. So:

• Only search for unsloth/<Model Name>-MTP-GGUF on HF to find the MTP version, e.g., unsloth/Qwen3.6-35B-A3B-MTP-GGUF. Regular quantization repos like bartowski do not have it.
• The filename contains MTP, e.g., …-MTP-UD-IQ4_XS.gguf. If it doesn't contain it, there's no draft head, and speculative decoding is impossible.
• Not all models have an unsloth MTP version — unsloth mainly adds it to the Qwen series. If it's not available, use a regular GGUF; it will still run, just without MTP acceleration.

Configuration without MTP: Remove MTP from the download command, switch to a regular repo (e.g., bartowski/Qwen3.6-35B-A3B-GGUF); remove the spec-type and spec-draft-n-max lines from the config file, leave the rest unchanged.

Router Mode and Full Preset

If you want to manage multiple models or have them start automatically on boot, upgrade to router mode: write all models and parameters to ~/.config/llama.cpp/models.ini, load with --models-preset, and use --models-max 1 to limit residency to one model at a time (24G can only fit one), with automatic LRU eviction:

# ~/.config/llama.cpp/models.ini
[qwen3.6-mtp-instruct]
model = ~/models/Qwen3.6-35B-A3B-MTP-UD-IQ4_XS.gguf
ngl = 99                      # All layers on GPU (99=all)
ctx-size = 196608             # 192K Context
chat-template = chatml        # OpenAI chat template, recommended even without tools
chat-template-kwargs = {"enable_thinking": false}
flash-attn = 1
cache-type-k = q4_0
cache-type-v = q4_0
spec-type = draft-mtp          # Next two lines only needed for MTP models
spec-draft-n-max = 2

Pitfall: chat-template=chatml must be set explicitly — when using the model's native template, tool call prefixes may generate incorrect tokens causing premature EOS (llama.cpp #19513). This applies even without MTP or tool calls.

Adding a new model only requires adding an entry in the INI file; the Open WebUI list updates automatically. See the "Full Configuration" section for the systemd unit and enable commands.

6. Persistent Scheduling: Models Online, Power Tidal (Linux)

Running fast is temporary; persistent non-wasteful operation is daily life. The following is based on Linux systemd and AMD sysfs interfaces (WSL2 users follow this section directly; the distribution comes with systemd; pure Windows users skip this, the core idea is the same: run a background process to manage llama-server, switching power profiles based on GPU load):

Three mechanisms:

① Boot Warmup. Router lazy loading relies on systemd ExecStartPost to send a request after startup to load the model into VRAM. The warmup payload must use -d @file — inline JSON will have its quotes swallowed by systemd. warmup.json is just one line: {"model":"qwen3.6-mtp-instruct","messages":[{"role":"user","content":"hi"}],"max_tokens":1}.

② Power Switches Based on Real Load. llama-gpu-sync.timer triggers every 15s, checking if gpu_busy_percent ≥ 50% to switch to COMPUTE (Tier 5); if idle for more than 90s, it drops back to BOOTUP_DEFAULT (Tier 0), only touching 0/5, not the 3d_full_screen used by games. The key is using real GPU occupancy rather than "whether the model is loaded" — resident but not inferring should stay in the power-saving tier. AMD implements this via sysfs pp_power_profile_mode; NVIDIA uses nvidia-smi -pl.

③ Whole Card Release During Gaming. gamemode (available in most distributions) detects Steam game launches and stops llama-server, giving all VRAM to the game, and restarts it upon exit. The model remains in page cache, with a ~6s warmup to resume:

# ~/.config/gamemode.ini
[custom]
start = systemctl --user stop llama-server
end   = systemctl --user start llama-server

Full systemd unit, power switching script, sudoers passwordless, and enable commands are in the "Full Configuration" section.

7. Performance Baseline

llama.cpp benchmarking terms:

• pp512 (prompt processing): Speed of processing 512 input tokens in parallel (token/s), determining how fast "reading long context" is.
• tg128 (text generation): Speed of continuously generating 128 output tokens, i.e., the intuitive "typing" speed — human reading is about 5–10 t/s; 100+ means "the screen floods before you finish speaking".
• MTP Draft Acceptance Rate: The proportion of draft head guesses verified by the main model; the higher, the greater the equivalent speedup.

Actual measurements on this machine (Qwen3.6-35B-A3B · IQ4_XS · 192K):

Metric	Value	Concept
pp512	~3150 t/s	Long context prefill is not laggy
tg128	~111 t/s	Far exceeds reading speed, feels instant
MTP Acceptance Rate	~52%	Draft head hits about half the time
Resident VRAM	21.8 / 24 GB	2.2G remaining

Test it yourself: llama-bench one command, compare different cards and quantizations:

llama-bench -m ~/models/Qwen3.6-35B-A3B-MTP-UD-IQ4_XS.gguf -ngl 99
# Output is two lines: pp512 / tg128
# Must stop the resident service first: systemctl --user stop llama-server (Linux)

8. After Running: Connect Clients

llama-server exposes an OpenAI-compatible endpoint (http://localhost:18080/v1); any client that accepts a custom base_url can connect — Open WebUI, IDE plugins, CLI agents, just point the base_url here.

At this point, the model is ready to use. But "what to do with it, which tasks belong to it, which should be left to Claude Opus, and whether to configure cloud fallback" is another topic — that's where local models truly deserve scrutiny, covered in a separate article: see local-llm-usage.md.

9. Pitfalls Collection

• systemd inline JSON has quotes swallowed → Warmup must use -d @file.
• Model ran on integrated GPU when both integrated and discrete GPUs were present → Lock to discrete GPU using GGML_VK_VISIBLE_DEVICES=0 (Vulkan) or CUDA_VISIBLE_DEVICES=0 (CUDA).
• Quantized KV without flash-attn → --flash-attn is a prerequisite for cache-type q4_0/q8_0; otherwise, it's ignored or causes an error.
• Tool calls causing premature EOS → Explicitly specify chat-template=chatml to avoid native template bugs (llama.cpp #19513).
• Thinking resident = always high power → Switch profiles using gpu_busy_percent (AMD) or nvidia-smi (NVIDIA); drop to power-saving tier when idle.
• Wrong VRAM budget calculation → First estimate Model Size + KV (Layers × Context × Precision) + Buffer + Desktop, leave 1–2G headroom, don't push to zero.

10. Full Configuration (Linux)

The previous sections explained "what to do" and only pasted key snippets; here, all files in the entire chain are listed completely. Create them in order, then enable.

WSL2 users follow this section directly; pure Windows users use Task Scheduler to replace systemd, and handle power switching separately for AMD/NVIDIA.

Prerequisites: Scripts + Passwordless Sudo

llama-profile-sync calls gpu-profile-set, which requires root to write to pp_power_profile_mode (AMD GPU sysfs); systemctl's ExecStopPost is similar. User units cannot prompt for sudo passwords, so passwordless access must be configured:

sudo tee /usr/local/bin/gpu-profile-set <<'EOF'
#!/bin/sh
case "$1" in
    bootup_default) idx=0 ;;
    3d_full_screen) idx=1 ;;
    power_saving)   idx=2 ;;
    video)          idx=3 ;;
    vr)             idx=4 ;;
    compute)        idx=5 ;;
    [0-5])          idx=$1 ;;
    *) echo "Unknown profile: $1" >&2; exit 1 ;;
esac
printf "%s\n" "$idx" > /sys/class/drm/card0/device/pp_power_profile_mode
EOF
sudo chmod 755 /usr/local/bin/gpu-profile-set

sudo tee /usr/local/bin/llama-profile-sync <<'EOF'
#!/bin/bash
PROFILE_PATH="/sys/class/drm/card0/device/pp_power_profile_mode"
BUSY_PATH="/sys/class/drm/card0/device/gpu_busy_percent"
STATE="/tmp/llama-profile-last-busy"
BUSY_THRESHOLD=50
HOLD_SECONDS=90

current_idx() { grep '\*' "$PROFILE_PATH" | awk '{print $1}'; }

set_profile() {
    local want=$1 cur
    cur=$(current_idx)
    case "$cur" in 0|5) ;; *) return 0 ;; esac
    case "$want" in
        compute)        [ "$cur" = "5" ] && return 0 ;;
        bootup_default) [ "$cur" = "0" ] && return 0 ;;
    esac
    sudo /usr/local/bin/gpu-profile-set "$want"
}

if ! systemctl --user is-active --quiet llama-server 2>/dev/null; then
    set_profile bootup_default; exit 0
fi
busy=$(cat "$BUSY_PATH" 2>/dev/null || echo 0)
now=$(date +%s)
if [ "${busy:-0}" -ge "$BUSY_THRESHOLD" ]; then
    echo "$now" > "$STATE"
    set_profile compute
else
    last=$(cat "$STATE" 2>/dev/null || echo 0)
    [ $((now - last)) -lt "$HOLD_SECONDS" ] && set_profile compute || set_profile bootup_default
fi
EOF
sudo chmod 755 /usr/local/bin/llama-profile-sync

# Passwordless sudo (replace USERNAME with your own):
# echo "USERNAME ALL=(ALL) NOPASSWD: /usr/local/bin/gpu-profile-set" | sudo tee /etc/sudoers.d/llama-profile

pp_power_profile_mode / gpu_busy_percent are AMD GPU sysfs interfaces. NVIDIA power management uses nvidia-smi -pl or nvidia-persistenced; these scripts do not apply.

systemd User Units (Three Files)

# ~/.config/systemd/user/llama-server.service
[Unit]
Description=llama.cpp server (router mode, Vulkan)
After=network.target

[Service]
Environment=GGML_VK_VISIBLE_DEVICES=0     # Vulkan lock to discrete GPU; CUDA users delete this line or change to CUDA_VISIBLE_DEVICES=0
ExecStart=/usr/bin/llama-server \
    --models-preset %h/.config/llama.cpp/models.ini \
    --models-max 1 \
    --host 0.0.0.0 \
    --port 18080 \
    --metrics
# Warmup: Preload model into VRAM; -d @file avoids systemd swallowing quotes
ExecStartPost=-/usr/bin/bash -c 'for i in $(seq 1 60); do curl -sf --max-time 90 http://127.0.0.1:18080/v1/chat/completions -H "Content-Type: application/json" -d @%h/.config/llama.cpp/warmup.json >/dev/null && exit 0; sleep 2; done'
# When service stops, GPU returns to power-saving tier
ExecStopPost=/usr/bin/sudo /usr/local/bin/gpu-profile-set bootup_default   # AMD power management; NVIDIA users delete this line
Restart=on-failure
RestartSec=5

[Install]
WantedBy=default.target

# ~/.config/systemd/user/llama-gpu-sync.service
[Unit]
Description=Sync GPU profile with llama-server model state
After=llama-server.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/llama-profile-sync

# ~/.config/systemd/user/llama-gpu-sync.timer
[Unit]
Description=Poll llama-server model state for GPU profile switching

[Timer]
OnBootSec=15s
OnUnitActiveSec=15s

[Install]
WantedBy=timers.target

Warmup

echo '{"model":"qwen3.6-mtp-instruct","messages":[{"role":"user","content":"hi"}],"max_tokens":1}' \
  > ~/.config/llama.cpp/warmup.json

Enable

# linger — Key for persistent models: service remains alive after logout
sudo loginctl enable-linger "$USER"

# Reload + Enable
systemctl --user daemon-reload
systemctl --user enable --now llama-server.service
systemctl --user enable --now llama-gpu-sync.timer

# Verify
systemctl --user status llama-server
systemctl --user status llama-gpu-sync.timer

--user is not --system: GPU context (DRM render node) is in the user session; running a system unit might fail to open the device; permissions are also much smaller.

Related Documents

• local-llm-usage.md — After running: Boundaries (Local vs Opus), What it can do, llm-jobs automation, Practical lessons learned
• software.md — Software Stack List
• timers-and-crons.md — Timed Tasks like llama-gpu-sync.timer
• monitoring.md — Monitoring Architecture (GPU metrics via node_exporter → Prometheus → Grafana)