---
title: Local LLM Deployment Notes
url: https://doc.liz6.com/en/homelab/local-llm
locale: en
area: homelab
tags:
- homelab
date: 2026-07-01
modified: 2026-08-24
description: 'Running local LLMs on consumer-grade GPUs boils down to a budgeting problem: model weights, KV cache, and the desktop environment compete for the same VRAM. How you allocate it determines the maximum model size and context length you can run. This article uses an RX 7900 XTX (24GB) as a case study to provide a selection methodology and a fully reproducible configuration, extending to 16GB and 12GB GPUs. All commands are compatible with Linux; WSL2 follows the same logic.'
---
# Local LLM Deployment Notes
> Running local LLMs on consumer-grade GPUs boils down to a budgeting problem: model weights, KV cache, and the desktop environment compete for the same VRAM. How you allocate it determines the maximum model size and context length you can run. This article uses an RX 7900 XTX (24GB) as a case study to provide a selection methodology and a fully reproducible configuration, extending to 16GB and 12GB GPUs. All commands are compatible with Linux; WSL2 follows the same logic.
## 1. Constraints: An Inequality, Independent of the GPU
All decisions for local deployment—which quantization to choose, what precision to set for KV cache, and how long to make the context—are solving the same problem:
> **Model Weights + KV Cache + Runtime Buffer + Desktop Headroom ≤ VRAM Capacity**
This inequality holds for 24GB, 16GB, and 12GB cards; only the right-hand side differs. Treat it as a budget to allocate, and it becomes the universal starting point for deployment. My machine has a 7900 XTX (24GB), which serves as the example below.
## 2. Step 1: Choose the Backend Based on Your GPU
`llama.cpp` supports multiple GPU backends. **You do not need to install any vendor-specific AI training stacks** (CUDA Toolkit / ROCm / oneAPI); it relies solely on the runtime provided by the GPU driver. Choose based on your card:
| Your GPU | Backend | Compile Flag |
|----------|----------|--------------|
| **NVIDIA** (GTX 10 series and above) | CUDA | `-DGGML_CUDA=ON` |
| **AMD** (RX series / Radeon VII) | Vulkan | `-DGGML_VULKAN=ON` |
| **Intel Arc** | Vulkan | `-DGGML_VULKAN=ON` |
| **Apple Silicon** (M series) | Metal | `-DGGML_METAL=ON` |
Common questions:
**"Do I need to install the CUDA Toolkit for NVIDIA cards?"** No. `llama.cpp` only depends on the CUDA runtime provided by the GPU driver; you don't need to install the multi-gigabyte CUDA Toolkit.
**"Why is it Vulkan for AMD instead of ROCm?"** ROCm is for training with PyTorch/vLLM; it's cumbersome to install (especially on Windows) and unnecessary for GGUF inference. Vulkan uses the runtime provided by the GPU driver and works out-of-the-box on both Linux and Windows.
**"What if I have both integrated and discrete GPUs?"** Vulkan users can lock to the discrete GPU using `GGML_VK_VISIBLE_DEVICES=0` (0 = first card); CUDA users use `CUDA_VISIBLE_DEVICES=0`. Metal users typically don't need to worry about this.
> **Windows Users**: Run all commands below in WSL2. WSL2 can directly access GPUs (NVIDIA cards work with official drivers; AMD cards require kernel 5.15+ and the official AMD WSL driver). The compilation and installation process is identical to Linux. First, install a distribution with `wsl --install`, then follow the steps.
Now, let's address the core issue: how to allocate VRAM.
## 3. How to Choose: Spend VRAM Where It Counts
There are only four knobs to adjust: parameter count and weight quantization are fixed when downloading the file; context length and KV precision are startup parameters that can be adjusted each time. Let's look at the solution for my machine first, then explain how to make the same decisions for your own card.
### My Solution: Prioritize Model Size, Then Slice KV by Workload
I chose **Qwen3.6-35B-A3B** (MoE, 35B total parameters but only 3B active per step, offering small-model speed with large-model capacity). The real trade-off isn't jumping back and forth between 35B and 8B; it's preserving the 35B model's expressiveness and consistency, while preparing two KV tiers for the same model: **Q8_0 + 128K for daily interaction**, and **q4_0 + 192K for when ultra-long input is truly needed**.
Breaking down this 24G:
- **Model Weights: 16.96 GiB** (IQ4_XS). MoE makes it fast; IQ4_XS provides sufficient quality.
- **Capacity Tier: q4_0 + flash-attn** compresses 192K context into about 5G. If you switch to f16, the same context would require 4x the VRAM and overflow; Q8_0 would start eating into headroom and significantly slow down around 100K.
- **Interaction Tier: Q8_0 + 128K limit**. The client only exposes 120K and starts compressing old messages around 55K. It rarely fills the full 128K in normal use, trading off for more stable generation trajectories. It can still run if it exceeds 100K occasionally, but the first-token latency becomes unsuitable for real-time chat.
- **Capacity Tier Resident: 21.8G, remaining 2.2G**. Idle power draw is low, with no pressure on the desktop.
I previously ran a conservative configuration with Q3_K_XL and short context; now I stick with IQ4_XS weights and only switch KV tiers based on load. **Default to Q8 interaction tier; switch to Q4 capacity tier only for ultra-long inputs**; if the desktop occasionally struggles with memory pressure, I have a Q3 weight tier as a fallback.
### Where Are the Knobs, and When Are They Set?
Apply this to your own card. The four knobs fall into two groups:
### Step 1: Choose Parameter Count + Weight Quantization (Fixed at Download)
**Choose Parameter Count** using the "Tier Table" below: 30–35B on 24G, 14B on 16G, 8B on 12G.
**Choose Weight Quantization**: A single model on Hugging Face will have a row of GGUF files; the filename suffix indicates the quantization level:
- Numbers ≈ bit count: `Q8_0` (8-bit, near-lossless, largest) > `Q6_K` > `Q5_K_M` > `Q4_K_M` (4-bit, the most universal balance point) > `Q3_K_M` (saves VRAM, quality drops).
- `IQ4_XS` / `IQ3_XXS` are i-quant: smaller at the same bit count, with similar quality, requiring imatrix (provided by the publisher). Use these if VRAM is tight.
- `_S / _M / _L` = Small / Medium / Large variants; `_M` is the most common within the same bit count.
**How to Choose**: GGUF file size ≈ VRAM usage. Compare your weight budget against the file size and pick the largest that fits:
```text
Weight Budget = VRAM − KV Cache − Desktop Headroom (leave ~2G)
Example: 24 − 5 (192K q4_0) − 2 ≈ 17G → 35B using IQ4_XS (actual 16.96G) fits perfectly;
switching to Q5_K_M (~24G) would not fit.
```
Experience priority: Aim for `Q4_K_M` first; if it doesn't fit, go down to `IQ4_XS` → `IQ3`; if you have surplus, go up to `Q5_K_M` / `Q6_K`. Don't use below Q3 unless you have no choice.
### Step 2: Set KV Precision (Adjusted at Startup)
KV precision and context length are both startup parameters, adjustable each time. KV relies on three switches:
| Switch | Function | Common Value |
|--------|----------|--------------|
| `--flash-attn` / `-fa` | **Prerequisite for quantized KV, must be on** | On / Off |
| `--cache-type-k` / `-ctk` | K cache precision | `f16` (default) / `q8_0` / `q4_0` |
| `--cache-type-v` / `-ctv` | V cache precision | Same as above |
K and V are set **independently**. The precision ladder is `f16 → q8_0 → q4_0`, with VRAM usage roughly `1 → 1/2 → 1/4`—this is why q4_0 can support about 4x the context of f16. Three typical tiers:
```bash
# Aggressive: Long context priority
llama-server ... -fa -ctk q4_0 -ctv q4_0 -c 196608
# Conservative: Quality priority, ~96K is the comfortable full-VRAM tier on my machine; leave headroom for service limits
llama-server ... -fa -ctk q8_0 -ctv q8_0 -c 98304
# Asymmetric: K is more sensitive to quantization; preserve K, compress V for extreme VRAM savings
llama-server ... -fa -ctk q8_0 -ctv q4_0
```
**How to Verify Usage**: The startup log prints `KV self size = ...`; check how much it takes. In terms of quality, q8_0 is basically lossless, and q4_0 is hard to perceive in daily conversation and coding, though it may occasionally struggle with precise retrieval in long texts. Commands to check VRAM usage vary by platform: `nvidia-smi` (NVIDIA), `cat /sys/class/drm/card0/device/mem_info_vram_used` (AMD Linux), Task Manager → Performance → GPU (Windows).
> Pitfall: Setting `-ctv q4_0` without `-fa` will cause an error or be ignored—quantizing the V cache depends on flash attention.
### Starting Points for Different VRAM Sizes
Pushing the same budget formula down (**these are estimates; calibrate with your own card**):
| VRAM | Realistic Choice |
|------|------------------|
| **24G** | 30–35B MoE (IQ4/Q4) or 32B dense IQ4; long context relies on q4_0 KV |
| **16G** | 14B dense Q4–Q5, or 30B MoE Q3 + moderate layer offload; medium context |
| **12G** | 8–9B Q4–Q5 comfortable (e.g., Qwen3-8B GGUF ~5.4G), or 14B Q4 compact |
| **8G** | 7–8B Q4, short context |
**How to Cut When VRAM is Insufficient, in Priority Order**: First, lower KV precision (f16→q8_0→q4_0) to preserve context → then reduce context length → finally, switch to a smaller model or more aggressive weight quantization. Don't forget that the desktop and games are also competing for this card; leave a few GB for them.
Decisions made, now let's pull the model and run it.
## 4. Running It: From Installation to First Reply
**Install llama.cpp** (must include server + corresponding backend, in priority order):
1. Use [official release binaries](https://github.com/ggerganov/llama.cpp/releases) — cross-platform, download and extract to use, already includes all backends.
2. Linux distros have packages: `apt install llama-cpp` (Debian/Ubuntu), `pacman -S llama.cpp-vulkan` (Arch).
3. If neither is available, compile from source (replace `` with the switch selected in Section 2, e.g., `-DGGML_CUDA=ON`):
```bash
git clone https://github.com/ggerganov/llama.cpp --depth 1
cd llama.cpp
cmake -B build -DLLAMA_CURL=ON
cmake --build build --config Release -j
# Artifacts are in build/bin/
```
**Download the model** to your local model directory:
```bash
pip install -U "huggingface_hub[cli]"
# Search HF for " GGUF", look for quantized repos from unsloth / bartowski
hf download unsloth/Qwen3.6-35B-A3B-MTP-GGUF \
--include "*IQ4_XS*.gguf" --local-dir ~/models
```
Large model GGUFs are often sharded (`…-00001-of-0000N.gguf`); `--include "*IQ4_XS*"` downloads all at once; `llama.cpp` will automatically load the rest when pointed to the first shard.
**Start** (single model, suitable for initial testing):
```bash
llama-server \
-m ~/models/Qwen3.6-35B-A3B-MTP-UD-IQ4_XS.gguf \
-ngl 99 -c 196608 \
--flash-attn --cache-type-k q4_0 --cache-type-v q4_0 \
--host 0.0.0.0 --port 18080
```
Vulkan users with multiple GPUs should add `GGML_VK_VISIBLE_DEVICES=0` in front to lock to the discrete GPU; CUDA users skip this.
`http://localhost:18080` is an OpenAI-compatible endpoint. Verify it's running:
```bash
curl http://localhost:18080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"gpt-3.5-turbo","messages":[{"role":"user","content":"hi"}]}'
```
> What is `MTP` in the path, why use unsloth's GGUF, and is it necessary? Section 5 covers this specifically.
Single-model commands are suitable for testing; for daily persistent use, the **router mode** is recommended (models.ini centrally manages multiple models, with LRU eviction as needed). MTP acceleration and tidal scheduling are also based on router mode, which we'll expand on below.
## 5. Speedup: MTP Speculative Decoding
### What Is It
MTP (Multi-Token Prediction) is a **speculative decoding** technique: it predicts multiple tokens in one forward pass instead of generating them sequentially token by token. On my machine with Qwen3.6-35B-A3B (IQ4_XS), MTP is off at ~75 t/s and on at ~111 t/s, a ~50% speedup. Your actual improvement will vary depending on the model and quantization.
Standard `llama.cpp` models do not have this capability—it comes from **unsloth** (a team specializing in GGUF quantization and acceleration) which **embeds a draft head into the GGUF during quantization**. The draft head "guesses" the next 2 tokens, and the main model verifies them in parallel in one forward pass. Correct guesses are gained; incorrect guesses are rolled back. No separate draft model is needed, and it doesn't consume extra VRAM:
### How to Find It, and What If You Don't Use It?
**MTP is not a native model feature; it was added by unsloth during quantization.** Therefore:
- **Only search for `unsloth/-MTP-GGUF` on HF to find the MTP version**, e.g., [`unsloth/Qwen3.6-35B-A3B-MTP-GGUF`](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF). Standard quantization repos like bartowski do not have it.
- **The filename contains "MTP"**, e.g., `…-MTP-UD-IQ4_XS.gguf`. If it's not there, it doesn't have the draft head and cannot do speculative decoding.
- **Not all models have an unsloth MTP version** — unsloth mainly adds it to the Qwen series. If it's not available, use a standard GGUF; it will still run, just without MTP acceleration.
**Configuration without MTP**: Remove `MTP` from the download command and switch to a standard repo (e.g., `bartowski/Qwen3.6-35B-A3B-GGUF`); remove the `spec-type` and `spec-draft-n-max` lines from the config file; everything else remains the same.
### Router Mode and Full Preset
If you want to manage multiple models or have them persist and auto-start, upgrade to **router mode**: write all models and parameters to `~/.config/llama.cpp/models.ini`, load with `--models-preset`, and set `--models-max 1` to limit residency to one model at a time (24G can only hold one), with LRU auto-eviction:
```ini
# ~/.config/llama.cpp/models.ini
[qwen3.6-mtp-instruct]
model = ~/models/Qwen3.6-35B-A3B-MTP-UD-IQ4_XS.gguf
ngl = 99 # All layers on GPU (99=all)
ctx-size = 196608 # 192K context
chat-template = chatml # OpenAI chat template, recommended even without tools
chat-template-kwargs = {"enable_thinking": false}
flash-attn = 1
cache-type-k = q4_0
cache-type-v = q4_0
spec-type = draft-mtp # Next two lines only needed for MTP models
spec-draft-n-max = 2
```
> Pitfall: `chat-template=chatml` must be set explicitly — when using the model's native template, tool call prefixes may generate incorrect tokens leading to premature EOS (llama.cpp #19513). This applies even if you're not using MTP or tool calls.
Adding a new model only requires adding an entry in the INI file; the Open WebUI list updates automatically. See the "Full Configuration" section for the systemd unit and enable commands.
## 6. Persistent Scheduling: Models Online, Power Tidal (Linux)
Running fast is temporary; running persistently without wasting power is daily life. The following is based on Linux systemd and AMD sysfs interfaces (WSL2 users follow this section directly; the distro comes with systemd; pure Windows users skip this, the core idea is the same: run a background process to manage llama-server, switching power tiers based on GPU load):
Three mechanisms:
**① Boot Warmup.** Router lazy-loading relies on systemd `ExecStartPost` to send a request after startup to load the model into VRAM. The warmup payload must use `-d @file` — inline JSON will have its quotes swallowed by systemd. `warmup.json` is just one line: `{"model":"qwen3.6-mtp-instruct","messages":[{"role":"user","content":"hi"}],"max_tokens":1}`.
**② Power Tiers Switch Based on Real Load.** `llama-gpu-sync.timer` triggers every 15s. The criterion is `gpu_busy_percent ≥ 50%` to switch to COMPUTE (Tier 5); if idle for over 90s, it drops back to BOOTUP_DEFAULT (Tier 0). It only touches 0/5, not the `3d_full_screen` used by games. The key is **using real GPU occupancy rather than "whether the model is loaded"** — if it's resident but not inferring, it should stay in the power-saving tier. AMD achieves this via sysfs `pp_power_profile_mode`; NVIDIA uses `nvidia-smi -pl`.
**③ Entire Card Yielded During Gaming.** [gamemode](https://github.com/FeralInteractive/gamemode) (available in most distros) detects Steam game launches and stops llama-server, dedicating all VRAM to the game, then restarts it upon exit. The model remains in page cache, and warmup takes ~6s to restore:
```ini
# ~/.config/gamemode.ini
[custom]
start = systemctl --user stop llama-server
end = systemctl --user start llama-server
```
> **Full systemd unit, power switching script, sudoers passwordless config, and enable commands are in the "Full Configuration" section.**
## 7. Performance Baseline
`llama.cpp` benchmarking terms:
- **pp512 (prompt processing)**: Speed of processing 512 input tokens in parallel (token/s), determining how fast "reading long context" is.
- **tg128 (text generation)**: Speed of continuously generating 128 output tokens, i.e., the intuitive "typing" speed — human reading is about 5–10 t/s; 100+ means "the screen is flooded before the sentence is finished."
- **MTP Draft Acceptance Rate**: The proportion of draft tokens guessed by the draft head that are verified by the main model; the higher, the greater the equivalent speedup.
My Machine Measurements (Qwen3.6-35B-A3B · IQ4_XS · 192K):
| Metric | Value | What It Means |
|--------|-------|---------------|
| pp512 | ~3150 t/s | Short baseline prefill is fast; cannot be linearly extrapolated to 100K |
| tg128 | ~111 t/s | Far exceeds reading speed, feels instantaneous |
| MTP Acceptance Rate | ~52% | Draft head hits about half the time |
| Resident VRAM | 21.8 / 24 GB | Remaining 2.2G |
**Measure for Yourself**: `llama-bench` in one command, compare different cards and quantizations:
```bash
llama-bench -m ~/models/Qwen3.6-35B-A3B-MTP-UD-IQ4_XS.gguf -ngl 99
# Output is two lines: pp512 / tg128
# If the resident service is running, stop it first: systemctl --user stop llama-server (Linux)
```
### Roleplay Load Measurements: 8B Large Window, 35B, and KV Precision
On 2026-08-24, I ran a local A/B test for SillyTavern's long conversation load. The question isn't "which model has a larger nominal window," but three more practical ones:
1. How much roleplay quality is lost by switching to 8B for a large window?
2. Does dropping 35B's KV from Q8_0 to q4_0 stably degrade writing style or fact retention?
3. Although 100K–192K can be configured, is re-reading such long history every turn still suitable for real-time chat?
Model quality and long-context A/B testing used only synthetic scenarios and random fact codes. Short scenarios used the same prompts and sampling conditions; long texts scattered 8 target facts at different positions in the text, scored by strict JSON mapping; subjective quality was summarized by double-blind scoring with swapped A/B order. The turn estimates below only aggregate tokens from local existing records; no body text is output or retained. The sample size is small; scores are for local selection only, not a universal model leaderboard.
#### Model Size: 8B's Large Window Cannot Make Up for Quality Gap
| Metric | 35B-A3B | 8B | Interpretation |
|--------|---------|----|----------------|
| Short Scenario Double-Blind Score | **8.64** | 6.36 | Gap is mainly in writing style, character believability, continuity, and avoiding repetition |
| Warmup Generation Speed | **126.7 t/s** | 107.4 t/s | MoE's active parameters are small; 35B total params ≠ 35B calculated per token |
| Median Full Response Latency | 3.74s | **3.53s** | Actual feel is basically the same tier |
| Cold Load | 12.73s | **3.47s** | 8B's clear advantage is only faster first load |
8B did not trade faster warmup inference for quality; on my backend, 35B-A3B actually has higher generation throughput. If the task is immersive long conversation, especially requiring the model to actively advance, maintain character state, and avoid looping in the same scene, **do not downgrade the main model to 8B just for the window specification**.
#### Long Context: Being Able to Fit It Doesn't Mean It's Stable or Fast Enough
The times in the table below are wall-clock times for the entire non-streaming request; context lengths are approximate; different model tokenizers cause slight deviations in actual input.
| Target Context | 35B · Q8_0 KV | 35B · q4_0 KV | 8B · Q8_0 KV |
|----------------|---------------|---------------|--------------|
| 22K | 8/8 · ~11.3s | 8/8 · ~14–15s | 6/8 · ~23.5s |
| 52K | — | 8/8 · ~39.9s | 8/8 · ~72.5s |
| 100K | 8/8 · ~111.5s | 8/8 · ~101.7s | 6/8 · ~231.8s |
| 160K | — | 8/8 · ~215.1s | Connection interrupted after ~5 minutes, backend only processed ~73% |
Here are three counter-intuitive results:
- **8B's nominal long window is not a stable utilization capability.** It passed all at 52K, but only got 6/8 at both 22K and 100K, indicating that smaller models are more sensitive to fact positions and specific inputs, not simply "shorter is more accurate."
- **8B is actually slower on long inputs.** 100K is ~2.3x slower than 35B q4_0; 160K is already unusable within the current routing connection timeout.
- **The main cost of ultra-long windows is the first-token wait per turn.** 35B q4_0's 160K passed all, but took ~215 seconds. Since chat re-reads history every turn, 192K is more of a capacity limit or offline document tier, not a real-time workspace that should be maxed out long-term.
#### KV Precision: Changes Generation Trajectory, But No Evidence of Stable Degradation
Same 35B, same weights, same seed and prompt, only switching KV cache precision:
| Control | Q8_0 | q4_0 |
|---------|------|------|
| 22K Precise Retrieval | 8/8 | 8/8 |
| 100K Precise Retrieval | 8/8 | 8/8 |
| Short Scenario Double-Blind Score | **9.13** | 8.38 |
| 22K Long Scenario Double-Blind Score | 7.90 | **9.30** |
| Thought Format Tag Leakage | 0/5 | 1/5 |
Q8_0 is better in short scenarios, q4_0 is better in long scenarios, and the scoring direction reverses. This shows that KV quantization is sufficient to send deterministic generation onto different token trajectories; a single output may be better or worse; but combined with 22K–160K retrieval passing all, **this round observed no evidence that q4_0 continuously degrades text quality or context precision**. On the other hand, q4_0's one format tag leakage and small sample fluctuations show it shouldn't be called "completely lossless."
Therefore, adopt a dual-tier approach rather than betting on a single one:
| Use Case | Server Context | KV | Client Budget | Reason for Choice |
|----------|----------------|----|---------------|-------------------|
| Daily SillyTavern / Roleplay | 131,072 | Q8_0 / Q8_0 | 122,880 | Default tier; preserves generation stability, far larger than 32K |
| Ultra-long Documents / Temporary Continuation of Very Long History | 196,608 | q4_0 / q4_0 | Switch temporarily per task | 100K–160K retrieval is stable, better capacity and bandwidth efficiency |
#### SillyTavern's Actual Compression and Output Budget
The current interaction tier doesn't wait until 120K is full to summarize:
- Summary calls use an independent 8,192 context / 160 output token non-streaming preset, but still point to the same 35B router, avoiding repeated unloading and reloading of the model for summaries.
- Auto-summary is on, delay 2, batch 2; it starts **preparing** per-segment summaries in the first few turns.
- The threshold for actually excluding old original text and injecting the summary is 45% of the client window, i.e., `122880 × 45% = 55296` tokens.
- After reaching the threshold, the last user message is retained; old original text is taken over by short/long summaries; thus, the session doesn't end at a fixed number of turns, but gradually transitions from verbatim history to compressed memory.
- In a minimal sample on my machine that only counts tokens, not viewing body text, the median for a full turn is about 425 tokens. Pure math suggests ~130 turns to trigger; deducting character cards, world books, system prompts, and considering reply fluctuations, **conservatively estimate 80–120 turns of original history understanding**. Long replies trigger earlier, short replies later.
- Chinese is not 1 character = 1 token. My machine's tokenizer sample is about 1 token ≈ 1.62 Chinese characters; originally 1200 tokens was already about 1900 characters; the current 1600 token limit is about 2600 characters. `max_tokens` is a limit, not a target length; setting it to 1600 is to avoid hard truncation of natural transitions, not to encourage filling every turn.
The final decision can be condensed into one sentence:
> **Preserve 35B roleplay quality; use 128K-level Q8 + compress at ~55K for daily use; switch to 192K Q4 only when you truly need to read over 100K. Don't switch to 8B just for a nominal large window, and don't mistake the startup context limit for a length suitable for real-time chat.**
## 8. After It Runs: Connecting Clients
`llama-server` exposes an **OpenAI-compatible endpoint** (`http://localhost:18080/v1`); any client that accepts a custom base_url can connect — Open WebUI, IDE plugins, CLI agents, just point the base_url there.
The model is ready to use at this point. But **"what to do with it, which tasks belong to it, which should be left to Claude Opus, and whether to configure cloud fallback" is another topic** — that's where local models are truly worth pondering, covered in a separate post: see [local-llm-usage.md](local-llm-usage.md).
## 9. Pitfall Collection
- **systemd inline JSON has quotes swallowed** → warmup must use `-d @file`.
- **With integrated and discrete GPUs, the model ran on the integrated GPU** → Lock to discrete GPU using `GGML_VK_VISIBLE_DEVICES=0` (Vulkan) or `CUDA_VISIBLE_DEVICES=0` (CUDA).
- **Quantized KV without flash-attn** → `--flash-attn` is a prerequisite for `cache-type q4_0/q8_0`; otherwise, it's ignored or errors.
- **Tool calls causing premature EOS** → Explicitly specify `chat-template=chatml` to avoid native template bugs (llama.cpp #19513).
- **Thinking resident = always high power** → Switch tiers using `gpu_busy_percent` (AMD) or `nvidia-smi` (NVIDIA); drop to power-saving tier when idle.
- **Wrong VRAM budget calculation** → First estimate `Model Size + KV (layers × ctx × precision) + Buffer + Desktop`; leave 1–2G headroom; don't push to zero.
## 10. Full Configuration (Linux)
The previous sections explained "what to do" and only pasted key snippets; here, all files in the entire chain are listed completely, created in order, and finally enabled.
> WSL2 users follow this section directly; pure Windows users use Task Scheduler to replace systemd, and handle power switching separately for AMD/NVIDIA cards.
### Prerequisites: Scripts + Passwordless sudo
`llama-profile-sync` calls `gpu-profile-set`, which needs root to write to `pp_power_profile_mode` (AMD GPU sysfs); `ExecStopPost` in systemctl does the same. User units cannot pop up sudo password prompts; passwordless access must be configured:
```bash
sudo tee /usr/local/bin/gpu-profile-set <<'EOF'
#!/bin/sh
case "$1" in
bootup_default) idx=0 ;;
3d_full_screen) idx=1 ;;
power_saving) idx=2 ;;
video) idx=3 ;;
vr) idx=4 ;;
compute) idx=5 ;;
[0-5]) idx=$1 ;;
*) echo "Unknown profile: $1" >&2; exit 1 ;;
esac
printf "%s\n" "$idx" > /sys/class/drm/card0/device/pp_power_profile_mode
EOF
sudo chmod 755 /usr/local/bin/gpu-profile-set
sudo tee /usr/local/bin/llama-profile-sync <<'EOF'
#!/bin/bash
PROFILE_PATH="/sys/class/drm/card0/device/pp_power_profile_mode"
BUSY_PATH="/sys/class/drm/card0/device/gpu_busy_percent"
STATE="/tmp/llama-profile-last-busy"
BUSY_THRESHOLD=50
HOLD_SECONDS=90
current_idx() { grep '\*' "$PROFILE_PATH" | awk '{print $1}'; }
set_profile() {
local want=$1 cur
cur=$(current_idx)
case "$cur" in 0|5) ;; *) return 0 ;; esac
case "$want" in
compute) [ "$cur" = "5" ] && return 0 ;;
bootup_default) [ "$cur" = "0" ] && return 0 ;;
esac
sudo /usr/local/bin/gpu-profile-set "$want"
}
if ! systemctl --user is-active --quiet llama-server 2>/dev/null; then
set_profile bootup_default; exit 0
fi
busy=$(cat "$BUSY_PATH" 2>/dev/null || echo 0)
now=$(date +%s)
if [ "${busy:-0}" -ge "$BUSY_THRESHOLD" ]; then
echo "$now" > "$STATE"
set_profile compute
else
last=$(cat "$STATE" 2>/dev/null || echo 0)
[ $((now - last)) -lt "$HOLD_SECONDS" ] && set_profile compute || set_profile bootup_default
fi
EOF
sudo chmod 755 /usr/local/bin/llama-profile-sync
# Passwordless sudo (replace username with your own):
# echo "USERNAME ALL=(ALL) NOPASSWD: /usr/local/bin/gpu-profile-set" | sudo tee /etc/sudoers.d/llama-profile
```
> `pp_power_profile_mode` / `gpu_busy_percent` are AMD GPU sysfs interfaces. NVIDIA power management uses `nvidia-smi -pl` or `nvidia-persistenced`; these scripts do not apply.
### systemd User Units (Three Files)
```ini
# ~/.config/systemd/user/llama-server.service
[Unit]
Description=llama.cpp server (router mode, Vulkan)
After=network.target
[Service]
Environment=GGML_VK_VISIBLE_DEVICES=0 # Vulkan lock to discrete GPU; CUDA users delete this line or change to CUDA_VISIBLE_DEVICES=0
ExecStart=/usr/bin/llama-server \
--models-preset %h/.config/llama.cpp/models.ini \
--models-max 1 \
--host 0.0.0.0 \
--port 18080 \
--metrics
# Warmup: Preload model into VRAM; -d @file avoids systemd swallowing quotes
ExecStartPost=-/usr/bin/bash -c 'for i in $(seq 1 60); do curl -sf --max-time 90 http://127.0.0.1:18080/v1/chat/completions -H "Content-Type: application/json" -d @%h/.config/llama.cpp/warmup.json >/dev/null && exit 0; sleep 2; done'
# When service stops, GPU returns to power-saving tier
ExecStopPost=/usr/bin/sudo /usr/local/bin/gpu-profile-set bootup_default # AMD power management; NVIDIA users delete this line
Restart=on-failure
RestartSec=5
[Install]
WantedBy=default.target
```
```ini
# ~/.config/systemd/user/llama-gpu-sync.service
[Unit]
Description=Sync GPU profile with llama-server model state
After=llama-server.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/llama-profile-sync
```
```ini
# ~/.config/systemd/user/llama-gpu-sync.timer
[Unit]
Description=Poll llama-server model state for GPU profile switching
[Timer]
OnBootSec=15s
OnUnitActiveSec=15s
[Install]
WantedBy=timers.target
```
### Warmup
```bash
echo '{"model":"qwen3.6-mtp-instruct","messages":[{"role":"user","content":"hi"}],"max_tokens":1}' \
> ~/.config/llama.cpp/warmup.json
```
### Enable
```bash
# linger — Key for persistent models: service survives logout
sudo loginctl enable-linger "$USER"
# reload + enable
systemctl --user daemon-reload
systemctl --user enable --now llama-server.service
systemctl --user enable --now llama-gpu-sync.timer
# Verify
systemctl --user status llama-server
systemctl --user status llama-gpu-sync.timer
```
> `--user` is not `--system`: GPU context (DRM render node) is in the user session; running a system unit might fail to open the device; permissions are also much smaller.
## Related Docs
- [local-llm-usage.md](local-llm-usage.md) — After running: Boundaries (Local vs Opus), What it can do, llm-jobs automation, Practical lessons learned
- [software.md](software.md) — Software Stack List
- [timers-and-crons.md](timers-and-crons.md) — Timers and Crons like llama-gpu-sync.timer
- [monitoring.md](monitoring.md) — Monitoring Architecture (GPU metrics via node_exporter → Prometheus → Grafana)