Local LLM Deployment Notes

Running local LLMs on consumer-grade GPUs boils down to a budgeting problem: model weights, KV cache, and the desktop environment compete for the same VRAM. How you allocate it determines the maximum model size and context length you can run. This article uses an RX 7900 XTX (24GB) as a case study to provide a selection methodology and a fully reproducible configuration, extending to 16GB and 12GB GPUs. All commands are compatible with Linux; WSL2 follows the same logic.

1. Constraints: An Inequality, Independent of the GPU

All decisions for local deployment—which quantization to choose, what precision to set for KV cache, and how long to make the context—are solving the same problem:

Model Weights + KV Cache + Runtime Buffer + Desktop Headroom ≤ VRAM Capacity

This inequality holds for 24GB, 16GB, and 12GB cards; only the right-hand side differs. Treat it as a budget to allocate, and it becomes the universal starting point for deployment. My machine has a 7900 XTX (24GB), which serves as the example below.

2. Step 1: Choose the Backend Based on Your GPU

llama.cpp supports multiple GPU backends. You do not need to install any vendor-specific AI training stacks (CUDA Toolkit / ROCm / oneAPI); it relies solely on the runtime provided by the GPU driver. Choose based on your card:

Your GPU	Backend	Compile Flag
NVIDIA (GTX 10 series and above)	CUDA	-DGGML_CUDA=ON
AMD (RX series / Radeon VII)	Vulkan	-DGGML_VULKAN=ON
Intel Arc	Vulkan	-DGGML_VULKAN=ON
Apple Silicon (M series)	Metal	-DGGML_METAL=ON

Common questions:

"Do I need to install the CUDA Toolkit for NVIDIA cards?" No. llama.cpp only depends on the CUDA runtime provided by the GPU driver; you don't need to install the multi-gigabyte CUDA Toolkit.

"Why is it Vulkan for AMD instead of ROCm?" ROCm is for training with PyTorch/vLLM; it's cumbersome to install (especially on Windows) and unnecessary for GGUF inference. Vulkan uses the runtime provided by the GPU driver and works out-of-the-box on both Linux and Windows.

"What if I have both integrated and discrete GPUs?" Vulkan users can lock to the discrete GPU using GGML_VK_VISIBLE_DEVICES=0 (0 = first card); CUDA users use CUDA_VISIBLE_DEVICES=0. Metal users typically don't need to worry about this.

Windows Users: Run all commands below in WSL2. WSL2 can directly access GPUs (NVIDIA cards work with official drivers; AMD cards require kernel 5.15+ and the official AMD WSL driver). The compilation and installation process is identical to Linux. First, install a distribution with wsl --install, then follow the steps.

Now, let's address the core issue: how to allocate VRAM.

3. How to Choose: Spend VRAM Where It Counts

There are only four knobs to adjust: parameter count and weight quantization are fixed when downloading the file; context length and KV precision are startup parameters that can be adjusted each time. Let's look at the solution for my machine first, then explain how to make the same decisions for your own card.

My Solution: Prioritize Model Size, Then Slice KV by Workload

I chose Qwen3.6-35B-A3B (MoE, 35B total parameters but only 3B active per step, offering small-model speed with large-model capacity). The real trade-off isn't jumping back and forth between 35B and 8B; it's preserving the 35B model's expressiveness and consistency, while preparing two KV tiers for the same model: Q8_0 + 128K for daily interaction, and q4_0 + 192K for when ultra-long input is truly needed.

Breaking down this 24G:

• Model Weights: 16.96 GiB (IQ4_XS). MoE makes it fast; IQ4_XS provides sufficient quality.
• Capacity Tier: q4_0 + flash-attn compresses 192K context into about 5G. If you switch to f16, the same context would require 4x the VRAM and overflow; Q8_0 would start eating into headroom and significantly slow down around 100K.
• Interaction Tier: Q8_0 + 128K limit. The client only exposes 120K and starts compressing old messages around 55K. It rarely fills the full 128K in normal use, trading off for more stable generation trajectories. It can still run if it exceeds 100K occasionally, but the first-token latency becomes unsuitable for real-time chat.
• Capacity Tier Resident: 21.8G, remaining 2.2G. Idle power draw is low, with no pressure on the desktop.

I previously ran a conservative configuration with Q3_K_XL and short context; now I stick with IQ4_XS weights and only switch KV tiers based on load. Default to Q8 interaction tier; switch to Q4 capacity tier only for ultra-long inputs; if the desktop occasionally struggles with memory pressure, I have a Q3 weight tier as a fallback.

Where Are the Knobs, and When Are They Set?

Apply this to your own card. The four knobs fall into two groups:

Step 1: Choose Parameter Count + Weight Quantization (Fixed at Download)

Choose Parameter Count using the "Tier Table" below: 30–35B on 24G, 14B on 16G, 8B on 12G.

Choose Weight Quantization: A single model on Hugging Face will have a row of GGUF files; the filename suffix indicates the quantization level:

• Numbers ≈ bit count: Q8_0 (8-bit, near-lossless, largest) > Q6_K > Q5_K_M > Q4_K_M (4-bit, the most universal balance point) > Q3_K_M (saves VRAM, quality drops).
• IQ4_XS / IQ3_XXS are i-quant: smaller at the same bit count, with similar quality, requiring imatrix (provided by the publisher). Use these if VRAM is tight.
• _S / _M / _L = Small / Medium / Large variants; _M is the most common within the same bit count.

How to Choose: GGUF file size ≈ VRAM usage. Compare your weight budget against the file size and pick the largest that fits:

Weight Budget = VRAM − KV Cache − Desktop Headroom (leave ~2G)
Example: 24 − 5 (192K q4_0) − 2 ≈ 17G → 35B using IQ4_XS (actual 16.96G) fits perfectly;
         switching to Q5_K_M (~24G) would not fit.

Experience priority: Aim for Q4_K_M first; if it doesn't fit, go down to IQ4_XS → IQ3; if you have surplus, go up to Q5_K_M / Q6_K. Don't use below Q3 unless you have no choice.

Step 2: Set KV Precision (Adjusted at Startup)

KV precision and context length are both startup parameters, adjustable each time. KV relies on three switches:

Switch	Function	Common Value
--flash-attn / -fa	Prerequisite for quantized KV, must be on	On / Off
--cache-type-k / -ctk	K cache precision	f16 (default) / q8_0 / q4_0
--cache-type-v / -ctv	V cache precision	Same as above

K and V are set independently. The precision ladder is f16 → q8_0 → q4_0, with VRAM usage roughly 1 → 1/2 → 1/4—this is why q4_0 can support about 4x the context of f16. Three typical tiers:

# Aggressive: Long context priority
llama-server ... -fa -ctk q4_0 -ctv q4_0 -c 196608
# Conservative: Quality priority, ~96K is the comfortable full-VRAM tier on my machine; leave headroom for service limits
llama-server ... -fa -ctk q8_0 -ctv q8_0 -c 98304
# Asymmetric: K is more sensitive to quantization; preserve K, compress V for extreme VRAM savings
llama-server ... -fa -ctk q8_0 -ctv q4_0

How to Verify Usage: The startup log prints KV self size = ...; check how much it takes. In terms of quality, q8_0 is basically lossless, and q4_0 is hard to perceive in daily conversation and coding, though it may occasionally struggle with precise retrieval in long texts. Commands to check VRAM usage vary by platform: nvidia-smi (NVIDIA), cat /sys/class/drm/card0/device/mem_info_vram_used (AMD Linux), Task Manager → Performance → GPU (Windows).

Pitfall: Setting -ctv q4_0 without -fa will cause an error or be ignored—quantizing the V cache depends on flash attention.

Starting Points for Different VRAM Sizes

Pushing the same budget formula down (these are estimates; calibrate with your own card):

VRAM	Realistic Choice
24G	30–35B MoE (IQ4/Q4) or 32B dense IQ4; long context relies on q4_0 KV
16G	14B dense Q4–Q5, or 30B MoE Q3 + moderate layer offload; medium context
12G	8–9B Q4–Q5 comfortable (e.g., Qwen3-8B GGUF ~5.4G), or 14B Q4 compact
8G	7–8B Q4, short context

How to Cut When VRAM is Insufficient, in Priority Order: First, lower KV precision (f16→q8_0→q4_0) to preserve context → then reduce context length → finally, switch to a smaller model or more aggressive weight quantization. Don't forget that the desktop and games are also competing for this card; leave a few GB for them.

Decisions made, now let's pull the model and run it.

4. Running It: From Installation to First Reply

Install llama.cpp (must include server + corresponding backend, in priority order):

1. Use official release binaries — cross-platform, download and extract to use, already includes all backends.
2. Linux distros have packages: apt install llama-cpp (Debian/Ubuntu), pacman -S llama.cpp-vulkan (Arch).
3. If neither is available, compile from source (replace <YOUR_BACKEND> with the switch selected in Section 2, e.g., -DGGML_CUDA=ON):

git clone https://github.com/ggerganov/llama.cpp --depth 1
cd llama.cpp
cmake -B build <YOUR_BACKEND> -DLLAMA_CURL=ON
cmake --build build --config Release -j
# Artifacts are in build/bin/

Download the model to your local model directory:

pip install -U "huggingface_hub[cli]"
# Search HF for "<model name> GGUF", look for quantized repos from unsloth / bartowski
hf download unsloth/Qwen3.6-35B-A3B-MTP-GGUF \
  --include "*IQ4_XS*.gguf" --local-dir ~/models

Large model GGUFs are often sharded (…-00001-of-0000N.gguf); --include "*IQ4_XS*" downloads all at once; llama.cpp will automatically load the rest when pointed to the first shard.

Start (single model, suitable for initial testing):

llama-server \
  -m ~/models/Qwen3.6-35B-A3B-MTP-UD-IQ4_XS.gguf \
  -ngl 99 -c 196608 \
  --flash-attn --cache-type-k q4_0 --cache-type-v q4_0 \
  --host 0.0.0.0 --port 18080

Vulkan users with multiple GPUs should add GGML_VK_VISIBLE_DEVICES=0 in front to lock to the discrete GPU; CUDA users skip this.

http://localhost:18080 is an OpenAI-compatible endpoint. Verify it's running:

curl http://localhost:18080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-3.5-turbo","messages":[{"role":"user","content":"hi"}]}'

What is MTP in the path, why use unsloth's GGUF, and is it necessary? Section 5 covers this specifically.

Single-model commands are suitable for testing; for daily persistent use, the router mode is recommended (models.ini centrally manages multiple models, with LRU eviction as needed). MTP acceleration and tidal scheduling are also based on router mode, which we'll expand on below.

5. Speedup: MTP Speculative Decoding

What Is It

MTP (Multi-Token Prediction) is a speculative decoding technique: it predicts multiple tokens in one forward pass instead of generating them sequentially token by token. On my machine with Qwen3.6-35B-A3B (IQ4_XS), MTP is off at ~75 t/s and on at ~111 t/s, a ~50% speedup. Your actual improvement will vary depending on the model and quantization.

Standard llama.cpp models do not have this capability—it comes from unsloth (a team specializing in GGUF quantization and acceleration) which embeds a draft head into the GGUF during quantization. The draft head "guesses" the next 2 tokens, and the main model verifies them in parallel in one forward pass. Correct guesses are gained; incorrect guesses are rolled back. No separate draft model is needed, and it doesn't consume extra VRAM:

How to Find It, and What If You Don't Use It?

MTP is not a native model feature; it was added by unsloth during quantization. Therefore:

• Only search for unsloth/<model name>-MTP-GGUF on HF to find the MTP version, e.g., unsloth/Qwen3.6-35B-A3B-MTP-GGUF. Standard quantization repos like bartowski do not have it.
• The filename contains "MTP", e.g., …-MTP-UD-IQ4_XS.gguf. If it's not there, it doesn't have the draft head and cannot do speculative decoding.
• Not all models have an unsloth MTP version — unsloth mainly adds it to the Qwen series. If it's not available, use a standard GGUF; it will still run, just without MTP acceleration.

Configuration without MTP: Remove MTP from the download command and switch to a standard repo (e.g., bartowski/Qwen3.6-35B-A3B-GGUF); remove the spec-type and spec-draft-n-max lines from the config file; everything else remains the same.

Router Mode and Full Preset

If you want to manage multiple models or have them persist and auto-start, upgrade to router mode: write all models and parameters to ~/.config/llama.cpp/models.ini, load with --models-preset, and set --models-max 1 to limit residency to one model at a time (24G can only hold one), with LRU auto-eviction:

# ~/.config/llama.cpp/models.ini
[qwen3.6-mtp-instruct]
model = ~/models/Qwen3.6-35B-A3B-MTP-UD-IQ4_XS.gguf
ngl = 99                      # All layers on GPU (99=all)
ctx-size = 196608             # 192K context
chat-template = chatml        # OpenAI chat template, recommended even without tools
chat-template-kwargs = {"enable_thinking": false}
flash-attn = 1
cache-type-k = q4_0
cache-type-v = q4_0
spec-type = draft-mtp          # Next two lines only needed for MTP models
spec-draft-n-max = 2

Pitfall: chat-template=chatml must be set explicitly — when using the model's native template, tool call prefixes may generate incorrect tokens leading to premature EOS (llama.cpp #19513). This applies even if you're not using MTP or tool calls.

Adding a new model only requires adding an entry in the INI file; the Open WebUI list updates automatically. See the "Full Configuration" section for the systemd unit and enable commands.

6. Persistent Scheduling: Models Online, Power Tidal (Linux)

Running fast is temporary; running persistently without wasting power is daily life. The following is based on Linux systemd and AMD sysfs interfaces (WSL2 users follow this section directly; the distro comes with systemd; pure Windows users skip this, the core idea is the same: run a background process to manage llama-server, switching power tiers based on GPU load):

Three mechanisms:

① Boot Warmup. Router lazy-loading relies on systemd ExecStartPost to send a request after startup to load the model into VRAM. The warmup payload must use -d @file — inline JSON will have its quotes swallowed by systemd. warmup.json is just one line: {"model":"qwen3.6-mtp-instruct","messages":[{"role":"user","content":"hi"}],"max_tokens":1}.

② Power Tiers Switch Based on Real Load. llama-gpu-sync.timer triggers every 15s. The criterion is gpu_busy_percent ≥ 50% to switch to COMPUTE (Tier 5); if idle for over 90s, it drops back to BOOTUP_DEFAULT (Tier 0). It only touches 0/5, not the 3d_full_screen used by games. The key is using real GPU occupancy rather than "whether the model is loaded" — if it's resident but not inferring, it should stay in the power-saving tier. AMD achieves this via sysfs pp_power_profile_mode; NVIDIA uses nvidia-smi -pl.

③ Entire Card Yielded During Gaming. gamemode (available in most distros) detects Steam game launches and stops llama-server, dedicating all VRAM to the game, then restarts it upon exit. The model remains in page cache, and warmup takes ~6s to restore:

# ~/.config/gamemode.ini
[custom]
start = systemctl --user stop llama-server
end   = systemctl --user start llama-server

Full systemd unit, power switching script, sudoers passwordless config, and enable commands are in the "Full Configuration" section.

7. Performance Baseline

llama.cpp benchmarking terms:

• pp512 (prompt processing): Speed of processing 512 input tokens in parallel (token/s), determining how fast "reading long context" is.
• tg128 (text generation): Speed of continuously generating 128 output tokens, i.e., the intuitive "typing" speed — human reading is about 5–10 t/s; 100+ means "the screen is flooded before the sentence is finished."
• MTP Draft Acceptance Rate: The proportion of draft tokens guessed by the draft head that are verified by the main model; the higher, the greater the equivalent speedup.

My Machine Measurements (Qwen3.6-35B-A3B · IQ4_XS · 192K):

Metric	Value	What It Means
pp512	~3150 t/s	Short baseline prefill is fast; cannot be linearly extrapolated to 100K
tg128	~111 t/s	Far exceeds reading speed, feels instantaneous
MTP Acceptance Rate	~52%	Draft head hits about half the time
Resident VRAM	21.8 / 24 GB	Remaining 2.2G

Measure for Yourself: llama-bench in one command, compare different cards and quantizations:

llama-bench -m ~/models/Qwen3.6-35B-A3B-MTP-UD-IQ4_XS.gguf -ngl 99
# Output is two lines: pp512 / tg128
# If the resident service is running, stop it first: systemctl --user stop llama-server (Linux)

Roleplay Load Measurements: 8B Large Window, 35B, and KV Precision

On 2026-08-24, I ran a local A/B test for SillyTavern's long conversation load. The question isn't "which model has a larger nominal window," but three more practical ones:

1. How much roleplay quality is lost by switching to 8B for a large window?
2. Does dropping 35B's KV from Q8_0 to q4_0 stably degrade writing style or fact retention?
3. Although 100K–192K can be configured, is re-reading such long history every turn still suitable for real-time chat?

Model quality and long-context A/B testing used only synthetic scenarios and random fact codes. Short scenarios used the same prompts and sampling conditions; long texts scattered 8 target facts at different positions in the text, scored by strict JSON mapping; subjective quality was summarized by double-blind scoring with swapped A/B order. The turn estimates below only aggregate tokens from local existing records; no body text is output or retained. The sample size is small; scores are for local selection only, not a universal model leaderboard.

Model Size: 8B's Large Window Cannot Make Up for Quality Gap

Metric	35B-A3B	8B	Interpretation
Short Scenario Double-Blind Score	8.64	6.36	Gap is mainly in writing style, character believability, continuity, and avoiding repetition
Warmup Generation Speed	126.7 t/s	107.4 t/s	MoE's active parameters are small; 35B total params ≠ 35B calculated per token
Median Full Response Latency	3.74s	3.53s	Actual feel is basically the same tier
Cold Load	12.73s	3.47s	8B's clear advantage is only faster first load

8B did not trade faster warmup inference for quality; on my backend, 35B-A3B actually has higher generation throughput. If the task is immersive long conversation, especially requiring the model to actively advance, maintain character state, and avoid looping in the same scene, do not downgrade the main model to 8B just for the window specification.

Long Context: Being Able to Fit It Doesn't Mean It's Stable or Fast Enough

The times in the table below are wall-clock times for the entire non-streaming request; context lengths are approximate; different model tokenizers cause slight deviations in actual input.

Target Context	35B · Q8_0 KV	35B · q4_0 KV	8B · Q8_0 KV
22K	8/8 · ~11.3s	8/8 · ~14–15s	6/8 · ~23.5s
52K	—	8/8 · ~39.9s	8/8 · ~72.5s
100K	8/8 · ~111.5s	8/8 · ~101.7s	6/8 · ~231.8s
160K	—	8/8 · ~215.1s	Connection interrupted after ~5 minutes, backend only processed ~73%

Here are three counter-intuitive results:

• 8B's nominal long window is not a stable utilization capability. It passed all at 52K, but only got 6/8 at both 22K and 100K, indicating that smaller models are more sensitive to fact positions and specific inputs, not simply "shorter is more accurate."
• 8B is actually slower on long inputs. 100K is ~2.3x slower than 35B q4_0; 160K is already unusable within the current routing connection timeout.
• The main cost of ultra-long windows is the first-token wait per turn. 35B q4_0's 160K passed all, but took ~215 seconds. Since chat re-reads history every turn, 192K is more of a capacity limit or offline document tier, not a real-time workspace that should be maxed out long-term.

KV Precision: Changes Generation Trajectory, But No Evidence of Stable Degradation

Same 35B, same weights, same seed and prompt, only switching KV cache precision:

Control	Q8_0	q4_0
22K Precise Retrieval	8/8	8/8
100K Precise Retrieval	8/8	8/8
Short Scenario Double-Blind Score	9.13	8.38
22K Long Scenario Double-Blind Score	7.90	9.30
Thought Format Tag Leakage	0/5	1/5

Q8_0 is better in short scenarios, q4_0 is better in long scenarios, and the scoring direction reverses. This shows that KV quantization is sufficient to send deterministic generation onto different token trajectories; a single output may be better or worse; but combined with 22K–160K retrieval passing all, this round observed no evidence that q4_0 continuously degrades text quality or context precision. On the other hand, q4_0's one format tag leakage and small sample fluctuations show it shouldn't be called "completely lossless."

Therefore, adopt a dual-tier approach rather than betting on a single one:

Use Case	Server Context	KV	Client Budget	Reason for Choice
Daily SillyTavern / Roleplay	131,072	Q8_0 / Q8_0	122,880	Default tier; preserves generation stability, far larger than 32K
Ultra-long Documents / Temporary Continuation of Very Long History	196,608	q4_0 / q4_0	Switch temporarily per task	100K–160K retrieval is stable, better capacity and bandwidth efficiency

SillyTavern's Actual Compression and Output Budget

The current interaction tier doesn't wait until 120K is full to summarize:

• Summary calls use an independent 8,192 context / 160 output token non-streaming preset, but still point to the same 35B router, avoiding repeated unloading and reloading of the model for summaries.
• Auto-summary is on, delay 2, batch 2; it starts preparing per-segment summaries in the first few turns.
• The threshold for actually excluding old original text and injecting the summary is 45% of the client window, i.e., 122880 × 45% = 55296 tokens.
• After reaching the threshold, the last user message is retained; old original text is taken over by short/long summaries; thus, the session doesn't end at a fixed number of turns, but gradually transitions from verbatim history to compressed memory.
• In a minimal sample on my machine that only counts tokens, not viewing body text, the median for a full turn is about 425 tokens. Pure math suggests ~130 turns to trigger; deducting character cards, world books, system prompts, and considering reply fluctuations, conservatively estimate 80–120 turns of original history understanding. Long replies trigger earlier, short replies later.
• Chinese is not 1 character = 1 token. My machine's tokenizer sample is about 1 token ≈ 1.62 Chinese characters; originally 1200 tokens was already about 1900 characters; the current 1600 token limit is about 2600 characters. max_tokens is a limit, not a target length; setting it to 1600 is to avoid hard truncation of natural transitions, not to encourage filling every turn.

The final decision can be condensed into one sentence:

Preserve 35B roleplay quality; use 128K-level Q8 + compress at ~55K for daily use; switch to 192K Q4 only when you truly need to read over 100K. Don't switch to 8B just for a nominal large window, and don't mistake the startup context limit for a length suitable for real-time chat.

8. After It Runs: Connecting Clients

llama-server exposes an OpenAI-compatible endpoint (http://localhost:18080/v1); any client that accepts a custom base_url can connect — Open WebUI, IDE plugins, CLI agents, just point the base_url there.

The model is ready to use at this point. But "what to do with it, which tasks belong to it, which should be left to Claude Opus, and whether to configure cloud fallback" is another topic — that's where local models are truly worth pondering, covered in a separate post: see local-llm-usage.md.

9. Pitfall Collection

• systemd inline JSON has quotes swallowed → warmup must use -d @file.
• With integrated and discrete GPUs, the model ran on the integrated GPU → Lock to discrete GPU using GGML_VK_VISIBLE_DEVICES=0 (Vulkan) or CUDA_VISIBLE_DEVICES=0 (CUDA).
• Quantized KV without flash-attn → --flash-attn is a prerequisite for cache-type q4_0/q8_0; otherwise, it's ignored or errors.
• Tool calls causing premature EOS → Explicitly specify chat-template=chatml to avoid native template bugs (llama.cpp #19513).
• Thinking resident = always high power → Switch tiers using gpu_busy_percent (AMD) or nvidia-smi (NVIDIA); drop to power-saving tier when idle.
• Wrong VRAM budget calculation → First estimate Model Size + KV (layers × ctx × precision) + Buffer + Desktop; leave 1–2G headroom; don't push to zero.

10. Full Configuration (Linux)

The previous sections explained "what to do" and only pasted key snippets; here, all files in the entire chain are listed completely, created in order, and finally enabled.

WSL2 users follow this section directly; pure Windows users use Task Scheduler to replace systemd, and handle power switching separately for AMD/NVIDIA cards.

Prerequisites: Scripts + Passwordless sudo

llama-profile-sync calls gpu-profile-set, which needs root to write to pp_power_profile_mode (AMD GPU sysfs); ExecStopPost in systemctl does the same. User units cannot pop up sudo password prompts; passwordless access must be configured:

sudo tee /usr/local/bin/gpu-profile-set <<'EOF'
#!/bin/sh
case "$1" in
    bootup_default) idx=0 ;;
    3d_full_screen) idx=1 ;;
    power_saving)   idx=2 ;;
    video)          idx=3 ;;
    vr)             idx=4 ;;
    compute)        idx=5 ;;
    [0-5])          idx=$1 ;;
    *) echo "Unknown profile: $1" >&2; exit 1 ;;
esac
printf "%s\n" "$idx" > /sys/class/drm/card0/device/pp_power_profile_mode
EOF
sudo chmod 755 /usr/local/bin/gpu-profile-set

sudo tee /usr/local/bin/llama-profile-sync <<'EOF'
#!/bin/bash
PROFILE_PATH="/sys/class/drm/card0/device/pp_power_profile_mode"
BUSY_PATH="/sys/class/drm/card0/device/gpu_busy_percent"
STATE="/tmp/llama-profile-last-busy"
BUSY_THRESHOLD=50
HOLD_SECONDS=90

current_idx() { grep '\*' "$PROFILE_PATH" | awk '{print $1}'; }

set_profile() {
    local want=$1 cur
    cur=$(current_idx)
    case "$cur" in 0|5) ;; *) return 0 ;; esac
    case "$want" in
        compute)        [ "$cur" = "5" ] && return 0 ;;
        bootup_default) [ "$cur" = "0" ] && return 0 ;;
    esac
    sudo /usr/local/bin/gpu-profile-set "$want"
}

if ! systemctl --user is-active --quiet llama-server 2>/dev/null; then
    set_profile bootup_default; exit 0
fi
busy=$(cat "$BUSY_PATH" 2>/dev/null || echo 0)
now=$(date +%s)
if [ "${busy:-0}" -ge "$BUSY_THRESHOLD" ]; then
    echo "$now" > "$STATE"
    set_profile compute
else
    last=$(cat "$STATE" 2>/dev/null || echo 0)
    [ $((now - last)) -lt "$HOLD_SECONDS" ] && set_profile compute || set_profile bootup_default
fi
EOF
sudo chmod 755 /usr/local/bin/llama-profile-sync

# Passwordless sudo (replace username with your own):
# echo "USERNAME ALL=(ALL) NOPASSWD: /usr/local/bin/gpu-profile-set" | sudo tee /etc/sudoers.d/llama-profile

pp_power_profile_mode / gpu_busy_percent are AMD GPU sysfs interfaces. NVIDIA power management uses nvidia-smi -pl or nvidia-persistenced; these scripts do not apply.

systemd User Units (Three Files)

# ~/.config/systemd/user/llama-server.service
[Unit]
Description=llama.cpp server (router mode, Vulkan)
After=network.target

[Service]
Environment=GGML_VK_VISIBLE_DEVICES=0     # Vulkan lock to discrete GPU; CUDA users delete this line or change to CUDA_VISIBLE_DEVICES=0
ExecStart=/usr/bin/llama-server \
    --models-preset %h/.config/llama.cpp/models.ini \
    --models-max 1 \
    --host 0.0.0.0 \
    --port 18080 \
    --metrics
# Warmup: Preload model into VRAM; -d @file avoids systemd swallowing quotes
ExecStartPost=-/usr/bin/bash -c 'for i in $(seq 1 60); do curl -sf --max-time 90 http://127.0.0.1:18080/v1/chat/completions -H "Content-Type: application/json" -d @%h/.config/llama.cpp/warmup.json >/dev/null && exit 0; sleep 2; done'
# When service stops, GPU returns to power-saving tier
ExecStopPost=/usr/bin/sudo /usr/local/bin/gpu-profile-set bootup_default   # AMD power management; NVIDIA users delete this line
Restart=on-failure
RestartSec=5

[Install]
WantedBy=default.target

# ~/.config/systemd/user/llama-gpu-sync.service
[Unit]
Description=Sync GPU profile with llama-server model state
After=llama-server.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/llama-profile-sync

# ~/.config/systemd/user/llama-gpu-sync.timer
[Unit]
Description=Poll llama-server model state for GPU profile switching

[Timer]
OnBootSec=15s
OnUnitActiveSec=15s

[Install]
WantedBy=timers.target

Warmup

echo '{"model":"qwen3.6-mtp-instruct","messages":[{"role":"user","content":"hi"}],"max_tokens":1}' \
  > ~/.config/llama.cpp/warmup.json

Enable

# linger — Key for persistent models: service survives logout
sudo loginctl enable-linger "$USER"

# reload + enable
systemctl --user daemon-reload
systemctl --user enable --now llama-server.service
systemctl --user enable --now llama-gpu-sync.timer

# Verify
systemctl --user status llama-server
systemctl --user status llama-gpu-sync.timer

--user is not --system: GPU context (DRM render node) is in the user session; running a system unit might fail to open the device; permissions are also much smaller.

Related Docs

• local-llm-usage.md — After running: Boundaries (Local vs Opus), What it can do, llm-jobs automation, Practical lessons learned
• software.md — Software Stack List
• timers-and-crons.md — Timers and Crons like llama-gpu-sync.timer
• monitoring.md — Monitoring Architecture (GPU metrics via node_exporter → Prometheus → Grafana)