--- title: 'What Local Large Models Can Help With: Boundaries, Scenarios, and Automation' url: https://doc.liz6.com/en/homelab/local-llm-usage locale: en area: homelab tags: - homelab date: 2026-07-10 modified: 2026-07-16 description: 'What to do with a local large model once it is running. The core is to define boundaries: a 35B local small model cannot handle complex production logic—that is the job of Claude Code + Opus; locally, only handle daily, non-complex, fault-tolerant miscellaneous tasks. But in this narrow niche, it has two characteristics that cloud services cannot replace: offline autonomy and privacy staying on-device. This article clarifies where the boundaries lie, what it can help with (using the real-world running openclaw daily assistant + llm-jobs daily report/harvest as examples), and how to turn an idea into a persistent task, while directly answering whether "configuring cloud fallback" is contradictory. See local-llm.md for deployment.' --- # What Local Large Models Can Help With: Boundaries, Scenarios, and Automation > Getting the model [running and persistent](local-llm.md) is just the starting point. The real question is: what exactly should you do with a 35B small model running on a consumer-grade GPU? The answer in this article is restrained—it cannot handle complex production logic (that is the job of Claude Code + Opus); its niche is narrow: **daily, non-complex, fault-tolerant miscellaneous tasks**. But in this narrow niche, it has two characteristics that cloud services cannot replace: **offline autonomy and privacy staying on-device**. Understanding this boundary tells you which tasks are worth the effort. ## 1. Define Boundaries First: What Local Models Do Not Do After installing a local model, the first reaction is often "use it to replace ChatGPT / Copilot"—then disappointment sets in. Because in pure capability, a 35B model that can run on a consumer GPU **cannot beat frontier cloud models**, especially in "production" tasks like multi-step reasoning, long-chain agents, and precise coding. One-sentence positioning: > **Complex logic, production delivery, quality-required tasks → Claude Code + Opus; local models only handle daily, non-complex, fault-tolerant miscellaneous tasks.** Why local shouldn't handle production: Small parameter models **will fail** where precision and multi-step reasoning are required, and the cost of failure in production is high (wrong code, deviated agents). The local savings in money and privacy are not enough to offset this cost. **Leaving complex tasks to Opus is the first and most important rule for defining the boundary for local models.** ## 2. Local's Foundation: Offline + Privacy Since it can't beat the cloud in capability, why use local? Because there are two things **even the strongest cloud cannot do**: - **Offline autonomy**: No internet, cloud inaccessible, cloud blocked, cloud price hikes or rate limits—local keeps running. It does not depend on any external service being online. - **Privacy stays on-device**: Private data fed to it is not uploaded, does not enter any third-party logs or training sets. These two are the real reasons "some tasks should be done locally." ### On the Contradiction of Cloud Fallback A popular practice is "local first, cloud fallback": if local stops (e.g., gamemode stops the server while gaming), it automatically falls back to the cloud, ensuring uninterrupted usage. Sounds great, but be clear-headed— > **Once cloud fallback is configured, local is no longer offline and no longer purely private.** As long as one fallback occurs, that one instance's data goes to the cloud. Fallback and local's two core characteristics are **naturally contradictory**. So don't blindly attach fallback to all local tasks. The honest approach is to **split by scenario into two categories**: | Scenario Nature | Why Local Was Chosen | Configure Fallback? | Example | |---|---|---|---| | **Availability Priority** | Local is just the "default for saving money"; offline/privacy are not hard requirements | ✅ Yes | openclaw daily assistant: look up info, schedule; if local stops, fallback to cloud to ensure assistant stays online | | **Purity Priority** | Local is chosen "because of offline/privacy" | ❌ Never | daily-harvest scans sessions containing private code and keys; fallback to cloud = sending this data out | This line is exactly what is drawn in this machine's configuration: openclaw's `primary = local` / `fallbacks = deepseek` (availability priority), while llm-jobs' harvest directly `exit 3` and wait for retry when local is unavailable, **never going to the cloud** (purity priority). The judgment criterion is simple: **if "sending this task to the cloud once" violates the original intention of using local, then it should not have fallback.** ## 3. So What Can It Actually Help With? In the narrow niche of "daily miscellaneous tasks + offline/privacy," the three attributes of local models unlock different types of scenarios. Below, ✅ indicates what is truly running on this machine, 💡 indicates extendable ideas. ### Zero Marginal Cost → Dare to Normalize Daily Miscellaneous Tasks With pay-per-use cloud models, you subconsciously "calculate before using"; with local zero cost, you dare to run low-value but high-frequency miscellaneous tasks daily. - ✅ **openclaw daily Q&A**: Look up info, schedule, casual questions; no pain in asking many times. These "single instance low value, but frequent" tasks are only affordable with zero cost. - ✅ **Daily report synthesis**: `daily-report` at 09:00 every day deduplicates and merges yesterday's fragmented entries into human language, combined with system inspection, pending items, and Friday's "Weekly Perspective" to push to WeChat. Must run daily, multiple calls per day; paying per call on the cloud feels painful. - ✅ **Weekly Perspective**: Friday's daily report adds a section—the model reads the past 7 days' completed entries, summarizes energy allocation, points out stalled items, and gives one suggestion for next week. Cross-document synthesis is annoying to do manually; zero cost for the model to do weekly. - 💡 **Extension**: Auto-generate one-sentence summaries for each saved article / RSS; monthly rolling summary. ### Privacy Stays on-Device → Dare to Feed Private Data There is a type of data "you really want to process with AI, but dare not give to the cloud." Local is the only place you dare to feed. - ✅ **daily-harvest**: Scans all Claude Code project sessions for the day (including private projects, code, possible keys) + openclaw memory, extracts "what was actually done" to fill back into the daily report. This data **must not go to the cloud**; only local dares to feed it like this. - 💡 **Extension**: Private diary / chat logs / financial receipts for local RAG retrieval; photo OCR archiving. ### Persistent Embedding → Dare to Weld It Into Automation The model runs persistently on `localhost:18080`; any script or timer can call it at any time, no need to wake any cloud service, no network round-trip concerns. - ✅ **llm-jobs**: systemd timer calls the local model on time, collect → extract → deliver, full local closed loop (detailed in next section). - ✅ **openclaw default model**: Daily agent's `primary` points to local. - ✅ **System inspection noise reduction**: `health-snapshot` only reads raw signals (failed units, backup results, ZFS water level, 24h tens of thousands of err-level logs aggregated by source, SSH auth failures); the model is responsible for noise reduction and priority sorting, producing the "🩺 System" section of the daily report. Judgments like "radicale logs INFO as err is long-term noise" or "SSH has 2FA, brute force attempts don't need reporting" cannot be written with rules; the model handles it in one sentence; when all green, only outputs "✅ All normal." - 💡 **Extension**: Periodic distillation and maintenance of memory. ## 4. Turning Ideas Into Persistent Tasks: llm-jobs Framework All those automations above look the same when implemented: **collect data → call local model once → write file / deliver**. Instead of writing each task separately (each hand-rolling curl to call the model, each sending WeChat), why not extract a unified framework? `~/.local/lib/llm-jobs/` is this framework—a unified landing point for tasks that will continuously grow in the category of "periodically calling local models" (daily reports, harvest, future weekly/monthly reports, memory maintenance); adding new tasks only requires writing one collection script. ### Why This Layer of Abstraction The daily report was originally an isolated script, with the only data source `~/doc/daily/`, and this directory relies on two agents, Claude Code and openclaw (Miko), to "call `doc-daily-log` when they remember" in a best-effort manner—if someone forgets to record, the completed work disappears entirely from the daily report (e.g., the 2026-07-03 daily report was originally empty). The problem splits into two parts: - **Comprehensiveness**: Work done didn't reliably enter daily. → Use `daily-harvest` to retrieve from **real work traces** (session records). - **Quality**: Pure string parsing passes duplicates and inconsistent granularity through as is. → Use local model synthesis. And these two are essentially "periodically calling local models," so extract a unified framework. ### Architecture Decision: systemd vs openclaw cron Both are "periodic," but the two paths differ greatly: | | systemd timer + script | openclaw cron | |---|---|---| | Task Model | Deterministic pipeline (collect → call model → deliver) | Wake an **agent session** (with tool/memory/channel context) | | Observability | journald → [Loki](monitoring.md), can Grafana alert | openclaw internal logs, outside monitoring stack | | Dependencies | Only depends on llama-server (also systemd) | Depends on openclaw gateway being online | | Debugging | `--dry-run`, `systemctl start`, pure scripts can be versioned | Must reproduce in agent context, non-deterministic | > **Rule: Pipeline tasks go to systemd, agent tasks use openclaw cron.** Daily reports and harvest are "read file → call model once → write file/send message" pipelines, no need for tool loops, go to systemd. openclaw cron is only reserved for scheduled tasks where "Miko really needs to think and act" (currently none). ### Structure: Three-Layer Decoupling ``` ~/.local/lib/llm-jobs/ bin/llm-call Unified model entry: POST localhost:18080; default closes thinking per request (--think opens), empty content treated as failure and retries with higher token, local down exit 3 bin/notify-weixin Unified delivery: send to WeChat bot; --spool persists messages with expired tokens (ret=-2) to disk for queuing bin/health-snapshot Read-only collection of inspection signals (failed units/backup results/ZFS/journal err aggregation/SSH auth), feeds the "system" section of daily-report to model for noise reduction jobs/daily-harvest Early morning: scan yesterday's sessions → map-reduce extract completed items + pending → supplement into daily jobs/daily-report Morning: read daily + real-time inspection → synthesize four sections → deliver jobs/notify-flush Hourly: resend WeChat messages queued in spool README.md Conventions + task registration form ``` - **`bin/llm-call` is the only model entry** — "periodically calling local model" has only one tested implementation; model name/endpoint/fallback strategy are centrally managed. Changing model only requires changing this one place. - **`bin/notify-weixin` delivery is independent** — Adding bark/email/multi-channel later only touches this layer. - **Template unit `llm-job@.service`** — `ExecStart=%h/.local/lib/llm-jobs/jobs/%i`, adding tasks doesn't require creating new service files. Convention: Task scripts **only call shared primitives in `bin/`**, do not each hand-roll curl to call the model or send WeChat. ### Complete Closed Loop harvest's scheduled instance passes `--date yesterday` via systemd drop-in (manual run still defaults to today); at 04:00 yesterday is completely over, the supplemented data is exactly what needs to be reported at 09:00. It doesn't matter if the machine sleeps at night and misses the trigger; `Persistent=true` will make up the run after waking. **daily-harvest (04:00)**: Find all Claude Code sessions from yesterday `~/.claude/projects/*/*.jsonl`, compress each into `USER:/AI:` plain text (remove thinking and tool noise, truncate to limit) and feed to the model **map** to extract two types of things— "meaningful work actually completed" and "explicitly said to do but not finished," plus openclaw's own distilled memory. Completed item candidates + **manual** entries already in today's daily **reduce**, only pick truly completed and manually unrecorded ones, output JSON, write back line by line with `hv_*` tag, physically delete old `hv_` blocks first then recalculate each time (**idempotent**, can run repeatedly without stacking). Pending candidates deduplicate against completed items and write to `daily/.pending` (entire chain is best-effort, failure only loses this section). Entire process uses local model + local write. **daily-report (09:00)**: Synthesize four sections to push to WeChat— ① "🩺 System": `health-snapshot` collects real-time inspection signals, model reduces noise into brief report (if model unavailable, degrade to only listing failed units, inspection never blocks sending); ② "📝 Yesterday's Completion": read all entries from yesterday's daily, model deduplicates and merges into human language, **if model unavailable fallback to "original grouping"** (no cloud, just no LLM synthesis), daily report must be sent; ③ "🔜 Pending": directly read `.pending` file, if none, entire section does not appear; ④ "📊 Weekly Perspective": Friday only, model summarizes past 7 days' entries. Add date/weather (`wttr.in`), all sections undergo unified **deterministic post-processing** (empty lines between entries, full-width indentation for bullet points—do not expect model layout stability), `notify-weixin --spool` delivers. **Fallback (Purity Priority Manifestation)**: After harvest completes, write `~/.local/state/llm-jobs/harvest-.ok` marker; if local model unavailable (stopped by gamemode), write `.deferred` and `exit 3`—**do not write fake empty results, do not go to cloud**, systemd marks failure into Loki. Before 09:00 report sends, if it finds no `.ok` marker from yesterday, first make up run `daily-harvest --date ` (model recovered at this time) then read daily. Covers all scenarios "machine sleeping at early morning / model stopped by gamemode." Note the premise of this fallback chain is that **`.ok` marker must be honest**—the accident on 2026-07-10 (see "Blood and Tears" in next section) was exactly empty results mistakenly written as `.ok`, short-circuiting the entire fallback chain. ### Add a New Task 1. Write `~/.local/lib/llm-jobs/jobs/` (executable, supports `--dry-run`, only calls shared primitives in `bin/`) 2. Create `~/.config/systemd/user/llm-job@.timer` (copy daily-report, change `OnCalendar`) 3. `systemctl --user daemon-reload && systemctl --user enable --now llm-job@.timer` ### Operations ```bash # Manual run (no send/write) ~/.local/lib/llm-jobs/jobs/daily-report --dry-run ~/.local/lib/llm-jobs/jobs/daily-harvest --dry-run [--date YYYY-MM-DD] # Manual trigger real task systemctl --user start llm-job@daily-report.service # View logs (journald → also into Loki) journalctl --user -u llm-job@daily-report.service -n 50 systemctl --user list-timers | grep llm-job ``` If WeChat `notify-weixin` returns `ret=-2` (context token expired), `--spool` caller's messages will be persisted to disk for queuing, `notify-flush` automatically resends hourly; sending a message to the bot on WeChat refreshes the token to accelerate delivery. ## 5. Several Practical Tips for Using Local Models Well All stepped out from these real tasks above, transferable to any "call local model" scenario: - **Turn off thinking for daily tasks, and must turn off on request side**: Tasks like organizing, summarizing, categorizing do not need chain of thought; turning off saves a lot of latency. Key is using request-level `chat_template_kwargs: {"enable_thinking": false}` (can override server preset), **do not rely on server preset**—preset is shared, might be changed for other scenarios one day (see next blood and tear). `bin/llm-call` now defaults to closing thinking on request side, `--think` explicitly opens when needed. - **Make good use of long context, don't chop it yourself**: Local can open very long windows (this machine 64K–192K), feed entire session, entire document. harvest once raised single session truncation limit from 10k to 40k—truncation cuts off completion summary in the **middle** of the session, causing omissions. - **Structured output must verify + retry**: Letting model output JSON requires fault-tolerant parsing, failure retry (`bin/llm-call` built-in retry 2 times). Small models occasionally run out of format under temp. - **Judgment prompts use positive extraction, do not give easy exit of "output none if nothing"** (Blood and Tear): harvest's map early used negative framework ("output none if not discussion"), under temp would **randomly** judge entire productive sessions as "none," losing a whole day's work (same input 1st time outputs 6 items, 2nd time "none"). Changed to "programming sessions usually have output, read full text then extract, only pure consultation outputs none" then 5/5 stable. **LLM judgment prompts do not give convenient exit of "output nothing," it will unstable lose data in sampling.** - **Do not over-merge**: "Merge similar items" too aggressive in reduce stage, makes same input sometimes 19 items, sometimes collapse into 4 vague big titles. **Maintain granularity**, leave merging to downstream (daily report generation) step. - **HTTP 200 can also be fake success—empty content must be treated as failure** (Blood and Tear, 2026-07-10): To fix codex lag, changed server preset `qwen3.6-mtp-thinking` from `enable_thinking:false` to `reasoning = on`, thinking chain separated into `reasoning_content`, batch task small `max_tokens` (512) all burned on thinking—returned 200 but `content=""`, `finish_reason=length`. harvest treated empty output as "this session has no work," 17 sessions with real work all judged "none," and wrote `.ok` success marker, morning fallback short-circuited, daily report sent "no records yesterday." And **giving thinking budget doesn't win**: tested 512→2048 it thinks longer still burns out. Fix method two-pronged: ①`llm-call` treats empty content as failure (`length` doubles max_tokens immediately retries, exhausted `exit 3` goes deferred); ② batch tasks default request side close thinking (previous point). Deeper lesson: **shared inference instance server preset is implicit contract for all callers**—changing preset for one scenario, must review all other callers. - **Do not expect model layout stability, use deterministic post-processing for display layer**: Writing layout requirements like "empty line between each item" in prompt, small model execution unstable. Daily report method is model only manages content, script uniformly rearranges (empty lines between entries, full-width space indentation for bullet points), no matter how model output format drifts, reader sees constant layout. - **Give enough tokens for structured output, and failure must never disguise as success** (Blood and Tear): Busy day candidates can reach hundreds (54 sessions → 163 candidates), reduce output JSON very long, `max_tokens=1500` directly **truncates** → JSON illegal → parsing failure. Early on treated parsing failure **silently as "no new additions," also wrote `.ok` success marker**, so morning fallback not triggered, whole day's work silently evaporated (2026-07-06). Three iron rules: ① Give enough output tokens and retry with higher tier (6000→9000); ② Judge empty must distinguish `None` (failure) from `[]` (truly no new additions); ③ **Failure must have degradation not discard**—this machine harvest degrades to "group by project directly write raw candidates" when reduce repeatedly fails (downstream daily report merges later), only model truly unavailable `.deferred`+`exit 3`. **Any path where "failure treated as empty result" will quietly swallow data in automation.** - **Agent / tool calls must open `--jinja`**: Otherwise model "can spit toolcall tags but not execute"—llama-server's native tool parsing (parse output into structured `tool_calls`) only takes effect when `--jinja` is opened. Pure conversation unaffected, but when agent uses must open. ## Related Documents - [local-llm.md](local-llm.md) — Deployment: VRAM budget, quantization selection, MTP, persistent tidal scheduling, complete systemd config - [timers-and-crons.md](timers-and-crons.md) — All systemd timer overview (including llm-job@ two) - [monitoring.md](monitoring.md) — journald → Loki log aggregation (llm-jobs failures can alert here)