--- title: 'Full-Stack Monitoring: Prometheus + Grafana + Loki + Tempo + Bark' url: https://doc.liz6.com/en/homelab/monitoring locale: en area: homelab tags: - homelab date: 2026-07-01 modified: 2026-07-16 description: 'Metrics (Prometheus) + Logs (Loki) + Traces (Tempo) all feed into Grafana, with alerts pushed to iPhone via Bark. Metrics use two sets: pull (local network node_exporter) and push (remote otel-collector fleet agent, via rathole tunnel + mTLS);…' --- # Full-Stack Monitoring: Prometheus + Grafana + Loki + Tempo + Bark > Metrics (Prometheus) + Logs (Loki) + Traces (Tempo) all feed into Grafana, with alerts pushed to iPhone via Bark. Metrics use two sets: pull (local network node_exporter) and push (remote otel-collector fleet agent, via rathole tunnel + mTLS); logs go journald → otelcol-logs → Loki; traces go otelcol-home tail sampling → Tempo. Using local host `li-home-0` as the instance, this covers full-stack architecture, alert rules, collection comparison, labeling scheme, severity determination, and pitfall records. ## 1. Overview Monitoring Architecture: Metrics + Logs + Traces → Grafana, Alerts → Bark → iPhone Remote Fleet otel-collector host_metrics 30s file_storage queue mTLS client cert rathole TCP Tunnel ↓ Caddy mTLS Termination Path Restriction + Revocation otelcol-home OTLP Receiver metrics pipeline traces pipeline tail sampling Prometheus Tempo :4317 node_exporter (pull) journald otelcol-logs Loki :3100 Grafana Metrics · Logs · Traces Bark APNs → iPhone metrics-fleet: Remote otel-collector (fleet agent) → OTLP+mTLS → rathole tunnel → Caddy gateway → otelcol-home (metrics+trace pipeline) Local: node_exporter → Prometheus (pull) · journald → otelcol-logs → Loki · otelcol-home traces → Tempo (gRPC, 720h retention) Grafana unified query for Metrics (Prometheus) + Logs (Loki) + Traces (Tempo) · Alerts via Bark webhook → APNs → iPhone (local bark-server, no tunnel/public internet dependency) All components native systemd · bind 127.0.0.1 · Docker only runs applications ## 2. Metrics Collection: Two Models | | Traditional Pull (node_exporter) | Remote Push (metrics-fleet) | |---|---|---| | Use Case | Local network nodes, local host | Public internet remote servers | | Collector | node_exporter / LHM exporter | otel-collector-contrib (host_metrics) | | Metric Naming | `node_*`, `windows_*`, `lhm_*` | `system_cpu_*`, `system_memory_*` | | Network | Prometheus actively pulls, requires reachable ports | Remote actively pushes, zero inbound ports | | Transport Security | None (unencrypted on LAN) | mTLS + Path Restriction (only OTLP write allowed) | | Onboarding Method | Manually edit `prometheus.yml` `static_configs` | `add-server.sh ` | | Dashboard | One independent JSON per host | Single template, `$host` dropdown to switch | | Transit Dependency | None | rathole tunnel → Transit VPS → Home | ### Pull Model: scrape_configs Prometheus pulls these targets every 15s: local host node_exporter (:9100), llama-server (:18080), Home Assistant (:8123), Windows machines (9182/9183), N100 side router (:9100), Xiaomi router (:9184), self-monitoring (:9090). ### Push Model: OTLP Receiver Remote servers collect host_metrics via otel-collector, authenticate via mTLS + rathole encrypted tunnel, and push to the Prometheus remote write endpoint at home: Push Model: Remote metrics pushed home via mTLS + rathole tunnel Remote otel-collector host_metrics Collection OTLP+mTLS Cert Public Entry otlp.liz6.com:9443 DNS → Transit VPS rathole Tunnel TCP Passthrough · Ciphertext Only Home Caddy mTLS Gateway Verify Cert + Revocation Check Only Allow /api/v1/otlp/* otelcol-home :4318 batch Processing Prometheus remote write :9090 mTLS terminates only at Caddy; rathole passes ciphertext end-to-end, so we never see plaintext traffic. Prometheus config `out_of_order_time_window=30m` to accommodate out-of-order arrival across the public internet. > Prometheus config `out_of_order_time_window: 30m` accommodates out-of-order arrival across the public internet. ### Grafana Dashboards Provisioning directory: 7 host panels (CPU / GPU / Memory / Disk / Cooling, per-node JSON) + 1 server status template (`$host` dropdown auto-discovers all host_names pushed via OTLP) + Network monitoring panel. Panel expression example: ```promql # CPU: 100*(1-avg by(host_name)(system_cpu_utilization_ratio{host_name="$host",state="idle"})) # Memory: 100*system_memory_utilization_ratio{host_name="$host",state="used"} # Liveness check (push model has no up metric, use last reported time instead): time()-max(timestamp(system_memory_usage_bytes{host_name="$host"})) ``` ## 3. Security: CA and mTLS Security for the remote push chain is ensured by client certificates issued by a private CA: ``` ~/infra/metrics-fleet/ca/ ├── make-ca.sh # One-time CA creation (EC P-256, 10 years) ├── issue-server-cert.sh # Issue server cert → /etc/caddy/metrics-fleet/ ├── issue-client-cert.sh # Issue client cert → out/clients// ├── revoke-client-cert.sh # Revoke + update Caddy revoked.caddy + reload └── out/clients// # Issued client certs (one per remote host) ``` Caddy acts as an mTLS gateway on 127.0.0.1:9443: verifies client cert (require_and_verify, trust CA), checks revocation list → 403, only allows `/api/v1/otlp/*` with strip_prefix → otelcol-home :4318. > The public IP of each remote server must be added to the DIRECT rules in mihomo on both the local host and the N100, ensuring rathole connections and SSH do not go through the proxy (transport connections must use native routing—see [rathole-tunnel.md](rathole-tunnel.md) section 4.3). ## 4. Alerts: Grafana + Bark Push to iPhone Prometheus handles collection, Grafana handles evaluation and notification. Alert chain: Alert Chain: Grafana Rule Evaluation → bark-server (local) → APNs → iPhone Grafana Rule Evaluation Evaluated every 30s webhook bark-server 127.0.0.1:8085 Local direct connection, no tunnel APNs iPhone Lock Screen Push Why use local bark-server: Alert notifications cannot depend on tunnels/public internet—if the chain is down, alerts can't be sent. Local direct connection has zero external dependencies. **Why use local bark-server instead of public webhook**: If alert notifications go via rathole tunnel or public internet—when the tunnel is down, alerts can't be sent. Local bark-server connects directly to iPhone, zero external dependencies, not bound to infrastructure. ### Contacts and Notification Strategy ```yaml # /etc/grafana/provisioning/alerting/contact-points.yaml contactPoints: - name: bark-home receivers: - type: webhook settings: url: "http://127.0.0.1:8085/" # Local bark-server, bypass tunnel maxAlerts: 2 # APNs 4KB limit, >2 alerts causes PayloadTooLarge title: "[{{ .Status }}] {{ .CommonLabels.grafana_folder }}{{ if .CommonLabels.alertname }} · {{ .CommonLabels.alertname }}{{ end }}" message: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}" policies: - receiver: bark-home group_by: ["grafana_folder", "alertname"] # Merge similar, send different separately group_wait: 30s # Initial delay to wait for similar alerts group_interval: 5m # Interval for subsequent alerts in same group repeat_interval: 4h # Re-alert every 4h if unresolved ``` > `maxAlerts: 2` is key—bark passes the entire webhook JSON to APNs, each alert is ~1.1–1.4KB, exceeding 2 alerts surpasses APNs 4KB limit, bark returns 500. ### Alert Rules (13 total) | Rule | Severity | Condition | Silence | |------|----------|-----------|---------| | Side router down | critical | `up{job="li-home-router"}==0` | 3m | | Router unreachable | critical | `ping_loss{target="gateway"}>30%` | 2m | | High external packet loss | warning | `ping_loss{target="223.5.5.5"}>20%` | 3m | | High TCP retransmission | warning | `Tcp_RetransSegs / Tcp_OutSegs > 10%` | 5m | | DNS all slow | warning | All three domain resolutions >100ms | 5m | | mihomo TUN failure | critical | `nikki` network interface metrics missing | 2m | | CN node disk low | warning | Root partition remaining <5GB | 5m | | Water pump stopped | critical | fan7 RPM <1500 | 5m | | VRM overheating | warning | temp5 >90°C | 2m | | GPU hotspot overheating | warning | 7900XTX junction temp >100°C | 2m | | N100 CPU overload | warning | CPU >90% | 5m | | Local exporter down | critical | `up{job="local-info"}==0` | 2m | | rathole service unreachable | critical | `rathole_service_up < 1` (tagged by service) | 3m | > Rules are managed via Grafana provisioning (`/etc/grafana/provisioning/alerting/home-network.yaml`), versioned in git, no need to enter Web UI to change rules. Each alert describes the phenomenon in `annotations.summary`, and provides troubleshooting steps in `annotations.runbook`—alert notifications only push summary, runbook is expanded in Grafana. ### Metrics Pitfalls - **N100 side router onboarding**: Prometheus scraping N100 requires going through local host proxy, add DIRECT rule to bypass. ## 5. Log Monitoring (Loki + OpenTelemetry) Fills the gap of "only metrics, no logs". All local system logs (kernel + systemd services + Docker containers) go uniformly into journald → otelcol-logs → Loki → Grafana Explore for retrieval. ### Design Trade-offs - **Choose OpenTelemetry, not Promtail/Vector**: Local host already runs otelcol-home, introducing Vector would duplicate collector tech stack. OTel also makes remote host log onboarding nearly free (reuse same mTLS/rathole/Caddy tunnel). - **Loki uses native OTLP endpoint** `/otlp/v1/logs` (old `lokiexporter` is deprecated). - **Docker changed to journald log driver**: Container logs with `CONTAINER_NAME` go into journald, collected by same `journaldreceiver` → has real container names, single data source. - **native systemd**, bind `127.0.0.1`, 90-day retention, no log dashboard (use Explore directly). ### Data Flow Log Data Flow: journald → otelcol-logs → Loki → Grafana Kernel dmesg systemd Services Docker Containers journald Persistent /var/log/journal SystemMaxUse=8G otelcol-logs journaldreceiver + OTTL resource_detection+batch User=otelcol-logs Loki 90d Grafana Explore Key: journaldreceiver puts fields into body (Map), OTTL first extracts attributes from body then sets (body, body["MESSAGE"]) to restore. Docker side /etc/docker/daemon.json: log-driver=journald + tag={{.Name}} + live-restore=true Loki side: common.instance_addr=127.0.0.1 (otherwise ring broadcast IP self-connection fails), discover_log_levels=true ### Components | Component | Location | Notes | |------|------|------| | Loki 3.7 | `/usr/local/bin/loki`, conf `/etc/loki` | `:3100`, OTLP `/otlp/v1/logs`, user `loki` | | otelcol-logs | `/usr/local/bin/otelcol-contrib` (0.154, reuse fleet binary) | unit `/etc/systemd/system/otelcol-logs.service`, no root needed (group `systemd-journal`) | | Grafana Data Source | `/etc/grafana/provisioning/datasources/loki.yaml` | uid `loki-local` | ### Labeling Scheme Low cardinality goes to **index labels**, high cardinality goes to **structured metadata** (queryable but not indexed): | Index Label | Meaning | |----------|------| | `host_name` | resource `host.name` (whole machine name, aligns with metrics) | | `service_name` | = `host.name`, Loki native grouping field | | `unit` | systemd service name; Docker container = container name (overwritten by OTTL from `CONTAINER_NAME`) | | `container` | Docker container name (container logs only) | | `level` | See "Severity Determination" below | | `transport` | kernel / stdout / syslog / journal / audit | ### Severity Determination (Hybrid Strategy) journald's `PRIORITY` is only the true severity for native sources (systemd-journal/syslog/kernel); `PRIORITY` for container logs and raw stdout/stderr only represents the stream (stdout=6/stderr=3), meaningless. 1. **Baseline**: Containers and non-stdout sources get severity based on PRIORITY (0–2 critical / 3 error / 4 warn / 5–6 info / 7 debug); container logs default to `info`. 2. **Body Override**: OTTL `IsMatch` identifies `[error]`/`level=info`/`"level":"warn"` and uppercase standalone words `WARNING`/`ERROR` from message body. 3. Loki side `discover_log_levels: true`—Loki recognizes `level` attribute and fills `detected_level` accordingly (used for Grafana severity visualization). ### Query Examples (Grafana Explore) ```logql {unit="sshd.service"} {level="error"} {container="adguardhome"} {transport="kernel"} {service_name="li-home-0"} |= "timeout" ``` ## 6. Log Pitfalls 1. **Loki single binary ring self-connection failure**: Default broadcasts network card IP, but gRPC only listens on 127.0.0.1 → connection refused. Must set `common.instance_addr: 127.0.0.1`. 2. **journaldreceiver puts entire log in body (Map)**: Fields are `body["PRIORITY"]` not attributes. OTTL first extracts attributes from body, finally `set(body, body["MESSAGE"])`. 3. **service_name semantics**: What we want is whole machine identifier, resource-level. Loki `discover_service_name` only reads resource attributes → set `service.name = host.name` in OTTL resource context. 4. **Docker container unit**: Under journald driver, `_SYSTEMD_UNIT=docker.service` is useless → OTTL overwrites with `CONTAINER_NAME`. 5. **Don't trust PRIORITY for severity**: adguard/HA/owntracks write info+warning all to stderr → PRIORITY all 3, if trusted they all become error. Don't set `discover_log_levels` to false—otherwise `detected_level` disappears, Grafana histogram collapses to single line. 6. **Colored logs = byte arrays**: MESSAGEs containing ANSI are stored as int-slice by journald, OTTL cannot decode. **Root fix = disable color at source**: containers add `NO_COLOR=1`+`PY_COLORS=0`; Rust services add `Environment=NO_COLOR=1`+`RUST_LOG_STYLE=never`. 7. **node_exporter sgcc spam**: `sgcc_ts.prom` has timestamps (orphan files owned by root), textfile collector doesn't support. Remove from `textfile_collector/` to fix. 8. **runlike rebuild container loses data**: `uvx runlike` outputs errors for anonymous/named volumes, need to explicitly pin volumes via `docker inspect .Mounts`. ## 7. Traces: Tempo + tail sampling Traces and metrics share the same OTLP entry point (otelcol-home :4318), internally split by pipeline: Trace Chain: Applications → otelcol-home → tail sampling → Tempo → Grafana Applications/Services OTLP trace exporter otelcol-home :4318 (same metrics entry) pipeline: memory_limiter → tail_sampling → batch Tempo :4317 · gRPC Grafana Explore tempo-local Traces and metrics share the same OTLP entry point (otelcol-home :4318), internally split by pipeline, non-interfering. tail_sampling ensures span tree integrity, avoiding broken chains from inconsistent parent/child span sampling. ### Sampling Strategy Uses **tail sampling** (not head sampling), ensuring span tree integrity is not truncated: | Strategy | Condition | Sampling Rate | |------|------|--------| | errors-only | `status_code == ERROR` | 100% | | latency-sampling | Latency >500ms | 100% | | probabilistic | Other normal requests | 10% | > tail sampling has 10s decision delay compared to head sampling (waits for span tree to be complete before deciding), but avoids "parent span sampled, child span not sampled" broken chains—crucial for troubleshooting distributed call chains. ### Tempo Backend ```yaml # /etc/tempo/config.yaml distributor: receivers: otlp: protocols: grpc: endpoint: 127.0.0.1:4317 ingester: max_block_duration: 5m lifecycler: ring: kvstore: { store: inmemory } replication_factor: 1 ``` Single instance inmemory ring (no cluster requirement), 720h (30 days) retention. Same pattern as Loki: native systemd, bind `127.0.0.1`, Grafana Explore directly retrieves by Trace ID. ### Onboarding Method Application side sets `OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318` (local) or `https://otlp.liz6.com:9443` (remote, via fleet tunnel). SDK automatically generates traces. Remote applications reuse existing fleet collector's mTLS certificates—metrics and traces share the same pipeline and security system. ## 8. Future Expansion: Remote Fleet Logs Reuse existing metrics-fleet transport (no need to modify Caddy, `/api/v1/otlp/*` already covers `v1/logs`): 1. Remote otelcol-fleet agent adds journald receiver + logs pipeline → `otlphttp` to `otlp.liz6.com:9443` 2. Home otelcol-home adds logs pipeline → `otlphttp` to Loki `/otlp/v1/logs` 3. `add-server.sh`/CA/naming all reused, `host_name`/`service_name` are each node's name ## Related Documents - [network-architecture.md](network-architecture.md) — Network topology and daemons - [rathole-tunnel.md](rathole-tunnel.md) — Tunnel architecture (transport connections bypass mihomo) - [server-lifecycle.md](server-lifecycle.md) — Remote server onboarding script (add-server)