Full-Stack Monitoring: Prometheus + Grafana + Loki + Tempo + Bark

Metrics (Prometheus) + Logs (Loki) + Traces (Tempo) converge into Grafana; alerts are pushed to iPhone via Bark. Metrics use two modes: pull (LAN node_exporter) and push (remote otel-collector fleet agent via rathole tunnel + mTLS); logs go journald → otelcol-logs → Loki; traces go otelcol-home tail sampling → Tempo. Using local instance li-home-0 as the example, this covers full-stack architecture, alert rules, collection comparison, labeling scheme, severity determination, and pitfall records.

1. Overview

2. Metrics Collection: Two Models

	Traditional Pull (node_exporter)	Remote Push (metrics-fleet)
Use Case	LAN nodes, local machine	Public internet remote servers
Collector	node_exporter / LHM exporter	otel-collector-contrib (host_metrics)
Metric Naming	node_*, windows_*, lhm_*	system_cpu_*, system_memory_*
Network	Prometheus actively pulls, requires reachable ports	Remote actively pushes, zero inbound ports
Transport Security	None (plaintext on LAN)	mTLS + Path Restriction (only allow OTLP write)
Onboarding Method	Manually edit prometheus.yml static_configs	add-server.sh <ssh-alias>
Dashboard	One independent JSON per host	Single template, $host dropdown to switch
Transit Dependency	None	rathole tunnel → Transit VPS → Home

Pull Model: scrape_configs

Prometheus pulls these targets every 15s: local node_exporter (:9100), llama-server (:18080), Home Assistant (:8123), Windows machines (9182/9183), N100 side router (:9100), Xiaomi router (:9184), self-monitoring (:9090).

Push Model: OTLP Receiver

Remote servers collect host_metrics via otel-collector, authenticate via mTLS + rathole encrypted tunnel, and push to the remote write endpoint of Prometheus at home:

Prometheus config out_of_order_time_window: 30m accommodates out-of-order arrival across the public internet.

Grafana Dashboards

Provisioning directory: 7 host panels (CPU / GPU / Memory / Disk / Cooling, JSON per node) + 1 server status template ($host dropdown auto-discovers all host_names pushed via OTLP) + Network monitoring panel.

Panel expression example:

# CPU: 100*(1-avg by(host_name)(system_cpu_utilization_ratio{host_name="$host",state="idle"}))
# Memory: 100*system_memory_utilization_ratio{host_name="$host",state="used"}
# Liveness check (push model has no up metric, use last reported time instead):
time()-max(timestamp(system_memory_usage_bytes{host_name="$host"}))

3. Security: CA and mTLS

Security of the remote push chain is ensured by client certificates issued by a private CA:

~/infra/metrics-fleet/ca/
├── make-ca.sh              # One-time CA creation (EC P-256, 10 years)
├── issue-server-cert.sh    # Issue server cert → /etc/caddy/metrics-fleet/
├── issue-client-cert.sh    # Issue client cert → out/clients/<name>/
├── revoke-client-cert.sh   # Revoke + update Caddy revoked.caddy + reload
└── out/clients/<name>/     # Issued client certs (one per remote machine)

Caddy acts as an mTLS gateway on 127.0.0.1:9443: verifies client certificate (require_and_verify, trust CA), checks revocation list → 403, only allows /api/v1/otlp/* with strip_prefix → otelcol-home :4318.

The public IP of each remote server must be added to the DIRECT rules of mihomo on both the local machine and the N100 to ensure rathole connections and SSH bypass the proxy (transport connections must use native routing — see rathole-tunnel.md section 4.3).

4. Alerts: Grafana + Bark Push to iPhone

Prometheus handles collection, Grafana handles evaluation and notification. Alert chain:

Why use local bark-server instead of public webhook: If alert notifications go through rathole tunnel or public internet—when the tunnel is down, alerts can't be sent. Local bark-server connects directly to iPhone, zero external dependencies, not bound to infrastructure.

Contacts and Notification Strategy

# /etc/grafana/provisioning/alerting/contact-points.yaml
contactPoints:
  - name: bark-home
    receivers:
      - type: webhook
        settings:
          url: "http://127.0.0.1:8085/<token>"   # Local bark-server, bypass tunnel
          maxAlerts: 2                            # APNs 4KB limit, >2 alerts causes PayloadTooLarge
          title: "[{{ .Status }}] {{ .CommonLabels.grafana_folder }}{{ if .CommonLabels.alertname }} · {{ .CommonLabels.alertname }}{{ end }}"
          message: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}"

policies:
  - receiver: bark-home
    group_by: ["grafana_folder", "alertname"]     # Merge same type, send different types separately
    group_wait: 30s                               # Initial delay to wait for same-type alerts
    group_interval: 5m                            # Interval for subsequent alerts in same group
    repeat_interval: 4h                           # Re-alert every 4h if unresolved

maxAlerts: 2 is key—bark passes the entire webhook JSON to APNs, each alert is about 1.1–1.4KB, exceeding 2 alerts surpasses APNs 4KB limit, bark returns 500.

Alert Rules (14 total)

Rule	Severity	Condition	Silence
Side Router Down	critical	up{job="li-home-router"}==0	3m
Router Unreachable	critical	ping_loss{target="gateway"}>30%	2m
High External Packet Loss	warning	ping_loss{target="223.5.5.5"}>20%	3m
High TCP Retransmission	warning	Tcp_RetransSegs / Tcp_OutSegs > 10%	5m
DNS All Slow	warning	All three domain resolutions >100ms	5m
mihomo TUN Failure	critical	nikki network interface metrics missing	2m
CN Node Disk Low	warning	Root partition remaining <5GB	5m
Water Pump Stopped	critical	fan7 RPM <1500	5m
VRM Overheating	warning	temp5 >90°C	2m
GPU Hotspot Overheating	warning	7900XTX Junction Temp >100°C	2m
N100 CPU Overload	warning	CPU >90%	5m
Local Exporter Down	critical	up{job="local-info"}==0	2m
rathole Service Unreachable	critical	rathole_service_up < 1 (tagged by service)	3m
Daily Report Email Delivery Failure	warning	Loki:llm-job@daily-report failed or notify-email: Send Failed	1m

Rules are managed via Grafana provisioning (/etc/grafana/provisioning/alerting/home-network.yaml), git versioned, no need to enter Web UI to change rules. Each alert describes the phenomenon in annotations.summary, and provides troubleshooting steps in annotations.runbook—alert notifications only push summary, runbook stays in Grafana to expand and view. Daily reports go via local email; when notify-email fails, systemd/Loki triggers this Bark, alert channel doesn't depend on email.

Metrics Pitfalls

• N100 Side Router Onboarding: Prometheus scraping N100 needs to go through local proxy to work, add DIRECT rule to bypass.

5. Log Monitoring (Loki + OpenTelemetry)

Fills the gap of "only metrics, no logs". All local system logs (kernel + systemd services + Docker containers) go into journald → otelcol-logs → Loki → Grafana Explore for search.

Design Trade-offs

• Choose OpenTelemetry, not Promtail/Vector: Local already runs otelcol-home, introducing Vector would duplicate collector tech stack. OTel also makes remote host log onboarding nearly free (reuse same mTLS/rathole/Caddy tunnel).
• Loki uses native OTLP endpoint /otlp/v1/logs (old lokiexporter is deprecated).
• Docker changed to journald log driver: Container logs with CONTAINER_NAME go into journald, collected by same journaldreceiver → has real container names, single data source.
• native systemd, bind 127.0.0.1, 90-day retention, no log dashboard (direct Explore).

Data Flow

Components

Component	Location	Notes
Loki 3.7	/usr/local/bin/loki, conf /etc/loki	:3100, OTLP /otlp/v1/logs, user loki
otelcol-logs	/usr/local/bin/otelcol-contrib (0.154, reuse fleet binary)	unit /etc/systemd/system/otelcol-logs.service, no root needed (group systemd-journal)
Grafana Data Source	/etc/grafana/provisioning/datasources/loki.yaml	uid loki-local

Labeling Scheme

Low cardinality goes to index labels, high cardinality goes to structured metadata (queryable but not indexed):

Index Label	Meaning
host_name	resource host.name (whole machine name, align with metrics)
service_name	= host.name, Loki native grouping field
unit	systemd service name; Docker container = container name (overridden by OTTL from CONTAINER_NAME)
container	Docker container name (container logs only)
level	See "Severity Determination" below
transport	kernel / stdout / syslog / journal / audit

Severity Determination (Hybrid Strategy)

journald's PRIORITY is only the true level for native sources (systemd-journal/syslog/kernel); PRIORITY for container logs and raw stdout/stderr only represents stream (stdout=6/stderr=3), meaningless.

1. Baseline: Containers and non-stdout sources get levels by PRIORITY (0–2 critical / 3 error / 4 warn / 5–6 info / 7 debug); container logs default to info.
2. Body Override: OTTL IsMatch identifies [error]/level=info/"level":"warn" and standalone uppercase words WARNING/ERROR from message body.
3. Loki side discover_log_levels: true—Loki identifies level attribute and fills detected_level accordingly (used for Grafana level visualization).

Query Examples (Grafana Explore)

{unit="sshd.service"}
{level="error"}
{container="adguardhome"}
{transport="kernel"}
{service_name="li-home-0"} |= "timeout"

6. Log Pitfalls

1. Loki single binary ring self-connection failure: Default broadcasts NIC IP, but gRPC only listens on 127.0.0.1 → connection refused. Must set common.instance_addr: 127.0.0.1.
2. journaldreceiver puts entire log in body(Map): Fields are body["PRIORITY"] not attributes. OTTL first extracts attributes from body, finally set(body, body["MESSAGE"]).
3. service_name semantics: Want whole machine identifier, which is resource-level. Loki discover_service_name only reads resource attributes → set service.name = host.name in OTTL resource context.
4. Docker container unit: Under journald driver _SYSTEMD_UNIT=docker.service is useless → OTTL overrides with CONTAINER_NAME.
5. Don't trust PRIORITY for levels: adguard/HA/owntracks write info+warning all to stderr → PRIORITY all 3, if trusted they all become error. Don't set discover_log_levels to false—otherwise detected_level disappears, Grafana histogram collapses to single line.
6. Colored logs = byte arrays: MESSAGE with ANSI is stored as int-slice by journald, OTTL cannot decode. Root fix = disable color at source: containers add NO_COLOR=1+PY_COLORS=0; Rust services add Environment=NO_COLOR=1+RUST_LOG_STYLE=never.
7. node_exporter sgcc spam: sgcc_ts.prom with timestamps (root all orphan files), textfile collector doesn't support. Remove from textfile_collector/ to fix.
8. runlike rebuild container loses data: uvx runlike outputs errors for anonymous/named volumes, need to explicitly pin volumes via docker inspect .Mounts.

7. Traces: Tempo + tail sampling

Traces and metrics share the same OTLP entry point (otelcol-home :4318), split internally by pipeline:

Sampling Strategy

Uses tail sampling (not head sampling), ensuring complete span trees without truncation:

Strategy	Condition	Sample Rate
errors-only	status_code == ERROR	100%
latency-sampling	Latency >500ms	100%
probabilistic	Other normal requests	10%

tail sampling has 10s decision delay compared to head sampling (waits for span tree to collect before deciding), but avoids "parent span sampled, child span not sampled" broken chains—which is crucial for troubleshooting distributed call chains.

Tempo Backend

# /etc/tempo/config.yaml
distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 127.0.0.1:4317
ingester:
  max_block_duration: 5m
  lifecycler:
    ring:
      kvstore: { store: inmemory }
      replication_factor: 1

Single instance inmemory ring (no cluster needs), 720h (30 days) retention. Same pattern as Loki: native systemd, bind 127.0.0.1, Grafana Explore directly searches by Trace ID.

Onboarding Method

App side set OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 (local) or https://otlp.liz6.com:9443 (remote, via fleet tunnel). SDK automatically generates traces. Remote apps reuse existing fleet collector's mTLS certificates—metrics and traces share the same pipeline and security system.

8. Future Expansion: Remote Fleet Logs

Reuse existing metrics-fleet transport (no need to modify Caddy, /api/v1/otlp/* already covers v1/logs):

1. Remote otelcol-fleet agent add journald receiver + logs pipeline → otlphttp to otlp.liz6.com:9443
2. Home otelcol-home add logs pipeline → otlphttp to Loki /otlp/v1/logs
3. add-server.sh/CA/naming all inherited, host_name/service_name are each node name

Related Documents

• network-architecture.md — Network topology and daemons
• rathole-tunnel.md — Tunnel architecture (transport connections bypass mihomo)
• server-lifecycle.md — Remote server onboarding script (add-server)