Full-Stack Monitoring: Prometheus + Grafana + Loki + Tempo + Bark

Metrics (Prometheus) + Logs (Loki) + Traces (Tempo) all feed into Grafana, with alerts pushed to iPhone via Bark. Metrics use two sets: pull (local network node_exporter) and push (remote otel-collector fleet agent, via rathole tunnel + mTLS); logs go journald → otelcol-logs → Loki; traces go otelcol-home tail sampling → Tempo. Using local host li-home-0 as the instance, this covers full-stack architecture, alert rules, collection comparison, labeling scheme, severity determination, and pitfall records.

1. Overview

2. Metrics Collection: Two Models

	Traditional Pull (node_exporter)	Remote Push (metrics-fleet)
Use Case	Local network nodes, local host	Public internet remote servers
Collector	node_exporter / LHM exporter	otel-collector-contrib (host_metrics)
Metric Naming	node_*, windows_*, lhm_*	system_cpu_*, system_memory_*
Network	Prometheus actively pulls, requires reachable ports	Remote actively pushes, zero inbound ports
Transport Security	None (unencrypted on LAN)	mTLS + Path Restriction (only OTLP write allowed)
Onboarding Method	Manually edit prometheus.yml static_configs	add-server.sh <ssh-alias>
Dashboard	One independent JSON per host	Single template, $host dropdown to switch
Transit Dependency	None	rathole tunnel → Transit VPS → Home

Pull Model: scrape_configs

Prometheus pulls these targets every 15s: local host node_exporter (:9100), llama-server (:18080), Home Assistant (:8123), Windows machines (9182/9183), N100 side router (:9100), Xiaomi router (:9184), self-monitoring (:9090).

Push Model: OTLP Receiver

Remote servers collect host_metrics via otel-collector, authenticate via mTLS + rathole encrypted tunnel, and push to the Prometheus remote write endpoint at home:

Prometheus config out_of_order_time_window: 30m accommodates out-of-order arrival across the public internet.

Grafana Dashboards

Provisioning directory: 7 host panels (CPU / GPU / Memory / Disk / Cooling, per-node JSON) + 1 server status template ($host dropdown auto-discovers all host_names pushed via OTLP) + Network monitoring panel.

Panel expression example:

# CPU: 100*(1-avg by(host_name)(system_cpu_utilization_ratio{host_name="$host",state="idle"}))
# Memory: 100*system_memory_utilization_ratio{host_name="$host",state="used"}
# Liveness check (push model has no up metric, use last reported time instead):
time()-max(timestamp(system_memory_usage_bytes{host_name="$host"}))

3. Security: CA and mTLS

Security for the remote push chain is ensured by client certificates issued by a private CA:

~/infra/metrics-fleet/ca/
├── make-ca.sh              # One-time CA creation (EC P-256, 10 years)
├── issue-server-cert.sh    # Issue server cert → /etc/caddy/metrics-fleet/
├── issue-client-cert.sh    # Issue client cert → out/clients/<name>/
├── revoke-client-cert.sh   # Revoke + update Caddy revoked.caddy + reload
└── out/clients/<name>/     # Issued client certs (one per remote host)

Caddy acts as an mTLS gateway on 127.0.0.1:9443: verifies client cert (require_and_verify, trust CA), checks revocation list → 403, only allows /api/v1/otlp/* with strip_prefix → otelcol-home :4318.

The public IP of each remote server must be added to the DIRECT rules in mihomo on both the local host and the N100, ensuring rathole connections and SSH do not go through the proxy (transport connections must use native routing—see rathole-tunnel.md section 4.3).

4. Alerts: Grafana + Bark Push to iPhone

Prometheus handles collection, Grafana handles evaluation and notification. Alert chain:

Why use local bark-server instead of public webhook: If alert notifications go via rathole tunnel or public internet—when the tunnel is down, alerts can't be sent. Local bark-server connects directly to iPhone, zero external dependencies, not bound to infrastructure.

Contacts and Notification Strategy

# /etc/grafana/provisioning/alerting/contact-points.yaml
contactPoints:
  - name: bark-home
    receivers:
      - type: webhook
        settings:
          url: "http://127.0.0.1:8085/<token>"   # Local bark-server, bypass tunnel
          maxAlerts: 2                            # APNs 4KB limit, >2 alerts causes PayloadTooLarge
          title: "[{{ .Status }}] {{ .CommonLabels.grafana_folder }}{{ if .CommonLabels.alertname }} · {{ .CommonLabels.alertname }}{{ end }}"
          message: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}"

policies:
  - receiver: bark-home
    group_by: ["grafana_folder", "alertname"]     # Merge similar, send different separately
    group_wait: 30s                               # Initial delay to wait for similar alerts
    group_interval: 5m                            # Interval for subsequent alerts in same group
    repeat_interval: 4h                           # Re-alert every 4h if unresolved

maxAlerts: 2 is key—bark passes the entire webhook JSON to APNs, each alert is ~1.1–1.4KB, exceeding 2 alerts surpasses APNs 4KB limit, bark returns 500.

Alert Rules (13 total)

Rule	Severity	Condition	Silence
Side router down	critical	up{job="li-home-router"}==0	3m
Router unreachable	critical	ping_loss{target="gateway"}>30%	2m
High external packet loss	warning	ping_loss{target="223.5.5.5"}>20%	3m
High TCP retransmission	warning	Tcp_RetransSegs / Tcp_OutSegs > 10%	5m
DNS all slow	warning	All three domain resolutions >100ms	5m
mihomo TUN failure	critical	nikki network interface metrics missing	2m
CN node disk low	warning	Root partition remaining <5GB	5m
Water pump stopped	critical	fan7 RPM <1500	5m
VRM overheating	warning	temp5 >90°C	2m
GPU hotspot overheating	warning	7900XTX junction temp >100°C	2m
N100 CPU overload	warning	CPU >90%	5m
Local exporter down	critical	up{job="local-info"}==0	2m
rathole service unreachable	critical	rathole_service_up < 1 (tagged by service)	3m

Rules are managed via Grafana provisioning (/etc/grafana/provisioning/alerting/home-network.yaml), versioned in git, no need to enter Web UI to change rules. Each alert describes the phenomenon in annotations.summary, and provides troubleshooting steps in annotations.runbook—alert notifications only push summary, runbook is expanded in Grafana.

Metrics Pitfalls

• N100 side router onboarding: Prometheus scraping N100 requires going through local host proxy, add DIRECT rule to bypass.

5. Log Monitoring (Loki + OpenTelemetry)

Fills the gap of "only metrics, no logs". All local system logs (kernel + systemd services + Docker containers) go uniformly into journald → otelcol-logs → Loki → Grafana Explore for retrieval.

Design Trade-offs

• Choose OpenTelemetry, not Promtail/Vector: Local host already runs otelcol-home, introducing Vector would duplicate collector tech stack. OTel also makes remote host log onboarding nearly free (reuse same mTLS/rathole/Caddy tunnel).
• Loki uses native OTLP endpoint /otlp/v1/logs (old lokiexporter is deprecated).
• Docker changed to journald log driver: Container logs with CONTAINER_NAME go into journald, collected by same journaldreceiver → has real container names, single data source.
• native systemd, bind 127.0.0.1, 90-day retention, no log dashboard (use Explore directly).

Data Flow

Components

Component	Location	Notes
Loki 3.7	/usr/local/bin/loki, conf /etc/loki	:3100, OTLP /otlp/v1/logs, user loki
otelcol-logs	/usr/local/bin/otelcol-contrib (0.154, reuse fleet binary)	unit /etc/systemd/system/otelcol-logs.service, no root needed (group systemd-journal)
Grafana Data Source	/etc/grafana/provisioning/datasources/loki.yaml	uid loki-local

Labeling Scheme

Low cardinality goes to index labels, high cardinality goes to structured metadata (queryable but not indexed):

Index Label	Meaning
host_name	resource host.name (whole machine name, aligns with metrics)
service_name	= host.name, Loki native grouping field
unit	systemd service name; Docker container = container name (overwritten by OTTL from CONTAINER_NAME)
container	Docker container name (container logs only)
level	See "Severity Determination" below
transport	kernel / stdout / syslog / journal / audit

Severity Determination (Hybrid Strategy)

journald's PRIORITY is only the true severity for native sources (systemd-journal/syslog/kernel); PRIORITY for container logs and raw stdout/stderr only represents the stream (stdout=6/stderr=3), meaningless.

1. Baseline: Containers and non-stdout sources get severity based on PRIORITY (0–2 critical / 3 error / 4 warn / 5–6 info / 7 debug); container logs default to info.
2. Body Override: OTTL IsMatch identifies [error]/level=info/"level":"warn" and uppercase standalone words WARNING/ERROR from message body.
3. Loki side discover_log_levels: true—Loki recognizes level attribute and fills detected_level accordingly (used for Grafana severity visualization).

Query Examples (Grafana Explore)

{unit="sshd.service"}
{level="error"}
{container="adguardhome"}
{transport="kernel"}
{service_name="li-home-0"} |= "timeout"

6. Log Pitfalls

1. Loki single binary ring self-connection failure: Default broadcasts network card IP, but gRPC only listens on 127.0.0.1 → connection refused. Must set common.instance_addr: 127.0.0.1.
2. journaldreceiver puts entire log in body (Map): Fields are body["PRIORITY"] not attributes. OTTL first extracts attributes from body, finally set(body, body["MESSAGE"]).
3. service_name semantics: What we want is whole machine identifier, resource-level. Loki discover_service_name only reads resource attributes → set service.name = host.name in OTTL resource context.
4. Docker container unit: Under journald driver, _SYSTEMD_UNIT=docker.service is useless → OTTL overwrites with CONTAINER_NAME.
5. Don't trust PRIORITY for severity: adguard/HA/owntracks write info+warning all to stderr → PRIORITY all 3, if trusted they all become error. Don't set discover_log_levels to false—otherwise detected_level disappears, Grafana histogram collapses to single line.
6. Colored logs = byte arrays: MESSAGEs containing ANSI are stored as int-slice by journald, OTTL cannot decode. Root fix = disable color at source: containers add NO_COLOR=1+PY_COLORS=0; Rust services add Environment=NO_COLOR=1+RUST_LOG_STYLE=never.
7. node_exporter sgcc spam: sgcc_ts.prom has timestamps (orphan files owned by root), textfile collector doesn't support. Remove from textfile_collector/ to fix.
8. runlike rebuild container loses data: uvx runlike outputs errors for anonymous/named volumes, need to explicitly pin volumes via docker inspect .Mounts.

7. Traces: Tempo + tail sampling

Traces and metrics share the same OTLP entry point (otelcol-home :4318), internally split by pipeline:

Sampling Strategy

Uses tail sampling (not head sampling), ensuring span tree integrity is not truncated:

Strategy	Condition	Sampling Rate
errors-only	status_code == ERROR	100%
latency-sampling	Latency >500ms	100%
probabilistic	Other normal requests	10%

tail sampling has 10s decision delay compared to head sampling (waits for span tree to be complete before deciding), but avoids "parent span sampled, child span not sampled" broken chains—crucial for troubleshooting distributed call chains.

Tempo Backend

# /etc/tempo/config.yaml
distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 127.0.0.1:4317
ingester:
  max_block_duration: 5m
  lifecycler:
    ring:
      kvstore: { store: inmemory }
      replication_factor: 1

Single instance inmemory ring (no cluster requirement), 720h (30 days) retention. Same pattern as Loki: native systemd, bind 127.0.0.1, Grafana Explore directly retrieves by Trace ID.

Onboarding Method

Application side sets OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 (local) or https://otlp.liz6.com:9443 (remote, via fleet tunnel). SDK automatically generates traces. Remote applications reuse existing fleet collector's mTLS certificates—metrics and traces share the same pipeline and security system.

8. Future Expansion: Remote Fleet Logs

Reuse existing metrics-fleet transport (no need to modify Caddy, /api/v1/otlp/* already covers v1/logs):

1. Remote otelcol-fleet agent adds journald receiver + logs pipeline → otlphttp to otlp.liz6.com:9443
2. Home otelcol-home adds logs pipeline → otlphttp to Loki /otlp/v1/logs
3. add-server.sh/CA/naming all reused, host_name/service_name are each node's name

Related Documents

• network-architecture.md — Network topology and daemons
• rathole-tunnel.md — Tunnel architecture (transport connections bypass mihomo)
• server-lifecycle.md — Remote server onboarding script (add-server)