Prometheus Operations Manual

Validate discovery, scraping, storage, queries and rules separately. up=1 means a scrape succeeded, not that the application is healthy. Missing series, failed scrapes and application failures are distinct.

Task	Start here
Target down / missing metrics	Discovery and scraping
Slow query / odd graph	Window, step and cardinality
Change config / rules	Validate and reload
Storage and backup	TSDB scope

Establish that the target still exists

Check readiness separately from discovery

# Check that the instance can serve queries
curl -fsS http://127.0.0.1:9090/-/ready

Each request contains an explicit base URL; replace consistently and use the actual proxy authentication/TLS options.

# Inspect scrape URL, final labels, and latest errors
curl -fsS http://127.0.0.1:9090/api/v1/targets |
  jq '.data.activeTargets[] | {labels,scrapeUrl,health,lastError,lastScrape}'

Use lastError to distinguish timeout, TLS, refusal, and parsing issues; verify lastScrape is recent.

# Inspect dropped discovery labels when a target is absent
curl -fsS 'http://127.0.0.1:9090/api/v1/targets?state=dropped' |
  jq '.data.droppedTargets[] | .discoveredLabels'

Retention limits can truncate this list; absence still requires discovery/configuration checks.

# Query scrape success at the current evaluation time
curl -fsS --get http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=up' | jq .

Check top-level status before data.result. Narrow by actual job/instance; an empty array is not zero.

Result	Meaning	Next check
up = 0	Target exists, scrape failed	lastError, timeout, DNS, TLS, auth, format
up = 1	Scrape succeeded, application may still fail	Application metrics and user-facing probes
No up series	Discovery/relabel or time/label mismatch	Discovered/dropped targets and selectors

Fetch scrapeUrl from Prometheus’s network and inspect metrics. A listening exporter does not establish matching DNS, credentials or trust. Query actual post-relabel labels. Readiness is not all-target health.

step changes evaluation spacing, not scraping

Choose instant or range evaluation

# Evaluate each counter’s five-minute rate now
curl -fsS --get http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=rate(http_requests_total[5m])' | jq .

Use an existing counter metric. The result is an increase per second, not the window’s total requests.

# Re-evaluate every 30 seconds over an explicit UTC range
curl -fsS --get http://127.0.0.1:9090/api/v1/query_range \
  --data-urlencode 'query=rate(http_requests_total[5m])' \
  --data-urlencode 'start=2026-09-10T00:00:00Z' \
  --data-urlencode 'end=2026-09-10T01:00:00Z' \
  --data-urlencode 'step=30s' | jq .

This is an alternative query mode. Start with the affected interval and inspect resultType, timestamps, values, and any warnings/infos.

Choose a populated time range. URL encoding protects expression syntax. Each range step evaluates the expression again; rate’s 5m is its sample window. Smaller steps do not create raw samples. Apply rate before aggregation to detect individual resets; short windows may lack two samples.

Preparing the visual

The model counts a full Cartesian product, not guaranteed actual occupancy. Unbounded user/request IDs and raw URLs create churn; prefer route templates and bounded labels, keeping detail in logs. Recording rules materialize extra series; they do not remove raw scraped cardinality.

Verify both files and runtime state

Before deployment: identify and validate active files

# Check configuration arguments and service identity
systemctl cat prometheus.service

Substitute actual config/rule paths, retain previous files, and use a compatible promtool version.

# Validate configuration and referenced rules
promtool check config /etc/prometheus/prometheus.yml

Publish only after success; this does not replace actual scrape/authentication tests.

# Validate target rule files when editing rules separately
promtool check rules /etc/prometheus/rules/*.yml

This is a focused alternative check; do not mechanically repeat identical validation. The glob must match real files.

# Run authored rule-behavior tests
promtool test rules /etc/prometheus/tests/alerts.test.yml

This is a test specification, not a rule file. Define input series and expected alerts/expressions around thresholds, for duration, recovery, and missing data.

After deployment: signal, result, and timestamp

# Request reload from the verified main process
sudo systemctl kill --signal=HUP --kill-whom=main prometheus.service

Signal delivery is not successful loading. Record the time and verify below.

# Inspect this configuration reload’s logs
journalctl -u prometheus.service --since "5 minutes ago" --no-pager

# Confirm the running configuration matches the intended content
curl -fsS http://127.0.0.1:9090/api/v1/status/config | jq -r '.data.yaml'

# Check reload success and the last successful timestamp together
curl -fsS --get http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query={__name__=~"prometheus_config_last_reload_successful|prometheus_config_last_reload_success_timestamp_seconds"}' | jq .

Requires self-scraping and a fresh scrape. Expect success=1 and a timestamp matching this operation; an old 1 is insufficient. On failure restore, validate/reload, then check targets and rules.

Adapt paths and target the verified main process. POST /-/reload requires lifecycle support; do not expose control endpoints casually. Check success and its timestamp. File checks cannot catch all runtime permissions/network failures. Use promtool rule unit tests for thresholds, for durations and recovery.

Inspect cardinality, retention and disk before deletion

Identify space consumption

# Inspect the actual data path and retention flags
curl -fsS http://127.0.0.1:9090/api/v1/status/flags | jq '.data'

Use storage.tsdb.path below; map container paths to the backing volume.

# Inspect head and label cardinality statistics
curl -fsS http://127.0.0.1:9090/api/v1/status/tsdb | jq '.data'

Compare against this instance’s history and workload rather than a universal limit.

# Inspect free space on the data filesystem
df -hT /var/lib/prometheus

# Look for WAL, compaction, corruption, or write errors
journalctl -u prometheus.service --since "1 hour ago" --no-pager

Back up through an already enabled snapshot API

Requires an approved, access-restricted --web.enable-admin-api deployment and enough free space. This creates a snapshot without deleting data.

# Create a snapshot including current head data
curl -fsS -X POST 'http://127.0.0.1:9090/api/v1/admin/tsdb/snapshot?skip_head=false' | jq .

Record data.name. Copy snapshots/name from the data directory to independent backup storage and verify it. A same-disk snapshot is not an off-host copy and excludes later writes.

Restore to an isolated directory with a compatible version and notification delivery disabled. Verify representative historical queries, coverage, and series before planning production recovery; keep matching configuration records.

Confirm storage.tsdb.path and correlate head series, churn, query cost and retention. There is no universal million-series failure threshold. Size retention is not a hard cap on every directory byte; reserve space for WAL/head too. Fix cardinality sources before deleting old data.

Snapshots and deletion need explicitly enabled admin APIs. For snapshot backups decide head inclusion and separately retain config, rules, service settings and auth; restore to an isolated instance. delete_series, tombstones and compaction are distinct destructive stages, not a universal cleanup shortcut. Trace notifications through the actual Alertmanager/Grafana deployment.

HTTP API · Querying basics · Metric practices · Storage · Rule testing