Prometheus Operations Manual

Prometheus is a time-series database combined with a monitoring and alerting engine. Core concepts: target (collection target), metric (indicator), label (dimension), rule (recording rule + alerting rule), TSDB (time-series storage).

Target

# API query target status
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {labels, health, lastError}'
# health: "up" = normal, "down" = collection failed
# lastError: reason for the last failure (timeout/connection refused/format error)

# Check which instances have an up metric of 0 (unreachable)
curl -s 'http://localhost:9090/api/v1/query?query=up==0' | jq '.data.result[] | .metric.instance'

# Web UI: http://localhost:9090/targets
# → You can see each target's labels, last scrape time, scrape duration, errors

Configuration and Hot Reload

# Check configuration syntax
promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/*.yml

# Hot reload (no restart, no data loss, only reloads rules + targets)
kill -HUP $(pidof prometheus)
# Or: curl -X POST http://localhost:9090/-/reload

Typical prometheus.yml structure:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['localhost:9100', '192.168.1.7:9100']
        labels: { env: 'prod' }

TSDB Management

# Data directory (a block directory is generated every 2 hours)
du -sh /var/lib/prometheus/data
ls /var/lib/prometheus/data/           # See block directories like 01xxxx-02xxxx

# TSDB status (number of series, memory usage, number of blocks)
curl -s http://localhost:9090/api/v1/status/tsdb | jq '.data'
# headStats.numSeries: number of currently active series (> 1M needs attention)
# headStats.chunkCount: number of chunks in memory

# Delete old data (requires --web.enable-admin-api startup parameter, disabled by default)
curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="old-job"}'
# After marking for deletion, clean_tombstones is required
curl -X POST http://localhost:9090/api/v1/admin/tsdb/clean_tombstones

# Snapshot (full data backup)
curl -X POST http://localhost:9090/api/v1/admin/tsdb/snapshot
# → /var/lib/prometheus/data/snapshots/<timestamp>/
# Snapshots can be copied directly to another Prometheus instance

Query Debugging

# Instant query
curl -s 'http://localhost:9090/api/v1/query?query=rate(node_cpu_seconds_total[5m])' | jq

# Range query (step = sampling interval)
curl -s 'http://localhost:9090/api/v1/query_range?query=up&start=...&end=...&step=60s' | jq

# List all label values
curl -s http://localhost:9090/api/v1/label/__name__/values | jq '.data[:20]'
curl -s http://localhost:9090/api/v1/label/job/values | jq

Troubleshooting

# Target down
curl http://<target>:<port>/metrics              # Check if the exporter is exposing data
# → timeout → firewall/systemd/port binding issue
# → 200 but empty → exporter is running but has no data

# High memory usage
# → High cardinality labels (e.g., request_id, user_id) create a new series for each unique value
# → Check: curl http://localhost:9090/api/v1/status/tsdb | jq '.data.headStats.numSeries'
# → Over 1 million → consider reducing label dimensions or using recording rules for pre-aggregation

# Slow queries/timeouts
# → Check the query's range and step: larger range + smaller step → more data points
# → Use recording rules to pre-calculate common queries