Grafana Operations Manual

Grafana is a data visualization platform. Core concepts: datasource, dashboard (collection of panels), panel (individual chart), alert rule, contact point (notification channel), provisioning (file-driven configuration management).

Provisioning

Grafana supports automatic creation of dashboards and alert rules via files—no manual UI interaction required. File changes take effect automatically after ~10s; use systemctl reload grafana to force a refresh.

/etc/grafana/provisioning/
├── dashboards/
│   ├── host-provider.yaml              # provider config: which folder, where to read JSON
│   └── json-all/host/*.json            # dashboard JSON files
├── alerting/
│   └── rules.yaml                      # alert rules (includes datasourceUid)
└── datasources/
    └── prometheus.yaml                 # datasource definition

provider yaml:

apiVersion: 1
providers:
- name: host
  folder: Host Status                   # target folder (created if it doesn't exist)
  type: file
  options:
    path: /etc/grafana/provisioning/dashboards/json-all/host

datasource yaml:

apiVersion: 1
datasources:
- name: Prometheus
  type: prometheus
  url: http://localhost:9090
  access: proxy
  uid: dfguyfutpcnpca                   # fixed UID (referenced in dashboard JSON)

Dashboard JSON Debugging

# View datasource and panels of a dashboard
jq '.panels[] | {title, targets: .targets[].expr}' dashboard.json

# View template variables
jq '.templating.list[] | {name, query, current}' dashboard.json

# Batch modify datasource uid
sed -i 's/"uid": "old-uid"/"uid": "new-uid"/g' dashboard.json

Unified Storage Operations (Grafana 12+)

Grafana 12 uses SQLite to store all objects (unified storage). Dashboards created via the UI are stored in the database, not in files.

systemctl stop grafana
cp /var/lib/grafana/grafana.db grafana.db.bak   # backup first!

sqlite3 /var/lib/grafana/grafana.db <<SQL
-- View dashboards and folders created via UI
SELECT name, kind FROM resource WHERE kind IN ('dashboard', 'folder');

-- Delete UI-created dashboard (provisioned ones are managed by files, not here)
DELETE FROM resource WHERE name = 'old-dashboard' AND kind = 'dashboard';
DELETE FROM resource_history WHERE name = 'old-dashboard';

-- Clean up history table (can occupy significant space)
SELECT COUNT(*) FROM resource_history;
DELETE FROM resource_history;
SQL

systemctl start grafana

Alerts

# List contact points
curl -s -u admin:<pass> http://localhost:3000/api/v1/provisioning/contact-points | jq

# Export contact points including secrets
curl -s -u admin:<pass> 'http://localhost:3000/api/v1/provisioning/contact-points/export?decrypt=true&format=json' | jq

# Test alert template (does not send notifications, only renders)
curl -s -u admin:<pass> -X POST \
  http://localhost:3000/api/alertmanager/grafana/config/api/v1/templates/test \
  -H 'Content-Type: application/json' \
  -d '{"name":"test","template":"{{ define \"test\" }}Alert: {{ .Alerts.Firing | len }} firing{{ end }}","alerts":[{"Status":"firing","Labels":{"alertname":"Test"}}]}'

Troubleshooting

journalctl -u grafana -f --no-pager

# Dashboard not showing data
# → Check if the UID pointed to by the panel's datasource exists
curl -s -u admin:<pass> http://localhost:3000/api/datasources | jq '.[].uid'
# → Check if the datasource itself is reachable: curl http://localhost:9090/api/v1/query?query=up

# Alert not triggering
# → Check if the rule's datasourceUid is correct
grep -r datasourceUid /etc/grafana/provisioning/alerting/
# → Check evaluation interval: /etc/grafana/grafana.ini → [alerting] → evaluation_interval

# Grafana fails to start
# → Often due to DB corruption or permission issues: journalctl -u grafana -e
# → Check if /var/lib/grafana/grafana.db is writable