Grafana Operations Manual

Separate data sources, queries, panels and alerting, and identify each object’s configuration owner. UI editing does not imply persistence back to provisioning files. Internal database tables are not a supported object-management API.

Establish query results before panel rendering

Service and database health

# Inspect the process and startup result
systemctl status grafana-server.service --no-pager

Substitute the installed unit; for containers use the relevant container logs.

# Read recent errors from this boot
journalctl -u grafana-server.service -b -n 100 --no-pager

# Inspect health response and version
curl -fsS http://127.0.0.1:3000/api/health | jq .

Check database status. HTTP success does not prove data-source queries work; refusal suggests listener/path issues, while 401/403 suggests access rules.

Identify the data-source UID, then reproduce the query

# Enter a read-scoped service-account token in this shell
read -r -s GRAFANA_TOKEN

Input is hidden; press Enter to finish. Use the correct organization. The following API requests reference this variable.

# List visible data sources in this organization
curl -fsS -H "Authorization: Bearer ${GRAFANA_TOKEN:?Load a read-scoped token}" \
  http://127.0.0.1:3000/api/datasources | jq ".[] | {uid,name,type,url}"

Match the panel UID. An empty list may indicate organization or permission issues rather than deletion.

# Request plugin health for the verified UID
curl -fsS -H "Authorization: Bearer ${GRAFANA_TOKEN:?Load a read-scoped token}" \
  http://127.0.0.1:3000/api/datasources/uid/prometheus-main/health | jq .

Replace prometheus-main; the plugin must implement health checking. Distinguish unknown UID from unsupported APIs. Then reproduce expression, time range, and variables in Explore.

# Clear the token variable when API checks are complete
unset GRAFANA_TOKEN

Adapt service/port and use a scoped service account in the intended organization. health is not a check of every data source. Datasource requests commonly originate from Grafana’s server; browser access does not establish server/container connectivity.

UID identifies references; files own provisioned config

apiVersion: 1
datasources:
  - name: Prometheus
    uid: prometheus-main
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    editable: false

Use the configured provisioning/datasources directory and a hostname resolvable from Grafana. Keep stable UIDs for references. Datasource provisioning loads at startup; not all provisioning follows dashboard polling. Use the supported type-specific admin reload API or a planned restart, not an assumed systemctl reload.

allowUiUpdates can permit dashboard UI saves without writing the source file; later source updates can overwrite them. Review exported UI changes into source control. Delete through the configuration owner with the relevant deletion/prune settings, not internal resource-table edits.

Verify objects after provisioning deployment

# Identify startup arguments, identity, and configuration source
systemctl cat grafana-server.service

Locate --config, GF_PATHS_PROVISIONING, and mounts. Retain the previous file before editing YAML.

# Reload startup configuration during a planned maintenance window
sudo systemctl restart grafana-server.service

This interrupts the instance and may affect rule evaluation. Follow rollout policy for replicas; this systemd option is an alternative to an Admin API reload.

# Inspect provisioning load errors
journalctl -u grafana-server.service --since "5 minutes ago" --no-pager

Repeat health and UID checks, then open a real panel. On failure restore the source file and reload through the same workflow.

Firing is upstream of notification policy

Silences generally suppress notifications, not evaluation. No Data/Error settings can generate distinct instances, not a metric value of zero. Preview templates and routes first; contact-point tests actually send messages. Update file-managed alerts through their provisioning workflow.

Separate evaluation from delivery failures

# Read evaluation and notification logs around the event
journalctl -u grafana-server.service --since "30 minutes ago" --no-pager

Correlate rule UID, organization, and time with the UI. Missing delivery entries may reflect log level rather than proof of no delivery.

Back up the actual backend and configuration

Identify SQLite, PostgreSQL or MySQL from configuration, not version assumptions. Back up DB, settings, provisioning, plugins and the encryption secret_key. Restore separately and verify source/dashboard UIDs and alerts. Use the database guide for consistent backups. A binary downgrade does not guarantee compatibility with migrated storage.

Grafana provisioning · Query inspector · Alerting · Backup