Webhooks and Callbacks

The "reverse API" where the server actively POSTs to a client-registered URL—real-time push upon event occurrence, replacing inefficient polling. HMAC signatures prevent forgery, while retries and idempotent deduplication ensure reliability. Almost all SaaS platforms (GitHub, Stripe, Grafana) offer webhook integrations.

Overview

A webhook is a "reverse API" where the server actively POSTs to a client-registered URL—real-time push upon event occurrence, replacing inefficient polling. The core security mechanism is HMAC signature verification (to prevent forgery), and the core reliability mechanism is retries plus idempotent deduplication. Almost all SaaS platforms (GitHub, Stripe, Grafana) provide webhooks. DingTalk/WeChat bot webhooks are common integration points for IM applications.

Principles

Webhook = Server actively POSTs to a client-registered URL (user-defined HTTP callback):

Traditional Polling:  Client → GET /api/status (every 30s) → Wasteful, Latent
Webhook:       Server → POST https://client/hook (on event) → Real-time, Zero Polling

Payload + Signature:
  POST /hook HTTP/1.1
  Content-Type: application/json
  X-Signature-256: sha256=3a2b1c...
  X-Delivery: uuid-1234  ← Unique event ID, for deduplication

  {"event":"alert","status":"firing"}

HMAC Signature Verification

Receiver verifies: Compute HMAC(payload_body, secret) using the shared secret → Compare with the header. Mismatch → Reject (forgery or man-in-the-middle tampering).

Retry Strategy

Non-2xx response → Exponential backoff retry (5s, 25s, 125s, ...). Repeated failures → Disable webhook.

Deduplication: delivery UUID → Record processed event IDs → Duplicate → Skip. Idempotent operations (same effect regardless of how many times executed) provide a stronger guarantee.

Grafana → DingTalk Webhook

POST https://oapi.dingtalk.com/robot/send?access_token=<token>
Content-Type: application/json

{
  "msgtype": "actionCard",
  "actionCard": {
    "title": "CPU high on li-home-0",
    "text": "### Alert\n- Instance: li-home-0\n[View](https://grafana.liz6.com/...)",
    "singleTitle": "View",
    "singleURL": "https://grafana.liz6.com/d/..."
  }
}

Grafana's built-in DingTalk channel has the singleURL hardcoded to point to Grafana itself → Change to a custom webhook actionCard → singleURL points to Loki drilldown.

References

Grafana: grafana.com/docs/grafana/latest/alerting/configure-notifications
DingTalk: open.dingtalk.com

Keywords: webhook, HMAC, idempotency, retry, Grafana, DingTalk, signature