Webhooks and Callbacks

A Webhook is an HTTP callback integration method: the event source sends notifications to an address configured by the receiver, which triggers processing based on the content. It reduces the wait time for active polling but is not a unified standard for reliable message protocols; retries, signatures, ordering, history queries, and success status codes are all determined by specific platform contracts.

Taking order status notifications as an example, the source creates event E42 and sends it via HTTP to the receiver; the receiver first reliably persists the event, then a worker updates the local view. Even if the source subsequently resends E42, or if E43 arrives before E42, processing must remain correct.

Receiver: Separate HTTP Response from Business Completion

The receive path can limit request size and type, verify signatures and necessary timestamps, parse and validate events, and then write the raw content and processing status to a persistent Inbox. After successful submission, quickly return a platform-recognized 2xx, and let workers consume asynchronously. This avoids executing all complex work within the HTTP timeout window while retaining a recovery entry point after a crash.

If you return success first and then try to push the event to an in-memory queue, the process exiting in between could permanently lose the event. If recovery depends on database or queue persistence, you cannot pretend to have received success when submission fails. On the other hand, duplicate events that have already been reliably queued can return success to avoid meaningless retries from the source.

GitHub documentation considers a response timeout of over 10 seconds as a delivery failure and explicitly states it will not automatically retry failed webhooks; retries can be performed manually or via custom programs. Stripe has its own automatic retry and resend rules. You cannot write "all non-2xx responses back off by 5, 25, 125 seconds, and automatically disable after multiple failures." GitHub Failed Deliveries, Stripe Webhooks

Signature Verification: Process Raw Bytes First, Then Understand JSON

HMAC uses a shared secret as the key and the byte sequence specified by the platform as the message. GitHub's X-Hub-Signature-256 uses sha256= followed by the hexadecimal digest; other platforms may sign timestamps, versions, or other fields together. Therefore, you cannot uniformly implement all platforms as HMAC(body). GitHub Signature Verification

The same JSON object can have different whitespace, key ordering, and escape representations. If a framework parses and then serializes first, even if the object content is identical, the bytes may differ; signature verification should preserve the original request body as required by the protocol. Below, we verify the HMAC algorithm using GitHub's published test vectors, and then check that modifications to the body are rejected.

import hashlib, hmac, json
secret = b"It's a Secret to Everybody"
body = b'Hello, World!'
expected = '757107ea0eb2509fc211221cce984b8a37570b6d7586c22c46f4379c8b043e17'
assert hmac.new(secret, body, hashlib.sha256).hexdigest() == expected

def verify(raw, signature):
    if not signature.startswith('sha256='):
        return False
    value = signature[7:]
    if len(value) != 64 or any(ch not in '0123456789abcdef' for ch in value):
        return False
    calculated = hmac.new(secret, raw, hashlib.sha256).hexdigest()
    return hmac.compare_digest(calculated, value)

assert verify(body, 'sha256=' + expected)
assert not verify(body + b' ', 'sha256=' + expected)
a, b = b'{"a":1,"b":2}', b'{ "b": 2, "a": 1 }'
assert json.loads(a) == json.loads(b)
assert hmac.digest(secret, a, 'sha256') != hmac.digest(secret, b, 'sha256')
print('Public HMAC vector, tamper rejection, and JSON byte difference checks passed')

This code only demonstrates this signature format and is not a universal SDK for all providers. Real receivers must also handle duplicate signature headers, multiple valid keys during rotation, and platform-specific signature versions; avoid simple loops that leak prefix matching time during comparison.

Why Replay Protection is Needed Even with Valid Signatures

If an attacker obtains a valid old request, the HMAC check may still succeed. If the platform signs the delivery time, the time window should be checked against a trusted clock; if the protocol does not sign a timestamp, do not invent an unsigned header as proof. The time the message event occurred is also different from the current delivery time; legitimate resends may carry events that happened long ago.

Retry windows, manual replay windows, and deduplication record retention periods must work together. If you only keep deduplication keys for five minutes but allow an administrator to resend the same event a month later, you cannot expect to still prevent duplicate business effects.

Three Types of IDs: Business Event, Delivery Attempt, and Resource

An event ID represents a fact of an event, a delivery attempt ID is used to track HTTP sends, and a resource ID represents the affected order or alert. The same event may have multiple attempts; the same resource may generate multiple different events. Whether specific fields remain unchanged during resends depends on the platform; do not judge solely based on the name containing "delivery".

GitHub's X-GitHub-Delivery can be used to identify the same delivery and its retries; Stripe suggests recording processed event IDs and points out that in some scenarios, different Event objects may correspond to related business changes. Idempotency keys should be determined based on platform event semantics and business actions, rather than concatenating all sources into a scopeless UUID set. GitHub Best Practices, Stripe Duplicate Events

GitHub's body HMAC does not cover arbitrary HTTP headers, so the Delivery header helps handle normal duplicate deliveries but cannot alone prevent an attacker holding a valid old body from replaying with modified headers. Sensitive actions must still deduplicate based on business identity and operation conditions within the signed body; do not assign cryptographic guarantees to unsigned headers that they do not have.

Deduplication and Business Submission: Must Cover the Crash Window

When updating views in a local database, you can place Inbox status changes and business modifications in the same transaction. The following local model uses unique keys for deduplication and ensures that "event processed" does not commit independently before business effects during an injected crash. The example is a transaction semantic verification of in-memory SQLite, not simulating disk failures or actual concurrent deployments.

import sqlite3
con = sqlite3.connect(':memory:')
con.execute('CREATE TABLE inbox(source TEXT, event TEXT, PRIMARY KEY(source,event))')
con.execute('CREATE TABLE counter(value INTEGER NOT NULL)')
con.execute('INSERT INTO counter VALUES (0)')
con.commit()

def apply(source, event, crash=False):
    with con:
        cursor = con.execute('INSERT OR IGNORE INTO inbox VALUES (?,?)', (source, event))
        if cursor.rowcount == 0:
            return 'duplicate'
        con.execute('UPDATE counter SET value=value+1')
        if crash:
            raise RuntimeError('Crash model before commit')
    return 'applied'

try:
    apply('provider:tenant-a', 'E42', crash=True)
except RuntimeError:
    pass
assert con.execute('SELECT count(*) FROM inbox').fetchone()[0] == 0
assert con.execute('SELECT value FROM counter').fetchone()[0] == 0
assert apply('provider:tenant-a', 'E42') == 'applied'
assert apply('provider:tenant-a', 'E42') == 'duplicate'
assert con.execute('SELECT value FROM counter').fetchone()[0] == 1
print('Deduplication and business effect rollback together, retry once to take effect passed')

Production asynchronous Inboxes usually have states like pending/processing/done and retry counts; you cannot skip uncompleted tasks just because a row is found. Concurrent workers also need mechanisms like task claiming, leases, or row locks to avoid executing the same message simultaneously.

When calling external deduction, sending SMS, or controlling devices, the other party's side effects are not within the local database transaction. You need to use idempotency keys, operation queries, and compensation supported by the other party; the Outbox can avoid missing commands after local state updates, but it will not make any external HTTP operation automatically exactly-once.

What happens if a crash occurs between two writes? Will it redo or miss?

All three strategies encounter the same crash point, but after recovery, they may result in two effects, zero effects, or one effect. The key is not whether the deduplication table is placed "before" or "after," but whether the two persistent states have a common commit boundary.

Out-of-Order: Decide if Events Represent Snapshots or Changes

If E42 represents order version 8 and E43 represents version 9, when E43 arrives first, the local record already has version 9, and receiving version 8 should not directly overwrite it to the old state. You can use conditional version updates; if the source only provides "resource changed" notifications, query the current state based on the trusted resource ID before updating.

Event timestamps are not always strictly increasing version numbers, and the order of different resources may not be related. The semantics of deletion, cancellation, restoration, and reconstruction should be clearly defined by the business state machine; do not simply process all events as "last receiver wins." Stripe explicitly does not guarantee events are delivered in generation order. Stripe Event Ordering

Outside of Webhooks, repair paths should still be retained: regularly compare with source status or historical events to find gaps and replay. Polling is not necessarily inefficient; low-frequency reconciliation can make up for webhook retry exhaustion, configuration errors, and temporary downtime.

Sender: Event Creation, Backoff, and Address Boundaries

When designing your own sending service, business changes and pending events should be persisted together, and then the sender sends HTTP. A timeout only means no confirmation was seen; the target may have already completed; record the event ID, attempt, response, and next time for each attempt, use bounded backoff with jitter, and avoid simultaneous retries after failure recovery.

Configurable callback URLs are the server's outbound access capabilities, requiring verification of scheme, resolution results, and the actual address after redirection according to the product's allowed target scope, and execution via egress constraints. Checking the string only once at registration is insufficient to handle DNS address changes; callback responses should not be treated as arbitrary new URLs for continued access.

Grafana to Message Bot: Preserve Alert Semantics, Then Convert Cards

Alert notifications may be a group of alerts, including firing and resolved, labels, annotations, and source links. When mapping them to bot cards, you need to decide on aggregation methods, resolution notifications, button targets, and body length, rather than just taking the first array item as the entire alert group. Grafana Notification Templates

If the goal is to point buttons to Loki queries, place controllable log query links in explicit fields, and then encode them into target JSON via templates or adaptation services. Current Grafana documentation supports webhook Custom Payload templates; specific availability depends on the version and deployment; template serialization functions are more reliable than manually splicing quotes. Bots may also require independent tokens, timestamp signatures, or business response codes; do not assume Grafana's HMAC configuration is automatically compatible with another platform. Custom JSON Payload

The original DingDing button behavior belongs to specific integration implementations; after upgrades, check according to the actual version, and do not write "singleURL is always hardcoded" as a protocol nature. If built-in endpoints cannot meet the required URLs or signatures, an adaptation service can complete field mapping, and include non-success business results from the bot API in retry states; HTTP 200 may also contain application-level errors.

When troubleshooting a notification, trace the source creation, HTTP delivery, Inbox submission, worker status, and final external result along the event ID. Looking only at the receiver returning 200 cannot determine if the card the user finally sees is correct.