Webhook failures are rarely caused by one variable alone. A request may contain the wrong signature input, arrive after a retry, receive an unexpected response code, or be processed twice because the handler has no durable event record. This guide provides a repeatable webhook debugging workflow: capture the complete request, verify authentication, trace delivery and retries, compare payloads, and make processing idempotent. It also shows what to monitor over time so an integration remains diagnosable after deployment.
Overview
A webhook is an HTTP request sent by one system to notify another system that an event occurred. The receiving application usually has to do four things correctly:
- Accept the request at the expected URL and method.
- Authenticate the sender using the provider's documented mechanism.
- Parse and validate the payload without changing data needed for verification.
- Record and process the event safely, including retries and duplicate deliveries.
When debugging, avoid starting with application business logic. First establish whether the request reached the intended endpoint, whether the raw request was preserved, what response the endpoint returned, and whether the provider attempted delivery again. A small delivery log can turn an ambiguous “webhook failed” report into a specific diagnosis.
Use a temporary inspection endpoint or a protected request-capture tool in a non-production environment when you need to see headers, query parameters, the raw body, and timing. Do not expose secrets or retain unrestricted production payloads. Redact authorization values, signing secrets, personal data, and tokens before sharing logs.
What to track
Capture the complete request
Record enough information to compare a successful and unsuccessful delivery. Useful fields include the received timestamp in UTC, HTTP method, route, host, request ID, provider event ID, content type, selected headers, body length, and the response status and duration. Store the raw body separately from the parsed object when signature verification depends on the exact bytes received.
Headers are often as important as the JSON payload. Look for the provider's event identifier, timestamp, signature, delivery attempt marker, and content digest if supplied. Header names and signing formats differ, so treat provider documentation as the source of truth rather than assuming that every webhook uses the same convention.
Check signature verification in the correct order
A common failure occurs when middleware parses or reformats JSON before the signature is checked. Many schemes sign a string made from the timestamp and raw request body. Re-serializing the body can change whitespace, escaping, or key order and produce a different input.
A provider-agnostic verification sequence looks like this:
- Read the raw request bytes or exact raw text.
- Extract the signature and any signed timestamp from the documented headers.
- Construct the signed message exactly as specified.
- Compute the expected digest with the configured secret.
- Compare values using a constant-time comparison where supported.
- Reject stale, malformed, or unverifiable requests before business processing.
Keep secret values out of logs. If verification fails, log a safe reason such as “missing signature,” “timestamp outside accepted window,” or “digest mismatch,” not the secret or the complete authorization header. If a provider supports secret rotation, determine whether the handler must temporarily try an active and a previous secret, and make that behavior explicit.
Trace response codes, timeouts, and retries
For each delivery attempt, connect the provider's event ID to your server's request ID. Track the response status, response body classification, processing duration, and whether the request reached application code. A connection failure, timeout, client error, and server error point to different parts of the system.
- No request record: investigate DNS, routing, firewall rules, TLS, gateway configuration, and the configured webhook URL.
- Authentication failure: inspect the raw body, secret selection, timestamp handling, and clock differences.
- Client error: compare the request against the endpoint contract and check required headers or fields.
- Server error: inspect application logs, dependency failures, database constraints, and unhandled exceptions.
- Timeout: move slow work out of the request path and acknowledge only after the event has been safely accepted.
Do not assume that a successful HTTP response means the entire downstream workflow completed. It may only mean that the event was accepted into a queue or durable inbox. Define that boundary clearly in logs and operational documentation.
Design for duplicate events
Retries are normal in distributed systems, and a delivery can be repeated even when the first attempt eventually succeeds. The handler should therefore be idempotent: processing the same event more than once should not create an unintended second charge, shipment, account change, or notification.
Use a stable provider event ID, or another documented unique event key, as an idempotency key. A durable event table can track values such as event_id, event_type, received_at, status, attempt_count, and processed_at. Enforce uniqueness at the database layer where possible. The typical flow is:
- Verify the request.
- Insert the event into an inbox or event table with a unique key.
- If the key already exists, return the safe duplicate outcome without repeating side effects.
- Process the recorded event through a worker or controlled transaction.
- Mark the event completed, or retain an error state for retry and investigation.
Cadence and checkpoints
Webhook debugging is not only an incident activity. Review the integration on a monthly or quarterly cadence, depending on its importance and change frequency. The goal is to detect drift before the next provider or application change turns it into an outage.
| Checkpoint | What to review | Useful evidence |
|---|---|---|
| Delivery health | Accepted, rejected, timed-out, and failed deliveries | Provider delivery history and server request logs |
| Latency | Time to receive, acknowledge, queue, and complete events | Request timing and worker metrics |
| Authentication | Signature failures, secret rotation readiness, and clock handling | Redacted verification outcomes |
| Retries | Repeated event IDs and events approaching retry limits | Event table and provider attempt data |
| Payload compatibility | Unknown fields, missing fields, and event-type coverage | Schema validation and sampled payloads |
| Recovery | Replay, quarantine, and manual remediation procedures | Runbook and non-production test results |
Use a small event-state model so developers and operators describe the same situation. For example: received means the request passed transport checks; verified means authentication passed; queued means durable acceptance completed; processing means a worker is acting on it; completed means the intended side effects succeeded; and failed means investigation or a controlled retry is required. Keep rejected authentication attempts separate from verified events that later failed in application processing.
How to interpret changes
Look for patterns rather than isolated failures. A sudden increase in signature mismatches suggests a secret, raw-body, timestamp, or deployment issue. A rise in duplicate events with stable processing times may indicate provider retries caused by response handling or an acknowledgement path that is too slow. A rise in processing time with unchanged delivery volume can point to a database, queue, or third-party dependency.
Compare event types separately. A global success rate can hide one failing event type or one customer configuration. Segment logs by endpoint, environment, event type, response status, and delivery attempt. Also compare the provider's view with your application's view: if the provider reports repeated attempts but the application records only one request, investigate the network or gateway layer; if the application records all attempts, inspect idempotency and response behavior.
When testing a fix, use a known event and record its expected state transitions. Confirm that an invalid signature cannot create a durable business side effect, that a duplicate event does not repeat the side effect, and that a worker failure leaves enough information for safe replay. Test both a fast successful path and a slow or unavailable dependency path.
When to revisit
Revisit this guide monthly or quarterly as part of integration maintenance, and immediately after a webhook provider changes its event format, signature scheme, retry behavior, endpoint configuration, or delivery terminology. Recheck it after changing request middleware, API gateways, queues, database transactions, secrets, deployment regions, or downstream services. A routine review is also warranted after an incident involving missing, late, rejected, or duplicate events.
Make the review practical:
- Choose several recent successful events and one failed or retried event.
- Trace each from provider delivery ID to request log, event record, worker job, and final business state.
- Confirm that sensitive headers and payload fields are redacted in operational logs.
- Run a non-production test for invalid signatures, duplicate delivery, malformed payloads, and slow downstream processing.
- Update the runbook with the current endpoint, event states, replay procedure, owner, and escalation path.
- Record the review date and the next checkpoint so the integration does not become an undocumented dependency.
For related API troubleshooting, use a JSON formatter and validator to inspect safe payload copies, and consult the JSON and API debugging toolkit when comparing or transforming request data. The essential habit is simple: preserve the original evidence, separate authentication from processing, track every delivery attempt, and make repeated events safe by design.
