How to correlate incidents with deploys using evidence

To understand how to correlate incidents with deploys, line up the release’s traffic exposure with the first failing requests, then compare affected and unaffected versions. Put deploy markers beside errors, latency and logs, using the same service and environment identity. Inspect the changed code or configuration for a mechanism before calling the deploy a cause.

This is one part of change intelligence: configuration, flags, infrastructure and dependency changes also belong in the investigation. The process below starts with deployment records you can retrieve today and builds toward a timeline another responder can verify.

1. Define the incident window from telemetry

Start with the user-visible symptom, not the notification time. Find the earliest affected requests and a healthy window before them. Record the service, environment, route, region if available, and whether every request or only a cohort is failing.

Use the four golden signals from Google’s SRE book: latency, traffic, errors and saturation. For a web service, useful views include request counts, failure rates, latency distributions and the resources limiting its work. Google’s guidance also distinguishes successful-request latency from failed-request latency; a fast failure can make an aggregate latency graph look deceptively healthy.

As a starting investigation window, try 15 minutes before the first failure through the present, then widen it if the behavior depends on delayed jobs or accumulated state. That is a working choice, not a provider requirement or universal alert threshold. Check the graph’s aggregation window: a five-minute bucket cannot establish which of two events seconds apart happened first.

Keep a request or trace link for representative failures. Record the query and time bounds, not just a screenshot. An alert may arrive after the failure began, so the last deploy before the page is not necessarily the last deploy before impact.

2. Collect deployment events with the right meaning

GitHub: requests and externally reported statuses

GitHub’s deployment event represents a request to deploy a ref. External tooling performs the deployment and reports statuses. Subscribe to deployment_status for progress; the webhook includes both deployment and deployment_status. GitHub does not emit that webhook for an inactive status.

For a historical investigation, use these documented read endpoints, substituting your owner, repository and deployment ID:

GET /repos/{owner}/{repo}/deployments?environment=production
GET /repos/{owner}/{repo}/deployments/{deployment_id}/statuses

The deployment response includes id, sha, ref, environment, created_at and statuses_url. The status response includes state, created_at, environment, environment_url and log_url. Follow pagination for the incident window. Keep both records: deployment creation time and successful status time are not interchangeable.

A success state is evidence of what the integration reported. Confirm whether your deploy job reports it after upload, readiness checks or traffic cutover. Do not treat a commit push or a green build as a production marker. The GitHub integration page provides context on the repository side of an investigation; the serving platform remains the source for runtime exposure.

Vercel: distinguish readiness from promotion

The Vercel webhook reference documents deployment.created, deployment.ready, deployment.error and deployment.promoted. For production correlation, the last one is especially useful: Vercel says it fires after a production deployment is promoted to start serving production traffic, whether promotion follows a successful build automatically or happens explicitly.

Save the top-level id, type and createdAt, plus payload.deployment.id, the project ID and deployment links. Treat deployment.ready as readiness, not evidence that the production domain has switched. Keep the original payload because field placement differs between event types; for example, the ready event documents payload.target, while the promoted event has no target field in its listed payload.

Also preserve deployment.rollback. Vercel documents it as acceptance of an instant rollback request, possibly before traffic finishes switching. Verify recovery from request telemetry rather than placing “recovered” at that webhook timestamp. For platform context, see monitoring Vercel in production.

Cloudflare Workers: versions and traffic assignments

Cloudflare distinguishes a version from a deployment. A version captures Worker state; a deployment assigns traffic to one version or splits it between two. Record deployment creation time, version IDs and traffic percentages. An uploaded version may never have served production.

From the Worker’s project directory, use the documented Wrangler commands:

npx wrangler deployments list
npx wrangler versions list
npx wrangler tail my-worker --status error --version-id <VERSION_ID>

The list commands expose recent deployment/version history, and the tail filters live invocation logs by status and version. The tail is for current reproduction, not retrieval of old requests. It can sample on busy Workers, so absence of an exception in the stream is not proof of absence.

Compare errors on each serving version and retain the exact exception. Cloudflare defines error 1101 as a Worker throwing a JavaScript exception; use the error 1101 guide to move from the platform error to the code path. Keep changes to storage resources separately: Cloudflare says their state is not tracked by Worker versions.

Kubernetes: the rollout is an interval

For a Deployment called checkout, use the Kubernetes deployment guide’s inspection commands, substituting your namespace and name:

kubectl -n production rollout history deployment/checkout
kubectl -n production rollout status deployment/checkout
kubectl -n production describe deployment checkout
kubectl -n production get rs

History shows revisions; describe shows conditions and events such as ScalingReplicaSet; ReplicaSets help identify the old and new Pod templates. Save these observations during the incident and link them to the image and source revision recorded by your delivery system. A ReplicaSet’s pod-template-hash identifies a Pod template, not a Git SHA.

Only Pod-template changes trigger a deployment rollout. Scaling does not create a revision. Record scaling and configuration edits as separate changes rather than assuming rollout history is exhaustive. Kubernetes reports stalled progress with ProgressDeadlineExceeded; its docs explicitly say it takes no action beyond reporting that condition. A successful rollout likewise needs application-level health checks before you conclude the service recovered.

3. Normalize the records into one change timeline

Use a small common record across sources. This example is an application-defined schema, not a Vercel or GitHub payload:

{
  "source": "vercel",
  "source_event_id": "example-promotion-42",
  "service": "checkout",
  "environment": "production",
  "kind": "deployment_promoted",
  "version": "example-release-b",
  "occurred_at": "2026-10-06T14:04:00Z",
  "observed_at": "2026-10-06T14:04:03Z"
}

Keep source links, commit identity when available and the raw record alongside it. Normalize timestamps to UTC and preserve whether they represent requested, ready, serving or completed states. Deduplicate with source event IDs and retain arrival time so a late webhook does not appear to be a late deploy.

Put markers on the dashboard you already use. Grafana annotations can display descriptions, tags and links on supported panels, and tag filters can select events for the current service. Represent a rolling update as a region annotation when you know its start and end. Filter production events apart from previews.

4. Join deploy markers to metrics and logs

Make your own application logs carry service, environment, release/version, route, outcome and request/trace identity where available. These are recommended fields, not platform-provided guarantees. The release value must describe the code handling that request, rather than the latest release your CI happened to upload.

Compare failure rates with request denominators by version or cohort. A candidate receiving less traffic can have fewer total errors while failing a larger share of requests. Check comparable routes, regions and flag exposure. Use logs and traces to establish whether failures follow the changed path.

The SRE canary chapter explains why per-population breakdowns matter and why before/after evaluation can be misleading. Shared backends can contaminate both candidate and control. If both versions slow down, investigate a common dependency and concurrent changes before concluding the release is irrelevant.

5. Work the hypothesis: a checkout example

The following timeline and measurements are fictional, to illustrate the method. They are not platform benchmarks or alert defaults.

Suppose a comparable five-minute window contains 60 failures among 2,000 checkout requests on B, versus 2 among 2,000 on A. Those fictional counts yield 3% and 0.1% respectively. The comparison strengthens the hypothesis, but it still needs a mechanism.

The failing requests contain a parser exception in a code path changed by B. The diff assumes an upstream field is always present; the saved failing response lacks it. The 14:06 flag cannot explain failures beginning at 14:05, so investigate the release independently and check whether the flag worsened the later period.

After rollback, confirm which release serves the requests and whether the failure rate returns to its normal range. Save the diff, failing example, comparison query and recovery window. If traffic volume also collapsed or another team changed the upstream, recovery has competing explanations that belong in the incident notes.

6. Keep the result reproducible

Finish with a short conclusion: candidate change, exposed population, mechanism, supporting evidence, contradictions and remaining unknowns. Use “correlated” until the mechanism is supported. Do not infer health from missing logs or label every nearby deploy as contributing.

Automate the records that were hard to retrieve and test collection with the next ordinary release. Retain config and flag events alongside deploys. For an exception investigation on another hosting platform, see Vercel FUNCTION_INVOCATION_FAILED. For prevention before shipping, see production controls for AI coding agents. The goal is a timeline that narrows the investigation while leaving room for evidence to disprove the first guess.

Running on Vercel? See how Polylane monitors Vercel in production.