Get started Dashboard
Continuous integration ·
Part 1 of Shipping safely: deploys, config and change risk

How to fix a broken build on main

Explore with AI

To fix a broken build on main, read the failing job's log first, then pick one of three moves: rerun only if the failure is unrelated to the code, revert the breaking commit if the cause is not obvious within minutes, or push a small fix if it is. Confirm the next run on main passes, then add the protection that would have caught it.

On this page

To fix a broken build on main, read the failing job’s log, then rerun, revert or fix forward depending on what it shows. Find the commit that broke the build, confirm the next run on main is green, and add the branch protection that would have stopped it.

A red main blocks everyone. Nobody can trust a merge, the next deploy waits, and every change merged on top makes the breaking one harder to isolate. Martin Fowler’s description of continuous integration puts it plainly in a practice called Fix Broken Builds Immediately: getting main green again comes before new work. The steps below use GitHub Actions and the GitHub CLI, but the order works on any CI.

1. Make one person own it

Say in the team channel that main is red and who is on it. Two people fixing the same failure in parallel produce two conflicting fixes, and nobody fixing it because everyone assumed someone else was is the more common failure.

Until main is green, hold merges that are not the fix. Every extra commit widens the range you have to search and can hide a second failure behind the first.

2. Read the failing job’s log

The red cross only tells you that a check failed. Find the run and read only the failed steps:

gh run list --branch main --status failure --limit 5
gh run view <run-id> --log-failed

gh run view --log-failed prints the log for the failed steps only, which is usually the twenty lines that matter out of thousands. Start from the first error: later errors are often consequences of it.

Sort what you find into two kinds:

  • The code is wrong. A compile error, a failing assertion, a type error, a lint rule, a missing import. These fail again on every run.
  • The run was unlucky. A runner that lost its network, a package registry that timed out, a job that was cancelled or ran out of disk. These are about the machine.

3. Choose: rerun, revert or fix forward

Rerun only when the failure is clearly outside your code. GitHub re-runs use the same commit SHA, so a real bug fails again and a rerun just costs time. Re-run only the failed jobs, with debug logging if the cause is unclear:

gh run rerun <run-id> --failed --debug

If a test passes on rerun, note it. A test that fails and passes on the same commit is flaky, and a flaky test on main teaches the team to ignore red builds.

Revert when the cause is not obvious. If you cannot name the cause within a few minutes, revert the breaking commit. A revert returns main to code that already passed, which is the fastest safe way back to green, and the author can fix the change on a branch without the whole team waiting. Fowler’s article recommends the same: when the cause is not obvious, take main back to the last known good build.

git revert <breaking-sha>

Fix forward when the cause is clear and the fix is small. A missing import, a renamed function one caller still uses, a snapshot that needs updating: push the fix through a normal pull request so CI runs on it before it reaches main.

4. Find the commit that broke it

The breaking commit sits between the last green run on main and the first red one. List the runs with their commits:

gh run list --branch main --limit 20 --json headSha,conclusion,createdAt
git log --oneline <last-green-sha>..<first-red-sha>

If that range holds one commit, you have it. If it holds several, let git bisect test them for you with the command that fails:

git bisect start <first-red-sha> <last-green-sha>
git bisect run npm test
git bisect reset

git bisect run treats exit code 0 as good, 125 as “skip this commit”, and any other code from 1 to 127 as bad. Use the narrowest command that reproduces the failure: running one test file is faster and more reliable than the whole suite.

If every commit in the range passes on its own, the build broke because two changes met. That is the case step 6 prevents.

5. Confirm main is green

Watch the run on main after your fix or revert lands, and treat the job as done only when it passes:

gh run watch <run-id> --exit-status

Then tell the team main is green and merges can resume. If you reverted, leave a note on the original pull request with the failing log so the author starts from the evidence.

6. Stop the next one

Most broken builds on main come from changes that were tested against an older main. Three settings close that gap.

  • Required status checks. In branch protection, require the CI checks that matter, so nothing merges on red.
  • Up-to-date branches. Requiring branches to be up to date before merging makes each pull request run against the current main. It is safe but slow on a busy repository, because every merge sends the others back to rebase.
  • A merge queue. A merge queue tests each pull request together with the ones ahead of it before anything reaches main. If a queued pull request fails its checks, it is removed from the queue and the rest continue. Workflows must listen for the merge_group event or the queue waits forever:
on:
  pull_request:
  merge_group:

With a merge queue in place, two pull requests that are green alone but red together fail in the queue instead of on main.

Where Polylane fits

Polylane detects issues and fixes them by opening pull requests. When the build on main breaks, it works out why, and the fix arrives ready for your team’s normal review.

See Polylane for Continuous Integration.

Common questions.

Should I rerun a failed build on main?

Only when the log shows a failure outside your code: a runner that lost its network, a registry that timed out, a cancelled job. A rerun uses the same commit, so a real bug fails again. Rerunning until it passes hides a flaky test instead of fixing it.

Is it better to revert or fix forward?

Revert when the cause is not obvious or the fix is more than a few lines. A revert returns main to a state that already passed, which is the fastest safe green. Fix forward when the cause is clear and the change is small enough to review in minutes.

How do I find which commit broke the build?

Compare the last green run on main with the first red one: the breaking commit is in that range. If the range holds several commits, run git bisect with the failing command to test each one automatically.

Why did main break when every pull request passed?

Each pull request was tested against an older main. Two changes can pass alone and fail together. Requiring branches to be up to date before merging, or using a merge queue, tests each change against what main will actually become.

How long can I rerun a GitHub Actions workflow?

GitHub lets you re-run a workflow up to 30 days after its initial run. The re-run uses the same commit SHA as the original, so it tests exactly the same code.

Can a tool fix the build on main for me?

Polylane can. When the build on your main branch breaks, it works out why and opens a pull request with the fix, ready for your team's normal review.

Sources (7)

  1. Continuous Integration, Fix Broken Builds Immediately (Martin Fowler)
  2. Re-running workflows and jobs (GitHub Docs)
  3. gh run view (GitHub CLI manual)
  4. gh run list (GitHub CLI manual)
  5. git-bisect (Git documentation)
  6. About protected branches (GitHub Docs)
  7. Managing a merge queue (GitHub Docs)

About the author

Boris Tane

Founder of Polylane

Boris Tane is the founder of Polylane. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.

Related

Nobody should be on-call. Polylane watches your infra, finds what broke, and writes the fix.

Get started for free