PR impact analysis
How BugBrain analyzes a pull request — a diff analyst maps changed paths to impacted user-facing areas, an ImpactSignal behavior graph sharpens selection over time, and advisory-first gating only fails a required check on an introduced high-severity issue.
When a pull request opens, the useful question isn't "does the whole app still work?" — it's "did this change break anything a user would notice?" BugBrain answers that by analyzing the diff, testing the areas the change actually touches, and reporting back as a GitHub Check. The design is careful and conservative: it focuses effort where the risk is, and it never blocks a merge over a bug that was already there.
Behavior-layer test impact analysis#
The first step is figuring out what the change means for users. A diff analyst reads the pull request's changes and performs behavior-layer Test Impact Analysis: instead of just noting which files changed, it reasons about which user-facing areas and flows those changes affect.
From that impacted set, BugBrain selects the test cases that exercise those areas and runs a focused test against the PR's preview or staging deployment. A change to the checkout code triggers checkout-area tests; a change to a profile page triggers profile tests. You get a relevant, fast signal instead of a full-suite run on every commit.
The ImpactSignal behavior graph: getting smarter with use#
Mapping code to behavior is hard to get perfectly right from static analysis alone, so BugBrain learns from history. An ImpactSignal behavior graph records the co-occurrence between changed-path prefixes and the user-facing areas that turned out to be affected, along with how often changes to those paths led to regressions.
On future pull requests, BugBrain uses this graph to expand the selected areas with the ones that a given kind of change has historically tended to break. In other words, the more pull requests flow through, the better BugBrain gets at predicting the blast radius of a change. Selection sharpens with usage — this accumulated behavior graph is the part that's hard for a generic tool to replicate.
Advisory-first gating#
A pre-merge check that fails the build over flaky or pre-existing problems quickly gets ignored. BugBrain avoids that with advisory-first gating:
- By default the GitHub Check is advisory (neutral). It tells you what it found without blocking the merge.
- A check only fails a required gate when all of these are true: the repository is set to required gating, the issue is introduced — meaning its signature wasn't seen in the project's baseline runs (it's genuinely new, not pre-existing) — the issue is high or critical severity, and it's oracle-confirmed (the judgment layer is confident it's real).
So a long-standing bug never fails your PR, a low-severity nit never fails your PR, and a run that itself failed or only partially completed never gates. The only thing that can block a merge is a new, serious, confirmed regression that this change introduced — exactly the case you'd want to catch before merging.
New, serious, and confirmed — all three
The gate is intentionally narrow. "Introduced" rules out pre-existing bugs, the severity bar rules out nits, and oracle confirmation rules out noise. That's what keeps a required check trustworthy instead of something teams learn to override.
Related#
Frequently asked questions
How does BugBrain decide what to test on a pull request?
A diff analyst reads the PR's changes and performs behavior-layer Test Impact Analysis — it maps the changed code paths to the user-facing areas they affect and selects the test cases that exercise those areas, rather than running everything.
Will a pre-existing bug fail my pull request?
No. Gating is advisory-first. A required check only fails on an introduced issue — one with a new signature not seen in the project's baseline — that is high or critical severity and oracle-confirmed. Bugs that already existed never block the PR.
What is the ImpactSignal behavior graph?
It records which changed code paths historically co-occur with regressions in which user-facing areas. Over time this lets BugBrain expand its test selection toward the areas a given kind of change tends to break — so selection sharpens with usage.
Is the check blocking by default?
No. By default the check is advisory (neutral). It only becomes a blocking gate when a repository is explicitly set to required gating, and even then only an introduced, confirmed high-severity issue fails it.