BugBrainDocs
bugbrain.techOpen app

PR quality gate

How to set up a PR quality gate in BugBrain — install the GitHub App, link a repo, and have BugBrain analyze each pull request's diff, test the impacted areas against the preview URL, and post an advisory GitHub Check. A metered add-on.

PR automation is a pre-merge quality gate: when a pull request opens, BugBrain analyzes the change, tests the parts of your app it touches, and posts the result back to GitHub as a Check and a comment. This guide covers setting it up and how the gate decides. It is a metered add-on, so it must be enabled for your workspace first.

What it is#

PR automation connects to GitHub through the BugBrain GitHub App. On each pull request it:

  1. Analyzes the diff with Test Impact Analysis to work out which user-facing areas the change affects.
  2. Selects the impacted test cases — the existing cases that cover those areas, plus any flows you've marked as protected.
  3. Runs a focused exploratory test against the pull request's preview or staging URL.
  4. Posts an advisory GitHub Check and a comment summarizing what it found.

It's GitHub-only today.

Why use it#

  • Catch regressions before merge — problems surface on the PR, where they're cheapest to fix.
  • Focused, not exhaustive — it tests what the diff touched, so checks stay fast and relevant.
  • Advisory by default — it informs reviewers without blocking, and only gates when you explicitly turn that on.

Before you start#

PR automation is a metered, feature-flag-gated add-on. Before it works for your workspace:

  • A super-admin must turn on the pr-automation feature flag and grant a monthly PR-review quota above zero. If the flag is off or the quota is 0, the project's PR Automation tab shows a "not enabled" notice with an upgrade or contact path.
  • You need admin or owner access in the workspace to install the GitHub App and link a repository.
  • You need a project whose preview or staging URL the pull request can be tested against.

One review per PR head, metered

Each pull request analysis counts one PR review against your monthly quota. BugBrain reserves a review per commit so re-running checks on the same change doesn't double-spend your quota.

Set it up#

  1. Install the BugBrain GitHub App

    Install the BugBrain GitHub App on the GitHub organization or account that owns your repository.

  2. Enter the installation id

    In the project's PR Automation tab, enter the installation id from the GitHub App install.

  3. Link a repository

    Link the repo and set its options: the protected flows you always want exercised, and the gating mode — advisory or required.
The PR Automation project tab
Linking a repo: protected flows and the advisory-or-required gating mode.

How the gate decides#

PR automation is advisory-first. By default the Check is informational and never fails a build. It can fail a pull request only when all of these are true:

  • The repo link's gating mode is set to required, and
  • BugBrain finds an introduced issue — one whose signature wasn't seen in the project's baseline runs (a problem this PR appears to have added), and
  • that issue is high or critical severity, and
  • it's oracle-confirmed (BugBrain's verdict checker is confident it's real).

A pre-existing bug never fails a PR — only newly introduced problems can gate. A run that itself failed or only partly completed never gates either.

Advisory until you choose otherwise

Leave gating on advisory to build trust — reviewers see the Check and comment without it blocking merges. Switch a repo to required once you're confident BugBrain's introduced-issue calls match your bar.

Tips#

  • Set protected flows for the journeys you never want broken (checkout, login, signup) so they're always exercised regardless of what the diff touched.
  • Keep the preview/staging URL stable and representative — the test runs against it, so a broken preview produces a failed run, not a useful gate.
  • Marking false positives in issues keeps the baseline honest, which sharpens what counts as an introduced issue for gating.
PR impact analysisHow BugBrain decides what a diff affects.API keysProgrammatic access for CI and the GitHub on-ramp.Webhook eventsThe pr.check.completed event and others you can subscribe to.

Frequently asked questions

Will PR automation block my pull requests?

Only if you ask it to. It's advisory-first — the Check is informational by default. It can fail a PR only when the repo link's gating mode is set to required and BugBrain finds an introduced, high-or-critical, oracle-confirmed issue. A pre-existing bug never fails a PR.

What's an "introduced" issue?

An issue with a signature (a fingerprint) that hasn't been seen in the project's baseline runs — in other words, a problem the pull request appears to have added, not one that was already there.

Which source hosts are supported?

GitHub only today, via the BugBrain GitHub App. Other hosts aren't supported yet.

How does BugBrain know what to test on a PR?

It analyzes the diff with Test Impact Analysis to work out which user-facing areas changed, selects the existing test cases that cover those areas, and runs a focused exploratory test against the PR's preview or staging URL.

Why do I see a "not enabled" notice?

PR automation is a metered add-on. It needs the pr-automation feature flag on and a monthly PR-review quota above zero. If either is missing, the page shows a "not enabled" notice with an upgrade or contact path.