Automating Flaky Test Burn-In with AI (2026)

A test that passes once tells you almost nothing. A test that passes fifty times in a row, under deliberate timing chaos, tells you it is probably deterministic. That gap is where flaky test burn-in lives, and in 2026 the interesting part is not the repetition - it is using AI to read the failures across those runs and tell you why a test is flaky, not just that it is.

This post walks through an automated burn-in pipeline: what burn-in is, how to inject jitter, latency, and CPU pressure to force race conditions into the open, an AI loop that classifies the flake cause across runs, and a CI gate that blocks merge when a newly added test cannot survive the gauntlet. The prize is stopping flakiness at the PR boundary, before it poisons your main branch and trains your team to ignore red.

What is burn-in and why does one green run lie?

Burn-in is running a test repeatedly - typically 20 to 50 times for UI and integration tests - to expose non-determinism before the code reaches your main branch. The premise is simple: flakiness is a probability, not a boolean. A test with a 5% failure rate passes 95% of the time, so a single CI run gives it a green light while it quietly waits to fail in someone else’s PR next week.

The usual culprits behind that 5%:

• Race conditions - the test asserts before the app finishes an async update, and only loses the race occasionally.
• Timeout antipatterns - a hardcoded waitForTimeout(500) that is usually enough, until the machine is busy.
• Test-order dependencies - a test that only passes when another test ran first and left shared state behind.
• External dependencies - a real network call, a clock, or a third-party service that is usually fast.

Burn-in makes the probability visible. Run the test 40 times; if it fails even once, it is not ready. For the diagnosis half of this problem, pair burn-in with AI Root Cause Analysis for CI Test Failures.

How do you run burn-in in CI?

Start with the simplest version: repeat the changed test N times and collect every result. Playwright has a native --repeat-each flag, which is the cleanest entry point.

# .github/workflows/burn-in.yml
name: Flaky Test Burn-In
on:
  pull_request:
    paths:
      - 'tests/**'

jobs:
  burn-in:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Find changed test files
        id: changed
        run: |
          FILES=$(git diff --name-only origin/${{ github.base_ref }}...HEAD \
            -- 'tests/**/*.spec.ts' | tr '\n' ' ')
          echo "files=$FILES" >> "$GITHUB_OUTPUT"

      - name: Install
        run: npm ci && npx playwright install --with-deps chromium

      - name: Burn-in changed tests (30x, JSON report)
        if: steps.changed.outputs.files != ''
        env:
          CHAOS: '1'
        run: |
          npx playwright test ${{ steps.changed.outputs.files }} \
            --repeat-each=30 \
            --workers=4 \
            --reporter=json > burn-in-report.json || true

      - name: Classify failures with AI
        if: steps.changed.outputs.files != ''
        run: node scripts/classify-flakes.mjs burn-in-report.json

      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: burn-in-report
          path: burn-in-report.json

Two details matter. We only burn in changed test files (git diff against the base branch), so cost scales with the PR, not the suite. And we pipe a JSON reporter to a file so the AI step downstream has structured data, not scraped console text. The || true keeps the workflow alive so the classifier always runs, even on failure - the gate decision happens after diagnosis.

How do you inject chaos to force races out?

Clean repetition catches obvious flakiness. Chaos injection catches the subtle stuff by widening the timing windows where races hide. The three cheap, high-value levers:

• Jitter - random delays before actions so ordering assumptions break.
• Network latency - throttle or delay responses so the app is mid-load when the test asserts.
• CPU pressure - saturate cores so timers fire late and hardcoded waits expire.

In Playwright you can wire jitter and latency directly through a fixture and route interception, gated by an environment variable so chaos only runs during burn-in.

// fixtures/chaos.ts
import { test as base } from '@playwright/test';

const CHAOS = process.env.CHAOS === '1';

function jitterMs(): number {
  // Variable delay widens timing windows so latent races surface.
  return 50 + Math.floor(Math.random() * 250);
}

export const test = base.extend({
  page: async ({ page }, use) => {
    if (CHAOS) {
      // Inject variable network latency on every response.
      await page.route('**/*', async (route) => {
        await new Promise((r) => setTimeout(r, jitterMs()));
        await route.continue();
      });
    }
    await use(page);
  },
});

export { expect } from '@playwright/test';

For CPU pressure, run a background load generator during the burn-in job so timers and hardcoded waits are stressed system-wide. Keep it crude and disposable.

// scripts/cpu-pressure.mjs
// Spin a fraction of cores with busy work for the duration of burn-in.
// Launch in the CI step background, kill it when the run finishes.
import os from 'node:os';
import { Worker, isMainThread } from 'node:worker_threads';

if (isMainThread) {
  const cores = Math.max(1, Math.floor(os.cpus().length / 2));
  for (let i = 0; i < cores; i++) new Worker(new URL(import.meta.url));
  console.log(`CPU pressure on ${cores} cores. Kill this process to stop.`);
} else {
  // Never-ending busy loop in each worker thread.
  while (true) Math.sqrt(Math.random() * Math.random());
}

A test that survives 30 clean runs but fails twice under latency plus CPU pressure just told you it has a race condition it would eventually hit in production. That is a catch, not a nuisance. Keep chaos moderate, though - if you inject so much delay that the app genuinely times out, you are testing your chaos config, not the code.

How does the AI classify the flake cause?

This is the step that turns burn-in from a dumb gate into a diagnosis. After N runs you have a JSON report with a mix of passes and failures. Instead of a human reading 40 logs, an LLM reads the distinct failure signatures across runs and classifies each into a cause category with a suggested fix.

// scripts/classify-flakes.mjs
import { readFile } from 'node:fs/promises';
import { callModel } from './llm-client.mjs';

const CATEGORIES = ['race', 'timeout', 'test-order', 'external', 'genuine-bug'];

function extractFailures(report) {
  const failures = [];
  for (const suite of report.suites ?? []) {
    for (const spec of walkSpecs(suite)) {
      const runs = spec.tests?.[0]?.results ?? [];
      const failed = runs.filter((r) => r.status !== 'passed');
      if (failed.length > 0) {
        failures.push({
          title: spec.title,
          totalRuns: runs.length,
          failedRuns: failed.length,
          // Dedupe near-identical errors to keep the prompt small.
          signatures: dedupe(failed.map((r) => trimError(r.error?.message ?? ''))),
        });
      }
    }
  }
  return failures;
}

const SYSTEM = `You triage flaky test failures from a burn-in run.
For each test you receive: title, how many of N runs failed, and the distinct
error signatures. Classify the ROOT CAUSE into exactly one of:
- race: assertion fires before an async update completes
- timeout: hardcoded wait / element not ready in time
- test-order: passes only depending on state left by another test
- external: real network, clock, or third-party dependency
- genuine-bug: the code is actually broken, not flaky
Return JSON: [{ title, cause, confidence, evidence, suggestedFix }].
Base "cause" on the signatures and the fail ratio. A low, intermittent fail
ratio with timing-related errors points to race; a consistent failure points
to genuine-bug. Output JSON only.`;

async function main() {
  const report = JSON.parse(await readFile(process.argv[2], 'utf8'));
  const failures = extractFailures(report);

  if (failures.length === 0) {
    console.log('Burn-in clean. No flakes to classify.');
    process.exit(0);
  }

  const verdict = JSON.parse(
    await callModel({ system: SYSTEM, user: JSON.stringify(failures, null, 2) }),
  );

  console.log('\n=== BURN-IN FLAKE CLASSIFICATION ===');
  for (const v of verdict) {
    console.log(`\n[${v.cause.toUpperCase()}] (${v.confidence}) ${v.title}`);
    console.log(`  evidence: ${v.evidence}`);
    console.log(`  suggested fix: ${v.suggestedFix}`);
  }

  // Any flaky NEW test fails the gate. See "genuine-bug" as an even harder no.
  process.exit(1);
}

// walkSpecs, dedupe, trimError omitted for brevity.
main();

A few things keep this honest and affordable. We dedupe error signatures before sending them so 30 identical stack traces become one, which slims the prompt - a pattern worth extending suite-wide, see Token-Efficient AI Testing in CI/CD. We ask for a confidence field so low-confidence classifications get human eyes. And crucially, the classification is a lead for triage, not an autofix - the model tells you where to look; an engineer confirms and fixes.

The CI gate: block merge on flaky new tests

The gate rule is blunt on purpose: if a newly added or changed test fails any burn-in run, the PR cannot merge. The classify-flakes script exits non-zero on any flake, which fails the job, which blocks the merge through branch protection.

That last row is the important nuance. Do not point this gate at your entire legacy suite on day one, or every PR turns red on flakes nobody in the PR introduced. Quarantine known flakes into a separate, non-blocking lane and pay them down deliberately, while the gate stays green for genuinely stable new work. The gate’s job is to stop the bleeding - new flakiness - not to boil the ocean.

Where does this fit with the rest of your AI testing?

Burn-in is the enforcement layer; root-cause analysis is the diagnosis layer. Together they close the loop: burn-in catches the flake at the PR boundary, the classifier says why, and an engineer fixes it before it ever merges. For the deeper diagnostic side see AI Root Cause Analysis for CI Test Failures, and for the broader agentic picture see the pillar, Building a Self-Operating QA Agent with Playwright MCP. If you are new to the whole space, start with AI-Augmented Test Automation: The 2026 Guide.

The takeaway

Flakiness is cheapest to kill at the moment it is born - in the PR that introduces it. Burn-in plus chaos injection forces the non-determinism into the open where a single CI run would have missed it, and an AI classification loop turns 40 confusing logs into one actionable verdict. Gate the merge on new tests, quarantine the legacy backlog, and treat the AI’s cause label as a well-informed lead rather than gospel. Do that and your main branch stops being the place where flaky tests go to hide. The exact run counts and chaos levels are yours to tune - [verify] against your own flake budget and CI minutes before you turn the gate on hard.