Automated Root Cause Analysis for CI Test Failures with AI (2026)

Every SDET knows the CI morning ritual: twelve tests went red overnight, and now someone has to open each one, squint at the stack trace, replay the trace, and decide whether it is a real regression, a locator that broke because marketing reshuffled the header, or the payment sandbox timing out again. Most of that triage is pattern-matching that a model can do in seconds. This post shows how to build automated root cause analysis that feeds a Playwright trace.zip and console logs into an LLM, classifies each failure into a clear taxonomy, and annotates the pull request - while keeping humans firmly in charge of what the labels mean.

This builds on our pillar guide, Building a Self-Operating QA Agent with Playwright MCP, and pairs naturally with AI Flaky Test Burn-In and Race Condition Detection for the flakiness side of the problem.

What does a Playwright trace actually contain?

Before you can classify a failure, you need to understand the richest artifact you already have. A Playwright trace is a trace.zip file - a self-contained recording you normally open in Trace Viewer, but which is also a structured input for a model.

Inside a trace you get:

• Actions - the ordered timeline of every step the test took, with timing, and exactly which action failed.
• DOM snapshots - a before-and-after capture of the page for each action, so you can see the state the test acted on and the state it produced.
• Network - every request and response, including status codes, so you can spot a 500 or a timed-out call.
• Console - all browser console output, including uncaught JS errors.
• Screenshots - visual frames of the run.

That is almost exactly the evidence a human uses to diagnose a failure. The whole strategy is to extract and compress this into a form a model can reason over cheaply.

How do you extract and summarize a trace for a model?

You do not send the raw trace.zip to an LLM - it is a binary archive, and even unzipped it is far too large and noisy to be token-efficient. You summarize it into a compact, high-signal text digest first.

Playwright’s trace is a zip of JSONL event files plus resources. A small extraction script pulls out the parts that matter: the failing action and its error, the last DOM snapshot around that action, any non-200 network responses, and console errors. Everything else - successful actions, image resources, redundant snapshots - is dropped.

// summarize-trace.ts - reduce a trace.zip to a high-signal digest
import AdmZip from 'adm-zip';

export function summarizeTrace(tracePath: string) {
  const zip = new AdmZip(tracePath);
  const events = zip.getEntries()
    .filter((e) => e.entryName.endsWith('.trace') || e.entryName.endsWith('.jsonl'))
    .flatMap((e) => e.getData().toString('utf8').trim().split('\n'))
    .map((line) => JSON.parse(line));

  const actions = events.filter((e) => e.type === 'action');
  const failed = actions.find((a) => a.error);
  const consoleErrors = events
    .filter((e) => e.type === 'console' && e.messageType === 'error')
    .map((e) => e.text);
  const badNetwork = events
    .filter((e) => e.type === 'resource' && e.status && e.status >= 400)
    .map((e) => `${e.status} ${e.method} ${e.url}`);

  return {
    failingAction: failed?.apiName ?? 'unknown',
    error: failed?.error?.message ?? 'no error captured',
    lastSelector: failed?.selector ?? null,
    consoleErrors: consoleErrors.slice(0, 10),
    badNetwork: badNetwork.slice(0, 10),
    domExcerpt: (failed?.snapshot?.html ?? '').slice(0, 2000),
  };
}

[verify exact trace event field names against your Playwright version - the schema is internal and can change between releases.] The principle holds regardless: keep the failing action, the error, the surrounding DOM, network errors, and console errors; discard the rest. This is also where token discipline pays off - see Token-Efficient AI Testing in CI/CD.

What taxonomy should you classify failures into?

A freeform “why did this fail?” answer is useless for automation. Give the model a fixed taxonomy and force it to pick one label, cite evidence, and score its confidence.

Four categories cover the vast majority of CI failures, and each maps to a different owner: a bug goes to developers, UI drift goes to whoever owns locators (or a self-healing step), infrastructure goes to the platform team, and a test defect goes to the SDET. That routing is the entire point of the classification.

Ask for structured output:

You are triaging a failed Playwright test. Using ONLY the evidence below,
classify the failure into exactly one label:
  actual_bug | ui_drift | flaky_infrastructure | test_defect

Return JSON: { "label": ..., "confidence": 0.0-1.0, "evidence": "...",
"suggested_owner": "..." }

Rules:
- Cite specific evidence (the error, a status code, a console line).
- If evidence is ambiguous or thin, lower confidence below 0.6.
- Do not guess a bug when a network timeout better explains it.

EVIDENCE:
<the trace digest + test name + retry status>

The confidence score is what keeps this safe: anything below a threshold gets routed to a human instead of trusted.

How do you wire it into CI?

The classifier runs as a separate, non-blocking, on-failure step. It never gates the build - a red test stays red - it only adds context.

First, make Playwright emit a trace on failure:

// playwright.config.ts
export default defineConfig({
  retries: 2,                    // retry status is itself a flakiness signal
  use: { trace: 'retain-on-failure' },
});

Then add a CI job that runs only when the test job fails, collects the traces, classifies them, and posts the results to the PR. In GitHub Actions:

# .github/workflows/triage.yml (excerpt)
triage:
  needs: test
  if: failure()                  # only run when tests failed
  runs-on: ubuntu-latest
  continue-on-error: true        # never let triage break the pipeline
  steps:
    - uses: actions/checkout@v4
    - uses: actions/download-artifact@v4
      with: { name: playwright-traces, path: traces/ }
    - run: node scripts/triage.js traces/   # summarize -> classify -> write report.md
    - uses: actions/github-script@v7
      with:
        script: |
          const body = require('fs').readFileSync('report.md', 'utf8');
          await github.rest.issues.createComment({
            ...context.repo, issue_number: context.issue.number, body,
          });

The continue-on-error: true line is not optional. If the model API is down or over quota, triage must degrade to “no annotation,” never to “broken build.” The failures are still visible in the normal test output; the AI comment is a bonus layer of context.

The PR comment ends up looking like a triage table an engineer can scan in seconds:

How do you keep humans in the loop?

This is the part you do not compromise on. The classifier triages; it never adjudicates. Three guardrails keep it honest:

1. It cannot change build status. A failing test fails the build regardless of what the model labels it. There is no “AI said flaky, so we auto-pass” path - that is how real bugs ship.
2. Low confidence routes to a human. Anything under your threshold (start around 0.6 and tune) is labeled needs_review rather than given a category, so thin evidence never masquerades as a diagnosis.
3. Labels are auditable. Every classification cites specific evidence and links to the stored trace.zip, so an engineer can open Trace Viewer and check the model’s reasoning in one click. When the model is wrong, you see exactly why.

Over time you track the classifier’s accuracy against what engineers actually concluded, and use that to tune the prompt and threshold. A model that is right 85% of the time on triage is enormously useful even though you would never let it touch the build gate. For the specific case of separating true flakiness from race conditions, hand the flaky_infrastructure bucket to the burn-in workflow in AI Flaky Test Burn-In and Race Condition Detection, and for the broader agent architecture this fits into, see The Tri-Agent Testing Pipeline.

The bottom line

The trace.zip you already generate on failure is a near-perfect diagnostic record - actions, DOM, network, and console in one file. Automated root cause analysis just puts a model in front of it: summarize the trace, classify against a fixed taxonomy with a confidence score, and annotate the PR with evidence-backed labels routed to the right owner. Wire it as a non-blocking on-failure step so it can never break your pipeline, keep every decision auditable, and never let the AI touch the build gate. Done that way, it turns the CI morning ritual from an hour of manual triage into a two-minute scan of a labeled table.

If you want a pipeline that captures traces and triages failures out of the box, that is exactly what our CI/CD Test Infrastructure and Test Automation Framework Engineering teams build.