Autonomous Exploratory Testing with Multimodal LLMs (2026)

Your scripted suite is green and your staging site is still broken. That is not a contradiction, it is the whole problem with deterministic tests: they only check what you told them to check. Everything else - the dead link three levels deep in the footer, the modal that renders half off-screen on a narrow viewport, the uncaught exception firing silently on page load - sails through untouched because you never wrote an assertion for it. Autonomous exploratory testing with a multimodal LLM is how you go looking for those bugs without scripting each one in advance. This post covers the agent loop, what it catches, how to keep it on a leash, and the limits you should be honest about.

This is a companion to our pillar, Building a Self-Operating QA Agent with Playwright MCP. If you want the deterministic counterpart, see how confirmed findings become specs in From User Stories to Playwright TypeScript Specs with AI.

What is autonomous exploratory testing with a multimodal LLM?

Traditional automation is a recording of decisions you already made: go here, click this, assert that. Autonomous exploratory testing hands the decisions to the agent. You point a multimodal LLM at a running app, give it a goal (“explore the checkout flow and report anything broken”), and it chooses each next action itself based on what it sees.

The word multimodal is load-bearing. The agent does not just read text; it consumes two views of every page at once - a structured accessibility snapshot (the ARIA tree of roles, names, and states) and a screenshot (the rendered pixels). The snapshot tells it what the page is; the screenshot tells it what the page looks like. Together they let the agent reason about a page the way a human tester does on their first pass through an unfamiliar build.

This is not a replacement for your assertions. It is the exploratory half of testing - the part that has always relied on a skilled human clicking around - finally made scalable.

How does the agent loop work?

Every autonomous run is the same four-step loop repeated until a budget runs out.

1. Navigate. The agent takes an action - open a URL, click a link, fill a field, submit a form.
2. Observe. It captures the new state: the accessibility snapshot, a screenshot, and any console or network events that fired.
3. Reason. The LLM compares what it sees against what it expected. Is this page blank? Did an error fire? Does the layout look wrong? Where should it go next to widen coverage?
4. Report. It logs anything suspicious as a finding, then loops back to navigate.

The observe step is where Playwright MCP earns its place. Rather than gluing screenshots and DOM dumps together yourself, the MCP server exposes a clean browser_snapshot (accessibility tree) and browser_take_screenshot to the agent as tools, alongside navigation and console-reading tools. The agent calls them in a loop and you supply the orchestration and guardrails.

import { chromium, type Page, type ConsoleMessage } from '@playwright/test';

type Finding = {
  route: string;
  kind: 'dead-link' | 'console-error' | 'http-error' | 'layout' | 'blank-page';
  detail: string;
};

async function observe(page: Page): Promise<{ snapshot: string; screenshot: Buffer }> {
  // The accessibility snapshot is the compact, stable view of the page.
  const snapshot = await page.accessibility.snapshot().then((s) => JSON.stringify(s));
  const screenshot = await page.screenshot({ fullPage: true });
  return { snapshot, screenshot };
}

In a Playwright MCP setup the observe step above is replaced by tool calls the agent makes directly, which keeps the browser state on the server and the reasoning in the model.

What does it catch that scripted tests miss?

This is the entire value proposition, so it is worth being specific. Scripted tests defend the known knowns. The agent hunts the unknown unknowns - the failures you would only find by actually looking.

The pattern: anything that requires judgment rather than a pre-written equality check is where the agent shines, and where scripts are structurally blind.

How do you capture console and network errors?

Findings are only useful if they carry evidence. Wire the listeners before the agent starts moving, and buffer events per route so each finding is self-contained.

function attachErrorCapture(page: Page, findings: Finding[], currentRoute: () => string) {
  // Uncaught exceptions in page scripts.
  page.on('pageerror', (err) => {
    findings.push({ route: currentRoute(), kind: 'console-error', detail: err.message });
  });

  // console.error and console.warn calls.
  page.on('console', (msg: ConsoleMessage) => {
    if (msg.type() === 'error' || msg.type() === 'warning') {
      findings.push({ route: currentRoute(), kind: 'console-error', detail: msg.text() });
    }
  });

  // Any failed HTTP response.
  page.on('response', (res) => {
    if (res.status() >= 400) {
      findings.push({
        route: currentRoute(),
        kind: 'http-error',
        detail: `${res.status()} ${res.url()}`,
      });
    }
  });
}

Now a “layout looks wrong” finding from the LLM arrives with the console error that probably caused it, which cuts triage time dramatically.

How do you bound the crawl?

An unbounded agent is a liability. Left alone it will click the Delete button, wander off to external domains, follow an infinite calendar of “next month” links, and spend your token budget doing it. Three guardrails keep it useful.

Route allowlist. Constrain navigation to your own origin and block destructive or off-limits paths. Do this at the harness level, not by asking the model nicely.

Step and time budget. Cap the number of actions and set a wall-clock timeout. A budget also forces the agent to prioritize, which tends to produce broader coverage than an aimless walk.

Staging with seeded data. Never point an exploratory agent at production. Run it against staging with a known dataset and a scoped auth session so a stray click cannot touch real customers.

const ALLOW = /^https:\/\/staging\.example\.com\//;
const BLOCK = /\/(logout|delete|admin\/destroy)/;
const MAX_STEPS = 40;

function isAllowed(url: string): boolean {
  return ALLOW.test(url) && !BLOCK.test(url);
}

async function runBounded(page: Page, chooseNextUrl: () => Promise<string>) {
  const visited = new Set<string>();
  for (let step = 0; step < MAX_STEPS; step++) {
    const next = await chooseNextUrl();
    if (!isAllowed(next) || visited.has(next)) continue;
    visited.add(next);
    await page.goto(next, { waitUntil: 'networkidle' });
    // observe, reason, report...
  }
}

The chooseNextUrl step is where the LLM decides, but isAllowed and MAX_STEPS are non-negotiable code the model never overrides.

How do you triage the findings?

The agent produces leads, not verdicts, and this is the part teams underestimate. A raw run mixes real regressions with false alarms: a “layout anomaly” that is actually a lazy-loaded image mid-render, a console warning from a third-party script you cannot fix, a “dead link” that is a deliberate 401 behind auth.

A practical triage flow:

1. Deduplicate. The same console error on every page is one finding, not forty.
2. Filter known noise. Suppress third-party warnings and expected auth responses with an ignore list.
3. Rank by signal. An uncaught exception outranks a cosmetic layout note.
4. Confirm manually. A human reproduces the top findings before anything gets filed.
5. Encode confirmed bugs. Every confirmed regression becomes a deterministic Playwright spec so it fails the build the same way forever after.

That last step is how autonomous and deterministic testing fit together. Exploration is a discovery engine that feeds your regression suite; it is not the regression suite itself.

What are the honest limits?

Sell this internally as a discovery tool and it delivers. Sell it as a replacement for assertions and it will burn your credibility. The real constraints:

• Non-deterministic. Two runs take different paths and surface different findings. You cannot gate a build on “the agent was happy today.”
• Needs human triage. False positives are inherent to judgment-based observation. Budget for the review, not just the run.
• Not an oracle. The agent does not know your business rules. It catches “this looks broken,” not “this discount should have been 15 percent.”
• Cost and latency. Every step is model inference plus a screenshot. This runs nightly or on-demand, not on every commit.

Contrast that with a deterministic spec: same input, same result, cheap, fast, and safe to block a merge on. You want both. Let the agent roam staging and hand you a list of things worth looking at; let your assertions stand guard on the paths that must never break.

Where does this fit in a 2026 SDET workflow?

The mature pattern in 2026 is a two-speed suite. Fast, deterministic Playwright specs run on every pull request and defend known behavior. A bounded multimodal exploratory agent runs nightly against staging, files ranked findings, and feeds new specs back into the fast suite as bugs are confirmed. The agent widens coverage into territory you never scripted; the specs make sure that once a bug is found, it stays found.

If you are standing this up, start with the pillar, Building a Self-Operating QA Agent with Playwright MCP, then wire the observe step onto the accessibility tree as described in Accessibility-Tree Driven Locators for Resilient Tests. Explore to find, assert to defend, and never confuse the two.