📖 The Story: The Agent That Couldn’t Find the Button

A team wired Playwright into a coding agent (Claude Code) to auto-fix failing tests. The agent could read the ARIA snapshot — a text representation of the page’s accessibility tree — and it correctly identified that a “Submit” button existed. But when the agent tried to click it, the click landed on an invisible overlay that covered the button.

The ARIA snapshot said the button was there, but it didn’t say where. The agent had no way to know the button was visually obscured by a modal overlay. It clicked the coordinates Playwright reported for the button, but those coordinates were under the overlay, so the click hit the overlay instead.

The team spent a week trying to teach the agent to check CSS visibility, z-index, and pointer-events — a rabbit hole of heuristics that worked for some overlays but not others.

If the ARIA snapshot had included bounding boxes, the agent would have immediately seen that the button’s box overlapped with the overlay’s box, and it would have dismissed the overlay first.

🎯 Why It Matters

🌍 Real World Example

A team uses an MCP-driven coding agent to triage failing E2E tests. When a test fails, the agent reads the ARIA snapshot to understand the page state. Before 1.60, the snapshot said “there is a button labeled Submit” — but the agent couldn’t tell if the button was visible, covered by an overlay, or off-screen. With ariaSnapshot({ boxes: true }), the snapshot now says “there is a button labeled Submit [box=120,296,560,48]” — the agent can compare this against the overlay’s box and determine the button is covered, then dismiss the overlay before retrying the click.

🧒 Explain Like I'm 10

Imagine you’re describing a room to a friend over the phone. You say “there’s a chair, a table, and a lamp.” Your friend knows what’s in the room, but not where anything is.

Now imagine you say “there’s a chair at position (3, 5), a table at (7, 5), and a lamp at (9, 2).” Your friend can now draw a map of the room and tell you if the chair is blocking the door.

ARIA snapshots with boxes do the same thing for web pages — they tell AI agents not just what’s on the page, but where each thing is.

🎓 Professional Explanation

Playwright’s ariaSnapshot() API produces a YAML-like text representation of the page’s accessibility tree. It lists every semantic element (headings, buttons, textboxes, links) with their roles and accessible names. This is invaluable for debugging and for asserting page structure without brittle CSS selectors.

Playwright 1.60 adds a boxes option that appends each element’s bounding box as [box=x,y,width,height] in the snapshot output. The coordinates are relative to the page’s top-left corner, in CSS pixels. This transforms the ARIA snapshot from a semantic-only representation into a full layout map that AI agents can reason about.

📊 The Flow

⚙️ The ariaSnapshot({ boxes: true }) API

The API lives on both Page and Locator. Pass { boxes: true } to append bounding boxes to each element in the snapshot. The output is a YAML-like string you can assert against, log for debugging, or feed to an AI agent.

import { test, expect } from class="tk-str">'@playwright/test';

test(class="tk-str">'capture aria snapshot with bounding boxes', async ({ page }) => {
  await page.goto(class="tk-str">'https:class="tk-com">//app.example.com/login');

  const snapshot = await page.ariaSnapshot({ boxes: true });
  console.log(snapshot);
  class=class="tk-str">"tk-com">// Output:
  class=class="tk-str">"tk-com">// - main:
  class=class="tk-str">"tk-com">//   - heading class="tk-str">"Welcome back" [box=120,80,560,40]
  class=class="tk-str">"tk-com">//   - textbox class="tk-str">"Email" [box=120,160,560,48]
  class=class="tk-str">"tk-com">//   - textbox class="tk-str">"Password" [box=120,228,560,48]
  class=class="tk-str">"tk-com">//   - button class="tk-str">"Sign in" [box=120,296,560,48]
});

class=class="tk-str">"tk-com">// Assert against a stored snapshot (without boxes for stability)
test(class="tk-str">'page structure matches golden', async ({ page }) => {
  await page.goto(class="tk-str">'https:class="tk-com">//app.example.com/dashboard');
  await expect(page).toMatchAriaSnapshot();
});

class=class="tk-str">"tk-com">// toMatchAriaSnapshot() now works on a Page directly (1.60+)
class=class="tk-str">"tk-com">// Equivalent to: expect(page.locator(class="tk-str">'body')).toMatchAriaSnapshot()

class=class="tk-str">"tk-com">// Feed the snapshot to an AI agent for triage
test(class="tk-str">'agent triages failing login', async ({ page }) => {
  await page.goto(class="tk-str">'https:class="tk-com">//app.example.com/login');
  await page.getByRole(class="tk-str">'button', { name: class="tk-str">'Sign in' }).click();

  class=class="tk-str">"tk-com">// Capture the page state for the agent
  const snapshot = await page.ariaSnapshot({ boxes: true });

  class=class="tk-str">"tk-com">// Agent reads: class="tk-str">"button 'Sign in' [box=120,296,560,48]"
  class=class="tk-str">"tk-com">//              class="tk-str">"dialog 'Error' [box=100,200,600,300]"
  class=class="tk-str">"tk-com">// Agent concludes: the error dialog overlaps the sign-in button
  class=class="tk-str">"tk-com">// Agent action: click the dialog's close button first, then retry
  const agentAnalysis = await callAgent(`Page snapshot: \
${snapshot}`);
  console.log(agentAnalysis);
});

🔄 Before vs After

Before: Semantic-only ARIA snapshot (1.59 and earlier)

const snapshot = await page.ariaSnapshot();
console.log(snapshot);
class=class="tk-str">"tk-com">// Output:
class=class="tk-str">"tk-com">// - main:
class=class="tk-str">"tk-com">//   - heading class="tk-str">"Welcome back"
class=class="tk-str">"tk-com">//   - textbox class="tk-str">"Email"
class=class="tk-str">"tk-com">//   - textbox class="tk-str">"Password"
class=class="tk-str">"tk-com">//   - button class="tk-str">"Sign in"

class=class="tk-str">"tk-com">// No coordinates — AI agents can't reason about layout

After: ARIA snapshot with boxes (1.60+)

const snapshot = await page.ariaSnapshot({ boxes: true });
console.log(snapshot);
class=class="tk-str">"tk-com">// Output:
class=class="tk-str">"tk-com">// - main:
class=class="tk-str">"tk-com">//   - heading class="tk-str">"Welcome back" [box=120,80,560,40]
class=class="tk-str">"tk-com">//   - textbox class="tk-str">"Email" [box=120,160,560,48]
class=class="tk-str">"tk-com">//   - textbox class="tk-str">"Password" [box=120,228,560,48]
class=class="tk-str">"tk-com">//   - button class="tk-str">"Sign in" [box=120,296,560,48]

class=class="tk-str">"tk-com">// AI agents can now reason about overlaps and visibility

⚠️ Common Mistakes

Mistake 1: Including boxes in toMatchAriaSnapshot() goldens

If you store a snapshot with [box=...] as the golden, it will fail on every environment with different viewport sizes or DPI. Always use toMatchAriaSnapshot() without boxes for assertions, and ariaSnapshot({ boxes: true }) only for logging and AI consumption.

Mistake 2: Assuming boxes reflect visibility

A bounding box tells you the element’s position and size, but not whether it’s visually visible. An element with display: none still has a box (often 0,0,0,0). An element covered by an overlay has a valid box but is not clickable. Always combine box data with visibility checks for clickability decisions.

Mistake 3: Confusing page-level and viewport-level coordinates

The boxes are relative to the page’s top-left corner, not the viewport. If the page is scrolled, an element at [box=120,2000,560,48] is below the visible viewport. AI agents must compare the box’s y-coordinate against the current scroll position and viewport height to determine if the element is on-screen.

✅ Best Practices

1. Use boxes for debugging only, never in golden snapshots for assertion — coordinates are environment-dependent.
2. Combine boxes with visibility checks — a valid box doesn’t mean the element is visible or clickable.
3. Use toMatchAriaSnapshot() on a Page (new in 1.60) for full-page structure assertions — it’s cleaner than expect(page.locator('body')).
4. Feed boxed snapshots to AI agents via MCP for triage — the coordinates enable spatial reasoning.
5. Log boxed snapshots on test failure in a custom reporter so engineers can see the layout state at the moment of failure.

🏗️ Senior Engineer Deep Dive

Why boxes matter for agentic testing

The accessibility tree tells an AI agent what’s on the page (a button labeled “Submit”), but not where it is or whether it’s interactable. Bounding boxes close that gap. An agent reading button 'Submit' [box=120,296,560,48] alongside dialog 'Error' [box=100,200,600,300] can compute that the dialog’s box (100,200 to 700,500) overlaps the button’s box (120,296 to 680,344), concluding the button is covered. Without boxes, the agent would have to issue separate getBoundingClientRect() calls for every element — a round-trip per element that slows triage significantly.

Boxes vs visual regression screenshots

Visual regression (toHaveScreenshot()) captures the rendered pixels — it catches CSS breakages but produces large image files and requires manual review of diffs. ARIA snapshots with boxes capture the semantic structure plus coordinates — they catch layout breakages (element moved, resized, overlapped) without pixel comparison, and they’re text-based so they diff cleanly in code review. Use both: screenshots for visual style, boxed ARIA for layout structure.

💼 Interview Questions

🏋️ Practical Exercise

🚀 Mini Project

📝 Quiz

❓ FAQ

Q: Can I get boxes for a specific locator instead of the whole page?

A: Yes. locator.ariaSnapshot({ boxes: true }) returns the snapshot for just that locator’s subtree. This is useful when you only care about a specific component’s layout, not the full page.

Q: Do boxes work with shadow DOM?

A: Yes. Playwright’s accessibility tree traverses shadow DOM boundaries, and the bounding boxes reflect the rendered position including any shadow DOM offsets. This is one of the few APIs that gives you accurate coordinates for shadow-DOM elements.

Q: How do boxes interact with CSS transforms?

A: The boxes reflect the post-transform rendered position. An element rotated 45 degrees will have a box that’s the axis-aligned bounding rectangle of the rotated element, which may be larger than the element’s pre-transform dimensions. AI agents should be aware that rotated/scaled elements have boxes that don’t perfectly match their visual footprint.

📦 Summary