Skip to content

Commit a5c4437

Browse files
committed
cherry-pick(#37693): docs: use VS Code images for test agents
1 parent 6d13fbe commit a5c4437

File tree

5 files changed

+118
-124
lines changed

5 files changed

+118
-124
lines changed
33.9 KB
Loading
27.7 KB
Loading
32.1 KB
Loading

docs/src/test-agents-js.md

Lines changed: 118 additions & 122 deletions
Original file line numberDiff line numberDiff line change
@@ -5,178 +5,178 @@ title: "Agents"
55

66
# Playwright Agents
77

8-
## Test Coverage in 1-2-3
8+
Playwright comes with three Playwright Agents out of the box: **🎭 planner**, **🎭 generator** and **🎭 healer**.
99

10-
Playwright’s agentic workflow makes it possible to generate test coverage in three straightforward steps.
11-
These steps can be performed independently, manually, or as chained calls in an agentic loop.
10+
These agents can be used independently, sequentially, or as the chained calls in the agentic loop.
11+
Using them sequentially will produce test coverage for your product.
1212

13-
1. **Plan**: A planning agent explores the app and produces a test plan in `specs/*.md`.
13+
* **🎭 planner** explores the app and produces a Markdown test plan
1414

15-
2. **Generate**: A generating agent transforms the plan into `tests/*.spec.ts` files. It executes actions against your site to verify selectors and flows, then emits testing code and assertions.
15+
* **🎭 generator** transforms the Markdown plan into the Playwright Test files
1616

17-
3. **Heal**: A healing agent executes the test suite and automatically repairs failing tests by applying diffs in place.
17+
* **🎭 healer** executes the test suite and automatically repairs failing tests
1818

1919
### Getting Started
2020

21-
In order to use Playwright Agents, you must add their definitions to your project using
21+
Start with adding Playwright Agent definitions to your project using
2222
the `init-agents` command. These definitions should be regenerated whenever Playwright
23-
is updated.
23+
is updated to pick up new tools and instructions.
2424

25-
You need to run this command for each agentic loop you will be using:
26-
27-
```bash
28-
# Generate agent files for each agentic loop
29-
# Visual Studio Code
25+
```bash tab=bash-vscode
3026
npx playwright init-agents --loop=vscode
31-
# Claude Code
27+
```
28+
29+
```bash tab=bash-claude
3230
npx playwright init-agents --loop=claude
33-
# opencode
31+
```
32+
33+
```bash tab=bash-opencode
3434
npx playwright init-agents --loop=opencode
3535
```
3636

37-
Once the agents have been generated, you can use your AI tool of choice to command these agents to build Playwright Tests. Playwright splits this into three steps with one agent per step:
37+
Once the agents have been generated, you can use your AI tool of choice to command these agents to build Playwright Tests.
38+
3839

39-
## 1. Plan
40+
## 🎭 Planner
4041

41-
The planning agent explores your app environment and produces a test plan for one or many scenarios and user flows.
42+
Planner agent explores your app and produces a test plan for one or many scenarios and user flows.
4243

4344
**Input**
4445

45-
* A clear request to the planning agent (e.g., “Generate a plan for guest checkout.”)
46-
* A live app entry point (URL) or a seed Playwright test that sets up the environment necessary to talk to your app
47-
* A Product Requirement Document (PRD) (optional)
46+
* A clear request to the planner (e.g., “Generate a plan for guest checkout.”)
47+
* A `seed test` that sets up the environment necessary to interact with your app
48+
* *(optional)* A Product Requirement Document (PRD) for context
4849

49-
**Example Prompt**
50+
**Prompt**
51+
52+
<img src={require("../images/test-agents/planner-prompt.png").src} alt="planner prompt" width="472"/>
5053

51-
```markdown
52-
<agent:planner> Generate a test plan for "Guest Checkout" scenario.
53-
Use `seed.spec.ts` as a seed test for the plan.
54+
> - Notice how the `seed.spec.ts` is included in the context of the planner.
55+
> - Planner will run this test to execute all the initialization necessary for your test including the global setup, project dependencies and all the necessary fixtures and hooks.
56+
> - Planner will also use this seed test as an example of all the generated tests. Alternatively, you can mention the file name in the prompt.
57+
58+
```js title="Example: seed.spec.ts"
59+
import { test, expect } from './fixtures';
60+
61+
test('seed', async ({ page }) => {
62+
// this test uses custom fixtures from ./fixtures
63+
});
5464
```
5565

5666
**Output**
5767

58-
* A Markdown test plan saved to `specs/[scenario name].md`. The plan is human-readable but precise enough for test generation.
68+
* A Markdown test plan saved as `specs/basic-operations.md`.
69+
* The plan is human-readable but precise enough for test generation.
5970

6071
<details>
61-
<summary>Example: specs/guest-checkout.md</summary>
72+
<summary>Example: <b>specs/basic-operations.md</b></summary>
6273

6374
```markdown
64-
# Feature: Guest Checkout
75+
# TodoMVC Application - Basic Operations Test Plan
6576

66-
## Purpose
67-
Allow a user to purchase without creating an account.
77+
## Application Overview
6878

69-
## Preconditions
70-
- Test seed `tests/seed.spec.ts`.
71-
- Payment sandbox credentials available via env vars.
79+
The TodoMVC application is a React-based todo list manager that demonstrates standard todo application functionality. The application provides comprehensive task management capabilities with a clean, intuitive interface. Key features include:
7280

73-
## Scenarios
81+
- **Task Management**: Add, edit, complete, and delete individual todos
82+
- **Bulk Operations**: Mark all todos as complete/incomplete and clear all completed todos
83+
- **Filtering System**: View todos by All, Active, or Completed status with URL routing support
84+
- **Real-time Counter**: Display of active (incomplete) todo count
85+
- **Interactive UI**: Hover states, edit-in-place functionality, and responsive design
86+
- **State Persistence**: Maintains state during session navigation
7487

75-
### SC-1: Add single item to cart and purchase
76-
**Steps**
77-
1. Open home page.
78-
2. Search for "Wireless Mouse".
79-
3. Open product page and add to cart.
80-
4. Proceed to checkout as guest.
81-
5. Fill shipping and payment details.
82-
6. Confirm order.
88+
## Test Scenarios
8389

84-
**Expected**
85-
- Cart count increments after item is added.
86-
- Checkout page shows item, price, tax, and total.
87-
- Order confirmation number appears; status is "Processing".
90+
### 1. Adding New Todos
8891

89-
### SC-2: Tax and shipping recalculation on address change
90-
**Steps**
91-
1. Start checkout with a CA address.
92-
2. Change state to NY.
92+
**Seed:** `tests/seed.spec.ts`
9393

94-
**Expected**
95-
- Tax and shipping values recalculate.
94+
#### 1.1 Add Valid Todo
9695

97-
## Data
98-
- Product SKU: `WM-123`
99-
- Payment: sandbox card `4111 1111 1111 1111`, valid expiry, CVV `123`.
96+
**Steps:**
97+
1. Click in the "What needs to be done?" input field
98+
2. Type "Buy groceries"
99+
3. Press Enter key
100100

101-
## Methodology
102-
*Optional notes about testing methodology*
101+
**Expected Results:**
102+
- Todo appears in the list with unchecked checkbox
103+
- Counter shows "1 item left"
104+
- Input field is cleared and ready for next entry
105+
- Todo list controls become visible (Mark all as complete checkbox)
106+
107+
#### 1.2 Add Multiple Todos
108+
...
103109
```
104110
</details>
105111

106-
## 2. Generate
112+
## 🎭 Generator
107113

108-
The generating agent uses the Markdown plan to produce executable Playwright tests.
109-
It verifies selectors and assertions live against the application. Playwright supports
114+
Generator agent uses the Markdown plan to produce executable Playwright Tests.
115+
It verifies selectors and assertions live as it performs the scenarios. Playwright supports
110116
generation hints and provides a catalog of assertions for efficient structural and
111117
behavioral validation.
112118

113119
**Input**
114120

115121
* Markdown plan from `specs/`
116122

117-
**Example Prompt**
123+
**Prompt**
118124

119-
```markdown
120-
<agent:generator> Generate tests for the guest checkout plan under `specs/`.
121-
```
125+
<img src={require("../images/test-agents/generator-prompt.png").src} alt="generator prompt" width="472"/>
126+
127+
> - Notice how the `basic-operations.md` is included in the context of the generator.
128+
> - This is how generator knows where to get the test plan from. Alternatively, you can mention the file name in the prompt.
122129
123130
**Output**
124131

125132
* A test suite under `tests/`
126133
* Generated tests may include initial errors that can be healed automatically by the healer agent
127134

128135
<details>
129-
<summary>Example: tests/guest-checkout.spec.ts</summary>
136+
<summary>Example: <b>tests/add-valid-todo.spec.ts</b></summary>
130137

131138
```ts
132-
import { test, expect } from '@playwright/test';
133-
134-
test.describe('Guest Checkout', () => {
135-
test('SC-1: add item and purchase', async ({ page }) => {
136-
await page.goto('/');
137-
await page.getByRole('searchbox', { name: /search/i }).fill('Wireless Mouse');
138-
await page.getByRole('button', { name: /search/i }).click();
139-
140-
await page.getByRole('link', { name: /wireless mouse/i }).click();
141-
await page.getByRole('button', { name: /add to cart/i }).click();
142-
143-
// Assertion: cart badge increments
144-
await expect(page.getByTestId('cart-badge')).toHaveText('1');
145-
146-
await page.getByRole('link', { name: /checkout/i }).click();
147-
await page.getByRole('button', { name: /continue as guest/i }).click();
148-
149-
// Fill checkout form
150-
await page.getByLabel('Email').fill(process.env.CHECKOUT_EMAIL!);
151-
await page.getByLabel('Full name').fill('Alex Guest');
152-
await page.getByLabel('Address').fill('1 Market St');
153-
await page.getByLabel('City').fill('San Francisco');
154-
await page.getByLabel('State').selectOption('CA');
155-
await page.getByLabel('ZIP').fill('94105');
156-
157-
// Payment (sandbox)
158-
const frame = page.frameLocator('[data-testid="card-iframe"]');
159-
await frame.getByLabel('Card number').fill('4111111111111111');
160-
await frame.getByLabel('MM / YY').fill('12/30');
161-
await frame.getByLabel('CVC').fill('123');
162-
163-
await page.getByRole('button', { name: /pay/i }).click();
164-
165-
// Assertions: confirmation invariants
166-
await expect(page).toHaveURL(/\/orders\/\w+\/confirmation/);
167-
await expect(page.getByRole('heading', { name: /thank you/i })).toBeVisible();
168-
await expect(page.getByTestId('order-status')).toHaveText(/processing/i);
169-
170-
// Optional visual check
171-
await expect(page.locator('[data-testid="order-summary"]')).toHaveScreenshot();
139+
// spec: specs/basic-operations.md
140+
// seed: tests/seed.spec.ts
141+
142+
import { test, expect } from '../fixtures';
143+
144+
test.describe('Adding New Todos', () => {
145+
test('Add Valid Todo', async ({ page }) => {
146+
// 1. Click in the "What needs to be done?" input field
147+
const todoInput = page.getByRole('textbox', { name: 'What needs to be done?' });
148+
await todoInput.click();
149+
150+
// 2. Type "Buy groceries"
151+
await todoInput.fill('Buy groceries');
152+
153+
// 3. Press Enter key
154+
await todoInput.press('Enter');
155+
156+
// Expected Results:
157+
// - Todo appears in the list with unchecked checkbox
158+
await expect(page.getByText('Buy groceries')).toBeVisible();
159+
const todoCheckbox = page.getByRole('checkbox', { name: 'Toggle Todo' });
160+
await expect(todoCheckbox).toBeVisible();
161+
await expect(todoCheckbox).not.toBeChecked();
162+
163+
// - Counter shows "1 item left"
164+
await expect(page.getByText('1 item left')).toBeVisible();
165+
166+
// - Input field is cleared and ready for next entry
167+
await expect(todoInput).toHaveValue('');
168+
await expect(todoInput).toBeFocused();
169+
170+
// - Todo list controls become visible (Mark all as complete checkbox)
171+
await expect(page.getByRole('checkbox', { name: '❯Mark all as complete' })).toBeVisible();
172172
});
173173
});
174174
```
175175
</details>
176176

177-
## 3. Heal
177+
## 🎭 Healer
178178

179-
When a test fails, the healing agent:
179+
When the test fails, the healer agent:
180180

181181
* Replays the failing steps
182182
* Inspects the current UI to locate equivalent elements or flows
@@ -187,42 +187,38 @@ When a test fails, the healing agent:
187187

188188
* Failing test name
189189

190-
**Example Prompt**
190+
**Prompt**
191191

192-
```markdown
193-
<agent:healer> Fix all failing tests for the guest checkout scenario.
194-
```
192+
<img src={require("../images/test-agents/healer-prompt.png").src} alt="healer prompt" width="469"/>
195193

196194
**Output**
197195

198-
* A passing test, or a skipped test if the healer was unable to ensure correct functionality
196+
* A passing test, or a skipped test if the healer believes the that functionality is broken.
199197

200198
## Artifacts and Conventions
201199

202200
The static agent definitions and generated files follow a simple, auditable structure:
203201

204202
```bash
205203
repo/
206-
.{claude|copilot|vscode|...}/ # agent definitions, tools, guardrails
207-
specs/ # human-readable test plans
208-
checkout-guest.md
209-
account-settings.md
210-
tests/ # generated Playwright tests
211-
seed.spec.ts
212-
checkout-guest.spec.ts
213-
account-settings.spec.ts
204+
.github/ # agent definitions
205+
specs/ # human-readable test plans
206+
basic-operations.md
207+
tests/ # generated Playwright tests
208+
seed.spec.ts # seed test for environment
209+
tests/create/add-valid-todo.spec.ts
214210
playwright.config.ts
215211
```
216212

217213
### Agent Definitions
218214

219-
Agent definitions are collections of instructions and MCP tools. They are provided by
215+
Under the hood, agent definitions are collections of instructions and MCP tools. They are provided by
220216
Playwright and should be regenerated whenever Playwright is updated.
221217

222218
Example for Claude Code subagents:
223219

224220
```bash
225-
npx playwright init-agents --loop=claude
221+
npx playwright init-agents --loop=vscode
226222
```
227223

228224
### Specs in `specs/`

examples/todomvc/tests/seed.spec.ts

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,3 @@
1-
/* eslint-disable notice/notice */
2-
31
import { test, expect } from './fixtures';
42

53
test('seed', async ({ page }) => {

0 commit comments

Comments
 (0)