Add a Playwright e2e test #460

msabramo · 2025-05-28T01:34:49Z

Write a Playwright test that tests the app end-to-end (e2e).

Motivation and Context

Ensure things don't get broken

How Has This Been Tested?

abramowi at Marcs-MacBook-Pro-3 in ~/Code/OpenSource/inspector (playwright-test●)
$ npm run test:e2e

> @modelcontextprotocol/[email protected] test:e2e
> npm run test:e2e --workspace=client


> @modelcontextprotocol/[email protected] test:e2e
> playwright test e2e


Running 8 tests using 5 workers
  8 passed (7.9s)

To open last HTML report run:

  npx playwright show-report

Breaking Changes

No

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation update

Checklist

I have read the MCP Documentation
My code follows the repository's style guidelines
New and existing tests pass locally
I have added appropriate error handling
I have added or updated documentation as needed

Additional context

msabramo · 2025-05-28T02:22:11Z

Related PR (which has perhaps fallen into some disrepair): #354

olaservo · 2025-05-30T16:03:12Z

Thanks for setting this up. My draft PR I'd opened for feedback has been getting pretty moldy. A couple questions about this e2e setup:

Are you thinking these should just be run locally during development? Originally I assumed that we should have these run as part of CI so we're not relying on that. I think it would also be OK to work that in as separate PRs.
I'm making a similar assumption with adding Playwright configs - my original PR had some settings for browser, timeouts, etc. which we can add as we need them.
How do you think we should approach adding tests for different transports? In my original tests I was duplicating a lot of code.

Thanks again for working on this, it should help bridge some gaps with testing basic stuff efficiently.

cliffhall

The test ran successfully, but only after

I downloaded the browsers (additional step required beyond just npm install)
Launching the inspector (you can't just run test:e2e, you have to run start then test:e2e)

Just thinking about how to handle this in CI. We'd need to use caching to install playwright and browsers only if they aren't cached. And start the server, then wait for it to be up before running the tests.

name: Playwright Tests

on:
  push:
    branches: [ main ] 
  pull_request:
    branches: [ main ]

jobs:
  test:
    timeout-minutes: 5
    runs-on: ubuntu-latest 

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 18 

      # Cache Playwright browsers
      - name: Cache Playwright browsers
        id: cache-playwright
        uses: actions/cache@v4 
        with:
          path: ~/.cache/ms-playwright # The default Playwright cache path
          key: ${{ runner.os }}-playwright-${{ hashFiles('package-lock.json') }} # Cache key based on OS and package-lock.json
          restore-keys: |
            ${{ runner.os }}-playwright-

      - name: Install dependencies
        run: npm ci 

      - name: Install Playwright and browsers unless cached
        run: npx playwright install --with-deps
        if: steps.cache-playwright.outputs.cache-hit != 'true' 

      - name: Start Server
        run: npm run start 

      - name: Wait for server to start
        run: npx wait-on http://localhost:6274

      - name: Run Playwright tests
        run: npm run test:e2e

package.json

msabramo · 2025-06-04T06:48:58Z

@olaservo: I had been thinking that this would mainly be run in CI but I like to do things in baby steps so that I don't get overwhelmed and so reviewing is easier. My plan was to do the CI in a separate PR.

Though now I'm wondering if we should just go for it since @cliffhall points out that it's not trivial to set up the tests locally and a lot of people will simply not bother, so maybe it's not of much use if it's not also run in CI.

I probably should look at your PR and see what good stuff I can borrow re: config, timeouts, etc.

msabramo · 2025-06-05T07:23:07Z

Added .github/workflows/e2e_tests.yml (from @cliffhall) in 8a8212c. I guess a maintainer needs to approve the workflow to run so we can see how it works?

cliffhall · 2025-06-05T23:21:47Z

Added .github/workflows/e2e_tests.yml (from @cliffhall) in 8a8212c. I guess a maintainer needs to approve the workflow to run so we can see how it works?

@msabramo I tried but it failed on format check. Need to run prettier-fix to pass CI.

msabramo · 2025-06-07T01:20:38Z

@msabramo I tried but it failed on format check. Need to run prettier-fix to pass CI.

Done in df4c372.

so that the CI job can proceed

msabramo · 2025-06-07T15:32:08Z

I added support for https://github.com/daun/playwright-report-summary in cdb9180.

As a result, at https://github.com/modelcontextprotocol/inspector/actions/runs/15509171521?pr=460 you can see a nice Playwright test report:

msabramo · 2025-06-07T19:19:58Z

@olaservo @cliffhall

Co-authored-by: Cliff Hall <[email protected]>

msabramo · 2025-06-20T18:32:10Z

Still got the following after applying d5b4a6e:

msabramo · 2025-06-20T18:36:41Z

msabramo · 2025-06-20T18:37:37Z

Here's the markdown summary from the GH Actions log, manually pasted here so we can see how it looks:

🎭 Playwright E2E Test Results

✅ 12 passed

Details

12 tests across 1 suite
22.3 seconds
d5b4a6e
ℹ️ Test Environment: Ubuntu Latest, Node.js 18
Browsers: Chromium, Firefox

📊 View Detailed HTML Report (download artifacts)

msabramo · 2025-06-20T19:12:11Z

I wonder if 1ff39f1 caused something to trigger that asked a maintainer to approve execution?

cliffhall · 2025-06-20T19:45:43Z

I wonder if 1ff39f1 caused something to trigger that asked a maintainer to approve execution?

Nope. That's why I asked to remove it. It did not help, but could potentially be a risk.

This reverts commit 4b14b96.

This reverts commit 1ff39f1.

msabramo · 2025-06-20T20:36:50Z

I wonder if 1ff39f1 caused something to trigger that asked a maintainer to approve execution?

Nope. That's why I asked to remove it. It did not help, but could potentially be a risk.

Okay, reverted the two changes in 9ffe63e and 3e28357.

msabramo · 2025-06-20T20:37:47Z

Maybe once it gets merged, the comments will work on builds on main (probably not on PRs in forks, which is probably a good thing).

cliffhall · 2025-06-20T20:49:38Z

Maybe once it gets merged, the comments will work on builds on main (probably not on PRs in forks, which is probably a good thing).

Can you also remove the permissions block

This reverts commit d5b4a6e.

msabramo · 2025-06-20T20:55:37Z

Can you also remove the permissions block

Yup, in 465ee9a.

cliffhall · 2025-06-20T21:27:36Z

One last question: Is there a way to get it to stop trying to make the PR comment? It's still reporting that it can't do it.

…t from a fork

msabramo · 2025-06-21T02:17:50Z

One last question: Is there a way to get it to stop trying to make the PR comment? It's still reporting that it can't do it.

I think c747fe8 might do the trick. Let's see...

…t from a fork

msabramo · 2025-06-21T02:38:31Z

I think c747fe8 might do the trick. Let's see...

Now it skips that step entirely for forks, which I guess is fine.

It might be possible to have it create the report and just not comment, but not sure if it's worth it...

msabramo · 2025-06-21T03:37:13Z

cliffhall · 2025-06-23T15:36:24Z

I re-ran the jobs just to see, and it does not produce the failed annotation warning now.

cliffhall

LGTM! 👍

.github/workflows/e2e_tests.yml

cliffhall · 2025-06-23T15:38:28Z

Thanks for hanging in there @msabramo.

Fixed.

cliffhall reviewed Jun 2, 2025

View reviewed changes

package.json Outdated Show resolved Hide resolved

cliffhall added the waiting on submitter Waiting for the submitter to provide more info label Jun 4, 2025

msabramo added 15 commits June 6, 2025 19:33

Add a Playwright e2e test

226ad25

npm run prettier-fix

69da70a

Exclude e2e from Jest

8a13526

Make npm run test:e2e work

8babc44

Mention npm run test:e2e in CONTRIBUTING.md

3a28efe

Add more Playwright tests

89e1efc

Move test:e2e up with other two test scripts

c803d93

Add .github/workflows/e2e_tests.yml

4f6a4ce

npm run prettier-fix

66ad85a

Do npm run start in background

1df83d7

so that the CI job can proceed

Upload Playwright Report and Screenshots

826bc35

git add playwright.config.ts

a421f32

npm run prettier-fix

f06a6e1

Install Playwright dependencies

a2a58c2

e2e_tests.yml: playwright-report and test-result in client dir

705d7b4

msabramo force-pushed the playwright-test branch from e8b503d to 705d7b4 Compare June 7, 2025 02:34

Generate report with https://github.com/daun/playwright-report-summary

cdb9180

msabramo force-pushed the playwright-test branch from 990ff30 to cdb9180 Compare June 7, 2025 15:22

Take screenshots on failure

2c0af88

msabramo force-pushed the playwright-test branch from 862bd61 to 2c0af88 Compare June 7, 2025 15:36

Update .github/workflows/e2e_tests.yml

d5b4a6e

Co-authored-by: Cliff Hall <[email protected]>

msabramo added 2 commits June 20, 2025 11:39

Change retention-days from 30 to 2

eb89e01

Replace pull_request with pull_request_target

1ff39f1

Both pull_request and pull_request_target

4b14b96

msabramo added 2 commits June 20, 2025 13:35

Revert "Both pull_request and pull_request_target"

9ffe63e

This reverts commit 4b14b96.

Revert "Replace pull_request with pull_request_target"

3e28357

This reverts commit 1ff39f1.

Revert "Update .github/workflows/e2e_tests.yml"

465ee9a

This reverts commit d5b4a6e.

Only try to comment on PR if the PR is from the canonical repo and no…

c747fe8

…t from a fork

msabramo added 2 commits June 20, 2025 19:24

Only try to comment on PR if the PR is from the canonical repo and no…

94a35d0

…t from a fork

Revert

2cd7b74

Always gen test summary but only comment on same repo PRs

ebc249b

cliffhall approved these changes Jun 23, 2025

View reviewed changes

.github/workflows/e2e_tests.yml Outdated Show resolved Hide resolved

.github/workflows/e2e_tests.yml Outdated Show resolved Hide resolved

Merge branch 'main' into playwright-test

86eaabe

cliffhall merged commit 873b838 into modelcontextprotocol:main Jun 23, 2025
4 checks passed

Add a Playwright e2e test #460

Add a Playwright e2e test #460

Uh oh!

Conversation

msabramo commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation and Context

How Has This Been Tested?

Breaking Changes

Types of changes

Checklist

Additional context

Uh oh!

msabramo commented May 28, 2025

Uh oh!

olaservo commented May 30, 2025

Uh oh!

cliffhall left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

msabramo commented Jun 4, 2025

Uh oh!

msabramo commented Jun 5, 2025

Uh oh!

cliffhall commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

msabramo commented Jun 7, 2025

Uh oh!

msabramo commented Jun 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

msabramo commented Jun 7, 2025

Uh oh!

msabramo commented Jun 20, 2025

Uh oh!

msabramo commented Jun 20, 2025

Uh oh!

msabramo commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎭 Playwright E2E Test Results

Details

Uh oh!

msabramo commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cliffhall commented Jun 20, 2025

Uh oh!

msabramo commented Jun 20, 2025

Uh oh!

msabramo commented Jun 20, 2025

Uh oh!

cliffhall commented Jun 20, 2025

Uh oh!

msabramo commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cliffhall commented Jun 20, 2025

Uh oh!

msabramo commented Jun 21, 2025

Uh oh!

msabramo commented Jun 21, 2025

Uh oh!

msabramo commented Jun 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cliffhall commented Jun 23, 2025

Uh oh!

cliffhall left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cliffhall commented Jun 23, 2025

Uh oh!

Uh oh!

Uh oh!

msabramo commented May 28, 2025 •

edited

Loading

cliffhall commented Jun 5, 2025 •

edited

Loading

msabramo commented Jun 7, 2025 •

edited

Loading

msabramo commented Jun 20, 2025 •

edited

Loading

msabramo commented Jun 20, 2025 •

edited

Loading

msabramo commented Jun 20, 2025 •

edited

Loading

msabramo commented Jun 21, 2025 •

edited

Loading