-
Notifications
You must be signed in to change notification settings - Fork 245
Description
🧭 Chore Summary
Implement comprehensive Playwright test automation for the entire MCP Gateway Admin UI: establish robust E2E testing with make test-ui
, make test-ui-headed
, and make test-ui-report
targets, achieving 100% UI coverage across all admin dashboard workflows, HTMX interactions, and real-time features using Python and pytest.
🧱 Areas Affected
- Test infrastructure / Playwright configuration (Python)
- Build system / Make targets (
make test-ui
,make test-ui-headed
,make test-ui-debug
,make test-ui-report
) - GitHub Actions / CI pipeline (parallel test execution)
- Admin UI coverage (authentication, CRUD operations, modals, forms)
- Real-time features (HTMX updates, WebSocket connections, SSE streams)
- API endpoint testing (REST and MCP protocol validation)
- Cross-browser compatibility (Chromium, Firefox, WebKit)
- Performance testing (page load times, interaction responsiveness)
⚙️ Context / Rationale
Playwright ensures that the entire user experience works correctly across browsers and deployments. Every user interaction becomes an automated test, catching UI regressions, HTMX integration issues, authentication problems, and real-time feature failures before they impact users. This creates comprehensive UI coverage that validates both functionality and user workflows using Python's robust testing ecosystem.
What is Playwright with Python?
Playwright is a framework for Web Testing and Automation that allows testing across Chromium, Firefox, and WebKit with a single API. The Python implementation integrates seamlessly with pytest, providing reliable end-to-end testing with auto-wait capabilities, powerful selectors, and comprehensive assertions.
MCP Gateway UI Test Architecture:
graph TD
A[Pytest-Playwright Suite] --> B[Authentication Tests]
A --> C[Admin Dashboard Tests]
A --> D[CRUD Operations Tests]
A --> E[Real-time Features Tests]
A --> F[API Integration Tests]
B --> B1[Login Flow]
B --> B2[JWT Token Validation]
B --> B3[Session Management]
C --> C1[Navigation & Tabs]
C --> C2[Dashboard Overview]
C --> C3[Entity Listings]
D --> D1[Tools Management]
D --> D2[Resources Management]
D --> D3[Prompts Management]
D --> D4[Servers Management]
D --> D5[Gateways Management]
D --> D6[Roots Management]
E --> E1[HTMX Updates]
E --> E2[WebSocket Connections]
E --> E3[SSE Streams]
E --> E4[Form Validation]
F --> F1[MCP Protocol]
F --> F2[REST Endpoints]
F --> F3[Error Handling]
Basic Authentication Test Example:
# tests/playwright/test_auth.py
import re
import pytest
from playwright.sync_api import Page, expect
class TestAuthentication:
def test_should_login_with_valid_credentials(self, page: Page):
"""Test successful login with valid credentials."""
page.goto("/admin")
# Should redirect to login
expect(page).to_have_url(re.compile(r".*login"))
# Fill login form
page.fill('[name="username"]', "admin")
page.fill('[name="password"]', "password")
page.click('button[type="submit"]')
# Should redirect to dashboard
expect(page).to_have_url(re.compile(r".*admin"))
expect(page.locator("h1")).to_contain_text("MCP Gateway Admin")
# JWT token should be set
cookies = page.context.cookies()
jwt_cookie = next((c for c in cookies if c["name"] == "jwt_token"), None)
assert jwt_cookie is not None
assert jwt_cookie["httpOnly"] is True
def test_should_reject_invalid_credentials(self, page: Page):
"""Test login rejection with invalid credentials."""
page.goto("/admin/login")
page.fill('[name="username"]', "invalid")
page.fill('[name="password"]', "wrong")
page.click('button[type="submit"]')
expect(page.locator(".error-message")).to_contain_text("Invalid credentials")
expect(page).to_have_url(re.compile(r".*login"))
HTMX and Real-time Features Test Example:
# tests/playwright/test_htmx_interactions.py
import pytest
from playwright.sync_api import Page, expect
class TestHTMXInteractions:
@pytest.fixture(autouse=True)
def setup(self, page: Page):
"""Login before each test."""
page.goto("/admin/login")
page.fill('[name="username"]', "admin")
page.fill('[name="password"]', "password")
page.click('button[type="submit"]')
expect(page).to_have_url(re.compile(r".*admin"))
def test_should_load_tab_content_via_htmx(self, page: Page):
"""Test HTMX tab loading functionality."""
# Click on tools tab
page.click("#tab-tools")
# Wait for HTMX request to complete
page.wait_for_selector("#tools-panel", state="visible")
# Verify content loaded
expect(page.locator("#tools-panel")).to_be_visible()
expect(page.locator("#tools-table")).to_be_visible()
# Check for tools data
tool_rows = page.locator("#tools-table tbody tr")
expect(tool_rows).to_have_count_greater_than(0)
def test_should_create_new_tool_via_modal(self, page: Page):
"""Test tool creation through modal form."""
page.click("#tab-tools")
page.wait_for_selector("#tools-panel")
# Open create modal
page.click('button:has-text("Add Tool")')
expect(page.locator("#create-tool-modal")).to_be_visible()
# Fill form
page.fill('[name="name"]', "test-tool")
page.fill('[name="description"]', "Test tool description")
page.select_option('[name="integrationType"]', "REST")
page.fill('[name="url"]', "https://api.example.com/test")
# Submit form
page.click('#create-tool-modal button[type="submit"]')
# Wait for modal to close and table to update
expect(page.locator("#create-tool-modal")).to_be_hidden()
page.wait_for_selector("#tools-table")
# Verify new tool appears in table
expect(page.locator("#tools-table")).to_contain_text("test-tool")
WebSocket Real-time Testing Example:
# tests/playwright/test_websocket.py
import json
import pytest
from playwright.sync_api import Page, expect
class TestWebSocketConnections:
def test_should_establish_websocket_connection(self, page: Page):
"""Test WebSocket connection establishment."""
# Monitor WebSocket connections
ws_messages = []
def handle_websocket(ws):
def handle_frame(frame):
try:
message = json.loads(frame.payload)
ws_messages.append(message)
print(f"WS Frame received: {message}")
except json.JSONDecodeError:
print(f"Non-JSON frame: {frame.payload}")
ws.on("framereceived", handle_frame)
page.on("websocket", handle_websocket)
# Login
page.goto("/admin")
page.fill('[name="username"]', "admin")
page.fill('[name="password"]', "password")
page.click('button[type="submit"]')
# Navigate to real-time monitoring section
page.click("#tab-monitoring")
# Test real-time updates
page.click('button:has-text("Start Monitoring")')
# Wait for real-time data
page.wait_for_selector(".realtime-indicator.active")
expect(page.locator(".connection-status")).to_contain_text("Connected")
# Verify WebSocket messages received
page.wait_for_function(lambda: len(ws_messages) > 0)
assert len(ws_messages) > 0
API Integration Testing Example:
# tests/playwright/test_api_integration.py
import pytest
from playwright.sync_api import Page, expect, APIRequestContext
class TestAPIIntegration:
def test_should_handle_mcp_protocol_requests(self, page: Page):
"""Test MCP protocol API integration."""
# Track API calls
api_calls = []
def handle_request(route):
print(f"MCP API call: {route.request.url}")
api_calls.append(route.request.url)
route.continue_()
page.route("/api/mcp/**", handle_request)
# Login
page.goto("/admin")
page.fill('[name="username"]', "admin")
page.fill('[name="password"]', "password")
page.click('button[type="submit"]')
# Test tool execution
page.click("#tab-tools")
page.wait_for_selector("#tools-panel")
# Find and execute a tool
first_tool = page.locator("#tools-table tbody tr").first
first_tool.locator('button:has-text("Execute")').click()
# Wait for execution modal
expect(page.locator("#tool-execution-modal")).to_be_visible()
# Fill parameters and execute
page.fill('[name="tool-params"]', '{"test": "value"}')
page.click('button:has-text("Run Tool")')
# Wait for results
page.wait_for_selector(".tool-result", timeout=10000)
expect(page.locator(".tool-result")).to_be_visible()
def test_mcp_initialize_endpoint(self, page: Page, request: APIRequestContext):
"""Test MCP initialize endpoint directly."""
# Login to get JWT token
page.goto("/admin")
page.fill('[name="username"]', "admin")
page.fill('[name="password"]', "password")
page.click('button[type="submit"]')
# Get JWT token from cookies
cookies = page.context.cookies()
jwt_cookie = next((c for c in cookies if c["name"] == "jwt_token"), None)
assert jwt_cookie is not None
# Test MCP initialize endpoint
response = request.post("/api/mcp/initialize",
headers={"Cookie": f"jwt_token={jwt_cookie['value']}"},
data={
"jsonrpc": "2.0",
"method": "initialize",
"params": {
"protocolVersion": "2025-03-26",
"capabilities": {}
},
"id": 1
}
)
assert response.ok
data = response.json()
assert "result" in data
Cross-browser Performance Testing:
# tests/playwright/test_performance.py
import time
import pytest
from playwright.sync_api import Page, expect
class TestPerformance:
@pytest.mark.parametrize("browser_name", ["chromium", "firefox", "webkit"])
def test_should_load_admin_dashboard_quickly(self, page: Page, browser_name: str):
"""Test dashboard load performance across browsers."""
if page.context.browser.browser_type.name != browser_name:
pytest.skip(f"Running only on {browser_name}")
start_time = time.time()
page.goto("/admin")
page.fill('[name="username"]', "admin")
page.fill('[name="password"]', "password")
page.click('button[type="submit"]')
# Wait for dashboard to be fully loaded
page.wait_for_selector("#dashboard-content")
page.wait_for_load_state("networkidle")
load_time = (time.time() - start_time) * 1000
print(f"Dashboard load time in {browser_name}: {load_time:.2f}ms")
# Assert reasonable load time (adjust threshold as needed)
assert load_time < 5000
def test_should_meet_accessibility_standards(self, page: Page):
"""Test accessibility compliance."""
page.goto("/admin")
page.fill('[name="username"]', "admin")
page.fill('[name="password"]', "password")
page.click('button[type="submit"]')
# Test keyboard navigation
page.keyboard.press("Tab")
focused_element = page.locator(":focus")
expect(focused_element).to_be_visible()
# Test screen reader labels
buttons = page.locator("button")
count = buttons.count()
for i in range(count):
button = buttons.nth(i)
aria_label = button.get_attribute("aria-label")
text_content = button.text_content()
assert aria_label or text_content, f"Button {i} missing accessible label"
📦 Related Make Targets
Target | Purpose |
---|---|
make test-ui |
Run all Playwright tests headless with parallel execution |
make test-ui-headed |
Run Playwright tests in headed mode for debugging |
make test-ui-debug |
Run tests with debugging tools and inspector |
make test-ui-report |
Generate and open detailed HTML test report |
make test-ui-install |
Install Playwright browsers and dependencies |
make test-ui-codegen |
Generate test code using Playwright codegen tool |
make test-ui-trace |
Run tests with trace recording for detailed debugging |
make test-ui-screenshot |
Run visual regression tests with screenshot comparison |
make test-ui-mobile |
Run tests against mobile viewports |
make test-ui-api |
Run API-only tests without browser automation |
make test-ui-smoke |
Run critical path smoke tests only |
make test-ui-watch |
Run tests in watch mode for development |
Bold targets are mandatory; CI must fail if any UI tests fail or coverage thresholds are not met.
📋 Acceptance Criteria
-
make test-ui
passes 100% across all test suites with 0 failures. -
make test-ui-report
generates comprehensive HTML report with screenshots and traces. - Cross-browser testing covers Chromium, Firefox, and WebKit with consistent results.
- All authentication flows tested (login, logout, JWT validation, session expiry).
- Complete CRUD operations coverage for all entities (tools, resources, prompts, servers, gateways, roots).
- HTMX interactions fully tested (tab navigation, form submissions, real-time updates).
- WebSocket and SSE real-time features validated.
- API integration tests cover both MCP protocol and REST endpoints.
- Error handling and edge cases documented with test scenarios.
- Performance benchmarks established for critical user journeys.
- Visual regression testing prevents UI layout breaks.
- Mobile responsiveness validated across different viewport sizes.
- GitHub Actions integrates Playwright tests with parallel execution and artifact storage.
🛠️ Task List (suggested flow)
-
Playwright setup and configuration
pip install playwright pytest-playwright playwright install
Create
pytest.ini
:[pytest] addopts = --tb=short --strict-markers --disable-warnings --color=yes --browser chromium --browser firefox --browser webkit --headed --slowmo 100 testpaths = tests/playwright markers = smoke: marks tests as smoke tests slow: marks tests as slow auth: marks tests as authentication related htmx: marks tests as HTMX related api: marks tests as API related
Create
playwright.config.py
:import os from playwright.sync_api import Playwright def pytest_configure(config): """Configure Playwright for pytest.""" os.environ.setdefault("PLAYWRIGHT_BROWSERS_PATH", "~/.cache/ms-playwright") def pytest_playwright_setup(playwright: Playwright): """Setup Playwright browsers and configuration.""" return { "base_url": os.getenv("TEST_BASE_URL", "http://localhost:8000"), "screenshot": "only-on-failure", "video": "retain-on-failure", "trace": "retain-on-failure", }
-
Makefile integration
.PHONY: test-ui test-ui-headed test-ui-debug test-ui-report test-ui-install \ test-ui-codegen test-ui-trace test-ui-screenshot test-ui-smoke # Install Playwright and browsers test-ui-install: @echo "🎭 Installing Playwright..." pip install playwright pytest-playwright playwright install --with-deps @echo "✅ Playwright installation complete" # Run all UI tests headless test-ui: @echo "🧪 Running Playwright tests..." python -m pytest tests/playwright/ \ --browser chromium --browser firefox --browser webkit \ --html=reports/ui-test-report.html --self-contained-html \ --junitxml=reports/ui-test-results.xml \ -v # Run tests in headed mode for debugging test-ui-headed: @echo "🎭 Running Playwright tests in headed mode..." python -m pytest tests/playwright/ \ --headed --slowmo 500 \ --browser chromium \ -v -s # Run tests with debugging tools test-ui-debug: @echo "🔧 Running Playwright tests in debug mode..." PWDEBUG=1 python -m pytest tests/playwright/ \ --headed --slowmo 1000 \ --browser chromium \ -v -s # Generate and open test report test-ui-report: @echo "📊 Opening test report..." @if [ -f reports/ui-test-report.html ]; then \ python -c "import webbrowser; webbrowser.open('file://$(PWD)/reports/ui-test-report.html')"; \ else \ echo "❌ No test report found. Run 'make test-ui' first."; \ fi # Generate test code test-ui-codegen: @echo "🎬 Starting Playwright codegen..." playwright codegen http://localhost:8000/admin # Run with trace recording test-ui-trace: @echo "📹 Running tests with trace recording..." python -m pytest tests/playwright/ \ --tracing on \ --browser chromium \ -v # Visual regression testing test-ui-screenshot: @echo "📸 Running visual regression tests..." python -m pytest tests/playwright/test_visual.py \ --update-snapshots \ --browser chromium # Smoke tests only test-ui-smoke: @echo "🚀 Running smoke tests..." python -m pytest tests/playwright/ \ -m smoke \ --browser chromium \ -v # Mobile testing test-ui-mobile: @echo "📱 Running mobile tests..." python -m pytest tests/playwright/test_mobile.py \ --device "iPhone 12" --device "Pixel 5" \ -v # API testing without browser test-ui-api: @echo "🔌 Running API tests..." python -m pytest tests/playwright/test_api/ \ -v # Watch mode for development test-ui-watch: @echo "👀 Running tests in watch mode..." python -m pytest tests/playwright/ \ --headed \ --browser chromium \ -f -v
-
Test structure and organization
tests/playwright/ ├── conftest.py # Pytest fixtures and configuration ├── test_auth.py # Authentication tests ├── test_dashboard.py # Dashboard navigation tests ├── test_htmx_interactions.py # HTMX interaction tests ├── test_websocket.py # WebSocket real-time tests ├── test_api_integration.py # API integration tests ├── test_performance.py # Performance and accessibility tests ├── test_visual.py # Visual regression tests ├── test_mobile.py # Mobile responsiveness tests ├── entities/ │ ├── test_tools.py # Tools CRUD operations │ ├── test_resources.py # Resources CRUD operations │ ├── test_prompts.py # Prompts CRUD operations │ ├── test_servers.py # Servers CRUD operations │ ├── test_gateways.py # Gateways CRUD operations │ └── test_roots.py # Roots CRUD operations ├── api/ │ ├── test_mcp_protocol.py # MCP protocol API tests │ ├── test_rest_endpoints.py # REST API endpoint tests │ └── test_error_handling.py # API error handling tests ├── fixtures/ │ ├── auth_helpers.py # Authentication helper functions │ ├── test_data.py # Test data factories │ └── page_objects.py # Page object models └── screenshots/ # Visual regression baselines
-
Pytest configuration and fixtures
# tests/playwright/conftest.py import pytest import re from playwright.sync_api import Page, Browser, BrowserContext @pytest.fixture def admin_page(page: Page): """Provide a logged-in admin page.""" page.goto("/admin") # Handle redirect to login if needed if re.search(r"login", page.url): page.fill('[name="username"]', "admin") page.fill('[name="password"]', "password") page.click('button[type="submit"]') page.wait_for_url(re.compile(r".*admin")) return page @pytest.fixture def authenticated_context(browser: Browser) -> BrowserContext: """Create a browser context with admin authentication.""" context = browser.new_context() page = context.new_page() page.goto("/admin/login") page.fill('[name="username"]', "admin") page.fill('[name="password"]', "password") page.click('button[type="submit"]') page.wait_for_url(re.compile(r".*admin")) return context @pytest.fixture def test_tool_data(): """Provide test data for tool creation.""" return { "name": "test-api-tool", "description": "Test API tool for automation", "url": "https://api.example.com/test", "integrationType": "REST", "requestType": "GET", "headers": '{"Authorization": "Bearer test-token"}', "input_schema": '{"type": "object", "properties": {"query": {"type": "string"}}}' } @pytest.fixture(autouse=True) def setup_test_environment(page: Page): """Setup test environment before each test.""" # Set viewport for consistent testing page.set_viewport_size({"width": 1280, "height": 720}) # Set default timeout page.set_default_timeout(30000) # Set up request interception for debugging def log_request(route): print(f"Request: {route.request.method} {route.request.url}") route.continue_() page.route("**/*", log_request)
-
Page Object Model implementation
# tests/playwright/fixtures/page_objects.py from playwright.sync_api import Page, expect import re class AdminPage: """Page object for the admin dashboard.""" def __init__(self, page: Page): self.page = page def login(self, username: str = "admin", password: str = "password"): """Login to the admin interface.""" self.page.goto("/admin") if re.search(r"login", self.page.url): self.page.fill('[name="username"]', username) self.page.fill('[name="password"]', password) self.page.click('button[type="submit"]') self.page.wait_for_url(re.compile(r".*admin")) def navigate_to_tab(self, tab_name: str): """Navigate to a specific tab.""" self.page.click(f"#tab-{tab_name}") self.page.wait_for_selector(f"#{tab_name}-panel", state="visible") def create_tool(self, tool_data: dict): """Create a new tool via the modal form.""" self.navigate_to_tab("tools") self.page.click('button:has-text("Add Tool")') expect(self.page.locator("#create-tool-modal")).to_be_visible() self._fill_tool_form(tool_data) self.page.click('#create-tool-modal button[type="submit"]') expect(self.page.locator("#create-tool-modal")).to_be_hidden() def _fill_tool_form(self, data: dict): """Fill the tool creation form.""" for field, value in data.items(): if field == "integrationType": self.page.select_option(f'[name="{field}"]', value) else: self.page.fill(f'[name="{field}"]', value) def get_tool_by_name(self, name: str): """Get a tool row by name.""" return self.page.locator(f'#tools-table tbody tr:has-text("{name}")') def delete_tool(self, name: str): """Delete a tool by name.""" tool_row = self.get_tool_by_name(name) tool_row.locator('button:has-text("Delete")').click() # Confirm deletion in modal confirm_button = self.page.locator('#delete-confirmation-modal button:has-text("Confirm")') if confirm_button.is_visible(): confirm_button.click() class ToolsPage: """Page object specifically for tools management.""" def __init__(self, page: Page): self.page = page self.admin_page = AdminPage(page) def setup(self): """Setup the tools page.""" self.admin_page.login() self.admin_page.navigate_to_tab("tools") def execute_tool(self, tool_name: str, parameters: dict = None): """Execute a tool with optional parameters.""" tool_row = self.admin_page.get_tool_by_name(tool_name) tool_row.locator('button:has-text("Execute")').click() expect(self.page.locator("#tool-execution-modal")).to_be_visible() if parameters: import json self.page.fill('[name="tool-params"]', json.dumps(parameters)) self.page.click('button:has-text("Run Tool")') self.page.wait_for_selector(".tool-result", timeout=10000) return self.page.locator(".tool-result").text_content()
-
Entity CRUD testing
# tests/playwright/entities/test_tools.py import pytest from playwright.sync_api import Page, expect from tests.playwright.fixtures.page_objects import AdminPage, ToolsPage class TestToolsCRUD: """Test CRUD operations for tools.""" def test_create_new_tool(self, page: Page, test_tool_data): """Test creating a new tool.""" admin_page = AdminPage(page) admin_page.login() admin_page.create_tool(test_tool_data) # Verify tool appears in table admin_page.navigate_to_tab("tools") expect(admin_page.get_tool_by_name(test_tool_data["name"])).to_be_visible() def test_edit_existing_tool(self, page: Page, test_tool_data): """Test editing an existing tool.""" admin_page = AdminPage(page) admin_page.login() admin_page.create_tool(test_tool_data) # Edit the tool tool_row = admin_page.get_tool_by_name(test_tool_data["name"]) tool_row.locator('button:has-text("Edit")').click() expect(page.locator("#edit-tool-modal")).to_be_visible() # Update description new_description = "Updated test tool description" page.fill('[name="description"]', new_description) page.click('#edit-tool-modal button[type="submit"]') # Verify update expect(page.locator("#tools-table")).to_contain_text(new_description) def test_delete_tool(self, page: Page, test_tool_data): """Test deleting a tool.""" admin_page = AdminPage(page) admin_page.login() admin_page.create_tool(test_tool_data) # Delete the tool admin_page.delete_tool(test_tool_data["name"]) # Verify tool is removed expect(admin_page.get_tool_by_name(test_tool_data["name"])).not_to_be_visible() def test_tool_execution(self, page: Page): """Test tool execution functionality.""" tools_page = ToolsPage(page) tools_page.setup() # Execute a tool with parameters result = tools_page.execute_tool("test-tool", {"query": "test query"}) assert result is not None assert len(result) > 0 @pytest.mark.parametrize("integration_type", ["MCP", "REST"]) def test_tool_creation_by_type(self, page: Page, integration_type): """Test tool creation for different integration types.""" admin_page = AdminPage(page) admin_page.login() admin_page.navigate_to_tab("tools") page.click('button:has-text("Add Tool")') page.select_option('[name="integrationType"]', integration_type) # Verify request type options change based on integration type if integration_type == "MCP": expect(page.locator('[name="requestType"] option[value="SSE"]')).to_be_visible() else: expect(page.locator('[name="requestType"] option[value="GET"]')).to_be_visible()
-
Real-time features comprehensive testing
# tests/playwright/test_realtime_features.py import json import time import pytest from playwright.sync_api import Page, expect class TestRealtimeFeatures: """Test real-time features including WebSocket and SSE.""" def test_websocket_connection_lifecycle(self, page: Page): """Test complete WebSocket connection lifecycle.""" ws_events = [] connection_count = 0 def handle_websocket(ws): nonlocal connection_count connection_count += 1 def on_frame_sent(frame): ws_events.append(("sent", frame.payload)) def on_frame_received(frame): ws_events.append(("received", frame.payload)) def on_close(): ws_events.append(("close", None)) ws.on("framesent", on_frame_sent) ws.on("framereceived", on_frame_received) ws.on("close", on_close) page.on("websocket", handle_websocket) # Login and navigate to monitoring page.goto("/admin") page.fill('[name="username"]', "admin") page.fill('[name="password"]', "password") page.click('button[type="submit"]') page.click("#tab-monitoring") # Start monitoring page.click('button:has-text("Start Monitoring")') # Wait for connection establishment page.wait_for_selector(".connection-status:has-text('Connected')") assert connection_count > 0 # Send a test message page.click('button:has-text("Send Test Message")') # Wait for message exchange page.wait_for_function(lambda: len(ws_events) > 0) assert len(ws_events) > 0 # Stop monitoring page.click('button:has-text("Stop Monitoring")') # Verify connection closed page.wait_for_selector(".connection-status:has-text('Disconnected')") def test_server_sent_events(self, page: Page): """Test Server-Sent Events functionality.""" sse_messages = [] # Intercept SSE requests def handle_response(response): if "text/event-stream" in response.headers.get("content-type", ""): sse_messages.append(response.url) page.on("response", handle_response) # Login and enable SSE monitoring page.goto("/admin") page.fill('[name="username"]', "admin") page.fill('[name="password"]', "password") page.click('button[type="submit"]') page.click("#tab-monitoring") page.click('button:has-text("Enable SSE Updates")') # Verify SSE connection established page.wait_for_function(lambda: len(sse_messages) > 0) assert any("sse" in url for url in sse_messages) # Verify real-time updates appear page.wait_for_selector(".sse-indicator.active") expect(page.locator(".live-updates")).to_be_visible() def test_htmx_polling_updates(self, page: Page): """Test HTMX polling for live updates.""" page.goto("/admin") page.fill('[name="username"]', "admin") page.fill('[name="password"]', "password") page.click('button[type="submit"]') # Navigate to a tab with polling page.click("#tab-servers") # Get initial state initial_content = page.locator("#servers-table").text_content() # Wait for at least one polling update time.sleep(2) # Content should potentially update (or at least polling should occur) page.wait_for_selector("#servers-table") expect(page.locator("#servers-table")).to_be_visible()
-
Performance and accessibility testing
# tests/playwright/test_performance.py import time import pytest from playwright.sync_api import Page, expect class TestPerformanceAndAccessibility: """Test performance benchmarks and accessibility compliance.""" @pytest.mark.slow def test_dashboard_load_performance(self, page: Page): """Test dashboard loading performance.""" start_time = time.time() page.goto("/admin") page.fill('[name="username"]', "admin") page.fill('[name="password"]', "password") page.click('button[type="submit"]') # Wait for all content to load page.wait_for_selector("#dashboard-content") page.wait_for_load_state("networkidle") load_time = (time.time() - start_time) * 1000 print(f"Dashboard load time: {load_time:.2f}ms") # Performance assertions assert load_time < 5000, f"Dashboard took {load_time:.2f}ms to load" def test_keyboard_navigation(self, page: Page): """Test keyboard accessibility.""" page.goto("/admin") page.fill('[name="username"]', "admin") page.fill('[name="password"]', "password") page.click('button[type="submit"]') # Test tab navigation focusable_elements = [] for i in range(10): # Test first 10 tab stops page.keyboard.press("Tab") focused = page.locator(":focus") if focused.count() > 0: tag_name = focused.evaluate("el => el.tagName") focusable_elements.append(tag_name) # Should have navigated through several focusable elements assert len(focusable_elements) > 5 assert any(tag in focusable_elements for tag in ["BUTTON", "A", "INPUT"]) def test_screen_reader_labels(self, page: Page): """Test screen reader accessibility.""" page.goto("/admin") page.fill('[name="username"]', "admin") page.fill('[name="password"]', "password") page.click('button[type="submit"]') # Check buttons have accessible names buttons = page.locator("button") count = buttons.count() unlabeled_buttons = [] for i in range(count): button = buttons.nth(i) aria_label = button.get_attribute("aria-label") text_content = button.text_content() title = button.get_attribute("title") if not (aria_label or text_content or title): unlabeled_buttons.append(i) assert len(unlabeled_buttons) == 0, f"Found {len(unlabeled_buttons)} unlabeled buttons" @pytest.mark.parametrize("viewport", [ {"width": 1920, "height": 1080}, # Desktop {"width": 768, "height": 1024}, # Tablet {"width": 375, "height": 667}, # Mobile ]) def test_responsive_design(self, page: Page, viewport): """Test responsive design across viewports.""" page.set_viewport_size(viewport) page.goto("/admin") page.fill('[name="username"]', "admin") page.fill('[name="password"]', "password") page.click('button[type="submit"]') # Check that main navigation is accessible if viewport["width"] < 768: # Mobile: check for hamburger menu or mobile navigation mobile_nav = page.locator(".mobile-nav, .hamburger-menu, .nav-toggle") expect(mobile_nav).to_be_visible() else: # Desktop/Tablet: check for standard navigation main_nav = page.locator("#main-navigation, .nav-tabs") expect(main_nav).to_be_visible() # Ensure content is not overflowing body = page.locator("body") body_width = body.bounding_box()["width"] assert body_width <= viewport["width"]
-
Visual regression testing
# tests/playwright/test_visual.py import pytest from playwright.sync_api import Page, expect class TestVisualRegression: """Test visual regression to catch UI changes.""" def test_dashboard_visual_baseline(self, page: Page): """Capture visual baseline for dashboard.""" page.goto("/admin") page.fill('[name="username"]', "admin") page.fill('[name="password"]', "password") page.click('button[type="submit"]') # Wait for full load page.wait_for_selector("#dashboard-content") page.wait_for_load_state("networkidle") # Full page screenshot expect(page).to_have_screenshot("dashboard-full.png") def test_modal_dialogs_visual(self, page: Page): """Test visual appearance of modal dialogs.""" page.goto("/admin") page.fill('[name="username"]', "admin") page.fill('[name="password"]', "password") page.click('button[type="submit"]') # Test tool creation modal page.click("#tab-tools") page.click('button:has-text("Add Tool")') modal = page.locator("#create-tool-modal") expect(modal).to_have_screenshot("create-tool-modal.png") @pytest.mark.parametrize("tab", ["tools", "resources", "prompts", "servers"]) def test_tab_content_visual(self, page: Page, tab: str): """Test visual appearance of each tab.""" page.goto("/admin") page.fill('[name="username"]', "admin") page.fill('[name="password"]', "password") page.click('button[type="submit"]') page.click(f"#tab-{tab}") page.wait_for_selector(f"#{tab}-panel") tab_panel = page.locator(f"#{tab}-panel") expect(tab_panel).to_have_screenshot(f"{tab}-panel.png")
-
CI/CD integration
# .github/workflows/playwright.yml name: Playwright UI Tests on: push: branches: [main, develop] pull_request: branches: [main] jobs: test-ui: timeout-minutes: 60 runs-on: ubuntu-latest strategy: fail-fast: false matrix: python-version: ["3.11", "3.12"] browser: [chromium, firefox, webkit] steps: - name: ⬇️ Checkout source uses: actions/checkout@v4 with: fetch-depth: 1 - name: 🐍 Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} cache: pip - name: 📦 Install project dependencies run: | python -m pip install --upgrade pip pip install -e .[dev] pip install playwright pytest-playwright - name: 🎭 Install Playwright browsers run: | playwright install --with-deps ${{ matrix.browser }} - name: 🏗️ Start application run: | make run-dev & sleep 15 curl -f http://localhost:8000/health || exit 1 - name: 🧪 Run Playwright tests run: | python -m pytest tests/playwright/ \ --browser ${{ matrix.browser }} \ --html=reports/ui-test-report-${{ matrix.browser }}.html \ --self-contained-html \ --junitxml=reports/ui-test-results-${{ matrix.browser }}.xml \ --screenshot=only-on-failure \ --video=retain-on-failure \ --tracing=retain-on-failure \ -v - name: 📊 Upload test results uses: actions/upload-artifact@v4 if: always() with: name: playwright-report-${{ matrix.python-version }}-${{ matrix.browser }} path: | reports/ test-results/ retention-days: 30 - name: 📋 Publish test results uses: dorny/test-reporter@v1 if: always() with: name: Playwright Tests (${{ matrix.browser }}) path: reports/ui-test-results-${{ matrix.browser }}.xml reporter: java-junit
📖 References
- Playwright Python Documentation – Modern web testing framework · https://playwright.dev/python/
- pytest-playwright Plugin – Pytest integration for Playwright · https://github.com/microsoft/playwright-pytest
- Playwright Best Practices – Testing patterns and strategies · https://playwright.dev/python/docs/best-practices
- HTMX Testing Guide – Testing HTMX applications · https://htmx.org/docs/#testing
- WebSocket Testing – Testing real-time connections · https://playwright.dev/python/docs/network#websockets
- Visual Comparisons – Screenshot testing · https://playwright.dev/python/docs/test-screenshots
🧩 Additional Notes
- Python ecosystem integration: Leverage pytest fixtures, parametrization, and marks for comprehensive test organization.
- Start with smoke tests: Focus on critical user journeys first (login → dashboard → basic CRUD).
- Test real user scenarios: Simulate actual workflows rather than isolated component testing.
- Handle async operations: Use proper waits for HTMX requests, WebSocket connections, and API calls.
- Cross-browser consistency: Test across Chromium, Firefox, and WebKit to catch browser-specific issues.
- Mobile responsiveness: Validate UI works correctly on mobile viewports and touch interactions.
- Error state testing: Test how UI handles network failures, timeouts, and invalid responses.
- Accessibility validation: Include keyboard navigation, screen reader support, and ARIA attributes.
- Performance monitoring: Set realistic thresholds for load times and interaction responsiveness.
- Visual regression prevention: Use screenshot comparisons to catch unintended layout changes.
- Test data management: Use pytest fixtures and factories to create consistent test data across scenarios.
Playwright-Python Best Practices for MCP Gateway:
- Use pytest fixtures for reusable setup and teardown logic
- Implement page object models as Python classes for maintainable code
- Test both happy paths and error scenarios for all CRUD operations
- Validate real-time features with proper WebSocket and SSE handling
- Include performance benchmarks for critical user journeys
- Use pytest markers (@pytest.mark.smoke) for test categorization
- Maintain clear test organization with descriptive class and method names
- Leverage Python's async/await capabilities for concurrent test execution