Skip to content

[CHORE]: Implement comprehensive Playwright test automation for the entire MCP Gateway Admin UI with Makefile targets and GitHub Actions #255

@crivetimihai

Description

@crivetimihai

🧭 Chore Summary

Implement comprehensive Playwright test automation for the entire MCP Gateway Admin UI: establish robust E2E testing with make test-ui, make test-ui-headed, and make test-ui-report targets, achieving 100% UI coverage across all admin dashboard workflows, HTMX interactions, and real-time features using Python and pytest.


🧱 Areas Affected

  • Test infrastructure / Playwright configuration (Python)
  • Build system / Make targets (make test-ui, make test-ui-headed, make test-ui-debug, make test-ui-report)
  • GitHub Actions / CI pipeline (parallel test execution)
  • Admin UI coverage (authentication, CRUD operations, modals, forms)
  • Real-time features (HTMX updates, WebSocket connections, SSE streams)
  • API endpoint testing (REST and MCP protocol validation)
  • Cross-browser compatibility (Chromium, Firefox, WebKit)
  • Performance testing (page load times, interaction responsiveness)

⚙️ Context / Rationale

Playwright ensures that the entire user experience works correctly across browsers and deployments. Every user interaction becomes an automated test, catching UI regressions, HTMX integration issues, authentication problems, and real-time feature failures before they impact users. This creates comprehensive UI coverage that validates both functionality and user workflows using Python's robust testing ecosystem.

What is Playwright with Python?
Playwright is a framework for Web Testing and Automation that allows testing across Chromium, Firefox, and WebKit with a single API. The Python implementation integrates seamlessly with pytest, providing reliable end-to-end testing with auto-wait capabilities, powerful selectors, and comprehensive assertions.

MCP Gateway UI Test Architecture:

graph TD
    A[Pytest-Playwright Suite] --> B[Authentication Tests]
    A --> C[Admin Dashboard Tests]
    A --> D[CRUD Operations Tests]
    A --> E[Real-time Features Tests]
    A --> F[API Integration Tests]
    
    B --> B1[Login Flow]
    B --> B2[JWT Token Validation]
    B --> B3[Session Management]
    
    C --> C1[Navigation & Tabs]
    C --> C2[Dashboard Overview]
    C --> C3[Entity Listings]
    
    D --> D1[Tools Management]
    D --> D2[Resources Management]
    D --> D3[Prompts Management]
    D --> D4[Servers Management]
    D --> D5[Gateways Management]
    D --> D6[Roots Management]
    
    E --> E1[HTMX Updates]
    E --> E2[WebSocket Connections]
    E --> E3[SSE Streams]
    E --> E4[Form Validation]
    
    F --> F1[MCP Protocol]
    F --> F2[REST Endpoints]
    F --> F3[Error Handling]
Loading

Basic Authentication Test Example:

# tests/playwright/test_auth.py
import re
import pytest
from playwright.sync_api import Page, expect

class TestAuthentication:
    def test_should_login_with_valid_credentials(self, page: Page):
        """Test successful login with valid credentials."""
        page.goto("/admin")
        
        # Should redirect to login
        expect(page).to_have_url(re.compile(r".*login"))
        
        # Fill login form
        page.fill('[name="username"]', "admin")
        page.fill('[name="password"]', "password")
        page.click('button[type="submit"]')
        
        # Should redirect to dashboard
        expect(page).to_have_url(re.compile(r".*admin"))
        expect(page.locator("h1")).to_contain_text("MCP Gateway Admin")
        
        # JWT token should be set
        cookies = page.context.cookies()
        jwt_cookie = next((c for c in cookies if c["name"] == "jwt_token"), None)
        assert jwt_cookie is not None
        assert jwt_cookie["httpOnly"] is True

    def test_should_reject_invalid_credentials(self, page: Page):
        """Test login rejection with invalid credentials."""
        page.goto("/admin/login")
        
        page.fill('[name="username"]', "invalid")
        page.fill('[name="password"]', "wrong")
        page.click('button[type="submit"]')
        
        expect(page.locator(".error-message")).to_contain_text("Invalid credentials")
        expect(page).to_have_url(re.compile(r".*login"))

HTMX and Real-time Features Test Example:

# tests/playwright/test_htmx_interactions.py
import pytest
from playwright.sync_api import Page, expect

class TestHTMXInteractions:
    @pytest.fixture(autouse=True)
    def setup(self, page: Page):
        """Login before each test."""
        page.goto("/admin/login")
        page.fill('[name="username"]', "admin")
        page.fill('[name="password"]', "password")
        page.click('button[type="submit"]')
        expect(page).to_have_url(re.compile(r".*admin"))

    def test_should_load_tab_content_via_htmx(self, page: Page):
        """Test HTMX tab loading functionality."""
        # Click on tools tab
        page.click("#tab-tools")
        
        # Wait for HTMX request to complete
        page.wait_for_selector("#tools-panel", state="visible")
        
        # Verify content loaded
        expect(page.locator("#tools-panel")).to_be_visible()
        expect(page.locator("#tools-table")).to_be_visible()
        
        # Check for tools data
        tool_rows = page.locator("#tools-table tbody tr")
        expect(tool_rows).to_have_count_greater_than(0)

    def test_should_create_new_tool_via_modal(self, page: Page):
        """Test tool creation through modal form."""
        page.click("#tab-tools")
        page.wait_for_selector("#tools-panel")
        
        # Open create modal
        page.click('button:has-text("Add Tool")')
        expect(page.locator("#create-tool-modal")).to_be_visible()
        
        # Fill form
        page.fill('[name="name"]', "test-tool")
        page.fill('[name="description"]', "Test tool description")
        page.select_option('[name="integrationType"]', "REST")
        page.fill('[name="url"]', "https://api.example.com/test")
        
        # Submit form
        page.click('#create-tool-modal button[type="submit"]')
        
        # Wait for modal to close and table to update
        expect(page.locator("#create-tool-modal")).to_be_hidden()
        page.wait_for_selector("#tools-table")
        
        # Verify new tool appears in table
        expect(page.locator("#tools-table")).to_contain_text("test-tool")

WebSocket Real-time Testing Example:

# tests/playwright/test_websocket.py
import json
import pytest
from playwright.sync_api import Page, expect

class TestWebSocketConnections:
    def test_should_establish_websocket_connection(self, page: Page):
        """Test WebSocket connection establishment."""
        # Monitor WebSocket connections
        ws_messages = []
        
        def handle_websocket(ws):
            def handle_frame(frame):
                try:
                    message = json.loads(frame.payload)
                    ws_messages.append(message)
                    print(f"WS Frame received: {message}")
                except json.JSONDecodeError:
                    print(f"Non-JSON frame: {frame.payload}")
            
            ws.on("framereceived", handle_frame)
        
        page.on("websocket", handle_websocket)
        
        # Login
        page.goto("/admin")
        page.fill('[name="username"]', "admin")
        page.fill('[name="password"]', "password")
        page.click('button[type="submit"]')
        
        # Navigate to real-time monitoring section
        page.click("#tab-monitoring")
        
        # Test real-time updates
        page.click('button:has-text("Start Monitoring")')
        
        # Wait for real-time data
        page.wait_for_selector(".realtime-indicator.active")
        expect(page.locator(".connection-status")).to_contain_text("Connected")
        
        # Verify WebSocket messages received
        page.wait_for_function(lambda: len(ws_messages) > 0)
        assert len(ws_messages) > 0

API Integration Testing Example:

# tests/playwright/test_api_integration.py
import pytest
from playwright.sync_api import Page, expect, APIRequestContext

class TestAPIIntegration:
    def test_should_handle_mcp_protocol_requests(self, page: Page):
        """Test MCP protocol API integration."""
        # Track API calls
        api_calls = []
        
        def handle_request(route):
            print(f"MCP API call: {route.request.url}")
            api_calls.append(route.request.url)
            route.continue_()
        
        page.route("/api/mcp/**", handle_request)
        
        # Login
        page.goto("/admin")
        page.fill('[name="username"]', "admin")
        page.fill('[name="password"]', "password")
        page.click('button[type="submit"]')
        
        # Test tool execution
        page.click("#tab-tools")
        page.wait_for_selector("#tools-panel")
        
        # Find and execute a tool
        first_tool = page.locator("#tools-table tbody tr").first
        first_tool.locator('button:has-text("Execute")').click()
        
        # Wait for execution modal
        expect(page.locator("#tool-execution-modal")).to_be_visible()
        
        # Fill parameters and execute
        page.fill('[name="tool-params"]', '{"test": "value"}')
        page.click('button:has-text("Run Tool")')
        
        # Wait for results
        page.wait_for_selector(".tool-result", timeout=10000)
        expect(page.locator(".tool-result")).to_be_visible()

    def test_mcp_initialize_endpoint(self, page: Page, request: APIRequestContext):
        """Test MCP initialize endpoint directly."""
        # Login to get JWT token
        page.goto("/admin")
        page.fill('[name="username"]', "admin")
        page.fill('[name="password"]', "password")
        page.click('button[type="submit"]')
        
        # Get JWT token from cookies
        cookies = page.context.cookies()
        jwt_cookie = next((c for c in cookies if c["name"] == "jwt_token"), None)
        assert jwt_cookie is not None
        
        # Test MCP initialize endpoint
        response = request.post("/api/mcp/initialize", 
            headers={"Cookie": f"jwt_token={jwt_cookie['value']}"},
            data={
                "jsonrpc": "2.0",
                "method": "initialize",
                "params": {
                    "protocolVersion": "2025-03-26",
                    "capabilities": {}
                },
                "id": 1
            }
        )
        
        assert response.ok
        data = response.json()
        assert "result" in data

Cross-browser Performance Testing:

# tests/playwright/test_performance.py
import time
import pytest
from playwright.sync_api import Page, expect

class TestPerformance:
    @pytest.mark.parametrize("browser_name", ["chromium", "firefox", "webkit"])
    def test_should_load_admin_dashboard_quickly(self, page: Page, browser_name: str):
        """Test dashboard load performance across browsers."""
        if page.context.browser.browser_type.name != browser_name:
            pytest.skip(f"Running only on {browser_name}")
        
        start_time = time.time()
        
        page.goto("/admin")
        page.fill('[name="username"]', "admin")
        page.fill('[name="password"]', "password")
        page.click('button[type="submit"]')
        
        # Wait for dashboard to be fully loaded
        page.wait_for_selector("#dashboard-content")
        page.wait_for_load_state("networkidle")
        
        load_time = (time.time() - start_time) * 1000
        print(f"Dashboard load time in {browser_name}: {load_time:.2f}ms")
        
        # Assert reasonable load time (adjust threshold as needed)
        assert load_time < 5000

    def test_should_meet_accessibility_standards(self, page: Page):
        """Test accessibility compliance."""
        page.goto("/admin")
        page.fill('[name="username"]', "admin")
        page.fill('[name="password"]', "password")
        page.click('button[type="submit"]')
        
        # Test keyboard navigation
        page.keyboard.press("Tab")
        focused_element = page.locator(":focus")
        expect(focused_element).to_be_visible()
        
        # Test screen reader labels
        buttons = page.locator("button")
        count = buttons.count()
        
        for i in range(count):
            button = buttons.nth(i)
            aria_label = button.get_attribute("aria-label")
            text_content = button.text_content()
            
            assert aria_label or text_content, f"Button {i} missing accessible label"

📦 Related Make Targets

Target Purpose
make test-ui Run all Playwright tests headless with parallel execution
make test-ui-headed Run Playwright tests in headed mode for debugging
make test-ui-debug Run tests with debugging tools and inspector
make test-ui-report Generate and open detailed HTML test report
make test-ui-install Install Playwright browsers and dependencies
make test-ui-codegen Generate test code using Playwright codegen tool
make test-ui-trace Run tests with trace recording for detailed debugging
make test-ui-screenshot Run visual regression tests with screenshot comparison
make test-ui-mobile Run tests against mobile viewports
make test-ui-api Run API-only tests without browser automation
make test-ui-smoke Run critical path smoke tests only
make test-ui-watch Run tests in watch mode for development

Bold targets are mandatory; CI must fail if any UI tests fail or coverage thresholds are not met.


📋 Acceptance Criteria

  • make test-ui passes 100% across all test suites with 0 failures.
  • make test-ui-report generates comprehensive HTML report with screenshots and traces.
  • Cross-browser testing covers Chromium, Firefox, and WebKit with consistent results.
  • All authentication flows tested (login, logout, JWT validation, session expiry).
  • Complete CRUD operations coverage for all entities (tools, resources, prompts, servers, gateways, roots).
  • HTMX interactions fully tested (tab navigation, form submissions, real-time updates).
  • WebSocket and SSE real-time features validated.
  • API integration tests cover both MCP protocol and REST endpoints.
  • Error handling and edge cases documented with test scenarios.
  • Performance benchmarks established for critical user journeys.
  • Visual regression testing prevents UI layout breaks.
  • Mobile responsiveness validated across different viewport sizes.
  • GitHub Actions integrates Playwright tests with parallel execution and artifact storage.

🛠️ Task List (suggested flow)

  1. Playwright setup and configuration

    pip install playwright pytest-playwright
    playwright install

    Create pytest.ini:

    [pytest]
    addopts = 
        --tb=short
        --strict-markers
        --disable-warnings
        --color=yes
        --browser chromium
        --browser firefox
        --browser webkit
        --headed
        --slowmo 100
    testpaths = tests/playwright
    markers =
        smoke: marks tests as smoke tests
        slow: marks tests as slow
        auth: marks tests as authentication related
        htmx: marks tests as HTMX related
        api: marks tests as API related

    Create playwright.config.py:

    import os
    from playwright.sync_api import Playwright
    
    def pytest_configure(config):
        """Configure Playwright for pytest."""
        os.environ.setdefault("PLAYWRIGHT_BROWSERS_PATH", "~/.cache/ms-playwright")
        
    def pytest_playwright_setup(playwright: Playwright):
        """Setup Playwright browsers and configuration."""
        return {
            "base_url": os.getenv("TEST_BASE_URL", "http://localhost:8000"),
            "screenshot": "only-on-failure",
            "video": "retain-on-failure",
            "trace": "retain-on-failure",
        }
  2. Makefile integration

    .PHONY: test-ui test-ui-headed test-ui-debug test-ui-report test-ui-install \
            test-ui-codegen test-ui-trace test-ui-screenshot test-ui-smoke
    
    # Install Playwright and browsers
    test-ui-install:
    	@echo "🎭  Installing Playwright..."
    	pip install playwright pytest-playwright
    	playwright install --with-deps
    	@echo "✅  Playwright installation complete"
    
    # Run all UI tests headless
    test-ui:
    	@echo "🧪  Running Playwright tests..."
    	python -m pytest tests/playwright/ \
    		--browser chromium --browser firefox --browser webkit \
    		--html=reports/ui-test-report.html --self-contained-html \
    		--junitxml=reports/ui-test-results.xml \
    		-v
    
    # Run tests in headed mode for debugging
    test-ui-headed:
    	@echo "🎭  Running Playwright tests in headed mode..."
    	python -m pytest tests/playwright/ \
    		--headed --slowmo 500 \
    		--browser chromium \
    		-v -s
    
    # Run tests with debugging tools
    test-ui-debug:
    	@echo "🔧  Running Playwright tests in debug mode..."
    	PWDEBUG=1 python -m pytest tests/playwright/ \
    		--headed --slowmo 1000 \
    		--browser chromium \
    		-v -s
    
    # Generate and open test report
    test-ui-report:
    	@echo "📊  Opening test report..."
    	@if [ -f reports/ui-test-report.html ]; then \
    		python -c "import webbrowser; webbrowser.open('file://$(PWD)/reports/ui-test-report.html')"; \
    	else \
    		echo "❌  No test report found. Run 'make test-ui' first."; \
    	fi
    
    # Generate test code
    test-ui-codegen:
    	@echo "🎬  Starting Playwright codegen..."
    	playwright codegen http://localhost:8000/admin
    
    # Run with trace recording
    test-ui-trace:
    	@echo "📹  Running tests with trace recording..."
    	python -m pytest tests/playwright/ \
    		--tracing on \
    		--browser chromium \
    		-v
    
    # Visual regression testing
    test-ui-screenshot:
    	@echo "📸  Running visual regression tests..."
    	python -m pytest tests/playwright/test_visual.py \
    		--update-snapshots \
    		--browser chromium
    
    # Smoke tests only
    test-ui-smoke:
    	@echo "🚀  Running smoke tests..."
    	python -m pytest tests/playwright/ \
    		-m smoke \
    		--browser chromium \
    		-v
    
    # Mobile testing
    test-ui-mobile:
    	@echo "📱  Running mobile tests..."
    	python -m pytest tests/playwright/test_mobile.py \
    		--device "iPhone 12" --device "Pixel 5" \
    		-v
    
    # API testing without browser
    test-ui-api:
    	@echo "🔌  Running API tests..."
    	python -m pytest tests/playwright/test_api/ \
    		-v
    
    # Watch mode for development
    test-ui-watch:
    	@echo "👀  Running tests in watch mode..."
    	python -m pytest tests/playwright/ \
    		--headed \
    		--browser chromium \
    		-f -v
  3. Test structure and organization

    tests/playwright/
    ├── conftest.py                    # Pytest fixtures and configuration
    ├── test_auth.py                   # Authentication tests
    ├── test_dashboard.py              # Dashboard navigation tests
    ├── test_htmx_interactions.py      # HTMX interaction tests
    ├── test_websocket.py              # WebSocket real-time tests
    ├── test_api_integration.py        # API integration tests
    ├── test_performance.py            # Performance and accessibility tests
    ├── test_visual.py                 # Visual regression tests
    ├── test_mobile.py                 # Mobile responsiveness tests
    ├── entities/
    │   ├── test_tools.py              # Tools CRUD operations
    │   ├── test_resources.py          # Resources CRUD operations
    │   ├── test_prompts.py            # Prompts CRUD operations
    │   ├── test_servers.py            # Servers CRUD operations
    │   ├── test_gateways.py           # Gateways CRUD operations
    │   └── test_roots.py              # Roots CRUD operations
    ├── api/
    │   ├── test_mcp_protocol.py       # MCP protocol API tests
    │   ├── test_rest_endpoints.py     # REST API endpoint tests
    │   └── test_error_handling.py     # API error handling tests
    ├── fixtures/
    │   ├── auth_helpers.py            # Authentication helper functions
    │   ├── test_data.py               # Test data factories
    │   └── page_objects.py            # Page object models
    └── screenshots/                   # Visual regression baselines
    
  4. Pytest configuration and fixtures

    # tests/playwright/conftest.py
    import pytest
    import re
    from playwright.sync_api import Page, Browser, BrowserContext
    
    @pytest.fixture
    def admin_page(page: Page):
        """Provide a logged-in admin page."""
        page.goto("/admin")
        
        # Handle redirect to login if needed
        if re.search(r"login", page.url):
            page.fill('[name="username"]', "admin")
            page.fill('[name="password"]', "password")
            page.click('button[type="submit"]')
            page.wait_for_url(re.compile(r".*admin"))
        
        return page
    
    @pytest.fixture
    def authenticated_context(browser: Browser) -> BrowserContext:
        """Create a browser context with admin authentication."""
        context = browser.new_context()
        page = context.new_page()
        
        page.goto("/admin/login")
        page.fill('[name="username"]', "admin")
        page.fill('[name="password"]', "password")
        page.click('button[type="submit"]')
        page.wait_for_url(re.compile(r".*admin"))
        
        return context
    
    @pytest.fixture
    def test_tool_data():
        """Provide test data for tool creation."""
        return {
            "name": "test-api-tool",
            "description": "Test API tool for automation",
            "url": "https://api.example.com/test",
            "integrationType": "REST",
            "requestType": "GET",
            "headers": '{"Authorization": "Bearer test-token"}',
            "input_schema": '{"type": "object", "properties": {"query": {"type": "string"}}}'
        }
    
    @pytest.fixture(autouse=True)
    def setup_test_environment(page: Page):
        """Setup test environment before each test."""
        # Set viewport for consistent testing
        page.set_viewport_size({"width": 1280, "height": 720})
        
        # Set default timeout
        page.set_default_timeout(30000)
        
        # Set up request interception for debugging
        def log_request(route):
            print(f"Request: {route.request.method} {route.request.url}")
            route.continue_()
        
        page.route("**/*", log_request)
  5. Page Object Model implementation

    # tests/playwright/fixtures/page_objects.py
    from playwright.sync_api import Page, expect
    import re
    
    class AdminPage:
        """Page object for the admin dashboard."""
        
        def __init__(self, page: Page):
            self.page = page
        
        def login(self, username: str = "admin", password: str = "password"):
            """Login to the admin interface."""
            self.page.goto("/admin")
            
            if re.search(r"login", self.page.url):
                self.page.fill('[name="username"]', username)
                self.page.fill('[name="password"]', password)
                self.page.click('button[type="submit"]')
                self.page.wait_for_url(re.compile(r".*admin"))
        
        def navigate_to_tab(self, tab_name: str):
            """Navigate to a specific tab."""
            self.page.click(f"#tab-{tab_name}")
            self.page.wait_for_selector(f"#{tab_name}-panel", state="visible")
        
        def create_tool(self, tool_data: dict):
            """Create a new tool via the modal form."""
            self.navigate_to_tab("tools")
            self.page.click('button:has-text("Add Tool")')
            expect(self.page.locator("#create-tool-modal")).to_be_visible()
            
            self._fill_tool_form(tool_data)
            self.page.click('#create-tool-modal button[type="submit"]')
            expect(self.page.locator("#create-tool-modal")).to_be_hidden()
        
        def _fill_tool_form(self, data: dict):
            """Fill the tool creation form."""
            for field, value in data.items():
                if field == "integrationType":
                    self.page.select_option(f'[name="{field}"]', value)
                else:
                    self.page.fill(f'[name="{field}"]', value)
        
        def get_tool_by_name(self, name: str):
            """Get a tool row by name."""
            return self.page.locator(f'#tools-table tbody tr:has-text("{name}")')
        
        def delete_tool(self, name: str):
            """Delete a tool by name."""
            tool_row = self.get_tool_by_name(name)
            tool_row.locator('button:has-text("Delete")').click()
            
            # Confirm deletion in modal
            confirm_button = self.page.locator('#delete-confirmation-modal button:has-text("Confirm")')
            if confirm_button.is_visible():
                confirm_button.click()
    
    class ToolsPage:
        """Page object specifically for tools management."""
        
        def __init__(self, page: Page):
            self.page = page
            self.admin_page = AdminPage(page)
        
        def setup(self):
            """Setup the tools page."""
            self.admin_page.login()
            self.admin_page.navigate_to_tab("tools")
        
        def execute_tool(self, tool_name: str, parameters: dict = None):
            """Execute a tool with optional parameters."""
            tool_row = self.admin_page.get_tool_by_name(tool_name)
            tool_row.locator('button:has-text("Execute")').click()
            
            expect(self.page.locator("#tool-execution-modal")).to_be_visible()
            
            if parameters:
                import json
                self.page.fill('[name="tool-params"]', json.dumps(parameters))
            
            self.page.click('button:has-text("Run Tool")')
            self.page.wait_for_selector(".tool-result", timeout=10000)
            
            return self.page.locator(".tool-result").text_content()
  6. Entity CRUD testing

    # tests/playwright/entities/test_tools.py
    import pytest
    from playwright.sync_api import Page, expect
    from tests.playwright.fixtures.page_objects import AdminPage, ToolsPage
    
    class TestToolsCRUD:
        """Test CRUD operations for tools."""
        
        def test_create_new_tool(self, page: Page, test_tool_data):
            """Test creating a new tool."""
            admin_page = AdminPage(page)
            admin_page.login()
            admin_page.create_tool(test_tool_data)
            
            # Verify tool appears in table
            admin_page.navigate_to_tab("tools")
            expect(admin_page.get_tool_by_name(test_tool_data["name"])).to_be_visible()
        
        def test_edit_existing_tool(self, page: Page, test_tool_data):
            """Test editing an existing tool."""
            admin_page = AdminPage(page)
            admin_page.login()
            admin_page.create_tool(test_tool_data)
            
            # Edit the tool
            tool_row = admin_page.get_tool_by_name(test_tool_data["name"])
            tool_row.locator('button:has-text("Edit")').click()
            
            expect(page.locator("#edit-tool-modal")).to_be_visible()
            
            # Update description
            new_description = "Updated test tool description"
            page.fill('[name="description"]', new_description)
            page.click('#edit-tool-modal button[type="submit"]')
            
            # Verify update
            expect(page.locator("#tools-table")).to_contain_text(new_description)
        
        def test_delete_tool(self, page: Page, test_tool_data):
            """Test deleting a tool."""
            admin_page = AdminPage(page)
            admin_page.login()
            admin_page.create_tool(test_tool_data)
            
            # Delete the tool
            admin_page.delete_tool(test_tool_data["name"])
            
            # Verify tool is removed
            expect(admin_page.get_tool_by_name(test_tool_data["name"])).not_to_be_visible()
        
        def test_tool_execution(self, page: Page):
            """Test tool execution functionality."""
            tools_page = ToolsPage(page)
            tools_page.setup()
            
            # Execute a tool with parameters
            result = tools_page.execute_tool("test-tool", {"query": "test query"})
            assert result is not None
            assert len(result) > 0
        
        @pytest.mark.parametrize("integration_type", ["MCP", "REST"])
        def test_tool_creation_by_type(self, page: Page, integration_type):
            """Test tool creation for different integration types."""
            admin_page = AdminPage(page)
            admin_page.login()
            admin_page.navigate_to_tab("tools")
            
            page.click('button:has-text("Add Tool")')
            page.select_option('[name="integrationType"]', integration_type)
            
            # Verify request type options change based on integration type
            if integration_type == "MCP":
                expect(page.locator('[name="requestType"] option[value="SSE"]')).to_be_visible()
            else:
                expect(page.locator('[name="requestType"] option[value="GET"]')).to_be_visible()
  7. Real-time features comprehensive testing

    # tests/playwright/test_realtime_features.py
    import json
    import time
    import pytest
    from playwright.sync_api import Page, expect
    
    class TestRealtimeFeatures:
        """Test real-time features including WebSocket and SSE."""
        
        def test_websocket_connection_lifecycle(self, page: Page):
            """Test complete WebSocket connection lifecycle."""
            ws_events = []
            connection_count = 0
            
            def handle_websocket(ws):
                nonlocal connection_count
                connection_count += 1
                
                def on_frame_sent(frame):
                    ws_events.append(("sent", frame.payload))
                
                def on_frame_received(frame):
                    ws_events.append(("received", frame.payload))
                
                def on_close():
                    ws_events.append(("close", None))
                
                ws.on("framesent", on_frame_sent)
                ws.on("framereceived", on_frame_received)
                ws.on("close", on_close)
            
            page.on("websocket", handle_websocket)
            
            # Login and navigate to monitoring
            page.goto("/admin")
            page.fill('[name="username"]', "admin")
            page.fill('[name="password"]', "password")
            page.click('button[type="submit"]')
            page.click("#tab-monitoring")
            
            # Start monitoring
            page.click('button:has-text("Start Monitoring")')
            
            # Wait for connection establishment
            page.wait_for_selector(".connection-status:has-text('Connected')")
            assert connection_count > 0
            
            # Send a test message
            page.click('button:has-text("Send Test Message")')
            
            # Wait for message exchange
            page.wait_for_function(lambda: len(ws_events) > 0)
            assert len(ws_events) > 0
            
            # Stop monitoring
            page.click('button:has-text("Stop Monitoring")')
            
            # Verify connection closed
            page.wait_for_selector(".connection-status:has-text('Disconnected')")
        
        def test_server_sent_events(self, page: Page):
            """Test Server-Sent Events functionality."""
            sse_messages = []
            
            # Intercept SSE requests
            def handle_response(response):
                if "text/event-stream" in response.headers.get("content-type", ""):
                    sse_messages.append(response.url)
            
            page.on("response", handle_response)
            
            # Login and enable SSE monitoring
            page.goto("/admin")
            page.fill('[name="username"]', "admin")
            page.fill('[name="password"]', "password")
            page.click('button[type="submit"]')
            
            page.click("#tab-monitoring")
            page.click('button:has-text("Enable SSE Updates")')
            
            # Verify SSE connection established
            page.wait_for_function(lambda: len(sse_messages) > 0)
            assert any("sse" in url for url in sse_messages)
            
            # Verify real-time updates appear
            page.wait_for_selector(".sse-indicator.active")
            expect(page.locator(".live-updates")).to_be_visible()
        
        def test_htmx_polling_updates(self, page: Page):
            """Test HTMX polling for live updates."""
            page.goto("/admin")
            page.fill('[name="username"]', "admin")
            page.fill('[name="password"]', "password")
            page.click('button[type="submit"]')
            
            # Navigate to a tab with polling
            page.click("#tab-servers")
            
            # Get initial state
            initial_content = page.locator("#servers-table").text_content()
            
            # Wait for at least one polling update
            time.sleep(2)
            
            # Content should potentially update (or at least polling should occur)
            page.wait_for_selector("#servers-table")
            expect(page.locator("#servers-table")).to_be_visible()
  8. Performance and accessibility testing

    # tests/playwright/test_performance.py
    import time
    import pytest
    from playwright.sync_api import Page, expect
    
    class TestPerformanceAndAccessibility:
        """Test performance benchmarks and accessibility compliance."""
        
        @pytest.mark.slow
        def test_dashboard_load_performance(self, page: Page):
            """Test dashboard loading performance."""
            start_time = time.time()
            
            page.goto("/admin")
            page.fill('[name="username"]', "admin")
            page.fill('[name="password"]', "password")
            page.click('button[type="submit"]')
            
            # Wait for all content to load
            page.wait_for_selector("#dashboard-content")
            page.wait_for_load_state("networkidle")
            
            load_time = (time.time() - start_time) * 1000
            print(f"Dashboard load time: {load_time:.2f}ms")
            
            # Performance assertions
            assert load_time < 5000, f"Dashboard took {load_time:.2f}ms to load"
        
        def test_keyboard_navigation(self, page: Page):
            """Test keyboard accessibility."""
            page.goto("/admin")
            page.fill('[name="username"]', "admin")
            page.fill('[name="password"]', "password")
            page.click('button[type="submit"]')
            
            # Test tab navigation
            focusable_elements = []
            for i in range(10):  # Test first 10 tab stops
                page.keyboard.press("Tab")
                focused = page.locator(":focus")
                if focused.count() > 0:
                    tag_name = focused.evaluate("el => el.tagName")
                    focusable_elements.append(tag_name)
            
            # Should have navigated through several focusable elements
            assert len(focusable_elements) > 5
            assert any(tag in focusable_elements for tag in ["BUTTON", "A", "INPUT"])
        
        def test_screen_reader_labels(self, page: Page):
            """Test screen reader accessibility."""
            page.goto("/admin")
            page.fill('[name="username"]', "admin")
            page.fill('[name="password"]', "password")
            page.click('button[type="submit"]')
            
            # Check buttons have accessible names
            buttons = page.locator("button")
            count = buttons.count()
            
            unlabeled_buttons = []
            for i in range(count):
                button = buttons.nth(i)
                aria_label = button.get_attribute("aria-label")
                text_content = button.text_content()
                title = button.get_attribute("title")
                
                if not (aria_label or text_content or title):
                    unlabeled_buttons.append(i)
            
            assert len(unlabeled_buttons) == 0, f"Found {len(unlabeled_buttons)} unlabeled buttons"
        
        @pytest.mark.parametrize("viewport", [
            {"width": 1920, "height": 1080},  # Desktop
            {"width": 768, "height": 1024},   # Tablet
            {"width": 375, "height": 667},    # Mobile
        ])
        def test_responsive_design(self, page: Page, viewport):
            """Test responsive design across viewports."""
            page.set_viewport_size(viewport)
            
            page.goto("/admin")
            page.fill('[name="username"]', "admin")
            page.fill('[name="password"]', "password")
            page.click('button[type="submit"]')
            
            # Check that main navigation is accessible
            if viewport["width"] < 768:
                # Mobile: check for hamburger menu or mobile navigation
                mobile_nav = page.locator(".mobile-nav, .hamburger-menu, .nav-toggle")
                expect(mobile_nav).to_be_visible()
            else:
                # Desktop/Tablet: check for standard navigation
                main_nav = page.locator("#main-navigation, .nav-tabs")
                expect(main_nav).to_be_visible()
            
            # Ensure content is not overflowing
            body = page.locator("body")
            body_width = body.bounding_box()["width"]
            assert body_width <= viewport["width"]
  9. Visual regression testing

    # tests/playwright/test_visual.py
    import pytest
    from playwright.sync_api import Page, expect
    
    class TestVisualRegression:
        """Test visual regression to catch UI changes."""
        
        def test_dashboard_visual_baseline(self, page: Page):
            """Capture visual baseline for dashboard."""
            page.goto("/admin")
            page.fill('[name="username"]', "admin")
            page.fill('[name="password"]', "password")
            page.click('button[type="submit"]')
            
            # Wait for full load
            page.wait_for_selector("#dashboard-content")
            page.wait_for_load_state("networkidle")
            
            # Full page screenshot
            expect(page).to_have_screenshot("dashboard-full.png")
        
        def test_modal_dialogs_visual(self, page: Page):
            """Test visual appearance of modal dialogs."""
            page.goto("/admin")
            page.fill('[name="username"]', "admin")
            page.fill('[name="password"]', "password")
            page.click('button[type="submit"]')
            
            # Test tool creation modal
            page.click("#tab-tools")
            page.click('button:has-text("Add Tool")')
            
            modal = page.locator("#create-tool-modal")
            expect(modal).to_have_screenshot("create-tool-modal.png")
        
        @pytest.mark.parametrize("tab", ["tools", "resources", "prompts", "servers"])
        def test_tab_content_visual(self, page: Page, tab: str):
            """Test visual appearance of each tab."""
            page.goto("/admin")
            page.fill('[name="username"]', "admin")
            page.fill('[name="password"]', "password")
            page.click('button[type="submit"]')
            
            page.click(f"#tab-{tab}")
            page.wait_for_selector(f"#{tab}-panel")
            
            tab_panel = page.locator(f"#{tab}-panel")
            expect(tab_panel).to_have_screenshot(f"{tab}-panel.png")
  10. CI/CD integration

    # .github/workflows/playwright.yml
    name: Playwright UI Tests
    
    on:
      push:
        branches: [main, develop]
      pull_request:
        branches: [main]
    
    jobs:
      test-ui:
        timeout-minutes: 60
        runs-on: ubuntu-latest
        strategy:
          fail-fast: false
          matrix:
            python-version: ["3.11", "3.12"]
            browser: [chromium, firefox, webkit]
        
        steps:
          - name: ⬇️ Checkout source
            uses: actions/checkout@v4
            with:
              fetch-depth: 1
          
          - name: 🐍 Set up Python ${{ matrix.python-version }}
            uses: actions/setup-python@v5
            with:
              python-version: ${{ matrix.python-version }}
              cache: pip
          
          - name: 📦 Install project dependencies
            run: |
              python -m pip install --upgrade pip
              pip install -e .[dev]
              pip install playwright pytest-playwright
          
          - name: 🎭 Install Playwright browsers
            run: |
              playwright install --with-deps ${{ matrix.browser }}
          
          - name: 🏗️ Start application
            run: |
              make run-dev &
              sleep 15
              curl -f http://localhost:8000/health || exit 1
          
          - name: 🧪 Run Playwright tests
            run: |
              python -m pytest tests/playwright/ \
                --browser ${{ matrix.browser }} \
                --html=reports/ui-test-report-${{ matrix.browser }}.html \
                --self-contained-html \
                --junitxml=reports/ui-test-results-${{ matrix.browser }}.xml \
                --screenshot=only-on-failure \
                --video=retain-on-failure \
                --tracing=retain-on-failure \
                -v
          
          - name: 📊 Upload test results
            uses: actions/upload-artifact@v4
            if: always()
            with:
              name: playwright-report-${{ matrix.python-version }}-${{ matrix.browser }}
              path: |
                reports/
                test-results/
              retention-days: 30
          
          - name: 📋 Publish test results
            uses: dorny/test-reporter@v1
            if: always()
            with:
              name: Playwright Tests (${{ matrix.browser }})
              path: reports/ui-test-results-${{ matrix.browser }}.xml
              reporter: java-junit

📖 References


🧩 Additional Notes

  • Python ecosystem integration: Leverage pytest fixtures, parametrization, and marks for comprehensive test organization.
  • Start with smoke tests: Focus on critical user journeys first (login → dashboard → basic CRUD).
  • Test real user scenarios: Simulate actual workflows rather than isolated component testing.
  • Handle async operations: Use proper waits for HTMX requests, WebSocket connections, and API calls.
  • Cross-browser consistency: Test across Chromium, Firefox, and WebKit to catch browser-specific issues.
  • Mobile responsiveness: Validate UI works correctly on mobile viewports and touch interactions.
  • Error state testing: Test how UI handles network failures, timeouts, and invalid responses.
  • Accessibility validation: Include keyboard navigation, screen reader support, and ARIA attributes.
  • Performance monitoring: Set realistic thresholds for load times and interaction responsiveness.
  • Visual regression prevention: Use screenshot comparisons to catch unintended layout changes.
  • Test data management: Use pytest fixtures and factories to create consistent test data across scenarios.

Playwright-Python Best Practices for MCP Gateway:

  • Use pytest fixtures for reusable setup and teardown logic
  • Implement page object models as Python classes for maintainable code
  • Test both happy paths and error scenarios for all CRUD operations
  • Validate real-time features with proper WebSocket and SSE handling
  • Include performance benchmarks for critical user journeys
  • Use pytest markers (@pytest.mark.smoke) for test categorization
  • Maintain clear test organization with descriptive class and method names
  • Leverage Python's async/await capabilities for concurrent test execution

Metadata

Metadata

Labels

choreLinting, formatting, dependency hygiene, or project maintenance chorescicdIssue with CI/CD process (GitHub Actions, scaffolding)devopsDevOps activities (containers, automation, deployment, makefiles, etc)frontendFrontend development (HTML, CSS, JavaScript)good first issueGood for newcomershelp wantedExtra attention is neededtestingTesting (unit, e2e, manual, automated, etc)triageIssues / Features awaiting triage

Type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions