[CHORE]: Set up contract testing with Pact (pact-python) including Makefile and GitHub Actions targets

[crivetimihai]

🧭 Chore Summary — Set up contract testing with Pact (pact-python) including Makefile and GitHub Actions targets

Implement consumer-driven contract testing using Pact to ensure reliable API interactions between MCP Gateway and its consumers/providers. Add both consumer tests (Gateway calling MCP servers) and provider tests (clients calling Gateway APIs) with Makefile targets and GitHub Actions integration to prevent breaking API changes and replace brittle end-to-end integration tests.

---

🧱 Areas Affected

• Makefile — new contract testing targets
• GitHub Actions — consumer/provider verification workflow
• Test suite — Pact consumer and provider test modules
• API reliability — contract validation for MCP server federation
• CI/CD pipeline — contract verification quality gates
• Documentation — contract testing guide for MCP Gateway

---

⚙️ Context / Rationale

What is Contract Testing?

Contract testing ensures that services can communicate correctly by validating they adhere to a shared API contract. In consumer-driven contract testing, the consumer defines expectations that form a contract, and the provider is verified against it.

MCP Gateway Contract Testing Scenarios:

1. Gateway as Consumer (calling MCP servers):

# Consumer test: Gateway expects MCP server to respond correctly
pact.given("MCP server has tools available")
    .upon_receiving("a tools list request")
    .with_request(method="POST", path="/", 
                  body={"jsonrpc": "2.0", "method": "tools/list"})
    .will_respond_with(status=200,
                       body={"jsonrpc": "2.0", "result": {"tools": []}})

2. Gateway as Provider (serving clients):

# Provider test: Verify Gateway responds to tool invocation correctly  
pact.given("tool 'weather' is registered")
    .upon_receiving("a tool call request")
    .with_request(method="POST", path="/tools/call",
                  body={"tool": "weather", "arguments": {"city": "NYC"}})
    .will_respond_with(status=200,
                       body={"result": {"temperature": "72°F"}})

Why Pact for MCP Gateway?

• Fast & Reliable: Replace slow end-to-end tests with focused contract validation
• Federation Safety: Ensure federated MCP servers maintain compatible contracts
• API Stability: Catch breaking changes in Gateway APIs before deployment
• Consumer-Driven: Client needs drive the contract, ensuring backward compatibility

Pact Workflow:

1. Consumer tests generate pact files (JSON contracts)
2. Provider verification replays interactions against real Gateway APIs
3. CI fails if contracts are broken, preventing incompatible deployments

---

📦 New & Updated Make Targets

Target	Purpose
make pact-install	Install pact-python and verify Pact CLI availability
make pact-consumer-test	Run consumer tests (Gateway → MCP servers) generating pact files
make pact-provider-verify	Verify Gateway APIs against consumer contracts
make pact-publish	Publish pact files to Pact Broker with version tags
make pact-ci	CI-friendly: run consumer tests, verify provider, publish results

Existing test targets remain: test, coverage, mutmut-run.

---

📋 Acceptance Criteria

• pact-python installed and Pact CLI available for contract verification
• Consumer tests for Gateway calling MCP servers (tools/list, tools/call, etc.)
• Provider tests verifying Gateway APIs against client contracts
• GitHub Actions runs contract tests and publishes to Pact Broker
• CI fails on contract violations, preventing breaking API deployments
• Pact files stored in pacts/ directory with proper naming convention
• Documentation covers contract testing workflow and troubleshooting

---

🛠️ Task List

1. Add pact-python configuration

   # pyproject.toml
   [tool.pact]
   pact_dir = "pacts"
   pact_broker_base_url = "https://your-pact-broker.com"
   provider_name = "mcp-gateway"
   consumer_name = "mcp-gateway-client"
   
   # Provider verification
   provider_base_url = "http://localhost:4444"
   provider_states_setup_url = "http://localhost:4444/_pact/provider-states"

2. Add Makefile targets

   .PHONY: pact-install pact-consumer-test pact-provider-verify pact-publish pact-ci
   
   pact-install:
   	@echo "📥 Installing Pact tooling..."
   	@$(VENV_DIR)/bin/pip install pact-python
   	@$(VENV_DIR)/bin/pact-verifier --version
   
   pact-consumer-test: pact-install
   	@echo "🔄 Running consumer contract tests..."
   	@$(VENV_DIR)/bin/pytest tests/pact/consumer/ -v
   
   pact-provider-verify: pact-install
   	@echo "✅ Verifying provider contracts..."
   	@$(VENV_DIR)/bin/pytest tests/pact/provider/ -v
   
   pact-publish:
   	@echo "📤 Publishing pacts to broker..."
   	@$(VENV_DIR)/bin/pact-broker publish pacts/ \
   		--consumer-app-version $(shell git rev-parse HEAD) \
   		--tag $(shell git branch --show-current)
   
   pact-ci: pact-consumer-test pact-provider-verify pact-publish

3. Create consumer tests for MCP server interactions

   # tests/pact/consumer/test_mcp_server_consumer.py
   import pytest
   from pact import Consumer, Provider
   
   def test_mcp_tools_list_contract():
       """Contract test: Gateway expects MCP server tools/list response"""
       pact = Consumer("mcp-gateway").has_pact_with(Provider("mcp-server"))
       
       pact.given("MCP server is running with tools") \
           .upon_receiving("a tools/list request") \
           .with_request(method="POST", path="/",
                        body={"jsonrpc": "2.0", "method": "tools/list", "id": 1}) \
           .will_respond_with(status=200,
                             body={"jsonrpc": "2.0", "id": 1, 
                                   "result": {"tools": [{"name": "example"}]}})

4. Create provider tests for Gateway APIs

   # tests/pact/provider/test_gateway_provider.py
   import pytest
   from pact.verifier import Verifier
   
   def test_gateway_provider_contracts():
       """Verify Gateway APIs against consumer contracts"""
       verifier = Verifier(provider="mcp-gateway",
                          provider_base_url="http://localhost:4444")
       
       output, logs = verifier.verify_pacts("pacts/mcp-gateway-client-mcp-gateway.json")
       assert output == 0, f"Pact verification failed: {logs}"

5. Add provider states setup endpoint

   # mcpgateway/pact_provider_states.py
   @app.post("/_pact/provider-states")
   async def provider_states(request: dict):
       """Setup provider states for Pact verification"""
       state = request.get("state")
       if state == "tool 'weather' is registered":
           # Setup test data for this state
           await setup_weather_tool()
       return {"result": "success"}

6. GitHub Actions integration

   • Add Pact consumer test job that generates contracts
   • Add provider verification job that validates Gateway APIs
   • Configure Pact Broker publishing with authentication
   • Cache Pact CLI and dependencies for faster builds

7. Documentation

   • Add contract testing section to development guide
   • Document MCP server contract expectations
   • Provide troubleshooting guide for contract failures
   • Include examples of writing new Pact tests

8. Integration with existing MCP flows

   • Add contracts for tools/list, tools/call endpoints
   • Add contracts for resources/list, resources/read
   • Add contracts for prompts/list, prompts/get
   • Add contracts for federated gateway interactions

9. Final validation

   • make pact-consumer-test generates valid pact files
   • make pact-provider-verify validates Gateway against contracts
   • CI workflow publishes and verifies contracts end-to-end
   • Contract violations properly fail CI with clear error messages

---

📖 References

• Pact Documentation — https://docs.pact.io/
• pact-python GitHub — https://github.com/pact-foundation/pact-python
• FastAPI Pact Example — https://github.com/pactflow/example-provider-python-fastapi
• MCP Protocol Specification — https://modelcontextprotocol.io/docs/

---

🧩 Additional Notes

• MCP-specific contracts: Focus on critical MCP protocol interactions (tools, resources, prompts)
• Federation testing: Ensure federated gateway contracts remain compatible
• Provider states: Use provider states to set up test data for different scenarios
• Broker integration: Consider PactFlow or self-hosted Pact Broker for team collaboration
• Incremental adoption: Start with core Gateway APIs, expand to MCP server interactions
• Performance: Pact tests are fast unit tests, much faster than end-to-end integration tests
• Versioning: Use Git SHA for provider versions, branch names for consumer tags