[CHORE]: Implement comprehensive fuzz testing automation and Makefile targets (hypothesis, atheris, schemathesis , RESTler) #256
Description
🧭 Chore Summary
Implement comprehensive fuzz testing automation across the entire MCP Gateway codebase: establish robust property-based testing with make fuzz-all, make fuzz-hypothesis, make fuzz-api, and make fuzz-security targets, achieving deep coverage of JSON-RPC validation, JSONPath processing, API endpoints, and stateful business logic to uncover edge cases, security vulnerabilities, and crashes before they reach production.
🧱 Areas Affected
- Test infrastructure / Fuzz testing framework setup
- Build system / Make targets (
make fuzz-all,make fuzz-hypothesis,make fuzz-api,make fuzz-security) - GitHub Actions / CI pipeline (nightly and PR-based fuzzing)
- Core validation logic (JSON-RPC, JSONPath, schema validation)
- API endpoints coverage (OpenAPI-driven fuzzing)
- Stateful API workflows (authentication, CRUD sequences)
- Security testing (input sanitization, injection attacks)
- Native code integration (C extensions, memory safety)
⚙️ Context / Rationale
Fuzz testing ensures that every code path handles unexpected inputs gracefully without crashes, security vulnerabilities, or undefined behavior. By combining property-based testing, coverage-guided fuzzing, schema-driven API testing, and stateful security testing, we create a comprehensive safety net that discovers edge cases no human would think to test. This is critical for a gateway that processes arbitrary JSON-RPC requests, JSONPath expressions, and user-generated content.
What is Fuzz Testing?
Fuzz testing automatically generates thousands of random, malformed, or edge-case inputs to find bugs, crashes, security vulnerabilities, and unexpected behaviors. Different fuzzing approaches target different layers of the application stack.
MCP Gateway Fuzz Testing Architecture:
graph TD
A[Fuzz Testing Suite] --> B[Property-Based Testing]
A --> C[Coverage-Guided Fuzzing]
A --> D[API Schema Fuzzing]
A --> E[Stateful Security Fuzzing]
B --> B1[Hypothesis - Pure Python Logic]
B --> B2[JSON-RPC Validation]
B --> B3[JSONPath Processing]
B --> B4[Schema Validators]
C --> C1[Atheris - Native Code]
C --> C2[Memory Safety]
C --> C3[C Extension Fuzzing]
C --> C4[Deep Logic Paths]
D --> D1[Schemathesis - OpenAPI]
D --> D2[Endpoint Fuzzing]
D --> D3[Contract Validation]
D --> D4[Response Verification]
E --> E1[RESTler - Stateful]
E --> E2[Auth Sequences]
E --> E3[CRUD Workflows]
E --> E4[Security Injections]
Hypothesis Property-Based Testing Example:
# tests/fuzz/test_jsonrpc_fuzz.py
from hypothesis import given, strategies as st, settings, example
import pytest
import json
from mcpgateway.validation.jsonrpc import validate_request, JSONRPCError
class TestJSONRPCFuzzing:
@given(st.binary())
@example(b"") # Empty binary
@example(b"\x00\x01\x02") # Non-UTF8 bytes
def test_validate_request_handles_binary_input(self, raw_bytes):
"""Test that binary input never crashes the validator."""
try:
# This should either parse successfully or raise JSONRPCError
validate_request(raw_bytes)
except (JSONRPCError, ValueError, TypeError, UnicodeDecodeError):
# These are acceptable exceptions
pass
except Exception as e:
pytest.fail(f"Unexpected exception: {type(e).__name__}: {e}")
@given(st.text())
@example("") # Empty string
@example("null") # Valid JSON but invalid request
@example('{"incomplete":') # Malformed JSON
def test_validate_request_handles_text_input(self, text_input):
"""Test that text input never crashes the validator."""
try:
validate_request(text_input)
except (JSONRPCError, ValueError, TypeError, json.JSONDecodeError):
pass
except Exception as e:
pytest.fail(f"Unexpected exception: {type(e).__name__}: {e}")
@given(st.dictionaries(
keys=st.text(min_size=1, max_size=50),
values=st.recursive(
st.one_of(st.none(), st.booleans(), st.integers(), st.floats(), st.text()),
lambda children: st.lists(children) | st.dictionaries(st.text(), children),
max_leaves=20
),
max_size=20
))
def test_validate_request_handles_arbitrary_dicts(self, data):
"""Test arbitrary dictionary structures."""
try:
validate_request(data)
except (JSONRPCError, ValueError, TypeError):
pass
except Exception as e:
pytest.fail(f"Unexpected exception: {type(e).__name__}: {e}")JSONPath Fuzzing Example:
# tests/fuzz/test_jsonpath_fuzz.py
from hypothesis import given, strategies as st, assume
import pytest
from mcpgateway.config import jsonpath_modifier
class TestJSONPathFuzzing:
@given(st.text(min_size=1, max_size=200))
def test_jsonpath_modifier_never_crashes(self, path_expression):
"""Test that arbitrary JSONPath expressions never crash."""
try:
result = jsonpath_modifier(path_expression)
# If it succeeds, result should be valid
assert result is not None
except (ValueError, TypeError, AttributeError, KeyError, IndexError):
# These are acceptable exceptions for invalid paths
pass
except Exception as e:
pytest.fail(f"Unexpected exception: {type(e).__name__}: {e}")
@given(st.text().filter(lambda x: '$' in x))
def test_jsonpath_with_dollar_expressions(self, expression):
"""Test JSONPath expressions containing $ operators."""
try:
jsonpath_modifier(expression)
except (ValueError, TypeError, AttributeError, KeyError, IndexError):
pass
except Exception as e:
pytest.fail(f"Unexpected exception: {type(e).__name__}: {e}")Atheris Coverage-Guided Fuzzing Example:
# fuzzers/fuzz_jsonpath.py
#!/usr/bin/env python3
"""Coverage-guided fuzzing for JSONPath processing."""
import atheris
import sys
import json
from mcpgateway.config import jsonpath_modifier
def TestOneInput(data):
"""Fuzz target for JSONPath processing."""
fdp = atheris.FuzzedDataProvider(data)
try:
# Generate various JSONPath expressions
if fdp.remaining_bytes() < 1:
return
choice = fdp.ConsumeIntInRange(0, 3)
if choice == 0:
# Simple path expression
path = "$." + fdp.ConsumeUnicodeNoSurrogates(50)
elif choice == 1:
# Array access
path = "$[" + str(fdp.ConsumeIntInRange(0, 1000)) + "]"
elif choice == 2:
# Complex expression
path = fdp.ConsumeUnicodeNoSurrogates(100)
else:
# Raw input
path = fdp.ConsumeString(200)
# Test JSONPath modifier (should never crash)
jsonpath_modifier(path)
except (ValueError, TypeError, AttributeError, KeyError, IndexError):
# Expected exceptions for invalid input
pass
except Exception:
# Unexpected exceptions should be caught by Atheris
raise
def main():
atheris.instrument_all()
atheris.Setup(sys.argv, TestOneInput)
atheris.Fuzz()
if __name__ == "__main__":
main()Schemathesis API Fuzzing Integration:
# tests/fuzz/test_api_schema_fuzz.py
import schemathesis
import pytest
from fastapi.testclient import TestClient
from mcpgateway.main import app
# Create schema from FastAPI app
schema = schemathesis.from_asgi("/openapi.json", app)
@schema.parametrize()
@pytest.mark.fuzz
def test_api_endpoints_never_crash(case):
"""Test that all API endpoints handle arbitrary valid inputs."""
# Add authentication headers
case.headers = case.headers or {}
case.headers.update({
"Authorization": "Basic YWRtaW46Y2hhbmdlbWU=" # admin:changeme
})
# Execute the test case
response = case.call_asgi(app)
# Verify response is well-formed
case.validate_response(response)
# Additional checks
assert response.status_code < 500, f"Server error: {response.status_code}"
if response.headers.get("content-type", "").startswith("application/json"):
try:
response.json() # Should parse as valid JSON
except ValueError:
pytest.fail("Response claimed to be JSON but was not parseable")
class TestAPIFuzzingCustom:
def test_jsonrpc_endpoint_fuzzing(self):
"""Custom fuzzing for JSON-RPC endpoints."""
client = TestClient(app)
test_cases = [
{"jsonrpc": "2.0", "method": "ping", "id": 1},
{"jsonrpc": "2.0", "method": "initialize", "params": {}, "id": 2},
{"jsonrpc": "2.0", "method": "tools/list", "id": 3},
# Malformed cases
{"jsonrpc": "1.0", "method": "ping", "id": 1}, # Wrong version
{"method": "ping", "id": 1}, # Missing jsonrpc
{"jsonrpc": "2.0", "id": 1}, # Missing method
]
for case in test_cases:
response = client.post(
"/rpc",
json=case,
headers={"Authorization": "Basic YWRtaW46Y2hhbmdlbWU="}
)
# Should not crash (may return error responses)
assert response.status_code in [200, 400, 401, 422]RESTler Stateful Security Fuzzing Configuration:
# fuzzers/restler_config.py
"""Configuration for RESTler stateful fuzzing."""
RESTLER_CONFIG = {
"RestlerRootDirPath": "restler_results",
"SwaggerSpecFilePath": "openapi.json",
"GrammarOutputDirectoryPath": "restler_grammar",
"ReplayLogFilePath": "restler_replay.json",
"Settings": {
"max_combinations": 10000,
"max_request_execution_time": 30,
"target_ip": "127.0.0.1",
"target_port": 4444,
"authentication": {
"token": {
"type": "basic",
"username": "admin",
"password": "changeme"
}
},
"checkers": {
"enable_all": True,
"enable_body_utf8_checking": True,
"enable_response_time_checking": True,
"enable_memory_leak_checking": True
}
}
}📦 Related Make Targets
| Target | Purpose |
|---|---|
make fuzz-all |
Run complete fuzzing suite (hypothesis + atheris + api + security) |
make fuzz-hypothesis |
Property-based testing with Hypothesis |
make fuzz-atheris |
Coverage-guided fuzzing with Atheris |
make fuzz-api |
Schema-driven API fuzzing with Schemathesis |
make fuzz-security |
Stateful security fuzzing with RESTler |
make fuzz-install |
Install all fuzzing dependencies |
make fuzz-corpus |
Generate and manage fuzz testing corpora |
make fuzz-report |
Generate comprehensive fuzz testing report |
make fuzz-clean |
Clean fuzzing artifacts and results |
make fuzz-regression |
Run regression tests for known issues |
make fuzz-quick |
Fast fuzzing for CI/PR validation |
make fuzz-extended |
Extended fuzzing for nightly runs |
Bold targets are mandatory; CI must fail if fuzzing discovers crashes, security issues, or unexpected exceptions.
📋 Acceptance Criteria
-
make fuzz-allcompletes successfully with 0 crashes and 0 security issues. - Hypothesis tests cover all pure Python validation logic with property-based testing.
- Atheris fuzzing targets JSONPath, JSON-RPC, and other native code paths.
- Schemathesis validates all OpenAPI endpoints with comprehensive input generation.
- RESTler performs stateful security testing of authentication and CRUD workflows.
- Fuzzing corpus is maintained with interesting test cases for regression testing.
- CI integration runs quick fuzzing on PRs and extended fuzzing nightly.
- All fuzzing tools generate structured reports with reproducible test cases.
- Security checkers enabled for injection attacks, memory leaks, and timing issues.
- Performance benchmarks ensure fuzzing doesn't introduce significant overhead.
- Documentation provides clear guidance on interpreting and acting on fuzz findings.
- Regression tests prevent reintroduction of previously discovered issues.
🛠️ Task List (suggested flow)
-
Install fuzzing dependencies and setup
# Install core fuzzing tools pip install hypothesis pytest-hypothesis pip install atheris pip install schemathesis # Install RESTler (requires .NET) git clone https://github.com/microsoft/restler-fuzzer.git cd restler-fuzzer && ./build.sh
Create
requirements-fuzz.txt:hypothesis>=6.90.0 pytest-hypothesis>=0.19.0 atheris>=2.3.0 schemathesis>=3.19.0 pytest-xdist>=3.0.0 coverage>=7.0.0
-
Makefile integration
.PHONY: fuzz-all fuzz-hypothesis fuzz-atheris fuzz-api fuzz-security \ fuzz-install fuzz-corpus fuzz-report fuzz-clean fuzz-quick # Install all fuzzing dependencies fuzz-install: @echo "🔧 Installing fuzzing dependencies..." pip install -r requirements-fuzz.txt @echo "✅ Fuzzing tools installed" # Run complete fuzzing suite fuzz-all: fuzz-hypothesis fuzz-atheris fuzz-api fuzz-security @echo "🎯 Complete fuzzing suite finished" make fuzz-report # Property-based testing with Hypothesis fuzz-hypothesis: @echo "🧪 Running Hypothesis property-based tests..." python -m pytest tests/fuzz/ -v \ --hypothesis-show-statistics \ --hypothesis-profile=dev \ -x # Coverage-guided fuzzing with Atheris fuzz-atheris: @echo "🎭 Running Atheris coverage-guided fuzzing..." @mkdir -p corpus fuzzers/results python fuzzers/fuzz_jsonpath.py -runs=10000 -max_total_time=300 \ -artifact_prefix=fuzzers/results/ -print_final_stats=1 python fuzzers/fuzz_jsonrpc.py -runs=10000 -max_total_time=300 \ -artifact_prefix=fuzzers/results/ -print_final_stats=1 # Schema-driven API fuzzing fuzz-api: @echo "🌐 Running Schemathesis API fuzzing..." @echo "Starting test server..." make run-dev & sleep 10 schemathesis run http://localhost:4444/openapi.json \ --checks all \ --hypothesis-max-examples=1000 \ --auth admin:changeme \ --workers 4 \ --report=reports/schemathesis-report.json pkill -f "make run-dev" || true # Stateful security fuzzing with RESTler fuzz-security: @echo "🔐 Running RESTler stateful security fuzzing..." @mkdir -p restler_results make run-dev & sleep 10 curl -o openapi.json http://localhost:4444/openapi.json restler-fuzzer/restler/Restler compile --api_spec openapi.json restler-fuzzer/restler/Restler fuzz \ --grammar_file Compile/grammar.py \ --dictionary_file Compile/dict.json \ --settings Compile/engine_settings.json \ --time_budget 0:30:00 \ --target_ip 127.0.0.1 \ --target_port 4444 pkill -f "make run-dev" || true # Quick fuzzing for CI fuzz-quick: @echo "⚡ Running quick fuzzing for CI..." python -m pytest tests/fuzz/ -v \ --hypothesis-max-examples=100 \ -x python fuzzers/fuzz_jsonpath.py -runs=1000 -max_total_time=60 # Generate fuzzing report fuzz-report: @echo "📊 Generating fuzzing report..." python scripts/generate_fuzz_report.py # Clean fuzzing artifacts fuzz-clean: @echo "🧹 Cleaning fuzzing artifacts..." rm -rf corpus/ fuzzers/results/ restler_results/ rm -f openapi.json reports/schemathesis-report.json
-
Directory structure setup
tests/fuzz/ ├── __init__.py ├── conftest.py # Fuzzing test configuration ├── test_jsonrpc_fuzz.py # JSON-RPC validation fuzzing ├── test_jsonpath_fuzz.py # JSONPath processing fuzzing ├── test_schema_validation_fuzz.py # Pydantic schema fuzzing ├── test_api_schema_fuzz.py # Schemathesis integration └── test_security_fuzz.py # Security-focused fuzzing fuzzers/ ├── fuzz_jsonpath.py # Atheris JSONPath fuzzer ├── fuzz_jsonrpc.py # Atheris JSON-RPC fuzzer ├── fuzz_config_parser.py # Atheris config parsing fuzzer ├── restler_config.py # RESTler configuration └── results/ # Fuzzing artifacts corpus/ ├── jsonpath/ # JSONPath test cases ├── jsonrpc/ # JSON-RPC test cases └── api/ # API request test cases scripts/ └── generate_fuzz_report.py # Report generation -
Core fuzzing test implementation
# tests/fuzz/conftest.py import pytest from hypothesis import settings, Verbosity # Configure Hypothesis for fuzzing settings.register_profile("dev", max_examples=100, verbosity=Verbosity.normal) settings.register_profile("ci", max_examples=50, verbosity=Verbosity.quiet) settings.register_profile("thorough", max_examples=1000, verbosity=Verbosity.verbose) @pytest.fixture(scope="session") def fuzz_settings(): """Configure fuzzing settings based on environment.""" import os profile = os.getenv("HYPOTHESIS_PROFILE", "dev") settings.load_profile(profile)
# tests/fuzz/test_schema_validation_fuzz.py from hypothesis import given, strategies as st import pytest from mcpgateway.schemas import ToolCreate, ResourceCreate, PromptCreate class TestSchemaValidationFuzzing: @given(st.dictionaries( keys=st.text(min_size=1, max_size=50), values=st.one_of( st.none(), st.booleans(), st.integers(), st.floats(), st.text(), st.lists(st.text()) ) )) def test_tool_create_schema_robust(self, data): """Test ToolCreate schema with arbitrary data.""" try: tool = ToolCreate(**data) # If validation succeeds, object should be valid assert tool.name is not None except (ValueError, TypeError): # Expected for invalid data pass @given(st.text(min_size=0, max_size=10000)) def test_tool_create_name_field(self, name): """Test tool name field with various string inputs.""" try: tool = ToolCreate(name=name, url="http://example.com") assert len(tool.name) >= 1 # Name should not be empty after validation except ValueError: # Expected for invalid names pass
-
Atheris native code fuzzing
# fuzzers/fuzz_jsonrpc.py #!/usr/bin/env python3 """Coverage-guided fuzzing for JSON-RPC validation.""" import atheris import sys import json from mcpgateway.validation.jsonrpc import validate_request, JSONRPCError def TestOneInput(data): """Fuzz target for JSON-RPC validation.""" fdp = atheris.FuzzedDataProvider(data) try: if fdp.remaining_bytes() < 1: return choice = fdp.ConsumeIntInRange(0, 4) if choice == 0: # Valid-ish JSON-RPC structure request = { "jsonrpc": fdp.ConsumeUnicodeNoSurrogates(10), "method": fdp.ConsumeUnicodeNoSurrogates(50), "id": fdp.ConsumeIntInRange(0, 1000000) } if fdp.ConsumeBool(): request["params"] = {"test": fdp.ConsumeUnicodeNoSurrogates(100)} elif choice == 1: # Malformed JSON structure request = fdp.ConsumeUnicodeNoSurrogates(200) elif choice == 2: # Binary data request = fdp.ConsumeBytes(100) elif choice == 3: # Random dictionary request = {} for _ in range(fdp.ConsumeIntInRange(0, 10)): key = fdp.ConsumeUnicodeNoSurrogates(20) value = fdp.ConsumeUnicodeNoSurrogates(50) request[key] = value else: # Edge cases request = None if fdp.ConsumeBool() else [] # Test validation (should never crash) validate_request(request) except (JSONRPCError, ValueError, TypeError, json.JSONDecodeError, KeyError): # Expected exceptions pass except Exception: # Unexpected - let Atheris catch it raise def main(): atheris.instrument_all() atheris.Setup(sys.argv, TestOneInput) atheris.Fuzz() if __name__ == "__main__": main()
-
Schemathesis API testing integration
# tests/fuzz/test_api_comprehensive_fuzz.py import schemathesis from fastapi.testclient import TestClient from mcpgateway.main import app import pytest # Load schema from the FastAPI app schema = schemathesis.from_asgi("/openapi.json", app) class TestAPIFuzzingComprehensive: @schema.parametrize(endpoint="/admin/tools") def test_tools_endpoint_fuzzing(self, case): """Fuzz the tools management endpoints.""" case.headers = case.headers or {} case.headers["Authorization"] = "Basic YWRtaW46Y2hhbmdlbWU=" response = case.call_asgi(app) case.validate_response(response) # Tools endpoints should never return 500 assert response.status_code < 500 @schema.parametrize(endpoint="/rpc") def test_jsonrpc_endpoint_fuzzing(self, case): """Fuzz the JSON-RPC endpoint.""" case.headers = case.headers or {} case.headers["Authorization"] = "Basic YWRtaW46Y2hhbmdlbWU=" case.headers["Content-Type"] = "application/json" response = case.call_asgi(app) # JSON-RPC should handle malformed requests gracefully assert response.status_code in [200, 400, 401, 422] def test_authentication_fuzzing(self): """Test authentication with various malformed credentials.""" client = TestClient(app) auth_variants = [ "Basic invalid", "Bearer token123", "Basic " + "x" * 1000, # Very long auth "Digest username=test", "", None ] for auth in auth_variants: headers = {"Authorization": auth} if auth else {} response = client.get("/admin/tools", headers=headers) # Should return 401 or handle gracefully assert response.status_code in [401, 400, 422]
-
Security-focused fuzzing
# tests/fuzz/test_security_fuzz.py from hypothesis import given, strategies as st import pytest from fastapi.testclient import TestClient from mcpgateway.main import app class TestSecurityFuzzing: @given(st.text(min_size=1, max_size=1000)) def test_sql_injection_resistance(self, malicious_input): """Test resistance to SQL injection in various fields.""" client = TestClient(app) # Test in tool name field payload = { "name": malicious_input, "url": "http://example.com", "description": "test" } response = client.post( "/admin/tools", json=payload, headers={"Authorization": "Basic YWRtaW46Y2hhbmdlbWU="} ) # Should not crash, may reject invalid input assert response.status_code in [200, 201, 400, 422] @given(st.text().filter(lambda x: any(char in x for char in '<>"\'&'))) def test_xss_prevention(self, potentially_malicious): """Test XSS prevention in user inputs.""" client = TestClient(app) # Test in description field that might be rendered payload = { "name": "test-tool", "url": "http://example.com", "description": potentially_malicious } response = client.post( "/admin/tools", json=payload, headers={"Authorization": "Basic YWRtaW46Y2hhbmdlbWU="} ) # Should handle potentially malicious content safely assert response.status_code in [200, 201, 400, 422] if response.status_code in [200, 201]: # If accepted, should be properly escaped in responses tool_response = client.get( "/admin/tools", headers={"Authorization": "Basic YWRtaW46Y2hhbmdlbWU="} ) # Raw HTML tags should not appear in JSON responses assert "<script>" not in tool_response.text.lower() @given(st.integers(min_value=-2**31, max_value=2**31)) def test_integer_overflow_handling(self, large_int): """Test handling of integer overflow in numeric fields.""" client = TestClient(app) # Test in ID fields and numeric parameters response = client.get( f"/admin/tools/{large_int}", headers={"Authorization": "Basic YWRtaW46Y2hhbmdlbWU="} ) # Should handle large integers gracefully assert response.status_code in [200, 400, 404, 422]
-
Corpus management and reporting
# scripts/generate_fuzz_report.py #!/usr/bin/env python3 """Generate comprehensive fuzzing report.""" import json import os from pathlib import Path from datetime import datetime def generate_report(): """Generate comprehensive fuzzing report.""" report = { "timestamp": datetime.now().isoformat(), "summary": {}, "tools": {}, "findings": [], "corpus_stats": {} } # Hypothesis statistics if Path("reports/hypothesis-stats.json").exists(): with open("reports/hypothesis-stats.json") as f: report["tools"]["hypothesis"] = json.load(f) # Atheris results atheris_results = Path("fuzzers/results") if atheris_results.exists(): report["tools"]["atheris"] = { "artifacts": len(list(atheris_results.glob("*"))), "results_dir": str(atheris_results) } # Schemathesis results if Path("reports/schemathesis-report.json").exists(): with open("reports/schemathesis-report.json") as f: report["tools"]["schemathesis"] = json.load(f) # RESTler results restler_results = Path("restler_results") if restler_results.exists(): report["tools"]["restler"] = { "findings": len(list(restler_results.glob("**/bug_buckets/*"))), "results_dir": str(restler_results) } # Corpus statistics corpus_dir = Path("corpus") if corpus_dir.exists(): for subdir in corpus_dir.iterdir(): if subdir.is_dir(): report["corpus_stats"][subdir.name] = len(list(subdir.glob("*"))) # Write report report_file = Path("reports/fuzz-report.json") report_file.parent.mkdir(exist_ok=True) with open(report_file, "w") as f: json.dump(report, f, indent=2) print(f"📊 Fuzzing report generated: {report_file}") # Print summary print("\n🎯 Fuzzing Summary:") for tool, results in report["tools"].items(): print(f" {tool}: {len(results) if isinstance(results, list) else 'completed'}") if __name__ == "__main__": generate_report()
-
CI/CD integration
# .github/workflows/fuzz-testing.yml name: Fuzz Testing on: schedule: - cron: '0 2 * * *' # Nightly at 2 AM push: branches: [main] pull_request: branches: [main] jobs: fuzz-quick: name: Quick Fuzzing (PR) runs-on: ubuntu-latest if: github.event_name == 'pull_request' steps: - name: ⬇️ Checkout source uses: actions/checkout@v4 - name: 🐍 Set up Python uses: actions/setup-python@v5 with: python-version: "3.12" cache: pip - name: 📦 Install dependencies run: | python -m pip install --upgrade pip pip install -e .[dev] make fuzz-install - name: ⚡ Run quick fuzzing run: make fuzz-quick timeout-minutes: 10 - name: 📊 Upload results uses: actions/upload-artifact@v4 if: always() with: name: quick-fuzz-results path: reports/ fuzz-comprehensive: name: Comprehensive Fuzzing (Nightly) runs-on: ubuntu-latest if: github.event_name == 'schedule' || github.event_name == 'push' strategy: matrix: tool: [hypothesis, atheris, api, security] steps: - name: ⬇️ Checkout source uses: actions/checkout@v4 - name: 🐍 Set up Python uses: actions/setup-python@v5 with: python-version: "3.12" cache: pip - name: 📦 Install dependencies run: | python -m pip install --upgrade pip pip install -e .[dev] make fuzz-install - name: 🧪 Run fuzzing - ${{ matrix.tool }} run: make fuzz-${{ matrix.tool }} timeout-minutes: 60 - name: 📊 Upload results uses: actions/upload-artifact@v4 if: always() with: name: fuzz-results-${{ matrix.tool }} path: | reports/ fuzzers/results/ restler_results/ corpus/ fuzz-report: name: Generate Fuzz Report runs-on: ubuntu-latest needs: [fuzz-comprehensive] if: github.event_name == 'schedule' || github.event_name == 'push' steps: - name: ⬇️ Checkout source uses: actions/checkout@v4 - name: 📥 Download all artifacts uses: actions/download-artifact@v4 with: path: artifacts/ - name: 📊 Generate comprehensive report run: | python scripts/generate_fuzz_report.py cat reports/fuzz-report.json - name: 🚨 Check for critical findings run: | python -c " import json with open('reports/fuzz-report.json') as f: report = json.load(f) critical_findings = [f for f in report.get('findings', []) if f.get('severity') == 'critical'] if critical_findings: print(f'🚨 Found {len(critical_findings)} critical findings!') exit(1) else: print('✅ No critical findings detected') "
-
Documentation and best practices
# docs/fuzzing.md """ # Fuzz Testing Guide ## Overview This document describes the comprehensive fuzz testing setup for MCP Gateway. ## Tools Used - **Hypothesis**: Property-based testing for pure Python logic - **Atheris**: Coverage-guided fuzzing for native code paths - **Schemathesis**: OpenAPI schema-driven API fuzzing - **RESTler**: Stateful security-focused API fuzzing ## Running Fuzzing Tests ### Quick Fuzzing (CI/PR) ```bash make fuzz-quick
Full Fuzzing Suite
make fuzz-all
Individual Tools
make fuzz-hypothesis # Property-based testing make fuzz-atheris # Coverage-guided fuzzing make fuzz-api # API schema fuzzing make fuzz-security # Security fuzzing
Interpreting Results
Hypothesis Failures
- Review minimal failing examples
- Add them as regression tests
- Fix underlying logic issues
Atheris Crashes
- Examine crash artifacts in
fuzzers/results/ - Use
addr2linefor C stack traces - Report upstream if in dependencies
Schemathesis Failures
- Check for 5xx responses in unexpected cases
- Verify OpenAPI schema accuracy
- Improve input validation
RESTler Security Issues
- Review bug buckets in
restler_results/ - Prioritize authentication bypasses
- Fix injection vulnerabilities
"""
📖 References
- Hypothesis Documentation – Property-based testing framework · https://hypothesis.readthedocs.io/
- Atheris Fuzzer – Coverage-guided Python fuzzing · https://github.com/google/atheris
- Schemathesis – OpenAPI-driven API testing · https://github.com/schemathesis/schemathesis
- RESTler Fuzzer – Stateful REST API fuzzing · https://github.com/microsoft/restler-fuzzer
- OSS-Fuzz Integration – Continuous fuzzing · https://google.github.io/oss-fuzz/
- Fuzzing Best Practices – Security testing guide · https://owasp.org/www-community/Fuzzing
🧩 Additional Notes
- Start incrementally: Begin with Hypothesis tests for core validation logic, then expand to native code and API fuzzing.
- Corpus management: Save interesting test cases that uncover edge cases for regression testing and faster fuzzing.
- Security focus: Prioritize fuzzing authentication, input validation, and data processing paths for security vulnerabilities.
- Performance awareness: Monitor fuzzing overhead and use timeouts to prevent resource exhaustion.
- CI integration: Run quick fuzzing on PRs and comprehensive fuzzing nightly to balance speed and coverage.
- Artifact preservation: Save crashes, timeouts, and security findings for detailed analysis and bug reporting.
- Regression prevention: Convert fuzzing discoveries into permanent test cases to prevent reintroduction.
- Documentation updates: Keep fuzzing documentation current with tool configurations and interpretation guides.
Fuzz Testing Best Practices for MCP Gateway:
- Focus on JSON-RPC validation, JSONPath processing, and schema validation as high-value targets
- Use property-based testing to validate invariants in core business logic
- Combine multiple fuzzing approaches for comprehensive coverage
- Maintain seed corpora for consistent and fast fuzzing runs
- Integrate security-focused fuzzing to catch injection and authentication bypasses
- Monitor fuzzing metrics and adjust timeouts and coverage targets based on findings
- Document all critical findings and ensure they become regression tests