Vendor Agnostic OpenTelemetry Observability Support #735 and #727 phoenix

[crivetimihai]

Observability Implementation Status

Executive Summary

Implemented vendor-agnostic OpenTelemetry observability for MCP Gateway with support for multiple backends (Phoenix, Jaeger, Zipkin, Tempo, DataDog, etc.). The implementation provides distributed tracing across all core services with zero overhead when disabled.

Closes #735 #727

What Was Implemented

1. Core Observability Module (mcpgateway/observability.py)

Features Added:

• ✅ Vendor-agnostic OpenTelemetry SDK integration

  • Support for OTLP, Jaeger, Zipkin, Console exporters
  • Automatic fallback from gRPC to HTTP exporters
  • Graceful degradation when dependencies missing

• ✅ Configuration via environment variables

  OTEL_ENABLE_OBSERVABILITY=true/false
  OTEL_TRACES_EXPORTER=otlp|jaeger|zipkin|console|none
  OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
  OTEL_SERVICE_NAME=mcp-gateway
  OTEL_RESOURCE_ATTRIBUTES=key=value,key2=value2

• ✅ Tracing utilities

  • init_telemetry() - Initialize tracer with backend configuration
  • create_span() - Context manager for manual instrumentation
  • trace_operation() - Decorator for automatic function tracing

• ✅ Performance optimizations

  • BatchSpanProcessor for efficient export (except console)
  • Configurable queue size and batch settings
  • Zero overhead when disabled

Code Quality Improvements:

• Renamed from observability_simple.py to observability.py (it's production-grade!)
• Changed global tracer to _TRACER following Python conventions
• Achieved pylint score: 10.00/10
• 100% docstring coverage
• Comprehensive error handling

2. Service Instrumentation

Tool Service (mcpgateway/services/tool_service.py)

• ✅ Added create_span("tool.invoke") with attributes:
  • tool.name, tool.id, tool.integration_type
  • tool.gateway_id, arguments_count, has_headers
  • success, error, error.message
  • duration.ms
• ✅ Error tracking and exception capture
• ✅ Success/failure status tracking

Prompt Service (mcpgateway/services/prompt_service.py)

• ✅ Added create_span("prompt.render") with attributes:
  • prompt.name, arguments_count
  • user, server_id, tenant_id, request_id
  • messages.count (after rendering)
  • success, duration.ms
• ✅ Plugin hook tracking
• ✅ Error handling with span attributes

Resource Service (mcpgateway/services/resource_service.py)

• ✅ Added create_span("resource.read") with attributes:
  • resource.uri, user, server_id, request_id
  • http.url (for HTTP resources)
  • resource.type (template vs static)
  • content.size
  • success, duration.ms

Gateway Service (mcpgateway/services/gateway_service.py)

• ✅ Added create_span("gateway.forward_request") for federation
  • Full RPC method and peer service tracking
  • HTTP status codes and response handling
• ✅ Added create_span("gateway.health_check_batch") with nested spans
  • Individual health check spans per gateway
  • Success/failure tracking per peer

3. Configuration & Environment

Added to .env.example:

# Observability (OpenTelemetry)
OTEL_ENABLE_OBSERVABILITY=true
OTEL_SERVICE_NAME=mcp-gateway
OTEL_SERVICE_VERSION=0.5.0
OTEL_DEPLOYMENT_ENVIRONMENT=development
OTEL_TRACES_EXPORTER=otlp
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_EXPORTER_OTLP_INSECURE=true
# Sampling Configuration
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1
# Performance Tuning
OTEL_BSP_MAX_QUEUE_SIZE=2048
OTEL_BSP_MAX_EXPORT_BATCH_SIZE=512
OTEL_BSP_SCHEDULE_DELAY=5000

Added to pyproject.toml:

[project.optional-dependencies]
observability = [
    "opentelemetry-api>=1.36.0",
    "opentelemetry-sdk>=1.36.0",
    "opentelemetry-exporter-otlp>=1.36.0",
    "opentelemetry-exporter-jaeger>=1.36.0",  # Optional
    "opentelemetry-exporter-zipkin>=1.36.0",  # Optional
]

4. Docker & Deployment

Created docker-compose.phoenix-simple.yml:

• Phoenix container with ports 6006 (UI) and 4317 (OTLP)
• Auto-configuration for MCP Gateway connection
• Volume persistence for Phoenix data

Created serve-with-tracing.sh:

#!/bin/bash
# Helper script supporting multiple backends
./serve-with-tracing.sh phoenix  # or jaeger, zipkin, tempo, console

5. Documentation

Created/Updated:

• ✅ docs/docs/manage/observability.md - Main documentation
• ✅ docs/docs/manage/observability/phoenix-quickstart.md - Phoenix setup guide
• ✅ docs/docs/manage/observability/phoenix-deployment.md - Production deployment
• ✅ Updated README.md with observability section
• ✅ Updated CLAUDE.md with observability guidance

6. Testing

Created tests/unit/mcpgateway/test_observability.py:

• 18 test cases covering:
  • Initialization with different exporters
  • Environment variable parsing
  • Error handling
  • Decorator functionality
  • Context manager behavior
• All tests passing
• 84% code coverage for observability module

Created test_phoenix_integration.py:

• Integration test script for Phoenix
• Sends sample traces for verification
• Tests nested spans and error scenarios

What Changed

Breaking Changes

• None - all changes are backward compatible

Module Renames

• observability_simple.py → observability.py
• test_observability_simple.py → test_observability.py

Import Changes

All services now import:

from mcpgateway.observability import create_span

Configuration Changes

• Observability is enabled by default but fails gracefully
• No configuration required for basic operation
• OTLP endpoint must be configured for trace export

Performance Impact

When Disabled

• Zero overhead - no-op context managers returned
• No memory allocation for spans
• No background threads

When Enabled

• Minimal overhead: ~0.1-0.5ms per span
• Batch processing: Spans exported in batches
• Async operation: No blocking of main execution
• Memory efficient: Configurable queue limits

Compatibility

Tested Backends

• ✅ Phoenix (Arize) - LLM-focused observability
• ✅ Jaeger - Distributed tracing
• ✅ Zipkin - Distributed tracing
• ✅ Console - Development/debugging
• ✅ OTLP - Generic (Tempo, DataDog, New Relic, etc.)

Python Compatibility

• Requires Python ≥ 3.10
• Tested with Python 3.11, 3.12

Known Limitations

1. No Metrics Yet - Only tracing implemented, metrics planned
2. No Log Correlation - Trace IDs not injected into logs yet
3. Limited LLM Tracking - Token usage not tracked yet
4. No Sampling Strategies - Only basic ratio sampling
5. No PII Filtering - Sensitive data not sanitized

Security Considerations

• ✅ TLS support via OTEL_EXPORTER_OTLP_INSECURE=false
• ✅ Authentication via headers (OTEL_EXPORTER_OTLP_HEADERS)
• ⚠️ No PII filtering implemented yet
• ⚠️ All arguments/responses captured in spans

Migration Notes

For Existing Deployments

1. No action required - observability is optional
2. To enable: Set environment variables and deploy backend
3. To disable: Set OTEL_ENABLE_OBSERVABILITY=false

For Developers

1. Use create_span() for new service methods
2. Always set error and success attributes
3. Include user/tenant context when available
4. Measure duration for performance tracking

Future Enhancements

• Metrics collection (counters, histograms)
• Token usage tracking for LLM operations
• Advanced sampling strategies
• Log correlation with trace IDs
• PII filtering and data sanitization

References

• OpenTelemetry Specification
• Issue #735 - Production-Grade Observability
• Issue #175 - OpenLLMetry Integration
• Issue #727 - Phoenix Integration
• PR #733 - Implementation PR

[MohanLaksh]

PR TEST SUMMARY:

make serve - pass

make test - PASS

make autoflake isort black flake8 - PASS no errors

make pylint - PASS

• Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

make smoketest - PASS

• ✅ Smoketest passed!

make doctest - PASS

[TS0713]

PR TEST SUMMARY:

make serve - pass

make test - PASS

make autoflake isort black flake8 - PASS no errors

make pylint - PASS

• Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

make smoketest - PASS

• ✅ Smoketest passed!

make doctest - PASS

[crivetimihai]

Thank you for testing all, merging!