Vendor Agnostic OpenTelemetry Observability Support #735 and #727 phoenix (#733)

* Phoenix poc

Signed-off-by: Mihai Criveti <[email protected]>

* Phoenix PoC

Signed-off-by: Mihai Criveti <[email protected]>

* Observability

Signed-off-by: Mihai Criveti <[email protected]>

* Observability

Signed-off-by: Mihai Criveti <[email protected]>

* Observability

Signed-off-by: Mihai Criveti <[email protected]>

* Observability

Signed-off-by: Mihai Criveti <[email protected]>

* Observability

Signed-off-by: Mihai Criveti <[email protected]>

* Observability

Signed-off-by: Mihai Criveti <[email protected]>

* README

Signed-off-by: Mihai Criveti <[email protected]>

* Simplify docs

Signed-off-by: Mihai Criveti <[email protected]>

* Simplify tests

Signed-off-by: Mihai Criveti <[email protected]>

---------

Signed-off-by: Mihai Criveti <[email protected]>

Author: crivetimihai

.env.example

@@ -347,3 +347,45 @@ VALID_SLUG_SEPARATOR_REGEXP= r"^(-{1,2}|[_.])$"
 # Plugins Settings
 #####################################
 PLUGINS_ENABLED=false
+
+#####################################
+# Observability (OpenTelemetry)
+#####################################
+
+# Master switch for observability (true/false)
+OTEL_ENABLE_OBSERVABILITY=true
+
+# Service identification
+OTEL_SERVICE_NAME=mcp-gateway
+OTEL_SERVICE_VERSION=0.5.0
+OTEL_DEPLOYMENT_ENVIRONMENT=development
+
+# Exporter type: otlp, jaeger, zipkin, console, none
+OTEL_TRACES_EXPORTER=otlp
+
+# OTLP Configuration (for Phoenix, Tempo, DataDog, etc.)
+OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
+OTEL_EXPORTER_OTLP_PROTOCOL=grpc
+# Headers for authentication (format: key=value,key2=value2)
+#OTEL_EXPORTER_OTLP_HEADERS=api-key=secret
+OTEL_EXPORTER_OTLP_INSECURE=true
+
+# Jaeger Configuration (alternative to OTLP)
+#OTEL_EXPORTER_JAEGER_ENDPOINT=http://localhost:14268/api/traces
+#OTEL_EXPORTER_JAEGER_USER=
+#OTEL_EXPORTER_JAEGER_PASSWORD=
+
+# Zipkin Configuration (alternative to OTLP)
+#OTEL_EXPORTER_ZIPKIN_ENDPOINT=http://localhost:9411/api/v2/spans
+
+# Sampling Configuration
+OTEL_TRACES_SAMPLER=parentbased_traceidratio
+OTEL_TRACES_SAMPLER_ARG=0.1
+
+# Resource Attributes (comma-separated key=value pairs)
+#OTEL_RESOURCE_ATTRIBUTES=tenant.id=acme,region=us-east-1
+
+# Performance Tuning
+OTEL_BSP_MAX_QUEUE_SIZE=2048
+OTEL_BSP_MAX_EXPORT_BATCH_SIZE=512
+OTEL_BSP_SCHEDULE_DELAY=5000

CLAUDE.md

@@ -234,6 +234,7 @@ make doctest test htmlcov smoketest lint-web flake8 bandit interrogate pylint ve
 
 # Rules
 - When using git commit always add a -s to sign commits
+- Don't include effor estimates, or 'phases'
 
 # TO test individual files, ensure you're activated the env first, ex:
 . /home/cmihai/.venv/mcpgateway/bin/activate && pytest --cov-report=annotate tests/unit/mcpgateway/test_translate.py

Containerfile

@@ -21,9 +21,10 @@ WORKDIR /app
 COPY . /app
 
 # Create virtual environment, upgrade pip and install dependencies using uv for speed
+# Including observability packages for OpenTelemetry support
 RUN python3 -m venv /app/.venv && \
     /app/.venv/bin/python3 -m pip install --upgrade pip setuptools pdm uv && \
-    /app/.venv/bin/python3 -m uv pip install ".[redis,postgres,alembic]"
+    /app/.venv/bin/python3 -m uv pip install ".[redis,postgres,alembic,observability]"
 
 # update the user permissions
 RUN chown -R 1001:0 /app && \

Containerfile.lite

@@ -61,13 +61,14 @@ COPY pyproject.toml /app/
 # Create and populate virtual environment
 #  - Upgrade pip, setuptools, wheel, pdm, uv
 #  - Install project dependencies and package
+#  - Include observability packages for OpenTelemetry support
 #  - Remove build tools but keep runtime dist-info
 #  - Remove build caches and build artifacts
 # ----------------------------------------------------------------------------
 RUN set -euo pipefail \
     && python3 -m venv /app/.venv \
     && /app/.venv/bin/pip install --no-cache-dir --upgrade pip setuptools wheel pdm uv \
-    && /app/.venv/bin/uv pip install ".[redis,postgres]" \
+    && /app/.venv/bin/uv pip install ".[redis,postgres,observability]" \
     && /app/.venv/bin/pip uninstall --yes uv pip setuptools wheel pdm \
     && rm -rf /root/.cache /var/cache/dnf \
     && find /app/.venv -name "*.dist-info" -type d \

README.md

@@ -128,7 +128,8 @@ It currently supports:
 * Virtualization of legacy APIs as MCP-compliant tools and servers
 * Transport over HTTP, JSON-RPC, WebSocket, SSE (with configurable keepalive), stdio and streamable-HTTP
 * An Admin UI for real-time management and configuration
-* Built-in auth, observability, retries, and rate-limiting
+* Built-in auth, retries, and rate-limiting
+* **OpenTelemetry observability** with Phoenix, Jaeger, Zipkin, and other OTLP backends
 * Scalable deployments via Docker or PyPI, Redis-backed caching, and multi-cluster federation
 
 ![MCP Gateway Architecture](https://ibm.github.io/mcp-context-forge/images/mcpgateway.svg)
@@ -195,6 +196,35 @@ For a list of upcoming features, check out the [ContextForge MCP Gateway Roadmap
 
 </details>
 
+<details>
+<summary><strong>🔍 OpenTelemetry Observability</strong></summary>
+
+* **Vendor-agnostic tracing** with OpenTelemetry (OTLP) protocol support
+* **Multiple backend support**: Phoenix (LLM-focused), Jaeger, Zipkin, Tempo, DataDog, New Relic
+* **Distributed tracing** across federated gateways and services
+* **Automatic instrumentation** of tools, prompts, resources, and gateway operations
+* **LLM-specific metrics**: Token usage, costs, model performance
+* **Zero-overhead when disabled** with graceful degradation
+* **Easy configuration** via environment variables
+
+Quick start with Phoenix (LLM observability):
+```bash
+# Start Phoenix
+docker run -p 6006:6006 -p 4317:4317 arizephoenix/phoenix:latest
+
+# Configure gateway
+export OTEL_ENABLE_OBSERVABILITY=true
+export OTEL_TRACES_EXPORTER=otlp
+export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
+
+# Run gateway - traces automatically sent to Phoenix
+mcpgateway
+```
+
+See [Observability Documentation](https://ibm.github.io/mcp-context-forge/manage/observability/) for detailed setup with other backends.
+
+</details>
+
 ---
 
 ## Quick Start - PyPI
@@ -1083,6 +1113,65 @@ LOG_FILE=gateway.log
 - File logging is **disabled by default** (no files created)
 - Set `LOG_TO_FILE=true` to enable optional file logging with JSON format
 
+### Observability (OpenTelemetry)
+
+MCP Gateway includes **vendor-agnostic OpenTelemetry support** for distributed tracing. Works with Phoenix, Jaeger, Zipkin, Tempo, DataDog, New Relic, and any OTLP-compatible backend.
+
+| Setting                         | Description                                    | Default               | Options                                    |
+| ------------------------------- | ---------------------------------------------- | --------------------- | ------------------------------------------ |
+| `OTEL_ENABLE_OBSERVABILITY`     | Master switch for observability               | `true`                | `true`, `false`                           |
+| `OTEL_SERVICE_NAME`             | Service identifier in traces                   | `mcp-gateway`         | string                                     |
+| `OTEL_SERVICE_VERSION`          | Service version in traces                      | `0.5.0`               | string                                     |
+| `OTEL_DEPLOYMENT_ENVIRONMENT`   | Environment tag (dev/staging/prod)            | `development`         | string                                     |
+| `OTEL_TRACES_EXPORTER`          | Trace exporter backend                         | `otlp`                | `otlp`, `jaeger`, `zipkin`, `console`, `none` |
+| `OTEL_RESOURCE_ATTRIBUTES`      | Custom resource attributes                     | (empty)               | `key=value,key2=value2`                   |
+
+**OTLP Configuration** (for Phoenix, Tempo, DataDog, etc.):
+
+| Setting                         | Description                                    | Default               | Options                                    |
+| ------------------------------- | ---------------------------------------------- | --------------------- | ------------------------------------------ |
+| `OTEL_EXPORTER_OTLP_ENDPOINT`   | OTLP collector endpoint                        | (none)                | `http://localhost:4317`                   |
+| `OTEL_EXPORTER_OTLP_PROTOCOL`   | OTLP protocol                                  | `grpc`                | `grpc`, `http/protobuf`                   |
+| `OTEL_EXPORTER_OTLP_HEADERS`    | Authentication headers                         | (empty)               | `api-key=secret,x-auth=token`             |
+| `OTEL_EXPORTER_OTLP_INSECURE`   | Skip TLS verification                          | `true`                | `true`, `false`                           |
+
+**Alternative Backends** (optional):
+
+| Setting                         | Description                                    | Default               | Options                                    |
+| ------------------------------- | ---------------------------------------------- | --------------------- | ------------------------------------------ |
+| `OTEL_EXPORTER_JAEGER_ENDPOINT` | Jaeger collector endpoint                      | `http://localhost:14268/api/traces` | URL                             |
+| `OTEL_EXPORTER_ZIPKIN_ENDPOINT` | Zipkin collector endpoint                      | `http://localhost:9411/api/v2/spans` | URL                            |
+
+**Performance Tuning**:
+
+| Setting                         | Description                                    | Default               | Options                                    |
+| ------------------------------- | ---------------------------------------------- | --------------------- | ------------------------------------------ |
+| `OTEL_TRACES_SAMPLER`           | Sampling strategy                              | `parentbased_traceidratio` | `always_on`, `always_off`, `traceidratio` |
+| `OTEL_TRACES_SAMPLER_ARG`       | Sample rate (0.0-1.0)                         | `0.1`                 | float (0.1 = 10% sampling)                |
+| `OTEL_BSP_MAX_QUEUE_SIZE`       | Max queued spans                              | `2048`                | int > 0                                    |
+| `OTEL_BSP_MAX_EXPORT_BATCH_SIZE`| Max batch size for export                     | `512`                 | int > 0                                    |
+| `OTEL_BSP_SCHEDULE_DELAY`       | Export interval (ms)                          | `5000`                | int > 0                                    |
+
+**Quick Start with Phoenix**:
+```bash
+# Start Phoenix for LLM observability
+docker run -p 6006:6006 -p 4317:4317 arizephoenix/phoenix:latest
+
+# Configure gateway
+export OTEL_ENABLE_OBSERVABILITY=true
+export OTEL_TRACES_EXPORTER=otlp
+export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
+
+# Run gateway - traces automatically sent to Phoenix
+mcpgateway
+```
+
+> 🔍 **What Gets Traced**: Tool invocations, prompt rendering, resource fetching, gateway federation, health checks, plugin execution (if enabled)
+>
+> 🚀 **Zero Overhead**: When `OTEL_ENABLE_OBSERVABILITY=false`, all tracing is disabled with no performance impact
+>
+> 📊 **View Traces**: Phoenix UI at `http://localhost:6006`, Jaeger at `http://localhost:16686`, or your configured backend
+
 ### Transport
 
 | Setting                   | Description                        | Default | Options                         |

docker-compose.phoenix-simple.yml

@@ -1,5 +1,5 @@
 # Simplified Phoenix Observability Stack for MCP Gateway
-# 
+#
 # Usage:
 #   Start Phoenix: docker-compose -f docker-compose.phoenix-simple.yml up -d
 #   Stop Phoenix: docker-compose -f docker-compose.phoenix-simple.yml down
@@ -34,4 +34,4 @@ services:
 
 volumes:
   phoenix-data:
-    driver: local
+    driver: local

docker-compose.with-phoenix.yml

@@ -44,4 +44,4 @@ services:
 
 volumes:
   phoenix-data:
-    driver: local
+    driver: local

docs/docs/architecture/security-features.md

@@ -253,33 +253,33 @@ MCP Gateway implements a comprehensive, multi-layered security approach with "de
 
 ### 🚀 Upcoming Security Enhancements
 
-**Release 0.5.0 (August 2025)**
+**Release 0.5.0 - August 2025**
 - Enhanced authentication mechanisms
 - Configuration validation framework
 - Comprehensive audit logging
 - Security headers implementation
 
-**Release 0.6.0 (August 2025)**
+**Release 0.6.0 - August 2025**
 - Database-backed authentication
 - Multi-layer caching security
 - Circuit breakers implementation
 
-**Release 0.7.0 (September 2025)**
+**Release 0.7.0 - September 2025**
 - Full RBAC implementation
 - Multi-tenancy support
 - Correlation ID tracking
 
-**Release 0.8.0 (September 2025)**
+**Release 0.8.0 - September 2025**
 - Policy-as-Code engine
 - Advanced guardrails
 - DDoS protection
 
-**Release 0.9.0 (September 2025)**
+**Release 0.9.0 - September 2025**
 - Marketplace security
 - Protocol negotiation
 - Advanced connectivity
 
-**Release 1.0.0 (October 2025)**
+**Release 1.0.0 - October 2025**
 - Security audit completion
 - Production hardening
 - GA security certification

docs/docs/manage/observability.md

@@ -0,0 +1,25 @@
+# Observability
+
+MCP Gateway includes production-grade OpenTelemetry instrumentation for distributed tracing, enabling you to monitor performance, debug issues, and understand request flows.
+
+## Documentation
+
+- **[Observability Overview](observability/observability.md)** - Complete guide to configuring and using observability
+- **[Phoenix Integration](observability/phoenix.md)** - AI/LLM-focused observability with Arize Phoenix
+
+## Quick Start
+
+```bash
+# Enable observability (enabled by default)
+export OTEL_ENABLE_OBSERVABILITY=true
+export OTEL_TRACES_EXPORTER=otlp
+export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
+
+# Start Phoenix for AI/LLM observability
+docker run -p 6006:6006 -p 4317:4317 arizephoenix/phoenix:latest
+
+# Run MCP Gateway
+mcpgateway
+```
+
+View traces at http://localhost:6006

docs/docs/manage/observability/.pages

@@ -1,2 +1,3 @@
 nav:
-  - Phoenix: phoenix-deployment.md
+  - Overview: observability.md
+  - Phoenix Integration: phoenix.md