Skip to content

Commit 49b5151

Browse files
crivetimihaimadhav165
authored and
rakdutta1
committed
137 metadata timestamps (IBM#776)
* Metadata / creation dates Signed-off-by: Mihai Criveti <[email protected]> * Metadata / creation dates Signed-off-by: Mihai Criveti <[email protected]> * Metadata / creation dates Signed-off-by: Mihai Criveti <[email protected]> * Security headers CSP Signed-off-by: Mihai Criveti <[email protected]> * Display metadata for resources Signed-off-by: Madhav Kandukuri <[email protected]> * eslint fix Signed-off-by: Madhav Kandukuri <[email protected]> --------- Signed-off-by: Mihai Criveti <[email protected]> Co-authored-by: Madhav Kandukuri <[email protected]>
1 parent 263f170 commit 49b5151

File tree

21 files changed

+2406
-90
lines changed

21 files changed

+2406
-90
lines changed

‎docs/docs/manage/.pages

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ nav:
22
- index.md
33
- backup.md
44
- bulk-import.md
5+
- metadata-tracking.md
56
- export-import.md
67
- export-import-tutorial.md
78
- export-import-reference.md

‎docs/docs/manage/index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ Whether you're self-hosting, running in the cloud, or deploying to Kubernetes, t
1515
| [Export/Import Tutorial](export-import-tutorial.md) | Step-by-step tutorial for getting started with export/import |
1616
| [Export/Import Reference](export-import-reference.md) | Quick reference guide for export/import commands and APIs |
1717
| [Bulk Import](bulk-import.md) | Import multiple tools at once for migrations and team onboarding |
18+
| [Metadata Tracking](metadata-tracking.md) | 📊 **NEW** - Comprehensive audit trails and entity metadata tracking |
1819
| [Well-Known URIs](well-known-uris.md) | Configure robots.txt, security.txt, and custom well-known files |
1920
| [Logging](logging.md) | Configure structured logging, log destinations, and log rotation |
2021

‎docs/docs/manage/metadata-tracking.md

Lines changed: 353 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,353 @@
1+
# 📊 Metadata Tracking & Audit Trails
2+
3+
MCP Gateway provides comprehensive metadata tracking for all entities (Tools, Resources, Prompts, Servers, Gateways) to enable enterprise-grade audit trails, compliance monitoring, and operational troubleshooting.
4+
5+
---
6+
7+
## 🎯 **Overview**
8+
9+
Every entity in MCP Gateway now includes detailed metadata about:
10+
- **Who** created or modified the entity
11+
- **When** the operation occurred
12+
- **From where** (IP address, user agent)
13+
- **How** it was created (UI, API, bulk import, federation)
14+
- **Source tracking** for federated entities and bulk operations
15+
16+
---
17+
18+
## 📊 **Metadata Fields**
19+
20+
All entities include the following metadata fields:
21+
22+
| Category | Field | Description | Example Values |
23+
|----------|-------|-------------|----------------|
24+
| **Creation** | `created_by` | Username who created entity | `"admin"`, `"alice"`, `"anonymous"` |
25+
| | `created_at` | Creation timestamp | `"2024-01-15T10:30:00Z"` |
26+
| | `created_from_ip` | IP address of creator | `"192.168.1.100"`, `"10.0.0.1"` |
27+
| | `created_via` | Creation method | `"ui"`, `"api"`, `"import"`, `"federation"` |
28+
| | `created_user_agent` | Browser/client info | `"Mozilla/5.0"`, `"curl/7.68.0"` |
29+
| **Modification** | `modified_by` | Last modifier username | `"bob"`, `"system"`, `"anonymous"` |
30+
| | `modified_at` | Last modification timestamp | `"2024-01-16T14:22:00Z"` |
31+
| | `modified_from_ip` | IP of last modifier | `"172.16.0.1"` |
32+
| | `modified_via` | Modification method | `"ui"`, `"api"` |
33+
| | `modified_user_agent` | Client of last change | `"HTTPie/2.4.0"` |
34+
| **Source** | `import_batch_id` | Bulk import UUID | `"550e8400-e29b-41d4-a716-446655440000"` |
35+
| | `federation_source` | Source gateway name | `"gateway-prod-east"` |
36+
| | `version` | Change tracking version | `1`, `2`, `3`... |
37+
38+
---
39+
40+
## 🖥️ **Viewing Metadata**
41+
42+
### **Admin UI**
43+
44+
Metadata is displayed in the detail view modals for all entity types:
45+
46+
1. **Navigate** to any entity list (Tools, Resources, Prompts, Servers, Gateways)
47+
2. **Click "View"** on any entity
48+
3. **Scroll down** to the "Metadata" section
49+
50+
**Example metadata display:**
51+
```
52+
┌─ Metadata ──────────────────────────────────────┐
53+
│ Created By: admin │
54+
│ Created At: 1/15/2024, 10:30:00 AM │
55+
│ Created From: 192.168.1.100 │
56+
│ Created Via: ui │
57+
│ Last Modified By: alice │
58+
│ Last Modified At: 1/16/2024, 2:22:00 PM │
59+
│ Version: 3 │
60+
│ Import Batch: N/A │
61+
└─────────────────────────────────────────────────┘
62+
```
63+
64+
### **API Responses**
65+
66+
All entity read endpoints include metadata fields in JSON responses:
67+
68+
```bash
69+
# Get tool with metadata
70+
curl -H "Authorization: Bearer $TOKEN" \
71+
http://localhost:4444/tools/abc123
72+
73+
{
74+
"id": "abc123",
75+
"name": "example_tool",
76+
"description": "Example tool",
77+
"createdBy": "admin",
78+
"createdAt": "2024-01-15T10:30:00Z",
79+
"createdFromIp": "192.168.1.100",
80+
"createdVia": "ui",
81+
"createdUserAgent": "Mozilla/5.0...",
82+
"modifiedBy": "alice",
83+
"modifiedAt": "2024-01-16T14:22:00Z",
84+
"version": 3,
85+
"importBatchId": null,
86+
"federationSource": null,
87+
...
88+
}
89+
```
90+
91+
---
92+
93+
## 🔍 **Metadata by Source Type**
94+
95+
### **Manual Creation (UI/API)**
96+
- `created_via`: `"ui"` or `"api"`
97+
- `created_by`: Authenticated username
98+
- `created_from_ip`: Client IP address
99+
- `federation_source`: `null`
100+
- `import_batch_id`: `null`
101+
102+
### **Bulk Import Operations**
103+
- `created_via`: `"import"`
104+
- `import_batch_id`: UUID linking related imports
105+
- `created_by`: User who initiated import
106+
- `federation_source`: `null`
107+
108+
### **Federation (MCP Server Discovery)**
109+
- `created_via`: `"federation"`
110+
- `federation_source`: Source gateway name
111+
- `created_by`: User who registered the gateway
112+
- `import_batch_id`: `null`
113+
114+
### **Legacy Entities (Pre-Metadata)**
115+
- All metadata fields: `null`
116+
- UI displays: `"Legacy Entity"`, `"Pre-metadata"`
117+
- `version`: `1` (automatically assigned)
118+
119+
---
120+
121+
## 🛡️ **Authentication Compatibility**
122+
123+
Metadata tracking works seamlessly across all authentication modes:
124+
125+
### **With Authentication (`AUTH_REQUIRED=true`)**
126+
```bash
127+
# Example: User "admin" creates a tool
128+
{
129+
"createdBy": "admin",
130+
"createdVia": "api",
131+
"createdFromIp": "192.168.1.100"
132+
}
133+
```
134+
135+
### **Without Authentication (`AUTH_REQUIRED=false`)**
136+
```bash
137+
# Example: Anonymous creation
138+
{
139+
"createdBy": "anonymous",
140+
"createdVia": "api",
141+
"createdFromIp": "192.168.1.100"
142+
}
143+
```
144+
145+
### **JWT vs Basic Authentication**
146+
- **JWT Authentication**: Extracts username from token payload (`username` or `sub` field)
147+
- **Basic Authentication**: Uses provided username directly
148+
- **Both formats handled gracefully** by the `extract_username()` utility
149+
150+
---
151+
152+
## 🔄 **Version Tracking**
153+
154+
Each entity maintains a version number that increments on modifications:
155+
156+
```bash
157+
# Initial creation
158+
POST /tools -> version: 1
159+
160+
# First update
161+
PUT /tools/123 -> version: 2
162+
163+
# Second update
164+
PUT /tools/123 -> version: 3
165+
```
166+
167+
Version tracking helps identify:
168+
- **Configuration drift** between environments
169+
- **Change frequency** for troubleshooting
170+
- **Rollback points** for recovery scenarios
171+
172+
---
173+
174+
## 📈 **Use Cases**
175+
176+
### **Security Auditing**
177+
- Track who created/modified sensitive configurations
178+
- Identify unauthorized changes by IP address
179+
- Monitor bulk import operations for compliance
180+
181+
### **Operational Troubleshooting**
182+
- Trace entity origins during incident response
183+
- Identify batch operations that may have caused issues
184+
- Understand federation dependencies between gateways
185+
186+
### **Compliance Reporting**
187+
- Generate audit reports for regulatory requirements
188+
- Track change management processes
189+
- Demonstrate access controls and change attribution
190+
191+
### **Development & Testing**
192+
- Identify test vs production entities
193+
- Track deployment-specific configurations
194+
- Monitor cross-environment migrations
195+
196+
---
197+
198+
## 🔧 **Configuration**
199+
200+
### **No Additional Setup Required**
201+
202+
Metadata tracking is **automatically enabled** for all new installations and upgrades:
203+
204+
- **Database migration** runs automatically on startup
205+
- **Existing entities** show graceful fallbacks for missing metadata
206+
- **No environment variables** needed - uses existing `AUTH_REQUIRED` setting
207+
208+
### **Proxy Support**
209+
210+
Metadata capture automatically handles reverse proxy scenarios:
211+
212+
```bash
213+
# Respects X-Forwarded-For headers
214+
X-Forwarded-For: 203.0.113.1, 192.168.1.1, 127.0.0.1
215+
# Records: created_from_ip = "203.0.113.1" (original client)
216+
```
217+
218+
### **Privacy Considerations**
219+
220+
The system captures IP addresses and user agents for audit purposes:
221+
222+
- **IP addresses**: Consider GDPR/privacy implications for EU deployments
223+
- **User agents**: May contain personally identifiable information
224+
- **Data retention**: Define policies for metadata archival
225+
- **Access control**: Metadata follows same permissions as parent entity
226+
227+
---
228+
229+
## 🚀 **Migration Guide**
230+
231+
### **Upgrading Existing Deployments**
232+
233+
1. **Automatic Migration**
234+
```bash
235+
# Migration runs automatically on startup
236+
# Or run manually:
237+
alembic upgrade head
238+
```
239+
240+
2. **Verify Migration**
241+
- Check admin UI - all entities show metadata sections
242+
- API responses include new metadata fields
243+
- Legacy entities display gracefully
244+
245+
3. **No Downtime Required**
246+
- All metadata columns are nullable
247+
- Existing functionality unchanged
248+
- Gradual adoption of metadata features
249+
250+
### **Metadata Backfill (Optional)**
251+
252+
For enhanced audit trails, optionally backfill known metadata:
253+
254+
```sql
255+
-- Backfill system-created entities
256+
UPDATE tools SET
257+
created_by = 'system',
258+
created_via = 'migration',
259+
version = 1
260+
WHERE created_by IS NULL;
261+
262+
-- Similar for other entity tables
263+
UPDATE gateways SET created_by = 'system', created_via = 'migration', version = 1 WHERE created_by IS NULL;
264+
UPDATE servers SET created_by = 'system', created_via = 'migration', version = 1 WHERE created_by IS NULL;
265+
UPDATE prompts SET created_by = 'system', created_via = 'migration', version = 1 WHERE created_by IS NULL;
266+
UPDATE resources SET created_by = 'system', created_via = 'migration', version = 1 WHERE created_by IS NULL;
267+
```
268+
269+
---
270+
271+
## 🔮 **Future Enhancements**
272+
273+
### **Enhanced Audit Features**
274+
- **Change history tracking** - Before/after state comparison
275+
- **Metadata-based filtering** - Search entities by creator, date, source
276+
- **Audit log export** - Generate compliance reports
277+
- **Custom metadata fields** - User-defined entity attributes
278+
279+
### **Cross-Gateway Features**
280+
- **Metadata synchronization** across federated gateways
281+
- **Trust scoring** based on metadata quality
282+
- **Provenance tracking** for complex federation scenarios
283+
284+
### **Analytics Integration**
285+
- **Usage pattern analysis** from metadata
286+
- **Creator activity dashboards**
287+
- **Import/export trend monitoring**
288+
289+
---
290+
291+
## 📋 **API Examples**
292+
293+
### **Creating Entities with Metadata**
294+
295+
Metadata is captured automatically - no additional parameters needed:
296+
297+
```bash
298+
# Create tool - metadata captured automatically
299+
curl -X POST http://localhost:4444/tools \
300+
-H "Authorization: Bearer $TOKEN" \
301+
-H "Content-Type: application/json" \
302+
-d '{
303+
"name": "example_tool",
304+
"url": "http://example.com/api",
305+
"integration_type": "REST",
306+
"request_type": "GET"
307+
}'
308+
309+
# Response includes metadata
310+
{
311+
"id": "abc123",
312+
"name": "example_tool",
313+
"createdBy": "admin",
314+
"createdAt": "2024-01-15T10:30:00Z",
315+
"createdVia": "api",
316+
"version": 1,
317+
...
318+
}
319+
```
320+
321+
### **Filtering by Metadata (Future)**
322+
323+
```bash
324+
# Future enhancement - filter by creator
325+
GET /tools?created_by=admin
326+
327+
# Filter by creation method
328+
GET /tools?created_via=federation
329+
330+
# Filter by date range
331+
GET /tools?created_after=2024-01-01&created_before=2024-01-31
332+
```
333+
334+
---
335+
336+
## ❓ **FAQ**
337+
338+
### **Q: Will this affect existing deployments?**
339+
A: No breaking changes. Existing entities show graceful fallbacks, all APIs work unchanged.
340+
341+
### **Q: What happens if authentication is disabled?**
342+
A: Metadata still works - `created_by` will be `"anonymous"` instead of a username.
343+
344+
### **Q: How much storage does metadata require?**
345+
A: Minimal - approximately 13 additional nullable text columns per entity.
346+
347+
### **Q: Can I disable metadata tracking?**
348+
A: Not currently - metadata is core to the audit system. All fields are optional and backwards compatible.
349+
350+
### **Q: How do I export metadata for compliance?**
351+
A: Use the standard export functionality - metadata is included in all entity exports.
352+
353+
This comprehensive metadata system provides enterprise-grade audit capabilities while maintaining full backwards compatibility and operational simplicity.

‎docs/docs/overview/features.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,16 @@ adding auth, caching, federation, and an HTMX-powered Admin UI.
123123
* **FastAPI** + Jinja2 + HTMX + Alpine.js
124124
* Tailwind CSS for styling
125125

126+
??? info "📊 Audit & Metadata Tracking"
127+
128+
* **Comprehensive metadata** for all entities (Tools, Resources, Prompts, Servers, Gateways)
129+
* **Creation tracking** - who, when, from where, how
130+
* **Modification history** - change attribution and versioning
131+
* **Federation source** tracking for MCP server entities
132+
* **Bulk import** batch identification
133+
* **Auth-agnostic** - works with/without authentication
134+
* **Backwards compatible** - legacy entities show graceful fallbacks
135+
126136
---
127137

128138
## 🗄 Persistence, Caching & Observability

‎docs/docs/overview/index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ This section introduces what the Gateway is, how it fits into the MCP ecosystem,
1414
- Federation of multiple MCP servers into one composable catalog
1515
- Protocol enforcement, health monitoring, and registry centralization
1616
- A visual Admin UI to manage everything in real time
17+
- **Comprehensive audit trails** with metadata tracking for all entities
1718
- **Comprehensive doctest coverage** ensuring all code examples are tested and verified
1819

1920
Whether you're integrating REST APIs, local functions, or full LLM agents, MCP Gateway standardizes access and transport - over HTTP, WebSockets, SSE, StreamableHttp or stdio.

0 commit comments

Comments
 (0)