SignalWire + OpenAI Voice Assistant

Build an AI phone assistant that actually understands and responds naturally to your callers.

This project connects SignalWire's telephony platform with OpenAI's GPT-4 Realtime API to create voice assistants that can answer phone calls, have natural conversations, and help callers with real information—all in real-time.

Table of Contents

• Introduction
• Prerequisites
• Quick Start
  • 1. Setup SignalWire
  • 2. Clone & Configure
  • 3. Test with ngrok
• How It Works
• Configuration
• Production Deployment
• Development
• Troubleshooting
• Project Structure

Introduction

This application creates a bidirectional audio streaming bridge between phone calls and OpenAI's Realtime API. The result is an AI assistant that can:

Table of Contents

• Introduction
• Prerequisites
• Quick Start
  • 1. Setup SignalWire
  • 2. Clone & Configure
  • 3. Test with ngrok
• How It Works
• Configuration
• Production Deployment
• Development
• Troubleshooting
• Project Structure

Introduction

This application creates a bidirectional audio streaming bridge between phone calls and OpenAI's Realtime API. The result is an AI assistant that can:

• Have natural, flowing conversations with zero buffering delays
• Answer questions and provide information in real-time
• Check the weather for any US city
• Tell the current time
• Handle interruptions naturally (no more talking over each other!)

All with crystal-clear HD voice quality and true real-time bidirectional communication.

🔍 Technical Overview

How the System Works

1. Incoming Call → SignalWire receives the call and streams audio via WebSocket to our server
2. Audio Processing → Our TypeScript server forwards the audio stream to OpenAI's Realtime API using the official SDK
3. Function Call Processing → When AI needs information (weather, time, etc.), function calls are processed locally on our server
4. AI Response → OpenAI processes speech and function results in real-time, generating audio responses
5. Audio Feedback → AI responses stream back through our WebSocket server to SignalWire
6. Caller Hears AI → SignalWire feeds the AI audio directly back into the call

Built With

• @openai/agents - OpenAI's official SDK for GPT-4 Realtime API
• @openai/agents-realtime - Real-time audio streaming with OpenAI
• Fastify - High-performance web framework
• TypeScript - Type-safe JavaScript

Prerequisites

You'll need:

1. Node.js 20+ - Download here
2. OpenAI API Key - Get one here (requires paid account)
3. SignalWire Account - Sign up free (for phone integration)
4. ngrok (for local development) - Install ngrok to expose your local server
5. Docker (optional) - Install Docker for containerized deployment

Quick Start

Follow these three high-level steps to get your AI voice assistant running:

1. Setup SignalWire

📞 Configure SignalWire for Voice Streaming

Create Your SignalWire Project

Follow the SignalWire Getting Started Guide to:

• Create your SignalWire project
• Set up your workspace

Sign up for free at SignalWire

Create a cXML Webhook Resource

Before you can assign webhook URLs, you need to create a cXML webhook resource:

1. In your SignalWire dashboard, go to My Resources
2. Click Create Resource
3. Select Script as the resource type, then select cXML
4. Set the resource to Handle Using as External Url
5. Set the Primary Script URL to your server's webhook endpoint (you'll configure this in step 3):

   https://your-ngrok-url.ngrok.io/incoming-call
   

   🚨 Critical: You MUST include /incoming-call at the end of your URL

6. Give it a descriptive name (e.g., "AI Voice Assistant")
7. Create the resource

📖 Learn More: SignalWire Call Fabric Resources Guide

Create a SIP Address

To test your AI assistant, create a SIP address that connects to your cXML resource:

1. From the resource page of the resource you just created, click the Addresses & Phone Numbers tab
2. Click Add to create a new address
3. Select SIP Address as the address type
4. Fill out the address information
5. Save the configuration

📖 Learn More: SignalWire Call Fabric Addresses Guide 📖 Learn More: SignalWire Call Fabric Addresses Guide

💡 Tip: You can also purchase a regular phone number and link it to your cXML resource if you prefer traditional phone number calling.

2. Clone & Configure

⚙️ Install and Set Up Your Code

Clone and Install

Option 1: Try in Replit

📝 Note: Clicking the button above will take you to Replit where you can import this GitHub repository. After importing, you'll need to configure your OpenAI API key as a Replit Secret. Add OPENAI_API_KEY as a secret in your Repl.

Option 2: Clone Locally

git clone <repository-url>
cd cXML-realtime-agent-stream
npm install

Add Your API Key

Add Your API Key

Choose ONE method based on how you'll run the app:

☁️ Option A: Replit (using Replit Secrets)

• Go to the "Secrets" tab in your Repl (lock icon in sidebar)
• Add a new secret: OPENAI_API_KEY with your API key value
• Learn more about Replit Secrets

🔵 Option B: Local Development (using .env file)

cp .env.example .env
# Edit .env and add your OpenAI API key:
# OPENAI_API_KEY=sk-your-actual-api-key-here

🐳 Option C: Docker Deployment (using secrets folder)

mkdir -p secrets
echo "sk-your-actual-api-key-here" > secrets/openai_api_key.txt

Note: Use only ONE method. Replit uses Secrets, local development uses .env, and Docker uses the secrets folder.

🔑 Get Your API Key: OpenAI Platform (requires paid account)

3. Test with ngrok

🌐 Expose Your Local Server & Test

Start Your Server

For Local Development:

npm run build
npm start

For Docker:

docker-compose up --build signalwire-assistant

✅ Your AI assistant is now running at http://localhost:5050/incoming-call

Expose with ngrok

In a new terminal, run:

npx ngrok http 5050

You'll get a public URL like: https://abc123.ngrok.io

Update SignalWire Webhook

1. Go back to your SignalWire cXML resource (from Step 1)
2. Update the Primary Script URL to:

   https://abc123.ngrok.io/incoming-call
   

3. Save the configuration

⚠️ Important: ngrok URLs change each time you restart it. Update your SignalWire webhook URL whenever you restart ngrok.

Test Your Assistant

Call the SIP address you created in Step 1:

• Using a SIP Phone or Softphone, dial: sip:[email protected]
• Replace with the actual SIP address you created

The call flow will be:

Your SIP call → SignalWire → ngrok → Your local server → OpenAI → Response → Caller

📱 Alternative: If you purchased a regular phone number and linked it to your cXML resource, you can call that number directly.

---

How It Works

Phone Call → SignalWire → Your Server → OpenAI → Real-time Response → Caller

1. Someone calls your SignalWire number
2. SignalWire streams the audio to your server via WebSocket
3. Your server forwards it to OpenAI's Realtime API
4. OpenAI processes speech and generates responses instantly
5. Responses stream back to the caller in real-time

The magic is in the real-time streaming—there's no "recording, processing, playing back." It's a continuous, natural conversation.

Configuration

Environment Variables

Configure your assistant using the following variables. Each variable is handled differently depending on your deployment method:

Variable	Local Development	Docker Deployment	Type	Required
OPENAI_API_KEY	.env file	Docker secrets file (secrets/openai_api_key.txt)	Secret	Yes
PORT	.env file	docker-compose environment section	Environment Variable	No
AUDIO_FORMAT	.env file	docker-compose environment section	Environment Variable	No

Setting Up Variables

For Local Development: Create a .env file in your project root:

OPENAI_API_KEY=sk-your-actual-api-key-here
PORT=5050  # optional, defaults to 5050
AUDIO_FORMAT=pcm16  # optional

For Docker Deployment:

• OPENAI_API_KEY: Create secrets/openai_api_key.txt with your API key
• PORT: Already configured in docker-compose.yml (can be modified there)
• AUDIO_FORMAT: Already configured in docker-compose.yml (can be modified there)

Audio Format Options

• pcm16 - High Definition Audio (24kHz) - Crystal clear voice quality, best for demos
• g711_ulaw - Standard Telephony (8kHz) - Traditional phone quality (default)

🔐 Security Note: Docker uses secrets for sensitive data like API keys, while regular environment variables are used for configuration options.

Customize Your Assistant

Edit src/config.ts to change your AI's personality:

export const AGENT_CONFIG = {
  voice: 'alloy',  // Choose: alloy, echo, fable, onyx, nova, shimmer
  instructions: `Your custom personality here...`
}

Add New Capabilities

Create new tools in src/tools/ - see weather.tool.ts for an example.

Production Deployment

For production deployment, we recommend using Docker. See the Docker Setup Guide for:

• External secrets management
• Health checks and monitoring
• Docker Swarm configuration
• Troubleshooting tips

Development

# Development with hot reload
npm run dev

# Type checking
npm run typecheck

# View debug logs
DEBUG=openai-agents:* npm run dev

Troubleshooting

Common Issues & Solutions

"Missing OPENAI_API_KEY"

• Make sure your .env file exists and contains your actual API key

"SignalWire client connection error"

• Ensure your webhook URL is publicly accessible (use ngrok for local testing)
• Check that port 5050 is not blocked

Audio quality issues

• HD voice requires L16@24000h codec in SignalWire webhook
• Standard quality: Remove the codec parameter

Can't receive calls

• Verify SignalWire webhook is set to your public URL with /incoming-call endpoint
• Check ngrok is still running and URL hasn't changed
• Common mistake: Using base URL without /incoming-call (calls won't work!)
• Look at console logs for connection messages

Project Structure

src/
├── config.ts          # AI assistant configuration
├── index.ts           # Server setup
├── routes/            # HTTP endpoints
│   ├── webhook.ts     # Handles incoming calls
│   ├── streaming.ts   # WebSocket audio streaming
│   └── health.ts      # Health check endpoint
├── tools/             # AI capabilities (weather, time, etc.)
└── transports/        # SignalWire ↔ OpenAI bridge

---

Built with TypeScript, Fastify, and WebSockets. MIT Licensed.