voice-agent-ai-sdk

Streaming voice/text agent SDK built on AI SDK with optional WebSocket transport.

Features

• Streaming text generation via AI SDK streamText with multi-step tool calling.
• Chunked streaming TTS — text is split at sentence boundaries and converted to speech in parallel as the LLM streams, giving low time-to-first-audio.
• Audio transcription via AI SDK experimental_transcribe (e.g. Whisper).
• Barge-in / interruption — user speech cancels both the in-flight LLM stream and pending TTS, saving tokens and latency.
• Memory management — configurable sliding-window on conversation history (maxMessages, maxTotalChars) and audio input size limits.
• Serial request queue — concurrent sendText / audio inputs are queued and processed one at a time, preventing race conditions.
• Graceful lifecycle — disconnect() aborts all in-flight work; destroy() permanently releases every resource.
• WebSocket transport with a full protocol of stream, tool, and speech lifecycle events.
• Works without WebSocket — call sendText() directly for text-only or server-side use.

Prerequisites

• Node.js 20+
• pnpm
• OpenAI API key

Setup

1. Install dependencies:

   pnpm install

2. Configure environment variables in .env:

   OPENAI_API_KEY=your_openai_api_key VOICE_WS_ENDPOINT=ws://localhost:8080

VOICE_WS_ENDPOINT is optional for text-only usage.

VoiceAgent usage (as in the demo)

Minimal end-to-end example using AI SDK tools, streaming text, and streaming TTS:

import "dotenv/config";
import { VoiceAgent } from "./src";
import { tool } from "ai";
import { z } from "zod";
import { openai } from "@ai-sdk/openai";

const weatherTool = tool({
   description: "Get the weather in a location",
   inputSchema: z.object({ location: z.string() }),
   execute: async ({ location }) => ({ location, temperature: 72, conditions: "sunny" }),
});

const agent = new VoiceAgent({
   model: openai("gpt-4o"),
   transcriptionModel: openai.transcription("whisper-1"),
   speechModel: openai.speech("gpt-4o-mini-tts"),
   instructions: "You are a helpful voice assistant.",
   voice: "alloy",
   speechInstructions: "Speak in a friendly, natural conversational tone.",
   outputFormat: "mp3",
   streamingSpeech: {
      minChunkSize: 40,
      maxChunkSize: 180,
      parallelGeneration: true,
      maxParallelRequests: 2,
   },
   // Memory management (new in 0.1.0)
   history: {
      maxMessages: 50,       // keep last 50 messages
      maxTotalChars: 100_000, // or trim when total chars exceed 100k
   },
   maxAudioInputSize: 5 * 1024 * 1024, // 5 MB limit
   endpoint: process.env.VOICE_WS_ENDPOINT,
   tools: { getWeather: weatherTool },
});

agent.on("text", ({ role, text }) => {
   const prefix = role === "user" ? "👤" : "🤖";
   console.log(prefix, text);
});

agent.on("chunk:text_delta", ({ text }) => process.stdout.write(text));
agent.on("speech_start", ({ streaming }) => console.log("speech_start", streaming));
agent.on("audio_chunk", ({ chunkId, format, uint8Array }) => {
   console.log("audio_chunk", chunkId, format, uint8Array.length);
});

await agent.sendText("What's the weather in San Francisco?");

if (process.env.VOICE_WS_ENDPOINT) {
   await agent.connect(process.env.VOICE_WS_ENDPOINT);
}

Configuration options

The agent accepts:

Option	Required	Default	Description
model	yes	—	AI SDK chat model (e.g. openai("gpt-4o"))
transcriptionModel	no	—	AI SDK transcription model (e.g. openai.transcription("whisper-1"))
speechModel	no	—	AI SDK speech model (e.g. openai.speech("gpt-4o-mini-tts"))
instructions	no	"You are a helpful voice assistant."	System prompt
stopWhen	no	stepCountIs(5)	Stopping condition for multi-step tool loops
tools	no	{}	AI SDK tools map
endpoint	no	—	Default WebSocket URL for connect()
voice	no	"alloy"	TTS voice
speechInstructions	no	—	Style instructions passed to the speech model
outputFormat	no	"mp3"	Audio output format (mp3, opus, wav, …)
streamingSpeech	no	see below	Streaming TTS chunk tuning
history	no	see below	Conversation memory limits
maxAudioInputSize	no	10485760 (10 MB)	Maximum accepted audio input in bytes

streamingSpeech

Key	Default	Description
minChunkSize	50	Min characters before a sentence is sent to TTS
maxChunkSize	200	Max characters per chunk (force-split at clause boundary)
parallelGeneration	true	Start TTS for upcoming chunks while the current one plays
maxParallelRequests	3	Cap on concurrent TTS requests

history

Key	Default	Description
maxMessages	100	Max messages kept in history (0 = unlimited). Oldest are trimmed in pairs.
maxTotalChars	0 (unlimited)	Max total characters across all messages. Oldest are trimmed when exceeded.

Methods

Method	Description
sendText(text)	Process text input. Returns a promise with the full assistant response. Requests are queued serially.
sendAudio(base64Audio)	Transcribe base64 audio and process the result.
sendAudioBuffer(buffer)	Same as above, accepts a raw Buffer / Uint8Array.
transcribeAudio(buffer)	Transcribe audio to text without generating a response.
generateAndSendSpeechFull(text)	Non-streaming TTS fallback (entire text at once).
interruptSpeech(reason?)	Cancel in-flight TTS only (LLM stream keeps running).
interruptCurrentResponse(reason?)	Cancel both the LLM stream and TTS. Used for barge-in.
connect(url?) / handleSocket(ws)	Establish or attach a WebSocket. Safe to call multiple times.
disconnect()	Close the socket and abort all in-flight work.
destroy()	Permanently release all resources. The agent cannot be reused.
clearHistory()	Clear conversation history.
getHistory() / setHistory(msgs)	Read or restore conversation history.
registerTools(tools)	Merge additional tools into the agent.

Read-only properties

Property	Type	Description
connected	boolean	Whether a WebSocket is connected
processing	boolean	Whether a request is currently being processed
speaking	boolean	Whether audio is currently being generated / sent
pendingSpeechChunks	number	Number of queued TTS chunks
destroyed	boolean	Whether destroy() has been called

Events

Event	Payload	When
text	{ role, text }	User input received or full assistant response ready
chunk:text_delta	{ id, text }	Each streaming text token from the LLM
chunk:reasoning_delta	{ id, text }	Each reasoning token (models that support it)
chunk:tool_call	{ toolName, toolCallId, input }	Tool invocation detected
tool_result	{ name, toolCallId, result }	Tool execution finished
speech_start	{ streaming }	TTS generation begins
speech_complete	{ streaming }	All TTS chunks sent
speech_interrupted	{ reason }	Speech was cancelled (barge-in, disconnect, error)
speech_chunk_queued	{ id, text }	A text chunk entered the TTS queue
audio_chunk	{ chunkId, data, format, text, uint8Array }	One TTS chunk is ready
audio	{ data, format, uint8Array }	Full non-streaming TTS audio
transcription	{ text, language }	Audio transcription result
audio_received	{ size }	Raw audio input received (before transcription)
history_trimmed	{ removedCount, reason }	Oldest messages evicted from history
connected / disconnected	—	WebSocket lifecycle
warning	string	Non-fatal issues (empty input, etc.)
error	Error	Errors from LLM, TTS, transcription, or WebSocket

Run (text-only check)

This validates LLM + tool + streaming speech without requiring WebSocket:

pnpm demo

Expected logs include text, chunk:text_delta, tool events, and speech chunk events.

Run (WebSocket check)

1. Start the local WS server:

   pnpm ws:server
   

2. In another terminal, run the demo:

   pnpm demo
   

The demo will:

• run sendText() first (text-only sanity check), then
• connect to VOICE_WS_ENDPOINT if provided,
• emit streaming protocol messages (text_delta, tool_call, audio_chunk, response_complete, etc.).

Browser voice client (HTML)

A simple browser client is available at example/voice-client.html.

What it does:

• captures microphone speech using Web Speech API (speech-to-text)
• sends transcript to the agent via WebSocket (type: "transcript")
• receives streaming audio_chunk messages and plays them in order

How to use:

1. Start your agent server/WebSocket endpoint.
2. Open example/voice-client.html in a browser (Chrome/Edge recommended).
3. Connect to ws://localhost:8080 (or your endpoint), then click Start Mic.

Scripts

• pnpm build – build TypeScript
• pnpm dev – watch TypeScript
• pnpm demo – run demo client
• pnpm ws:server – run local test WebSocket server

Notes

• If VOICE_WS_ENDPOINT is empty, WebSocket connect is skipped.
• The sample WS server sends a mock transcript message for end-to-end testing.
• Streaming TTS uses chunk queueing and supports interruption (interrupt).