Realtime Voice & Conversations

This is the API behind a live InfraVoice call. You create a session against one of your agents, open a WebSocket, and stream raw microphone audio into it; the platform runs speech recognition, your agent's LLM (including tool calls), and text-to-speech, and streams the reply audio back to you along with a running commentary of JSON events — transcripts, turn timings, tool invocations, barge-in signals.

When the call ends, the same conversation lands in a durable conversation record: the full transcript, per-turn provider and latency telemetry, tool inputs and outputs, per-turn audio (kept for 30 days — the rest of the record has no expiry), and a summary. That record is what you pull for QA, analytics, or your own CRM.

Base URL: https://your-infravoice-host/api/v1 · WebSocket base: wss://your-infravoice-host/api/v1 Auth (conversations): Authorization: Bearer usf-… — your organization key, from Account settings → Organization key Auth (voice sessions): none is checked. The session_id is the credential — see Auth and envelopes. Audio wire format: raw PCM, 16 kHz, 16-bit signed little-endian, mono, both directions.

Endpoints

Everything on this page, in one screen. Two services back these routes and they answer in two different response shapes — read Auth and envelopes before you write a parser.

Voice pipeline — Python, raw JSON, no bearer token checked.

Conversations — Go, enveloped, Authorization: Bearer usf-… required.

Call flow at a glance

1. Start a session from your backend — POST /voice/sessions/start returns a session_id.
2. Open the WebSocket within 30 seconds at wss://your-infravoice-host/api/v1/voice/ws/{session_id}?profile=browser_speakers. Pre-warmed provider connections (and the stored session config) are purged after 30 seconds.
3. Wait for pipeline_ready. Do not stream microphone audio, and do not show a "listening" indicator, until this event arrives. Audio sent before it is not guaranteed to be transcribed.
4. Stream and play. Continuous binary PCM frames out; binary frames back. After each bot utterance finishes playing out of the speaker, send {"type":"playback_done"}.
5. Hang up by closing the socket. That is the reliable end-of-call.
6. Read the transcript a few seconds later via GET /conversations/by-session/{session_id}, then GET /conversations/{id}.

Draft vs published. Everything you change about an agent in the dashboard — prompt, providers, voice, tools, telephony settings — is saved to a draft. A session resolves the agent's published configuration, so a draft change has no effect on live calls until you call POST /api/v1/agents/{id}/versions/publish. If a call is using the wrong prompt, wrong voice, or is missing a tool you just added, publish the agent and start a fresh session — a live socket will not pick up a mid-call change. This is almost always the explanation.

Auth and envelopes

The voice pipeline does not authenticate requests. POST /voice/sessions/start, /text, /end and the WebSocket accept no bearer token, no organization key, and no cookie. The only thing that binds a live socket to your account is the session_id returned by /sessions/start, which resolves to your agent, your workspace, and your credit balance.

• Call /sessions/start from your backend, never from the browser. Your agent_id and workspace_id should not be shipped to a client that a stranger can read.
• Treat session_id as a short-lived secret. Hand it to exactly one client, for exactly one call. Anyone holding it can talk to your agent on your credits.
• Billing is still enforced. The WebSocket checks your workspace balance at connect time and closes with an insufficient_balance error frame if it is empty. A missing token will not stop a call; a missing balance will.

The conversation endpoints behave normally: they require Authorization: Bearer usf-… (your organization key) or a dashboard JWT, and every row is fenced to your workspace. The same organization key also works on the telephony and campaign APIs.

The two services also do not share a response shape.

So POST /voice/sessions/start gives you {"session_id": …} at the top level, while GET /conversations/{id} gives you {"success":true,"data":{ "conversation": …, "messages": […] }}. A single "unwrap .data" helper applied to both will crash on the voice endpoints. List endpoints on the Go side also carry pagination in meta, never in data.

Rate limiting differs too: /api/v1/voice/* is exempt from the platform rate limiter — the concurrency cap on voice sockets is what bounds it. The conversation endpoints are rate limited normally.

Start a session

POST https://your-infravoice-host/api/v1/voice/sessions/start

Creating a session is the expensive part of a call: the platform loads your agent, assembles the system prompt (agent handbook + attached knowledge base + any dynamic context you pass), writes the resolved config into Redis so any pipeline worker can serve the socket, and opens speculative connections to your providers. The agent must already exist and have ASR/LLM/TTS providers configured on its published version.

curl -X POST https://your-infravoice-host/api/v1/voice/sessions/start \
  -H "Content-Type: application/json" \
  -d '{
    "user_id": "user_123",
    "agent_id": "0f2b9a4c-1e77-4c3a-9c11-7d0d2b6f8a51",
    "workspace_id": "ws_7a1c",
    "welcome_mode": "tts",
    "language": "en",
    "dynamic_variables": { "customer_name": "Priya" }
  }'

Response — raw JSON, no envelope:

{
  "session_id": "5f9d3b02-6c41-4f8e-b0a7-2e5a9d17c4bb",
  "agent_id": "0f2b9a4c-1e77-4c3a-9c11-7d0d2b6f8a51",
  "welcome_message": "Hi Priya, thanks for calling — how can I help?",
  "status": "active"
}

welcome_message is returned after variable substitution, so you can display exactly what the caller is about to hear.

# Start a voice session from your backend, then hand session_id to one client.
import requests
 
res = requests.post(
    "https://your-infravoice-host/api/v1/voice/sessions/start",
    json={
        "user_id": "user_123",
        "agent_id": "0f2b9a4c-1e77-4c3a-9c11-7d0d2b6f8a51",
        "workspace_id": "ws_7a1c",
        "welcome_mode": "tts",
        "language": "en",
        "kb_mode": "tool",
        "knowledge_base_ids": ["kb_returns_policy"],
        "dynamic_variables": {"customer_name": "Priya", "plan": "Pro"},
    },
    timeout=30,
)
session = res.json()          # no {"data": …} wrapper here
print(session["session_id"])  # connect the WebSocket within 30 seconds

One session, one socket. A session_id is for a single connection. Reconnecting after a drop means starting a new session — and, because the conversation record follows the session, a new transcript row. Start the session when the user is about to talk, not when your page loads.

Inject text into a live session

POST https://your-infravoice-host/api/v1/voice/sessions/{session_id}/text

Feeds a turn straight to the LLM as if the caller had said it, skipping speech recognition. The agent's reply is spoken over the existing WebSocket — this endpoint does not return the answer. Use it for typed input in a hybrid chat/voice widget, or to have an operator whisper a line into a running call.

curl -X POST https://your-infravoice-host/api/v1/voice/sessions/5f9d3b02-6c41-4f8e-b0a7-2e5a9d17c4bb/text \
  -H "Content-Type: application/json" \
  -d '{"text": "What are your opening hours on Sunday?", "role": "user"}'

{
  "session_id": "5f9d3b02-6c41-4f8e-b0a7-2e5a9d17c4bb",
  "status": "injected",
  "text": "What are your opening hours on Sunday?"
}

Returns 404 if the session id is unknown and 409 if the session exists but no WebSocket has connected yet — there is nowhere to speak the answer. The equivalent WebSocket message, {"type":"text_inject","text":"…","role":"user"}, is usually easier because it is already on the right connection.

End a session

POST https://your-infravoice-host/api/v1/voice/sessions/{session_id}/end

curl -X POST https://your-infravoice-host/api/v1/voice/sessions/5f9d3b02-6c41-4f8e-b0a7-2e5a9d17c4bb/end

{ "session_id": "5f9d3b02-6c41-4f8e-b0a7-2e5a9d17c4bb", "status": "ended" }

Closing the socket is the reliable hang-up. /end is served by whichever pipeline worker receives the HTTP request, and it looks the session up in that worker's memory. In a multi-worker deployment it can return 404 for a session that is very much alive on a sibling worker. Close the WebSocket; use /end only as a belt-and-braces cleanup for a session whose socket never connected.

The WebSocket

wss://your-infravoice-host/api/v1/voice/ws/{session_id}

This is not a JSON-envelope protocol. It is a dual-mode socket: binary frames carry audio, text frames carry JSON control events, in both directions. Your client must branch on frame type.

Connecting

Pick honestly. On speakers, the microphone hears the bot's own voice, so the server uses conservative barge-in thresholds to avoid the agent interrupting itself. On a headset there is no echo path, so it can be far more responsive to the caller cutting in.

The gateway enforces a global cap on concurrent voice sockets. When it is saturated the upgrade is refused with 503 SERVICE_UNAVAILABLE before the pipeline is ever reached — retry with backoff rather than hammering.

Audio in and out

Sending:

• Raw PCM, 16 kHz, 16-bit signed little-endian, mono, as binary WebSocket frames.
• No framing header, no base64, no JSON wrapper. Just the samples.
• Send continuously, in small chunks (20–50 ms is typical). Do not gate on your own local voice detection.
• Frames shorter than 10 bytes are discarded.

Turn-taking is entirely server-side: a Silero voice-activity detector decides when the caller started and stopped speaking. There is no client message for "I started talking" or "I'm done talking." If you send audio only while you think someone is speaking, you will clip the beginning of every utterance and confuse endpointing.

Receiving:

• The same format: PCM 16 kHz s16le mono, as binary frames.
• Chunks may begin with a 44-byte RIFF/WAV header. Strip it before appending to a playback buffer, or you will hear a click at every chunk boundary. Check for the ASCII bytes RIFF at offset 0 and skip 44 bytes if present.
• Queue and play in arrival order. Do not start playback before pipeline_ready.

Messages you send

Only two message types are acted on:

{ "type": "playback_done" }

The authoritative "my speaker has finished all queued audio" signal. It resets the idle monitor's countdown. Send it after every bot utterance completes playback. Skip it and the agent will interrupt itself with "Are you still there?" while it is still talking, because from the server's point of view the caller has been silent since the audio was generated.

{ "type": "text_inject", "text": "I'd like to change my address", "role": "user" }

A typed turn, bypassing speech recognition. Identical in effect to POST /voice/sessions/{id}/text.

session.end does nothing. Some reference clients send {"type":"session.end"} on hang-up. No server handler exists for it. It is harmless, but it is not how you end a call — closing the socket is.

Events you receive

Every text frame from the server is a JSON object with a type and a server_ts (epoch milliseconds), plus type-specific fields:

{ "type": "transcription", "server_ts": 1753549812345, "text": "I need to change my flight", "turn": 3, "speech_duration_ms": 1840, "asr_latency_ms": 210 }

Turn numbers matter. Every event carries the turn it belongs to, and events for a superseded turn can still arrive after the caller has moved on — especially in tool-using turns, which emit multiple steps. Discard anything whose turn you have already abandoned.

Telephony sockets

For phone calls you do not open the socket yourself. The telephony service issues TwiML that points the carrier's media stream at wss://your-infravoice-host/api/v1/voice/ws/twilio/{callSid}, with session_id and agent_id passed as stream customParameters. The carrier performs its own handshake (connected → start → media frames), and the pipeline then runs the identical handler behind a telephony serializer. Two differences matter when debugging a phone call against your browser experience:

• Audio is mulaw at 8 kHz, not PCM at 16 kHz — the carrier's format, converted at the edge.
• The JSON event stream is suppressed on telephony sockets. There is no pipeline_ready, no transcription, no turn_complete on the wire. Everything you would have watched live is still written to the conversation record, so for phone calls you read the transcript afterwards rather than following it on the socket.

This URL is documented so you recognise it in logs. It is not an endpoint you call: configure a phone number on the agent and the platform wires it up.

Barge-in and tools

Interruption is entirely server-driven. There is no client-to-server "interrupt" or "cancel" message; sending one does nothing. What you control is how gracefully you react to the server's signals.

The recommended sequence, and the one the reference client implements:

1. user_started_speaking arrives → duck playback (drop the volume). Do not stop it. Voice detection fires on coughs, door slams and speaker echo, and killing audio on every trigger makes the agent feel broken.
2. An interim_transcription follows → this is a real barge-in. Now cut playback and flush your audio queue.
3. phase_idle arrives instead (or ~2 seconds pass with no interim result) → it was a false trigger. Restore volume and let the agent finish.

Then use turn numbers as the safety net: every tts_audio_start carries the caller turn it answers. If audio arrives for a turn the caller already interrupted — a later step of an abandoned tool chain, for instance — drop it instead of playing it.

Tool calls on the wire

When your agent uses tools, a single caller turn produces several steps, and the event stream lets you narrate it:

{ "type": "tool_calls_started", "server_ts": 1753549813001, "turn": 3, "step": 1, "parallel": true, "tools": ["lookup_booking", "check_availability"] }
{ "type": "tool_call_in_progress", "server_ts": 1753549813010, "turn": 3, "step": 1, "tool_name": "lookup_booking", "tool_call_id": "call_a1", "tool_input": { "pnr": "QZ71KP" } }
{ "type": "tool_call_result", "server_ts": 1753549813402, "turn": 3, "step": 1, "tool_name": "lookup_booking", "tool_call_id": "call_a1", "result": "{\"flight\":\"IX-812\",\"date\":\"2026-08-03\"…", "tool_execution_ms": 392 }
{ "type": "llm_step_start", "server_ts": 1753549813410, "turn": 3, "step": 2 }

Because the model is re-invoked after tools return, a tool-using turn costs roughly double the model latency of a plain turn. That is the usual explanation for one noticeably slower reply in an otherwise snappy call — turn_complete.timing.steps[] will show you exactly where the time went. tool_call_result.result is truncated to 200 characters on the wire; retrieve the complete tool_input/tool_output from the conversation record afterwards.

Failure frames

The socket sends a JSON error frame and then closes:

The most common of these by far is Session not found caused by connecting too late — the session config and its pre-warmed providers are cleaned up after 30 seconds if nothing connects.

Conversations

Every call — browser or phone — produces a conversation record. These endpoints are the normal, fully authenticated platform API: Authorization: Bearer usf-…, Go response envelope, workspace-fenced.

List conversations

GET https://your-infravoice-host/api/v1/conversations

A request with no filter is never a global list — it is silently scoped to your own workspace. agent_id and user_id are additionally joined against agent ownership, so an id belonging to someone else returns nothing rather than data.

curl "https://your-infravoice-host/api/v1/conversations?agent_id=0f2b9a4c-1e77-4c3a-9c11-7d0d2b6f8a51&limit=20&offset=0" \
  -H "Authorization: Bearer usf-YOUR_ORG_KEY"

{
  "success": true,
  "data": [
    {
      "id": "c1a2b3c4-0000-4a11-9f00-8d2e6b7c1122",
      "session_id": "5f9d3b02-6c41-4f8e-b0a7-2e5a9d17c4bb",
      "user_id": "user_123",
      "agent_id": "0f2b9a4c-1e77-4c3a-9c11-7d0d2b6f8a51",
      "status": "completed",
      "total_turns": 14,
      "total_duration_ms": 187400,
      "avg_response_latency_ms": 1420,
      "summary": "Caller rebooked flight IX-812 to 5 August and confirmed the fare difference.",
      "metadata": { "call_direction": "inbound" },
      "started_at": "2026-07-26T09:14:02Z",
      "ended_at": "2026-07-26T09:17:09Z",
      "created_at": "2026-07-26T09:14:02Z",
      "updated_at": "2026-07-26T09:17:11Z",
      "call_id": "call_8812fa",
      "session_count": 3,
      "segment_count": 3,
      "segment_pattern": "ai→human→ai"
    }
  ],
  "error": null,
  "meta": { "page": 1, "per_page": 20, "total": 137, "total_pages": 7 },
  "timestamp": "2026-07-26T09:20:00Z"
}

summary is the post-call summary, populated asynchronously after the call ends — expect it to be absent on a conversation you fetch the instant the socket closes. The last four fields only appear on transferred calls; see Transferred calls.

Get one conversation

GET https://your-infravoice-host/api/v1/conversations/{id}

The full record: header, every transcript turn with its provider and latency telemetry, and every tool call.

curl https://your-infravoice-host/api/v1/conversations/c1a2b3c4-0000-4a11-9f00-8d2e6b7c1122 \
  -H "Authorization: Bearer usf-YOUR_ORG_KEY"

data contains three keys:

{
  "success": true,
  "data": {
    "conversation": {
      "id": "c1a2b3c4-0000-4a11-9f00-8d2e6b7c1122",
      "session_id": "5f9d3b02-6c41-4f8e-b0a7-2e5a9d17c4bb",
      "agent_id": "0f2b9a4c-1e77-4c3a-9c11-7d0d2b6f8a51",
      "status": "completed",
      "total_turns": 14,
      "total_duration_ms": 187400,
      "avg_response_latency_ms": 1420,
      "summary": "Caller rebooked flight IX-812 to 5 August…",
      "started_at": "2026-07-26T09:14:02Z",
      "ended_at": "2026-07-26T09:17:09Z"
    },
    "messages": [
      {
        "id": "m_0003",
        "conversation_id": "c1a2b3c4-0000-4a11-9f00-8d2e6b7c1122",
        "turn_number": 3,
        "role": "user",
        "content": "I need to change my flight",
        "audio_url": "https://…/turn_3_user.wav?X-Amz-Signature=…",
        "asr_provider": "usf",
        "asr_model": "usf-asr-en",
        "asr_latency_ms": 210,
        "asr_confidence": 0.94,
        "total_latency_ms": 210,
        "created_at": "2026-07-26T09:14:31Z"
      },
      {
        "id": "m_0004",
        "conversation_id": "c1a2b3c4-0000-4a11-9f00-8d2e6b7c1122",
        "turn_number": 3,
        "role": "assistant",
        "content": "I can help with that — I've found booking QZ71KP…",
        "tool_calls": [{ "id": "call_a1", "name": "lookup_booking" }],
        "llm_provider": "openai",
        "llm_model": "gpt-4.1-mini",
        "llm_latency_ms": 880,
        "tokens_input": 1840,
        "tokens_output": 96,
        "tts_provider": "cartesia",
        "tts_latency_ms": 240,
        "total_latency_ms": 1512,
        "internal_overhead_ms": 32,
        "created_at": "2026-07-26T09:14:33Z"
      }
    ],
    "tool_calls": [
      {
        "id": "tc_01",
        "message_id": "m_0004",
        "conversation_id": "c1a2b3c4-0000-4a11-9f00-8d2e6b7c1122",
        "tool_name": "lookup_booking",
        "tool_input": { "pnr": "QZ71KP" },
        "tool_output": { "flight": "IX-812", "date": "2026-08-03", "fare_class": "Y" },
        "status": "success",
        "latency_ms": 392,
        "error_message": null,
        "created_at": "2026-07-26T09:14:32Z"
      }
    ]
  },
  "error": null,
  "timestamp": "2026-07-26T09:20:00Z"
}

A few things worth knowing about this payload:

• Messages are paired by turn_number, not one-per-turn: a caller user message and the agent's assistant reply share a turn number. Group by it to render a conversation.
• Latency telemetry is per stage. asr_latency_ms + llm_latency_ms + tts_latency_ms + internal_overhead_ms is the anatomy of total_latency_ms. When a call feels slow, this tells you which vendor to blame — usually the model's time-to-first-token.
• audio_url is minted fresh on every read as a short-lived pre-signed URL. Download it or use it immediately; do not cache it, store it in your database, or email it — it will expire. Re-fetch the conversation to get a new one. (When object storage is not configured, recordings are served from a platform path instead; those paths are unguessable rather than authenticated, so treat them as secrets too.) Per-turn recordings are also deleted on the platform retention schedule (30 days by default, set per deployment rather than per workspace) — if you need them long-term, download them into your own storage.
• tool_calls here are complete — full tool_input and tool_output objects, unlike the 200-character truncation on the live socket.
• A conversation outside your workspace returns 404, not 403, so ids cannot be probed.

GET /api/v1/conversations/{id}/messages returns just the messages array (same shape, freshly signed audio URLs) when you don't need the header or the tool calls.

Resolve a session id

GET https://your-infravoice-host/api/v1/conversations/by-session/{sessionID}

You often hold a voice session_id rather than a conversation id — it's what /sessions/start gave you, and it's what a billing charge line references. This maps one to the other and returns the conversation object.

curl https://your-infravoice-host/api/v1/conversations/by-session/5f9d3b02-6c41-4f8e-b0a7-2e5a9d17c4bb \
  -H "Authorization: Bearer usf-YOUR_ORG_KEY"

Returns 404 when the session belongs to another workspace — deliberately indistinguishable from "no such session".

For aggregates, GET /api/v1/conversations/summary?workspace_id=ws_7a1c covers all conversations in the workspace, not just the page you are looking at (workspace_id is required; optional status and agent_id narrow it). It returns total, active, completed, failed, total_turns, total_duration_ms and avg_latency_ms. For a single agent's volume, GET /api/v1/conversations/agent/{agentID}/count returns {"count": n} more cheaply.

Transferred calls

When a call is transferred to a human agent — and again if it comes back to the AI afterwards — the platform does not keep writing into one conversation record. Each stint is its own conversation row, with its own id, its own session_id, and its own slice of the transcript. One phone call, three records.

What ties them together is call_id. That is the stitch key. Every sibling stint of the same physical call carries the same call_id.

GET /api/v1/conversations already does the stitching for you: sibling stints are folded into a single list row, and that row carries four extra fields:

These four fields are absent on ordinary single-stint calls — their presence is itself the signal that the row you are looking at is a folded, multi-segment call.

Two consequences that trip people up:

The folded row's id and session_id are the earliest stint, but its total_turns, total_duration_ms and avg_response_latency_ms are call-level aggregates. Those numbers describe the whole call; that id describes only its first leg.

So GET /conversations/{id} on a folded row returns a transcript shorter than total_turns promised. Nothing is truncated and nothing is lost — the detail endpoint faithfully returns the one stint you asked for, and the remaining turns live in the sibling records sharing that call_id. If your integration reports "the transcript is cut off right where the caller asked for a human", this is what happened.

Practical guidance:

• Treat segment_count > 1 as "this is a multi-part call" and surface it in your UI, exactly as the dashboard's call history does with its "N segments" pill.
• Never compare messages.length against the folded row's total_turns and conclude data is missing.
• Where you hold a sibling's own session_id — from a webhook, a billing charge, or your own records — resolve it with GET /api/v1/conversations/by-session/{sessionID} and fetch that stint's transcript the normal way.
• There is no /segments endpoint. Segment information is exposed only as those folded fields on list rows; don't go looking for one.

Writing your own records

The voice pipeline writes conversation records for you. These endpoints exist for the case where you run your own voice or chat pipeline and want its transcripts to live in InfraVoice alongside the platform's — so QA scoring, analytics and call history cover both.

Supply your own id on create rather than reading it back — message logging is asynchronous, and generating the id yourself removes the race where a turn arrives before you know what to attach it to. agent_id must be an agent in your own workspace.

Finalising with PUT /conversations/{id} is the meaningful one: it flips the record to completed and triggers post-call processing — summary generation and QA scoring — provided you include agent_id. Skip the PUT and the record stays active forever with no summary.

curl -X PUT https://your-infravoice-host/api/v1/conversations/c1a2b3c4-0000-4a11-9f00-8d2e6b7c1122 \
  -H "Authorization: Bearer usf-YOUR_ORG_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "status": "completed",
    "total_turns": 14,
    "total_duration_ms": 187400,
    "avg_response_latency_ms": 1420,
    "agent_id": "0f2b9a4c-1e77-4c3a-9c11-7d0d2b6f8a51",
    "user_id": "user_123"
  }'

Live transcript streaming. A server-sent-events stream of a single conversation's turns exists and powers the dashboard's live views, but it is not currently authenticated. Until it is gated, treat it as an internal dashboard affordance rather than an API to build on, and never place a conversation id where a third party can read it. For durable transcripts, poll GET /api/v1/conversations/{id} or read the record once the call ends.

Errors

The two services on this page report failures differently.

Voice pipeline (/api/v1/voice/*) — plain HTTP status codes with raw JSON bodies, and JSON error frames on the socket.

Conversations (/api/v1/conversations/*) — the standard envelope with success: false and an error object.

On-socket failures (insufficient_balance, Session not found, Server at capacity) are covered in Failure frames.