Realtime Speech To Speech

WSS

realtime

Bidirectional WebSocket endpoint for real-time speech translation. This endpoint is served by realtime-api-server at wss://realtime.camb.ai/v1/realtime, separate from the /apis/live-tts/ws and /streaming-transcription/listen WebSocket endpoints.

GET /v1/realtime
Host: realtime.camb.ai
x-api-key: <YOUR_API_KEY>

Sessions run in one of two modes, selected with the mode query parameter or the mode field of the first session.update. fast (the default) accepts audio almost immediately and translates with the lowest latency, but supports fewer languages. slow supports the full language list and translates more accurately, at the cost of a 30s+ cold boot before the session is ready. mode replaced a model parameter that took engine codenames. Both are still accepted — iris resolves to fast, and lilac/violet/orchid to slow — so existing integrations keep working while you migrate. Authenticate with the x-api-key WebSocket request header. If your client cannot set WebSocket headers, send credentials in the first session.update event instead.

Quickstart

Use the SDK (Python or TypeScript) — it handles the session lifecycle (including the session.starting cold-boot wait), surfaces the server’s binary audio frames as typed audio events, and exposes typed events. Input and output audio are PCM16, mono, 24 kHz. The example below streams a WAV file and writes the translated speech to another WAV.

import asyncio
import os
import wave

from camb.client import CambAI
from camb.realtime import ServerEventType
from camb.live_transcription import FileAudioSource


async def main():
    client = CambAI(api_key=os.environ["CAMB_API_KEY"])
    session = await client.realtime.connect(
        source_language="en-us",
        target_language="de-de",
    )

    out_audio = bytearray()

    @session.on(ServerEventType.TEXT_DONE)
    def _(event):
        print("translation:", event.text)

    @session.on(ServerEventType.AUDIO_DELTA)
    def _(event):
        out_audio.extend(event.data)  # raw PCM16 mono 24 kHz

    async with session:
        await session.wait_until_ready()
        # Input WAV must be 16-bit PCM, mono, 24 kHz.
        await session.stream_audio(FileAudioSource("input_24k_mono.wav", real_time=True))

    with wave.open("translated.wav", "wb") as out:
        out.setnchannels(1)
        out.setsampwidth(2)
        out.setframerate(24000)
        out.writeframes(bytes(out_audio))


asyncio.run(main())

import fs from "node:fs";

import { CambClient, RealtimeServerEventType } from "@camb-ai/sdk";

const client = new CambClient({ apiKey: process.env.CAMB_API_KEY });
const session = await client.realtime.connect({
  sourceLanguage: "en-us",
  targetLanguage: "de-de",
});

const outChunks = [];
let resolveDone;
const audioDone = new Promise((r) => (resolveDone = r));

session.on(RealtimeServerEventType.TextDone, (event) =>
  console.log("translation:", event.text),
);
session.on(RealtimeServerEventType.AudioDelta, (event) =>
  outChunks.push(Buffer.from(event.data)), // raw PCM16 mono 24 kHz
);
session.on(RealtimeServerEventType.AudioDone, () => resolveDone());

await session.waitUntilReady();

// Input WAV must be 16-bit PCM, mono, 24 kHz. This skips the standard 44-byte
// header and streams the PCM data at real-time pace in 100 ms slices.
const data = fs.readFileSync("input_24k_mono.wav").subarray(44);
const chunkSize = 24000 * 2 * 0.1;
for (let i = 0; i < data.length; i += chunkSize) {
  await session.sendAudio(data.subarray(i, i + chunkSize));
  await new Promise((r) => setTimeout(r, 100));
}

await Promise.race([audioDone, new Promise((r) => setTimeout(r, 30_000))]);
await session.close();

// Raw PCM16 mono 24 kHz; wrap in a WAV header to play it (see the tutorial).
fs.writeFileSync("translated.pcm", Buffer.concat(outChunks));

See the Realtime Speech Translation tutorial for the microphone quickstart, the full event list, and configuration. The sections below document the underlying wire protocol for reference (for example, if you are building a client in a language without an SDK).

Integration in 4 steps

Open the realtime socket

Connect to wss://realtime.camb.ai/v1/realtime. This endpoint is not under the client.camb.ai/apis namespace used by the other WebSocket API references.

async with websockets.connect(
    "wss://realtime.camb.ai/v1/realtime",
    additional_headers=[("x-api-key", "YOUR_API_KEY")],
) as ws:
    ...

Send session.update as the first message

The first WebSocket message must be a JSON session.update event. The server waits up to 60 seconds for it.

{
  "type": "session.update",
  "session": {
    "source_language": "en-us",
    "target_language": "de-de",
    "output_modalities": ["text", "audio"]
  }
}

The server responds with session.created, then session.updated.

Stream input audio

Send microphone audio as base64-encoded bytes in input_audio_buffer.append. Only text WebSocket messages are parsed as realtime events.

{
  "type": "input_audio_buffer.append",
  "audio": "<base64_audio_bytes>"
}

Each decoded audio payload can be up to 256 KiB.

Read translated output

Listen for transcript, translated text, and translated audio events. response.text.delta values are additive for the current response, and response.audio.delta contains base64-encoded synthesized audio bytes.

Authentication

Prefer the WebSocket request header:

x-api-key: <YOUR_API_KEY>

The initial session.update event can also carry credentials:

{
  "type": "session.update",
  "session": {
    "source_language": "en-us",
    "target_language": "de-de",
    "output_modalities": ["text", "audio"]
  },
  "auth": {
    "api_key": "<YOUR_API_KEY>"
  }
}

If both the request header and auth object are present, the request header credential is used.

Reference

The AsyncAPI spec above documents every client and server event. Quick lookup:

Session configuration

Field	Type	Required	Notes
`source_language`	string	Yes	Source language tag, for example `en-us`. Must be a supported language, or `auto` to detect it.
`target_language`	string	Yes, unless `source_language` is `auto`	Target language tag, for example `de-de`. Must be a supported language. With `auto` in `slow` mode it may be omitted until you know it.
`speaker_setup`	string	No	Only with `auto`: `changing_speakers` (default) or `single_known_speaker`. See Speaker setup.
`source_language_candidates`	string[]	No	Only with `auto`: up to 10 languages detection may choose from.
`output_modalities`	string[]	No	Defaults to `["text", "audio"]`.
`voice`	object	No	Output voice selection. Defaults to `{ "type": "default" }`. See Voice selection.

Voice selection

By default, translated speech is synthesized with a built-in voice for the target language. To synthesize the translation with one of your own cloned voices, include a voice object in the session configuration:

{
  "type": "session.update",
  "session": {
    "source_language": "en-us",
    "target_language": "de-de",
    "output_modalities": ["text", "audio"],
    "voice": { "type": "cloned", "voice_id": 147320 }
  }
}

Field	Type	Required	Notes
`type`	string	Yes	`"default"` to use the built-in voice, or `"cloned"` to use one of your voices.
`voice_id`	integer	When `type` is `"cloned"`	ID of a voice you own. Get it from List Voices or Create Custom Voice.

The voice must belong to your account — stock/built-in voice IDs are rejected with an error event. Omitting voice (or sending { "type": "default" }) uses the built-in voice. The resolved selection is echoed back in session.created. If you use the SDK, pass voice_id (Python) or voiceId (TypeScript) to realtime.connect() and it builds this voice object for you:

session = await client.realtime.connect(
    source_language="en-us",
    target_language="de-de",
    voice_id=147320,
)

const session = await client.realtime.connect({
  sourceLanguage: "en-us",
  targetLanguage: "de-de",
  voiceId: 147320,
});

For the most natural-sounding results, choose a voice whose reference language matches your target_language. A large mismatch between the voice’s native language and the translation language can reduce clarity and accent accuracy.

Automatic source language detection

Set source_language to auto and the server works out which language is being spoken, and keeps listening for changes. It reports the language with session.source_language_changed, and labels every completed transcript with the language it was identified as:

{
  "type": "session.update",
  "session": {
    "mode": "slow",
    "source_language": "auto",
    "target_language": "en-us",
    "source_language_candidates": ["de-de", "fr-fr", "en-us"]
  }
}

{ "type": "session.source_language_changed", "language": "de-DE" }
{
  "type": "conversation.item.input_audio_transcription.completed",
  "transcript": "Können Sie mir bitte den Bericht schicken?",
  "item_id": "7b0d5c1e-...",
  "language": "de-DE",
  "passthrough": false
}

How it behaves by default:

First detection. The language is set after the first full sentence.
Switching. Once set, the language switches after about two sentences, or 4 seconds of speech, in the new language. A single word, a name, or one sentence in another language doesn’t switch it, and an utterance in the current language cancels a pending switch. Speech counts as each sentence ends, so a speaker who keeps talking without a pause still switches it.
Utterances that can’t be identified. Very short or ambiguous utterances (“ok”, a number) have no language and don’t count towards a switch.
Same language as the target. Speech already in the target language passes through as text: its transcript arrives with passthrough: true, and nothing is translated or spoken for it.
Candidates. Naming the languages you expect in source_language_candidates makes detection faster and more accurate. Without candidates, every language the server can detect is considered. A candidate that can’t be detected automatically is rejected when the session starts, with an error naming it.

Both modes support auto. In fast mode, set a starting target_language, which must be one of the supported languages; every later target must be one too.

Changing languages during the session

A session started with auto accepts further session.update events that change source_language, target_language, or both. Nothing else can change once the session is running.

Send	Effect
`{"target_language": "fr-fr"}`	Translate into French from the next utterance on.
`{"target_language": ""}`	Stop translating and only transcribe (`slow` mode only).
`{"source_language": "de-de"}`	Fix the source language. Detection keeps labelling transcripts but stops switching.
`{"source_language": "auto"}`	Go back to detecting the source language.

{ "type": "session.update", "session": { "target_language": "fr-fr" } }

The server answers with session.updated, and with session.source_language_changed when the source language changed. A change the session can’t make is answered with an error that has a code, and the session carries on:

`error.code`	Meaning
`session_update_field_not_supported`	The update named a field other than `source_language` or `target_language`, or the session wasn’t started with `auto`.
`target_language_unsupported_for_mode`	The session’s mode can’t translate into that language (for example, a language outside `fast` mode’s list). The previous target stays.

Speaker setup

speaker_setup tells detection who speaks into the session:

changing_speakers (default): a remote party, a meeting room, or a broadcast. The language switches as soon as there is enough evidence.
single_known_speaker: your own microphone. The language switches only after about 4 seconds of speech in the new language, however many sentences that is. And if the session hears its own target language for that long, it doesn’t switch. On speakers rather than headphones, that is usually the other party coming through. Instead the server sends:
{ "type": "session.source_language_switch_confirmation_requested", "language": "de-DE" }
To accept it, set the source language yourself: {"type": "session.update", "session": {"source_language": "de-de"}}. To reject it, do nothing.

If your client can tell that an utterance was echo or crosstalk (for example, it matches audio you just played), send its item_id back within about half a second, before the switch applies, so it doesn’t count towards a switch:

{ "type": "input_audio_buffer.ignore_for_language_detection", "item_id": "7b0d5c1e-..." }

Translating both sides of a call

A session translates one direction. For a conversation, open one session per direction and link them: each session translates into the language the other session detects. Whenever one session sends session.source_language_changed, send its language to the other session as target_language. Neither person has to pick a language.

import asyncio
import json
import os

import websockets

URL = "wss://realtime.camb.ai/v1/realtime?mode=slow"
HEADERS = [("x-api-key", os.environ["CAMB_API_KEY"])]


async def open_session(speaker_setup):
    session = await websockets.connect(URL, additional_headers=HEADERS)
    await session.send(json.dumps({
        "type": "session.update",
        "session": {
            "mode": "slow",
            "source_language": "auto",
            "speaker_setup": speaker_setup,
            "output_modalities": ["text", "audio"],
        },
    }))
    return session


async def main():
    local = await open_session("single_known_speaker")  # your microphone
    remote = await open_session("changing_speakers")  # the other party's audio
    ready = {local: asyncio.Event(), remote: asyncio.Event()}

    async def set_target_language(session, language):
        await ready[session].wait()  # a session takes changes once it has started
        await session.send(json.dumps({
            "type": "session.update",
            "session": {"target_language": language},
        }))

    async def read(session, other_session, play_translated_audio):
        async for message in session:
            if isinstance(message, bytes):
                play_translated_audio(message)  # PCM16, mono, 24 kHz
                continue
            event = json.loads(message)
            if event["type"] == "session.created":
                ready[session].set()
            elif event["type"] == "session.source_language_changed":
                # This side's speaker uses event["language"]: translate the other side into it.
                asyncio.create_task(set_target_language(other_session, event["language"]))

    await asyncio.gather(
        read(local, remote, play_to_the_other_party),
        read(remote, local, play_to_you),
        stream_microphone(local),  # input_audio_buffer.append, as in step 3 above
        stream_call_audio(remote),
    )


asyncio.run(main())

play_to_the_other_party, play_to_you, stream_microphone and stream_call_audio are your own audio plumbing. Use single_known_speaker for the microphone session, and wire up session.source_language_switch_confirmation_requested if your users may be on speakers. A confirmed switch arrives as session.source_language_changed like any other, so the link passes it on. The Python and TypeScript SDKs do this linking for you. connect_session_pair (connectSessionPair in TypeScript) opens both sessions, sets their speaker setups, and keeps each session’s target language equal to the other’s source language:

pair = await client.realtime.connect_session_pair(mode="slow")
async with pair:
    await pair.wait_until_ready()
    await asyncio.gather(
        pair.local_speaker_session.stream_audio(microphone),
        pair.other_party_session.stream_audio(call_audio),
    )

const pair = await client.realtime.connectSessionPair({ mode: RealtimeMode.Slow });
await pair.waitUntilReady();
await Promise.all([
  pair.localSpeakerSession.stream(microphone),
  pair.otherPartySession.stream(callAudio),
]);

See Detecting the source language in the SDK tutorial.

Supported languages

source_language and target_language accept the BCP-47 tags below (case-insensitive). Pick any supported language as the source and any supported language as the target.

Supported realtime languages (14)

Code	Language
`ar-ae`	Arabic (United Arab Emirates)
`ar-eg`	Arabic (Egypt)
`ar-sa`	Arabic (Saudi Arabia)
`de-de`	German (Germany)
`en-gb`	English (United Kingdom)
`en-us`	English (United States)
`es-es`	Spanish (Spain)
`fr-ca`	French (Canada)
`fr-fr`	French (France)
`hi-in`	Hindi (India)
`ja-jp`	Japanese (Japan)
`ko-kr`	Korean (Korea)
`pt-br`	Portuguese (Brazil)
`zh-cn`	Chinese (Mandarin, Simplified)

Client events

Event	Support
`session.update`	Required first message. Later ones may change `source_language` and `target_language` in sessions started with `auto`; see Changing languages during the session.
`input_audio_buffer.append`	Supported after activation.
`input_audio_buffer.ignore_for_language_detection`	Supported in sessions started with `auto`. Stops an utterance counting towards a source language switch.
`input_audio_buffer.clear`	Recognized but not supported; returns an `unsupported_event` error and the session carries on.
`input_audio_buffer.commit`	Recognized but not supported; returns an `unsupported_event` error and the session carries on.
`response.cancel`	Recognized but not supported; returns `error`.

Server events

Event	Description
`session.created`	Session has been authorized, started, and activated.
`session.updated`	Active session configuration.
`conversation.item.input_audio_transcription.completed`	Completed user transcript. In sessions started with `auto`, it also carries `item_id`, `language`, and `passthrough`.
`response.text.delta`	Additive translated text delta.
`response.text.done`	Final translated text.
`response.audio.delta`	Base64-encoded translated audio bytes.
`response.audio.done`	Current translated audio response is complete.
`session.source_language_changed`	The source language was detected, switched, or set by the client. Sessions started with `auto` only.
`session.source_language_switch_confirmation_requested`	A `single_known_speaker` session heard its own target language and is asking before switching.
`error`	Something went wrong. See Errors.

Limits

Limit	Value
Initial `session.update` timeout	60 seconds
Idle timeout (no audio or events from the client)	60 seconds
Maximum session duration	4 hours
Maximum client event text size	1 MiB
Maximum decoded audio payload per `input_audio_buffer.append`	256 KiB
Audio encoding in `input_audio_buffer.append.audio`	Base64-encoded audio bytes

Billing

Active sessions are charged in billing windows and finalized on close, failure, or billing stop. If billing stops a session, for example when credits run out, the server sends an error event with code insufficient_credits (or billing_stopped) and closes the session.

Errors

Errors arrive as an error event:

{
  "type": "error",
  "error": {
    "type": "authentication_error",
    "code": "invalid_api_key",
    "message": "The API key is invalid or has expired. Check the key, or create a new one.",
    "param": null
  }
}

Field	Description
`type`	The category: `invalid_request_error`, `authentication_error`, `permission_error`, `billing_error`, `rate_limit_error`, or `server_error`.
`code`	Stable and machine-readable. Branch on this, not on `message`.
`message`	A description written for you, safe to log or show.
`param`	The field at fault, for example `session.target_language`, or `null`.

When an error ends the session, the server closes the WebSocket right after the error event. The close frame’s reason is the same code, so a client that missed the event can still tell what happened. The close code is standard:

Close code	Meaning
`1000`	Normal end, including reaching the maximum session duration.
`1008`	A problem you can fix: credentials, credits, a bad request, a timeout.
`1009`	A message over the size limit.
`1011`	A failure on our side. Retrying usually works.
`1012`	The server is restarting. Reconnect to start a new session.
`1013`	Too many sessions right now. Try again shortly.

Errors that end the session:

`code`	When
`missing_api_key`	No `x-api-key` header and no `auth` in the first `session.update`.
`invalid_api_key`	The API key is invalid or has expired.
`permission_denied`	The credentials aren’t allowed to start this session.
`insufficient_credits`	Not enough credits to start, or credits ran out during the session.
`billing_stopped`	Billing stopped the session for another reason.
`billing_unavailable`	The session couldn’t be billed, so it was stopped.
`rate_limited`	Too many sessions are being started.
`voice_not_found`	The `voice.voice_id` isn’t a voice in your account.
`session_update_timeout`	No `session.update` arrived within 60 seconds of connecting.
`first_event_not_session_update`	The first message wasn’t a `session.update` event.
`invalid_session_configuration`	The session configuration is invalid; `param` names the field.
`unsupported_language`	The mode can’t serve the requested language; `param` names the field.
`automatic_language_detection_unavailable`	`source_language: "auto"` isn’t available.
`event_too_large`	An event was over 1 MiB.
`input_audio_too_large`	An audio chunk was over 256 KiB.
`idle_timeout`	No audio or events from the client for 60 seconds.
`keepalive_timeout`	The client stopped answering WebSocket pings.
`max_session_duration_reached`	The session reached 4 hours.
`client_too_slow`	The client isn’t reading the session’s output fast enough.
`session_start_failed`	The session couldn’t start because of a failure on our side.
`session_failed`	The session failed while running because of a failure on our side.
`server_shutting_down`	The server is restarting.

Errors the session survives. The request is refused and the session carries on:

`code`	When
`invalid_event`	An event wasn’t valid JSON, had an unknown `type`, or had a field of the wrong shape. The message says what was wrong.
`unsupported_event`	A recognized event this server doesn’t support.
`session_update_field_not_supported`	A later `session.update` changed something other than languages. See Changing languages during the session.
`target_language_unsupported_for_mode`	The session’s mode can’t translate into that language.
`session_update_failed`	A valid language change couldn’t be applied.

If the request is refused before the WebSocket opens (an unsupported mode query parameter), the server answers with HTTP 400 and the same error object as the body.

Messages

session.starting

type:object

type

type:string

required

session.starting

Session Created

type:object

Sent after authorization, startup, and activation complete.

type

type:string

required

session.created

session

type:object

required

mode

type:enum

Latency/quality tradeoff. fast accepts audio almost immediately and translates with the lowest latency, but supports fewer languages. slow supports the full language list and translates more accurately, at the cost of a 30s+ cold boot before the session is ready. May also be supplied as a mode query parameter on the upgrade URL; a value here takes precedence.

Available options: fast, slow

model

type:enum

deprecated

Deprecated alias for mode, kept so pre-rename integrations keep working. The retired engine codenames resolve to a mode: iris to fast, and lilac/violet/orchid to slow. mode takes precedence if both are sent. Also accepted as a model query parameter on the upgrade URL.

Available options: fast, slow, iris, lilac, violet, orchid

source_language

type:string

required

Source language tag, for example en-US, or auto to have the server detect the language being spoken. See Automatic source language detection.

target_language

type:string

Target language tag, for example de-DE. With source_language: "auto" it may be omitted in slow mode: the session then transcribes without translating until a target is set with a later session.update. fast mode always needs one.

speaker_setup

type:enum

Only with source_language: "auto". Who speaks into the session, which decides how readily the detected language switches. changing_speakers suits a remote party, a meeting room or a broadcast. single_known_speaker suits the local microphone: its language switches only after sustained speech, so the other party heard through the loudspeakers cannot switch it, and a switch to the session's own target language is sent as session.source_language_switch_confirmation_requested instead of being applied.

Available options: changing_speakers, single_known_speaker

source_language_candidates

type:array

Only with source_language: "auto". The languages detection may choose from. Naming the languages you expect makes detection faster and more accurate; when omitted, every language the server can detect is considered.

item

type:string

output_modalities

type:array

item

type:enum

Available options: text, audio

voice

type:object

Output voice selection. Use the built-in voice or one of your cloned voices.

type:string

required

Durable realtime session ID.

Session Updated

type:object

Sent immediately after session.created with the active session configuration.

type

type:string

required

session.updated

session

type:object

required

mode

type:enum

Available options: fast, slow

model

type:enum

deprecated

Available options: fast, slow, iris, lilac, violet, orchid

source_language

type:string

required

Source language tag, for example en-US, or auto to have the server detect the language being spoken. See Automatic source language detection.

target_language

type:string

speaker_setup

type:enum

Available options: changing_speakers, single_known_speaker

source_language_candidates

type:array

item

type:string

output_modalities

type:array

item

type:enum

Available options: text, audio

voice

type:object

Output voice selection. Use the built-in voice or one of your cloned voices.

conversation.item.input_audio_transcription.delta

type:object

type

type:string

required

conversation.item.input_audio_transcription.delta

delta

type:string

required

Newly transcribed source-language text. Additive within one utterance; resets after the matching conversation.item.input_audio_transcription.completed.

Input Audio Transcription Completed

type:object

Completed user transcript produced by the realtime pipeline.

type

type:string

required

conversation.item.input_audio_transcription.completed

transcript

type:string

required

item_id

type:string

Identifies the utterance, for input_audio_buffer.ignore_for_language_detection. Sent only when the source language is detected automatically.

language

type:string

The language the utterance was identified as, for example de-DE. Sent only when the source language is detected automatically, and omitted when the utterance was too short or too ambiguous to identify.

passthrough

type:boolean

true when the utterance was already in the target language, so it was passed through as text and was neither translated nor spoken. Sent only when the source language is detected automatically.

Response Text Delta

type:object

Incremental translated text. The delta is additive for the current response.

type

type:string

required

response.text.delta

delta

type:string

required

Additive translated text delta for the current response.

Response Text Done

type:object

Final translated text for the current response.

type

type:string

required

response.text.done

text

type:string

required

Final translated text.

Response Audio Delta

type:string

Base64-encoded synthesized output audio bytes.

Response Audio Done

type:object

Current assistant audio response is complete.

type

type:string

required

response.audio.done

Error

type:object

Structured error for unsupported recognized events and billing stop decisions.

type

type:string

required

error

type:object

required

type

type:enum

required

The error's category.

Available options: invalid_request_error, authentication_error, permission_error, billing_error, rate_limit_error, server_error

code

type:enum

required

Stable, machine-readable error code; branch on this, not on message. When the error ends the session, the WebSocket close frame repeats it as the close reason.

Available options: missing_api_key, invalid_api_key, permission_denied, insufficient_credits, billing_stopped, billing_unavailable, rate_limited, voice_not_found, session_update_timeout, first_event_not_session_update, invalid_event, invalid_session_configuration, unsupported_language, automatic_language_detection_unavailable, event_too_large, input_audio_too_large, unsupported_event, session_update_field_not_supported, target_language_unsupported_for_mode, session_update_failed, idle_timeout, keepalive_timeout, max_session_duration_reached, client_too_slow, session_start_failed, session_failed, server_shutting_down

message

type:string

required

A description written for the caller, safe to log or show.

param

type:stringnull

required

The field at fault, for example session.target_language, or null.

Source Language Changed

type:object

The session's source language changed: detected for the first time, detected as having switched, or set by the client. Sent only when the source language is detected automatically.

type

type:string

required

session.source_language_changed

language

type:string

required

The new source language.

previous_language

type:string

The source language before the change. Omitted when the language was detected for the first time.

Source Language Switch Confirmation Requested

type:object

A single_known_speaker session heard sustained speech in its own target language, which is usually the other party coming through the loudspeakers, so it did not switch. Confirm with session.update setting source_language, or ignore it.

type

type:string

required

session.source_language_switch_confirmation_requested

language

type:string

required

The language the session heard sustained speech in, which is its own target language.

Update Session

type:object

First client event. Authorizes and activates the realtime session.

type

type:string

required

session.update

session

type:object

required

The first session.update carries the full session configuration. A later one may only change languages.

mode

type:enum

Available options: fast, slow

model

type:enum

deprecated

Available options: fast, slow, iris, lilac, violet, orchid

source_language

type:string

required

Source language tag, for example en-US, or auto to have the server detect the language being spoken. See Automatic source language detection.

target_language

type:string

speaker_setup

type:enum

Available options: changing_speakers, single_known_speaker

source_language_candidates

type:array

item

type:string

output_modalities

type:array

item

type:enum

Available options: text, audio

voice

type:object

Output voice selection. Use the built-in voice or one of your cloned voices.

source_language

type:string

A language tag fixes the source language, which stops detection from switching it (this is also how a switch confirmation request is confirmed). auto resumes detection.

target_language

type:string

Translate into this language from the next utterance on. An empty string stops translating (slow mode only). In fast mode it must be one of fast mode's languages, or the update is refused with target_language_unsupported_for_mode.

auth

type:object

api_key

type:string

required

Append Input Audio

type:object

Append base64-encoded microphone audio bytes to the realtime input stream.

type

type:string

required

input_audio_buffer.append

audio

type:string

required

Base64-encoded audio bytes. The decoded payload can be up to 256 KiB.

Clear Input Audio Buffer

type:object

Recognized but not supported in this version. The server responds with an error event.

type

type:string

required

input_audio_buffer.clear

Commit Input Audio Buffer

type:object

Recognized but not supported in this version. The server responds with an error event.

type

type:string

required

input_audio_buffer.commit

Cancel Response

type:object

Recognized but not supported in this version. The server responds with an error event.

type

type:string

required

response.cancel

Input Audio Buffer Ignore For Language Detection

type:object

Stop counting an utterance the client recognised as echo or crosstalk towards a source language switch. Only has an effect while the switch it supports is still pending.

type

type:string

required

input_audio_buffer.ignore_for_language_detection

item_id

type:string

required

The item_id of a conversation.item.input_audio_transcription.completed event.

Last modified on October 22, 2026

Messages

Getting Started

Models

Tutorials

SDK Guides

Hosting Platforms

Integrations

API Reference

Other Products

Release Logs

Realtime Speech To Speech

Quickstart

Integration in 4 steps

Authentication

Reference

Session configuration

Voice selection

Automatic source language detection

Changing languages during the session

Speaker setup

Translating both sides of a call

Supported languages

Client events

Server events

Limits

Billing

Errors

​Quickstart

​Integration in 4 steps

​Authentication

​Reference

​Session configuration

​Voice selection

​Automatic source language detection

​Changing languages during the session

​Speaker setup

​Translating both sides of a call

​Supported languages

​Client events

​Server events

​Limits

​Billing

​Errors

Quickstart

Integration in 4 steps

Authentication

Reference

Session configuration

Voice selection

Automatic source language detection

Changing languages during the session

Speaker setup

Translating both sides of a call

Supported languages

Client events

Server events

Limits

Billing

Errors