Build a conversation.
Make it yours.

Connect with JavaScript

Run on Node.js with npm install ws. Save the example as realtime-node.mjs, set SOPHEA_URL to the matching HTTPS API origin and SOPHEA_DEVICE_KEY to your key, then run node realtime-node.mjs. Both company and assistant keys work with their matching service.

This example sends text and saves a playable reply.wav. For voice input, set SOPHEA_INPUT_PCM to a raw mono PCM16, 24 kHz file. It sends audio at real-time speed and reports first received audio plus server timing metrics. For a private CA, set NODE_EXTRA_CA_CERTS to the trusted CA file.

// npm install ws
// Set SOPHEA_URL (HTTPS origin) and SOPHEA_DEVICE_KEY, then:
// node examples/realtime-node.mjs
// Optional: SOPHEA_INPUT_PCM=/path/to/mono-24000hz-s16le.pcm
import WebSocket from 'ws';
import { readFileSync, writeFileSync } from 'node:fs';
import { setTimeout as sleep } from 'node:timers/promises';

if (!process.env.SOPHEA_URL || !process.env.SOPHEA_DEVICE_KEY) {
  throw new Error('Set SOPHEA_URL and SOPHEA_DEVICE_KEY.');
}
const url = new URL('/v1/realtime', process.env.SOPHEA_URL);
if (url.protocol !== 'https:') throw new Error('Use a trusted HTTPS origin.');
url.protocol = 'wss:';
url.searchParams.set('model', 'sophea-realtime');
const input = process.env.SOPHEA_INPUT_PCM
  ? readFileSync(process.env.SOPHEA_INPUT_PCM) : null;
if (input && (input.length < 4800 || input.length > 30 * 48000 || input.length % 2)) {
  throw new Error('Input must be 0.1–30 seconds of raw mono PCM16 at 24 kHz, without a WAV header.');
}
const ws = new WebSocket(url, {
  headers: { Authorization: `Bearer ${process.env.SOPHEA_DEVICE_KEY}` },
  handshakeTimeout: 15000,
});
const send = event => ws.send(JSON.stringify(event));
const chunks = [];
let started = 0, firstAudioMs = null, finished = false, submitted = false;
function fail(message) {
  if (finished) return;
  finished = true;
  console.error(message);
  process.exitCode = 1;
  ws.terminate();
}
const deadline = setTimeout(() => fail('Conversation timed out.'), 120000);
ws.on('error', () => fail('Connection failed. Check the URL, key and trusted CA.'));
ws.on('close', () => {
  clearTimeout(deadline);
  if (!finished) fail('Connection closed before the reply completed.');
});
ws.on('message', async raw => {
  try {
    const event = JSON.parse(raw.toString());
    if (event.type === 'session.created') {
      send({ type: 'session.update', session: {
        audio: { input: { turn_detection: null }, output: { voice: 'Vivian' } },
        // Company keys: add instructions and tools here if needed.
        // Assistant keys: omit instructions to keep the published prompt.
      }, extra_body: { tts_engine: 'qwen', language: 'auto' } });
    } else if (event.type === 'session.updated' && !submitted) {
      submitted = true;
      if (input) {
        for (let offset = 0; offset < input.length; offset += 960) {
          if (finished) return;
          send({ type: 'input_audio_buffer.append',
            audio: input.subarray(offset, offset + 960).toString('base64') });
          await sleep(20); // 20 ms of audio, paced in real time.
        }
        send({ type: 'input_audio_buffer.commit' });
      } else {
        send({ type: 'conversation.item.create', item: {
          type: 'message', role: 'user',
          content: [{ type: 'input_text', text: 'Hello!' }],
        } });
      }
      started = Date.now();
      send({ type: 'response.create' });
    } else if (event.type === 'response.output_audio.delta') {
      firstAudioMs ??= Date.now() - started;
      chunks.push(Buffer.from(event.delta, 'base64'));
    } else if (event.type === 'response.output_audio_transcript.delta') {
      process.stdout.write(event.delta);
    } else if (event.type === 'error') {
      fail(event.error?.message || 'The API rejected the request.');
    } else if (event.type === 'response.done') {
      if (event.response.status !== 'completed' || !chunks.length) {
        fail('Reply did not complete with audio.');
        return;
      }
      const pcm = Buffer.concat(chunks);
      const header = Buffer.alloc(44);
      header.write('RIFF', 0); header.writeUInt32LE(36 + pcm.length, 4);
      header.write('WAVEfmt ', 8); header.writeUInt32LE(16, 16);
      header.writeUInt16LE(1, 20); header.writeUInt16LE(1, 22);
      header.writeUInt32LE(24000, 24); header.writeUInt32LE(48000, 28);
      header.writeUInt16LE(2, 32); header.writeUInt16LE(16, 34);
      header.write('data', 36); header.writeUInt32LE(pcm.length, 40);
      writeFileSync('reply.wav', Buffer.concat([header, pcm]));
      console.log('\nSaved reply.wav', {
        firstReceivedAudioMs: firstAudioMs,
        serverMetrics: event.response.metadata,
      }); // Receive timing does not include speaker playback.
      finished = true;
      clearTimeout(deadline);
      ws.close();
    }
  } catch {
    fail('Could not process this response or write reply.wav.');
  }
});

This is server/device JavaScript. Keep the key in your backend or trusted device. Browser apps should connect through your authenticated backend; do not ship a permanent key in browser code. Sophea does not provide the OpenAI ephemeral-key or WebRTC endpoints.

Connect with Python

Install openai. Set SOPHEA_URL to your API origin and SOPHEA_DEVICE_KEY to your key. Use trusted HTTPS. This example sends text and saves the agent’s reply as playable audio.

import asyncio, base64, os, wave
from openai import AsyncOpenAI

async def main():
    async with AsyncOpenAI(
        api_key=os.environ["SOPHEA_DEVICE_KEY"],
        base_url=os.environ["SOPHEA_URL"].rstrip("/") + "/v1",
    ) as client:
        async with client.realtime.connect(model="sophea-realtime") as connection:
            await connection.session.update(session={
                "audio": {"input": {"turn_detection": None}}
            })
            await connection.conversation.item.create(item={
                "type": "message", "role": "user",
                "content": [{"type": "input_text", "text": "Hello!"}]
            })
            await connection.response.create()
            with wave.open("reply.wav", "wb") as output:
                output.setnchannels(1)
                output.setsampwidth(2)
                output.setframerate(24000)
                async for event in connection:
                    if event.type == "response.output_audio.delta":
                        output.writeframes(base64.b64decode(event.delta))
                    elif event.type == "error":
                        raise RuntimeError(event.error.message)
                    elif event.type == "response.done":
                        if event.response.status != "completed":
                            raise RuntimeError("Response did not complete")
                        break

asyncio.run(main())

This API implements an OpenAI Realtime GA WebSocket subset. Use model sophea-realtime and Sophea voice names. WebRTC, SIP, semantic VAD, images and per-response configuration are not supported.

Voice, language & instructions

Call GET /api/platform/voice-api/voices with Authorization: Bearer YOUR_KEY to discover available voices and languages. Select a voice before the first audio response. Enabled languages: English, French, German, Spanish, Italian, Portuguese, Russian, Japanese, Korean, Chinese and Greek. Qwen streams text and audio over a persistent upstream WebSocket; Greek uses Echo’s default voice. Other languages and Supertonic are disabled. Voice identity changes when switching between Qwen and Echo. Streaming Qwen speed is fixed at 1.0. Inspect native_languages, fallback_engine and fallback_voice in the catalogue. Echo accepts Greek or automatic. Speech recognition must also support your selected language.

await connection.send({
    "type": "session.update",
    "session": {"audio": {"output": {"voice": "Vivian"}}},
    "extra_body": {"tts_engine": "qwen", "language": "en"}
})

send({
  type: "session.update",
  session: { audio: { output: { voice: "Vivian" } } },
  extra_body: { tts_engine: "qwen", language: "auto" }
});

extra_body is a Sophea event extension; send it with Python connection.send or the JavaScript send helper above. Update settings between responses and after clearing or committing pending audio. To customize behavior, set session.instructions. Omit it to keep the published assistant’s instructions. Voice pacing and tool permissions still apply.

Saved designed voices: create a preview in the platform’s Voice Designer, name it and choose Save voice. The authenticated voice catalogue includes your workspace’s saved IDs in voices and their display names in each engine’s saved_voices map. Set session.audio.output.voice to that custom_… ID and extra_body.tts_engine to qwen. Saved voices use Qwen Base with the original generated reference and support incremental text/audio streaming. They support Qwen’s ten native languages; Greek switches to Echo, so it does not preserve the designed voice. Removing a saved voice makes it unavailable for future sessions and Realtime turns. Update any assistants using it before removal.

Automatic language switching: use extra_body: {"language":"auto"} with a Qwen voice. Clear language changes update the reply language. Requests such as “reply in Chinese” persist even if the user continues speaking Greek. Say “follow my language again” or set language: "auto" again between turns to clear that preference. Clear short greetings and corrections such as “hi” and “I said HI” can switch languages. Acknowledgements such as “OK” and ambiguous mixed speech keep the current language. Generated replies are checked before synthesis; a mismatch or check failure uses a localized retry message. A specific code such as el fixes the language; Echo stays Greek. Input recognition remains automatic when the output preference changes. Accuracy depends on the recognizer’s transcript; this does not guarantee word-by-word mixed-language transcription.

In response.done.metadata, inspect sophea_tts_engine and sophea_tts_voice for Qwen routing, sophea_output_language, sophea_language_reason, sophea_language_detection_ms, sophea_output_language_check and sophea_output_language_check_ms. Detection and output checking are included in first-audio timing and token usage. State belongs to the connection; after reconnecting, reapply any required language setting.

Investigating pauses: response.done.metadata also reports sophea_asr_ms (recognition duration), sophea_asr_wait_ms (recognition time still pending when the response started), sophea_service_setup_ms, sophea_generation_ms (including language checks, grounding and server tools), and sophea_tts_wait_ms (approved text to first audio). When first-phrase preparation is enabled, sophea_first_phrase_prefetch reports whether it was used or discarded and sophea_prefetch_synthesis_ms measures preparation. These stages can overlap; do not add all fields together. Server timings exclude device playback buffering.

Add previous user and assistant messages through conversation.item.create before response.create. Use text content with input_text for user messages and output_text for assistant messages. Use session.update for session-wide instructions.

Stream audio in both directions

For automatic turn detection, use audio.input.turn_detection: {"type":"server_vad"}. On input_audio_buffer.speech_started, stop queued playback and truncate the previous assistant audio to what was heard. Do not also create manual responses for VAD-triggered turns.

Let the agent call your functions

Company keys support client-side functions supplied in session.tools as flat objects with type: "function", name, description and a JSON parameters schema. Assistant keys use the published agent’s server tools.

When a completed response contains a function_call, validate its name and JSON arguments against your own allowlist, run the function, and return its result:

await connection.conversation.item.create(item={
    "type": "function_call_output",
    "call_id": call.call_id,
    "output": json.dumps(result)
})
await connection.response.create()

Import json in your client. Handle each call ID once. Results belong to the connection that requested them. Return results within 30 seconds; a timeout does not prove an external action failed, so do not blindly retry side effects. Up to 16 functions and 16,000 characters per result are supported.

Direct device protocol

Use this alternative if your device sends binary audio without the Realtime SDK. Create a session through the reference below, then connect to the returned ws_path on this origin with ?ticket= plus the URL-encoded ws_ticket. Connect before expires_in; tickets are single-use secrets.

This protocol uses 16,000 Hz mono PCM16 binary frames, not the Realtime interface’s 24 kHz base64 audio. Prefer 20 ms / 640-byte frames at real-time speed. Parse text frames as JSON events and play binary frames as audio.

Handle errors and limits

Keys are scoped to their service and can expire or be revoked. Keep them out of URLs, logs and browser bundles. Reconnect with a fresh session after a connection closes.

Sessions last at most 30 minutes and close after 60 seconds without input. Keep audio paced in real time. Direct frames are limited to 32,000 bytes and 100 frames/second. Keep Realtime JSON messages below 256 KiB and send short audio chunks. Close codes: 4401 invalid access, 4408 timeout, 4409 audio limit, 4429 throttled, 1009 message too large.

API reference

Authorize with your key, without the Bearer prefix. Credentials stay in memory and clear on reload. Creating a session uses account quota. Try it out does not connect a microphone. Use the SDK for the WebSocket endpoint.