Build realtime voice features with Wiro AI, such as live voice conversation, streaming text-to-speech and streaming speech-to-text, by starting a task through the REST API and streaming PCM audio over the Wiro WebSocket. Use when the user wants a voice assistant, a live voice conversation, streaming text-to-speech or speech-to-text, or WebSocket audio in an app built on Wiro.
Wiro Realtime Voice
A Wiro realtime model keeps a session open. One REST call starts a task, then raw PCM audio and JSON events flow over a WebSocket until the session ends. For the REST basics (auth headers, model search, prices, task lookup), use the wiro-ai skill.
When to use it
| Mode | Client sends | Server sends | Good for |
|---|---|---|---|
| Voice conversation | Microphone audio | AI audio, plus TRANSCRIPT_USER: and TRANSCRIPT_AI: text | Voice assistants, interactive voice chat |
| Text-to-speech | Text in the Run call, no microphone | AI audio only | Narration, voiceover, assistants |
| Speech-to-text | Microphone audio | TRANSCRIPT_USER: text only, no audio | Live dictation, captioning, meeting transcription |
For a finished audio file rather than a live stream, a standard text-to-speech model through Run fits better (see wiro-ai).
Start a session
- Pick a model and read its parameters with
POST /Tool/Detail. They differ per model. - From server-side code (see Credentials in apps), create the task with
POST https://api.wiro.ai/v1/Run/{owner}/{model}. Use this asynchronous Run; the/syncvariant is for finite models. - Open
wss://socket.wiro.ai/v1and register withtask_info. - Wait for
task_stream_ready. Send no audio before it. - Stream microphone frames, play the audio frames that come back, or both.
- Send
task_session_end, wait fortask_postprocess_end, then close the socket.
Run body for openai/gpt-realtime-mini from the docs, sent with x-api-key: YOUR_API_KEY:
{
"voice": "marin",
"system_instructions": "You are a friendly assistant.",
"input_audio_format": "audio/pcm",
"output_audio_format": "audio/pcm",
"input_audio_rate": "24000",
"output_audio_rate": "24000"
}
The other docs examples send {"text": "Hello, this is a realtime text-to-speech demo."} to openmoss/moss-tts-realtime and {"transcriptionDelay": "balanced"} to mistralai/voxtral-mini-4b-realtime-2602. Run returns the task id and the task token. Check result and errors before you connect:
{"result": true, "errors": [], "taskid": "TASK_ID", "socketaccesstoken": "TASK_TOKEN"}
The client sends these JSON text messages, plus binary audio frames (see Audio):
{"type": "task_info", "tasktoken": "TASK_TOKEN"}
{"type": "task_session_interrupt", "tasktoken": "TASK_TOKEN"}
{"type": "task_session_end", "tasktoken": "TASK_TOKEN"}
The server sends:
| Event | Meaning |
|---|---|
task_stream_ready | The session is ready. Start the microphone. In TTS, audio starts arriving. |
| Binary frame | AI audio (conversation and TTS). |
task_output | A transcript in message, prefixed TRANSCRIPT_USER: (the person) or TRANSCRIPT_AI: (the AI). |
task_stream_end | Conversation: the AI finished its turn. TTS: audio for this segment is done. STT: no more transcripts will come. |
task_cost | turnCost, cumulativeCost and usage, the provider's raw breakdown (its shape varies). |
task_end | The model process exited. Keep listening. |
task_postprocess_end | The last event. Close the socket now. |
task_cancel | The task was cancelled or killed. Treat it as the end. |
{"type": "task_output", "message": "TRANSCRIPT_USER:What's the weather like today?"}
The socket can also send the general task events from https://wiro.ai/docs/websocket (task_queue, task_start and so on); the examples treat them as information only. Strip the transcript prefix and update the UI on every message; speech-to-text sends partial results as words are recognized.
- Conversation turns: keep the microphone streaming. To cut the AI off, stop local playback at once and send
task_session_interrupt. The server stops that reply, sends a finaltask_stream_endfor it, and the session goes on. Models whose provider does its own barge-in (the OpenAI realtime family and ElevenLabs) may also stop on their own when the person starts talking. The example apps rely on that withgpt-realtime-miniand silence their local playback. - Text-to-speech: the stream usually ends by itself once the text is spoken (
task_stream_end,task_cost,task_end,task_postprocess_end). Let queued audio finish before you close the player. Sendtask_session_endto stop early. - Speech-to-text: keep sending audio until you end the session. The model returns text only when it hears speech.
Audio
- Both directions use raw PCM: signed 16-bit (Int16), 24,000 Hz, mono, little-endian. If the model takes format or rate parameters, send the values above (
audio/pcm,24000) so the Run call matches your pipeline. For speech-to-text, always send 24 kHz; the server converts it for the model. - A binary frame is the task token as UTF-8 bytes, one
|byte (0x7C), then the PCM bytes. Send audio only as binary frames. - On receive, check for a binary frame before you parse JSON, then take everything after the first
|byte as PCM. - Chunks: the docs use 200 ms (4,800 samples, 9,600 bytes). The mobile examples send 20 ms (480 samples) so barge-in reacts faster.
- Playback: convert Int16 to Float32 (divide by 32768), schedule each chunk at
max(currentTime, nextPlayTime), then advancenextPlayTimeby the chunk's length, so chunks play back to back without gaps or clicks. - Browser capture (docs sample):
getUserMediawithechoCancellationandnoiseSuppression, anAudioContextat 24,000 Hz, and an AudioWorklet that buffers samples and converts them to Int16. - Mobile (examples): use one audio graph for capture and playback so the OS echo canceller hears both sides. For conversation, the iOS examples set
AVAudioSessionto.playAndRecordwith mode.voiceChat, and the Android examples record fromVOICE_COMMUNICATIONand addAcousticEchoCancelerwhen the device has one. On Android, play TTS-only audio on a media track; a voice-communication track without a call route plays from the earpiece. - Permissions: conversation and STT need the microphone, TTS does not. iOS needs
NSMicrophoneUsageDescriptionplus a runtime request. Android needsRECORD_AUDIO(requested at runtime) andINTERNET. Test on a physical device; the examples note that simulator and emulator microphones are not a usable voice path.
Framing an outgoing chunk, adapted from ReactNative-Realtime src/realtime/binaryFrame.ts (MIT):
import { utf8ToBytes } from '@noble/hashes/utils.js';
const SEPARATOR = '|'.charCodeAt(0); // 0x7C
function encode(taskToken: string, payload: Uint8Array): Uint8Array {
const prefix = utf8ToBytes(taskToken);
const frame = new Uint8Array(prefix.length + 1 + payload.length);
frame.set(prefix, 0);
frame[prefix.length] = SEPARATOR;
frame.set(payload, prefix.length + 1);
return frame; // send as a binary WebSocket message
}
Choose a model and know the cost
- Conversation models in the docs:
openai/gpt-realtime-mini,openai/gpt-realtimeandelevenlabs/realtime-conversational-ai. - Text-to-speech and speech-to-text: browse https://wiro.ai/models with the Realtime TTS or Realtime STT category. The docs use
openmoss/moss-tts-realtimeandmistralai/voxtral-mini-4b-realtime-2602; the example apps also offeropenbmb/voxcpm2for speech. - Typical parameters: voice and system instructions for conversation; the text and an optional reference audio clip for voice cloning for TTS; a transcription delay that trades speed for accuracy, and sometimes a language, for STT. Confirm them with
POST /Tool/Detailbefore you build the Run body. - Each model has its own price, shown on its model page and at https://wiro.ai/product/pricing (
wiro-aireads it from the API). A session is charged while it runs; eachtask_costevent givesturnCostfor the latest turn and the running total incumulativeCost.
Money rules:
- Before you start a session for the user, a test call included, tell them the model and the expected cost.
- A session bills while it runs. End sessions the user no longer needs. Ask before long or open-ended ones, such as always-on listening, a session left open in the background, or test loops.
- In apps, end the session when the user leaves the screen, and cap how long one can run. The example apps hang up when their screen is left, and the Flutter one also ends a session when its own timer runs out.
Credentials in apps
- Never ship a Wiro API key or secret in a browser or mobile app. Build config values (
Secrets.xcconfig,secrets.properties,--dart-define,.env) end up inside the binary; the examples use them for local runs only. - The key is needed for one call:
POST /Run. The WebSocket subscribe, the audio frames and the session end are authorized by the per-task token alone, so the key can stay on your server. - The pattern: your backend holds
WIRO_API_KEY(plusWIRO_API_SECRETfor a Signature project), decides who may start a session and for how long, calls Run, and returns onlytaskidandsocketaccesstokento the app. It also stops tasks for the app withPOST /Task/Cancel(taskid, while queued) orPOST /Task/Kill(socketaccesstoken, once running). - Each example app keeps those calls behind a
WiroApiClientinterface (runModel,terminateTaskBestEffort), so you can point it at your backend. The Swift README section "Shipping to users" shows how. - The task token expires with the task. Keep tokens and keys out of logs.
- Run auth headers:
x-api-key, orx-api-key,x-nonceandx-signaturefor a Signature project. Thewiro-aiskill shows the signing.
Errors and cleanup
- End every session the same way:
task_session_end, a bounded wait fortask_postprocess_end, then close the socket. If a client disconnects without it, the server stops the session, but do not rely on that. - A Run response with
result: falseor entries inerrorshas no task token, so there is no session to open. The conversation examples retry network, server-unavailable and timeout failures with backoff, and never retry authentication errors or rejected requests. - If
task_stream_readydoes not arrive in time, close the socket and have the backend stop that task before you try again, so nothing keeps running. - If the socket drops mid-conversation, the examples first resubscribe to the same task (a new socket, the same
task_infoand token) within a small retry budget, and start a new task only if that fails. A new task is a fresh assistant with no memory of the conversation. - The server can end a session too, for example when the wallet balance runs out; you still get
task_costandtask_end. Treattask_postprocess_endortask_cancelas the end whenever they arrive, and do not start a replacement on your own. - On every exit path (stop, error, leaving the screen), stop the microphone and release the audio session. The examples also guard the microphone so two features never hold it at once.
Example apps
| Repo | Platform | What it shows |
|---|---|---|
| Swift-Realtime | iOS, Swift and SwiftUI | One AVAudioEngine graph; the README section "Shipping to users" sketches a backend client |
| Kotlin-Realtime | Android, Kotlin and Jetpack Compose | One AudioRecord and AudioTrack graph, session lifecycle tests with fakes |
| Dart-Realtime | Flutter on Android and iOS | A native echo-cancelling audio package and a session time limit |
| ReactNative-Realtime | React Native and TypeScript on Android and iOS | The protocol in pure TypeScript, native audio behind one WiroPcmAudio TurboModule |
Each app has the same three tabs over one protocol: conversation with barge-in (openai/gpt-realtime-mini), live transcription (mistralai/voxtral-mini-4b-realtime-2602) and streaming text-to-speech (openmoss/moss-tts-realtime or openbmb/voxcpm2). Their auth, networking, realtime and audio layers have no UI dependency, so you can lift them into your own app. To run one, you need a Wiro project and API key from https://wiro.ai/panel/project/new.
More
- General Wiro skill (REST API, MCP server, auth, prices, tasks):
npx skills add wiroai/Wiro-Plugin --skill wiro-ai - Realtime voice conversation: https://wiro.ai/docs/realtime-voice-conversation
- Realtime text-to-speech: https://wiro.ai/docs/realtime-text-to-speech
- Realtime speech-to-text: https://wiro.ai/docs/realtime-speech-to-text
- WebSocket events: https://wiro.ai/docs/websocket
- Run a model: https://wiro.ai/docs/run-a-model
- Cancel and kill tasks: https://wiro.ai/docs/tasks
- Models: https://wiro.ai/models
- Pricing: https://wiro.ai/product/pricing
- Downloads
- 120All time
- Security audit
- Review
- Last updated
- 1w ago
- Current version
- v1.0.1
- License
- MIT-0
