ESP32 Voice Assistant for Openclaw

Hands-free voice assistant for OpenClaw on an ESP32-S3-BOX-3 — on-device wake word, switchable xAI Grok / ElevenLabs STT+TTS, no Home Assistant required.

Darren Robinson@darrenjrobinson

Install

openclaw skills install @darrenjrobinson/esp32-voice-assistant

ESP32 Voice Assistant for OpenClaw

Give OpenClaw a hardware voice: an ESP32-S3-BOX-3 running stock ESPHome firmware talks directly to a Docker bridge — on-device wake word, VAD, STT, OpenClaw as the agent brain, then streamed sentence-chunked TTS back to the speaker. No Home Assistant in the voice path, and zero firmware changes: the bridge impersonates the Home Assistant voice client over the ESPHome native API (TCP 6053), so you never reflash or edit device YAML.

text

wake word (on-device) → ESP32-S3-BOX-3 → bridge (Docker): VAD → STT (xAI|ElevenLabs)
  → spoken ack → OpenClaw /v1/chat/completions → TTS (xAI|ElevenLabs, streamed)
  → device fetches reply audio (HTTP :10400, FLAC)

The author's device answers to "Yo Marvin" — a custom-trained wake word. Yours can use any standard ESPHome wake word (e.g. "Hey Jarvis") or a custom one you train yourself; the process is covered in Hardware Voice Assistant for OpenClaw.

STT and TTS providers are independently switchable between xAI Grok Voice and ElevenLabs — mix and match (e.g. cheap xAI STT + an ElevenLabs custom voice for TTS).

Hardware & prerequisites

ESP32-S3-BOX-3 running an ESPHome voice-assistant firmware with on-device wake word (the stock wake-word-voice-assistants config or a derivative — standard or custom-trained wake word). Other ESPHome voice devices should work; the bridge adapts to the advertised media format.
Docker host on the same LAN — the device must be able to reach the bridge's HTTP port (default 10400).
OpenClaw with the chat interface enabled — the bridge talks to OpenClaw's OpenAI-compatible /v1/chat/completions endpoint, which is only served when the chat interface is turned on in your OpenClaw configuration. It must be reachable from the bridge (default compose config expects it on the Docker host, port 18789).
xAI API key and/or ElevenLabs API key — at least one, matching your STT_PROVIDER/TTS_PROVIDER selection. ELEVENLABS_VOICE_ID is also required when TTS is ElevenLabs.

Setup

Clone the repo and create the config:

bash

git clone https://github.com/darrenjrobinson/voice-esp32-openclaw.git
cd voice-esp32-openclaw
cp .env.example .env

In OpenClaw, make sure the chat interface is enabled so /v1/chat/completions is being served, and note its URL and API key.
Edit .env — at minimum: STT_PROVIDER/TTS_PROVIDER, the matching API key(s), OPENCLAW_URL, OPENCLAW_API_KEY, ESP32_HOST (and ESP32_NOISE_PSK if the device API is encrypted), and BRIDGE_ADVERTISE_HOST (the Docker host's LAN IP — required inside Docker). A commented reference copy ships with this skill as example.env.
Disconnect Home Assistant's voice subscription for the device (disable the ESPHome integration entry or its assist satellite entity). The protocol allows only one voice-assistant subscriber — with two, the wake word silently does nothing.

Start the bridge and verify:

bash

docker compose --profile bridge up -d --build
docker compose logs -f voice-bridge
# expect: "Connected to <device> (…)" and "device announcement format: flac @ 48000 Hz"

Say the wake word. Done.

Configuration reference (all in `.env`)

Variable	Default	Purpose
`STT_PROVIDER` / `TTS_PROVIDER`	`xai`	`xai` or `elevenlabs`, independently — any mix works
`XAI_VOICE`	`eve`	Any of the 26 built-in xAI voices or a custom voice ID
`ELEVENLABS_VOICE_ID`	—	Required for ElevenLabs TTS; list yours with `python scripts/elevenlabs_voices.py`
`ELEVENLABS_TTS_MODEL`	`eleven_flash_v2_5`	Lowest-latency, half-price model
`OPENCLAW_SESSION_KEY`	`agent:main:voice`	Session identity sent to OpenClaw (`OPENCLAW_SESSION_MODE=user` sends it as the OpenAI `user` field, `header` as `X-Session-Key`)
`OPENCLAW_TIMEOUT_SECONDS`	`240`	Max wait for the agent's reply — tool-heavy turns can run minutes
`BRIDGE_ACK_PHRASE`	`On it.`	Spoken while the agent thinks; synthesized once and cached; empty disables
`BRIDGE_VOLUME`	`1.0`	Device volume set before each reply (`0` = leave alone)
`VAD_THRESHOLD`	`500`	Speech gate — the BOX-3 mic runs quiet, values near `60` work well; per-turn level stats are logged for tuning
`VAD_SILENCE_SECONDS` / `VAD_MAX_SECONDS`	`0.8` / `10`	End-of-utterance silence and capture cap
`TTS_STREAMING`	`true`	Stream synthesis for fastest time-to-first-audio

Per-turn cost metrics (USD + ElevenLabs credits) are appended to out/metrics.jsonl. A typical turn costs ~$0.005 (all-xAI) to ~$0.015 (ElevenLabs TTS).

Troubleshooting

.env changes don't apply: containers read .env only at creation. From the repo directory run docker compose --profile bridge up -d (add --force-recreate if it reports nothing changed). restart is not enough.
401 or connection refused from OpenClaw: confirm the chat interface is enabled in OpenClaw (it serves /v1/chat/completions), OPENCLAW_URL is reachable from inside the container, and OPENCLAW_API_KEY matches.
Wake word does nothing: something else (usually Home Assistant) still holds the voice-assistant subscription — only one subscriber is allowed.
No reply audio: check BRIDGE_ADVERTISE_HOST is the Docker host's LAN IP and port 10400 is reachable from the device.
Device quirks (chunked-transfer rejection, FLAC-only decoding, wedged announcements, volume handling) are already handled by the bridge — see the README's "Device quirks" table before suspecting the device.
Rollback: docker compose --profile bridge down, then re-enable the ESPHome integration in Home Assistant — the device reverts to the HA Assist pipeline immediately; nothing on the device was ever modified.