Work / Real-time voice avatar engine / 2026
openvideo
A browser engine where you talk to a 3D avatar that answers with lip sync, emotion cues and lets you interrupt it mid-sentence.
The boundary
Stays inside
- Provider keys for STT, LLM and TTS
- Persona and system prompt
- Turn metrics
Allowed to cross
- 16 kHz PCM from the mic
- Final transcripts
- Audio segments with word timings
The browser holds no credentials. It sends sound and gets sound back.
Context
Voice agents feel slow because each stage waits for the last, and they talk over you because nothing listens while they speak.
Idea
Stream every stage. LLM text is cut at soft punctuation into segments of up to 220 characters, each sent to TTS as it arrives. Two spoken words from the user abort the reply and rewrite it as interrupted.
Craft
- A shared protocol package defines JSON headers paired with binary PCM frames, paced in real time.
- Cue tags like [e:emotion g:gesture] are parsed out of the model text and timed to the audio.
- Metrics log the gaps that matter: final transcript to first token, first segment and first audio.
- Falls back to a pulsing orb when no avatar model is present.
Threat model
| Asset | Threat | Control |
|---|---|---|
| Provider API keys | Exposure in client code | Server side only, behind the WebSocket |
| System prompt | Override from the client | Persona is server owned |
| Spend | Runaway generation | Barge-in aborts the stream; context compaction |
Outcome
A working monorepo with server, web client, protocol and test harness, including Deepgram TTS integration under tests.