Interactive Avatar: Integration Overview

This page shows how to assemble the Interactive Avatar primitives into a real Bring-Your-Own (BYO) application: the end-to-end architecture, how data flows, who is responsible for what, and the tradeoffs you'll weigh in production. For the shared mental model behind this split, see Concepts.

Architecture

You bring a Python LiveKit voice agent and attach a single component, synthesia.AvatarSession, to it. That component does three things:

1. Authenticates with Synthesia using your workspace API key.
2. Dispatches a hosted avatar worker into your LiveKit room, where it joins as a normal participant with the identity synthesia-avatar-agent.
3. Re-routes your agent's audio output to the worker over a LiveKit data stream, so the avatar drives lip-sync from whatever speech your agent produces.

No GPU or video code runs in your agent process—rendering is fully hosted.

[your Python LiveKit Agent + synthesia.AvatarSession]
        │  1. authenticate (API key)
        │  2. dispatch worker
        ▼
[Synthesia API gateway] ──▶ [Avatar worker: GPU render + lip-sync]
        │                                   │
        │  3. worker joins your room as 'synthesia-avatar-agent'
        │  4. your agent's speech audio ──▶ worker (LiveKit data stream)
        │  5. lip-synced video + audio ──▶ published to the room
        ▼
[LiveKit room]  ◀──────────────▶  [end user: any web/mobile LiveKit client]

The avatar is a regular LiveKit participant, so any web or mobile LiveKit client renders it with no Synthesia-specific frontend code.

Data flow

Trace a single turn through the system:

1. The end user speaks in your LiveKit room (or your realtime model listens directly to their audio).
2. Your agent does what it always does—runs your speech model, or your STT → LLM → TTS pipeline—and produces speech audio.
3. The plugin intercepts that audio (it replaces session.output.audio) and forwards it to the avatar worker over a LiveKit data stream, instead of publishing it straight to the room.
4. The avatar worker renders lip-synced video and publishes both video and audio back into the room as the synthesia-avatar-agent participant.
5. The end user's client subscribes to that participant like any other and displays the talking avatar.

The important consequence: the audio the user hears comes from the avatar worker, not directly from your agent—which is why the avatar and the voice stay in sync.

Who owns what

The dividing line: you own the conversation and the room; Synthesia owns the rendered avatar. This integration is designed for teams that want to connect their own LLM, agent, or conversational logic and use Synthesia only for the avatar layer.

Audio routing (and the one rule that matters)

When you attach the avatar, the component transparently replaces session.output.audio, so the avatar lip-syncs to your agent's speech and you write no avatar integration code yourself. Two rules follow from this:

• Attach the avatar before session.start(). Attaching after will not work.
• Don't reassign session.output.audio after attaching the avatar, or lip-sync will break.

Because the re-routing operates on session.output.audio—which is model-agnostic—the avatar attaches identically whether you run a realtime model or a component pipeline.

Choosing an integration path

The avatar attaches the same way regardless of how your agent generates speech. The choice is about your speech stack, not the avatar:

Because TTS is BYO, Synthesia voices are not available through this API—pick any TTS provider compatible with your LiveKit pipeline. Whichever path you choose, match the model's speaking voice to your avatar's persona so audio and face don't clash.

See the Quickstart guides for complete, runnable code.

Production considerations

What the current sources support today:

• Keep secrets server-side. The Synthesia API key, LiveKit API key/secret, and model provider keys belong in your agent's environment or a secret manager—never in frontend code. The plugin mints the avatar's room token in-process, so your LiveKit secret never leaves your agent.
• Deploy with the dependency declared. When deploying to LiveKit Cloud, install the plugin with uv add <wheel-url> so it's recorded in pyproject.toml; an environment-only install won't be present in the Cloud build. (See the Quickstart guides.) Your agent worker process is the only component you deploy—rendering is always Synthesia-hosted, as described in Architecture above.
• Handle cold starts. avatar.start() waits up to join_timeout (default 30s) for the worker to join; raise it if the worker cold-starts slowly, and treat SynthesiaTimeoutError as retryable.
• Handle mid-session drops. Subscribe to the error event (a SynthesiaConnectionError) and to session_ended. A dropped avatar track is retryable—back off and retry per the taxonomy below—but whether retrying resumes the same session and room state, or always requires a fresh session, isn't yet documented; until confirmed, treat it as a new start() call.
• Distinguish retryable from terminal failures. RateLimitedError (honor retry_after), SynthesiaTimeoutError, and SynthesiaConnectionError are retryable; SynthesiaAuthError, UnknownAvatarError, and QuotaExceededError are not. Full taxonomy in the Errors reference.
• Video quality is a subscriber-side lever. The avatar's resolution is set by the hosted worker (the plugin has no quality knob); to keep it crisp, have your client subscribe at full resolution rather than downscaling to a small tile.
• Concurrency is capped. Interactive Avatar sessions count against your plan's concurrent-session limit: 1 for Freemium, 100 for all paid plans. A separate minute-based cap also applies; its specific threshold isn't published. Exceeding either raises QuotaExceededError (HTTP 402) from the plugin, or a 429 concurrency_limit_exceeded problem from POST /api/interactive-avatars/sessions if you call the REST API directly. No rate-limit headers are sent on a concurrency denial—nothing predicts when a slot frees up.