Interactive Avatar: Tools Quickstart

A Synthesia Interactive Avatar that fills in a booking form on the page as you speak to it, and reacts out loud when you type into the form yourself. Built on the minimal quickstart: a Python LiveKit agent rendered as a photoreal, lip-synced avatar in your browser.

The pattern is LLM tool calling plus shared UI state, synced both ways over LiveKit data channels:

• agent → browser: the LLM calls a @function_tool (update_field), which publishes the change on a data topic, and the browser renders it.
• browser → agent: your manual edits (and the Submit click) publish back on the same topic; the agent updates its state and the avatar acknowledges out loud.

The form is a stand-in for any state your app shares with the avatar: a booking, an onboarding flow, a support ticket, a shopping cart.

Prerequisites

• Python 3.10–3.14
• LiveKit Cloud project (free tier works): LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET from cloud.livekit.io
• Synthesia API key (Interactive Avatars)
• OpenAI API key from platform.openai.com

Run it

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

cp .env.example .env   # then fill in the three keys

python agent.py dev    # terminal 1: the agent worker
python server.py       # terminal 2: frontend at http://localhost:8080

Open http://localhost:8080, click Start, and allow the microphone. Once the avatar joins, the form unlocks. Try both directions:

1. Speak: "I'd like to book something. I'm Ada Lovelace, next Tuesday, about analytical engines." Watch the fields flash blue as the avatar fills them.
2. Type: put your email straight into the form; the avatar notices and confirms it out loud.
3. Confirm verbally (or click Submit) and the agent validates, freezes the form, and logs the record.

Don't use python agent.py console. It runs a mock room and the avatar will silently never appear. Always test through a real room, like this frontend.

How it works

One data topic (form), three message shapes, and server-side state as the single source of truth:

you speak ─► STT ─► LLM calls update_field(field, value)
                        └─► agent state updated ─► publish_data ─► the form fills in

you type  ─► browser publishData ─► data_received handler
                        └─► agent state updated ─► generate_reply ─► the avatar reacts aloud

Everything tools-specific in agent.py is two functions. The tool (agent → browser):

@function_tool
async def update_field(self, field: str, value: str) -> dict:
    self.form[field] = value.strip()
    await self._publish({"type": "field", "field": field, "value": self.form[field]})
    return {"status": "ok"}

And the data handler (browser → agent):

def on_data(packet):
    msg = json.loads(packet.data)
    agent.form[msg["field"]] = msg["value"]
    session.generate_reply(instructions=f"The visitor typed {msg['field']} themselves — acknowledge it.")

The authoritative form lives in FormAgent.form, not in the LLM's conversation memory or the DOM, so neither side can drift. The model reads it back with get_form, and submit_form validates against it, so a value you typed counts exactly like one the avatar heard.

Two choices worth copying:

• Preemptive generation is off (turn_handling in agent.py). The framework can speculatively start a reply on an eager transcript; with side-effecting tools, that speculation could call update_field with a half-heard value and then be discarded.
• Browser input is untrusted. The handler whitelists field names and caps value length before touching state. Keep that shape when you extend the protocol.

Make it yours

• The form: edit FIELDS in agent.py and the matching <input> ids in index.html (the two lists must stay in sync), and mention the new fields in build_instructions().
• What Submit does: submit_form currently logs the record; replace the logger.info line with a POST to your booking API or CRM.
• More tools: any @function_tool on FormAgent becomes something the avatar can do, like fetching availability before offering dates, looking up the visitor by email, or navigating the page.
• Voice CARTESIA_VOICE_ID (Cartesia voices), via .env.
• Avatar: SYNTHESIA_AVATAR_ID, via .env, must be the Interactive ID of a synthetic or Personal avatar your workspace can use. In Synthesia Studio, open the avatar's ••• menu and choose Copy Interactive ID. Stock actor-based avatars such as Ryan or Ada can't be used as interactive avatars on any plan.
• Models: swap the stt= / llm= / tts= lines for any LiveKit-supported provider; tool calling is provider-independent.

server.py mints room tokens with sync_streams=True (avatar audio/video sync) and a RoomAgentDispatch matching the worker's agent_name (see Security & production). Keep both when you build your own token endpoint, and keep SYNTHESIA_API_KEY server-side: it's a workspace-bound secret that must never reach frontend code.

Security & production

Demo only. This is a local quickstart. Don't deploy it as-is.

• The /token endpoint is unauthenticated. The demo server binds to localhost and mints 15-minute, room-scoped tokens, so exposure is limited to your machine. But anyone who can reach the endpoint can dispatch an agent worker, which costs money, so a production endpoint must sit behind your app's authentication.
• Agent dispatch is explicit. The worker sets agent_name and only joins rooms whose token requests it via RoomAgentDispatch (server.py), so keep the two names in sync. Don't remove agent_name: an unnamed worker auto-joins every new room in the LiveKit project.
• Data-channel messages come from the browser. Treat them as untrusted user input on the agent side. The handler already whitelists fields and caps lengths; keep doing that as you extend it.
• Form contents are PII in a real deployment. The demo logs the submitted record; route it somewhere appropriate before collecting real data.

Troubleshooting

Source for this recipe: interactive-avatar-quickstarts/tools.