avatar

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 6, 2026 License: MIT Imports: 8 Imported by: 0

Documentation

Overview

Package avatar provides lip-sync avatar support for omni-livekit voice agents.

This package enables voice agents to display real-time lip-synced video avatars that move their mouths in sync with generated speech. It supports multiple avatar providers (Tavus, Anam, Simli, etc.) through a pluggable architecture.

Architecture

The avatar system consists of three main components:

  1. Session - Manages the avatar lifecycle (start, wait for join, close)
  2. AudioDestination - Streams TTS audio to the avatar provider
  3. RPC Handlers - Handle playback control (start, finish, clear buffer)

The avatar provider joins the LiveKit room as a separate participant and publishes video+audio tracks on behalf of the agent using the "lk.publish_on_behalf" participant attribute.

Basic Usage 

// Create avatar session with a provider
session, err := tavus.New(tavus.Config{
    APIKey: os.Getenv("TAVUS_API_KEY"),
    FaceID: "your-face-id",
})

// Start the avatar
err = session.Start(ctx, avatar.StartOptions{
    Room:          agent.Room(),
    AgentIdentity: "meeting-pm",
    LiveKitURL:    os.Getenv("LIVEKIT_URL"),
    // ...
})

// Wait for avatar to join
err = session.WaitForJoin(ctx, 10*time.Second)

// Get audio destination for TTS output
audioOut := session.AudioOutput()

// Stream TTS audio to avatar
audioOut.CaptureFrame(ctx, pcmFrame)
audioOut.Flush(ctx)

// Handle interruption
audioOut.ClearBuffer(ctx)

Providers

Avatar providers are implemented in sub-packages:

  • avatar/tavus - Tavus avatar provider
  • avatar/anam - Anam avatar provider
  • avatar/simli - Simli avatar provider

Each provider implements the Session interface and handles provider-specific API calls to create avatar sessions.

Audio Streaming

Audio is streamed from the agent to the avatar using LiveKit's ByteStream API. The DataStreamAudioOutput implementation handles:

  • Streaming PCM audio frames to the avatar
  • Managing stream lifecycle (create, write, close)
  • RPC-based playback control

Playback Control

The avatar communicates playback state via RPC:

  • lk.playback_started - Avatar started speaking
  • lk.playback_finished - Avatar finished speaking (with position and interrupted flag)
  • lk.clear_buffer - Agent requests playback interruption

Index

Constants

View Source 
const (
	// RPCPlaybackStarted is sent by the avatar when it starts speaking.
	RPCPlaybackStarted = "lk.playback_started"

	// RPCPlaybackFinished is sent by the avatar when it finishes speaking.
	RPCPlaybackFinished = "lk.playback_finished"

	// RPCClearBuffer is sent by the agent to interrupt playback.
	RPCClearBuffer = "lk.clear_buffer"
)

RPC method names for playback control. These must match the methods implemented by avatar workers.

View Source 
const AudioStreamTopic = "lk.audio_stream"

AudioStreamTopic is the ByteStream topic for audio data.

Variables

View Source 
var (
	// ErrSessionNotStarted indicates that Start() was not called before
	// attempting an operation that requires an active session.
	ErrSessionNotStarted = errors.New("avatar: session not started")

	// ErrSessionAlreadyStarted indicates that Start() was called on a
	// session that is already running.
	ErrSessionAlreadyStarted = errors.New("avatar: session already started")

	// ErrAvatarJoinTimeout indicates that the avatar participant did not
	// join the room within the specified timeout.
	ErrAvatarJoinTimeout = errors.New("avatar: join timeout")

	// ErrAvatarTrackTimeout indicates that the avatar did not publish
	// the expected track (video/audio) within the timeout.
	ErrAvatarTrackTimeout = errors.New("avatar: track publish timeout")

	// ErrProviderUnavailable indicates that the avatar provider API
	// is unreachable or returned an error.
	ErrProviderUnavailable = errors.New("avatar: provider unavailable")

	// ErrProviderAuthFailed indicates that authentication with the
	// avatar provider failed (invalid API key, etc.).
	ErrProviderAuthFailed = errors.New("avatar: provider authentication failed")

	// ErrProviderRateLimited indicates that the avatar provider has
	// rate-limited the request.
	ErrProviderRateLimited = errors.New("avatar: provider rate limited")

	// ErrInvalidConfig indicates that the avatar configuration is
	// invalid or incomplete.
	ErrInvalidConfig = errors.New("avatar: invalid configuration")

	// ErrRPCTimeout indicates that an RPC call to the avatar timed out.
	ErrRPCTimeout = errors.New("avatar: RPC timeout")

	// ErrRPCFailed indicates that an RPC call to the avatar failed.
	ErrRPCFailed = errors.New("avatar: RPC failed")

	// ErrStreamClosed indicates that the audio stream was closed
	// unexpectedly.
	ErrStreamClosed = errors.New("avatar: stream closed")

	// ErrAvatarDisconnected indicates that the avatar participant
	// disconnected from the room unexpectedly.
	ErrAvatarDisconnected = errors.New("avatar: disconnected")
)

Sentinel errors for avatar operations.

Functions

func GenerateAvatarToken 

func GenerateAvatarToken(opts TokenOptions) (string, error)

GenerateAvatarToken creates a JWT token for an avatar to join a room.

The token includes the special "lk.publish_on_behalf" attribute that allows the avatar participant to publish tracks that appear in the UI as if they came from the agent participant.

Types

type AudioConfig 

type AudioConfig struct {
	// SampleRate is the audio sample rate in Hz.
	// Default: 24000 (common for avatar providers)
	SampleRate int

	// Channels is the number of audio channels.
	// Default: 1 (mono)
	Channels int

	// Encoding is the audio encoding format.
	// Default: "linear16" (PCM16 little-endian)
	Encoding string
}

AudioConfig holds audio configuration for avatar sessions.

func DefaultAudioConfig 

func DefaultAudioConfig() AudioConfig

DefaultAudioConfig returns the default audio configuration.

func (AudioConfig) FrameDuration 

func (c AudioConfig) FrameDuration(frameSize int) int

FrameDuration returns the duration in milliseconds for the given frame size.

func (AudioConfig) FrameSize 

func (c AudioConfig) FrameSize(durationMs int) int

FrameSize returns the number of bytes per frame for the given duration in ms.

type AudioDestination 

type AudioDestination interface {
	// CaptureFrame sends a PCM16 audio frame to the avatar.
	//
	// The frame should be little-endian PCM16 at the configured sample rate.
	// Frames are typically 20ms of audio (e.g., 960 samples at 48kHz).
	//
	// This method may block if the underlying transport is congested.
	CaptureFrame(ctx context.Context, frame []byte) error

	// Flush marks the end of an audio segment (utterance).
	//
	// The avatar should speak all buffered audio before accepting more.
	// This closes the current ByteStream and prepares for the next utterance.
	Flush(ctx context.Context) error

	// ClearBuffer interrupts current playback.
	//
	// Use this when the user interrupts the agent. The avatar should
	// stop speaking immediately and discard any buffered audio.
	//
	// This sends an RPC to the avatar worker and waits for acknowledgment.
	ClearBuffer(ctx context.Context) error

	// SampleRate returns the expected input sample rate in Hz.
	// Common values: 16000, 24000, 48000
	SampleRate() int

	// Channels returns the expected number of audio channels.
	// Typically 1 (mono).
	Channels() int

	// OnPlayback registers a callback for playback events.
	// Multiple callbacks can be registered.
	OnPlayback(callback PlaybackCallback)

	// Close releases resources associated with this audio destination.
	// After Close(), no more frames should be sent.
	Close() error
}

AudioDestination receives audio frames and forwards them to an avatar.

Implementations include:

  • DataStreamAudioOutput: Streams audio to a remote avatar via LiveKit ByteStream
  • QueueAudioOutput: In-memory queue for testing

The typical usage flow is:

  1. Call CaptureFrame() for each PCM audio frame from TTS
  2. Call Flush() when the utterance is complete
  3. Call ClearBuffer() to interrupt playback (e.g., user interruption)

type AvatarMetadata 

type AvatarMetadata struct {
	// Kind identifies this as an avatar participant.
	Kind string `json:"kind"`

	// Provider is the avatar provider name.
	Provider string `json:"provider,omitempty"`

	// AgentIdentity is the identity of the agent this avatar represents.
	AgentIdentity string `json:"agent_identity,omitempty"`
}

AvatarMetadata is the standard metadata structure for avatar participants.

func DefaultAvatarMetadata 

func DefaultAvatarMetadata(provider, agentIdentity string) AvatarMetadata

DefaultAvatarMetadata returns the default metadata for avatar participants.

type BaseSession 

type BaseSession struct {
	// contains filtered or unexported fields
}

BaseSession provides common functionality for avatar session implementations. Provider-specific sessions can embed this struct.

func NewBaseSession 

func NewBaseSession(provider, avatarIdentity string) *BaseSession

NewBaseSession creates a new BaseSession.

func (*BaseSession) AudioOutput 

func (s *BaseSession) AudioOutput() AudioDestination

AudioOutput returns the audio destination.

func (*BaseSession) AvatarIdentity 

func (s *BaseSession) AvatarIdentity() string

AvatarIdentity returns the avatar participant identity.

func (*BaseSession) Callbacks 

func (s *BaseSession) Callbacks() *SessionCallbacks

Callbacks returns the session callbacks.

func (*BaseSession) EmitError 

func (s *BaseSession) EmitError(err error)

EmitError emits an error event if a callback is registered.

func (*BaseSession) EmitMetrics 

func (s *BaseSession) EmitMetrics(metrics Metrics)

EmitMetrics emits metrics if a callback is registered.

func (*BaseSession) EmitPlaybackFinished 

func (s *BaseSession) EmitPlaybackFinished(position float64, interrupted bool)

EmitPlaybackFinished emits a playback finished event if a callback is registered.

func (*BaseSession) EmitPlaybackStarted 

func (s *BaseSession) EmitPlaybackStarted()

EmitPlaybackStarted emits a playback started event if a callback is registered.

func (*BaseSession) IsStarted 

func (s *BaseSession) IsStarted() bool

IsStarted returns true if the session has been started.

func (*BaseSession) MarkStarted 

func (s *BaseSession) MarkStarted()

MarkStarted marks the session as started.

func (*BaseSession) Provider 

func (s *BaseSession) Provider() string

Provider returns the provider name.

func (*BaseSession) Room 

func (s *BaseSession) Room() *lksdk.Room

Room returns the room reference.

func (*BaseSession) SetAudioOutput 

func (s *BaseSession) SetAudioOutput(out AudioDestination)

SetAudioOutput sets the audio destination.

func (*BaseSession) SetCallbacks 

func (s *BaseSession) SetCallbacks(callbacks *SessionCallbacks)

SetCallbacks sets the session callbacks.

func (*BaseSession) SetRoom 

func (s *BaseSession) SetRoom(room *lksdk.Room)

SetRoom sets the room reference.

func (*BaseSession) StartTime 

func (s *BaseSession) StartTime() time.Time

StartTime returns when the session was started.

type DataStreamAudioOutput 

type DataStreamAudioOutput struct {
	// contains filtered or unexported fields
}

DataStreamAudioOutput streams audio to a remote avatar via LiveKit ByteStream.

It handles:

  • Creating ByteStream writers for each utterance
  • Registering RPC handlers for playback events
  • Sending clear buffer requests

func NewDataStreamAudioOutput 

func NewDataStreamAudioOutput(cfg DataStreamConfig) (*DataStreamAudioOutput, error)

NewDataStreamAudioOutput creates a new DataStreamAudioOutput.

func (*DataStreamAudioOutput) CaptureFrame 

func (o *DataStreamAudioOutput) CaptureFrame(ctx context.Context, frame []byte) error

CaptureFrame sends a PCM16 audio frame to the avatar.

func (*DataStreamAudioOutput) Channels 

func (o *DataStreamAudioOutput) Channels() int

Channels returns the expected number of audio channels.

func (*DataStreamAudioOutput) ClearBuffer 

func (o *DataStreamAudioOutput) ClearBuffer(ctx context.Context) error

ClearBuffer interrupts current playback.

func (*DataStreamAudioOutput) Close 

func (o *DataStreamAudioOutput) Close() error

Close releases resources.

func (*DataStreamAudioOutput) Flush 

func (o *DataStreamAudioOutput) Flush(ctx context.Context) error

Flush marks the end of an audio segment.

func (*DataStreamAudioOutput) OnPlayback 

func (o *DataStreamAudioOutput) OnPlayback(callback PlaybackCallback)

OnPlayback registers a callback for playback events.

func (*DataStreamAudioOutput) SampleRate 

func (o *DataStreamAudioOutput) SampleRate() int

SampleRate returns the expected input sample rate.

type DataStreamConfig 

type DataStreamConfig struct {
	// Room is the LiveKit room.
	// Required.
	Room *lksdk.Room

	// DestinationIdentity is the avatar participant identity.
	// Required.
	DestinationIdentity string

	// Audio is the audio configuration.
	// Optional, defaults to DefaultAudioConfig().
	Audio AudioConfig

	// RPCTimeout is the timeout for RPC calls.
	// Default: 5 seconds
	RPCTimeout time.Duration
}

DataStreamConfig configures the DataStreamAudioOutput.

type Metrics 

type Metrics struct {
	// AvatarJoinLatency is the time from Start() to avatar joining the room.
	AvatarJoinLatency time.Duration

	// PlaybackLatency is the time from audio send to avatar speech start.
	// This is measured per utterance.
	PlaybackLatency time.Duration

	// Provider is the avatar provider name.
	Provider string

	// Timestamp is when the metrics were collected.
	Timestamp time.Time
}

Metrics contains avatar performance metrics.

type PlaybackCallback 

type PlaybackCallback func(event PlaybackEvent)

PlaybackCallback is called when playback events occur.

type PlaybackEvent 

type PlaybackEvent struct {
	// Type is the type of playback event.
	Type PlaybackEventType

	// Position is the playback position in seconds when the event occurred.
	// Only meaningful for PlaybackFinished events.
	Position float64

	// Interrupted indicates if playback was interrupted by ClearBuffer().
	// Only meaningful for PlaybackFinished events.
	Interrupted bool
}

PlaybackEvent represents a playback state change from the avatar.

type PlaybackEventType 

type PlaybackEventType string

PlaybackEventType identifies the type of playback event. 

const (
	// PlaybackStarted indicates the avatar started speaking.
	PlaybackStarted PlaybackEventType = "started"

	// PlaybackFinished indicates the avatar finished speaking.
	// Check Position and Interrupted for details.
	PlaybackFinished PlaybackEventType = "finished"
)

type ProviderError 

type ProviderError struct {
	Provider string // Provider name (e.g., "tavus", "anam")
	Op       string // Operation that failed
	Err      error  // Underlying error
}

ProviderError wraps an error from an avatar provider with additional context.

func NewProviderError 

func NewProviderError(provider, op string, err error) *ProviderError

NewProviderError creates a new ProviderError.

func (*ProviderError) Error 

func (e *ProviderError) Error() string

Error implements the error interface.

func (*ProviderError) Unwrap 

func (e *ProviderError) Unwrap() error

Unwrap returns the underlying error.

type QueueAudioOutput 

type QueueAudioOutput struct {
	// contains filtered or unexported fields
}

QueueAudioOutput is an in-memory AudioDestination for testing.

It captures all audio frames sent to it, allowing tests to verify that the correct audio data is being produced.

func NewQueueAudioOutput 

func NewQueueAudioOutput(config AudioConfig) *QueueAudioOutput

NewQueueAudioOutput creates a new QueueAudioOutput with the given config.

func (*QueueAudioOutput) CaptureFrame 

func (q *QueueAudioOutput) CaptureFrame(ctx context.Context, frame []byte) error

CaptureFrame stores a PCM16 audio frame in the queue.

func (*QueueAudioOutput) Channels 

func (q *QueueAudioOutput) Channels() int

Channels returns the configured number of channels.

func (*QueueAudioOutput) ClearBuffer 

func (q *QueueAudioOutput) ClearBuffer(ctx context.Context) error

ClearBuffer simulates interrupting playback.

func (*QueueAudioOutput) Close 

func (q *QueueAudioOutput) Close() error

Close marks the queue as closed.

func (*QueueAudioOutput) Flush 

func (q *QueueAudioOutput) Flush(ctx context.Context) error

Flush marks the end of an audio segment.

func (*QueueAudioOutput) FrameCount 

func (q *QueueAudioOutput) FrameCount() int

FrameCount returns the number of frames captured.

func (*QueueAudioOutput) Frames 

func (q *QueueAudioOutput) Frames() [][]byte

Frames returns all captured frames.

func (*QueueAudioOutput) OnPlayback 

func (q *QueueAudioOutput) OnPlayback(callback PlaybackCallback)

OnPlayback registers a callback for playback events.

func (*QueueAudioOutput) Reset 

func (q *QueueAudioOutput) Reset()

Reset clears all captured data and resets state.

func (*QueueAudioOutput) SampleRate 

func (q *QueueAudioOutput) SampleRate() int

SampleRate returns the configured sample rate.

func (*QueueAudioOutput) SimulatePlaybackFinished 

func (q *QueueAudioOutput) SimulatePlaybackFinished(position float64, interrupted bool)

SimulatePlaybackFinished simulates a playback finished event.

func (*QueueAudioOutput) SimulatePlaybackStarted 

func (q *QueueAudioOutput) SimulatePlaybackStarted()

SimulatePlaybackStarted simulates a playback started event.

func (*QueueAudioOutput) TotalBytes 

func (q *QueueAudioOutput) TotalBytes() int

TotalBytes returns the total number of bytes captured.

func (*QueueAudioOutput) WasCleared 

func (q *QueueAudioOutput) WasCleared() bool

WasCleared returns true if ClearBuffer() was called.

func (*QueueAudioOutput) WasFlushed 

func (q *QueueAudioOutput) WasFlushed() bool

WasFlushed returns true if Flush() was called.

type Session 

type Session interface {
	// AvatarIdentity returns the participant identity of the avatar worker.
	// This is the identity that appears in the room and publishes video.
	AvatarIdentity() string

	// Provider returns the provider name (e.g., "tavus", "anam", "simli").
	Provider() string

	// Start initializes the avatar session.
	//
	// This method should:
	//  1. Create a session with the avatar provider's API
	//  2. Generate a LiveKit token for the avatar to join the room
	//  3. Configure the audio output to stream to the avatar
	//
	// The avatar will join the room asynchronously. Use WaitForJoin()
	// to wait for the avatar to be ready.
	Start(ctx context.Context, opts StartOptions) error

	// WaitForJoin blocks until the avatar participant joins the room
	// and publishes the expected tracks (typically video).
	//
	// Returns an error if the timeout is exceeded or the context is cancelled.
	WaitForJoin(ctx context.Context, timeout time.Duration) error

	// AudioOutput returns the audio destination for streaming TTS audio
	// to the avatar. Returns nil if the session is not started.
	AudioOutput() AudioDestination

	// Close disconnects the avatar and cleans up resources.
	//
	// This method should:
	//  1. End the session with the avatar provider
	//  2. Remove the avatar participant from the room
	//  3. Clean up any registered RPC handlers
	Close(ctx context.Context) error
}

Session manages a lip-sync avatar that publishes video to the room.

Implementations of this interface handle provider-specific API calls to create avatar sessions and manage their lifecycle.

The typical flow is:

  1. Create a Session with provider-specific configuration
  2. Call Start() to initialize the avatar and connect it to the room
  3. Call WaitForJoin() to wait for the avatar to be ready
  4. Use AudioOutput() to stream TTS audio to the avatar
  5. Call Close() when done to clean up resources

type SessionCallbacks 

type SessionCallbacks struct {
	// OnMetricsCollected is called when avatar metrics are available.
	OnMetricsCollected func(metrics Metrics)

	// OnPlaybackStarted is called when the avatar starts speaking.
	OnPlaybackStarted func()

	// OnPlaybackFinished is called when the avatar finishes speaking.
	// The position is the playback position in seconds when stopped.
	// The interrupted flag indicates if playback was interrupted by
	// a ClearBuffer() call.
	OnPlaybackFinished func(position float64, interrupted bool)

	// OnAvatarJoined is called when the avatar participant joins the room.
	OnAvatarJoined func(identity string)

	// OnAvatarLeft is called when the avatar participant leaves the room.
	OnAvatarLeft func(identity string)

	// OnError is called when an error occurs.
	// This is for non-fatal errors that don't cause the session to fail.
	OnError func(err error)
}

SessionCallbacks defines optional event callbacks for avatar sessions.

type StartOptions 

type StartOptions struct {
	// Room is the LiveKit room the agent has joined.
	// Required.
	Room *lksdk.Room

	// AgentIdentity is the identity of the agent participant.
	// The avatar will publish tracks on behalf of this identity
	// using the lk.publish_on_behalf attribute.
	// Required.
	AgentIdentity string

	// LiveKitURL is the LiveKit server URL for the avatar to connect to.
	// This should match the URL the agent connected to.
	// Required.
	LiveKitURL string

	// LiveKitAPIKey is used to generate tokens for the avatar.
	// Required.
	LiveKitAPIKey string

	// LiveKitAPISecret is used to generate tokens for the avatar.
	// Required.
	LiveKitAPISecret string

	// Callbacks configures optional event callbacks.
	Callbacks *SessionCallbacks
}

StartOptions configures avatar session startup.

func (*StartOptions) Validate 

func (o *StartOptions) Validate() error

Validate checks that all required fields are set.

type TokenOptions 

type TokenOptions struct {
	// APIKey is the LiveKit API key.
	// Required.
	APIKey string

	// APISecret is the LiveKit API secret.
	// Required.
	APISecret string

	// RoomName is the room the avatar will join.
	// Required.
	RoomName string

	// AvatarIdentity is the participant identity for the avatar.
	// Required.
	AvatarIdentity string

	// AvatarName is the display name for the avatar participant.
	// Optional, defaults to AvatarIdentity.
	AvatarName string

	// PublishOnBehalf is the identity of the agent participant.
	// The avatar will publish tracks that appear as if they're from this participant.
	// Required.
	PublishOnBehalf string

	// TTL is the token validity duration.
	// Default: 5 minutes
	TTL time.Duration

	// Metadata is optional participant metadata.
	Metadata string
}

TokenOptions configures avatar token generation.

Source Files

View all Source files

Directories

Path Synopsis
Package tavus provides Tavus avatar integration for omni-livekit voice agents.
 Package tavus provides Tavus avatar integration for omni-livekit voice agents.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.