omni-livekit

README

Omni-LiveKit

LiveKit voice gateway for WebRTC-based voice AI applications.

Overview

LiveKit provides pure WebRTC voice without phone network involvement. While PSTN gateways (Twilio, Telnyx, Vonage, Plivo) also offer WebRTC SDKs, they primarily bridge browser/app connections to the phone network. LiveKit is designed for direct browser-to-browser and app-to-app communication.

┌───────────────┐        ┌─────────────────┐        ┌───────────────────┐
│  Browser/App  │◄──────►│  LiveKit Cloud  │◄──────►│   OmniVoice       │
│   (WebRTC)    │ WebRTC │    or Server    │ WebRTC │   Voice Gateway   │
└───────────────┘        └─────────────────┘        └───────────────────┘

Use Cases

Use Case	PSTN Gateways	LiveKit
Phone calls (PSTN)	Yes	No
Browser WebRTC	Yes (bridges to PSTN)	Yes (native)
Mobile apps	WebRTC SDK available	WebRTC SDK available
Browser-to-browser	Via PSTN bridge	Direct WebRTC
Latency	300-500ms (PSTN hop)	<200ms
Cost model	Per-minute telephony	Infrastructure only

Installation

go get github.com/plexusone/omni-livekit

Native Dependencies

This package requires native audio libraries for Opus encoding/decoding and resampling:

macOS:

brew install opus opusfile libsoxr

Ubuntu/Debian:

apt-get install libopus-dev libopusfile-dev libsoxr-dev

Fedora:

dnf install opus-devel opusfile-devel soxr-devel

Build Tags

Voice agents must be built with -tags opus to enable proper Opus codec support:

# Required for voice agents
go run -tags opus ./cmd/voice-agent
go build -tags opus ./cmd/voice-agent

# Without -tags opus, audio won't be decoded properly and STT will fail

Quick Start

Go Backend (AI Agent)

package main

import (
    "context"
    "log"
    "os"

    "github.com/plexusone/omni-livekit/omnivoice/gateway"
    coregateway "github.com/plexusone/omnivoice-core/gateway"
)

func main() {
    // Create gateway implementing coregateway.WebRTCGateway interface
    gw, err := gateway.New(gateway.Config{
        LiveKitURL:    os.Getenv("LIVEKIT_URL"),
        LiveKitAPIKey: os.Getenv("LIVEKIT_API_KEY"),
        LiveKitSecret: os.Getenv("LIVEKIT_API_SECRET"),
        RoomName:      "voice-agent",
        AgentIdentity: "ai-agent",
        SampleRate:    24000,
    })
    if err != nil {
        log.Fatal(err)
    }

    // Handle participant joins (WebRTC-specific handler)
    gw.OnParticipantJoined(func(p *coregateway.ParticipantInfo) error {
        log.Printf("Participant joined: %s (%s)", p.DisplayName, p.Identity)
        return nil // Accept participant
    })

    ctx := context.Background()
    if err := gw.Start(ctx); err != nil {
        log.Fatal(err)
    }
}

Generate Client Token

import "github.com/plexusone/omni-livekit/room"

client, _ := room.NewClient(room.Config{
    APIKey:    os.Getenv("LIVEKIT_API_KEY"),
    APISecret: os.Getenv("LIVEKIT_API_SECRET"),
    URL:       os.Getenv("LIVEKIT_URL"),
})

// Generate token for web/mobile client
token, _ := client.GenerateClientToken("voice-agent", "user-123", "John")

React Frontend

import { LiveKitRoom, useVoiceAssistant } from '@livekit/components-react';

function VoiceAgent() {
    const token = await fetchTokenFromBackend();

    return (
        <LiveKitRoom
            serverUrl={process.env.LIVEKIT_URL}
            token={token}
            connect={true}
            audio={true}
        >
            <VoiceUI />
        </LiveKitRoom>
    );
}

Interface

omni-livekit implements the coregateway.WebRTCGateway interface from omnivoice-core:

type WebRTCGateway interface {
    Name() ProviderName
    Start(ctx context.Context) error
    Stop() error
    OnParticipantJoined(handler ParticipantHandler)
    JoinRoom(ctx context.Context, roomName string) error
    LeaveRoom() error
    CurrentRoom() string
    GetSession(participantID string) (WebRTCSession, bool)
    ListSessions() []WebRTCSession
    GenerateClientToken(roomName, identity, displayName string) (string, error)
}

This is different from the Gateway interface used by PSTN providers (Twilio, Telnyx, etc.) which uses phone numbers and MakeCall().

Architecture

Audio Flow

Client (Browser/Mobile)
    │
    ▼ WebRTC Audio Track (Opus)
    │
┌───────────────────────────────────────┐
│           LiveKit Server              │
└───────────────────────────────────────┘
    │
    ▼ WebRTC Audio Track (Opus)
    │
┌───────────────────────────────────────┐
│       Go Backend (omni-livekit)       │
│                                       │
│  ┌─────────────────────────────────┐  │
│  │  PCMRemoteTrack                 │  │
│  │  (Opus → PCM16 decode)          │  │
│  └─────────────┬───────────────────┘  │
│                ▼                      │
│  ┌─────────────────────────────────┐  │
│  │  Voice Pipeline                 │  │
│  │  STT → LLM → TTS                │  │
│  │  (or Voice-to-Voice)            │  │
│  └─────────────┬───────────────────┘  │
│                ▼                      │
│  ┌─────────────────────────────────┐  │
│  │  PCMLocalTrack                  │  │
│  │  (PCM16 → Opus encode)          │  │
│  └─────────────────────────────────┘  │
└───────────────────────────────────────┘

Key Differences from PSTN Gateways

Aspect	PSTN Gateways	LiveKit
Primary use	Phone network access	Browser/app-to-app
Connection model	Phone number dialing	Room-based participation
Signaling	HTTP webhooks + WebRTC	WebRTC/WebSocket
Audio format	mulaw 8kHz (PSTN), Opus (WebRTC)	Opus (configurable rate)
Identity	Phone number or SIP URI	Participant ID
Session model	Call legs (inbound/outbound)	Room participants

Human Participation

When an AI agent joins a LiveKit room, humans can join via several methods:

Option 1: LiveKit Meet (No Code Required)

LiveKit provides a free hosted web UI. Generate a join URL with token:

// Generate token and URL
token, _ := client.GenerateClientToken("my-room", "user-123", "Alice")
joinURL := fmt.Sprintf("https://meet.livekit.io/custom?liveKitUrl=%s&token=%s",
    url.QueryEscape(serverURL), token)
// Share joinURL with the human participant

The human opens the link in their browser, grants mic/camera permissions, and joins.

Option 2: LiveKit Cloud Dashboard

If using LiveKit Cloud, create rooms and generate join links from the dashboard UI.

Option 3: Custom Frontend

Build your own UI with LiveKit's client SDKs:

Platform	SDK	Install
Web (JS/TS)	livekit-client	npm install livekit-client
React	@livekit/components-react	npm install @livekit/components-react
iOS	LiveKit Swift SDK	Swift Package Manager
Android	LiveKit Android SDK	Maven
Flutter	livekit_client	pub.dev

React Example:

import { LiveKitRoom, VideoConference } from '@livekit/components-react';
import '@livekit/components-styles';

function MeetingRoom({ token, serverUrl }) {
  return (
    <LiveKitRoom token={token} serverUrl={serverUrl} connect={true}>
      <VideoConference />
    </LiveKitRoom>
  );
}

Minimal HTML/JS:

<script src="https://unpkg.com/livekit-client/dist/livekit-client.umd.js"></script>
<script>
  const room = new LivekitClient.Room();
  await room.connect('wss://your-server.livekit.cloud', token);

  // Enable microphone
  await room.localParticipant.setMicrophoneEnabled(true);
</script>

Recommendation

Use Case	Recommended Approach
Testing/Demos	LiveKit Meet
Quick prototype	LiveKit React components
Production app	Custom frontend with full UX control

Environment Variables

# LiveKit Cloud or self-hosted server
export LIVEKIT_URL="wss://your-app.livekit.cloud"
export LIVEKIT_API_KEY="your-api-key"
export LIVEKIT_API_SECRET="your-api-secret"

Audio Configuration

The gateway supports configurable sample rates for optimal voice AI performance:

gateway.Config{
    SampleRate: 24000,  // 24kHz for high quality (OpenAI Realtime API)
    // SampleRate: 16000, // 16kHz for most STT/TTS
    Channels: 1,        // Mono audio
}

Session Events

session, ok := gw.GetSession(participantID)
if !ok {
    return
}

// WebRTCSession provides participant info
fmt.Printf("Room: %s\n", session.RoomName())
fmt.Printf("Participant: %s\n", session.Participant().DisplayName)

// Handle events
for event := range session.Events() {
    switch event.Type {
    case coregateway.EventUserTranscript:
        fmt.Printf("User said: %s\n", event.Data)
    case coregateway.EventAgentTranscript:
        fmt.Printf("Agent said: %s\n", event.Data)
    case coregateway.EventInterruption:
        fmt.Println("User interrupted agent")
    }
}

// Send audio to participant
samples := make([]int16, 480) // 20ms at 24kHz
session.SendAudio(samples)

Integration with OmniVoice

import (
    "github.com/plexusone/omni-livekit/omnivoice/gateway"
    "github.com/plexusone/omnivoice"
    _ "github.com/plexusone/omnivoice/providers/all"
)

// Use OmniVoice for STT/TTS
stt, _ := omnivoice.GetSTTProvider("deepgram", omnivoice.WithAPIKey(apiKey))
tts, _ := omnivoice.GetTTSProvider("elevenlabs", omnivoice.WithAPIKey(apiKey))

// LiveKit gateway receives audio → STT → LLM → TTS → sends audio

Voice Pipeline Modes

Standard Pipeline (STT → LLM → TTS):

Audio In → STT → Text → LLM → Text → TTS → Audio Out

This mode uses separate providers for speech-to-text, language model, and text-to-speech.

Voice-to-Voice (Lower Latency):

Audio In → Voice Model → Audio Out

OmniVoice supports direct voice-to-voice with:

• Deepgram - Nova-3 Voice Agent
• Google - Gemini Live
• OpenAI - Realtime API

Voice-to-voice eliminates the text intermediate step, reducing latency for real-time conversations.

Avatars

The avatar package provides a unified interface for visual representations of voice agents.

Supported Providers

Provider	Type	Status	Best For
none	Audio-only	✅ Built-in	Voice-only agents
static	Pre-encoded H.264	✅ Built-in	Simple visual presence
Tavus	Realistic humans	✅ Implemented	Video-trained digital twins
HeyGen	Realistic humans	🔜 Planned	Highest quality realism
D-ID	Photo-to-avatar	🔜 Planned	Quick setup from photo
bitHuman	Animated + realistic	🔜 Planned	Animated characters, edge/CPU
Simli	Realistic + animated	🔜 Planned	Budget-friendly

See Avatar Provider Comparison for detailed evaluation.

Quick Start with Factory

import (
    "github.com/plexusone/omni-livekit/avatar"
    _ "github.com/plexusone/omni-livekit/avatar/tavus" // Register Tavus provider
)

// Configure avatar
result, err := avatar.Setup(avatar.SetupConfig{
    Provider: "tavus",  // or "static", "" (none)
    Tavus: avatar.TavusConfig{
        APIKey: os.Getenv("TAVUS_API_KEY"),
        PalID:  os.Getenv("TAVUS_PAL_ID"),  // Optional
    },
    LiveKitURL:       os.Getenv("LIVEKIT_URL"),
    LiveKitAPIKey:    os.Getenv("LIVEKIT_API_KEY"),
    LiveKitAPISecret: os.Getenv("LIVEKIT_API_SECRET"),
})

switch result.Mode {
case avatar.SetupModeStatic:
    // Apply static image to agent options
    agentOpts.MediaMode = agent.AudioWithImage
    agentOpts.Image.H264Path = result.StaticImage.H264Path

case avatar.SetupModeLive:
    // Start live avatar session after agent joins room
    err := result.Session.Start(ctx, avatar.StartOptions{
        Room:          agent.Room(),
        AgentIdentity: "my-agent",
        // ... LiveKit credentials
    })
    defer result.Session.Close(ctx)

    // Wait for avatar to join
    result.Session.WaitForJoin(ctx, 30*time.Second)

    // Stream TTS audio to avatar for lip-sync
    audioOut := result.Session.AudioOutput()
    audioOut.CaptureFrame(ctx, ttsAudio)
    audioOut.Flush(ctx)
}

Static Image Avatar

For a simple visual presence without lip-sync:

result, _ := avatar.Setup(avatar.SetupConfig{
    Provider: avatar.ProviderStatic,
    StaticImage: avatar.StaticImageConfig{
        H264Path: "/path/to/avatar.h264",  // Pre-encoded H.264
        // Or: UseDefault: true            // Use embedded default
    },
})

Live Lip-Sync Avatar (Tavus)

For realistic talking head video with Tavus:

result, _ := avatar.Setup(avatar.SetupConfig{
    Provider: avatar.ProviderTavus,
    Tavus: avatar.TavusConfig{
        APIKey: os.Getenv("TAVUS_API_KEY"),
        PalID:  "your-pal-id",  // Or use default
        FaceID: "optional-face-override",
    },
    // LiveKit credentials for avatar token generation
    LiveKitURL:       os.Getenv("LIVEKIT_URL"),
    LiveKitAPIKey:    os.Getenv("LIVEKIT_API_KEY"),
    LiveKitAPISecret: os.Getenv("LIVEKIT_API_SECRET"),
})

The avatar joins the LiveKit room as a participant with synchronized lip movements.

For animated characters or other avatar styles, see the Avatar Provider Comparison.

OmniMeet Integration

This package also provides the LiveKit provider for OmniMeet, the PlexusOne meeting abstraction.

import "github.com/plexusone/omni-livekit/omnimeet"

provider, _ := omnimeet.NewProvider(omnimeet.Config{
    APIKey:    os.Getenv("LIVEKIT_API_KEY"),
    APISecret: os.Getenv("LIVEKIT_API_SECRET"),
    ServerURL: os.Getenv("LIVEKIT_URL"),
})

// Create meeting
m, _ := provider.CreateMeeting(ctx, meeting.CreateRequest{
    Name: "Team Standup",
})

// Join as agent
factory := provider.(provider.AgentParticipantFactory)
agent, _ := factory.CreateAgentParticipant(provider.AgentParticipantOptions{
    AutoSubscribe: true,
})

See the OmniMeet LiveKit Provider documentation for details.

Related Packages

• omnimeet-core - Meeting abstraction core
• omnivoice-core - Voice interfaces
• omnivoice - STT/TTS providers
• livekit/server-sdk-go - LiveKit Go SDK

License

MIT License