audio

package module
v1.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 21, 2026 License: MIT Imports: 12 Imported by: 0

README

audio

github.com/caitunai/audio is a Go package for streaming audio processing. It decodes encoded audio frames to PCM, optionally applies denoise models, runs streaming VAD, and returns speech segments, silence gaps, audio level stats, and real-time VAD events.

The package is designed as a reusable SDK:

  • It does not read application configuration directly.
  • It does not depend on Gin, Viper, WebSocket, or application storage code.
  • It does not write archive files or business data.
  • Callers own transport, config loading, logging fields, and persistence.

Features

  • Frame decoding:
    • Opus via github.com/hraban/opus
    • Speex via github.com/caitunai/speex
    • LC3 via github.com/caitunai/lc3
  • Denoise providers:
    • none
    • dpdfnet
    • gtcrn
  • Streaming VAD:
    • Silero VAD v5/v6 ONNX models
    • 16 kHz windows are processed in 512-sample chunks
    • VAD events are emitted while frames are processed
  • Result metadata:
    • speech segments
    • silence gaps between segments
    • RMS, dBFS, peak, silent flag
    • VAD probability stats and processed window count

For the complete VAD state machine, input/output formats, tuning guidance, and ASR integration rules, see VAD.md. For all denoise providers, defaults, tensor mappings, and validation rules, see DENOISE.md.

Installation

go get github.com/caitunai/audio

When developing this package inside the ASR application, the parent project uses:

replace github.com/caitunai/audio => ./third_party/audio
ONNX Runtime

VAD and denoise providers use ONNX Runtime through cgo. The current cgo flags expect the headers and library in common Homebrew/Linux locations.

macOS with Homebrew:

brew install onnxruntime

Expected path:

/opt/homebrew/opt/onnxruntime/

Linux default include paths:

/usr/local/include/onnxruntime
/usr/include/onnxruntime

The package intentionally relies on github.com/streamer45/silero-vad-go/speech for ONNX Runtime linker flags so multiple cgo files do not emit duplicate -lonnxruntime or duplicate rpath flags.

At the moment this package uses the github.com/iflamed/silero-vad-go fork through a replace entry because the fork keeps the upstream module path:

replace github.com/streamer45/silero-vad-go => github.com/iflamed/silero-vad-go v0.0.0-20260707034153-8ea1a1ba2fdc

Applications that consume this module and need the same linker behavior should keep the same replace until the fork is released with a stable module path or upstream accepts the linker flag changes.

Quick Start

package main

import (
	"github.com/caitunai/audio"
	"github.com/rs/zerolog/log"
)

func main() {
	cfg := audio.ProcessingConfig{
		Codec: audio.CodecConfig{
			Codec:      audio.CodecOpus,
			SampleRate: audio.DefaultSampleRate,
		},
		Denoise: audio.DenoiseConfig{
			Provider:   audio.DenoiseProviderNone,
			SampleRate: audio.DefaultSampleRate,
			Enabled:    true,
		},
		VAD: audio.VADConfig{
			ModelPath:            "models/silero_vad.onnx",
			ModelVersion:         audio.DefaultVADModelVersion,
			SampleRate:           audio.DefaultSampleRate,
			MinSilenceDurationMS: audio.DefaultVADMinSilenceDurationMS,
			SpeechPadMS:          audio.DefaultVADSpeechPadMS,
			MaxBufferedSeconds:   audio.DefaultVADMaxBufferedSeconds,
			Threshold:            audio.DefaultVADThreshold,
			Enabled:              true,
		},
	}

	session, err := audio.NewProcessingSession(log.Logger, cfg)
	if err != nil {
		panic(err)
	}
	defer session.Close()

	// For each encoded WebSocket/audio frame:
	frame := []byte{}
	frameResult, err := session.ProcessFrame(frame)
	if err != nil {
		panic(err)
	}
	_ = frameResult.Samples // decoded and optionally denoised PCM, valid until the next frame
	_ = frameResult.Events  // streaming VAD events

	finalResult, err := session.Finish()
	if err != nil {
		panic(err)
	}
	_ = finalResult
}

Core Pipeline

encoded frame payload
  -> FrameDecoder.Decode
  -> Denoiser.Process
  -> VADRunner.Process
  -> ProcessingFrameResult{Samples, Events}
  -> FinishWithOutput()
  -> ProcessingFinishResult{flush Samples, flush Events, tail-only VADResult}

ProcessingFrameResult.Samples may alias an internal decoder buffer. Consume or copy it before processing the next frame. Completed segments are delivered by streaming speech_end events and are not retained by the VAD runner. VADResult.Segments contains only tail segments first delivered during finish; applications that need complete history must store or persist events themselves.

Codec Configuration

type CodecConfig struct {
	Codec              string
	SpeexMode          string
	SampleRate         int
	LC3FrameDurationUS int
	LC3Bitrate         int
	LC3FrameBytes      int
}

Supported codec values:

  • audio.CodecOpus
  • audio.CodecSpeex
  • audio.CodecLC3

Defaults:

  • empty codec defaults to Opus
  • empty sample rate defaults to the VAD sample rate passed by the caller
  • Speex mode is inferred from sample rate:
    • 8000 Hz: narrowband
    • 16000 Hz: wideband
    • 32000 Hz: ultra-wideband
  • LC3 frame duration defaults to the LC3 package default
  • LC3 bitrate defaults to the LC3 package default

You can also use the decoder directly:

decoder, buffer, err := audio.NewFrameDecoder(audio.CodecConfig{
	Codec:      audio.CodecLC3,
	SampleRate: 16000,
})
if err != nil {
	panic(err)
}
defer decoder.Close()

pcm, err := decoder.Decode(payload, buffer)

Denoise Configuration

type DenoiseConfig struct {
	Provider        string
	ModelPath       string
	InputName       string
	StateInputName  string
	OutputName      string
	StateOutputName string
	SampleRate      int
	WindowSamples   int
	FrameSamples    int
	Enabled         bool
}

Supported providers:

  • audio.DenoiseProviderNone
  • audio.DenoiseProviderDPDFNet
  • audio.DenoiseProviderGTCRN

Use audio.NormalizeDenoiseConfig before persisting or displaying effective settings. Use audio.ValidateDenoiseConfig to validate user-provided values.

No Denoise

Use this when input audio has already been denoised.

audio.DenoiseConfig{
	Provider:   audio.DenoiseProviderNone,
	SampleRate: 16000,
	Enabled:    true,
}

provider=none is normalized to disabled denoise and does not require a model path.

DPDFNet
audio.DenoiseConfig{
	Provider:        audio.DenoiseProviderDPDFNet,
	ModelPath:       "models/dpdfnet8.onnx",
	SampleRate:      16000,
	WindowSamples:   320,
	FrameSamples:    160,
	InputName:       "spec",
	StateInputName:  "state_in",
	OutputName:      "spec_e",
	StateOutputName: "state_out",
	Enabled:         true,
}
GTCRN
audio.DenoiseConfig{
	Provider:       audio.DenoiseProviderGTCRN,
	ModelPath:      "models/gtcrn_simple.onnx",
	SampleRate:     16000,
	WindowSamples:  512,
	FrameSamples:   160,
	InputName:      "mix",
	StateInputName: "",
	OutputName:     "enh",
	StateOutputName:"",
	Enabled:        true,
}

GTCRN can infer state tensor names when StateInputName and StateOutputName are empty and the model exposes compatible state inputs/outputs.

VAD Configuration

type VADConfig struct {
	ModelPath            string
	ModelVersion         string
	SampleRate           int
	MinSilenceDurationMS int
	SpeechPadMS          int
	MaxBufferedSeconds   int
	Threshold            float32
	Enabled              bool
}

Supported sample rates:

  • 8000
  • 16000

Supported Silero model versions:

  • v5
  • v6

Default values:

  • sample rate: 16000
  • model version: v5
  • threshold: 0.35
  • minimum silence duration: 500 ms
  • speech padding: 30 ms
  • max buffered duration: 600 s

When VAD is disabled, frames are still decoded and optionally denoised, but no speech segments are emitted.

Resource Lifecycle

Call Close for every object that owns native resources:

  • ProcessingSession
  • FrameDecoder
  • Denoiser
  • VADRunner

ProcessingSession.Finish() automatically flushes denoise, finalizes VAD, and closes the session. Calling Close() again is safe. Spectral DPDFNet/GTCRN processing compensates overlap latency by withholding startup-delay samples and draining the same number of tail samples during finish, so callers must not replace Finish/FinishWithOutput with Close on a normal stream ending.

Model runtimes for DPDFNet, GTCRN, and Silero VAD are cached internally to avoid loading a new model for every audio stream. Per-stream state remains isolated in each denoiser/VAD runner instance.

Error Handling

Errors are package-level sentinel errors and are joined with lower-level causes when useful. Callers can classify errors with errors.Is.

Common errors:

  • ErrAudioCodecInvalid
  • ErrAudioDecode
  • ErrAudioDenoiseConfig
  • ErrAudioDenoiseInit
  • ErrAudioDenoiseProcess
  • ErrAudioVADInvalidConfig
  • ErrAudioVADDetect
  • ErrSileroVADUnavailable
  • ErrAudioProcessingClosed

MaxBufferedSeconds controls downstream retained PCM capacity; it is not a maximum stream duration. ProcessingSession keeps cumulative timing counters but does not retain all decoded PCM or reject a long-lived stream solely because its total sample count exceeds that capacity.

Testing

Run package tests:

GOCACHE="$(pwd)/.cache/go-build" go test ./...

Some tests and runtime paths require ONNX Runtime and cgo:

CGO_ENABLED=1 go test ./...

The ASR parent repository keeps sample Silero test files in third_party/silero-vad-go/testfiles. If this module is tested outside the parent repository and those files are unavailable, the integration tests that depend on them are skipped automatically.

Publishing Notes

Before publishing a release:

  1. Ensure go test ./... passes on macOS and Linux with ONNX Runtime installed.
  2. Tag the repository with a semantic version, for example v0.1.0.
  3. Keep codec packages released separately:
    • github.com/caitunai/speex
    • github.com/caitunai/lc3
  4. Document model file compatibility for DPDFNet, GTCRN, and Silero VAD.
  5. Keep application-specific archive and WebSocket code outside this module.

Documentation

Index

Constants

View Source 
const (
	CodecOpus  = "opus"
	CodecSpeex = "speex"
	CodecLC3   = "lc3"
)
View Source 
const (
	DefaultSampleRate               = 16000
	DefaultVADModelVersion          = "v5"
	DefaultVADThreshold             = 0.35
	DefaultVADMinSilenceDurationMS  = 500
	DefaultVADSoftSilenceDurationMS = 64
	DefaultVADMaxSpeechDurationMS   = 30000
	DefaultVADSpeechPadMS           = 30
	DefaultVADMaxBufferedSeconds    = 600
	DenoiseProviderNone             = "none"
	DenoiseProviderDPDFNet          = "dpdfnet"
	DenoiseProviderGTCRN            = "gtcrn"
	DefaultDenoiseProvider          = DenoiseProviderDPDFNet
	DefaultDenoiseWindowSamples     = 320
	DefaultDenoiseFrameSamples      = 160
	DefaultDenoiseInputName         = "spec"
	DefaultDenoiseStateInputName    = "state_in"
	DefaultDenoiseOutputName        = "spec_e"
	DefaultDenoiseStateOutputName   = "state_out"
	DefaultGTCRNWindowSamples       = 512
	DefaultGTCRNFrameSamples        = 160
	DefaultGTCRNInputName           = "mix"
	DefaultGTCRNOutputName          = "enh"
)

Variables

View Source 
var (
	ErrOpusDecoderInit   = errors.New("opus decoder init failed")
	ErrOpusDecode        = errors.New("opus decode failed")
	ErrSpeexDecoderInit  = errors.New("speex decoder init failed")
	ErrSpeexDecode       = errors.New("speex decode failed")
	ErrSpeexClose        = errors.New("speex close failed")
	ErrLC3DecoderInit    = errors.New("lc3 decoder init failed")
	ErrLC3Decode         = errors.New("lc3 decode failed")
	ErrLC3Close          = errors.New("lc3 close failed")
	ErrAudioDecode       = errors.New("audio decode failed")
	ErrAudioCodecInvalid = errors.New("audio codec invalid")
)
View Source 
var (
	ErrAudioPCMBufferLimit   = errors.New("audio pcm buffer limit exceeded")
	ErrSileroVADUnavailable  = errors.New("silero vad unavailable")
	ErrAudioVADDetect        = errors.New("audio vad detect failed")
	ErrAudioVADClose         = errors.New("audio vad close failed")
	ErrAudioVADInvalidConfig = errors.New("audio vad config invalid")
	ErrAudioDenoiseInit      = errors.New("audio denoise init failed")
	ErrAudioDenoiseProcess   = errors.New("audio denoise process failed")
	ErrAudioDenoiseClose     = errors.New("audio denoise close failed")
	ErrAudioDenoiseConfig    = errors.New("audio denoise config invalid")
	ErrAudioProcessingClosed = errors.New("audio processing session closed")
)

Functions

func ClearDenoiseModelCache 

func ClearDenoiseModelCache()

func ClearVADDetectorCache 

func ClearVADDetectorCache()

func ValidateDenoiseConfig 

func ValidateDenoiseConfig(cfg DenoiseConfig) error

func ValidateVADConfig 

func ValidateVADConfig(cfg VADConfig) error

Types

type CodecConfig 

type CodecConfig struct {
	Codec              string
	SpeexMode          string
	SampleRate         int
	LC3FrameDurationUS int
	LC3Bitrate         int
	LC3FrameBytes      int
}

type DenoiseConfig 

type DenoiseConfig struct {
	Provider        string
	ModelPath       string
	InputName       string
	StateInputName  string
	OutputName      string
	StateOutputName string
	SampleRate      int
	WindowSamples   int
	FrameSamples    int
	Enabled         bool
}

func NormalizeDenoiseConfig 

func NormalizeDenoiseConfig(cfg DenoiseConfig) DenoiseConfig

type Denoiser 

type Denoiser interface {
	Enabled() bool
	Process(samples []float32) ([]float32, error)
	Flush() ([]float32, error)
	Close() error
}

func NewDenoiser 

func NewDenoiser(cfg DenoiseConfig) (Denoiser, error)

type FrameDecoder 

type FrameDecoder interface {
	Decode(payload []byte, buffer []float32) ([]float32, error)
	Close() error
}

func NewFrameDecoder 

func NewFrameDecoder(cfg CodecConfig) (FrameDecoder, []float32, error)

type ProcessingConfig 

type ProcessingConfig struct {
	Denoise DenoiseConfig
	Codec   CodecConfig
	VAD     VADConfig
}

type ProcessingFinishResult added in v1.0.1 

type ProcessingFinishResult struct {
	Samples   []float32
	Events    []VADEvent
	VADResult VADResult
}

type ProcessingFrameResult 

type ProcessingFrameResult struct {
	// Samples aliases the decoder buffer and must be consumed before the next frame.
	Samples []float32
	Events  []VADEvent
}

type ProcessingSession 

type ProcessingSession struct {
	// contains filtered or unexported fields
}

func NewProcessingSession 

func NewProcessingSession(logger zerolog.Logger, cfg ProcessingConfig) (*ProcessingSession, error)

func (*ProcessingSession) AudioLevel 

func (s *ProcessingSession) AudioLevel() SignalStats

func (*ProcessingSession) Close 

func (s *ProcessingSession) Close() error

func (*ProcessingSession) Codec 

func (s *ProcessingSession) Codec() string

func (*ProcessingSession) DecodeOpusFrame 

func (s *ProcessingSession) DecodeOpusFrame(payload []byte) error

func (*ProcessingSession) Finish 

func (s *ProcessingSession) Finish() (VADResult, error)

func (*ProcessingSession) FinishWithOutput added in v1.0.1 

func (s *ProcessingSession) FinishWithOutput() (result ProcessingFinishResult, err error)

func (*ProcessingSession) MaxPCMSamples 

func (s *ProcessingSession) MaxPCMSamples() int

func (*ProcessingSession) ProcessFrame 

func (s *ProcessingSession) ProcessFrame(payload []byte) (ProcessingFrameResult, error)

func (*ProcessingSession) ProcessOpusFrame 

func (s *ProcessingSession) ProcessOpusFrame(payload []byte) (ProcessingFrameResult, error)

func (*ProcessingSession) SampleRate 

func (s *ProcessingSession) SampleRate() int

type SignalStats 

type SignalStats struct {
	RMS    float64 `json:"rms"`
	DBFS   float64 `json:"dbfs"`
	Peak   float32 `json:"peak"`
	Silent bool    `json:"silent"`
}

func AnalyzeSignal 

func AnalyzeSignal(samples []float32) SignalStats

type SilenceGap 

type SilenceGap struct {
	PreviousSegmentIndex int     `json:"previous_segment_index"`
	NextSegmentIndex     int     `json:"next_segment_index"`
	StartAt              float64 `json:"start_at"`
	EndAt                float64 `json:"end_at"`
	Duration             float64 `json:"duration"`
}

type SpeechSegment 

type SpeechSegment struct {
	Index     int     `json:"index"`
	StartAt   float64 `json:"start_at"`
	EndAt     float64 `json:"end_at"`
	Duration  float64 `json:"duration"`
	OpenEnded bool    `json:"open_ended"`
}

type VADConfig 

type VADConfig struct {
	ModelPath             string
	ModelVersion          string
	SampleRate            int
	MinSilenceDurationMS  int
	SoftSilenceDurationMS int
	MaxSpeechDurationMS   int
	SpeechPadMS           int
	MaxBufferedSeconds    int
	Threshold             float32
	Enabled               bool
}

type VADEvent 

type VADEvent struct {
	Type    VADEventType  `json:"type"`
	Segment SpeechSegment `json:"segment"`
	Gap     SilenceGap    `json:"gap"`
	HasGap  bool          `json:"has_gap"`
}

type VADEventType 

type VADEventType string
const (
	VADEventSpeechStart  VADEventType = "speech_start"
	VADEventSpeechEnd    VADEventType = "speech_end"
	VADEventSoftBoundary VADEventType = "speech_soft_boundary"
)

type VADResult 

type VADResult struct {
	Segments      []SpeechSegment `json:"segments"`
	Gaps          []SilenceGap    `json:"gaps"`
	AudioLevel    SignalStats     `json:"audio_level"`
	AudioDuration float64         `json:"audio_duration"`
	VADLastProb   float32         `json:"vad_last_probability"`
	VADMaxProb    float32         `json:"vad_max_probability"`
	SampleRate    int             `json:"sample_rate"`
	SampleCount   int             `json:"sample_count"`
	SegmentCount  int             `json:"segment_count"`
	VADWindows    int             `json:"vad_windows"`
	VADEnabled    bool            `json:"vad_enabled"`
}

type VADRunner 

type VADRunner interface {
	Enabled() bool
	Process(samples []float32) ([]VADEvent, error)
	Finish(sampleCount int) (VADResult, error)
	Close() error
}

func NewVADRunner 

func NewVADRunner(cfg VADConfig) (VADRunner, error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.