voice

README

something-voice

something-voice is a standalone Go library for continuous, local speech transcription. It captures microphone audio with malgo, recognizes general speech with the streaming sherpa-onnx Go API, prints each finalized utterance once, and delivers the same finalized utterance to the caller.

It does not use Windows Speech Recognition, SAPI grammars, the Web Speech API, a cloud service, or a command-line speech process. It does not execute commands or attach meanings to speech.

Setup

The module path is github.com/MisterKeke/something-voice.

Pinned dependencies:

github.com/k2-fsa/sherpa-onnx-go v1.13.8
github.com/gen2brain/malgo v0.11.25

The native packages require cgo and a working C toolchain. Supported build targets are Windows, Linux, and macOS on architectures supported by the pinned sherpa-onnx Go packages. Other operating systems, and builds with cgo disabled, return ErrUnsupported at startup.

Download a general-purpose streaming model before creating a recognizer. The documented English example is sherpa-onnx-streaming-zipformer-en-2023-06-26, available from the sherpa-onnx ASR model release. Its directory contains four files used by this package:

encoder-epoch-99-avg-1-chunk-16-left-128.onnx
decoder-epoch-99-avg-1-chunk-16-left-128.onnx
joiner-epoch-99-avg-1-chunk-16-left-128.onnx
tokens.txt

The model documentation lists other online transducer models and languages. Set ModelConfig.Language or Config.Language as descriptive metadata and provide the corresponding model files. Language selection is determined by the model; the online transducer API does not translate a language tag into a model.

The sherpa-onnx Go API is a cgo wrapper around prebuilt platform native libraries. At runtime, ship the native files for the selected platform and architecture with the application. In particular, the upstream Go documentation says Windows users may need to copy DLLs from the pinned sherpa-onnx-go-windows package's lib/x86_64-pc-windows-gnu (Win64), lib/i686-pc-windows-gnu (Win32), or corresponding arm64 directory beside the executable. See the upstream Go API installation notes for platform-specific details. malgo requires cgo; it does not require an extra library on Windows/macOS and links -ldl on Linux/BSD systems.

Library API

ctx, cancel := context.WithCancel(context.Background())
defer cancel()

recognizer, err := voice.New(voice.Config{
    Language: "en-US",
    Model: voice.ModelConfig{
        Encoder: "/models/encoder-epoch-99-avg-1-chunk-16-left-128.onnx",
        Decoder: "/models/decoder-epoch-99-avg-1-chunk-16-left-128.onnx",
        Joiner:  "/models/joiner-epoch-99-avg-1-chunk-16-left-128.onnx",
        Tokens:  "/models/tokens.txt",
        ModelType: "zipformer2",
    },
    OnFinal: func(utterance voice.Utterance) {
        // Match utterance.Text in the importing application.
    },
})
if err != nil {
    return err
}

session, err := recognizer.Start(ctx)
if err != nil {
    return err
}
if err := session.WaitReady(); err != nil {
    return err
}

for utterance := range session.Finals() {
    _ = utterance // The library has already printed this line to Output.
}
return session.Wait()

Config.Output defaults to os.Stdout; inject a bytes.Buffer or another in-memory io.Writer in tests. The library never opens a transcript file. OnFinal and Session.Finals() are alternative ways to receive actionable finalized text; do not print it again in the caller if you want one console line per utterance. Session.Partials() and OnPartial are informational and are never printed or delivered as final events.

Call session.Stop() or recognizer.Stop() to cancel capture and wait until the microphone, sherpa stream/recognizer, malgo device/context, and worker have been released. Context cancellation has the same cleanup guarantee. Start and Stop are serialized so a session cannot be replaced while its resources are still being released.

Runtime behavior and privacy

The malgo callback only copies one bounded PCM chunk into an in-memory queue. Recognition runs on the session worker, away from the callback. If the queue fills, the newest chunk is dropped, the session is canceled, and ErrAudioOverload is reported; the queue never grows without bound. A malgo stop notification ends the session with ErrMicrophoneDisconnected.

The library retains microphone samples and event text in memory only. It never writes audio, transcripts, temporary recordings, or transcript logs to a file or database, and it never sends audio or transcripts over the network. Model and native runtime files are read locally. Microphone permission is controlled by the operating system and must be granted by the host application/user.

Final text is emitted only when sherpa-onnx reports an endpoint. A partial may change several times while the user is speaking and is not a command. Only speech recognized by the selected model can be printed; this library does not claim word-perfect transcription.

Errors include ErrMissingModel, ErrNoMicrophone, ErrDeviceUnavailable, ErrMicrophonePermission, ErrMicrophoneDisconnected, ErrRecognizerInit, ErrRecognition, ErrAudioOverload, ErrUnsupported, and ErrOutput. Use errors.Is when handling them.

Demo

cmd/voice-demo is a small terminal listener. It uses the library's own stdout printing and waits for Ctrl+C:

go run ./cmd/voice-demo -encoder /models/encoder.onnx -decoder /models/decoder.onnx -joiner /models/joiner.onnx -tokens /models/tokens.txt

Use -microphone with a malgo device ID or name; leave it empty for the OS default. The demo does not match phrases or perform actions.

Licensing and attribution

The library source in this repository is MIT licensed; see LICENSE.

This project includes or links to these external components:

• sherpa-onnx, including its Go bindings and native runtime, is Apache-2.0 licensed. Preserve upstream notices when redistributing native files.
• malgo is released into the public domain under its included Unlicense/public-domain dedication.
• The suggested sherpa-onnx-streaming-zipformer-en-2023-06-26 model is a separate model artifact. Review the license and attribution included with the exact model archive before redistribution or commercial deployment; this repository does not redistribute model weights and does not assume that the sherpa-onnx source license covers them.

Manual verification

Per the project request, tests, builds, the demo, and other development commands were not run. Run these manually from the repository root after installing native prerequisites:

go mod tidy
go test ./...
go vet ./...
go build ./...
go run ./cmd/voice-demo -encoder <encoder.onnx> -decoder <decoder.onnx> -joiner <joiner.onnx> -tokens <tokens.txt>

For Something-v2, tag this breaking rewrite as v0.2.0 and import github.com/MisterKeke/something-voice@v0.2.0.

Documentation

Overview

Package voice provides continuous, local speech transcription.

The package feeds microphone audio to a sherpa-onnx streaming recognizer. It prints finalized utterances itself and also exposes those utterances to the importing application. It never interprets spoken text as a command.

Index

• Variables
• type Config
• type ModelConfig
• type Partial
• type Recognizer
  • func New(config Config) (*Recognizer, error)
  • func (r *Recognizer) Language() string
  • func (r *Recognizer) Start(ctx context.Context) (*Session, error)
  • func (r *Recognizer) Stop() error
• type Session
  • func (s *Session) Done() <-chan struct{}
  • func (s *Session) Errors() <-chan error
  • func (s *Session) Finals() <-chan Utterance
  • func (s *Session) Partials() <-chan Partial
  • func (s *Session) Ready() <-chan struct{}
  • func (s *Session) Stop() error
  • func (s *Session) Wait() error
  • func (s *Session) WaitReady() error
• type Utterance

Variables

var (
	// ErrInvalidConfiguration indicates invalid library configuration.
	ErrInvalidConfiguration = errors.New("voice: invalid configuration")
	// ErrMissingModel indicates that a configured model file is missing or not a regular file.
	ErrMissingModel = errors.New("voice: missing model file")
	// ErrUnsupported indicates that the current platform or build cannot capture audio.
	ErrUnsupported = errors.New("voice: speech recognition is unsupported")
	// ErrNoMicrophone indicates that no capture device is available.
	ErrNoMicrophone = errors.New("voice: no microphone is available")
	// ErrDeviceUnavailable indicates that the requested microphone cannot be found.
	ErrDeviceUnavailable = errors.New("voice: requested microphone is unavailable")
	// ErrMicrophonePermission indicates that the audio backend could not open the microphone.
	ErrMicrophonePermission = errors.New("voice: microphone permission or device-open failure")
	// ErrMicrophoneDisconnected indicates that the active microphone stopped unexpectedly.
	ErrMicrophoneDisconnected = errors.New("voice: microphone disconnected")
	// ErrRecognizerInit indicates that sherpa-onnx could not create its native recognizer.
	ErrRecognizerInit = errors.New("voice: recognizer initialization failed")
	// ErrRecognition indicates a failure while processing audio.
	ErrRecognition = errors.New("voice: recognition failed")
	// ErrAudioOverload indicates that the bounded audio queue overflowed.
	ErrAudioOverload = errors.New("voice: audio processing queue overflowed")
	// ErrOutput indicates that the configured output writer failed.
	ErrOutput = errors.New("voice: output writer failed")
	// ErrSessionActive indicates that a recognizer already has a listening session.
	ErrSessionActive = errors.New("voice: a listening session is already active")
	// ErrStopped indicates that a session was explicitly stopped.
	ErrStopped = errors.New("voice: listening session stopped")
)

Types

type Config

type Config struct {
	Model ModelConfig

	// Language is descriptive metadata for the selected model. Model.Language
	// is used when Language is empty.
	Language string
	// Microphone selects a device by malgo ID or device name. Empty selects the
	// operating system's default capture device.
	Microphone string
	// Output receives exactly one newline-terminated line per finalized
	// utterance. Nil uses os.Stdout. The library never opens a transcript file.
	Output io.Writer
	// OnFinal receives finalized utterances after the library writes them.
	OnFinal func(Utterance)
	// OnPartial receives changing partial recognition text. Partials are never
	// written to Output and are never delivered through Finals.
	OnPartial func(Partial)

	// SampleRate and FeatureDim default to 16000 and 80, respectively.
	SampleRate int
	FeatureDim int
	// AudioQueueSize bounds microphone audio retained in memory. When full,
	// the session stops with ErrAudioOverload rather than growing memory.
	AudioQueueSize int
	// EventQueueSize bounds the exported final and partial event channels.
	EventQueueSize int
}

Config controls a Recognizer.

type ModelConfig

type ModelConfig struct {
	Encoder string
	Decoder string
	Joiner  string
	Tokens  string

	// ModelType is optional for most models. It defaults to zipformer2, which
	// is the type used by the documented English Zipformer example.
	ModelType string
	// Provider is a sherpa-onnx provider such as cpu. It defaults to cpu.
	Provider string
	// NumThreads controls native model computation and defaults to one.
	NumThreads int
	// Language is descriptive metadata for the selected model (for example,
	// en-US). The online transducer API infers language from the model files.
	Language string
}

ModelConfig identifies one sherpa-onnx online transducer model.

A general-purpose streaming Zipformer model needs encoder, decoder, joiner, and tokens files. The files are read by the native runtime; this library never downloads or copies them.

type Partial

type Partial struct {
	Text      string
	Timestamp time.Time
}

Partial is an interim recognition result. It is informational only.

type Recognizer

type Recognizer struct {
	// contains filtered or unexported fields
}

Recognizer owns one local model configuration and permits one session at a time.

func New

func New(config Config) (*Recognizer, error)

New validates model paths and creates a local recognizer configuration. It does not access the microphone or initialize the native model.

func (*Recognizer) Language

func (r *Recognizer) Language() string

Language returns the descriptive language metadata configured for the model.

func (*Recognizer) Start

func (r *Recognizer) Start(ctx context.Context) (*Session, error)

Start begins microphone capture and continues until Stop, context cancellation, a microphone failure, or a recognition failure. Setup errors are reported by Session.WaitReady and Session.Errors.

func (*Recognizer) Stop

func (r *Recognizer) Stop() error

Stop stops the active session and waits for its workers and native resources to be released. It is safe to call repeatedly.

type Session

type Session struct {
	// contains filtered or unexported fields
}

func (*Session) Done

func (s *Session) Done() <-chan struct{}

Done returns a channel closed only after all native resources and workers have been released.

func (*Session) Errors

func (s *Session) Errors() <-chan error

Errors returns zero or one terminal error and is closed when the session ends.

func (*Session) Finals

func (s *Session) Finals() <-chan Utterance

Finals returns a bounded stream of finalized utterances. It is closed when the session ends.

func (*Session) Partials

func (s *Session) Partials() <-chan Partial

Partials returns a bounded, informational stream of changing partial text. A slow partial consumer may cause partials to be dropped; finalized text is never replaced by a partial.

func (*Session) Ready

func (s *Session) Ready() <-chan struct{}

Ready returns a channel closed after native setup succeeds or fails.

func (*Session) Stop

func (s *Session) Stop() error

Stop requests cancellation, waits for cleanup, and returns the session's terminal error. It is safe to call repeatedly.

func (*Session) Wait

func (s *Session) Wait() error

Wait blocks until the session ends and returns its terminal error. Explicit Stop returns ErrStopped; parent-context cancellation returns that context's error. A nil result means the backend ended normally.

func (*Session) WaitReady

func (s *Session) WaitReady() error

WaitReady waits for microphone and recognizer setup to complete.

type Utterance

type Utterance struct {
	Text      string
	Timestamp time.Time
}

Utterance is one finalized speech segment. It is the only recognition result intended for command matching by an importing application.