go-whistle

README

go-whistle

A pure-Go port of Whistle, Cactus Compute's 16.9 MB on-device speech-to-text model (55M parameters, 7 languages: en, de, fr, es, it, nl, pl).

Status: working. Transcription with language detection, keyword biasing, the depth ladder and word timestamps, live streaming and embeddings all run in pure Go and match the reference engine closely (see Parity). The model runs straight from the packed 2/4-bit whistle.cact. The engine is still faster: on an Apple M5 Pro, a median 18.8 ms to the first token against its 6.9 ms and 1085 against 1499 tok/s (bench/RESULTS.md). On amd64 the float products still run in pure Go (PLAN.md 12.3). See PLAN.md for the roadmap.

Quickstart

Install the CLI (Go 1.27 or later, which an older go command downloads by itself; pure Go, no cgo) and fetch the pinned model files:

go install github.com/cwbudde/go-whistle/cmd/whistle@latest
whistle download

ok       whistle.cact (sha256 verified)
ok       whistle.safetensors (sha256 verified)
ok       config.json (sha256 verified)
whistle b358ddad ready in models/whistle

download verifies each file by size and SHA-256 and resumes interrupted downloads. The commands below use models/whistle/whistle.cact by default (--model picks another, also whistle.safetensors) and the fixture clips in testdata/audio. A clone has all 32; without one, fetch the two the examples start with:

mkdir -p testdata/audio
for c in ls-other-3080-5040-0018 ls-other-3080-5040-0011; do
	curl -sSfLo testdata/audio/$c.wav https://raw.githubusercontent.com/cwbudde/go-whistle/main/testdata/audio/$c.wav
done

Check the install

whistle doctor

model      models/whistle/whistle.cact
size       16919407 bytes (16.9 MB)
format     .cact (2/4-bit CQ weights with the tokenizer)
sha256     b6e02f048568ac5d01a2042556c658061e699acbc0aa2a1439f52f3d461dffeb = pinned Cactus-Compute/whistle@b358ddad whistle.cact
files      whistle.safetensors ok, config.json ok
weights    packed: CQ matrices straight from the file (whistle.Load's default; WithFloatWeights dequantises)
go         go1.27.1 darwin/arm64
kernels    NEON: GEMM, dot, axpy, CQ LUT GEMV, softmax exp
workers    GOMAXPROCS 15, 15 CPUs
load       ok in 9ms; languages en de fr es it nl pl
self-test  2 s tone burst: embedding 25×512, transcript [en] "Thank you." in 50ms
clip       testdata/audio/ls-other-3080-5040-0018.wav: [en] "Directed for your master." in 37ms
all checks passed

It exits with 1 when a check fails: a missing or modified model file, a failed load or self-test.

Transcribe

whistle transcribe testdata/audio/ls-other-3080-5040-0018.wav

Directed for your master.

--json prints the engine's JSON object byte for byte; --word-timestamps adds each word's start, end and probability:

whistle transcribe --json --word-timestamps testdata/audio/ls-other-3080-5040-0018.wav

{
  "text": "Directed for your master.",
  "language": "en",
  "words": [
    { "word": "Directed", "start": 0.64, "end": 1.12, "probability": 0.894 },
    { "word": "for", "start": 1.12, "end": 1.52, "probability": 0.739 },
    { "word": "your", "start": 1.52, "end": 1.68, "probability": 0.98 },
    { "word": "master.", "start": 1.68, "end": 2.08, "probability": 0.962 }
  ],
  "ttft_ms": 27.3,
  "decode_tps": 860.3
}

--keywords names a file of words and phrases to favour, one per line:

printf "Cowley's\n" > keywords.txt
whistle transcribe testdata/audio/ls-other-3080-5040-0011.wav
whistle transcribe --keywords keywords.txt testdata/audio/ls-other-3080-5040-0011.wav

Here are some verses of colonies. Tell me how you like them.
Here are some verses of Cowley's. Tell me how you like them.

--language forces the language, and --depth 2..8 runs fewer decoder layers (faster, less accurate). Several files load the model once and print each text after its path, or a JSON line each:

whistle transcribe testdata/audio/ls-other-3080-5040-0011.wav testdata/audio/fleurs-de_de-10229344228128634115.wav

testdata/audio/ls-other-3080-5040-0011.wav: Here are some verses of colonies. Tell me how you like them.
testdata/audio/fleurs-de_de-10229344228128634115.wav: Jeder nimmt an der Gesellschaft Teil und benutzt Transportsysteme, was jeder beklagt sich über die Transportsysteme.

One call takes at most 30 s, as in the engine. A longer file (here the nine ls-clean clips of the stream fixtures joined, 44.91 s) fails with exit code 1:

$ whistle transcribe long.wav
whistle: long.wav: audio limit is 30 s: 718560 samples, at most 480000 (transcribe takes one pass of at most 30 s; whistle stream takes longer audio)

Stream

whistle stream transcribes audio of any length live, as the engine's stream API does: each chunk (--chunk seconds, default 1) is one call that re-transcribes the buffer and commits the words two consecutive passes agree on; the end of the input commits the rest.

whistle stream long.wav

   1.00 s  commit ""  pending "Come and call."
   2.00 s  commit ""  pending "Concord returned to its"
   3.00 s  commit "Concord returned to its"  pending "place amidst the 10."
   4.00 s  commit "place amidst the"  pending "tents."
   5.00 s  commit "tents."  pending "\"You will be frank with me.\""
…
  44.00 s  commit "help protect"  pending "yourself if worse come.\""
  44.91 s  commit "yourself if worse"  pending "comes to worst."
  44.91 s  stop   "comes to worst."
Concord returned to its place amidst the tents. "You will be frank with me?" "I always am." There was something in his air and manner that betrayed to the scout the utter confusion of the state of his mind. Four or five of the latter only lingered about the door of the prison of Uncas, wary but close observers of the manner of their captive. She found the door, but it was locked outside. Among other things on which she cast her eyes was a small crucifix of solid silver, standing on a cabinet near the window. "I am very tired of swimming about here. Oh, Mouse!" "I almost think I can remember feeling little different," Captain Martin said. "I shall give you a pistol to help protect yourself if worse comes to worst.

Without a file (or with -) it reads raw 16 kHz mono PCM from stdin: --format s16le (signed 16-bit little-endian, the default) or f32le (float32 little-endian in [-1, 1]). --json prints each call as the engine's stream JSON, and --commits-only only the calls that commit words plus the stop, as the engine's --audio-stream does:

ffmpeg -loglevel error -i testdata/audio/ls-clean-6930-75918-0000.wav -f s16le -ac 1 -ar 16000 - | whistle stream --json --commits-only

{"text":"Concord returned to its","words":[{"word":"Concord","start":0.56,"end":1.20,"probability":0.909},{"word":"returned","start":1.20,"end":1.60,"probability":0.843},{"word":"to","start":1.60,"end":1.84,"probability":0.990},{"word":"its","start":1.84,"end":2.00,"probability":0.998}],"pending":"place amidst the 10.","language":"en","received":3.00,"pass_ms":56.0}
{"text":"place amidst the","words":[{"word":"place","start":2.00,"end":2.24,"probability":0.998},{"word":"amidst","start":2.24,"end":2.80,"probability":0.977},{"word":"the","start":2.80,"end":2.96,"probability":0.995}],"pending":"tents.","language":"en","received":3.50,"pass_ms":63.0}
{"text":"tents.","words":[{"word":"tents.","start":2.96,"end":3.28,"probability":0.949}],"pending":"","language":"en","received":3.50,"pass_ms":0.0}

A microphone works the same way, for example arecord -q -f S16_LE -r 16000 -c 1 -t raw | whistle stream; Ctrl-C ends the stream and commits the pending words. --realtime plays a file no faster than real time.

Embed

whistle embed writes the encoder output, what the engine's needle_embed returns: one row of 512 values per 80 ms frame, at most 30 s of audio. JSON by default, or a NumPy .npy file:

whistle embed testdata/audio/ls-other-3080-5040-0018.wav | head -c 120
whistle embed --format npy -o clip.npy testdata/audio/ls-other-3080-5040-0018.wav

{"frames":33,"dim":512,"frame_ms":80,"embedding":[[-1.322341,-3.7772563,-1.3063984,-0.8534372,-1.0002542,-1.1180489,0.65

numpy.load("clip.npy") gives a float32 array of shape (33, 512).

Server

whistle serve loads the model once and serves it over HTTP on 127.0.0.1:8080 (--addr), with a log line per request on stderr. POST /transcribe takes a WAV file as the body, or raw 16 kHz mono PCM with ?format=s16le or f32le. Its options are query parameters: language, keywords (repeatable), depth and word_timestamps. It answers with the JSON whistle transcribe --json prints:

whistle serve
curl -s --data-binary @testdata/audio/ls-other-3080-5040-0018.wav 'http://127.0.0.1:8080/transcribe?word_timestamps=true'
ffmpeg -loglevel error -i testdata/audio/ls-clean-6930-75918-0000.wav -f s16le -ac 1 -ar 16000 talk.raw
curl -s --data-binary @talk.raw 'http://127.0.0.1:8080/transcribe?format=s16le&language=en'

{"text":"Directed for your master.","language":"en","words":[{"word":"Directed","start":0.64,"end":1.12,"probability":0.894},{"word":"for","start":1.12,"end":1.52,"probability":0.739},{"word":"your","start":1.52,"end":1.68,"probability":0.980},{"word":"master.","start":1.68,"end":2.08,"probability":0.962}],"ttft_ms":16.1,"decode_tps":947.3}
{"text":"Concord returned to its place amidst the tents.","language":"en","ttft_ms":14.0,"decode_tps":975.3}

Errors are JSON {"error":"..."}: 400 for bad options or audio, 408 for a body slower than --read-timeout, 413 for a body over --max-body-mib (16 MiB), 422 for audio over 30 s, 503 when --concurrency passes run and --queue requests wait. GET /healthz and GET /v1/info (the model's SHA-256, weights mode, languages and the limits) are there too.

GET /stream is a WebSocket (?format=s16le|f32le, language, keywords). Each binary message is a chunk of 16 kHz mono PCM for one stream call and gets one text message back, the call's stream JSON as whistle stream --json prints it. The text message {"type":"stop"} commits the rest, sends that result and closes the connection. A Go client with github.com/coder/websocket (error checks shortened to check):

c, _, err := websocket.Dial(ctx, "ws://127.0.0.1:8080/stream?format=s16le", nil)
check(err)

for i := 0; i < len(pcm); i += 32000 { // 1 s of s16le per call
	check(c.Write(ctx, websocket.MessageBinary, pcm[i:min(i+32000, len(pcm))]))

	_, msg, err := c.Read(ctx)
	check(err)
	fmt.Println(string(msg))
}

check(c.Write(ctx, websocket.MessageText, []byte(`{"type":"stop"}`)))

_, msg, err := c.Read(ctx) // the stop's result
check(err)
fmt.Println(string(msg))

_, _, err = c.Read(ctx) // then the server closes
fmt.Println("close:", websocket.CloseStatus(err))

With pcm read from talk.raw:

{"text":"","words":[],"pending":"Come and call.","language":"en","received":1.00,"pass_ms":18.5}
{"text":"","words":[],"pending":"Concord returned to its","language":"en","received":2.00,"pass_ms":21.3}
{"text":"Concord returned to its","words":[{"word":"Concord","start":0.56,"end":1.20,"probability":0.909},{"word":"returned","start":1.20,"end":1.60,"probability":0.843},{"word":"to","start":1.60,"end":1.84,"probability":0.990},{"word":"its","start":1.84,"end":2.00,"probability":0.998}],"pending":"place amidst the 10.","language":"en","received":3.00,"pass_ms":34.8}
{"text":"place amidst the","words":[{"word":"place","start":2.00,"end":2.24,"probability":0.998},{"word":"amidst","start":2.24,"end":2.80,"probability":0.977},{"word":"the","start":2.80,"end":2.96,"probability":0.995}],"pending":"tents.","language":"en","received":3.50,"pass_ms":34.0}
{"text":"tents.","words":[{"word":"tents.","start":2.96,"end":3.28,"probability":0.949}],"pending":"","language":"en","received":3.50,"pass_ms":0.0}
close: StatusNormalClosure

Ctrl-C (or SIGTERM) shuts the server down: running requests finish, and open streams get their stop result and a 1001 close. whistle serve --help lists the limits; the internal/server package doc has the whole protocol.

All commands exit with 0 on success, 1 when they fail and 2 for a wrong command line.

Web demo

web/ is a static page that runs the library in the browser: cmd/whistle-wasm, built with GOOS=js GOARCH=wasm, runs in a Web Worker, so the page stays responsive during a pass. Nothing leaves the tab. It is published at https://cwbudde.github.io/go-whistle/ from every push to main (.github/workflows/pages.yaml). Build and serve it locally:

just wasm          # web/whistle.wasm and Go's web/wasm_exec.js, both git-ignored
just wasm-serve    # http://127.0.0.1:8080/; just wasm-serve <addr> for another address

On the page:

• Model: fetch it from a URL, by default the pinned Hugging Face file whistle download uses (16.1 MiB), checked against its SHA-256 and kept in the browser's Cache Storage for the next visit; or pick a local whistle.cact.
• Options: language (detect or force), keywords (one per line), depth and word timestamps.
• File: Transcribe shows the text, the language, TTFT, tok/s, a table of the timed words and the engine JSON. A WAV of integer PCM is read by the CLI's own reader in Go, so the JSON is whistle transcribe --json's except for the timings; anything else (float WAV, MP3, …) the browser decodes and resamples to 16 kHz. Stream it in 1 s chunks runs the same file through the stream API, as whistle stream does.
• Microphone: an AudioWorklet resamples the input to 16 kHz mono and the stream API commits words live, with the pending ones in grey; a slow pass makes the next call longer rather than the queue.

Served over plain http from another machine (a LAN IP), the page still loads and checks the model, hashing it in Go instead of Web Crypto, but it cannot cache it, and the microphone needs https or localhost.

The worker calls globalThis.whistle: load(bytes), sha256(bytes), decodeWAV(bytes), transcribe(pcm, {language, keywords, word_timestamps, depth}) with the engine JSON as its result, and streamStart({language, keywords}), streamProcess(pcm) and streamStop() with the stream JSON; every call returns a promise, and an error rejects it. The Go side loads the model with whistle.LoadBytes.

WebAssembly runs on one thread without SIMD: about 25 tok/s and a TTFT of 0.3–0.8 s under Node, a 7 s clip in about 3 s in Chromium; see F-043. just wasm-test runs the package tests under Node (go_js_wasm_exec; -short, just wasm-test "" for the full corpus), then drives the worker headlessly against the CLI:

resampler 16000 Hz: 64000 samples (0 held back), 440 Hz error 1.4e-8, 12 kHz peak 5.0e-1, 1.4 ms per second of audio
resampler 22050 Hz: 63982 samples (18 held back), 440 Hz error 2.6e-6, 12 kHz peak 1.3e-5, 5.1 ms per second of audio
resampler 44100 Hz: 63982 samples (18 held back), 440 Hz error 2.4e-6, 12 kHz peak 6.1e-6, 7.2 ms per second of audio
resampler 48000 Hz: 63982 samples (18 held back), 440 Hz error 3.6e-6, 12 kHz peak 5.2e-6, 7.1 ms per second of audio
resampler 96000 Hz: 63983 samples (17 held back), 440 Hz error 4.4e-6, 12 kHz peak 3.6e-6, 12.8 ms per second of audio
whistle.wasm 0.1.0-wasm: model 16919407 bytes, SHA-256 checked, loaded in 53 ms; languages en de fr es it nl pl
ls-other-3080-5040-0018: [en] 2.64 s, go decoder, wasm ttft 328.7 ms, 22.4 tok/s, 823 ms in all; native ttft 19.2 ms, 735.8 tok/s; equal to whistle transcribe: true, JSON without timings true
ls-clean-5639-40744-0011: [en] 2.67 s, go decoder, wasm ttft 317.2 ms, 22.4 tok/s, 973 ms in all; native ttft 18.9 ms, 746 tok/s; equal to whistle transcribe: true, JSON without timings true
fleurs-de_de-10308608197319017935: [de] 6.90 s, go decoder, wasm ttft 805.4 ms, 22.4 tok/s, 2653 ms in all; native ttft 27.3 ms, 726.3 tok/s; equal to whistle transcribe: true, JSON without timings true
fleurs-fr_fr-10174413140250646001: [fr] 7.20 s, go decoder, wasm ttft 771 ms, 23.4 tok/s, 2856 ms in all; native ttft 26.6 ms, 784 tok/s; equal to whistle transcribe: true, JSON without timings true
ls-clean-6930-75918-0000 stream: 5 calls with the stop, "Concord returned to its place amidst the tents."; equal to whistle stream --json: true
PASS

Go API

The library is the package github.com/cwbudde/go-whistle/whistle:

go get github.com/cwbudde/go-whistle/whistle

package main

import (
	"fmt"
	"log"

	"github.com/cwbudde/go-whistle/whistle"
)

func main() {
	m, err := whistle.Load("models/whistle/whistle.cact")
	if err != nil {
		log.Fatal(err)
	}
	defer m.Close()

	// One call: a WAV file (or Transcribe on 16 kHz mono float32 samples), at most 30 s.
	res, err := m.TranscribeWAV("testdata/audio/ls-other-3080-5040-0011.wav", whistle.TranscribeOptions{
		Keywords:       []string{"Cowley's"},
		WordTimestamps: true,
	})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("[%s] %s\n", res.Language, res.Text)
	for _, w := range res.Words[:3] {
		fmt.Printf("%s %v-%v p=%.3f\n", w.Text, w.Start, w.End, w.Probability)
	}

	// The engine's JSON, byte for byte.
	js, err := res.MarshalJSON()
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(string(js[:60]))

	// Live audio of any length: about a second per Process call, Stop at the end.
	s, err := m.NewStream(whistle.StreamOptions{Language: "en"})
	if err != nil {
		log.Fatal(err)
	}
	second := make([]float32, whistle.SampleRate) // here: a second of silence
	for range 2 {
		r, err := s.Process(second)
		if err != nil {
			log.Fatal(err)
		}
		fmt.Printf("%.2f s: committed %q, pending %q\n", r.Received, r.Text, r.Pending)
	}
	fmt.Printf("stop: committed %q\n", s.Stop().Text)

	// The encoder output, one row of 512 values per 80 ms frame.
	e, err := m.Embed(second)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(e.Frames, e.Dim)
}

Run from a directory with models/whistle (whistle download) and the clip (see Quickstart), it prints:

[en] Here are some verses of Cowley's. Tell me how you like them.
Here 640ms-800ms p=0.380
are 800ms-880ms p=0.998
some 880ms-1.04s p=1.000
{"text":"Here are some verses of Cowley's. Tell me how you l
1.00 s: committed "", pending ""
2.00 s: committed "", pending ""
stop: committed ""
13 512

The package documentation (go doc github.com/cwbudde/go-whistle/whistle) covers the options, the JSON forms and the streaming rules; whistle/example_test.go has runnable examples.

Parity with the engine

Whistle's engine (libneedle3) is closed source, so this port is rebuilt from the published weights and checked against the engine as a black-box oracle. The engine runs W2A8 numerics, so the goal is equal transcripts, not equal bits. Measured with the default packed weights:

What	Go port	Engine	Details
Transcript and language equal to the engine's, 32 clips	31/32	—	F-031
LibriSpeech test-clean subset, 200 utterances, WER	5.35 %	5.30 %	F-031, F-037
Keyword cases equal to the engine's	45/53	—	F-029, F-037
Depth ladder 2–7 equal to the engine's	161/192	—	F-030
Word start and end within 80 ms of the engine's	97.4 % of 459 words	—	F-033
Embedding vs needle_embed, mean per-frame cosine	0.999827, every clip ≥ 0.999	—	F-027, F-037
Stream: calls with the engine's decision	797/930; final text equal on 98 of 108 runs	—	F-036
Stream at 1 s chunks: final text equal to offline	18/32	18/32	F-036
Result and stream JSON	byte-identical in fields, order and numbers	—	F-027, F-036

Every remaining difference is a near tie under our model, explained in the linked findings.

Limits

• 30 s per call for transcribe, Transcribe and Embed, the engine's limit (ErrTooLong, "audio limit is 30 s"). A stream has no limit; it trims committed audio once its buffer holds more than 15 s.
• Languages: en, de, fr, es, it, nl, pl, detected per call or forced.
• WAV input is integer PCM (8/16/24/32-bit, any channels and rate), as upstream's whistle.py reads it. IEEE float WAVs are rejected; convert them, or pipe raw f32le into whistle stream.
• Speed: slower than the engine, which is hand-tuned C++ with W2A8 kernels; see bench/RESULTS.md for the numbers on the same machine (median TTFT 2.72× the engine's, decode 0.72×) and PLAN.md 12.3 for the GEMM on amd64, which still runs in pure Go.

Why this is a re-implementation, not a translation

In cactus-compute/needle, Whistle is a thin Python wrapper around a closed-source C++ engine. So this port rebuilds the model from three sources:

• the published weights on Hugging Face: whistle.safetensors (FP32) and whistle.cact (2/4-bit quantised)
• the open JAX reference for the shared Needle 3 blocks
• the prebuilt engine, which is used only as a test oracle for encoder embeddings and transcripts

Goals

• No cgo, no Python and no ONNX at runtime
• Load both whistle.safetensors and the compact whistle.cact
• An API and JSON output compatible with upstream needle.transcribe, including language detection, keyword biasing, word timestamps, streaming and embeddings
• Transcripts that match the reference engine on a fixture set, and competitive CPU speed

License

Apache-2.0, the same as the upstream code and weights. Whistle and Needle are by Cactus Compute; this project is not affiliated with them.