README
¶
pocket-tts-go
An inference engine for Kyutai Pocket TTS
in pure Go: no Python, no cgo, standard library only.
One go build, one binary, one WAV.
go build ./cmd/pocket-tts
./pocket-tts -voice testdata/voices/french_24l/<voice>.safetensors -o hello.wav "Bonjour le monde."
Or as a library:
engine, _ := pockettts.Open(pockettts.Options{
Weights: ".../model.safetensors",
Tokenizer: ".../tokenizer.model",
Language: "french_24l",
})
defer engine.Close()
voice, _ := engine.LoadVoice("testdata/voices/french_24l/<voice>.safetensors")
sound, _ := engine.Synthesize("Bonjour le monde.", voice, nil)
sound is a []float32 at 24 kHz.
Synthesize takes a *Settings, or nil for the model's own values. Build one
from DefaultSettings(lang) and change what you mean to change: the fields are
not interpreted, so a zero is a zero — an EndThreshold of 0 is a real setting,
one that makes the model far more reluctant to declare itself done.
Settings.Frame receives each frame as soon as it is ready, so the sound can be
played during generation. Settings.Ctx, when set, stops the generation as soon
as it is cancelled — at the next frame — and Synthesize then returns the
context's error.
A voice can also be cloned from a recording, without training and without leaving Go:
voice, _ := engine.VoiceFromWAV("someone.wav") // mono, 24 kHz, 20 to 30 seconds
_ = engine.SaveVoice("someone.safetensors", voice)
Encoding is what costs — some four seconds for a recording of twenty-eight, plus two and a half for the transformer to listen to it. Saving the result turns the next start into a file read. Past seventy seconds of recording the model stops mid-sentence without reporting anything, so VoiceFromWAV refuses beyond that.
The weights, the tokenizer and the predefined voices can be found where the
upstream daemon downloads them: Locate(lang.WeightsPath()),
Locate(lang.TokenizerPath()) and Locate(lang.EmbeddingPath(name)) each
return a path in the Hugging Face cache, or the empty string. LocateVoices(lang)
lists the voices that are actually there.
Status
Two models, two answers. On an i7-9700K with eight threads, one sentence, the
voice alba, and the same end-of-speech threshold on both sides:
| golem | pocket_tts, PyTorch | |
|---|---|---|
french_24l — 24 transformer layers |
×2.94 real time | ×1.27 |
english_2026-01 — 6 layers |
×6.81 | ×4.03 |
| cloning: encoding a recording | ×7.72 real time | — |
first sound, french_24l |
175 ms | 320 ms |
first sound, english_2026-01 |
66 ms | 90 ms |
| ready to speak, warm | 30 ms | 3.5 s |
The two speaking figures were ×2.82 and ×6.55 until nn.MatMatBF16Rows learned
to block its rows and columns for Gemma's vision tower. That kernel is shared,
and the six percent it gave this engine was measured on identical work — the
same text and the same seed, both sides producing 14.2 seconds of sound — not
on the benchmark above, whose output length moves with the arithmetic. What the
same change cost is written beside the tolerance in pockettts_test.go: the
worst frame against PyTorch went from 0.159% of the scale to 0.194%, against a
bar of 0.5%.
go test ./pockettts -run xxx -bench 'Synthesis|FirstFrame' -benchtime 8x
python ref/bench_python.py french_24l alba 5
Both sides load the model and prepare the voice before the clock starts, and both throw the first synthesis away. What is timed is what a daemon does all day.
Both columns used to read the other way round on the six-layer model, and what turned them is under Throughput below: the audio decoder was spending its time entering kernels rather than computing in them, and one frame of it went from 17.8 ms to 6.3 ms. Then the barriers went — everything a section produced is now finished inside it — which is the last third. Where the transformer dominates — twenty-four layers, memory-bound matrix-vector products — golem was already ahead and is now more than twice PyTorch. Where it does not, the decoder was the whole difference.
What golem brings either way: a binary of a few megabytes instead of a gigabyte
of environment, thirty milliseconds to first readiness instead of three and a
half seconds, and a distribution that fits in a go install.
The method
No layer is deemed correct until its intermediate activations match PyTorch.
The scripts in ref/ load the real weights, inject a deterministic input, and
write every intermediate quantity into testdata/. The Go tests read these
files back, and no longer need Python.
Those recordings are not versioned here: they are what the model computes, and the model is Kyutai's. The command that writes them is at the bottom of this file, and every test that reads them skips until it has been run — the gaps in the table below are what it produced on this machine.
| stage | maximum gap with PyTorch |
|---|---|
| one transformer layer | 2×10⁻⁷ |
| the 24 layers, on the real voice state | 2×10⁻⁷ |
| flow net | 4×10⁻⁶ |
| audio decoder, eight chained frames | 5×10⁻⁷ |
| tokenizer | identical segmentation on 18 sentences, in both languages |
| end to end, from text to sound | 5×10⁻⁶ on the first frame, 3×10⁻⁴ on the eighth |
Every stage above is checked on both french_24l and english_2026-01.
The end-to-end drift is not a divergence of computation: generation is a loop that feeds back its own output, and the rounding gap accumulates there instead of repeating.
Randomness is treated as an input: the flow net starts from Gaussian noise, so the output is reproducible neither in Go nor in Python. The Python harness writes the noise it drew, the Go test reads it back, and both integrate the same flow.
What the engine does not do
It does not train, and it does not batch. One synthesis at a time, one voice at a time, no quantization and no GPU: the engine is a decoder and nothing else.
Cloning used to be out of scope and is not any more. It cost the Mimi encoder —
half the model, for work that happens once per voice — and that half is in Go
now: VoiceFromWAV reads a mono 24 kHz recording, encodes it, and hands back
the K/V caches of the transformer that listened to it, which is what a voice
file holds. SaveVoice writes one out in the same safetensors format the
Python daemon caches, so a voice encoded here and a voice downloaded from
upstream are the same file. No trip through Python is required for anything.
No voice ships with this repository — the voice states published with Pocket TTS
belong to Kyutai, and redistributing them is not this project's call to make.
Put one under testdata/voices/<language>/, or encode your own; every test that
needs a voice skips when there is none.
Languages
The thirteen models Kyutai ships — six languages, at six or twenty-four
layers — are all supported. Pass -language; the default is french_24l.
./pocket-tts -language english_2026-01 -voice voice.safetensors "Hello world."
Four things actually differ between them, and only four:
| values | |
|---|---|
| depth of the flow_lm transformer | 24 layers or 6 |
; folded into , before synthesis |
french_24l, german, german_24l |
| short inputs padded with spaces | english_2026-01 only |
| frames generated after the end is detected | 8 |
Two config keys look like they should matter and do not.
insert_bos_before_voice is applied when the voice state is built, on the
Python side, so it is already baked into the file this engine reads.
mimi.inner_dim sizes the encoder's downsampling and the speaker projection,
neither of which is on the decoding path.
Parity is checked against PyTorch on two languages — french_24l at 24 layers
and english_2026-01 at 6, which is also the one language needing the padding.
The other ten share their geometry and rules with one of those two, but have not
been run.
Throughput
In autoregressive generation at batch one, a transformer does not do large matrix products: it does a long series of matrix-vector products. Every frame therefore re-reads the whole of the weights. The limiting factor is not compute power, it is memory bandwidth — and on that ground Go is not at a disadvantage, which is what makes the project possible without a line of assembly.
Measured breakdown, per frame (the budget is 80 ms):
| stage | time | what it cannot go below |
|---|---|---|
| flow_lm and flow net, 24 layers, alone | 19 ms | 16 ms — 604 MB of weights, at the 38 GB/s eight cores sustain |
| Mimi decoder, alone | 6.3 ms | ~1 ms — 162 MMAC over eight AVX2 cores |
one frame of french_24l, the two pipelined |
29 ms |
The two floors are floors for something having the machine to itself, which in the pipeline neither has: the isolated times add up to 25 ms and a pipelined frame costs 29, and the four milliseconds between them are the two halves sharing eight cores. The transformer is close to its floor and there is little left in it. The decoder is six times above its own, and that is now a matter of how much of its arithmetic is spent in kernels rather than around them.
go test ./pockettts/internal/mimi -run xxx -bench Frame -benchtime 40x
go test ./pockettts/internal/flowlm -run xxx -bench AdvanceLatent -benchtime 20x
Two things mattered more than all the micro-optimizations put together:
Processing positions in blocks wherever they are available. The audio transformer receives sixteen positions together; processing them one by one made it re-read its twenty-five megabytes of weights sixteen times per frame — more than the whole flow_lm, for a tenth of the computation. The sweet spot is low: with bfloat16 weights, a batch of size L gives L multiply-accumulates per two bytes read, and the balance between memory and compute falls around L≈4. The text prompt benefits from the same treatment.
Finishing what a section produced, inside it. A section ends with every core waiting on the slowest, and what came after one here was always another pass over the same values on a single core: the bias of a projection, the activation over its outputs, the layer scale, the residual, the rotation of the queries and keys and the copy into the cache. Measured on one French sentence, that was 367 ms of the 2.29 s spent outside any section at all, plus a barrier for each of the 1 738 activations that had a section to itself.
They are now folded into the section that computed the values they read.
nn.Linear.ApplyRows, nn.GELURange and nn.MatMatBF16Rows exist for exactly
that: a worker that has computed rows [start, end) applies the bias, the
activation, the scale and the residual to those rows before it lets go. The
projection of the queries, keys and values is cut into heads rather than rows,
because the rotation needs a head whole — so the same worker rotates what it
projected and writes it into the cache. What is left outside the sections is
112 ms, and it is almost all the norms, which need every value of a position
before they can start.
Nothing about the arithmetic changed: the same values in the same order, and the fixtures hold it. The sentence went from ×2.43 to ×2.82 real time, the six-layer model from ×5.66 to ×6.55, and eight cores now give ×2.9 the speed of one where they gave ×2.4.
Pipelining. Generating a latent and decoding it are independent from one frame to the next. Decoding therefore runs on its own goroutine, and the two complement each other — the transformer mostly waits on memory, the decoder mostly on compute.
Making the inner loop long enough to be worth entering. The audio decoder
was seventeen milliseconds a frame for a hundred and sixty-two million
multiply-accumulates, which is more than an order of magnitude above what the
arithmetic costs, and the reason had nothing to do with the arithmetic. Its convolutions accumulate one scalar coefficient along one output
row at a time, which is the right shape when the row is a thousand samples long
and the wrong one at the top of the decoder, where the row is sixteen: the
512-channel input convolution was making 1.8 million kernel calls per frame to
multiply-add sixteen values each. Gathering each output position's window into
one contiguous vector turns the same layer into 57 000 dot products of length
512 — the same arithmetic, a thirtieth of the calls. nn/conv.go chooses
between the two forms per layer, on the length of the row and on whether the
gathered window still fits in cache. The two transposed convolutions at the top
of the decoder take the same treatment, a transposed convolution of stride S
being S ordinary ones interleaved. Attention in the audio transformer had the
same disease in a different place — a Go loop where a kernel belonged, and one
barrier per position where sixteen positions of eight heads make one section.
What is left to gain: the decoder is six times above its compute floor rather than eighteen, which is now the same kind of gap as everything else here rather than an outlier, and the flow_lm remains the memory-bound half — three milliseconds above a floor set by the bus, and nothing to do about that one.
One constraint not to lose sight of
The weights must stay in bfloat16 in memory, with the conversion happening in the kernel. Converting them to float32 once and for all looks simpler but doubles the amount re-read on every frame: 15.8 GB/s would be needed against 16.5 GB/s available. The margin disappears.
Layout
| path | role |
|---|---|
pockettts.go |
the API: open, load a voice, synthesize |
cmd/pocket-tts |
the command line |
internal/transformer |
the causal layer, shared by both models |
internal/flowlm |
the language model, the flow net, the voices |
internal/mimi |
the audio decoder |
internal/text |
text preparation |
internal/reference |
where the fixtures and the weights are found, for the tests |
languages.go |
what differs between the shipped models |
bench_test.go |
what a synthesis costs, end to end |
../tensors, ../nn, ../token/sentencepiece, ../audio |
the shared layer: safetensors and mmap, kernels, the unigram tokenizer, RIFF writing |
testdata/voices |
where a voice state goes, by language; none ship here |
ref/ |
the Python scripts that write the fixtures, and the one that times the reference |
Reproducing the fixtures
From an environment where pocket_tts is installed:
python ref/dump_layer.py <model.safetensors> testdata/layer0
python ref/dump_pipeline.py testdata/voices/french_24l/<voice>.safetensors testdata/pipeline
python ref/dump_pipeline.py testdata/voices/english_2026-01/<voice>.safetensors \
testdata/pipeline_en english_2026-01 "Hello world."
python ref/dump_tokens.py <french_24l/tokenizer.model> testdata/tokenizer/cases.json french
python ref/dump_tokens.py <tokenizer.model> testdata/tokenizer/cases_en.json english
The same environment runs ref/bench_python.py, which is what the table at the
top of this file compares golem against.
Nothing above is versioned here. The voice states go under
testdata/voices/<language>/, and the predefined ones can be fetched from
kyutai/pocket-tts-without-voice-cloning, which serves them already encoded.
The weights and the tokenizer are 672 MB and have no place in a git history
either — so every test that needs one of them skips cleanly when it cannot find
it; POCKET_TTS_WEIGHTS, POCKET_TTS_TOKENIZER and
POCKET_TTS_VOICE say where to look.
Documentation
¶
Overview ¶
Package pockettts synthesizes speech from text with the Kyutai Pocket TTS models, in pure Go.
The engine loads the weights once — the file is memory-mapped, nothing is copied — then synthesizes as many times as asked. A voice is a precomputed state, produced by the reference Python daemon from a sound excerpt; the Go engine reads it back and therefore has no need for the audio encoder.
engine, err := pockettts.Open(pockettts.Options{
Weights: ".../model.safetensors",
Tokenizer: ".../tokenizer.model",
Language: "french_24l",
})
voice, err := engine.LoadVoice(".../voice.safetensors")
sound, err := engine.Synthesize("Bonjour le monde.", voice, nil)
Index ¶
- Constants
- func Languages() []string
- func Locate(relative string) string
- func LocateVoices(l Language) []string
- type Engine
- func (m *Engine) Close() error
- func (m *Engine) Language() Language
- func (m *Engine) LoadVoice(path string) (*Voice, error)
- func (m *Engine) SaveVoice(path string, v *Voice) error
- func (m *Engine) Synthesize(t string, voice *Voice, r *Settings) ([]float32, error)
- func (m *Engine) VoiceFromSamples(samples []float32) (*Voice, error)
- func (m *Engine) VoiceFromWAV(path string) (*Voice, error)
- type Language
- type Options
- type Settings
- type Voice
Constants ¶
const DefaultLanguage = "french_24l"
DefaultLanguage is the one the engine uses when none is named.
const FramesPerSecond = 12.5
FramesPerSecond is the frame rate of the model.
const MaxCloneSeconds = 70
MaxCloneSeconds is the longest recording worth listening to.
The model was not trained on longer ones: past about seventy seconds it emits its end almost immediately and produces a few seconds of sound instead of the whole text, without any error to say so. Twenty to thirty seconds is enough to carry a voice, and is what the reference tooling recommends.
const SampleRate = 24000
SampleRate is the sampling rate of the sound produced.
Variables ¶
This section is empty.
Functions ¶
func Locate ¶ added in v0.2.0
Locate returns the absolute path of relative inside the Hugging Face cache, or the empty string when it is not there. relative is what WeightsPath, TokenizerPath or EmbeddingPath returned.
A missing file is not an error here: the caller knows what it was looking for and can say so better than this function can.
func LocateVoices ¶ added in v0.2.0
LocateVoices lists the predefined voices available for a language, by name and in order. An empty result means the model was never downloaded, or was downloaded without its voices.
Types ¶
type Engine ¶
type Engine struct {
// contains filtered or unexported fields
}
Engine holds the loaded weights. It is not safe for concurrent use: generation advances internal state. To synthesize in parallel, open one engine per goroutine — the weights being memory-mapped, the cost is that of the buffers, not that of the model.
func (*Engine) SaveVoice ¶ added in v0.3.0
SaveVoice writes a voice where LoadVoice can read it back, in the format the reference daemon wrote. Encoding a recording costs seconds; reading the file back costs milliseconds.
func (*Engine) Synthesize ¶
Synthesize returns the samples of the text, in [-1, 1], at 24 kHz.
func (*Engine) VoiceFromSamples ¶ added in v0.3.0
VoiceFromSamples is VoiceFromWAV for a recording already in memory, in [-1, 1] at SampleRate.
func (*Engine) VoiceFromWAV ¶ added in v0.3.0
VoiceFromWAV encodes a recording and returns the voice it holds.
The file must be at the model's sampling rate. Resampling is not done here: it is a choice with audible consequences, and a caller who has to make it should make it knowingly rather than have it made in passing.
type Language ¶
type Language struct {
Name string
// Layers is the depth of the flow_lm transformer.
Layers int
// RemoveSemicolons folds ";" into "," before synthesis. The models trained
// with it hesitate on a semicolon they never saw.
RemoveSemicolons bool
// PadShortInputs prefixes eight spaces to inputs of fewer than five words.
// The model generates poorly on very few tokens, and the padding buys it
// some.
PadShortInputs bool
// FramesAfterEnd is how many frames to keep generating once the end is
// detected, so the sentence can settle.
FramesAfterEnd int
}
Language holds what the engine needs to know about one language.
func LookupLanguage ¶
LookupLanguage returns a language by name.
func (Language) EmbeddingPath ¶ added in v0.2.0
EmbeddingPath is where a predefined voice sits inside a Pocket TTS snapshot. The file is the voice state the model starts from, already computed by Kyutai; nothing here encodes a voice from sound.
func (Language) TokenizerPath ¶
func (Language) WeightsPath ¶
WeightsPath is where the language's weights sit inside a Pocket TTS snapshot, and TokenizerPath where its tokenizer does. The tokenizer of english_2026-01 lives at the root of the repository rather than under its language directory — the one irregularity in the layout.
type Options ¶
type Options struct {
Weights string // model.safetensors of the language
Tokenizer string // tokenizer.model
// Language names which of the shipped models the weights belong to. It
// decides the depth of the transformer and the text rules; getting it wrong
// fails at load time rather than producing a wrong voice. Empty means
// DefaultLanguage.
Language string
}
Options describes what the engine needs in order to start.
type Settings ¶
type Settings struct {
Temperature float64 // the variance of the starting noise
EndThreshold float64 // beyond it, the model declares itself done
FramesAfterEnd int // enough to let the sentence settle
MaxTokens int // size of a segment
Seed uint64 // 0 for a random draw, which is a value and not a default
// Frame, if provided, receives each frame of samples as soon as it is
// ready. That is what allows the sound to be played during generation.
Frame func([]float32)
// Ctx, when set, stops the generation as soon as it is cancelled: at the
// next frame within a segment, and between two segments. Synthesize then
// returns the context's error, and the sound it had produced is dropped —
// a caller that cancels has already had those frames, through Frame, and
// has its own reason to stop.
Ctx context.Context
// contains filtered or unexported fields
}
Settings tunes the generation. Start from DefaultSettings and change what you mean to change: no field is interpreted, and a zero is a zero.
func DefaultSettings ¶ added in v0.2.0
DefaultSettings returns the model's own values, which are those of the reference daemon. Callers start from it and change what they mean to change.
The settings carry no zero-means-default rule: a caller that sets a field to zero gets zero. An end threshold of 0 is a legitimate setting — it makes the model far more reluctant to declare itself done — and the earlier rule turned it into -4 silently.
Source Files
Directories