sample

README

sample

A row of logits in, a token out.

The chain is llama.cpp's default one, in its order:

1. top-k keeps the k highest logits. With a vocabulary of 262144 and a k of 64, sorting the row would cost more than the draw it serves, so the selection runs through a heap of k entries — the root is the worst kept candidate, and anything that does not beat it is dropped on sight. Only the survivors are sorted.
2. top-p softmaxes the survivors and keeps the shortest prefix whose probabilities reach p, including the token that crosses the threshold. One candidate always survives: an empty distribution has nothing to draw from.
3. temperature divides what is left before the final softmax. Zero or less never reaches this point — Pick returns the highest logit straight away.
4. the draw walks the cumulative probabilities against one float from the generator.

Ties, everywhere, go to the lower identifier, which is what llama.cpp does and what makes a greedy run reproducible.

The generator is math/rand/v2's PCG, seeded from Params.Seed. A sampler owns it, so two samplers in the same process do not disturb each other and a seed plus a sequence of logit rows determines the tokens exactly.

s := sample.New(sample.Params{Temperature: 1, TopK: 64, TopP: 0.95, Seed: 7})
id := s.Pick(logits)

Defaults() returns the values Gemma 4's file declares for itself; gemma reads that file's own general.sampling.* keys and falls back to them.

What is not here: repetition and presence penalties, min-p, typical-p, locally typical sampling, mirostat, grammars. They are additions to this chain rather than changes to it, and nothing in golem asks for them yet.

Documentation

Overview

Package sample turns a row of logits into a token.

The chain is the one llama.cpp runs by default, in its order: top-k cuts the row down to the most likely few, top-p cuts it again at a share of the mass, temperature reshapes what is left, and a draw from the resulting distribution names the token. A temperature of zero short-circuits all of it and takes the highest logit.

Nothing here knows what model produced the row.

Index

• func Greedy(logits []float32) int32
• func Softmax(logits []float32) []float32
• type Params
  • func Defaults() Params
• type Sampler
  • func New(p Params) *Sampler
  • func (s *Sampler) Pick(logits []float32) int32

Functions

func Greedy

func Greedy(logits []float32) int32

Greedy is the highest logit. Ties go to the lower identifier, as llama.cpp does.

func Softmax

func Softmax(logits []float32) []float32

Softmax turns logits into probabilities, shifted by the largest so that a row of several hundred does not overflow the exponential.

Types

type Params

type Params struct {
	// Temperature divides the logits before the draw. Zero or less is greedy.
	Temperature float32
	// TopK keeps that many candidates. Zero or less keeps all of them.
	TopK int
	// TopP keeps the shortest prefix of the sorted candidates whose
	// probabilities reach it. One or more keeps all of them.
	TopP float32
	// Seed fixes the run: the same seed over the same logits draws the same
	// tokens.
	Seed uint64
}

func Defaults

func Defaults() Params

Defaults are the values Gemma 4's own file declares under general.sampling.

type Sampler

type Sampler struct {
	// contains filtered or unexported fields
}

func New

func New(p Params) *Sampler

func (*Sampler) Pick

func (s *Sampler) Pick(logits []float32) int32

Pick names a token. The logits are read, never written.