lang

Documentation

Overview

Package lang provides a tiny, pure-Go character bigram language model used as a prior over candidate strings. Image distance alone cannot separate visually near-identical candidates (especially behind heavy blur); a language prior breaks those ties toward plausible text, and flags implausible "recoveries".

The model is a Laplace-smoothed character bigram trained at init from a small embedded corpus of English prose and code. It is deliberately lightweight (no CGO, no large model): it ranks, it does not generate.

Index

• Constants
• func CorpusFor(l Language) string
• func DictionaryPrior() func(string) float64
• func DictionaryScore(s string) float64
• func FreqWeight(word string) float64
• func FreqWeightEN(word string) float64
• func InfiniPrior() func(string) float64
• func PriorFor(l Language) func(string) float64
• func WeightedScoreEN(s string) float64
• func WeightedScoreFR(s string) float64
• type Dict
  • func Dictionary() *Dict
  • func DictionaryFor(l Language) *Dict
  • func FrenchDictionary() *Dict
  • func (d *Dict) All() func(yield func(string) bool)
  • func (d *Dict) ByRuneLen(n int) []string
  • func (d *Dict) Contains(word string) bool
  • func (d *Dict) Score(s string) float64
• type Infini
  • func DefaultFrench() *Infini
  • func InfiniDefaultEnglish() *Infini
  • func InfiniFor(l Language) *Infini
  • func NewInfini(text string) *Infini
  • func (m *Infini) Score(s string) float64
• type Language
  • func ParseLanguage(s string) (Language, bool)
  • func (l Language) String() string
• type Model
  • func Default() *Model
  • func ModelFor(l Language) *Model
  • func New(text string) *Model
  • func (m *Model) Score(s string) float64
  • func (m *Model) TransitionLogProb(prev, next rune) float64

Constants

const BonusWord = 1.0

BonusWord is the plausibility bonus awarded per token that is a known dictionary word. The score returned by DictionaryScore is the mean of per-token bonuses (BonusWord when the token is in the dictionary, 0 otherwise), so it lies in [0, BonusWord] and never penalises a candidate.

The value 1.0 is chosen to be on the same order of magnitude as the character-bigram scores used by Model.Score, so that the two priors compose cleanly when summed via unpixel.WithPriors.

Functions

func CorpusFor

func CorpusFor(l Language) string

CorpusFor returns the raw embedded corpus text for l. It is provided so callers that need to build a fresh (non-shared) Model or Infini for concurrent use can do so without importing the corpus as a separate dependency.

func DictionaryPrior

func DictionaryPrior() func(string) float64

DictionaryPrior returns a plausibility scorer backed by the embedded English word list, ready for use with unpixel.WithPriors. The returned function is DictionaryScore expressed as a closure, making it trivial to compose with other priors:

res, _ := unpixel.Recover(ctx, img,
    unpixel.WithPriors(defaults.LanguageModel(), lang.DictionaryPrior()),
)

The returned scorer awards BonusWord per known dictionary word (mean over tokens), returning 0 for empty input. It never returns a negative value.

func DictionaryScore

func DictionaryScore(s string) float64

DictionaryScore scores s against the shared English dictionary. It is a convenience wrapper around Dictionary().Score(s); see Dict.Score for full semantics.

func FreqWeight

func FreqWeight(word string) float64

FreqWeight returns the Zipfian frequency weight for word in the French frequency list (exact match, accents preserved, no case folding). The return value is in [0, 1]:

• Ranked word at position r: (log(F+1)−log(r)) / log(F+1)
• In-dict but unranked: baseFreqWeight (0.15)
• OOV (not in either list): 0

FreqWeight is safe for concurrent use after the first call.

func FreqWeightEN

func FreqWeightEN(word string) float64

FreqWeightEN returns the Zipfian frequency weight for word in the English frequency list (exact match, lowercase, no case folding). The return value is in [0, 1]:

• Ranked word at position r: (log(F+1)−log(r)) / log(F+1)
• In-dict but unranked: baseFreqWeight (0.15)
• OOV (not in either list): 0

FreqWeightEN is safe for concurrent use after the first call.

func InfiniPrior

func InfiniPrior() func(string) float64

InfiniPrior returns a prior closure (Score) over the default French corpus, ready for use with unpixel.WithPriors:

res, _ := unpixel.Recover(ctx, img,
    unpixel.WithPriors(lang.InfiniPrior()),
)

func PriorFor

func PriorFor(l Language) func(string) float64

PriorFor returns a plausibility scorer for l: a fusion of a Zipfian frequency-weighted dictionary score and the variable-order character infini-gram, so common in-vocabulary words rank above rare ones and out-of-vocabulary strings still get graceful char-level scoring. Higher is more plausible. The returned function is safe to pass directly as unpixel.WithLanguageModel (or composed via unpixel.WithPriors).

Each call to PriorFor creates a dedicated Infini instance (not the shared singleton) so the returned closure can be called concurrently without races. The Infini cache inside the closure is private and is NOT shared.

Fusion:

score(s) = wDict*weightedScore(s) + wChar*infini.Score(s)

where wDict=0.5 and wChar=1.0. weightedScore is WeightedScoreEN for English and WeightedScoreFR for French: both apply the Zipfian rank→weight model from the embedded frequency lists so that "the"/"de" outrank equal-length rare words. The char-gram term provides language discrimination for OOV strings and distinguishes word-order permutations.

func WeightedScoreEN

func WeightedScoreEN(s string) float64

WeightedScoreEN scores s using the Zipfian frequency weight for each whitespace token. It returns the mean FreqWeightEN over all tokens (0 for empty input). Common words (e.g. "the", "and") outrank rare in-dict words.

func WeightedScoreFR

func WeightedScoreFR(s string) float64

WeightedScoreFR scores s using the Zipfian frequency weight for each whitespace token. It returns the mean FreqWeight over all tokens (0 for empty input). It is the frequency-aware counterpart to Dict.Score: instead of a flat BonusWord=1.0 per known token, each token contributes its FreqWeight so common words outrank rare equal-length words.

Types

type Dict

type Dict struct {
	// contains filtered or unexported fields
}

Dict is a set of known English words used as a plausibility prior. The zero value is not usable; obtain one with Dictionary.

func Dictionary

func Dictionary() *Dict

Dictionary returns the shared Dict built from the embedded word list. It is initialised exactly once (sync.Once) and is safe for concurrent use.

func DictionaryFor

func DictionaryFor(l Language) *Dict

DictionaryFor returns the appropriate Dict for l. DictionaryFor(English) == Dictionary(); DictionaryFor(French) == FrenchDictionary().

func FrenchDictionary

func FrenchDictionary() *Dict

FrenchDictionary returns the shared Dict built from the embedded French word list. Words preserve correct accents ("connaît", "liberté", "égalité"…). It is initialised exactly once and is safe for concurrent use. Contains uses the word as given — the list is entirely lowercase, so callers wanting case-insensitive lookup should fold with strings.ToLower first.

func (*Dict) All

func (d *Dict) All() func(yield func(string) bool)

All returns an iterator over every word in the dictionary in unspecified order. The iterator is safe to call multiple times; each call returns a fresh traversal.

func (*Dict) ByRuneLen

func (d *Dict) ByRuneLen(n int) []string

ByRuneLen returns all words in the dictionary whose rune length is exactly n. The result slice is cached after the first call for a given n; subsequent calls for the same n return the identical slice. An n that matches no words returns nil.

func (*Dict) Contains

func (d *Dict) Contains(word string) bool

Contains reports whether word (as given, no case folding) is in the dictionary. The embedded word list is lowercase, so callers that want case-insensitive lookup should fold with strings.ToLower first.

func (*Dict) Score

func (d *Dict) Score(s string) float64

Score scores s as a plausibility prior based on whole-word dictionary membership. It splits s on whitespace into tokens and returns the mean per-token bonus: BonusWord for each token that is a known dictionary word (after lowercasing), 0 otherwise. Single-character tokens that are valid words ("a", "i") are counted; other single-character tokens score 0.

The return value is in [0, BonusWord]. An empty string or a string with no tokens returns 0, so the prior never penalises a candidate — it only rewards recognisable text.

type Infini

type Infini struct {
	// contains filtered or unexported fields
}

Infini is a variable-order character language model backed by a suffix array over an embedded corpus: at each byte position it scores the next byte using the longest preceding context that still occurs in the corpus (infini-gram backoff), so it captures long-range structure where the data supports it and falls back gracefully where it does not. Unlike Model it preserves non-ASCII bytes, so it models accented text (e.g. French "connaît") correctly.

Inspiration: nathan-barry/tiny-infini-gram (infini-gram via Go's index/suffixarray). The standard-library suffix array (index/suffixarray) provides Lookup(-1) which returns all match positions in O(log n + k) time, so count(sub) = len(sa.Lookup(sub, -1)) with no external dependencies.

The zero value is not usable; use NewInfini or DefaultFrench.

func DefaultFrench

func DefaultFrench() *Infini

DefaultFrench returns the shared Infini model trained on the embedded French corpus. It is initialised exactly once (sync.Once). Not safe for concurrent Score calls — matches the Model contract.

func InfiniDefaultEnglish

func InfiniDefaultEnglish() *Infini

InfiniDefaultEnglish returns the shared Infini model trained on the embedded English corpus (corpus.txt). It is initialised exactly once. Not safe for concurrent Score calls — matches the Model and DefaultFrench contracts.

func InfiniFor

func InfiniFor(l Language) *Infini

InfiniFor returns the Infini model for the given language: InfiniFor(English) == InfiniDefaultEnglish(); InfiniFor(French) == DefaultFrench().

func NewInfini

func NewInfini(text string) *Infini

NewInfini builds an Infini model over text (lowercased, UTF-8 preserved so accented bytes survive as multi-byte sequences).

func (*Infini) Score

func (m *Infini) Score(s string) float64

Score returns the mean per-byte log-probability of s (higher = more plausible text). An empty string returns the floor log-prob without panicking. Case is folded to lower before scoring.

type Language

type Language int

Language identifies a supported natural language.

const (
	// English selects English word lists and character models.
	English Language = iota
	// French selects French word lists and character models (accent-aware).
	French
)

func ParseLanguage

func ParseLanguage(s string) (Language, bool)

ParseLanguage parses a language name or code (case-insensitive). Recognised values: "en", "english" for English; "fr", "french", "français" for French. The second return value reports whether the input was recognised.

func (Language) String

func (l Language) String() string

String returns the ISO 639-1 two-letter code for the language ("en" / "fr").

type Model

type Model struct {
	// contains filtered or unexported fields
}

Model is a character bigram language model. The zero value is not usable; use Default or New.

func Default

func Default() *Model

Default returns the shared model trained on the embedded corpus.

func ModelFor

func ModelFor(l Language) *Model

ModelFor returns the shared bigram Model for l. ModelFor(English) is equivalent to Default; ModelFor(French) returns a bigram model trained on the embedded French corpus (corpus_fr.txt). The returned model is safe for concurrent read-only use (Score, TransitionLogProb).

func New

func New(text string) *Model

New trains a bigram model on text.

func (*Model) Score

func (m *Model) Score(s string) float64

Score returns the mean per-character log-probability of s (higher = more plausible text). An empty string scores the unseen floor. Case is ignored.

func (*Model) TransitionLogProb

func (m *Model) TransitionLogProb(prev, next rune) float64

TransitionLogProb returns log P(next|prev) under the bigram model, applying the same ASCII-clamp and smoothing/backoff that Score uses: the observed bigram log-prob if present, the unigram log-prob minus 1 (observed character, unseen context), or unseen() for a completely unseen character.

It emits the exact per-edge factor summed by Score: for ASCII lowercase input with a ' ' start context, summing TransitionLogProb(' ', s[0]), TransitionLogProb(s[0], s[1]), … equals Score(s) × len(s).