README
¶
aitypo
Packages urlinsane's typo generators as learnable tasks: an oracle, a corpus, splits, and exact scoring.
d := aitypo.Data{Vowels: en.Vowels(), Homoglyphs: en.Homoglyphs(), Keyboard: rows}
tasks, _ := aitypo.Tasks(d).Select("co", "cs", "hr")
ex := aitypo.Emit(tasks, corpus, "datasets/domains/domain.lst")
aitypo.Assign(ex, aitypo.DefaultRatio, "")
aitypo.WriteJSONL(os.Stdout, ex)
{"task":"co","input":"google.com","expect":["gogle.com","googe.com",...],"split":"train"}
Why these functions are trainable
Every function in pkg/typo is a total, deterministic map from a name to a
set of names. CharacterOmission("google") is exactly
{gogle, googe, googl, goole, oogle} and nothing else, forever.
That is unusual, and it is the whole basis of this package. A task with an exact oracle needs no annotation, cannot disagree with itself, generates unlimited labelled data, and can be scored exactly rather than by a similarity metric someone picked.
The distinction that matters: Needs
Not every task asks a model the same kind of question.
Needs |
Tasks | What success means |
|---|---|---|
NeedsNothing |
co cs cr hi ho di do dhs tos bf |
The model learned a rule — delete one character, transpose two, insert at every gap |
NeedsLanguage |
hr hs cm vs gi gr cns ons sp |
The model memorised a table — this script's homoglyphs, this language's misspellings |
NeedsKeyboard |
acs aci rar |
The model memorised a layout |
Reporting one accuracy across both would average a rule a model learned with a lookup it memorised. For the table tasks the interesting experiment is generalisation off the table — homoglyphs for a script the data does not carry.
sp looks like a rule and is not. typo.SingularPluralise goes through
nlp.NewClient, so mouse/mice, child/children, person/people, datum/data and
the uncountables are dictionary entries, derivable from the input by nothing.
It sat under NeedsNothing until that was noticed, which credited a model with
a rule it had memorised as an English dictionary.
It is also the one table not gated on a Data field, because its table is
compiled into the library rather than supplied — so unlike every other
language task it is not silenced on a non-English corpus, and
sp("maison") is ["maisons"]. Read sp on non-English input as English
morphology applied to foreign strings.
bf is a rule but not a rule about characters: it models a bit flipping in a
resolver's memory, so its output is byte-level and may not be valid UTF-8.
Learnable; not evidence about plausible human typing.
Scoring is strict on purpose
Result.Exact() is the headline: the prediction must be the expected set, whole.
A generator's contract is the set — omission of google is five names, not four
of them and something plausible — so four right and one invented has not learned
the rule. Precision, Recall and F1 are there to say how it failed.
Both sides are normalised the same way (sorted, deduplicated, input dropped), so a model cannot inflate recall by repeating itself or echoing the input.
Summarize errors on a result naming a task the registry does not have, rather
than labelling it. Registry is a map, so a miss yields the zero Task and its
Needs is NeedsNothing — the zero value of the enum that separates a learned
rule from a memorised table. Scoring a corpus against Tasks(Data{}), which
deregisters every language and keyboard task, reported a homoglyph table and a
keyboard layout as learned rules at a perfect 1.000.
Summarize is macro-averaged: every input is one vote. Micro-averaging would
weight an input by how many variants it happens to produce, and the score would
mostly measure the corpus's length distribution.
Splits do not leak
Assign partitions by input name, not by example. One name yields an example
per task and those share almost all their structure, so putting google in
train for omission and in test for transposition measures memorisation of the
name rather than of the rule.
It is a stable hash of the name, not a shuffle:
- the same name lands in the same split on every machine and every run;
- adding names never moves the ones already there, so a corpus can grow without invalidating every earlier measurement.
Leakage asserts the property rather than trusting it — concatenating two
corpora, or splitting with two salts, breaks it silently, and the symptom is a
test score that looks unusually good.
It wraps pkg/typo, it does not copy it
This package began as a copied directory, and that is worth recording. A copy of
a generator set is the exact defect this repository spent a day retiring: acs,
aci and rar had been fixed rune-safe inside their own plugin directories
while eight sibling generators kept slicing bytes, and the split was the bug.
Four minutes after the copy was taken it was already two commits behind.
There is one implementation, in pkg/typo. Every oracle here calls it. A task
registry that reimplemented a generator would train a model against a second
definition of the truth.
What cannot be written
bf models a bit flipping in a resolver's memory, so it flips bits inside
multi-byte runes: bf("münchen") returns 16 byte sequences out of 64 that are
not text at all.
JSON strings are UTF-8 by definition and encoding/json coerces invalid input
to U+FFFD without erroring. Written anyway, those 16 come back changed, and
scoring the corpus against its own oracle then grades a model that reproduced it
exactly at 0.750 precision — with no error anywhere.
So WriteJSONL refuses, naming the task and the input — checking every example
before writing any, because bufio.Writer flushes at 4 KB and a refusal
partway through used to leave a half-encoded record on disk. It checks Source
too: a path need not be valid UTF-8 on Linux, and provenance that comes back
changed is a corpus lying about where it came from. Filter with
Representable if some examples are expected to be unwritable, or keep such a
corpus to ASCII inputs. bf over ASCII is fine and is not banned.
Cost
Generation is cheap per call and large in aggregate. 300 domains over 8 tasks is
2,382 examples and 50,682 output strings; the whole 17-task set over 2,000
domains does not finish in two minutes. Stats.Variants — the total size of all
expectation sets — is the figure that predicts training cost, not the example
count.
What this package is not
It trains nothing and imports no ML machinery. internal/model is the learner —
an HMM with Baum-Welch and dag-cbor model artifacts — and graph.BeliefModel is
where a learned function plugs back into the scanner. This package only builds
the data and scores the answers.
Documentation
¶
Overview ¶
Package aitypo packages urlinsane's typo generators as learnable tasks.
Every function in pkg/typo is a total, deterministic map from a name to a set of names: CharacterOmission("google") is exactly {gogle, googe, googl, goole, oogle} and nothing else, forever. That property is unusual and it is what makes these functions trainable. A task with an exact oracle needs no annotation, cannot disagree with itself, generates unlimited labelled data, and can be scored exactly rather than by a similarity metric someone chose.
So this package is not a model and trains nothing. It is the harness around the functions:
Task one generator, with its oracle and what it needs to run Corpus the names to run it over — the shipped datasets, or your own Example one (input, expected-set) pair, with a split assigned Score a model's predicted set against the oracle's, per task
It wraps pkg/typo, it does not copy it ¶
This package began as a copied directory. That is worth naming, because a copy of a generator set is the exact defect this repository spent a day retiring: acs, aci and rar had been fixed rune-safe in their own plugin directories while eight sibling generators kept slicing bytes, and the split was the bug. Four minutes after the copy was taken it was already two commits behind. There is one implementation, in pkg/typo, and everything here calls it.
What a model is actually being asked to learn ¶
Task.Needs is the interesting field, not the id. A generator with Needs == NeedsNothing encodes a rule — delete one character, transpose two, insert a hyphen at every gap — and a model that gets it right has learned the rule. A generator with NeedsLanguage or NeedsKeyboard is a table lookup: homoglyphs for a script, the neighbours of a key on a layout. A model can only reproduce those by memorising the table, so success there means something different, and generalisation to a script the table does not carry is the real test.
Splitting the two is why Needs exists. Reporting one accuracy number across both would average a rule a model learned with a table it memorised.
Index ¶
- Constants
- Variables
- func Assign(examples []Example, r Ratio, salt string)
- func Leakage(examples []Example) []string
- func Representable(e Example) bool
- func WriteJSONL(w io.Writer, examples []Example) error
- type Data
- type Example
- type Needs
- type Oracle
- type Ratio
- type Registry
- type Result
- type Stats
- type Summary
- type Task
Constants ¶
const ( SplitTrain = "train" SplitVal = "val" SplitTest = "test" )
Split names the three partitions.
Variables ¶
var DefaultRatio = Ratio{Train: 0.8, Val: 0.1, Test: 0.1}
DefaultRatio is 80/10/10.
Functions ¶
func Assign ¶
Assign labels every example with a split, grouped by input name.
Deterministic and stateless: the partition depends only on the name and the salt, so two machines building the same corpus agree, and adding a name never moves an existing one. A shuffle-and-slice would move every name each time the corpus grew, quietly invalidating every earlier measurement.
salt lets one corpus be re-split independently of another — pass the run or experiment name. An empty salt is fine and gives the canonical split.
func Leakage ¶
Leakage reports inputs that appear in more than one split.
It should always return nothing, because Assign groups by input. It exists because that is a property worth asserting rather than trusting: a corpus built by concatenating two files, or split by two different salts, breaks it silently, and the symptom is a test score that looks unusually good.
func Representable ¶
Representable reports whether an example survives a JSONL round trip.
JSON strings are UTF-8 by definition, and encoding/json silently coerces invalid input to U+FFFD rather than refusing it. Most generators cannot produce invalid UTF-8, but bf can and does: it models a bit flipping in a resolver's memory, so it flips bits inside multi-byte runes, and bf("münchen") returns 16 byte sequences out of 64 that are not text.
Written anyway, those 16 come back changed, and scoring the corpus against its own oracle then reports a model that reproduced it *exactly* as 75% precise. That is the failure this package is least able to detect from the outside, because nothing errors and the numbers look plausible.
func WriteJSONL ¶
WriteJSONL writes examples one per line.
JSONL because a corpus is appended to, streamed, and reviewed in a diff, and because every training stack reads it. One example per line means a corpus can be shuffled with sort -R, split with head, and inspected with grep without a parser.
Examples are written in the order given. Emit produces them in corpus order and task order, so a corpus built twice from the same inputs is byte identical — which is what lets a training run be identified by the hash of its data.
Types ¶
type Data ¶
type Data struct {
// Graphemes is the alphabet for insertion and replacement.
Graphemes []string
// Vowels feeds vowel swapping.
Vowels []string
// Homoglyphs maps a character to the characters that look like it.
Homoglyphs map[string][]string
// Homophones are groups of words that sound alike. Cross-language groups
// go here too — the generator does not care which it is given, and the
// difference is worth recording in the corpus Source rather than in a
// second task.
Homophones [][]string
// Misspellings are groups whose members are habitually confused.
Misspellings [][]string
// Numerals maps a digit to the words denoting it.
Numerals map[string][]string
// Keyboard is a layout as rows of characters, the shape
// pkg/typo's adjacency helpers expect.
Keyboard []string
}
Data is the reference data the table-driven tasks read.
Every field is optional and a nil one silences its tasks rather than erroring: a corpus for the pure rules should not require a language database to exist. Tasks whose data is missing generate empty expectations, which Emit drops, so an absent table shows up as a task with no examples rather than as a task that appears to have been learned perfectly.
The shapes match pkg/typo's parameters exactly so nothing has to be adapted here; fill them from internal/dataset, from pkg/kb, or by hand in a test.
type Example ¶
type Example struct {
Task string `json:"task"`
Input string `json:"input"`
Expect []string `json:"expect"`
Split string `json:"split,omitempty"`
// Source names where the input came from — a dataset path, a scan, a
// hand-written case. A corpus that cannot say what it is made of cannot be
// reproduced.
Source string `json:"source,omitempty"`
}
Example is one training record: an input and the complete set of outputs the oracle produces for it.
The whole set, not one pair. These generators are set-valued — omission of "google" is five names — and a corpus of (input, one-output) rows would teach a model to produce one variant and give it no way to learn when to stop.
func Emit ¶
Emit runs each task over each input and returns one example per pair that produced anything.
Inputs that produce nothing are dropped rather than written with an empty expectation. An empty set is a real answer — "example" has no doubled characters, so rar produces nothing — but a corpus of mostly-empty rows trains a model to answer nothing, and the honest place to teach "sometimes the answer is empty" is a deliberate sample rather than an accident of which names the corpus happened to hold. EmitWithEmpty is that deliberate sample.
func EmitWithEmpty ¶
EmitWithEmpty keeps the pairs whose expectation is empty.
func ReadJSONL ¶
ReadJSONL reads a corpus back.
A malformed line is an error naming the line number rather than a skipped record. A corpus that silently drops rows trains on less data than it reports, and the report is what an experiment is judged on.
type Needs ¶
type Needs uint8
Needs says what data a task's oracle reads beyond its input.
It is a property of the generator, not of a run: CharacterOmission reads nothing whatever you pass it, and HomoglyphSwapping is a table lookup even when the table is empty.
const ( // NeedsNothing is a pure string rule. The oracle is a function of the // input alone, so a model that reproduces it has learned the rule. NeedsNothing Needs = iota // NeedsLanguage reads a language's vocabulary: homoglyphs, homophones, // misspellings, vowels, graphemes, numerals. NeedsLanguage // NeedsKeyboard reads a layout's key adjacency. NeedsKeyboard )
type Oracle ¶
Oracle is a generator: a total, deterministic map from one name to the set of names it produces.
Total and deterministic are load-bearing. Every scoring number in this package assumes calling an oracle twice with the same input gives the same set, so a generator that read a clock, ranged a map, or depended on registration order would make an experiment unrepeatable without failing anywhere visible.
type Ratio ¶
type Ratio struct{ Train, Val, Test float64 }
Ratio is the fraction of *groups* — not examples — in each partition.
type Registry ¶
Registry is a set of tasks, addressable by id.
func Tasks ¶
Tasks builds the registry from d.
Every entry wraps a pkg/typo function directly. Nothing is reimplemented here, and nothing should be: the oracle has to be the same code the scanner runs, or a model trained to match this package would be trained against a second definition of the truth.
type Result ¶
type Result struct {
Task string
Input string
Expect []string
Predict []string
Hit []string // predicted and expected
Missed []string // expected, not predicted
Spuri []string // predicted, not expected
}
Result is one prediction scored against its oracle.
Both counts and the sets are kept. The counts are what a training loop reads; the sets are what a human reads when the number is disappointing, and losing them means re-running the model to find out what it actually said.
func Score ¶
Score compares one prediction against a task's oracle.
The prediction is deduplicated and the input itself is dropped from it before comparison, exactly as Expect does to the oracle's output. Otherwise a model could inflate recall by emitting the input and every variant twice, and the two sides would be measured under different rules.
func (Result) Exact ¶
Exact reports whether the prediction is the expected set exactly.
The headline metric for these tasks, and a strict one on purpose. A generator's contract is the whole set — omission of "google" is five names, not four of them and something plausible — so a model that gets four right and invents a fifth has not learned the rule. Precision and recall are there to say how it failed.
type Stats ¶
type Stats struct {
Examples int
Inputs int
Tasks int
Splits map[string]int
ByTask map[string]int
// Variants is the total size of all expectation sets — the number of
// output strings a model has to produce across the corpus, which is the
// figure that predicts training cost, not the example count.
Variants int
}
Stats is what a corpus is made of, for the line a training run should print before it starts.
type Summary ¶
type Summary struct {
Task string
Needs Needs
N int
ExactMatch float64 // fraction of inputs answered exactly
Precision float64 // macro-averaged over inputs
Recall float64
F1 float64
}
Summary aggregates results for one task.
func Summarize ¶
Summarize aggregates per task, macro-averaged over inputs.
It takes the Registry rather than a needs map, and errors on a result naming a task the registry does not have. Both halves are needed and only the first was here: Registry is itself a map, so reg[id] on a miss returned the zero Task, whose Needs is NeedsNothing — exactly the silent mislabelling the type was supposed to prevent.
It is reachable in normal use. Building a corpus with language and keyboard data and then scoring it against a registry built without them — which is what Tasks(Data{}) gives, and it deregisters hr, hs, acs, aci, vs and gi — reported a memorised homoglyph table and a memorised keyboard layout as learned rules, at a perfect 1.000. That is the one distinction this package exists to keep, so it is an error rather than a footnote.
Macro, not micro: micro-averaging would weight an input by how many variants it happens to produce, so a ten-character name would count ten times as much as a three-character one and the score would mostly measure the corpus's length distribution. Every input is one vote.
Results are grouped by task and returned in task order, so two runs of the same evaluation print identically.
type Task ¶
type Task struct {
// ID matches the algorithm id the scanner uses — co, cs, hr — so a result
// here names the same thing a scan does.
ID string
Title string
Needs Needs
// Oracle is the reference implementation. It is the label source, the
// evaluation target, and the definition of correct.
Oracle Oracle
}
Task is one generator, packaged for training.
Source Files