textmatch

Documentation

Overview

Package textmatch is the small lexical toolkit two subsystems needed at once.

It exists because comparing "is this sentence about the same thing as that one" turned up in two unrelated places — deciding whether a recalled value supersedes another, and deciding whether a proposed approach is one somebody already ruled out — and neither should have to import the other to ask.

Everything here is deliberately shallow. There is no stemmer, no tokenizer, no model. The judgements it supports are all of the form "close enough to be worth a human's attention", where a near-miss costs a wasted glance and a false confidence costs an afternoon.

Index

• Constants
• Variables
• func Akin(a, b string) bool
• func DifferentSubjects(a, b string) bool
• func DifferingValues(a, b string) bool
• func Flatten(s string) string
• func Negated(s string) bool
• func Overlap(a, b map[string]bool) float64
• func Subject(s string) map[string]bool
• func Values(s string) map[string]bool

Constants

const Related = 0.5

Related is the bar for "these two statements are about the same thing".

Half the shorter statement's distinctive words. Tuned by what it is used for: every consumer of this package acts on a match by suppressing something or interrupting someone, so a false positive is louder than a miss.

Variables

var Negations = []string{
	"not ", "no longer", "instead of", "rather than", "decided against",
	"drop ", "dropped", "cancel", "cancelled", "abandon", "scrap", "stop ",
	"reverted", "backed out", "we are staying", "sticking with",
}

Negations are the ways people call something off. Cheap and blunt, but the alternative is a model call on every read, and these are the phrasings that actually show up when a plan is cancelled or an approach is abandoned.

var Stopwords = map[string]bool{
	"the": true, "and": true, "for": true, "was": true, "are": true, "our": true,
	"that": true, "this": true, "with": true, "from": true, "have": true, "has": true,
	"will": true, "would": true, "into": true, "over": true, "under": true, "after": true,
	"before": true, "than": true, "then": true, "they": true, "were": true, "been": true,
	"but": true, "not": true, "its": true, "his": true, "her": true, "their": true,
	"about": true, "just": true, "only": true, "also": true, "more": true, "most": true,
	"what": true, "when": true, "where": true, "which": true, "whose": true,
	"does": true, "did": true, "should": true, "could": true, "shall": true,
	"bring": true, "make": true, "take": true, "give": true, "need": true,
	"want": true, "know": true, "tell": true, "keep": true, "come": true,
	"going": true, "doing": true, "being": true, "said": true, "says": true,
	"try": true, "trying": true, "tried": true, "instead": true, "maybe": true,

	"like": true, "likes": true, "liked": true, "prefer": true, "prefers": true,
	"preferred": true, "love": true, "loves": true, "hate": true, "hates": true,
	"dislike": true, "dislikes": true, "enjoy": true, "enjoys": true,
	"wants": true, "wanted": true, "uses": true, "used": true, "using": true,
}

Stopwords are the words that say nothing about the subject.

Two groups. The ordinary function words, and — less obviously — the interrogatives and colourless verbs that make up most of a question. "What should I do about the waveguide" is a question about the waveguide; leaving the scaffolding in drags every comparison toward a middling score where nothing is clearly related and nothing is clearly not.

Functions

func Akin

func Akin(a, b string) bool

Akin reports whether two words are the same term in different clothes.

A shared prefix of five characters catches the inflections that matter — manufactures/manufacturer, proposal/proposals, quote/quoted — without pulling in a stemmer. Exact matching missed all of them, and a question asked in the user's words rarely uses the same inflection as the note that answers it.

func DifferentSubjects

func DifferentSubjects(a, b string) bool

DifferentSubjects reports that each statement names something the other does not — the sign of two parallel facts rather than one fact restated.

The sibling of DifferingValues, for the case where the difference is a noun rather than a number. "kestrel handles checkout through a dedicated service" and "kestrel handles pricing through a dedicated service" share every word but one, embed at well over any dedup threshold, and are two facts. Values() finds no numbers in either, so the numeric guard waves them through and one of them is destroyed.

The test is mutual difference, not "the incoming says something new". That distinction is the whole of it, and getting it wrong the other way makes the guard useless:

• A restatement may add words. "I like my replies terse, without any preamble" says everything "I prefer terse replies with no preamble" says and one thing more. One subject set contains the other, and containment in either direction is a restatement.
• A parallel fact swaps one thing for another. "pricing" sits exactly where "checkout" sat, so each set holds a word the other lacks. Neither contains the other, and nothing but keeping both is safe.

Requiring mutual difference also means this never fires on a statement that merely elaborates, which is what a model does every time it re-extracts a fact it has already stated — the case the dedup threshold exists to catch.

func DifferingValues

func DifferingValues(a, b string) bool

DifferingValues reports that both statements assert a value and share none.

func Flatten

func Flatten(s string) string

Flatten collapses whitespace without shortening.

func Negated

func Negated(s string) bool

Negated reports whether a statement calls something off.

func Overlap

func Overlap(a, b map[string]bool) float64

Overlap measures containment, not Jaccard.

Jaccard asks how much two statements share as a proportion of everything either one says, which punishes the longer statement for being longer: "we are targeting a $199 retail price" against "final call: retail price is $249, that is locked for launch" scores 0.29 and slips under any sensible bar, so a superseded price gets handed over as though it were current. The question that actually matters is whether the shorter statement is *about* the same thing as the longer one, and that is containment.

func Subject

func Subject(s string) map[string]bool

Subject reduces a statement to its distinctive content words.

func Values

func Values(s string) map[string]bool

Values returns the normalised values a statement asserts.