similar

Documentation

Overview

Package similar answers "more like this" over a catalog: given a Work, which other Works does it most resemble (tasks/284)?

The method is a two-hop walk over the bipartite graph of Works and their attributes -- series, contributors, tags, subjects. From the focus Work, step out to each attribute it carries, then back in to every other Work carrying that attribute. Each shared attribute contributes to that Work's score.

Three things keep the walk honest, and none of them is optional:

• Rarity weighting. A shared attribute is worth weight/log2(df+2), where df is how many Works carry it. Two books sharing an obscure subject heading have told you far more than two books sharing "Fiction".
• A document-frequency cap. An attribute held by more than DFCapFraction of the catalog is skipped entirely: it cannot discriminate. The cap is a fraction rather than a constant so it scales with the collection.
• A singleton floor. An attribute only this Work carries links to nothing, and one shared by exactly two Works is the most informative case there is, so the floor is df >= 2 rather than any higher threshold.

Subjects additionally walk the SKOS concept tree: a Work's subjects are expanded upward through skos:broader (decayed per hop) and one step down through narrower. This is what makes the result feel like subject cataloging rather than string matching -- "Lesbian mothers" and "Lesbian parents" are neighbours in the tree even when no Work carries both IRIs.

Language is scored differently on purpose. It is a flat bonus applied only to Works some other signal has already scored, never a way onto the list: every Work shares a language with most of the catalog, so as a walk edge it would return the whole catalog in an arbitrary order.

The package is pure. It holds no store, no HTTP, no Hugo -- so the OPAC's build step and the admin's live endpoint can compute identical neighbours instead of drifting apart.

Index

• Variables
• type Index
  • func Build(works []Work, opts Options) *Index
  • func (ix *Index) Len() int
  • func (ix *Index) Neighbors(workID string, n int) []Scored
• type Options
  • func DefaultOptions() Options
• type Relation
• type Scored
• type Weights
• type Work

Variables

var DefaultWeights = Weights{
	RelSeries:      5,
	RelContributor: 3,
	RelTag:         2,
	RelSubject:     1,
}

DefaultWeights is the shipped tuning. Raising Tag lets one coincidental folksonomy label float unrelated books onto the rail; that is the failure mode this ordering exists to avoid.

Types

type Index

type Index struct {
	// contains filtered or unexported fields
}

Index is a built, read-only postings index. Safe for concurrent Neighbors.

func Build

func Build(works []Work, opts Options) *Index

Build indexes the catalog. Tombstoned Works are excluded outright: a retired record must not be recommended from elsewhere, and it has no neighbours of its own (tasks/280). Suppressed Works are kept -- the admin surface shows them, and the public projection never sees them because it drops them upstream. A repeated WorkID is indexed once, at its first occurrence. Duplicates are a caller error -- ScanSummaries over a prefix that also catches catalog.nq will produce them -- but the failure they cause is nasty and silent: the focus Work occupies several offsets, Neighbors excludes only the one it looked up, and the Work recommends itself at the top of its own rail.

Every attribute slice is normalized on the way in: sorted, de-duplicated, empty values dropped. A Work carrying one subject twice would otherwise post to it twice and count it twice, scoring a coincidence of serialization as evidence -- and neither caller de-duplicates subjects or tags upstream. Normalizing here rather than asking the callers to also means an ordering difference between the projector and the admin index cannot change anyone's neighbours.

func (*Index) Len

func (ix *Index) Len() int

Len reports how many Works the index scores over.

func (*Index) Neighbors

func (ix *Index) Neighbors(workID string, n int) []Scored

Neighbors returns up to n Works most similar to workID, best first. An unknown or tombstoned workID yields nothing rather than an error: it has no neighbours, which is the honest answer and the one every caller would otherwise write.

type Options

type Options struct {
	Weights Weights
	// DFCapFraction skips any attribute carried by more than this fraction of
	// the catalog. 0.20 means "a subject on a fifth of the catalog tells us
	// nothing". Values <= 0 or >= 1 disable the cap.
	DFCapFraction float64
	// LanguageBonus is added once to any candidate that shares a language with
	// the focus Work and was already scored by another signal.
	LanguageBonus float64
	// AvailabilityBonus nudges a holdable candidate above an identical one that
	// the reader cannot borrow.
	AvailabilityBonus float64
	// TreeDepth is how many skos:broader hops a subject expands upward;
	// TreeDecay multiplies the contribution per hop. Depth 0 disables the tree
	// walk. Narrower is always exactly one hop, at TreeDecay.
	TreeDepth int
	TreeDecay float64
	// Broader and Narrower resolve a subject IRI's neighbours in the concept
	// tree. Both may be nil, which disables the tree walk. They exist as hooks
	// so this package never imports a vocabulary index.
	Broader  func(iri string) []string
	Narrower func(iri string) []string
}

Options tunes the walk. The zero value is not useful; use DefaultOptions.

func DefaultOptions

func DefaultOptions() Options

DefaultOptions carries qllpoc's tuning.

type Relation

type Relation int

Relation names the attribute kinds the walk traverses.

const (
	RelSeries Relation = iota
	RelContributor
	RelTag
	RelSubject
)

type Scored

type Scored struct {
	WorkID string  `json:"workId"`
	Score  float64 `json:"score"`
	// Shared names the attributes that put this Work on the list, most
	// valuable first, so a cataloger asking "why is this here?" has an answer.
	Shared []string `json:"shared,omitempty"`
}

Scored is one neighbour and why it ranked.

type Weights

type Weights [numRelations]float64

Weights are the per-relation contributions, before rarity weighting. The defaults are qllpoc's, tuned against a 62k-work public-library catalog: two books in a series are related whatever else they share, a shared author is strong, a shared uncontrolled tag is suggestive, and a single shared subject heading is the weakest real evidence there is.

type Work

type Work struct {
	WorkID string
	// Tombstoned Works are excluded from the index outright (tasks/280).
	Tombstoned bool

	// Held earns the availability bonus: a neighbour the reader can actually
	// borrow outranks one they cannot. Both callers already agree on the
	// predicate -- project.Work.Held is defined as physical items or a live
	// availability identifier, which is ingest's HasAvailability || Items > 0.
	Held bool

	Series       []string
	Contributors []string
	Tags         []string
	Subjects     []string
	Languages    []string
}

Work is the scorer's input: the attributes a Work is scored on, and nothing else. It is deliberately not ingest.WorkSummary or project.Work.

Both surfaces must recommend the same neighbours -- an OPAC page and the admin editor disagreeing about what a Work resembles is the tasks/253 failure class, where the facet rail and the query disagreed about what was filtered. The guarantee is that each caller converts its own record into this one type (ingest.WorkSummary.SimilarWork, project.Work.SimilarWork), and a test drives both converters from the same graph and requires the results to be equal.

Values are opaque strings compared by equality, so the two callers must agree on spelling as well as shape: subjects are authority IRIs, contributors are agent labels, languages are local names ("en").