vocabsrc

package
v0.183.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 11, 2026 License: MIT Imports: 25 Imported by: 0

Documentation

Overview

Package vocabsrc manages public authority sources: a registry of live-suggest and downloadable vocabulary sources seeded with built-ins (id.loc.gov datasets, Wikidata, VIAF), snapshot download jobs that convert public SKOS RDF dumps into authority-tree N-Quads the vocab index loads atomically, and the live typeahead proxy the picker and enrichment reconcile through. Custom sources (GND, Getty, MeSH, Homosaurus, ...) are drop-in registry entries, not code -- see docs/authority-sources.md.

Index

Constants

View Source 
const (
	// FlavorSuggest2 is the id.loc.gov suggest2 API (any of its datasets --
	// and anything else speaking the same shape).
	FlavorSuggest2 = "suggest2"
	// FlavorWikidata is the Wikidata wbsearchentities action API.
	FlavorWikidata = "wikidata"
	// FlavorVIAF is the VIAF AutoSuggest API.
	FlavorVIAF = "viaf"
	// FlavorSearchFAST is OCLC's searchFAST fastsuggest API -- a
	// Solr-shaped response; the full documented parameter set is required
	// (the bare query/fl form 400s).
	FlavorSearchFAST = "searchfast"
)

Suggest response dialects.

Variables

View Source 
var ErrConflict = errors.New("vocabsrc: conflict")

ErrConflict reports an operation refused by the state it would leave behind -- deleting a source whose snapshot is still installed. The caller can make it succeed by doing something first, which is what separates it from ErrValidation.

View Source 
var ErrNotFound = errors.New("vocabsrc: not found")

ErrNotFound reports a missing source, job, or installed snapshot.

View Source 
var ErrValidation = errors.New("vocabsrc: invalid source")

ErrValidation reports a source description the registry refuses.

View Source 
var SuggestFlavors = []string{FlavorSuggest2, FlavorWikidata, FlavorVIAF, FlavorSearchFAST}

SuggestFlavors is the one allow-list of configurable suggest dialects, so the validator, the dispatcher, and the SPA dropdown derive from a single source rather than each keeping its own copy (a flavor added to the dispatcher but not the validator is exactly how searchfast became builtin-only).

Functions

func Convert 

func Convert(r io.Reader, scheme string) ([]byte, int, error)

Convert buffers ConvertTo -- for callers that want the converted bytes in hand; the install paths stream through ConvertTo into the blob store.

func ConvertTo 

func ConvertTo(w io.Writer, r io.Reader, scheme string, maxBytes int64) (int, error)

ConvertTo streams a SKOS N-Triples/N-Quads dump (gzipped or plain) into authority-tree N-Quads under the authority:<scheme> graph, keeping only the predicates the index reads. It decodes one statement at a time, so peak memory is a statement plus the concept-count set, not the dump. Common wrong-format uploads (zip archives, XML exports) are named outright -- publishers like OCLC FAST distribute both. maxBytes caps the decompressed input (0 = the 4GB default). Returns the distinct prefLabel-bearing concept count.

A malformed line refuses the whole dump, naming the line. It used to be skipped, and that was not a decision -- it was whatever libcodex's parser did. The dumps that trip it are the ones you most want refused: a 5,242,880-byte homosaurus-v4.nt, cut mid-IRI at exactly 5MiB, converts cleanly under a lenient parser and installs a vocabulary silently missing every concept after the cut. The subject pages it labels are then wrong, and nothing anywhere says so. Five real LC/Homosaurus/FAST dumps were parsed strictly to check this; the only one that failed was that truncated download.

The line number in that refusal is the dump's own. This used to feed 1MB chunks to rdf.ParseNQuads and add a running base to each SyntaxError.Line, because a bulk parser numbers from the start of the bytes it was handed; the streaming decoder numbers from the start of the stream, so the base is gone and cannot drift from the parser again.

func ValidSuggestFlavor added in v0.146.1 

func ValidSuggestFlavor(f string) bool

ValidSuggestFlavor reports whether f is a configurable suggest dialect.

Types

type CrosswalkEnricher 

type CrosswalkEnricher struct {
	Index *vocab.Index
	// Target is the scheme equivalents resolve into.
	Target string
}

CrosswalkEnricher walks every work subject's skos:exactMatch/closeMatch links into one target scheme: a work carrying a homosaurus heading whose term matches an LCSH heading gains the LCSH equivalent as a moderated suggestion. Purely local -- both vocabularies must be loaded in the index; no network.

func NewCrosswalk 

func NewCrosswalk(ix *vocab.Index, target string) *CrosswalkEnricher

NewCrosswalk wraps the index as a crosswalk enrichment source for one target scheme.

func (*CrosswalkEnricher) Enrich 

func (e *CrosswalkEnricher) Enrich(_ context.Context, works []ingest.WorkSummary) ([]ingest.Enrichment, error)

Enrich implements ingest.Enricher: for each work, every controlled subject's match links resolving in the target scheme (and not already on the work) become subject candidates, and each candidate's skos:broader ancestor chain rides along as standalone term metadata so hierarchy nodes stay labeled without a work carrying them.

func (*CrosswalkEnricher) Name 

func (e *CrosswalkEnricher) Name() string

Name implements ingest.Enricher; the registry key and enrichment graph.

type Enricher 

type Enricher struct {
	Src    Source
	Client *SuggestClient
	// Index, when set, upgrades each match with the locally-installed term
	// description (multilingual labels, broader edges) and rides its
	// skos:broader ancestor chain along as Enrichment.Terms;
	// a match whose scheme is not installed keeps the suggest-API label.
	Index *vocab.Index
	// MinConfidence drops weaker matches. Default 0.9 (exact label only).
	MinConfidence float64
}

Enricher adapts a suggest-capable source to the ingest.Enricher contract (the generalized locsh shape): a Work's uncontrolled tags reconcile against the source's typeahead API, exact normalized-label matches score 1.0 and prefix matches 0.6, and candidates at or above MinConfidence become controlled-subject enrichments.

func NewEnricher 

func NewEnricher(src Source, client *SuggestClient) *Enricher

NewEnricher wraps a registry source as an enrichment provider.

func (*Enricher) Enrich 

func (e *Enricher) Enrich(ctx context.Context, works []ingest.WorkSummary) ([]ingest.Enrichment, error)

Enrich implements ingest.Enricher: each distinct tag is looked up once per run, and matches at or above MinConfidence become subject candidates on every Work carrying the tag.

func (*Enricher) Name 

func (e *Enricher) Name() string

Name implements ingest.Enricher; the source name keys the enrichment graph.

type InstallInfo 

type InstallInfo struct {
	Source      string    `json:"source"`
	Scheme      string    `json:"scheme"`
	Terms       int       `json:"terms"`
	InstalledAt time.Time `json:"installedAt"`
	SnapshotURL string    `json:"snapshotUrl"`
}

InstallInfo is an installed snapshot's sidecar metadata, stored beside the .nq in the blob store so install state survives restarts.

type Job 

type Job struct {
	ID         string    `json:"id"`
	Source     string    `json:"source"`
	Scheme     string    `json:"scheme"`
	Requester  string    `json:"requester"`
	Status     Status    `json:"status"`
	Terms      int       `json:"terms,omitempty"`
	Error      string    `json:"error,omitempty"`
	CreatedAt  time.Time `json:"createdAt"`
	FinishedAt time.Time `json:"finishedAt,omitzero"`
}

Job is one vocabulary download: fetch the source's SKOS dump, convert it to authority-tree N-Quads, install, and swap the index.

type Service 

type Service struct {
	DB   store.Store
	Blob blob.Store
	// Index is the shared term index snapshots swap into.
	Index *vocab.Index
	// AuthoritiesPrefix roots the authority tree the index loads; installed
	// snapshots land under it at vocab/<name>.nq. Empty = "data/authorities/".
	AuthoritiesPrefix string
	// BaseSchemes is the deployment's configured scheme filter (nil = all
	// authority graphs load, so installs never need scheme bookkeeping).
	BaseSchemes []string
	// Suggest overrides the live-typeahead client (tests). nil = defaults.
	Suggest *SuggestClient
	// HTTPClient fetches snapshot dumps. nil = a 15-minute-timeout client.
	HTTPClient *http.Client
	// MaxSnapshotMB caps a snapshot dump's decompressed size (0 = the 4GB
	// default) -- the defensive ceiling against a hostile or
	// misconfigured endpoint.
	MaxSnapshotMB int
	Logger        *slog.Logger
	// Now overrides the clock (tests).
	Now func() time.Time
}

Service is the authority-source surface: registry CRUD, live suggest, and snapshot download/install/remove over the shared vocab index.

func (*Service) CacheTerm 

func (s *Service) CacheTerm(ctx context.Context, sugg Suggestion) error

CacheTerm writes a live pick's minimal term description (prefLabel, definition, exactMatch siblings) into the authorities tree under cache/<scheme>/ and swaps the index. A subject picked from a live source labels forever -- across saves and restarts -- and its exactMatch links join the crosswalk data. Re-caching an already-cached term is a no-op.

func (*Service) CreateDownload 

func (s *Service) CreateDownload(ctx context.Context, requester, sourceName string) (Job, error)

CreateDownload queues a download job for a snapshot-capable source. The worker loop (RunQueued) picks it up; installing the same source again is the refresh path -- the snapshot is overwritten in place.

func (*Service) DeleteSource 

func (s *Service) DeleteSource(ctx context.Context, name string) error

DeleteSource removes a stored registry entry. A built-in cannot be deleted (deleting a stored override restores the shipped definition).

It refuses with ErrConflict while a snapshot is installed, which is what the screen's tooltip has always promised: deleting the row out from under an install leaves an orphan whose Upload and Delete actions can only 404, and the admin was told the server would stop them. RemoveSnapshot first, then delete.

Deleting a stored *override* of a built-in is exempt: the shipped definition takes its place, so the install keeps a source and is never orphaned.

The refusal is not a statement about the vocabulary. An install still outlives its source row by other routes -- an offline lcat vocab-install, or a registry that reset because the deployment has no document store -- so Views still synthesizes the orphan, and RemoveSnapshot is still the way out of it.

func (*Service) GetJob 

func (s *Service) GetJob(ctx context.Context, id string) (Job, error)

GetJob returns one job.

func (*Service) GetSource 

func (s *Service) GetSource(ctx context.Context, name string) (Source, error)

GetSource resolves one source by name.

func (*Service) InstallUpload 

func (s *Service) InstallUpload(ctx context.Context, sourceName string, r io.Reader) (int, error)

InstallUpload installs a hand-supplied dump for a registered source -- the escape hatch when the publisher's download URL is unreachable (or the source has none). Same converter, snapshot layout, and index swap as a download; the sidecar records "upload" as the provenance. Synchronous: the caller holds the bytes, no worker round-trip.

func (*Service) Installed 

func (s *Service) Installed(ctx context.Context) ([]InstallInfo, error)

Installed lists the installed snapshots by reading the sidecars under the vocab/ subtree.

func (*Service) Jobs 

func (s *Service) Jobs(ctx context.Context) ([]Job, error)

Jobs lists the download jobs, newest first.

func (*Service) PutSource 

func (s *Service) PutSource(ctx context.Context, src Source) error

PutSource creates or replaces a registry entry -- the drop-in-config path for sources beyond the built-ins (and for overriding a built-in's URLs).

func (*Service) Reload 

func (s *Service) Reload(ctx context.Context) error

Reload rebuilds the shared index from the authorities tree with the effective schemes -- an atomic snapshot swap under concurrent readers.

func (*Service) RemoveCachedTerm added in v0.146.0 

func (s *Service) RemoveCachedTerm(ctx context.Context, scheme, id string) error

RemoveCachedTerm deletes one live-pick cache entry and reloads the index -- the undo for a click. If it was the scheme's last pick, the scheme drops out of cachedSchemes, so the reload also drops it from the filter unless a snapshot or the base filter still keeps it.

func (*Service) RemoveSnapshot 

func (s *Service) RemoveSnapshot(ctx context.Context, name string) (InstallInfo, error)

RemoveSnapshot deletes an installed snapshot, its install meta, and the sidecar index artifacts built from it, then reloads the index so the scheme's terms drop out.

The scheme comes from the install meta rather than the source registry, because removing a snapshot whose source row is already gone -- the orphan install Views synthesizes so it stays removable -- is exactly the case that leaves artifacts behind. The sidecar goes first: its manifest is what arms the scheme, so an interrupted removal degrades the scheme to the map loader instead of arming it on an index whose snapshot has been deleted. It returns the removed snapshot's InstallInfo so a caller can record what was uninstalled (its scheme and term count) in an audit note.

func (*Service) RunDownload 

func (s *Service) RunDownload(ctx context.Context, id string) error

RunDownload executes a QUEUED job (claiming it RUNNING first, so concurrent workers cannot double-run): fetch, convert, install, reload. Failures land in the job's Error; RunDownload errors only on store problems.

func (*Service) RunQueued 

func (s *Service) RunQueued(ctx context.Context) (int, error)

RunQueued drains QUEUED jobs once -- the worker-loop body.

func (*Service) Schemes 

func (s *Service) Schemes(ctx context.Context) ([]string, error)

Schemes computes the index's effective scheme filter: the configured base filter plus every installed snapshot's and cached live pick's scheme, or nil (= everything) when no base filter is configured.

func (*Service) Sources 

func (s *Service) Sources(ctx context.Context) ([]Source, error)

Sources merges the built-ins with the stored registry (a stored source overrides a built-in of the same name), sorted by name.

func (*Service) Views 

func (s *Service) Views(ctx context.Context) ([]SourceView, error)

Views assembles the download-list screen's rows.

type Source 

type Source struct {
	Name   string `json:"name"`
	Scheme string `json:"scheme"` // vocab scheme key its terms live under
	// License and Homepage surface in the download list so a deployment
	// sees what it is installing.
	License  string `json:"license,omitempty"`
	Homepage string `json:"homepage,omitempty"`
	// Live typeahead capability. SuggestFlavor selects the response dialect
	// (suggest2 | wikidata | viaf); SuggestDataset is the suggest2 dataset
	// path (e.g. "authorities/subjects").
	SuggestFlavor  string `json:"suggestFlavor,omitempty"`
	SuggestURL     string `json:"suggestUrl,omitempty"`
	SuggestDataset string `json:"suggestDataset,omitempty"`
	// SnapshotURL points at a downloadable SKOS RDF dump (N-Triples or
	// N-Quads, optionally gzipped) convertible into the vocab index.
	SnapshotURL string `json:"snapshotUrl,omitempty"`
	// Builtin marks a shipped source (not deletable; a stored source of the
	// same name overrides it).
	Builtin bool `json:"builtin,omitempty"`
}

Source is one public authority source. A source may offer live typeahead (SuggestURL), a downloadable SKOS dump (SnapshotURL), or both.

func Builtins 

func Builtins() []Source

Builtins returns the shipped sources: the id.loc.gov datasets (subjects, genre/form, children's subjects downloadable; the name authority file is live-only -- its dump is ~11M concepts), OCLC FAST (live-only), Wikidata, and VIAF.

func (Source) CanSnapshot 

func (s Source) CanSnapshot() bool

CanSnapshot reports whether the source offers a downloadable dump.

func (Source) CanSuggest 

func (s Source) CanSuggest() bool

CanSuggest reports whether the source offers live typeahead.

type SourceView 

type SourceView struct {
	Source
	Installed *InstallInfo `json:"installed,omitempty"`
	Job       *Job         `json:"job,omitempty"`
	// Orphan marks a row synthesized from an install with no source record behind
	// it. Such a row can only be removed: everything else the screen offers needs a
	// source to act on, and answers 404 without one. An empty
	// SnapshotURL is not a proxy for this -- an upload-only source has none either.
	Orphan bool `json:"orphan,omitempty"`
	// Sidecar reports whether the scheme is served from artifacts on disk rather
	// than resident maps, and ResidentTerms is the count it still holds in memory
	// (the whole scheme when map-backed, just the live-pick overlay when
	// sidecar-backed). They explain a process's memory profile scheme by scheme
	//; Installed.Terms is the snapshot size regardless of backend.
	Sidecar       bool `json:"sidecar,omitempty"`
	ResidentTerms int  `json:"residentTerms,omitempty"`
}

SourceView is the list surface: a source plus its install state and its most recent download job.

type Status 

type Status string

Status is the download-job lifecycle (the export shape). 

const (
	StatusQueued  Status = "QUEUED"
	StatusRunning Status = "RUNNING"
	StatusDone    Status = "DONE"
	StatusFailed  Status = "FAILED"
)

type SuggestClient 

type SuggestClient struct {
	// Client overrides the HTTP client (tests). Default: 10s timeout.
	Client *http.Client
}

SuggestClient queries a source's live typeahead API.

func (*SuggestClient) Suggest 

func (c *SuggestClient) Suggest(ctx context.Context, src Source, q string, limit int) ([]Suggestion, error)

Suggest queries one source for up to limit typeahead hits.

type Suggestion 

type Suggestion struct {
	Source      string `json:"source"`
	Scheme      string `json:"scheme"`
	ID          string `json:"id"`
	Label       string `json:"label"`
	Description string `json:"description,omitempty"`
	// Variants are the heading's variant/"used for" labels when the source
	// exposes them (suggest2's more.variantLabels) -- often the only context
	// a bare authorized heading carries.
	Variants   []string `json:"variants,omitempty"`
	ExactMatch []string `json:"exactMatch,omitempty"`
}

Suggestion is one live typeahead hit, source-tagged for the picker badge. ExactMatch carries sibling identifiers when the source exposes them (VIAF clusters map to LCNAF/GND/Wikidata) so a term created from the pick can record skos:exactMatch cross-references.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.