vocabsrc

package
v0.77.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 9, 2026 License: MIT Imports: 24 Imported by: 0

Documentation

Overview

Package vocabsrc manages public authority sources (tasks/067): a registry of live-suggest and downloadable vocabulary sources seeded with built-ins (id.loc.gov datasets, Wikidata, VIAF), snapshot download jobs that convert public SKOS RDF dumps into authority-tree N-Quads the vocab index loads atomically, and the live typeahead proxy the picker and enrichment reconcile through. Custom sources (GND, Getty, MeSH, Homosaurus, ...) are drop-in registry entries, not code -- see docs/authority-sources.md.

Index

Constants

View Source 
const (
	// FlavorSuggest2 is the id.loc.gov suggest2 API (any of its datasets --
	// and anything else speaking the same shape).
	FlavorSuggest2 = "suggest2"
	// FlavorWikidata is the Wikidata wbsearchentities action API.
	FlavorWikidata = "wikidata"
	// FlavorVIAF is the VIAF AutoSuggest API.
	FlavorVIAF = "viaf"
	// FlavorSearchFAST is OCLC's searchFAST fastsuggest API (tasks/132) -- a
	// Solr-shaped response; the full documented parameter set is required
	// (the bare query/fl form 400s).
	FlavorSearchFAST = "searchfast"
)

Suggest response dialects.

Variables

View Source 
var ErrNotFound = errors.New("vocabsrc: not found")

ErrNotFound reports a missing source, job, or installed snapshot.

View Source 
var ErrValidation = errors.New("vocabsrc: invalid source")

ErrValidation reports a source description the registry refuses.

Functions

func Convert 

func Convert(r io.Reader, scheme string) ([]byte, int, error)

Convert buffers ConvertTo -- for callers that want the converted bytes in hand; the install paths stream through ConvertTo into the blob store.

func ConvertTo 

func ConvertTo(w io.Writer, r io.Reader, scheme string, maxBytes int64) (int, error)

ConvertTo streams a SKOS N-Triples/N-Quads dump (gzipped or plain) into authority-tree N-Quads under the authority:<scheme> graph, keeping only the predicates the index reads. Lines are independent in N-Quads, so the input parses and emits in bounded chunks -- peak memory is the chunk plus the concept-count set, not the dump (tasks/110); malformed lines are skipped by the lenient parser. Common wrong-format uploads (zip archives, XML exports) are named outright -- publishers like OCLC FAST distribute both. maxBytes caps the decompressed input (0 = the 4GB default). Returns the distinct prefLabel-bearing concept count.

Types

type CrosswalkEnricher 

type CrosswalkEnricher struct {
	Index *vocab.Index
	// Target is the scheme equivalents resolve into.
	Target string
}

CrosswalkEnricher walks every work subject's skos:exactMatch/closeMatch links into one target scheme (tasks/072): a work carrying a homosaurus heading whose term matches an LCSH heading gains the LCSH equivalent as a moderated suggestion. Purely local -- both vocabularies must be loaded in the index; no network.

func NewCrosswalk 

func NewCrosswalk(ix *vocab.Index, target string) *CrosswalkEnricher

NewCrosswalk wraps the index as a crosswalk enrichment source for one target scheme.

func (*CrosswalkEnricher) Enrich 

func (e *CrosswalkEnricher) Enrich(_ context.Context, works []ingest.WorkSummary) ([]ingest.Enrichment, error)

Enrich implements ingest.Enricher: for each work, every controlled subject's match links resolving in the target scheme (and not already on the work) become subject candidates, and each candidate's skos:broader ancestor chain rides along as standalone term metadata (tasks/178) so hierarchy nodes stay labeled without a work carrying them.

func (*CrosswalkEnricher) Name 

func (e *CrosswalkEnricher) Name() string

Name implements ingest.Enricher; the registry key and enrichment graph.

type Enricher 

type Enricher struct {
	Src    Source
	Client *SuggestClient
	// Index, when set, upgrades each match with the locally-installed term
	// description (multilingual labels, broader edges) and rides its
	// skos:broader ancestor chain along as Enrichment.Terms (tasks/178);
	// a match whose scheme is not installed keeps the suggest-API label.
	Index *vocab.Index
	// MinConfidence drops weaker matches. Default 0.9 (exact label only).
	MinConfidence float64
}

Enricher adapts a suggest-capable source to the ingest.Enricher contract (the generalized locsh shape): a Work's uncontrolled tags reconcile against the source's typeahead API, exact normalized-label matches score 1.0 and prefix matches 0.6, and candidates at or above MinConfidence become controlled-subject enrichments.

func NewEnricher 

func NewEnricher(src Source, client *SuggestClient) *Enricher

NewEnricher wraps a registry source as an enrichment provider.

func (*Enricher) Enrich 

func (e *Enricher) Enrich(ctx context.Context, works []ingest.WorkSummary) ([]ingest.Enrichment, error)

Enrich implements ingest.Enricher: each distinct tag is looked up once per run, and matches at or above MinConfidence become subject candidates on every Work carrying the tag.

func (*Enricher) Name 

func (e *Enricher) Name() string

Name implements ingest.Enricher; the source name keys the enrichment graph.

type InstallInfo 

type InstallInfo struct {
	Source      string    `json:"source"`
	Scheme      string    `json:"scheme"`
	Terms       int       `json:"terms"`
	InstalledAt time.Time `json:"installedAt"`
	SnapshotURL string    `json:"snapshotUrl"`
}

InstallInfo is an installed snapshot's sidecar metadata, stored beside the .nq in the blob store so install state survives restarts.

type Job 

type Job struct {
	ID         string    `json:"id"`
	Source     string    `json:"source"`
	Scheme     string    `json:"scheme"`
	Requester  string    `json:"requester"`
	Status     Status    `json:"status"`
	Terms      int       `json:"terms,omitempty"`
	Error      string    `json:"error,omitempty"`
	CreatedAt  time.Time `json:"createdAt"`
	FinishedAt time.Time `json:"finishedAt,omitzero"`
}

Job is one vocabulary download: fetch the source's SKOS dump, convert it to authority-tree N-Quads, install, and swap the index.

type Service 

type Service struct {
	DB   store.Store
	Blob blob.Store
	// Index is the shared term index snapshots swap into.
	Index *vocab.Index
	// AuthoritiesPrefix roots the authority tree the index loads; installed
	// snapshots land under it at vocab/<name>.nq. Empty = "data/authorities/".
	AuthoritiesPrefix string
	// BaseSchemes is the deployment's configured scheme filter (nil = all
	// authority graphs load, so installs never need scheme bookkeeping).
	BaseSchemes []string
	// Suggest overrides the live-typeahead client (tests). nil = defaults.
	Suggest *SuggestClient
	// HTTPClient fetches snapshot dumps. nil = a 15-minute-timeout client.
	HTTPClient *http.Client
	// MaxSnapshotMB caps a snapshot dump's decompressed size (0 = the 4GB
	// default) -- the tasks/110 defensive ceiling against a hostile or
	// misconfigured endpoint.
	MaxSnapshotMB int
	Logger        *slog.Logger
	// Now overrides the clock (tests).
	Now func() time.Time
}

Service is the authority-source surface: registry CRUD, live suggest, and snapshot download/install/remove over the shared vocab index.

func (*Service) CacheTerm 

func (s *Service) CacheTerm(ctx context.Context, sugg Suggestion) error

CacheTerm writes a live pick's minimal term description (prefLabel, definition, exactMatch siblings) into the authorities tree under cache/<scheme>/ and swaps the index (tasks/072). A subject picked from a live source labels forever -- across saves and restarts -- and its exactMatch links join the crosswalk data. Re-caching an already-cached term is a no-op.

func (*Service) CreateDownload 

func (s *Service) CreateDownload(ctx context.Context, requester, sourceName string) (Job, error)

CreateDownload queues a download job for a snapshot-capable source. The worker loop (RunQueued) picks it up; installing the same source again is the refresh path -- the snapshot is overwritten in place.

func (*Service) DeleteSource 

func (s *Service) DeleteSource(ctx context.Context, name string) error

DeleteSource removes a stored registry entry. A built-in cannot be deleted (deleting a stored override restores the shipped definition).

func (*Service) GetJob 

func (s *Service) GetJob(ctx context.Context, id string) (Job, error)

GetJob returns one job.

func (*Service) GetSource 

func (s *Service) GetSource(ctx context.Context, name string) (Source, error)

GetSource resolves one source by name.

func (*Service) InstallUpload 

func (s *Service) InstallUpload(ctx context.Context, sourceName string, r io.Reader) (int, error)

InstallUpload installs a hand-supplied dump for a registered source -- the escape hatch when the publisher's download URL is unreachable (or the source has none). Same converter, snapshot layout, and index swap as a download; the sidecar records "upload" as the provenance. Synchronous: the caller holds the bytes, no worker round-trip.

func (*Service) Installed 

func (s *Service) Installed(ctx context.Context) ([]InstallInfo, error)

Installed lists the installed snapshots by reading the sidecars under the vocab/ subtree.

func (*Service) Jobs 

func (s *Service) Jobs(ctx context.Context) ([]Job, error)

Jobs lists the download jobs, newest first.

func (*Service) PutSource 

func (s *Service) PutSource(ctx context.Context, src Source) error

PutSource creates or replaces a registry entry -- the drop-in-config path for sources beyond the built-ins (and for overriding a built-in's URLs).

func (*Service) Reload 

func (s *Service) Reload(ctx context.Context) error

Reload rebuilds the shared index from the authorities tree with the effective schemes -- an atomic snapshot swap under concurrent readers.

func (*Service) RemoveSnapshot 

func (s *Service) RemoveSnapshot(ctx context.Context, name string) error

RemoveSnapshot deletes an installed snapshot and its sidecar, then reloads the index so the scheme's terms drop out.

func (*Service) RunDownload 

func (s *Service) RunDownload(ctx context.Context, id string) error

RunDownload executes a QUEUED job (claiming it RUNNING first, so concurrent workers cannot double-run): fetch, convert, install, reload. Failures land in the job's Error; RunDownload errors only on store problems.

func (*Service) RunQueued 

func (s *Service) RunQueued(ctx context.Context) (int, error)

RunQueued drains QUEUED jobs once -- the worker-loop body.

func (*Service) Schemes 

func (s *Service) Schemes(ctx context.Context) ([]string, error)

Schemes computes the index's effective scheme filter: the configured base filter plus every installed snapshot's and cached live pick's scheme, or nil (= everything) when no base filter is configured.

func (*Service) Sources 

func (s *Service) Sources(ctx context.Context) ([]Source, error)

Sources merges the built-ins with the stored registry (a stored source overrides a built-in of the same name), sorted by name.

func (*Service) Views 

func (s *Service) Views(ctx context.Context) ([]SourceView, error)

Views assembles the download-list screen's rows.

type Source 

type Source struct {
	Name   string `json:"name"`
	Scheme string `json:"scheme"` // vocab scheme key its terms live under
	// License and Homepage surface in the download list so a deployment
	// sees what it is installing.
	License  string `json:"license,omitempty"`
	Homepage string `json:"homepage,omitempty"`
	// Live typeahead capability. SuggestFlavor selects the response dialect
	// (suggest2 | wikidata | viaf); SuggestDataset is the suggest2 dataset
	// path (e.g. "authorities/subjects").
	SuggestFlavor  string `json:"suggestFlavor,omitempty"`
	SuggestURL     string `json:"suggestUrl,omitempty"`
	SuggestDataset string `json:"suggestDataset,omitempty"`
	// SnapshotURL points at a downloadable SKOS RDF dump (N-Triples or
	// N-Quads, optionally gzipped) convertible into the vocab index.
	SnapshotURL string `json:"snapshotUrl,omitempty"`
	// Builtin marks a shipped source (not deletable; a stored source of the
	// same name overrides it).
	Builtin bool `json:"builtin,omitempty"`
}

Source is one public authority source. A source may offer live typeahead (SuggestURL), a downloadable SKOS dump (SnapshotURL), or both.

func Builtins 

func Builtins() []Source

Builtins returns the shipped sources: the id.loc.gov datasets (subjects, genre/form, children's subjects downloadable; the name authority file is live-only -- its dump is ~11M concepts), OCLC FAST (live-only, tasks/132), Wikidata, and VIAF.

func (Source) CanSnapshot 

func (s Source) CanSnapshot() bool

CanSnapshot reports whether the source offers a downloadable dump.

func (Source) CanSuggest 

func (s Source) CanSuggest() bool

CanSuggest reports whether the source offers live typeahead.

type SourceView 

type SourceView struct {
	Source
	Installed *InstallInfo `json:"installed,omitempty"`
	Job       *Job         `json:"job,omitempty"`
}

SourceView is the list surface: a source plus its install state and its most recent download job.

type Status 

type Status string

Status is the download-job lifecycle (the tasks/038 export shape). 

const (
	StatusQueued  Status = "QUEUED"
	StatusRunning Status = "RUNNING"
	StatusDone    Status = "DONE"
	StatusFailed  Status = "FAILED"
)

type SuggestClient 

type SuggestClient struct {
	// Client overrides the HTTP client (tests). Default: 10s timeout.
	Client *http.Client
}

SuggestClient queries a source's live typeahead API.

func (*SuggestClient) Suggest 

func (c *SuggestClient) Suggest(ctx context.Context, src Source, q string, limit int) ([]Suggestion, error)

Suggest queries one source for up to limit typeahead hits.

type Suggestion 

type Suggestion struct {
	Source      string `json:"source"`
	Scheme      string `json:"scheme"`
	ID          string `json:"id"`
	Label       string `json:"label"`
	Description string `json:"description,omitempty"`
	// Variants are the heading's variant/"used for" labels when the source
	// exposes them (suggest2's more.variantLabels) -- often the only context
	// a bare authorized heading carries.
	Variants   []string `json:"variants,omitempty"`
	ExactMatch []string `json:"exactMatch,omitempty"`
}

Suggestion is one live typeahead hit, source-tagged for the picker badge. ExactMatch carries sibling identifiers when the source exposes them (VIAF clusters map to LCNAF/GND/Wikidata) so a term created from the pick can record skos:exactMatch cross-references.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.