ingest

Documentation

Overview

Facet dimensions over a WorkSummary (tasks/168, tasks/171, tasks/254).

These live next to WorkSummary rather than in the HTTP layer because two callers must agree on them exactly: the works listing, which counts and filters facets for the rail, and a batch/export selection, which resolves the same filters into a set of Works. When they drifted, "Export these results…" offered the entire catalog (tasks/254). One definition, two callers.

Package ingest defines the compile-time provider model (ARCHITECTURE §9a): a bibliographic source plugs in as a Go type satisfying Provider, is registered in a Registry, and runs through the shared identity + clustering pipeline (Run). The provider set is finite, first-party or deployment-authored, and a deployment builds its own lcat, so binding is static -- no dynamic loading. The interface is shaped so a future subprocess or WASM transport can implement it unchanged.

Index

• Constants
• func CleanText(s string) string
• func FacetValues(s WorkSummary, group string) []string
• func FoldsCase(group string) bool
• func Holdings(s WorkSummary) []string
• func MatchesFacets(s WorkSummary, filters map[string][]string) bool
• func MatchesGroup(s WorkSummary, group string, selected []string) bool
• func Needs(s WorkSummary) []string
• func RunEnrich(ctx context.Context, st blob.Store, prefix string, e Enricher) (int, error)
• func SimilarWorks(summaries []WorkSummary) []similar.Work
• func SplitExtra(v string) []string
• func Visibility(s WorkSummary) string
• type AuthoritySubject
• type Config
• type Enricher
• type Enrichment
• type ExtraProvider
• type Factory
• type Provider
• type ReconcileResult
  • func Reconcile(ctx context.Context, st blob.Store, prefix, feed string, ...) (ReconcileResult, error)
• type Record
• type Registry
  • func NewRegistry() *Registry
  • func (r *Registry) Names() []string
  • func (r *Registry) New(name string, cfg Config) (Provider, error)
  • func (r *Registry) Register(name string, f Factory) error
• type Result
  • func Run(prov Provider, out string) (Result, error)
  • func RunStore(ctx context.Context, prov Provider, st blob.Store, prefix string) (Result, []string, error)
• type Role
  • func (r Role) String() string
• type SubjectEnricher
• type SummarySource
• type TermDescriber
• type VerbatimProvider
• type WorkSummary
  • func ScanSummaries(ctx context.Context, st blob.Store, prefix string) ([]WorkSummary, map[string]string, error)
  • func SummariesOf(ctx context.Context, src SummarySource, st blob.Store, prefix string) ([]WorkSummary, map[string]string, error)
  • func SummarizeDataset(ds *rdf.Dataset) []WorkSummary
  • func SummarizeGrain(grain []byte) ([]WorkSummary, error)
  • func (s WorkSummary) Matches(q string) bool
  • func (s WorkSummary) SimilarWork() similar.Work

Constants

const (
	FacetVisibility = "visibility"
	FacetHoldings   = "holdings"
	FacetNeeds      = "needs"
	FacetSubject    = "subject"
	FacetTag        = "tag"
)

Facet group names. Anything else is an extras key (tasks/171).

const (
	// ReconcileReview flags the Work (lcat:withdrawnFromFeed) for the admin
	// review queue and changes nothing else.
	ReconcileReview = "review"
	// ReconcileAutoSuppress flags and suppresses in one pass, recording the
	// pass as the suppressor so a returning title un-suppresses itself.
	ReconcileAutoSuppress = "auto-suppress"
)

Reconciliation policies (tasks/078): what happens to a Work whose sole bib feed stopped listing it.

Functions

func CleanText

func CleanText(s string) string

CleanText normalizes a vendor free-text field: it resolves HTML character references, drops markup tags, and collapses the whitespace runs that stripping leaves. Text carrying neither "&" nor "<" passes through untouched, so a clean value (and a bare "<" as prose, e.g. "2 < 3") is byte-identical.

References decode to a bounded fixpoint (three passes): vendor feeds double-escape (&amp;#8212;), so a single pass would only peel one layer. The providers apply this to prose and transcribed titles at ingest, so grains store clean text and every downstream projection and export inherits it (tasks/081, tasks/089). Headings and identifiers are deliberately left untouched.

func FacetValues

func FacetValues(s WorkSummary, group string) []string

FacetValues returns a summary's values in one facet group. An unrecognized group name is read as an extras key, whose comma-joined value is split -- the lcat convention for multi-valued extras.

func FoldsCase

func FoldsCase(group string) bool

FoldsCase reports whether a group's values match case-insensitively. Tags are free text a cataloger typed; the controlled dimensions are not.

func Holdings

func Holdings(s WorkSummary) []string

Holdings lists the holdings signals a summary carries (multi-valued).

func MatchesFacets

func MatchesFacets(s WorkSummary, filters map[string][]string) bool

MatchesFacets reports whether a summary passes every group's filter: AND across groups, OR within one. This is the rule the facet rail draws and the rule an export selection must resolve by, or the count beside the button and the count in the file disagree (tasks/254).

func MatchesGroup

func MatchesGroup(s WorkSummary, group string, selected []string) bool

MatchesGroup reports whether a summary passes one group's filter. An empty selection always passes: an unselected facet means "any", not "none".

func Needs

func Needs(s WorkSummary) []string

Needs lists a summary's completeness gaps (multi-valued): the triage slices catalogers work through.

func RunEnrich

func RunEnrich(ctx context.Context, st blob.Store, prefix string, e Enricher) (int, error)

RunEnrich executes an enricher in direct (auto-approve) mode over every grain under prefix in the store: each returned Work's enrichment:<name> graph is dropped and replaced with the fresh assertions, so a re-run is idempotent, and returning an Enrichment with no Subjects explicitly clears a Work's statements from this source. Works absent from the result are left untouched. Returns the number of Works written.

func SimilarWorks

func SimilarWorks(summaries []WorkSummary) []similar.Work

SimilarWorks converts a batch, preserving order.

func SplitExtra

func SplitExtra(v string) []string

SplitExtra splits a comma-joined extras value into trimmed facet values. A single-valued extra passes through unchanged.

func Visibility

func Visibility(s WorkSummary) string

Visibility buckets a summary into exactly one value: what the public projection does with it. A withdrawn-but-kept Work is public by the curator's decision (tasks/078).

Types

type AuthoritySubject

type AuthoritySubject = bibframe.AuthoritySubject

AuthoritySubject is one controlled-vocabulary subject a provider asserts for a Work (authority URI + localized labels + skos:broader parents). It is an alias of the bibframe emission type, defined in bibframe to avoid an ingest<-bibframe import cycle while keeping the capability's shape visible here.

type Config

type Config struct {
	Feed   string
	Source string
	Params map[string]string
}

Config carries a provider's build-time configuration into its Factory. Feed overrides the provenance graph name (default: the registry key); Source is the primary input (a cache directory, a file, a URL); Params holds provider-specific extras so the registry stays uniform.

type Enricher

type Enricher interface {
	Name() string
	Enrich(ctx context.Context, works []WorkSummary) ([]Enrichment, error)
}

Enricher produces enrichments for batches of Works. This executes the RoleEnrich half of the provider model that Run reserves: enrichers never touch feed graphs -- their statements land in their own enrichment:<name> graph (direct mode) or in the moderation queue (a deployment decision made by the caller, not the enricher).

type Enrichment

type Enrichment struct {
	WorkID   string
	Subjects []AuthoritySubject
	// Terms carries standalone term descriptions -- typically the
	// skos:broader ancestor chains of Subjects -- asserted into the
	// enrichment graph as labels + hierarchy only, with no link to the Work
	// (tasks/178). They keep ancestor nodes labeled in projections that roll
	// subtree data up the hierarchy even when no work carries the ancestor
	// directly.
	Terms []AuthoritySubject
	// Confidence (0-1] qualifies queue-moderated enrichments; direct-mode
	// callers may threshold on it.
	Confidence float64
}

Enrichment is one Work's enrichment result: controlled subjects to assert.

type ExtraProvider

type ExtraProvider interface {
	Extras() map[string]string
}

ExtraProvider is an optional capability a Record may implement to carry per-Work display fields that are not part of BIBFRAME -- e.g. a cover URL, a personal rating, or a read date. Run writes them into the Work's feed provenance graph under bibframe.ExtraPred, and the projector surfaces them as catalog.json's `extra` object (tasks/026), which the Hugo module forwards to page params (tasks/022). A Record that does not implement it carries no extras, leaving the graph unchanged. For a clustered Work the first record's extras win, matching how shared Work metadata is taken.

type Factory

type Factory func(Config) (Provider, error)

Factory constructs a configured Provider from a Config. A deployment registers one per provider type, so adding a source is a Register call plus the provider's own package -- no libcat fork.

type Provider

type Provider interface {
	Name() string
	Role() Role
	Records(ctx context.Context) ([]Record, error)
}

Provider is one bibliographic source bound at compile time. Name is the provenance feed graph the provider's statements belong to (feed:<name>); Role fixes the graph class; Records yields the source's resolvable records for the shared pipeline. A network-backed provider honors ctx for cancellation.

type ReconcileResult

type ReconcileResult struct {
	// Flagged Works were newly marked withdrawn this pass.
	Flagged int
	// Suppressed Works were auto-suppressed this pass.
	Suppressed int
	// Cleared Works returned to the feed and lost their withdrawal flag.
	Cleared int
	// Unsuppressed Works returned and lost a reconcile-set suppression.
	Unsuppressed int
	// FlaggedIDs lists the newly withdrawn Works, sorted.
	FlaggedIDs []string
}

ReconcileResult counts what one pass did.

func Reconcile

func Reconcile(ctx context.Context, st blob.Store, prefix, feed string, present map[string]bool, policy, date string) (ReconcileResult, error)

Reconcile diffs the corpus against the Works a feed's latest scan resolved to (present) and flags the leavers: a Work is withdrawn only when this feed is its sole bib source and no curator has invested editorial statements in it -- items, tags, merges, or a keep decision all protect it. Returning Works lose the flag (and a reconcile-set suppression), so a title cycling out of and back into a collection round-trips with identity and editorial statements intact. Nothing is ever deleted.

type Record

type Record interface {
	Identity() identity.Record
	Work() codexbf.Work
	Instance() codexbf.Instance
}

Record is one resolvable bibliographic record: the keys that assign it stable two-tier ids (Identity) and the Work/Instance BIBFRAME it contributes once clustered. Any source that produces these three is an ingest Record -- an OverDrive item already is one -- so the pipeline is provider-agnostic.

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry maps a provider type key to its Factory. Registration is explicit (in main, not init), so the built-in set is auditable and a deployment composes its own set.

func NewRegistry

func NewRegistry() *Registry

NewRegistry returns an empty Registry.

func (*Registry) Names

func (r *Registry) Names() []string

Names returns the registered provider keys in sorted order.

func (*Registry) New

func (r *Registry) New(name string, cfg Config) (Provider, error)

New constructs the registered provider for name with cfg, erroring if the key is unknown. When cfg.Feed is empty it defaults to name, so the registry key doubles as the provenance graph unless a deployment overrides it.

func (*Registry) Register

func (r *Registry) Register(name string, f Factory) error

Register binds a provider type key to its Factory. It errors on an empty key, a nil factory, or a duplicate key, so a composition mistake fails at startup.

type Result

type Result struct {
	Stats           bibframe.BuildStats
	MintedWorks     int
	MintedInstances int
	Retired         int
	Conflicts       []string
	// WorkIDs are the Works this run's records resolved to -- the presence
	// set the feed reconciliation pass (tasks/078) diffs the corpus against.
	WorkIDs []string
}

Result is what one ingest run produced: the grain/instance counts, how many ids were freshly minted at each tier, how many merge-retired grains were dropped, and any resolver conflicts to surface. The pipeline returns this rather than printing so the CLI (or a cloud handler) owns presentation.

func Run

func Run(prov Provider, out string) (Result, error)

Run ingests a provider's records into canonical grains under out, the shared direct-BIBFRAME pipeline every ingest provider uses (ARCHITECTURE §4/§9). It seeds the resolver from any grains already under out (so re-ingest reuses ids and clusters editions), applies committed editorial merges/pins, resolves each record to stable two-tier ids, groups records into Works (first record wins shared metadata; a duplicate Instance -- e.g. a shared ISBN -- is emitted once), carries each Work's preserved editorial statements across the feed rewrite, writes one grain per Work plus catalog.nq, and drops the grains of merge-retired Works. The run is deterministic in record order. Only ingest-role providers are executed.

func RunStore

func RunStore(ctx context.Context, prov Provider, st blob.Store, prefix string) (Result, []string, error)

RunStore is Run over a blob.Store (tasks/050): the same prior-load / resolve / cluster / build pipeline, with grains written under ETag optimistic concurrency so a concurrent editorial save is never clobbered -- the copy-cataloging commit path. Unchanged grains are not rewritten, so re-committing identical records is byte-stable and touches nothing. It does not write catalog.nq: store-backed deployments regenerate the bulk artifacts from grains via the rebuild trigger. Returns the changed (or removed) grain paths for that trigger.

type Role

type Role int

Role is what a provider contributes to the graph, which fixes the provenance named graph its statements land in (ARCHITECTURE §5/§9).

const (
	// RoleIngest is a bibliographic source: its records are resolved and clustered
	// into Works, landing in the feed:<name> graph. OverDrive and MARC are ingest
	// providers.
	RoleIngest Role = iota
	// RoleEnrich is a provider that adds statements about existing entities
	// (authority labels, clustering hints, embeddings). Its statements land in the
	// editorial: or enrichment: graph, not feed:. Reserved: Run handles ingest
	// providers today; enrichment execution is future work.
	RoleEnrich
)

func (Role) String

func (r Role) String() string

String renders a Role for messages.

type SubjectEnricher

type SubjectEnricher interface {
	ControlledSubjects() []AuthoritySubject
}

SubjectEnricher is an optional capability a Record may implement to contribute controlled-vocabulary subjects for its Work -- e.g. by promoting free genre tags through an authority table (tasks/026). Run emits each as a bf:subject link to the authority URI plus its skos:prefLabel/broader statements in the feed graph, so the projector resolves them as controlled subjects with labels and hierarchy (tasks/012/015). For a clustered Work the first record's subjects win, matching how shared Work metadata and extras are taken; a Record that does not implement it contributes none.

type SummarySource

type SummarySource interface {
	SummariesWithPaths(ctx context.Context) ([]WorkSummary, map[string]string, error)
}

SummarySource yields the corpus's WorkSummaries plus each Work's grain path without a fresh corpus walk -- the seam a maintained index (the backend's workindex, tasks/106/109) plugs into where workers would otherwise each run their own ScanSummaries. Both return values are shared, read-only views.

type TermDescriber

type TermDescriber interface {
	DescribedTerms() []AuthoritySubject
}

TermDescriber is an optional capability a Record may implement to carry standalone vocabulary term descriptions -- typically the skos:broader ancestor chains of its ControlledSubjects (tasks/180). Run emits each as the term's skos:prefLabel/broader statements in the feed graph with NO bf:subject link to the Work, so hierarchy nodes above the asserted subjects stay labeled in the projection's term sideband (tasks/178) even though no Work carries them directly. For a clustered Work the first record's terms win, matching ControlledSubjects.

type VerbatimProvider

type VerbatimProvider interface {
	Verbatim() []string
}

VerbatimProvider is an optional capability a Record may implement to carry the crosswalk-lossy MARC fields of its source record, serialized field-exact via bibframe.EncodeVerbatimField. Run writes them into the Instance's feed graph under bibframe.PredMARCVerbatim, so a lossy tag is preserved verbatim rather than silently dropped, and MARC export / the MARC view reproduce it (tasks/049).

type WorkSummary

type WorkSummary struct {
	WorkID       string
	Title        string
	Contributors []string
	ISBNs        []string
	// Tags are the Work's uncontrolled subject labels (feed genres,
	// approved folksonomy) -- the raw material tag-to-controlled-term
	// reconciliation enrichers match against.
	Tags []string
	// Subjects are the Work's controlled subject IRIs (bf:subject with an
	// IRI object, any graph) -- what authority merges rewrite (tasks/046).
	Subjects []string
	// Series and Languages are the similarity scorer's two strongest signals
	// (tasks/284): two books in one series are related whatever else they
	// share, and a reader of one language is rarely served a book in another.
	// Series holds the Work's series statements, read off its bf:relation
	// series relations (tasks/309); Languages are bf:language local names ("en").
	Series    []string `json:",omitempty"`
	Languages []string `json:",omitempty"`
	// Visibility and holdings signals for the admin works list (tasks/078):
	// the editor deliberately shows everything, so each row says what the
	// public projection would do with it.
	Suppressed bool   `json:",omitempty"`
	Tombstoned bool   `json:",omitempty"`
	Withdrawn  string `json:",omitempty"` // date the feed reconciliation flagged it
	Kept       bool   `json:",omitempty"` // curator keeps it despite withdrawal
	// Items counts physical holdings across the Work's instances;
	// HasAvailability reports a live-availability identifier (a digital
	// holding as long as the Work is not withdrawn).
	Items           int  `json:",omitempty"`
	HasAvailability bool `json:",omitempty"`
	// Extras carries the Work's lcat:extra/* literals -- deployment-defined
	// key/value metadata the feed graph attaches (bibframe.addWorkExtras),
	// e.g. sources: "loc, mombian". The admin works view facets provenance
	// on configured keys (tasks/171); multi-valued keys are comma-joined by
	// convention and split at facet time.
	Extras map[string]string `json:",omitempty"`
}

WorkSummary is the slice of a Work an enricher reasons over: enough for a vocabulary lookup or reconciliation call without handing over the graph.

func ScanSummaries

func ScanSummaries(ctx context.Context, st blob.Store, prefix string) ([]WorkSummary, map[string]string, error)

ScanSummaries walks the grain tree and extracts a WorkSummary per Work, plus each Work's grain path.

func SummariesOf

func SummariesOf(ctx context.Context, src SummarySource, st blob.Store, prefix string) ([]WorkSummary, map[string]string, error)

SummariesOf reads summaries from src when one is wired, falling back to a fresh ScanSummaries walk of prefix.

func SummarizeDataset

func SummarizeDataset(ds *rdf.Dataset) []WorkSummary

SummarizeDataset is SummarizeGrain for callers that already hold the parsed dataset (the work index scans identity, summaries, and barcodes off one parse).

func SummarizeGrain

func SummarizeGrain(grain []byte) ([]WorkSummary, error)

SummarizeGrain extracts the WorkSummaries a grain carries (post-merge grains can hold several Works). Exported for callers that already hold the grain bytes, like the on-save authority auto-linker (tasks/046).

func (WorkSummary) Matches

func (s WorkSummary) Matches(q string) bool

Matches reports whether the summary matches a lowercased search query -- substring over title, id, contributors, tags, and ISBNs. One matcher serves the works listing and batch search selections (tasks/047), so a saved query means the same thing everywhere.

func (WorkSummary) SimilarWork

func (s WorkSummary) SimilarWork() similar.Work

SimilarWork converts an admin-side summary into the similarity scorer's input (tasks/284). It is one of exactly two converters; project.Work.SimilarWork is the other, and a test drives both from the same graph and requires equal results, because an OPAC rail and an admin panel that disagree about what a Work resembles is a bug the reader can see and nobody can explain.

Suppressed Works are kept. Suppression hides a Work from the public projection, which never reaches the scorer at all; the admin surface shows suppressed Works and so must recommend them. Tombstoned Works are passed through with the flag set and excluded by similar.Build -- retiring a record must not leave it recommended from elsewhere (tasks/280).

Held collapses the two digital/physical signals the summary carries separately into the one predicate project.Work.Held already publishes.