project

package
v0.229.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 14, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package project turns the canonical BIBFRAME graph into the catalog's derived data: the JSON a static site (the Hugo module's content adapter) and the search index consume (ARCHITECTURE §7). It is a read-only projection -- the graph stays the source of truth -- and it flattens each clustered Work, with its Instances and the union of its feed and editorial statements, into one record.

Index

Constants

View Source 
const SchemaVersion = 12

SchemaVersion is the catalog.json / facets.json / redirects.json schema version. The Hugo module and search-index builder read it to detect a projector/consumer mismatch. v2 added the per-Instance identifier scheme (ProviderID.Source) for the availability adapter. v3 split controlled subjects (authority URIs + resolved labels) from uncontrolled feed tags. v4 added per-Instance format (from the Instance's RDA media type) and the Work-level formats facet, so a clustered mixed-format Work exposes each format. v5 added subject skos:broader parents (Subject.Broader / SubjectFacet.Broader) so consumers render vocabulary hierarchy without re-reading the graph. v6 added the holdings signal (Instance.Held / Work.Held): physical items or a live-availability identifier whose feed still lists the Work. v7 added Work.Summary from bf:summary -- the description/abstract as first-class bibliographic data rather than an adopter extra. v8 added Subject.Scheme / SubjectFacet.Scheme -- the short vocabulary code derived from the authority-URI namespace -- so a multi-vocabulary corpus can facet and mint term pages per scheme instead of colliding same-label terms from different vocabularies. v9 made classifications {value, label} objects (Work.Classifications / Facets.Classifications): value stays the scheme code (what MARC 084 $a carries), label is the human text riding the classification node's rdfs:label -- the display-only channel -- so a facet can show "Fiction / Romance / Contemporary" while exports keep FIC027000. The code stays verbatim here on purpose. A Dewey number's prime mark (082 $a "813/.6") is data a MARC export must round-trip, so it is not the projector's to strip -- but a slash cannot survive being a URL path segment, so the Hugo module keys the classification taxonomy by the code's slug and shows the code . The URL needed the slug, not the record. v10 added Catalog.Terms -- the vocabulary sideband: every referenced subject term plus its transitive skos:broader ancestors, with whatever labels and broader edges the graph carries -- so a consumer can label hierarchy nodes no Work carries directly instead of minting them label-less. v11 added Work.Relations -- the editorial whole/part links (hasPart/partOf as {id, title}, restricted to works present in this projection) -- and the Instance series statement/enumeration (bf:seriesStatement 490$a, bf:seriesEnumeration 490$v), so the site cross-links parts and shows series lines. v12 moved series from the Instance to the Work and made them objects: Work.Series []{title, enumeration, issn, traced} replaces Instance.Series []string + Instance.SeriesEnumeration. libcodex v0.25.0 hangs one bf:relation per 490 on the Work, so each enumeration belongs to its own series instead of being paired by list position -- a pairing an RDF graph, being a set, could not preserve. 490$x (ISSN) and ind1=1 (traced) are carried for the first time .

View Source 
const SimilarSchemaVersion = 1

SimilarSchemaVersion is similar.json's own schema version.

It is deliberately not SchemaVersion. That number exists so a consumer can detect a projector/consumer mismatch in catalog.json, facets.json and redirects.json -- artifacts the Hugo module cannot render without. similar.json is optional on both sides: a module built before it ignores the file, and a module built after it renders no rail when the file is absent. Bumping SchemaVersion would force every adopter into a lockstep reproject to announce a mismatch that cannot occur.

Variables

View Source 
var SubjectSchemePrefixes = []SchemePrefix{
	{"https://homosaurus.org/", "homosaurus"},
	{"http://homosaurus.org/", "homosaurus"},
	{"http://id.worldcat.org/fast/", "fast"},
	{"https://id.worldcat.org/fast/", "fast"},
	{"http://id.loc.gov/authorities/subjects/", "lcsh"},
	{"https://id.loc.gov/authorities/subjects/", "lcsh"},
	{"http://id.loc.gov/authorities/childrensSubjects/", "lcshac"},
	{"https://id.loc.gov/authorities/childrensSubjects/", "lcshac"},
	{bibframe.LocalAuthorityNS, "local"},
}

SubjectSchemePrefixes is the namespace -> scheme table SchemeForURI consults, first match wins. `lcat project --subject-scheme` prepends deployment-specific entries, so a custom authority (or a different code for a known one) overrides these defaults.

Functions

func Feeds added in v0.99.0 

func Feeds(catalogNQ []byte) ([]string, error)

Feeds lists the provider names that appear as feed graphs in a catalog.nq, sorted. It answers "what could this catalog be projected as?", which is the question a caller needs when a requested provider projects nothing: the difference between a catalog that is empty and a catalog whose feeds simply were not asked for.

func FeedsDataset added in v0.131.0 

func FeedsDataset(ds *rdf.Dataset) []string

FeedsDataset is Feeds over an already-parsed dataset.

func LoadDataset added in v0.131.0 

func LoadDataset(path string) (*rdf.Dataset, error)

LoadDataset streams the N-Quads catalog at path and returns a dataset holding every work, editorial and alias quad, plus only the authority quads the projection actually reads.

The projection consumes an authority graph through exactly three indexes: buildLabelIndex (skos:prefLabel, rdfs:label), buildBroaderIndex (skos:broader) and termSideband's closure over the latter. Nothing reads skos:altLabel, skos:narrower, skos:related or the type triples -- together 45% of the quads in a corpus with LCSH installed -- and nothing reads a term no Work references, which is 99.99% of the rest. mergedView is graph-scoped to feed + editorial, so the authority quads never reach it at all.

So: keep the work graphs whole, and from the authority graphs keep only the label and broader quads of terms the works reach, transitively. On the demo playground that is 98 terms out of ~450,000, and the file is read twice rather than parsed once into a dataset that is thrown away.

Read twice, not slurped once. `os.ReadFile` on a 264MB catalog puts 264MB in the heap before a quad is parsed, and ParseNQuadsShared then keeps it resident for the life of the dataset because its terms alias that buffer. Two streaming passes cost one more read of a file the page cache already holds.

func SanitizeExtras added in v0.127.0 

func SanitizeExtras(cat *Catalog, public map[string]bool) int

SanitizeExtras drops every Work extra whose key is not on the public allowlist, and returns how many values it removed.

`lcat:extra/*` is the adopter passthrough, so it carries whatever the ingest mapping put there -- including institution-private holdings ("this library already has it"), acquisition flags, and internal notes. Those belong in the grains, which is where cataloguers work, and not on the public face. The two faces project from the same grains by design, so the only place to draw the line is here.

"sources" is exempt: SanitizeSources governs it, filtering *within* the value rather than dropping the key, and running both would let public-extras silently undo a configured public-sources allowlist.

The grains are untouched. This is a projection-time filter, exactly as SanitizeSources is.

func SanitizeSources added in v0.30.0 

func SanitizeSources(cat *Catalog, public map[string]bool) int

SanitizeSources rewrites every Work's extra "sources" attribution list to the public allowlist, dropping the key when nothing public remains, and returns how many attributions were stripped. Provenance under lcat:extra/sources may name sources whose attribution must not leak on a public surface; the same allowlist governs the nq download (export package). Values are comma-joined by convention and compared trimmed; kept values are re-joined ", " and keep their order.

func SchemeForURI 

func SchemeForURI(uri string) string

SchemeForURI returns the short vocabulary code for an authority URI, or "" when the namespace is not in SubjectSchemePrefixes.

func SourceSet added in v0.30.0 

func SourceSet(csv string) map[string]bool

SourceSet parses a comma-separated name list into an allowlist set. It backs both `public-sources` (source attributions, filtered by SanitizeSources and the export nq filter) and `public-extras` (extra keys, filtered by SanitizeExtras). Names are trimmed; empty entries are ignored, so "" yields an empty (strip-everything) set.

Types

type Catalog 

type Catalog struct {
	Version int    `json:"version"`
	Works   []Work `json:"works"`
	// Terms is the vocabulary sideband: one entry per referenced
	// subject term or transitive skos:broader ancestor that the graph
	// describes (labels and/or broader edges; bare URIs with nothing to say
	// are skipped). Sorted by ID.
	Terms []Term `json:"terms,omitempty"`
}

Catalog is the projected corpus: one record per Work, sorted by id.

func Merge added in v0.30.0 

func Merge(cats []*Catalog) *Catalog

Merge unions per-feed projections by work id. Project views one provenance graph at a time (feed:<provider> + editorial:), so a multi-feed corpus projects each feed separately and merges here: earlier catalogs win a shared id -- list the primary feed first, its records are richer than a sidecar's -- and the result is sorted by id like Project's output. The input catalog.nq must cover every feed's works; after a multi-feed ingest, `lcat serialize` regenerates it, since each ingest run rewrites catalog.nq with only its own run's works.

The vocabulary sideband (Catalog.Terms) merges by term id, field-wise: labels fill per language (earlier catalogs win a language, like works win an id), broader edges union, the first non-empty scheme sticks -- per-feed projections describe the same authority term from different coverage, so the union is at least as rich as any one input.

func Project 

func Project(catalogNQ []byte, provider string) (*Catalog, error)

Project reads a catalog.nq dataset and projects each Work into a Catalog record. Display/facet fields are drawn from the union of the provider's feed graph and the editorial graph, so curated subjects appear alongside feed data. Editorial lcat:overrides markers shadow the feed first: a property a cataloger claimed shows only its editorial values, and deleting the marker resurfaces the feed untouched.

catalogNQ must not be modified while the returned Catalog is in use: the parse is zero-copy (ParseNQuadsShared) -- one input-sized allocation saved at corpus scale -- so projected strings alias the buffer.

func ProjectDataset added in v0.131.0 

func ProjectDataset(ds *rdf.Dataset, provider string) *Catalog

ProjectDataset is Project over an already-parsed dataset. A multi-feed projection views one provenance graph at a time, so it used to reparse the whole catalog once per feed -- five full parses of a 1.76M-quad corpus for three feeds plus Feeds and Redirects. Pair it with LoadDataset.

func (*Catalog) Facets 

func (c *Catalog) Facets() Facets

Facets aggregates the catalog into per-dimension value counts, each value counted once per Work. Values are ordered by descending count then value, so the output is deterministic.

func (*Catalog) Similar added in v0.118.0 

func (c *Catalog) Similar(limit int) *SimilarIndex

Similar computes each Work's neighbours and returns the similar.json sidecar. limit <= 0 yields an empty index rather than an unbounded one.

The pass is parallel across Works. Neighbors is read-only on a built Index, so this is embarrassingly parallel, and it needs to be: at 62,602 Works a serial pass is ~84 s and ~43 GB of allocation churn, on top of a projector that already peaks near 2 GB. Build itself stays serial -- it is 44 ms.

func (*Catalog) SimilarOptions added in v0.118.0 

func (c *Catalog) SimilarOptions() similar.Options

SimilarOptions is the scorer configuration the projector uses: the defaults, wired to the catalog's own concept tree.

func (*Catalog) SimilarWorks added in v0.118.0 

func (c *Catalog) SimilarWorks() []similar.Work

SimilarWorks converts the catalog, preserving order.

type Classification 

type Classification struct {
	Value string `json:"value"`
	Label string `json:"label,omitempty"`
}

Classification is one classification of a Work: the scheme code (bf: classificationPortion -- what MARC 084 $a carries) plus the optional human label riding the classification node's rdfs:label. Facets and term pages key on Value; display prefers Label and falls back to Value, so a corpus without labels renders exactly as before.

type ClassificationFacet 

type ClassificationFacet struct {
	Value string `json:"value"`
	Label string `json:"label,omitempty"`
	Count int    `json:"count"`
}

ClassificationFacet is one classification facet value: the scheme code (the key), the optional human label (see Classification), and the number of Works carrying it.

type Contributor 

type Contributor struct {
	Name string `json:"name"`
	Role string `json:"role,omitempty"`
}

Contributor is an agent's display name and role.

type FacetValue 

type FacetValue struct {
	Value string `json:"value"`
	Count int    `json:"count"`
}

FacetValue is one facet value and the number of Works that carry it.

type Facets 

type Facets struct {
	Version         int                   `json:"version"`
	Languages       []FacetValue          `json:"languages,omitempty"`
	Subjects        []SubjectFacet        `json:"subjects,omitempty"`
	Tags            []FacetValue          `json:"tags,omitempty"`
	Formats         []FacetValue          `json:"formats,omitempty"`
	Contributors    []FacetValue          `json:"contributors,omitempty"`
	Classifications []ClassificationFacet `json:"classifications,omitempty"`
}

Facets is the precomputed facet index: for each facetable dimension, the distinct values and how many Works carry each. Emitting it saves the static site from aggregating the whole corpus in templates at build time.

type Instance 

type Instance struct {
	ID          string       `json:"id"`
	Format      string       `json:"format,omitempty"`
	ISBNs       []string     `json:"isbns,omitempty"`
	ProviderIDs []ProviderID `json:"providerIds,omitempty"`
	// Items are the Instance's physical holdings: call number,
	// shelving location, barcode, note -- never circulation state, which
	// stays live-only (ARCHITECTURE §5).
	Items []Item `json:"items,omitempty"`
	// Held is true when this Instance has >=1 item (physical) or a
	// live-availability identifier whose feed still lists the Work (digital,
	// unless the reconciliation flagged the Work withdrawn).
	Held bool `json:"held,omitempty"`
	// Publication statement transcribed from MARC 260/264: the
	// agent ($b), the place of publication ($a) and the date. These are
	// per-Instance -- a Work's 2020 hardback and 2022 ebook differ -- so they
	// render on the edition line, not the Work-level details.
	Publisher string `json:"publisher,omitempty"`
	Place     string `json:"place,omitempty"`
	Published string `json:"published,omitempty"`
}

Instance is one edition/format: its id, format (from its RDA media type), ISBNs, and the scheme-tagged provider ids the runtime availability adapter keys on.

type Item 

type Item struct {
	CallNumber string `json:"callNumber,omitempty"`
	Location   string `json:"location,omitempty"`
	Barcode    string `json:"barcode,omitempty"`
	Note       string `json:"note,omitempty"`
}

Item is one holding of an Instance (the minimal bf:Item model).

type ProviderID 

type ProviderID struct {
	Source string `json:"source,omitempty"`
	Value  string `json:"value"`
}

ProviderID is one non-ISBN identifier with its bf:source scheme, so a client-side availability adapter selects its key by scheme (e.g. OverDrive's "overdrive-reserve" Reserve ID vs the "overdrive" title id) rather than guessing from a flat list (ARCHITECTURE §9). Source is empty for an untagged identifier.

Every scheme is projected; the hugo module decides which ones reach the DOM, via data/lcat/availabilityAttrs.toml. The schemes its bundled adapters resolve are "overdrive-reserve" (OverDrive Reserve ID) and "daia" (DAIA document id, e.g. "ppn:12345"). A scheme with no adapter is still projected and simply never queried.

type Redirect 

type Redirect struct {
	From string `json:"from"`
	To   string `json:"to"`
}

Redirect is one retired-Work -> surviving-Work URL redirect (ARCHITECTURE §4): after an editorial merge the retired id must still resolve, so shared links and SEO survive.

type RedirectMap 

type RedirectMap struct {
	Version   int        `json:"version"`
	Redirects []Redirect `json:"redirects"`
}

RedirectMap is the redirect artifact emitted alongside catalog.json: every retired Work id and the surviving id it now resolves to, chains collapsed to the final survivor and sorted by retired id. An empty To is a tombstone -- retired with no successor.

Per the decision the projector emits the map and the host serves it. The host is the Hugo module, which publishes this file to /redirects.json and mints a meta-refresh stub for each merged id, and `lcat serve`, which reads the published map and answers 301 for a merge and 410 for a tombstone. Unlike catalog.json and facets.json this artifact is read at request time, so it must reach the served tree. Until nothing published it, nothing read it, and every retired permalink answered a bare 404.

func Redirects 

func Redirects(catalogNQ []byte) (RedirectMap, error)

Redirects builds the redirect map from a catalog.nq dataset by reading the editorial graph's lcat:mergedInto statements and collapsing merge chains (A->B->C yields A->C and B->C) to the final survivor. A merge cycle terminates at the last id reached rather than looping.

catalogNQ must not be modified while the returned map is in use: the parse is zero-copy (ParseNQuadsShared), so ids alias the buffer.

func RedirectsDataset added in v0.131.0 

func RedirectsDataset(ds *rdf.Dataset) RedirectMap

RedirectsDataset is Redirects over an already-parsed dataset, so a caller that projects several feeds parses the catalog once rather than once per feed .

type RelatedWork added in v0.72.0 

type RelatedWork struct {
	ID    string `json:"id"`
	Title string `json:"title,omitempty"`
}

RelatedWork is one linked work: its catalog id and display title.

type Relations added in v0.72.0 

type Relations struct {
	HasPart []RelatedWork `json:"hasPart,omitempty"`
	PartOf  []RelatedWork `json:"partOf,omitempty"`
}

Relations are a Work's whole/part cross-links (schema v11).

type SchemePrefix 

type SchemePrefix struct{ Prefix, Scheme string }

SchemePrefix maps an authority-URI namespace prefix to its short vocabulary code.

type Series added in v0.129.0 

type Series struct {
	// Title is the transcribed series statement (490$a, bf:mainTitle on the
	// bf:Series).
	Title string `json:"title"`
	// Enumeration is this Work's volume/part within the series (490$v).
	Enumeration string `json:"enumeration,omitempty"`
	// ISSN is the series ISSN (490$x), which the flat mapping silently dropped.
	ISSN string `json:"issn,omitempty"`
	// Traced marks a series the cataloger traced (490 ind1=1), i.e. one with a
	// corresponding added entry. Carried as bf:status mstatus/tr.
	Traced bool `json:"traced,omitempty"`
}

Series is one series membership: the transcribed statement, this Work's place in it, and the series ISSN.

Enumeration belongs to the *relation*, not to the series, which is why it can finally be per-series rather than one-per-Instance: "bk. 2 of Firebrand fiction" is a fact about this Work's membership, and the same series numbers a different Work differently.

type SimilarIndex added in v0.118.0 

type SimilarIndex struct {
	Version int `json:"version"`
	// Limit is how many neighbours were kept per Work, so a consumer can tell a
	// short rail from a truncated one.
	Limit int                          `json:"limit"`
	Works map[string][]SimilarNeighbor `json:"works"`
}

SimilarIndex is the similar.json sidecar: for each Work, its ranked neighbours. Works with no neighbours are omitted rather than stored empty.

type SimilarNeighbor added in v0.118.0 

type SimilarNeighbor struct {
	ID    string `json:"id"`
	Title string `json:"title"`
	// Shared lists the attribute values that put this Work on the list, most
	// significant first, capped by the scorer. A neighbour reached only through
	// the concept tree or a flat bonus can legitimately share nothing verbatim.
	Shared []string `json:"shared,omitempty"`
}

SimilarNeighbor is one recommended Work: enough to render a card without a second lookup, plus the shared attributes that answer the only question a librarian asks about a recommendation, which is "why is this here?".

type Subject 

type Subject struct {
	ID      string            `json:"id"`
	Labels  map[string]string `json:"labels,omitempty"`
	Broader []string          `json:"broader,omitempty"`
	// Scheme is the short vocabulary code derived from the URI's namespace
	// (SchemeForURI) -- "homosaurus", "fast", "lcsh", "local";
	// empty for an unrecognized authority.
	Scheme string `json:"scheme,omitempty"`
}

Subject is a controlled-vocabulary subject: a stable authority URI plus the human labels resolved from the authority's skos:prefLabel / rdfs:label statements in the graph, keyed by language tag (e.g. "en", "es"; "" for an untagged label). Links and facets key on ID; display uses Labels, falling back to ID when the authority provides none. Distinct from an uncontrolled feed Tag.

Broader holds the authority URIs of this term's skos:broader parents (sorted, deduped), so a consumer can render vocabulary hierarchy (breadcrumb trails, broader/narrower drill-down) without re-reading the graph. It is id-only: a parent's label resolves from the parent's own Subject/authority record.

type SubjectFacet 

type SubjectFacet struct {
	ID      string            `json:"id"`
	Labels  map[string]string `json:"labels,omitempty"`
	Broader []string          `json:"broader,omitempty"`
	// Scheme is the vocabulary code (see Subject.Scheme) so a consumer
	// renders one facet group per vocabulary.
	Scheme string `json:"scheme,omitempty"`
	Count  int    `json:"count"`
}

SubjectFacet is one controlled-subject facet value: the authority URI (the key), its resolved labels, its skos:broader parents (for hierarchy-aware facet drill-down), and the number of Works carrying it. Facets key on ID so a relabel does not churn the facet; display uses Labels.

type Term added in v0.34.0 

type Term struct {
	ID      string            `json:"id"`
	Labels  map[string]string `json:"labels,omitempty"`
	Broader []string          `json:"broader,omitempty"`
	Scheme  string            `json:"scheme,omitempty"`
}

Term is one controlled-vocabulary concept the catalog references -- a Work's subject or one of its skos:broader ancestors -- with the labels and broader edges resolved from the graph. The sideband exists for hierarchy nodes no Work carries directly: the browse-artifact builder unions subtree postings into ancestors (search.BuildBrowse), and without this it can only mint them label-less.

type Work 

type Work struct {
	ID       string `json:"id"`
	Title    string `json:"title"`
	Subtitle string `json:"subtitle,omitempty"`
	// Summary is the Work's description/abstract (bf:summary), first label wins
	// when a record carries several.
	Summary         string           `json:"summary,omitempty"`
	Contributors    []Contributor    `json:"contributors,omitempty"`
	Subjects        []Subject        `json:"subjects,omitempty"`
	Tags            []string         `json:"tags,omitempty"`
	Languages       []string         `json:"languages,omitempty"`
	Classifications []Classification `json:"classifications,omitempty"`
	// Formats is the union of the Work's Instances' formats (e.g. ebook, audiobook),
	// so a clustered mixed-format Work is faceted under each format it offers.
	Formats   []string   `json:"formats,omitempty"`
	Instances []Instance `json:"instances,omitempty"`
	// Held is true when any Instance is held: physical items, or
	// a live-availability identifier whose feed still lists the Work. Whether
	// unheld Works are hidden or merely badged is the importing site's call.
	Held bool `json:"held,omitempty"`
	// Extra holds the Work's non-BIBFRAME adopter display fields (e.g. cover, rating,
	// dateRead) a provider carried through the feed graph under bibframe.ExtraPred
	//. The Hugo module forwards it to page params. Omitted (nil)
	// when the corpus carries none, so a catalog without extras is unchanged.
	Extra map[string]string `json:"extra,omitempty"`
	// Relations are the Work's editorial whole/part links,
	// restricted to works present in this projection -- a link to a
	// suppressed, tombstoned, or foreign work is omitted rather than
	// rendered as a dead cross-link.
	Relations *Relations `json:"relations,omitempty"`
	// Series are the Work's series memberships, one per 490.
	// Work-level, because that is where libcodex >= v0.25.0 hangs them: a
	// bf:relation whose bf:relationship is .../relationship/series. They used
	// to be flat literals on the Instance, which paired a statement to its
	// enumeration by list position -- and an RDF graph is a set, so two 490s
	// sharing a $v collapsed to one triple and the pairing died.
	Series []Series `json:"series,omitempty"`
}

Work is the discovery unit as the static site sees it -- the display and facet fields of a bf:Work plus its Instances (the borrowable editions/formats).

func (Work) SimilarWork added in v0.118.0 

func (w Work) SimilarWork() similar.Work

SimilarWork converts a projected Work into the similarity scorer's input . It is one of exactly two converters -- ingest.WorkSummary.SimilarWork is the other -- and a test drives both from the same graph and requires equal results, so the OPAC's precomputed rail and the admin's live panel cannot drift.

Series is transcribed per Instance and hoisted to the Work, matching what the admin summary does. Tombstoned is always false: the projection drops retired Works upstream and never sees one.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.