nquads

Documentation

Overview

Package nquads is the generic mapped N-Quads ingest provider: it streams a dcterms-shaped .nq export into ingest records driven entirely by a declarative TOML mapping -- work-IRI prefix, predicate->field map, identifier URN schemes, and source-attestation tiers -- so a deployment sideloads an RDF export the way Aspen Discovery sideloads MARC: with a profile, not code. Works sharing identifier keys (e.g. ISBNs) with a primary feed merge in the shared clustering pipeline; unshared works mint as their own. Generalized from the queerbooks-demo collnq provider; extended it to the full coll-feed contract (per-format bucket grouping, contributions with roles, provisions, formats, topic tags, classifications, extras passthrough, non-key identifiers, and standalone term descriptions for the vocabulary sideband).

Index

• Constants
• func New(cfg ingest.Config) (ingest.Provider, error)
• type ClassificationMapping
• type IdentifierRule
  • func (r *IdentifierRule) UnmarshalTOML(v any) error
• type Mapping
  • func LoadMapping(path string) (*Mapping, error)
• type Provider
  • func (p *Provider) MergeSeeds() []ingest.MergeSeed
  • func (p *Provider) Name() string
  • func (p *Provider) Records(ctx context.Context) ([]ingest.Record, error)
  • func (p *Provider) Role() ingest.Role
• type SourcesMapping
• type StringList
  • func (s *StringList) UnmarshalTOML(v any) error

Constants

const ProviderName = "nquads"

ProviderName is the registry key and default provenance feed (feed:nquads).

Functions

func New

func New(cfg ingest.Config) (ingest.Provider, error)

New builds the provider from an ingest.Config: Source is the .nq path, Params["mapping"] the mapping TOML path, Feed overrides the provenance feed name, and Params["tentative"]="drop" drops works whose only attestation is a tentative source instead of ingesting them.

Types

type ClassificationMapping

type ClassificationMapping struct {
	// Prefix is the object-IRI prefix stripped to form the code value.
	Prefix string `toml:"prefix"`
	// Source is the bf:source scheme code (e.g. "bisacsh").
	Source string `toml:"source"`
}

ClassificationMapping maps "classification" predicate objects -- coded IRIs like urn:bisac:FIC000000 -- to BIBFRAME classifications: Value is the IRI minus Prefix, Label the IRI's harvested skos:prefLabel, Source the scheme code.

type IdentifierRule

type IdentifierRule struct {
	// Scheme is the legacy string form: "isbn" makes the value a cross-feed
	// ISBN merge key; any other scheme emits "<scheme>:<value>" as both a
	// SchemeID resolution key and a source-tagged Instance identifier.
	Scheme string
	// Class is the emitted identifier class: "Isbn" or "Identifier"
	// (default). Table form only.
	Class string
	// Source is the emitted bf:source tag; also the key namespace when Key
	// is true. Table form only.
	Source string
	// Key makes the identifier a resolution key: an "Isbn"-class value
	// merges cross-feed, any other joins as "<source>:<value>". Table form
	// only (the legacy string form is always a key).
	Key bool
}

IdentifierRule is one identifier prefix's handling. Exactly one of the two forms applies: the legacy string form sets Scheme (always a resolution key); the table form sets Class/Source/Key.

func (*IdentifierRule) UnmarshalTOML

func (r *IdentifierRule) UnmarshalTOML(v any) error

UnmarshalTOML implements toml.Unmarshaler for the string-or-table union.

type Mapping

type Mapping struct {
	// WorkPrefix is the IRI prefix of work subjects; the remainder is the
	// export's work id.
	WorkPrefix string `toml:"work-prefix"`
	// IDScheme namespaces the durable provider id ("<scheme>:<workid>")
	// minted for each work. Defaults to the feed name; keep it stable across
	// exports or every work re-mints.
	IDScheme string `toml:"id-scheme"`
	// Class is the BIBFRAME work class (default "Text").
	Class string `toml:"class"`
	// DefaultLanguage is the ISO 639-2/B code used when the export carries no
	// (mappable) language (default "eng").
	DefaultLanguage string `toml:"default-language"`
	// IdentityLanguage, when set, fixes the language in every record's
	// resolution identity (identity.WorkKey folds author|title|LANG) while
	// Work.Languages keeps the export's real values. Set it when replacing a
	// provider that keyed everything under one language, so freshly-minted
	// WorkIDs match the old provider's; leave empty to key by
	// the record's language.
	IdentityLanguage string `toml:"identity-language"`
	// IDOrder orders records for deterministic ingest: "lexical" (default) or
	// "numeric" (decimal ids without padding, shorter ids first).
	IDOrder string `toml:"id-order"`
	// Predicates maps record fields to the predicate IRIs that carry them.
	// Work-level fields: title, subtitle, creator, contributor, summary,
	// subject, tag, keyword, classification, source, language, group.
	// Bucket-level fields: identifier, publisher, issued, format.
	// Term-description fields (non-work subjects): prefLabel, broader.
	// A field may list several IRIs.
	//
	// A creator/contributor OBJECT may be a literal name or an agent NODE:
	// when the feed also describes the node (rdfs:label, plus owl:sameAs /
	// skos:exactMatch authority links), the label becomes the name and the
	// authority IRIs ride onto the contribution -- the agent emits as the
	// preferred IRI (VIAF > LCNAF > ISNI > ORCID > wikidata), the rest as
	// owl:sameAs, all read into ContributorIDs by the summary scan. No
	// mapping rule is needed; the description is harvested automatically.
	Predicates map[string]StringList `toml:"predicates"`
	// Identifiers maps object URN prefixes to identifier rules. The legacy
	// string form keeps its meaning: scheme "isbn" clusters cross-feed, any
	// other scheme becomes a durable source-tagged id key
	// ("<scheme>:<value>"). The table form ({class, source, key}) covers
	// non-key identifiers -- display ISBNs, ASINs, availability ids -- that
	// must ride the Instance without becoming resolution keys.
	Identifiers map[string]IdentifierRule `toml:"identifiers"`
	// Languages maps the export's language codes to ISO 639-2/B. An unmapped
	// three-letter code passes through; anything else falls back to
	// DefaultLanguage.
	Languages map[string]string `toml:"languages"`
	// KeywordSource is the bf:source on "keyword" topic subjects (e.g.
	// "overdrive"); "tag" topics always carry none.
	KeywordSource string `toml:"keyword-source"`
	// ExtrasPrefix, when set, harvests every work statement whose PREDICATE
	// starts with it as a display extra: key = the predicate remainder,
	// value = the literal, verbatim. First statement wins a key.
	// The [sources] mechanism owns its extra key; on a collision it wins.
	ExtrasPrefix string `toml:"extras-prefix"`
	// Classifications describes the "classification" field's objects.
	Classifications ClassificationMapping `toml:"classifications"`
	// Sources describes provenance attestation objects.
	Sources SourcesMapping `toml:"sources"`
}

Mapping declares how a deployment's N-Quads export maps onto ingest records -- the whole provider config, written as a TOML file so adopting a new dcterms-shaped export means editing a mapping, not writing Go. Only subjects under WorkPrefix are read as work records; skos prefLabel and broader statements on any other subject are harvested as authority-term descriptions for the works' controlled subjects (and their ancestor chains).

func LoadMapping

func LoadMapping(path string) (*Mapping, error)

LoadMapping reads and validates a mapping TOML file.

type Provider

type Provider struct {
	// contains filtered or unexported fields
}

Provider streams a catalog .nq file into ingest records, one per work IRI.

func (*Provider) MergeSeeds

func (p *Provider) MergeSeeds() []ingest.MergeSeed

MergeSeeds implements ingest.MergeSeeder: the feed's cluster-merge statements as resolver provider keys. Populated by Records (call it first).

func (*Provider) Name

func (p *Provider) Name() string

Name is the provenance feed the run writes (feed:<name>).

func (*Provider) Records

func (p *Provider) Records(ctx context.Context) ([]ingest.Record, error)

Records parses the export and returns one record per work subject, ordered per the mapping's id-order so ingest runs are deterministic. Records sharing a mapped "group" object (dcterms:isPartOf-style; self when absent) carry a shared grouping id, so the pipeline clusters them into one Work with one Instance each.

func (*Provider) Role

func (p *Provider) Role() ingest.Role

Role marks this an ingest-role provider.

type SourcesMapping

type SourcesMapping struct {
	// Prefix is stripped from the source IRI to form the source slug.
	Prefix string `toml:"prefix"`
	// ExtraKey is the work extra the joined slugs land under (default
	// "sources" -- the key the public-provenance allowlist governs).
	ExtraKey string `toml:"extra-key"`
	// Tentative lists source IRIs that do not confer confidence: a work
	// attested only by these is marked with the "tentative" extra and can be
	// dropped wholesale via Params["tentative"]="drop".
	Tentative []string `toml:"tentative"`
}

SourcesMapping describes the source-attestation objects a "source" field predicate points at.

type StringList

type StringList []string

StringList decodes from either a single TOML string or an array of strings, so the common one-IRI field stays one line.

func (*StringList) UnmarshalTOML

func (s *StringList) UnmarshalTOML(v any) error

UnmarshalTOML implements toml.Unmarshaler.