export

Documentation

Overview

Package export derives the downloadable catalog artifacts from an ingest output root (tasks/172): catalog.nq.gz is the corpus itself (the ingest serialization, blank-node labels already unique across grains), and catalog.mrc.gz / catalog.xml.gz are per-grain MARC round-trips via bibframe.DecodeGrainMARC, which honors editorial override shadows and verbatim sidecars (fidelity bounded by docs/marc-fidelity.md). A Manifest records sizes, sha256 digests, and record counts for a downloads page. This is the static-site counterpart of the backend's on-demand export service (backend/export); both read the same grains.

Index

• type File
• type Manifest
  • func Run(opts Options) (*Manifest, error)
• type Options

Types

type File

type File struct {
	Name    string `json:"name"`
	Bytes   int64  `json:"bytes"`
	SHA256  string `json:"sha256"`
	Records int    `json:"records"`
}

File is one artifact's manifest entry; the sha256 is over the compressed bytes (what a downloader verifies).

type Manifest

type Manifest struct {
	Generated string `json:"generated"`
	Works     int    `json:"works"`
	Files     []File `json:"files"`
}

Manifest is the downloads-page data file: what was generated when, from how many works, with per-artifact integrity and record counts.

func Run

func Run(opts Options) (*Manifest, error)

Run exports the corpus under opts.In to gzipped download artifacts under opts.Out and returns their manifest.

type Options

type Options struct {
	// In is the ingest output root: the directory holding catalog.nq and
	// data/works/ grains.
	In string
	// Out is the directory the gzipped artifacts are written to.
	Out string
	// PublicSources, when non-nil, is the allowlist of extra/sources names
	// permitted in the nq download; other attributions are stripped, matching
	// project.SanitizeSources on the catalog.json side, so no public surface
	// discloses more provenance than the site does. Nil keeps every source.
	// The on-disk graph of record stays complete either way.
	PublicSources map[string]bool
	// Log receives skip and strip warnings; nil means os.Stderr.
	Log io.Writer
}

Options configures one export run.