bundle

Documentation

Overview

Package bundle builds the documentation bundle memql.io serves: one set per release, attached to the GitHub Release as docs-<version>.tgz (memql#5717, bundle contract v2; docs/DOCS_STANDARD.md section 5).

It is the code the root gates run. docs_public_boundary_test.go checks the tree with Check on every pull request and docs_bundle_build_test.go builds it; the release lane runs the same Check before it builds, so a release cannot ship a page the pull-request gate would have refused.

The bundle:

• SELECTS the pages git tracks under docs/public whose front matter says audience public, status stable and exposure engine (a missing exposure reads as engine), plus the pages generated in process (the concept catalog at reference/concepts.md);
• REWRITES every relative link to a selected page into its absolute site route, /docs/<slug>/, fragment kept, and REFUSES every other relative link -- one that leaves docs/public, or names a page the bundle does not publish, or a path that is not a page -- unless the temporary allowlist names it, in which case the link is replaced by its text;
• STRIPS the HTML comments outside code, which are gate markers, never prose;
• WRITES manifest.json (schema 2), memql-docs-version, llms.txt, llms-full.txt, sitemap-docs.xml and the page tree, and packs them into a tarball that is byte-identical across builds of one commit.

Index

• Constants
• Variables
• func Route(p string) string
• func Slug(p string) string
• func ValidVersion(v string) bool
• type AllowEntry
• type Allowlist
  • func LoadAllowlist(path string) (Allowlist, error)
  • func ParseAllowlist(name string, data []byte) (Allowlist, error)
• type Area
• type Dater
• type Excluded
• type GeneratedPage
• type GitDater
  • func NewGitDater(root, ref string) (*GitDater, error)
  • func (d *GitDater) LastUpdated(repoPath string) (time.Time, error)
• type Manifest
• type ManifestPage
• type Meta
  • func GitMeta(root, version, siteURL string) (Meta, error)
• type NavArea
• type NavPage
• type Options
• type Page
• type RefusedOutError
  • func (e *RefusedOutError) Error() string
• type Result
  • func Build(o Options) (*Result, error)
• type SourceFile
  • func LoadTree(root string) ([]SourceFile, error)
• type Violation
  • func (v Violation) String() string

Constants

const (
	ManifestFile = "manifest.json"
	VersionFile  = "memql-docs-version"
	LLMSFile     = "llms.txt"
	LLMSFullFile = "llms-full.txt"
	SitemapFile  = "sitemap-docs.xml"
)

The files at the bundle's root, beside the page tree.

const (
	ExposureEngine   = "engine"
	ExposureInstance = "instance"
)

Exposure values. A page with no `exposure` key reads as engine until the docs-readers epic makes the key required.

const (
	// RuleLeavesPublic: a relative link resolves outside docs/public -- into
	// docs/internal, docs/superpowers, or the source tree. Nothing there is on
	// the site, and the design records and internal docs are not public.
	RuleLeavesPublic = "leaves-public"
	// RuleUnselectedTarget: a relative link names a docs/public page the
	// bundle does not publish (a draft, a historical page, an internal one).
	RuleUnselectedTarget = "unselected-target"
	// RuleUnrewritableTarget: a relative link names a docs/public path that is
	// not a page -- a directory, a JSON file, an image -- which has no route.
	RuleUnrewritableTarget = "unrewritable-target"
	// RuleMissingTarget: a relative link names nothing tracked under
	// docs/public at all.
	RuleMissingTarget = "missing-target"

	// RuleInstanceExposure: a public page marked `exposure: instance`.
	RuleInstanceExposure = "instance-exposure"
	// RuleUnknownExposure: an `exposure` value outside engine | instance.
	RuleUnknownExposure = "unknown-exposure"
	// RuleUnknownArea: a published page whose area is not in Areas.
	RuleUnknownArea = "unknown-area"
	// RuleRouteCollision: two pages served at one URL.
	RuleRouteCollision = "route-collision"
	// RuleStaleAllowlistEntry: an allowlist entry that matches no link, so the
	// list only ever shrinks.
	RuleStaleAllowlistEntry = "stale-allowlist-entry"
)

The boundary rules. The link rules can be allowlisted by (file, target, rule); the page rules cannot, because an allowlist entry names a link.

const ConceptCatalogPath = "reference/concepts.md"

ConceptCatalogPath is where the concept catalog sits in the bundle, relative to PublicDir. It is rendered in process and never written into docs/public.

const DefaultAllowlist = "docs/public_boundary_allowlist.toml"

DefaultAllowlist is the boundary allowlist, relative to the repository root.

const DefaultSiteURL = "https://memql.io"

DefaultSiteURL is the site the bundle's absolute URLs point at.

const HomePath = "overview/index.md"

HomePath is the page served as the docs home, /docs/.

const ManifestSchema = 2

ManifestSchema is the manifest's contract version. Schema 1 was the Python bundler's {version, pageCount, areas, nav}; schema 2 is this one.

const PublicDir = "docs/public"

PublicDir is the one tree the bundle reads, relative to the repository root.

Variables

var Areas = []Area{
	{ID: "overview", Title: "Overview", Order: 0},
	{ID: "concepts", Title: "Concepts", Order: 1},
	{ID: "language", Title: "Language", Order: 2},
	{ID: "ai", Title: "AI", Order: 3},
	{ID: "operate", Title: "Operate", Order: 4},
	{ID: "build", Title: "Build", Order: 5},
	{ID: "cockpit", Title: "Cockpit", Order: 6},
	{ID: "reference", Title: "Reference", Order: 7},
}

Areas is the ONE table of public documentation areas, in sidebar order. Before it, the bundler and the site each kept a list and the two disagreed on the order (build before operate in one, after it in the other). The manifest now carries this table, so the site reads the order instead of keeping its own.

A public page whose `area` is not here is a violation, not an "other" group: the site has nowhere to put it. Adding an area means adding it here and to the closed set in docs/DOCS_STANDARD.md section 2 and docs_front_matter_test.go, in the same change.

var ErrBadVersion = errors.New("the version is not X.Y.Z (bare, no leading v)")

ErrBadVersion refuses a version label that is not bare X.Y.Z.

var ErrShallowClone = errors.New("the repository is a shallow clone: every page would date to the tip commit; fetch the full history (actions/checkout fetch-depth: 0, or git fetch --unshallow)")

ErrShallowClone refuses to date pages in a shallow clone. At depth 1 every page's last commit is the tip, so every lastUpdated and every sitemap lastmod would read as the release day -- dates that look right and say nothing.

var ErrTree = errors.New("cannot read the repository")

ErrTree is a failure to read the repository: not a git checkout, or git absent.

var ErrUndated = errors.New("a page has no commit to date it by")

ErrUndated is a page git cannot date: tracked, but in no commit yet.

var ErrViolations = errors.New("the docs bundle breaks the public boundary")

ErrViolations is returned when the boundary check fails. Nothing is written.

Functions

func Route

func Route(p string) string

Route is a page's absolute site route. The site serves trailing slashes, so the home is /docs/ and every other page /docs/<slug>/.

func Slug

func Slug(p string) string

Slug is a page's route slug: the path minus `.md`, a trailing `/index` folded into its directory, and the docs home (overview/index.md) empty.

func ValidVersion

func ValidVersion(v string) bool

ValidVersion reports whether v is the bare X.Y.Z a bundle is labelled with.

Types

type AllowEntry

type AllowEntry struct {
	File   string // repository-relative page path
	Target string // the link target as written
	Rule   string
	Reason string // optional
	Line   int    // the entry's line in the allowlist file
}

AllowEntry lets one known link through the boundary check until the page is fixed. It names the link by the page that carries it, its target as written, and the rule it breaks. One entry covers every occurrence of that target on that page.

type Allowlist

type Allowlist struct {
	Path    string
	Entries []AllowEntry
}

Allowlist is the parsed allowlist file.

func LoadAllowlist

func LoadAllowlist(path string) (Allowlist, error)

LoadAllowlist reads the allowlist at path. A missing file is an empty list.

func ParseAllowlist

func ParseAllowlist(name string, data []byte) (Allowlist, error)

ParseAllowlist reads the allowlist's format: the subset of TOML the repository's other allowlist (.gitleaks.toml) is written in, and nothing more --

# a comment
[[allow]]
file   = "docs/public/build/build-tags.md"
target = "../../internal/design/platform-consolidation.md"
rule   = "leaves-public"
reason = "optional"

Basic strings only. Anything else -- another table, an unknown key, a duplicate entry, a rule no link can break -- is refused rather than guessed at, because a list that silently read less than it said would let links through that nobody allowed.

type Area

type Area struct {
	ID    string `json:"id"`
	Title string `json:"title"`
	Order int    `json:"order"`
}

Area is one section of the published docs: the `area` front-matter value, the sidebar title and its place in the order.

type Dater

type Dater interface {
	// LastUpdated returns the committer date of the last commit that touched
	// the repository-relative path.
	LastUpdated(repoPath string) (time.Time, error)
}

Dater answers when a page last changed.

type Excluded

type Excluded struct {
	Path   string `json:"path"`
	Reason string `json:"reason"`
}

Excluded is a page under docs/public the bundle leaves out, and why.

type GeneratedPage

type GeneratedPage struct {
	Path    string // relative to docs/public
	Content string // the whole page, front matter included
}

GeneratedPage is a page rendered in process at bundle time, such as the concept catalog. It is selected and link-checked like an authored page.

type GitDater

type GitDater struct {
	Root string
	Ref  string
}

GitDater dates pages from git history: the committer date of `git log -1 --format=%cI <ref> -- <path>`.

func NewGitDater

func NewGitDater(root, ref string) (*GitDater, error)

NewGitDater returns a dater for the repository at root, reading history from ref (HEAD when empty). It refuses a shallow clone.

func (*GitDater) LastUpdated

func (d *GitDater) LastUpdated(repoPath string) (time.Time, error)

LastUpdated implements Dater.

type Manifest

type Manifest struct {
	Schema     int     `json:"schema"`
	Version    string  `json:"version"`
	Tag        *string `json:"tag"` // null when the built commit carries no release tag
	Commit     string  `json:"commit"`
	ReleasedAt string  `json:"releasedAt"`
	SiteURL    string  `json:"siteUrl"`
	// Home is the path of the page served at /docs/.
	Home      string         `json:"home"`
	PageCount int            `json:"pageCount"`
	Areas     []Area         `json:"areas"`
	Pages     []ManifestPage `json:"pages"`
	// Nav is TRANSITIONAL: schema 1's nav tree, derived from Pages, kept only
	// because the site's current reader builds its sidebar from it. It goes
	// when the site reads Pages (docs-current PR 6, memql#5718).
	Nav []NavArea `json:"nav"`

	// Reserved: null until the change that fills each lands (docs-readers
	// PR 2 fills positioning; docs-generated PRs 1 to 3 the rest). A reader
	// treats null as "not produced by this release", never as empty.
	Positioning any `json:"positioning"`
	NodeTypes   any `json:"nodeTypes"`
	Providers   any `json:"providers"`
	Apps        any `json:"apps"`
	Diagrams    any `json:"diagrams"`
}

Manifest is manifest.json, the bundle's table of contents (schema 2). The site reads it to build its navigation, its meta tags and its sitemap.

type ManifestPage

type ManifestPage struct {
	Path               string `json:"path"`
	Slug               string `json:"slug"`
	URL                string `json:"url"`
	Title              string `json:"title"`
	Area               string `json:"area"`
	Order              int    `json:"order"` // position within its area
	Description        string `json:"description"`
	DescriptionDerived bool   `json:"descriptionDerived"`
	Exposure           string `json:"exposure"`
	SinceVersion       string `json:"sinceVersion"`
	LastUpdated        string `json:"lastUpdated"`
	Generated          bool   `json:"generated"`
}

ManifestPage is one published page.

type Meta

type Meta struct {
	Version string
	// Tag is the release tag at the built commit (v-prefixed or bare), or ""
	// when the commit carries none -- a local build between releases.
	Tag        string
	Commit     string
	ReleasedAt time.Time // the built commit's committer date
	SiteURL    string
}

Meta is what the manifest says about the build itself.

func GitMeta

func GitMeta(root, version, siteURL string) (Meta, error)

GitMeta reads the commit, its date and its release tag at HEAD of the repository at root. The tag is the one named v<version> or <version> that points at HEAD; release tags come in both spellings (bare through 0.14.0).

type NavArea

type NavArea struct {
	Area  string    `json:"area"`
	Pages []NavPage `json:"pages"`
}

NavArea and NavPage are schema 1's nav shape, unchanged.

type NavPage

type NavPage struct {
	Path         string `json:"path"`
	Title        string `json:"title"`
	SinceVersion string `json:"sinceVersion"`
}

NavPage is one nav entry.

type Options

type Options struct {
	// RepoRoot is the repository whose docs/public is read when Sources is nil.
	RepoRoot string
	// Sources, when set, replaces reading the tree from RepoRoot.
	Sources []SourceFile
	// Reference holds the pages generated in process.
	Reference []GeneratedPage
	// Allow is the boundary allowlist.
	Allow Allowlist
	// Check runs the selection and the boundary check and writes nothing.
	Check bool
	// Meta labels the build. Version is required, and is bare X.Y.Z.
	Meta Meta
	// Dater dates the pages. Required unless Check.
	Dater Dater
	// Out, when set, receives the bundle tree.
	Out string
	// Tarball, when set, receives the packed bundle.
	Tarball string
}

Options configures one build or check.

type Page

type Page struct {
	Path     string // relative to docs/public
	Slug     string // "" for the home, "operate/auth" for operate/auth/index.md
	URL      string // the site route, "/docs/<slug>/"
	Title    string
	Area     string
	Exposure string // always "engine": an instance page never reaches here

	SinceVersion string
	Description  string
	// DescriptionDerived is true when the page carries no `description` key
	// and Description was read from its first paragraph.
	DescriptionDerived bool
	Generated          bool
	// contains filtered or unexported fields
}

Page is one page the bundle publishes.

type RefusedOutError

type RefusedOutError struct{ Dir string }

RefusedOutError is the refusal to clear an output directory that does not hold a previous bundle.

func (*RefusedOutError) Error

func (e *RefusedOutError) Error() string

type Result

type Result struct {
	Selected int        `json:"pages"`
	Excluded []Excluded `json:"excluded"`
	// Violations are the ones the allowlist does not cover, stale allowlist
	// entries included. A non-empty list means ErrViolations.
	Violations []Violation `json:"violations"`
	// Allowlisted counts the distinct (file, target, rule) links the allowlist
	// let through.
	Allowlisted int `json:"allowlisted"`
	// Manifest is nil in check mode.
	Manifest *Manifest `json:"-"`
	// Files are the bundle's entries, root files and pages, sorted.
	Files []string `json:"-"`
}

Result is what a build or check found.

func Build

func Build(o Options) (*Result, error)

Build selects, checks and -- unless Check is set -- renders and writes the bundle. On a boundary violation it returns the Result and ErrViolations having written nothing.

type SourceFile

type SourceFile struct {
	// Path is relative to docs/public, slash-separated.
	Path string
	// Content is the file's bytes for a markdown file, nil for any other: a
	// non-markdown file matters only as a link target that exists.
	Content []byte
}

SourceFile is one tracked file under docs/public.

func LoadTree

func LoadTree(root string) ([]SourceFile, error)

LoadTree reads the files git tracks under docs/public in the repository at root. Tracked, never walked: a filesystem walk ships whatever untracked scratch file happens to sit in the checkout.

type Violation

type Violation struct {
	File     string `json:"file"`           // repository-relative
	Line     int    `json:"line,omitempty"` // 1-based, 0 for a page-level rule
	Target   string `json:"target,omitempty"`
	Resolved string `json:"resolved,omitempty"` // where the target lands, repository-relative
	Rule     string `json:"rule"`
	Detail   string `json:"detail,omitempty"`
}

Violation is one breach of the public boundary.

func (Violation) String

func (v Violation) String() string