scan

Documentation

Overview

Package scan walks a file tree, hands each eligible file to the extractor registry, and merges the results into one Result with repo-relative positions, resolved remote defs, and duplicate-id detection (docs/SPEC.md §16 passes 1 and 2).

It is a pure function over an fs.FS: no environment, no home directory, no network, no writes. The CLI decides what tree to hand in (usually os.DirFS(repo)) and what to do with the result. Because the input is an fs.FS, tests use fstest.MapFS and the conformance suite uses fixture directories, both without touching disk paths.

Index

• Constants
• Variables
• func CitesNothing(p string, c block.Carrier) bool
• func CountMatches(fsys fs.FS, pattern string) (int, error)
• func IsOwnOutput(src []byte) bool
• func MatchAny(patterns []string, path string) (bool, error)
• type Cache
• type Options
• type Problem
• type Result
  • func Scan(ctx context.Context, fsys fs.FS, opts Options) (Result, error)
• type Skip
• type SkipReason
  • func (r SkipReason) Unreadable() bool
• type StatCache
• type SymbolResolver

Constants

const (
	DefaultMaxFileKB    = 512
	DefaultMaxLineChars = 2000
)

Defaults for the limits that keep a scan bounded (§10 normalization notes).

const StateDir = ".ds"

Directories never descended into, regardless of config. Vendored and installed code is other people's; VCS metadata is never a document. StateDir is the tool's own directory (§6). It is never scanned: the ledger, refs, acks, and journal it holds quote directives and would otherwise define and cite themselves.

Variables

var (
	ErrDuplicateID = errors.New("scan: id defined more than once")
	// ErrSharedBlock is two different ids bound to the same lines of the same
	// file. It is a finding because the ids are then indistinguishable to
	// every later pass: both hash the same bytes, so a sentence citing either
	// is measured against the other's block and a change flags both or
	// neither. It is nearly always a directive that failed to bind where its
	// author aimed it and landed on a neighbour (bug 18); the honest remedy is
	// to move or remove one, which is why the message names both.
	ErrSharedBlock = errors.New("scan: two ids bound to the same block")
	// ErrCrossingBlocks is two blocks in one file that overlap without one
	// containing the other: each holds part of the other, so an edit to the
	// shared lines flags both. Nesting is not this and is never reported.
	ErrCrossingBlocks = errors.New("scan: two blocks overlap without one containing the other")
	ErrDefInGenerated = errors.New("scan: ds:def in a generated file will be overwritten by the next generation")
	// ErrBadStability is an unrecognised `stability=`. It is a finding
	// rather than a silent default because the fallback is `stable`, so a
	// typo in `frozen` would quietly relax the policy the author asked for
	// and the block would stop flagging changes they wanted flagged.
	ErrBadStability = errors.New("scan: unknown stability")
	// ErrBadType is an unrecognised `type=`, and ErrTypeMismatch a value that
	// does not have the shape its type declares. Both used to pass silently:
	// type= was read by render (for url) and printed by facts, and nothing
	// checked it.
	ErrBadType = errors.New("scan: unknown type")
	// ErrMalformedID is a def whose id is not in the §8 shape for this
	// workspace, `<label>-<suffix>` (bug 55).
	ErrMalformedID  = errors.New("scan: malformed id")
	ErrTypeMismatch = errors.New("scan: value does not match its type=")
	// ErrBareDirective is a directive written as a line of its own in a file
	// whose type does carry comments. It is almost always damage rather than a
	// choice: `ds def` used to insert an uncommented line into any file type it
	// did not recognise, which is invalid syntax in every language that has
	// any -- a bare line in a go.work stopped every build in that workspace.
	// Reporting it is how a repository already hit finds the places to fix.
	ErrBareDirective = errors.New("scan: directive is not inside a comment")
	// ErrBadID is a def id holding whitespace or a control character. A
	// quoted value allows both, and such an id was accepted without a word:
	// it can never be cited through a link, and a tab or newline in it split
	// rows of the files that store it.
	ErrBadID         = errors.New("scan: id contains whitespace or a control character")
	ErrRemoteMissing = errors.New("scan: remote def target file not found")
	ErrRemotePick    = errors.New("scan: remote def pick failed")
	// ErrRemoteSkipped is a file= target the scanner would not read as a
	// source either: over scan.max_file_kb, or binary.
	ErrRemoteSkipped = errors.New("scan: remote def target is not readable as text")
	ErrPick          = errors.New("scan: pick failed")
	ErrNoInclude     = errors.New("scan: no include patterns; nothing would be scanned")
	// ErrCfgBlockForm is `<!-- ds:cfg id=… -->` on a line of its own. cfg
	// puts a value inside a sentence, so it is link form only (§9.3); the
	// block form scanned and checked as a citation while render refused it
	// and left the comment in the page, so a page that passed check
	// published without its value (bug 67). It is reported here so check
	// fails where render would.
	ErrCfgBlockForm = errors.New("scan: ds:cfg is link form only; write [value](ds:cfg?id=…) in the sentence")
)

Sentinel problem errors produced at the scan level (extractor problems keep their own sentinels).

var ErrSymlink = errors.New("scan: file= target is a symbolic link")

ErrSymlink is a remote `file=` target that is a symbolic link; links are refused so a def cannot point outside the tree by indirection (§ Security).

var SkipReasonValues = []SkipReason{SkipExcluded, SkipNotIn, SkipTooLarge, SkipLongLine, SkipBinary, SkipReadError, SkipOwnOutput}

SkipReasonValues is the canonical order.

Functions

func CitesNothing

func CitesNothing(p string, c block.Carrier) bool

CitesNothing reports a reference that scan drops: anything in a format with no comment syntax, and a whole-line directive in a format that carries directives only in comments (bugs 139 and 140). It is exported so an editor reading an unsaved buffer drops the same references a scan of the saved file would.

func CountMatches

func CountMatches(fsys fs.FS, pattern string) (int, error)

CountMatches reports how many files under fsys one include pattern would select, honouring the same skipped directories as Scan. `doctor` uses it to flag globs that match nothing, which is how a misspelled path fails silently otherwise.

func IsOwnOutput

func IsOwnOutput(src []byte) bool

IsOwnOutput reports a file ds itself wrote: a `--json` report, or an ack log exported by `ds audit --export`. Both quote sentences, citations included, so scanned with the default `code = ["**"]` a report saved inside the repository became a phantom citation of every block it mentioned (bug 75), and `rename` rewrote the ids in an exported audit record (bug 74). The test is the content, not the name, because the user names the file.

func MatchAny

func MatchAny(patterns []string, path string) (bool, error)

MatchAny reports whether path matches any of the glob patterns, for callers outside this module that keep an allow-list of paths (`run.allow`).

Types

type Cache

type Cache interface {
	Get(path, hash string) (extract.Found, bool)
	Put(path, hash string, found extract.Found)
}

Cache is the incremental-extraction store. Keys are the path and the content hash; a hit returns what extracting the same bytes produced. The scanner serialises its calls, so an implementation needs no locking.

type Options

type Options struct {
	Prefix       string
	Repo         string
	Include      glob.Set
	Exclude      glob.Set
	Generated    glob.Set
	Secret       glob.Set
	Registry     *extract.Registry
	MaxFileKB    int
	MaxLineChars int
	// Pickers are plugin pick= schemes (§37.3 Picker), by scheme name.
	Pickers map[string]pick.Picker
	// Symbol resolves `pick=symbol:<name>` on a remote def: the block the
	// tier that scans the target file binds for that name, as an in-file
	// `ds def file#name` would, content and hash included. A scan has no
	// grammar of its own, so without it the scheme stays unsupported (bug
	// 133).
	Symbol SymbolResolver
	// Cache remembers extraction per file and content hash, so a check
	// after a small change re-extracts only what changed (§16 "incremental
	// by default"). nil extracts everything.
	Cache Cache
	// Workers bounds concurrent extraction; 0 means one per CPU.
	Workers int
}

Options configures a scan. Include and Exclude are compiled glob sets; Registry defaults to extract.Default().

type Problem

type Problem struct {
	Pos block.Position
	Err error
}

Problem is a scan-level or extractor-level issue with its location.

type Result

type Result struct {
	Defs     []block.Block
	Refs     []block.Reference
	Problems []Problem
	Skipped  []Skip
	// Files is the number of files actually extracted.
	Files int
	// Tier records which extractor handled each scanned file, for `doctor`.
	Tier map[string]string
	// Pages holds frontmatter declarations per markdown file that had any.
	Pages map[string]extract.Page
}

Result is everything a scan found, sorted by file then line so two scans of the same tree are byte-identical when serialised.

func Scan

func Scan(ctx context.Context, fsys fs.FS, opts Options) (Result, error)

Scan walks fsys and extracts. It stops early only on context cancellation or a walk error; per-file read errors become Skips.

type Skip

type Skip struct {
	File   string
	Reason SkipReason
}

Skip records a file that was not scanned and why.

type SkipReason

type SkipReason string

SkipReason is why a file was not scanned; reported, never silent.

const (
	SkipExcluded  SkipReason = "excluded"     // matched an exclude glob
	SkipNotIn     SkipReason = "not-included" // matched no include glob
	SkipTooLarge  SkipReason = "too-large"    // larger than MaxFileKB
	SkipLongLine  SkipReason = "long-line"    // a non-prose line past MaxLineChars, taken as minified
	SkipBinary    SkipReason = "binary"       // holds a NUL byte
	SkipReadError SkipReason = "read-error"   // could not be stat'd or read
	SkipOwnOutput SkipReason = "ds-output"    // a report ds itself wrote (IsOwnOutput)
)

func (SkipReason) Unreadable

func (r SkipReason) Unreadable() bool

Unreadable reports whether a file was skipped because of its form — too large, a line past the limit, binary, or unreadable — rather than by a choice in config. Such a file still exists and may still be cited, so its previous state is carried forward and it is reported, never dropped: a dropped doc took every citation in it, and any unreviewed change they carried, with it, and the check went green.

type StatCache

type StatCache interface {
	Cache
	Unchanged(path string, size int64, mod time.Time) (extract.Found, bool)
	Stamp(path string, size int64, mod time.Time)
}

StatCache is an optional upgrade a Cache may implement so an unchanged file is not even read: Unchanged answers from the size and modification time recorded for it, and Stamp records them once the scanner holds that file's extraction (a Get hit or a Put). On a large tree reading every file is most of what a check costs — 100,000 opens and reads took four seconds against the two the performance target allows (§33) — and a content-hash key can only be consulted after the read.

The scanner calls Unchanged only with a real modification time (a zero one, as fstest.MapFS reports, always reads) and after the size limit, so a hit skips exactly the read, the binary test and the long-line test that produced the entry. A file edited so quickly that its time does not move is the implementation's to guard against, as git does for its index: refuse to Stamp a time too close to the moment of stamping.

type SymbolResolver

type SymbolResolver func(path string, src []byte, name string) (block.Block, error)

SymbolResolver returns the block a tier binds for name in the file at path with content src, or an error naming why it cannot.