gate

package
v0.0.11 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 24, 2026 License: MIT Imports: 31 Imported by: 0

Documentation

Overview

Package gate runs the ingest gates of spec §7 over sources about to enter a topic. Stages are code and predicates are decisions; only survivors pay the next stage:

  1. exact dedupe (code): normalized URL, platform id, body hash against the topic's sources, quarantined ones included;
  2. pre-fetch relevance (bulk only): one request per ≤ 20 items of metadata; skipped items are never fetched;
  3. code quality checks (internal/quality);
  4. one fresh refetch of URL sources that a code flag or a short body marks as a possibly broken capture;
  5. post-fetch quality + relevance: the shared gate judgment (classify.JudgeGate) on the would-be document;
  6. near-duplicate: BM25 and same-domain title candidates, one request.

Quality gates and exact dedupe apply by default; relevance gates run in shadow until calibrated (session.RelevanceGateMode). Shadow never skips or quarantines: the would-be outcome is reported and the source is written with `triage: review` so the owner can label it.

Index

Constants

View Source 
const (
	TriageKept             = "kept"
	TriageReview           = "review"
	TriageQuarantined      = "quarantined"
	TriageSkipped          = "skipped"
	TriageDuplicateSkipped = "duplicate-skipped"
)

Triage outcomes. Kept, review and quarantined are also `triage` values; skipped (stage 2) and duplicate-skipped (stage 1) never reach the disk.

View Source 
const (
	ReasonOffTopic  = "off_topic"
	ReasonDuplicate = "duplicate"
	ReasonUndecided = "undecided"
)

Gate reasons that are not quality reasons. ReasonUndecided marks a source whose gate judgment could not be made (timeout, budget, invalid receipt, state too large...): it is written with `triage: review` so a failed judgment never reads as kept (spec §2 principle 3).

View Source 
const (
	StageDedupe        = "dedupe"
	StagePrefetch      = "prefetch"
	StageQuality       = "quality"
	StageRelevance     = "relevance"
	StageNearDuplicate = "near-duplicate"
	StageUndecided     = "undecided"
)

Stage names, for outcomes and review evidence.

View Source 
const (
	// NearBM25TopK is the number of BM25 candidates.
	NearBM25TopK = 5
	// NearTitleSimilarity is the same-domain title similarity floor.
	NearTitleSimilarity = 0.6
)

Near-duplicate candidate limits (spec §7 stage 6).

View Source 
const (
	RelationSameContent = "same_content"
	RelationNewVersion  = "new_version"
)

Near-duplicate relation options of the duplicate bank.

View Source 
const (
	ActionFetch  = "fetch"
	ActionReview = "review"
	ActionSkip   = "skip"
)

Pre-fetch actions.

View Source 
const (
	PrefetchCollectedPath = "collected_on_purpose_path"
	PrefetchRelevanceOff  = "relevance_off"
	// PrefetchExcluded marks an item whose would-be path matches
	// decisions.exclude: it is fetched without a pre-fetch call.
	PrefetchExcluded = "excluded"
)

Pre-fetch reasons for a fetch that was not decided by the role answer.

View Source 
const ExcludedNote = "decisions.exclude: kept without a judgment (no decision call)"

ExcludedNote is the evidence line of a source kept without a judgment because it matches decisions.exclude.

View Source 
const MaxItemsPerRequest = 20

MaxItemsPerRequest caps the items of one pre-fetch relevance request.

View Source 
const SkippedFile = "skipped.jsonl"

SkippedFile is the log of items skipped before fetch, under <topic>/.decisions/.

Variables

View Source 
var ErrSkippedNotFound = errors.New("gate: skipped item not found")

ErrSkippedNotFound is returned when a skipped id has no row.

Functions

func AppendSkipped 

func AppendSkipped(topicRoot string, row SkippedRow) error

AppendSkipped appends one row to skipped.jsonl.

func MarkRescued 

func MarkRescued(topicRoot string, row SkippedRow, stamp string) error

MarkRescued appends the row again with Rescued true.

func NeedsRefetch 

func NeedsRefetch(flags []quality.Flag, words int, refetchable bool) bool

NeedsRefetch is the stage-4 trigger: a URL source (refetchable) with a code flag or fewer than quality.MinWords content words.

func NormalizeURL 

func NormalizeURL(raw string) string

NormalizeURL returns the dedupe key of a URL: host lowercased without `www.`, the path without a trailing slash, the query without tracking parameters (`utm_*`, `fbclid`, `gclid`, `si`, `igshid`, ...) and sorted, no fragment. The scheme is dropped, so the http and https captures of one page are the same source. It returns "" for anything that is not an absolute http(s) URL.

func PlatformID 

func PlatformID(raw string) string

PlatformID returns the platform identity of a media URL: `youtube:<video id>` for watch, youtu.be, shorts, embed and live URLs, and `instagram:<shortcode>` for post, reel and IGTV URLs; "" otherwise.

func PrefetchAction 

func PrefetchAction(pOffTopic, pKept, quarantine, fetch float64) string

PrefetchAction bands one item: skip when P(off_topic) ≥ quarantine, fetch when P(core)+P(adjacent)+P(collected_on_purpose) ≥ fetch, else review.

func ReviewItem 

func ReviewItem(o Outcome, path, title string) (review.Item, bool)

ReviewItem returns the `gate` review-queue item of an outcome written at path (review band, shadow would-be quarantine, or gate quarantine), and false for a kept source. Accepting it keeps the source (restoring it from quarantine), rejecting it quarantines it with Reason.

func SkippedID 

func SkippedID(rawURL string) string

SkippedID derives the stable id of a skipped item from its URL: "sk-" + the first 10 hex digits of sha256(NormalizeURL(url)).

func SkippedPath 

func SkippedPath(topicRoot string) string

SkippedPath returns <topic>/.decisions/skipped.jsonl.

func TitleSimilarity 

func TitleSimilarity(a, b string) float64

TitleSimilarity is the token Jaccard similarity of two titles (tokens as corpus.Tokenize: lowercased letters/digits, stopwords dropped).

Types

type Checker 

type Checker struct {
	// contains filtered or unexported fields
}

Checker runs the per-source stages of one ingest run over a topic. It is not safe for concurrent use; ingest serializes items.

func NewChecker 

func NewChecker(s *session.Session) (*Checker, error)

NewChecker loads what the gates compare against: the topic's sources (dedupe, same-host lines, near-duplicate candidates) and the quarantined ones (dedupe only). Sources matching decisions.exclude are kept for exact dedupe but never become near-duplicate candidates (spec §14).

func (*Checker) Evaluate 

func (c *Checker) Evaluate(ctx context.Context, in Fetched) (Outcome, *corpus.Document, error)

Evaluate runs stages 3–6 over one fetched source and returns the outcome and the document to write (the refetched body when stage 4 kept it). A code flag that survives the refetch quarantines without any decision call (in apply mode). The error is fatal only (auth, cancellation, I/O).

func (*Checker) Register 

func (c *Checker) Register(doc *corpus.Document, quarantined bool)

Register adds a source written (or quarantined) by this run, so later items dedupe and near-dedupe against it. A source matching decisions.exclude is registered for exact dedupe only: it never becomes a near-duplicate candidate of a later item, since candidates are sent to the model (spec §14).

func (*Checker) Skip 

func (c *Checker) Skip(ref Ref) (Outcome, bool)

Skip returns the stage-1 outcome for ref when it duplicates an existing source and dedupe applies (quality gate mode apply): the source is not written. In shadow it returns false and Evaluate reports the would-be skip.

type Fetched 

type Fetched struct {
	// Title and Markdown are the fetched title and body.
	Title    string
	Markdown string
	// Build returns the would-be document (path, frontmatter with
	// provenance, body) for a title and body. The gate calls it again after
	// a refetch, so the judged document is always the one that is written.
	Build func(title, markdown string) (*corpus.Document, error)
	// RequestedURL, FinalURL, StatusCode and SiteName are what the fetcher
	// reported (SiteName feeds the "title equals the site name" rule).
	RequestedURL string
	FinalURL     string
	StatusCode   int
	SiteName     string
	// Refetch performs the one fresh stage-4 scrape (URL sources only; nil
	// disables stage 4).
	Refetch func(ctx context.Context) (*firecrawl.ScrapeResult, error)
	// PlatformID is the `youtube:<id>` / `instagram:<shortcode>` identity
	// when the source URL does not carry it.
	PlatformID string
	// Transcript adds the `no_speech_content` question and disables the thin
	// rule.
	Transcript bool
	// PrefetchReview marks an item stage 2 put in the review band; Shadow
	// names a would-be pre-fetch skip that a shadow gate did not apply.
	PrefetchReview bool
	PrefetchShadow string
	// PrefetchPOffTopic is the pre-fetch P(off_topic) (review evidence).
	PrefetchPOffTopic float64
	// PrefetchReceipt is the pre-fetch receipt key.
	PrefetchReceipt string
}

Fetched is one fetched source about to be written, as seen by stages 3–6.

type Index 

type Index struct {
	// contains filtered or unexported fields
}

Index is the exact-dedupe index of a topic's sources, quarantined ones included (stage 1).

func NewIndex 

func NewIndex(topicRoot string, sources []*corpus.Document) (*Index, error)

NewIndex builds the dedupe index from loaded sources plus every markdown file under raw/_quarantine/ (which corpus.Load never loads).

func (*Index) Add 

func (x *Index) Add(doc *corpus.Document, quarantined bool)

Add registers one source (also used for sources written earlier in the same run, so a URL listed twice is ingested once).

func (*Index) Find 

func (x *Index) Find(ref Ref) (Match, bool)

Find returns the existing source equal to ref, checking the platform id, then the normalized URL, then the body hash.

type Item 

type Item struct {
	// ID is the stable skipped id; derived from URL with SkippedID when
	// empty.
	ID          string
	URL         string
	Title       string
	Description string
	// Channel is the channel, uploader or site name.
	Channel string
	// Date is the publication date as text.
	Date string
	// Path is the would-be topic-relative path (for collected_on_purpose_paths).
	Path string
	// SourceKind is the would-be source kind (provenance).
	SourceKind string
}

Item is one bulk item before fetch: metadata only.

type Match 

type Match struct {
	// Path is the topic-relative path of the existing source (the quarantine
	// path for a quarantined one).
	Path string
	// Quarantined reports that the existing source is in raw/_quarantine/.
	Quarantined bool
	// By names what matched: url, platform_id or body_hash.
	By string
}

Match is an existing source equal to a new one.

func (Match) Describe 

func (m Match) Describe() string

Describe renders the match for reports: "duplicate of raw/x.md" or "duplicate of quarantined raw/_quarantine/x.md".

type NearCandidate 

type NearCandidate struct {
	Doc *corpus.Document
	// Origin is "bm25" or "title".
	Origin string
}

NearCandidate is one near-duplicate candidate proposed by code.

func NearCandidates 

func NearCandidates(doc *corpus.Document, sources []*corpus.Document, index *corpus.BM25) []NearCandidate

NearCandidates proposes the stage-6 candidates for doc: the BM25 top 5 over title + first 2k characters of every source, plus sources of the same host whose title similarity is ≥ 0.6; the document itself excluded, at most 10.

type Outcome 

type Outcome struct {
	// Triage is kept, review, quarantined, skipped or duplicate-skipped.
	Triage string `json:"triage"`
	// Reason is the triage_reason (off_topic, thin, error_page, paywall,
	// not_an_article, no_speech, duplicate); empty when kept.
	Reason string `json:"triage_reason,omitempty"`
	// Stage is the stage that set Triage.
	Stage string `json:"stage,omitempty"`
	// Quality is the value for the `quality` key: the quality reason when a
	// quality check or question fired at or above the review threshold.
	Quality string `json:"quality,omitempty"`
	// Purpose, Question, Probability and ReceiptKey describe the decision
	// behind Triage (for review items and labels).
	Purpose     decisions.Purpose `json:"purpose,omitempty"`
	Question    string            `json:"question,omitempty"`
	Probability float64           `json:"probability,omitempty"`
	ReceiptKey  string            `json:"receipt_key,omitempty"`
	// DuplicateOf is the existing source a duplicate matched.
	DuplicateOf string `json:"duplicate_of,omitempty"`
	// DuplicateQuarantined reports that DuplicateOf is quarantined.
	DuplicateQuarantined bool `json:"duplicate_quarantined,omitempty"`
	// Shadow lists would-be outcomes a shadow gate did not apply, such as
	// "quarantined:off_topic (relevance shadow)".
	Shadow []string `json:"shadow,omitempty"`
	// Supersedes lists `[[target]]` wikilinks of older versions (stage 6).
	Supersedes []string `json:"supersedes,omitempty"`
	// Flags are the code quality flags of the kept body.
	Flags []quality.Flag `json:"flags,omitempty"`
	// Refetched reports that the stage-4 refetch body was kept.
	Refetched bool `json:"refetched,omitempty"`
	// Undecided lists "<question>:<reason>" answers that could not be used.
	Undecided []string `json:"undecided,omitempty"`
	// Excluded reports that the source matches topic.yaml
	// `decisions.exclude`: no decision call was made about it (code checks
	// still ran) and it is kept without a judgment.
	Excluded bool `json:"excluded,omitempty"`
	// Evidence is one human-readable line for review items and logs.
	Evidence string `json:"evidence,omitempty"`
}

Outcome is what the gates decided about one source.

func (Outcome) Kept 

func (o Outcome) Kept() bool

Kept reports whether the source is written to raw/ (kept or review).

type PrefetchDecision 

type PrefetchDecision struct {
	Item Item `json:"-"`
	// Action is fetch, review (fetch and write with triage: review) or skip.
	Action string `json:"action"`
	// Reason explains a fetch without a role answer (collected path,
	// relevance off, undecided:<reason>).
	Reason     string  `json:"reason,omitempty"`
	Role       string  `json:"role,omitempty"`
	POffTopic  float64 `json:"p_off_topic"`
	PKept      float64 `json:"p_kept"`
	ReceiptKey string  `json:"receipt_key,omitempty"`
	// Shadow is the would-be action a shadow gate did not apply ("skip").
	Shadow string `json:"shadow,omitempty"`
}

PrefetchDecision is the pre-fetch verdict of one item.

func Prefetch 

func Prefetch(ctx context.Context, s *session.Session, items []Item, opts PrefetchOptions) ([]PrefetchDecision, error)

Prefetch runs stage 2 over bulk items: items whose would-be path matches decisions.exclude or collected_on_purpose_paths, and every item of a topic with relevance off, are fetched without a call; the rest are judged in requests of at most MaxItemsPerRequest items with the state `{contract, items:[{id,title,description,channel,date,provenance}]}` and one `role_item_{id}` question per item. In apply mode, skipped items are appended to skipped.jsonl and enqueued as `skip` review items; in shadow mode nothing is skipped and the would-be skip is returned in Shadow (the item is fetched and written with triage: review). Undecided items are fetched (a failed judgment is never a "no"). The error is fatal only.

type PrefetchOptions 

type PrefetchOptions struct {
	Batch string
	Query string
}

PrefetchOptions carries the run provenance recorded with every item.

type Ref 

type Ref struct {
	// URL is the source URL (normalized with NormalizeURL).
	URL string
	// PlatformID is a `youtube:<id>` / `instagram:<shortcode>` identity; when
	// empty it is derived from URL.
	PlatformID string
	// BodyHash is corpus.BodyHash of the body about to be written.
	BodyHash string
}

Ref identifies a source for exact dedupe. Empty fields are not compared.

type SkippedRow 

type SkippedRow struct {
	ID          string  `json:"id"`
	Time        string  `json:"time"`
	Batch       string  `json:"batch,omitempty"`
	Query       string  `json:"query,omitempty"`
	URL         string  `json:"url"`
	Title       string  `json:"title,omitempty"`
	Description string  `json:"description,omitempty"`
	Channel     string  `json:"channel,omitempty"`
	Date        string  `json:"date,omitempty"`
	POffTopic   float64 `json:"p_off_topic"`
	ReceiptKey  string  `json:"receipt_key,omitempty"`
	Rescued     bool    `json:"rescued"`
}

SkippedRow is one row of skipped.jsonl (plan schema). Rescue appends a new row with the same id and Rescued true; readers keep the last row per id.

func FindSkipped 

func FindSkipped(topicRoot, id string) (SkippedRow, error)

FindSkipped returns the latest row of one skipped id.

func LoadSkipped 

func LoadSkipped(topicRoot string) ([]SkippedRow, error)

LoadSkipped reads skipped.jsonl and returns the last row per id, in first appearance order (a missing file is empty; torn lines are ignored).

type Tally 

type Tally struct {
	Kept        int `json:"kept"`
	Review      int `json:"review"`
	Quarantined int `json:"quarantined"`
	Skipped     int `json:"skipped"`
	Duplicates  int `json:"duplicates"`
	// Undecided counts review sources whose gate judgment failed (a subset
	// of Review).
	Undecided int `json:"undecided"`
	// Excluded counts written sources matching decisions.exclude (never
	// judged).
	Excluded int `json:"excluded,omitempty"`
	// Shadow counts sources with at least one would-be outcome not applied.
	Shadow int `json:"shadow"`
}

Tally counts the outcomes of a run.

func (*Tally) Add 

func (t *Tally) Add(o Outcome)

Add counts one outcome.

func (Tally) Line 

func (t Tally) Line() string

Line renders the tally: "kept 2, review 1, quarantined 0, skipped 3, duplicates 1, undecided 0" (undecided sources are also counted in review), plus the excluded count when a source matched decisions.exclude.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.