types

package
v0.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 29, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package types holds the frozen domain types and the interfaces each package implements.

Layout: 

cmd/content-parity   CLI entry (check, benchmark, aggregate, scan, keys, dataset)
deploy               operator kit for publishing the key directory
signing              RFC 9421 and web-bot-auth (no project imports)
internal/types       this package
internal/schema      SQLite DDL
internal/fixture     local fixture servers
internal/fetch       default observation identities (honest UA)
internal/extract     semantic blocks and token estimates
internal/compare     block-level matching
internal/sitecontext robots, llms.txt, alternate links
internal/storage     SQLite persistence
internal/bodystore   compressed response bodies (local evidence)
internal/engine      check, benchmark, aggregate, and scan
internal/inject      machine-only text on stored bodies
internal/dataset     Tranco-backed pilot and benchmark lists
internal/config      JSON config file
internal/sitemap     sitemap.org urlset and index

Package signing is importable by third parties. It does not import this package. RequestSigner is satisfied implicitly.

Index

Constants

View Source 
const (
	ChallengeChannelHeader   = "header"
	ChallengeChannelLocation = "location"
	ChallengeChannelBody     = "body"

	ChallengeStrengthStrong = "strong"
	ChallengeStrengthWeak   = "weak"
)
View Source 
const (
	ProductName    = "content-parity"
	ProductVersion = "0.1.0"
	ProductUA      = "content-parity/0.1.0 (measurement; +https://github.com/Zulwatha/content-parity)"
	ProductToken   = "content-parity"

	// ImpersonateUA is a published GPTBot token. Sending it is not
	// default behavior. Fetch sends it only for IdentityImpersonate
	// when ImpersonateCrawler is set. Benchmark mode rejects that option.
	ImpersonateUA    = "Mozilla/5.0 AppleWebKit/537.36 (compatible; GPTBot/1.2; +https://openai.com/gptbot)"
	ImpersonateToken = "GPTBot"
)

Product identification. Every default HTTP identity sends ProductUA. Browser uses Chrome's own UA. Impersonate is the only identity that may send ImpersonateUA, and only when the caller opts in.

View Source 
const (
	CrawlerUA    = ImpersonateUA
	CrawlerToken = ImpersonateToken
)

CrawlerUA and CrawlerToken are the historical names for the impersonation strings. They are not used by default identities.

View Source 
const (
	DefaultTimeout     = 30 * time.Second
	DefaultRedirects   = 10
	DefaultRetries     = 2
	DefaultConcurrency = 4
	DefaultRetryWait   = 200 * time.Millisecond
	DefaultHostDelay   = time.Second

	DefaultHeartbeatInterval = 10 * time.Second
	DefaultHeartbeatStale    = 90 * time.Second
	DefaultBrowserTimeout    = 45 * time.Second
	DefaultBrowserRecycle    = 20
)

Timing defaults consumed by fetch and the benchmark command.

View Source 
const (
	PlatformCDN    = "cdn"
	PlatformEdge   = "edge"
	PlatformOrigin = "origin"
)
View Source 
const (
	WellKnownPath      = "/.well-known/http-message-signatures-directory"
	DirectoryMediaType = "application/http-message-signatures-directory+json"
	RequestTag         = "web-bot-auth"
	DirectoryTag       = "http-message-signatures-directory"
	SignatureAlgorithm = "ed25519"
	Algorithm          = SignatureAlgorithm
	RequestLabel       = "sig1"
	DirectoryLabel     = "binding"
)

Signing contract for package signing.

The package lives at github.com/Zulwatha/content-parity/signing and must import nothing under this module. Third parties may import it on its own.

Profile: RFC 9421 HTTP Message Signatures plus draft-meunier-webbotauth-httpsig-protocol-02.

Keys: crypto/ed25519 only. PKCS#8 PEM on disk. keyid is the base64url (no pad) SHA-256 JWK thumbprint of {"crv":"Ed25519","kty":"OKP","x":"..."} as in RFC 8037 A.3.

Request Sign:

  • set Signature-Agent to `sig1="<origin>"` (dictionary form)
  • cover "@authority" "@method" "@path" `signature-agent`;key="sig1"
  • params: created, expires (now + SignatureLifetime), keyid, alg="ed25519", nonce (32 random bytes, standard base64), tag="web-bot-auth"
  • label sig1
  • Signature value is an sf-byte-sequence (`:base64:`)

Directory:

  • path /.well-known/http-message-signatures-directory
  • media type application/http-message-signatures-directory+json
  • body: JWKS {keys:[{kty,crv,kid,x,use:"sig"}]}
  • Cache-Control: max-age matching DirectoryCacheAge
  • sign the response: tag="http-message-signatures-directory", label binding, cover `@authority`;req and content-digest (RFC 9530 sha-256), created, expires, keyid

Tests: RFC 9421 B.1.4 Ed25519 key and the EdDSA vectors in appendix E.2 of the protocol draft. No live network.

Constants the package should export: 

WellKnownPath      = "/.well-known/http-message-signatures-directory"
DirectoryMediaType = "application/http-message-signatures-directory+json"
RequestTag         = "web-bot-auth"
DirectoryTag       = "http-message-signatures-directory"
Algorithm          = "ed25519"
RequestLabel       = "sig1"
DirectoryLabel     = "binding"

AgentOrigin is the public origin that hosts the well-known directory. Signature-Agent must use this value. https is required except for loopback (tests and serve-keys).

CheckPublished GETs {Origin}{WellKnownPath} and verifies the current public key is in the JWKS. A failed check must abort a signed-mode benchmark at startup. Do not sign requests against an origin that does not publish the key.

View Source 
const AcceptHTML = "text/html, application/xhtml+xml, */*;q=0.1"

AcceptHTML is the Accept value for unsigned replicas and signed.

View Source 
const AcceptMarkdown = "text/markdown, text/x-markdown, text/plain;q=0.8, */*;q=0.1"

AcceptMarkdown is the Accept value for IdentityMarkdown.

View Source 
const AlterationFloor = 0.6

AlterationFloor is the minimum text similarity for an altered pair. Below this, blocks stay unmatched (missing/added).

View Source 
const DefaultBodyCap = 2 << 20

DefaultBodyCap is the maximum number of body bytes stored for one observation. The cap is recorded on the sidecar when it is hit.

View Source 
const DirectoryCacheAge = 24 * time.Hour

DirectoryCacheAge is Cache-Control max-age on the key directory.

View Source 
const ModulePath = "github.com/Zulwatha/content-parity"

ModulePath is the import path of this module. It is the repository URL without a scheme, so go get and go install can resolve it.

View Source 
const SignatureLifetime = 5 * time.Minute

SignatureLifetime is the created-to-expires window on signed requests.

Variables

View Source 
var ChallengeFamilies = []ChallengeFamily{
	cloudflareFamily,
	amazonWAFFamily,
	datadomeFamily,
	perimeterXFamily,
	anubisFamily,
	recaptchaFamily,
	hcaptchaFamily,
	httpAuthFamily,
}

ChallengeFamilies is the ordered registry. The classifier walks this list and records every match.

View Source 
var SignatureHeaderNames = []string{
	"Signature",
	"Signature-Input",
	"Signature-Agent",
}

SignatureHeaderNames are the only headers Sign may add to the unsigned request. The signed identity differs in nothing else.

View Source 
var TokenEstimateHeaders = []string{
	"X-Markdown-Tokens",
	"Markdown-Tokens",
	"X-Token-Count",
	"X-LLM-Tokens",
}

TokenEstimateHeaders are names used by edge markdown converters. First present header with a non-negative integer wins.

Functions

func BaselineEligible 

func BaselineEligible(a, b Observation) bool

BaselineEligible is true when both observations can sit in a comparison denominator. Empty is excluded. Uncertain is not.

func BaselineUnstable 

func BaselineUnstable(unsigned, control Observation, diff Comparison) bool

BaselineUnstable is true when the unsigned and control observations are not equivalent. The site then has no meaningful identity comparison. Both observations must exist; the caller decides that.

func BlockLess 

func BlockLess(a, b Block) bool

BlockLess is a side-independent total order on semantic fields. Index is ignored, as in Equal.

func CanonicalURL 

func CanonicalURL(raw string) string

CanonicalURL compares final URLs for redirect differences. Scheme and host are lowercased, default ports stripped, empty path becomes /. Query is kept. Fragment is dropped.

func ChallengeAsymmetry 

func ChallengeAsymmetry(a, b Observation) bool

ChallengeAsymmetry is the primary finding: one identity was challenged and the other was served. Never a content difference.

func ChallengeFamilyNames 

func ChallengeFamilyNames(sigs []ChallengeSignal) []string

ChallengeFamilyNames returns the families that produced sigs, in first-seen order.

func ContentComparable 

func ContentComparable(o Outcome) bool

ContentComparable is true when the response has usable page content. Uncertain is included: the challenge evidence was inconclusive, not a refusal to measure. Empty is not.

func ContentDiffers 

func ContentDiffers(a, b Observation, diff Comparison) bool

ContentDiffers is a block or representation difference between two served identities. Challenge pairs must not be passed here.

func CoreCoverage 

func CoreCoverage(core []Block, replicas ...[]Block) float64

CoreCoverage is len(core) / max replica length. Zero when every replica is empty.

func CoreTooSmall 

func CoreTooSmall(reps []Observation, core []Block) bool

CoreTooSmall is true when all replicas were served and produced blocks, but the intersection is empty.

func DomainSHA256 

func DomainSHA256(list string) string

func EstimateTokens 

func EstimateTokens(s string) int

EstimateTokens is ceil(runes/4) over s. Empty input is 0. Extract applies this to block texts joined by a single newline.

func HasStrongChallenge 

func HasStrongChallenge(sigs []ChallengeSignal) bool

HasStrongChallenge reports a signal that is enough to call the page an interstitial.

func IsChallenge 

func IsChallenge(status int, h http.Header, body []byte) bool

IsChallenge reports whether the response has a strong interstitial signal. A captcha widget string on an ordinary page is not one.

func LooksLikeHTML 

func LooksLikeHTML(contentType string) bool

LooksLikeHTML reports an HTML Content-Type.

func LooksLikeMarkdown 

func LooksLikeMarkdown(contentType string) bool

LooksLikeMarkdown reports a markdown Content-Type.

func MachineHasExtraText 

func MachineHasExtraText(diff Comparison) bool

MachineHasExtraText is the third aggregate predicate: the machine sequence contains text that is not in the browser sequence (DiffAdded when left is browser and right is machine).

func MarkdownFault 

func MarkdownFault(md, html Observation, htmlVsMarkdown Comparison) bool

MarkdownFault is lost body content in a real markdown variant. Navigation and interface chrome do not count. Missing Vary is MarkdownMissingVary, not this predicate.

func MarkdownLostBody 

func MarkdownLostBody(md, html Observation, htmlVsMarkdown Comparison) bool

MarkdownLostBody is true when the HTML sequence has a body block the markdown sequence is missing. Chrome-only loss is false. Words that appear in the markdown under another block kind are not lost.

func MarkdownMissingVary 

func MarkdownMissingVary(md, html Observation) bool

MarkdownMissingVary is true when a real markdown variant is missing Vary: Accept on the markdown side, or on the HTML side when HTML was served.

func MarkdownVariant 

func MarkdownVariant(md, html Observation) bool

MarkdownVariant is true when the markdown identity was served a markdown-typed body, or a type that differs from the HTML reference.

func NormalizeText 

func NormalizeText(s string) string

NormalizeText trims, then collapses every Unicode whitespace run to one ASCII space. Extract must run this on every block text.

func PathLooksLikeMarkdown 

func PathLooksLikeMarkdown(path string) bool

PathLooksLikeMarkdown is true for .md and .markdown paths.

func RedirectDiffers 

func RedirectDiffers(a, b Observation) bool

RedirectDiffers reports different final URLs after both served.

func ReportedTokens 

func ReportedTokens(h http.Header) (n int, name string, ok bool)

ReportedTokens reads a converter token estimate from headers. ok is false when none is present or parseable. Neither this value nor EstimateTokens is authoritative.

func ResolveAlternate 

func ResolveAlternate(pageURL, href string) string

ResolveAlternate is the comparison key for a markdown alternate. href is resolved against pageURL, then CanonicalURL. A variant hosted on another host is a different resource.

func RobotsToken 

func RobotsToken(k IdentityKind) string

RobotsToken is the robots.txt user-agent token used for k. Every identity, including impersonate, uses ProductToken. The tool never claims an allowance granted to another operator.

func SameBlocks 

func SameBlocks(a, b []Block) bool

SameBlocks reports identical sequences by Block.Equal, in order.

func SentToOrigin 

func SentToOrigin(o Outcome) bool

SentToOrigin is true when the tool performed an HTTP exchange. Unavailable and disallowed are client-side skips.

func ShuffleSeed 

func ShuffleSeed() uint64

ShuffleSeed is a non-zero seed from the clock. Tests that need a fixed order call ShuffledIdentities with a constant.

func StripFrontmatter 

func StripFrontmatter(src []byte) (body []byte, front string)

StripFrontmatter removes a leading YAML (---) or TOML (+++) fence from markdown. The fence body is returned as front. body is the rest, or the original input when no fence is present.

func TokenSource 

func TokenSource(blocks []Block) string

TokenSource joins block texts in order with newlines. Used as the input to EstimateTokens.

func TrioEligible 

func TrioEligible(reps []Observation) bool

TrioEligible is true when every unsigned replica can sit in the stable-core denominator. Empty replicas cannot. Uncertain can.

func TrioUnstable 

func TrioUnstable(reps []Observation, core []Block) bool

TrioUnstable is true when the three unsigned replicas do not support a meaningful core comparison: outcomes differ, final URLs differ, or the core is empty while at least one replica has blocks.

func UserAgent 

func UserAgent(k IdentityKind) string

UserAgent returns the User-Agent string fetch must send for k in default (honest) mode. Browser is empty: Chrome supplies its own. Impersonate also returns ProductUA here; fetch sends ImpersonateUA only when the caller has opted in.

func VaryCoversAccept 

func VaryCoversAccept(h http.Header) bool

VaryCoversAccept reports whether Vary lists Accept, or is "*".

Types

type Aggregates 

type Aggregates struct {
	Sites int `json:"sites"`

	// PrimaryFlags is how many site-context rows were flagged
	// primary before collapse. Larger than Sites when a host had
	// more than one primary page.
	PrimaryFlags int `json:"primary_flags,omitempty"`

	// Outcomes is a count per identity per outcome.
	Outcomes map[IdentityKind]map[Outcome]int `json:"outcomes"`

	SignedVsUnsignedContentRaw   Share `json:"signed_vs_unsigned_content_raw"`
	SignedVsUnsignedContentNet   Share `json:"signed_vs_unsigned_content_net"`
	SignedVsUnsignedRedirectRaw  Share `json:"signed_vs_unsigned_redirect_raw"`
	SignedVsUnsignedRedirectNet  Share `json:"signed_vs_unsigned_redirect_net"`
	SignedVsUnsignedChallengeRaw Share `json:"signed_vs_unsigned_challenge_raw"`
	SignedVsUnsignedChallengeNet Share `json:"signed_vs_unsigned_challenge_net"`

	SignedVsUnsignedLinkTargetRaw Share `json:"signed_vs_unsigned_link_target_raw"`
	SignedVsUnsignedLinkTargetNet Share `json:"signed_vs_unsigned_link_target_net"`

	// Unstable: the three unsigned replicas do not support a
	// meaningful core comparison on the primary page.
	Unstable Share `json:"unstable"`

	// CoreTooSmall: all three unsigned replicas were served with
	// blocks, but the intersection is empty. Counted in Unstable
	// and reported separately so it is not a silent drop.
	CoreTooSmall Share `json:"core_too_small"`

	// CoreCoverageMean is the mean of len(core)/max(replica
	// lengths) over primary pages whose three unsigned replicas
	// were served. N is that denominator.
	CoreCoverageMean Share `json:"core_coverage_mean"`

	// InteriorFound: an interior content URL was chosen (not a
	// homepage fallback), over sites.
	InteriorFound Share `json:"interior_found"`

	// Empty: observations that reached the origin and extracted
	// zero blocks, over observations sent to the origin.
	Empty Share `json:"empty"`

	// Uncertain: observations with weak challenge evidence, over
	// observations sent to the origin. Counted in neither the
	// served set nor the challenged set.
	Uncertain Share `json:"uncertain"`

	// MarkdownLostBodyContent: a real markdown variant is missing
	// HTML body content. Chrome-only loss is not a hit.
	MarkdownLostBodyContent Share `json:"markdown_lost_body_content"`

	// MarkdownMissingVary: a real markdown variant is missing
	// Vary: Accept. Separate from content loss.
	MarkdownMissingVary Share `json:"markdown_missing_vary"`

	// AlternateMismatch is a count of sites whose head and Link
	// header markdown alternates disagree. A finding, not a score.
	AlternateMismatch int `json:"alternate_mismatch"`

	// DuplicatePrimaryHosts is how many hosts had more than one
	// page flagged primary. Those hosts are collapsed to one page
	// before any share is computed. Zero means the denominator is
	// already one page per host.
	DuplicatePrimaryHosts int `json:"duplicate_primary_hosts,omitempty"`
}

Aggregates is the publishable summary: counts, shares, and platform breakdowns. It contains no site identifiers.

type Alternate 

type Alternate struct {
	Href     string          `json:"href"`
	Resolved string          `json:"resolved,omitempty"`
	Type     string          `json:"type"`
	Source   AlternateSource `json:"source"`
}

Alternate is a markdown alternate from the document head or Link header.

type AlternateMismatch 

type AlternateMismatch struct {
	OnlyHead   []AlternateRef `json:"only_head,omitempty"`
	OnlyHeader []AlternateRef `json:"only_header,omitempty"`
}

AlternateMismatch is a finding: head and Link header sets disagree. One-sided declarations count. Not a score.

func (AlternateMismatch) HasMismatch 

func (m AlternateMismatch) HasMismatch() bool

HasMismatch reports any href present in only one source.

type AlternateRef 

type AlternateRef struct {
	Declared string `json:"declared"`
	Resolved string `json:"resolved"`
}

AlternateRef is one one-sided declaration: what the site wrote and the absolute URL it resolves to.

type AlternateSource 

type AlternateSource string

AlternateSource says where a markdown alternate was declared. 

const (
	AlternateHead   AlternateSource = "head"
	AlternateHeader AlternateSource = "header"
)

type Block 

type Block struct {
	Kind   BlockKind `json:"kind"`
	Text   string    `json:"text"`
	Target string    `json:"target,omitempty"`
	Level  int       `json:"level,omitempty"`
	Index  int       `json:"index"`
	// Chrome is navigation or interface chrome: nav, page header,
	// page footer, complementary rails, and search widgets.
	// Article headers and footers inside main or article are not.
	// Comparison ignores this flag.
	Chrome bool `json:"chrome,omitempty"`
}

Block is one item in the normalized sequence.

func ProjectOntoCore 

func ProjectOntoCore(seq, core []Block) []Block

ProjectOntoCore keeps blocks from seq that appear in core, by Equal, preserving seq order and consuming each core key once.

func StableCore 

func StableCore(replicas ...[]Block) []Block

StableCore is the blocks that appear, by Equal, in every replica. Order and Index follow the first replica. Target is ignored, as in Equal: a changed href does not drop a block from the core.

func (Block) Equal 

func (b Block) Equal(o Block) bool

Equal compares visible text. Kind and Level must match. Target and Chrome are ignored: a changed href or a chrome flag is not a different text block.

type BlockDiff 

type BlockDiff struct {
	Op         DiffOp  `json:"op"`
	LeftIndex  int     `json:"left_index"`
	RightIndex int     `json:"right_index"`
	Left       *Block  `json:"left,omitempty"`
	Right      *Block  `json:"right,omitempty"`
	Confidence float64 `json:"confidence"`
	Evidence   string  `json:"evidence"`
}

BlockDiff is one explainable difference.

func MissingBodyBlocks 

func MissingBodyBlocks(diff Comparison, markdown []Block) []BlockDiff

MissingBodyBlocks are DiffMissing entries whose left block is body content that is not in the markdown sequence. Chrome, empty punctuation, short interface links, and the same words under a different block kind are omitted.

type BlockKind 

type BlockKind string

BlockKind is a semantic unit after extraction. Not a markup role. 

const (
	BlockHeading   BlockKind = "heading"
	BlockParagraph BlockKind = "paragraph"
	BlockListItem  BlockKind = "list_item"
	BlockLink      BlockKind = "link"
	BlockText      BlockKind = "text"
)

type BodyEvidence 

type BodyEvidence struct {
	RunID    string
	Domain   string
	PageURL  string
	Identity IdentityKind
	Body     []byte
	Capped   bool
	Cap      int
	RawBytes int
}

BodyEvidence is one stored response body. It is local evidence and is never part of a publishable output.

type BodyStore 

type BodyStore interface {
	Put(ctx context.Context, rec BodyEvidence) error
	Get(ctx context.Context, runID, domain, pageURL string, id IdentityKind) (BodyEvidence, bool, error)
}

BodyStore persists compressed response bodies outside the metrics database. Implementations must not write bodies into Store.

type CacheState 

type CacheState struct {
	CacheControl  string `json:"cache_control,omitempty"`
	Age           string `json:"age,omitempty"`
	Expires       string `json:"expires,omitempty"`
	XCache        string `json:"x_cache,omitempty"`
	CFCacheStatus string `json:"cf_cache_status,omitempty"`
	XCacheHits    string `json:"x_cache_hits,omitempty"`
}

CacheState is what the origin (or an edge) said about this representation. Recorded as observed. Not a score.

func CacheStateFromHeaders 

func CacheStateFromHeaders(h http.Header) CacheState

CacheStateFromHeaders reads the usual cache and edge headers.

func (CacheState) Present 

func (c CacheState) Present() bool

Present is true when any cache field was set on the response.

type ChallengeFamily 

type ChallengeFamily struct {
	Name      string
	Headers   []familyHeader
	Locations []familyLocation
	Body      []familyBody
	// Phrases, if set, replace the default interstitial-copy list
	// when deciding whether a contextual or widget body marker is
	// strong for this family. Empty means the default list.
	Phrases []string
}

ChallengeFamily is a named detector with its own signals and evidence rules. Adding a family does not require changing the classifier: append it to ChallengeFamilies.

type ChallengeSignal 

type ChallengeSignal struct {
	Family   string `json:"family,omitempty"`
	Channel  string `json:"channel"`
	Name     string `json:"name"`
	Evidence string `json:"evidence,omitempty"`
	Offset   int    `json:"offset,omitempty"`
	Strength string `json:"strength,omitempty"`
}

ChallengeSignal is one audited reason a response was classified as a challenge or left uncertain. Family names the detector that produced it. Channel is header, location, or body. Body signals include the byte offset of the marker and a snippet of surrounding text so the call can be checked.

func ChallengeSignals 

func ChallengeSignals(status int, h http.Header, body []byte) []ChallengeSignal

ChallengeSignals walks ChallengeFamilies. An empty slice means no challenge evidence. The classifier does not mention families by name.

func (ChallengeSignal) String 

func (s ChallengeSignal) String() string

func (ChallengeSignal) Strong 

func (s ChallengeSignal) Strong() bool

type Comparer 

type Comparer interface {
	Compare(left, right []Block) Comparison
}

Comparer matches two normalized block sequences.

The comparison is symmetric: Compare(B, A) is the mirror of Compare(A, B). Missing and added swap. Altered pairs swap indexes. Matched and MatchRatio are unchanged. If a tie rule would break that, change the rule.

Deterministic procedure:

  1. Longest common subsequence over Block.Equal. Those pairs are matches. They do not appear in Diffs. When skip-left and skip-right have the same length, skip the block that is greater under BlockLess. That choice does not depend on which sequence is on the left. Matched links whose Target differs are recorded on LinkTargets, not in Diffs. Evidence: `link target text=%q left=%s right=%s`.
  2. Remaining same-Kind pairs: compute text similarity 1 - levenshtein(runes)/max(len). Walk candidate pairs by descending similarity, then BlockLess of the pair's lesser block, then the greater block. No left-index tiebreak. First pair at or above AlterationFloor wins; no reuse. Those are DiffAltered. Confidence is the similarity. Evidence: `altered kind=%s sim=%.4f left=%d right=%d`.
  3. Unpaired left: DiffMissing, Confidence 1, LeftIndex set, RightIndex -1. Evidence: `missing kind=%s left=%d`.
  4. Unpaired right: DiffAdded, Confidence 1, RightIndex set, LeftIndex -1. Evidence: `added kind=%s right=%d`.

Diffs are ordered: altered by left index, then missing by left index, then added by right index. Order is not part of symmetry.

Matched is the LCS length. MatchRatio is Matched / max(left, right), or 1 if both sequences are empty.

Empty vs empty is a perfect match. Never scores a site.

type Comparison 

type Comparison struct {
	LeftIdentity  IdentityKind       `json:"left_identity,omitempty"`
	RightIdentity IdentityKind       `json:"right_identity,omitempty"`
	Diffs         []BlockDiff        `json:"diffs"`
	LinkTargets   []LinkTargetChange `json:"link_targets,omitempty"`
	Matched       int                `json:"matched"`
	LeftCount     int                `json:"left_count"`
	RightCount    int                `json:"right_count"`
	MatchRatio    float64            `json:"match_ratio"`
}

Comparison is the full match of two sequences. LeftIdentity and RightIdentity are filled by the caller.

func (Comparison) HasDiffs 

func (c Comparison) HasDiffs() bool

HasDiffs reports any missing, added, or altered text block. Link target changes are reported separately and do not count.

func (Comparison) HasLinkTargetChanges 

func (c Comparison) HasLinkTargetChanges() bool

HasLinkTargetChanges reports a matched link whose href moved.

type ContextProbe 

type ContextProbe interface {
	Probe(ctx context.Context, pageURL string, html []byte, headers http.Header) (SiteContext, error)
	FetchOrigin(ctx context.Context, pageURL string, pause func(context.Context) error) (SiteContext, error)
	WithPage(sc SiteContext, pageURL string, html []byte, headers http.Header) SiteContext
}

ContextProbe gathers origin metadata for one page.

pageURL is the page under test. html is the already-fetched HTML (unsigned or signed body); it may be empty. headers are that response's headers and MUST be read.

Probe GETs {origin}/robots.txt and {origin}/llms.txt with ProductUA. Do not follow redirects off origin. A missing file is not an error.

Robots: one RobotsRule per AllIdentities(), using RobotsToken (always ProductToken). Evaluate the tool's own group and the wildcard group only. Never select a group for another operator's token. Longest matching Allow/Disallow for that agent group wins; a more specific User-agent group beats *. Empty robots.txt or fetch failure means allowed.

Each Alternate stores Href (as declared) and Resolved (ResolveAlternate). One-sided mismatch entries are AlternateRef with both fields.

llms.txt: parse markdown links and bare http(s) URLs. IsMarkdown is true when the path ends in .md or .markdown, or the type looks like markdown.

Alternates come from both:

  • document head: link[rel~=alternate]
  • HTTP Link headers (RFC 8288)

Keep those whose type is text/markdown or text/x-markdown, or whose href path ends in .md / .markdown. Set Source to head or header. Compare href sets with ResolveAlternate(pageURL, href). Keys are resolved absolute CanonicalURLs. A variant on another host is a different resource. A difference, including a one-sided declaration, is Mismatch.

This package exists only to explain outcomes.

FetchOrigin GETs robots.txt and llms.txt. pause, if non-nil, is called before each GET so the caller can apply the same per-host delay as identity fetches. WithPage attaches head and Link-header alternates from an already-fetched page and does not touch the network. Probe is FetchOrigin with a nil pause, then WithPage.

type DiffOp 

type DiffOp string

DiffOp is one reported difference. Exact matches are not listed. 

const (
	// DiffMissing: present on the left, absent on the right.
	DiffMissing DiffOp = "missing"

	// DiffAdded: present on the right, absent on the left.
	DiffAdded DiffOp = "added"

	// DiffAltered: paired by kind, text is not identical.
	DiffAltered DiffOp = "altered"
)

type DirectoryChecker 

type DirectoryChecker interface {
	CheckPublished(ctx context.Context) error
}

DirectoryChecker confirms the published key set contains the current key. Implemented by signing.Agent.

type DirectoryServer 

type DirectoryServer interface {
	ServeDirectory(http.ResponseWriter, *http.Request)
}

DirectoryServer serves GET /.well-known/http-message-signatures-directory. Implemented by package signing without importing this package.

type Extracted 

type Extracted struct {
	Blocks      []Block `json:"blocks"`
	TokenCount  int     `json:"token_count"`
	Frontmatter string  `json:"frontmatter,omitempty"`
}

Extracted is the pipeline output for one response body.

type Extractor 

type Extractor interface {
	Extract(contentType string, body []byte) (Extracted, error)
}

Extractor turns HTML or markdown into the same block sequence.

Markdown is rendered to HTML first, then both inputs use one walker.

Walker, document order, on the body (or the whole tree if no body):

  • ignore script, style, noscript, template
  • h1-h6: one heading (Level 1-6, text = visible text), then child links only
  • p: one paragraph, then child links
  • li: one list_item, then child links
  • a not already emitted as a child link: one link
  • other visible text (div, section, article, td, span at block scope, and similar): one text run, skipped if empty after NormalizeText

A child link is Kind=link, Text=anchor text, Target=href as written (trimmed, not resolved, not lowercased).

Chrome is set when the block is emitted from nav, search, menu, page-level header/footer/aside, role=banner/contentinfo, or (when the document has main or article) anything outside those regions. Header and footer inside main or article are body content unless they have a chrome landmark role. The flag is not part of Equal.

TokenCount is EstimateTokens(TokenSource(blocks)). Empty or unparseable bodies return zero blocks and a nil error. Unknown content types: if the body looks like HTML, parse as HTML; otherwise treat as markdown.

Markdown that begins with a YAML (---) or TOML (+++) fence is passed through StripFrontmatter first. The fence is Frontmatter, never a block, so it cannot appear as missing or added content.

type FetchRequest 

type FetchRequest struct {
	URL      string
	Identity IdentityKind
}

FetchRequest is one identity against one URL.

type FetchResult 

type FetchResult struct {
	Identity    IdentityKind
	URL         string
	FinalURL    string
	Outcome     Outcome
	StatusCode  int
	Headers     http.Header
	ContentType string
	Body        []byte
	ByteSize    int64
	Duration    time.Duration
	Err         string
	Challenge   []ChallengeSignal
	Cache       CacheState
	Platform    Platform
}

FetchResult is the raw observation before extraction.

type Fetcher 

type Fetcher interface {
	Fetch(ctx context.Context, req FetchRequest) FetchResult
	FetchAll(ctx context.Context, rawURL string) []FetchResult
	Available(kind IdentityKind) (ready bool, reason string)
}

Fetcher retrieves a URL as one or all identities.

Fetch never panics.

Config values are used as written. Zero retries means no retries. Zero redirects means do not follow redirects (record the first Location as FinalURL if present). A concurrency below 1 is invalid for FetchAll and must run as 1 worker so the call cannot hang. Callers that want the documented constants use fetch.DefaultConfig.

Redirects: follow up to Config.Redirects, record FinalURL. Retry: StatusCode 429, 503, and transport errors, up to Config.Retries, honoring Retry-After when present, otherwise DefaultRetryWait then 4x. Do not retry 4xx other than 429. After retries, map the final response with ClassifyHTTP, passing the body so challenge markers can be found. Record Challenge signals, CacheStateFromHeaders, and PlatformFromHeaders. Transport and deadline errors are OutcomeTransport.

Browser: render with Chrome, return documentElement.outerHTML. If Chrome is missing, OutcomeUnavailable. Classify the browser result from the navigation status, headers, and body the same way as HTTP identities.

Unsigned replicas (unsigned, unsigned_b, unsigned_c): GET with ProductUA and AcceptHTML. They are the same request, fetched three times so a stable core can be taken. Historical identity "control" is the same request and is not in AllIdentities.

Markdown: GET with ProductUA and AcceptMarkdown.

Signed: build the unsigned request first (same method, URL, User-Agent, Accept, header names, values, and relative order), then RequestSigner.Sign. Sign adds only Signature, Signature-Input, and Signature-Agent. It must not change any other request property. The signature is the only variable. If the signer is missing or AgentOrigin is empty, signed is OutcomeUnavailable.

Impersonate: not in AllIdentities. Fetch runs it only when the caller sets ImpersonateCrawler. Then it sends ImpersonateUA. Benchmark mode must refuse that option. Robots for impersonate still uses ProductToken.

Every default request identifies the tool. Do not add headers that bypass access control. The engine applies robots before Fetch (OutcomeDisallowed is set there).

FetchAll runs AllIdentities with at most Config.Concurrency in flight. Order of the returned slice is AllIdentities(). The engine fetches a shuffled copy per site.

type IdentityKind 

type IdentityKind string

IdentityKind is one observation identity. 

const (
	// IdentityBrowser is a full Chrome profile with JavaScript.
	// Optional at runtime: missing Chrome is OutcomeUnavailable.
	IdentityBrowser IdentityKind = "browser"

	// IdentityUnsigned is a plain GET that sends ProductUA.
	// This is the tool's own unsigned request. Two more replicas
	// (unsigned_b, unsigned_c) are fetched so a stable core can
	// be taken across three identical requests.
	IdentityUnsigned IdentityKind = "unsigned"

	// IdentityUnsignedB is the second unsigned replica.
	IdentityUnsignedB IdentityKind = "unsigned_b"

	// IdentityUnsignedC is the third unsigned replica.
	IdentityUnsignedC IdentityKind = "unsigned_c"

	// IdentityControl is the historical name for a second unsigned
	// fetch. It is not in AllIdentities. Old databases may still
	// contain it. Fetch still accepts it as unsigned-like.
	IdentityControl IdentityKind = "control"

	// IdentityMarkdown is a plain GET that prefers markdown via Accept.
	IdentityMarkdown IdentityKind = "markdown"

	// IdentitySigned is a controlled variant of the unsigned identity:
	// the unsigned request plus signature headers. Nothing else changes.
	// That is the basis of the headline finding.
	IdentitySigned IdentityKind = "signed"

	// IdentityImpersonate sends a third-party crawler User-Agent.
	// It is not in AllIdentities. Fetch only runs it when the caller
	// opts in. Benchmark mode refuses that option. It is for testing
	// sites you own or have permission to test, never for published data.
	IdentityImpersonate IdentityKind = "impersonate"
)

func AllIdentities 

func AllIdentities() []IdentityKind

AllIdentities is the default observation set. FetchAll returns this order. The engine shuffles a copy per site before fetching so cache warming is not mistaken for an identity effect. Impersonate is not included.

func ShuffledIdentities 

func ShuffledIdentities(seed uint64) []IdentityKind

ShuffledIdentities returns AllIdentities in a per-seed order.

func UnsignedReplicas 

func UnsignedReplicas() []IdentityKind

UnsignedReplicas are the three unsigned fetches used to build the stable core. They send the same request.

func (IdentityKind) Machine 

func (k IdentityKind) Machine() bool

Machine reports whether k is a non-browser identity.

func (IdentityKind) UnsignedLike 

func (k IdentityKind) UnsignedLike() bool

UnsignedLike is true for the unsigned replicas, historical control, and signed. Those send the same method, URL, User-Agent, and Accept. Signed then adds only the signature headers.

func (IdentityKind) UnsignedReplica 

func (k IdentityKind) UnsignedReplica() bool

UnsignedReplica is true for the three unsigned fetches that form the stable core.

func (IdentityKind) Valid 

func (k IdentityKind) Valid() bool
type LLMSLink struct {
	Href       string `json:"href"`
	IsMarkdown bool   `json:"is_markdown"`
}

LLMSLink is one link parsed from llms.txt.

type LLMSTxt 

type LLMSTxt struct {
	Present bool       `json:"present"`
	URL     string     `json:"url"`
	Status  int        `json:"status"`
	Links   []LLMSLink `json:"links"`
}

LLMSTxt is the /llms.txt probe. Presence is HTTP 200 with a body.

type LinkTargetChange 

type LinkTargetChange struct {
	LeftIndex   int    `json:"left_index"`
	RightIndex  int    `json:"right_index"`
	Text        string `json:"text"`
	LeftTarget  string `json:"left_target"`
	RightTarget string `json:"right_target"`
	Evidence    string `json:"evidence"`
}

LinkTargetChange is a matched link whose visible text is the same and whose href changed. It is not a content difference.

type Observation 

type Observation struct {
	RunID              string
	Domain             string
	PageURL            string
	FinalURL           string
	Identity           IdentityKind
	Outcome            Outcome
	StatusCode         int
	Headers            http.Header
	ContentType        string
	ByteSize           int64
	TokenCount         int
	TokenCountReported int
	TokenCountHeader   string
	Challenge          []ChallengeSignal
	Cache              CacheState
	Platform           Platform
	Blocks             []Block
	Duration           time.Duration
	Err                string
	FetchedAt          time.Time
}

Observation is one identity's stored result. The raw body is kept in a BodyStore, not here, so the metrics database stays portable. TokenCount is our estimate. TokenCountReported is the converter header, if any. Neither is authoritative.

type Outcome 

type Outcome string

Outcome is the result of one identity's attempt. A completed HTTP exchange is not the same as a successful measurement. 

const (
	// OutcomeServed: the origin returned a representation of the
	// requested resource, and extraction produced at least one
	// block. Not a challenge, 429, or 401/403.
	OutcomeServed Outcome = "served"

	// OutcomeEmpty: the origin returned a finished exchange that
	// classified as served, but extraction produced zero blocks.
	// Distinct from served. Not a successful measurement.
	OutcomeEmpty Outcome = "empty"

	// OutcomeRateLimited: final status 429 after retries.
	OutcomeRateLimited Outcome = "rate_limited"

	// OutcomeBlocked: origin refused with 401/403 and no challenge
	// signal. Distinct from robots disallowed.
	OutcomeBlocked Outcome = "blocked"

	// OutcomeChallenged: interstitial or verification page. Often HTTP
	// 200. Detected from strong response signals, never from content
	// compare.
	OutcomeChallenged Outcome = "challenged"

	// OutcomeUncertain: a body marker is present but the surrounding
	// evidence is not enough to call the page a challenge. The
	// response is still measured. The outcome is a qualifier.
	OutcomeUncertain Outcome = "uncertain"

	// OutcomeRedirectLoop: redirect hop limit hit.
	OutcomeRedirectLoop Outcome = "redirect_loop"

	// OutcomeTransport: network failure, timeout, or 5xx after retries.
	OutcomeTransport Outcome = "transport"

	// OutcomeUnavailable: this process cannot run the identity
	// (no Chrome, no signer, no agent origin). No request was sent.
	OutcomeUnavailable Outcome = "unavailable"

	// OutcomeDisallowed: robots.txt forbids this identity. No request
	// was sent.
	OutcomeDisallowed Outcome = "disallowed"
)

func AllOutcomes 

func AllOutcomes() []Outcome

AllOutcomes is the stable print order for aggregate breakdowns.

func ClassifyHTTP 

func ClassifyHTTP(status int, h http.Header, body []byte, redirectLoop bool) Outcome

ClassifyHTTP maps a finished HTTP exchange to an outcome. redirectLoop is true when the hop limit stopped the client. 5xx after retries is OutcomeTransport. A strong challenge wins over 401/403. Weak body evidence is OutcomeUncertain.

func (Outcome) Valid 

func (o Outcome) Valid() bool

type PageRole 

type PageRole string

PageRole is whether a stored page is the homepage or the interior content URL chosen from it. 

const (
	PageHome     PageRole = "home"
	PageInterior PageRole = "interior"
)

type PageSelection 

type PageSelection struct {
	Role     PageRole `json:"role"`
	Primary  bool     `json:"primary"`
	Rule     string   `json:"rule,omitempty"`
	Evidence string   `json:"evidence,omitempty"`
	Source   string   `json:"source_url,omitempty"`
	Href     string   `json:"href,omitempty"`
}

PageSelection records how a target URL was chosen. It is audit evidence, not a score.

type Platform 

type Platform struct {
	Class        string `json:"class"` // cdn, edge, or origin
	Name         string `json:"name,omitempty"`
	MarkdownEdge bool   `json:"markdown_edge,omitempty"`
}

Platform is inferred from response headers only. No extra fetch.

func PlatformFromHeaders 

func PlatformFromHeaders(h http.Header) Platform

PlatformFromHeaders classifies CDN, edge platform, or origin server from headers already on the response.

func (Platform) Key 

func (p Platform) Key() string

Key is the stable aggregate bucket: class:name[+markdown].

type RequestSigner 

type RequestSigner interface {
	Sign(req *http.Request) error
}

RequestSigner attaches web-bot-auth headers to req. Implemented by package signing without importing this package.

type RobotsRule 

type RobotsRule struct {
	Identity   IdentityKind  `json:"identity"`
	Allowed    bool          `json:"allowed"`
	Agent      string        `json:"agent"`
	Pattern    string        `json:"pattern,omitempty"`
	CrawlDelay time.Duration `json:"crawl_delay,omitempty"`
}

RobotsRule is the robots.txt decision for one identity.

type Run 

type Run struct {
	ID           string
	StartedAt    time.Time
	FinishedAt   *time.Time
	Status       RunStatus
	DomainList   string
	Settings     RunSettings
	HeartbeatAt  *time.Time
	HeartbeatPID int
	StopReason   string
	LastHost     string
}

Run is one benchmark execution. Resume reuses an unfinished row after the heartbeat says the previous process is gone.

func (Run) HeartbeatLive 

func (r Run) HeartbeatLive(staleAfter time.Duration, alive func(int) bool) bool

HeartbeatLive is true when a process still owns this running row.

type RunSettings 

type RunSettings struct {
	DomainSHA256 string `json:"domain_sha256"`
	KeyID        string `json:"key_id,omitempty"`
	AgentOrigin  string `json:"agent_origin,omitempty"`
	Concurrency  int    `json:"concurrency"`
	Delay        string `json:"delay"`
	Timeout      string `json:"timeout"`
	Retries      int    `json:"retries"`
	Redirects    int    `json:"redirects"`
}

RunSettings is the parameter set that produced a run. Resume refuses to continue when these do not match the stored row.

func ParseRunSettings 

func ParseRunSettings(raw string) RunSettings

func (RunSettings) Empty 

func (s RunSettings) Empty() bool

func (RunSettings) Equal 

func (s RunSettings) Equal(o RunSettings) bool

func (RunSettings) Marshal 

func (s RunSettings) Marshal() string

func (RunSettings) Mismatch 

func (s RunSettings) Mismatch(o RunSettings) string

type RunStatus 

type RunStatus string

RunStatus is the lifecycle of one benchmark pass. 

const (
	RunRunning     RunStatus = "running"
	RunComplete    RunStatus = "complete"
	RunInterrupted RunStatus = "interrupted"
)

type Share 

type Share struct {
	Rate       float64          `json:"rate"`
	N          int              `json:"n"`
	ByPlatform map[string]Share `json:"by_platform,omitempty"`
}

Share is one rate with its denominator. Rate is 0 when N is 0. ByPlatform breaks the same share down by Platform.Key. Not a grade.

type SiteContext 

type SiteContext struct {
	Robots     []RobotsRule      `json:"robots"`
	RobotsRaw  []byte            `json:"-"`
	LLMSTxt    LLMSTxt           `json:"llms_txt"`
	Alternates []Alternate       `json:"alternates"`
	Mismatch   AlternateMismatch `json:"mismatch"`
	Selection  PageSelection     `json:"selection"`
}

SiteContext explains fetch outcomes. It is never scored and must not be mixed into Comparison.

type SiteContextRecord 

type SiteContextRecord struct {
	RunID   string      `json:"run_id"`
	Domain  string      `json:"domain"`
	PageURL string      `json:"page_url"`
	Context SiteContext `json:"context"`
}

SiteContextRecord is one persisted probe.

type Store 

type Store interface {
	CreateRun(ctx context.Context, domainList string, settings RunSettings) (Run, error)
	FinishRun(ctx context.Context, id string) error
	StopRun(ctx context.Context, id, reason string) error
	ResumeRun(ctx context.Context, id string, settings RunSettings) error
	SetRunSettings(ctx context.Context, id string, settings RunSettings) error
	TouchHeartbeat(ctx context.Context, id string, pid int, lastHost string) error
	Run(ctx context.Context, id string) (Run, error)
	LatestRunning(ctx context.Context) (Run, bool, error)
	LatestUnfinished(ctx context.Context) (Run, bool, error)

	SaveObservation(ctx context.Context, o Observation) error
	HasObservation(ctx context.Context, runID, domain, pageURL string, id IdentityKind) (bool, error)
	Observation(ctx context.Context, runID, domain, pageURL string, id IdentityKind) (Observation, bool, error)
	ObservationsByRun(ctx context.Context, runID string) ([]Observation, error)
	ObservationsByDomain(ctx context.Context, runID, domain string) ([]Observation, error)

	SaveSiteContext(ctx context.Context, rec SiteContextRecord) error
	SiteContext(ctx context.Context, runID, domain, pageURL string) (SiteContextRecord, bool, error)
	SiteContextsByRun(ctx context.Context, runID string) ([]SiteContextRecord, error)
}

Store persists runs, observations, and site context. SaveObservation upserts on (run_id, domain, page_url, identity).

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.