signals

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 10, 2026 License: AGPL-3.0 Imports: 19 Imported by: 0

Documentation

Overview

Tuning round 2: this file replaces round 1's sequential fetchPool with a bounded-concurrency version, plus the manifest.json fetch+parse (previously a blocking predecessor to the icon pool - cost data showed bimi(~120ms)+manifest(~245ms) stacking serially before icon fetches even started, contributing to 44/84 no-logo domains hitting the 2500ms wall clock mid-progress). ExtractAll now runs manifest.json and the whole icon-candidate pool AT THE SAME TIME.

Tuning round 3 (item 1): concurrent Fetch calls from manifest.json, favicon.ico, and every icon candidate all compete for the SAME shared, hard fetch-count budget (fetchsec.Client, 10 fetches/lookup). Round 2's design let each of them call client.Fetch independently and let the budget's own atomic counter sort out who wins - correct in the sense that the counter itself never over-commits (see fetchsec's budgetState), but under contention (more desired fetches than remaining slots) WHICH specific candidate wins a scarce slot became a function of goroutine scheduling, not Section 5 priority. ikea.com intermittently returned 0 logos instead of its real highest-priority candidate (apple-touch-icon) because of exactly this: on unlucky runs, a lower-priority concurrent fetch (manifest.json, favicon.ico) won the last available slot instead. admitByPriority is the fix: decide, synchronously and in priority order, WHICH candidates get to compete for the network at all, before any of them starts - only the admitted ones are then dispatched concurrently. This restores round 1's deterministic priority guarantee while keeping the actual network I/O concurrent.

Package signals implements Tier 1 brand-signal extraction per BRAND_API_SPEC_V1_2026-08.md Section 5, in the priority order specified there: BIMI, JSON-LD Organization, manifest.json, apple-touch-icon, link rel=icon, header/nav DOM heuristics, meta og tags, social anchors, and og:image as a flagged last resort. No stylesheet traversal, no font discovery, no headless browser, no LLM calls - see Classifier for where LLM-based conflict resolution would slot in later.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions 

func FindManifestLink(doc *html.Node, pageURL string) string

FindManifestLink returns the resolved href of <link rel="manifest">, if present.

func ParseHTML 

func ParseHTML(body []byte) (*html.Node, error)

ParseHTML parses raw homepage bytes into a DOM tree, tolerant of malformed markup (as real-world pages are).

func ValidateBIMISVG 

func ValidateBIMISVG(raw []byte) (ok bool, notes []string)

ValidateBIMISVG applies a lightweight proxy for SVG Tiny 1.2 Portable/ Secure conformance, per spec Section 5's "VALIDATE the record; do not blanket-trust": the fetched bytes must sniff as SVG, must sanitize cleanly (zero warnings - a conformant BIMI mark should never contain scripts/external refs/event handlers our sanitizer would strip; if it does, we do not trust it as a BIMI mark even though we still sanitized it), must declare a viewBox, and must stay under a plausible size cap. Full BIMI SVG Tiny PS schema validation (disallowed element/attribute enumeration beyond our sanitizer's blocklist) is out of scope for this prototype - see the extractor's ambiguity notes.

Types

type BIMILookup 

type BIMILookup struct {
	// contains filtered or unexported fields
}

BIMILookup is a handle to an in-progress or completed priority-(1) BIMI step (DNS TXT lookup, then - if a record validates - fetch + validate + sanitize the logo SVG it points at). Tuning round 2: BIMI's DNS lookup depends only on the domain, not on the homepage body, so it has no reason to wait for the homepage fetch to finish before starting - StartBIMILookup lets the caller (envelope.build) kick it off concurrently with normalize.ResolveTerminal's homepage fetch, then Wait() for it once the homepage is ready to be handed to ExtractAll.

func CompletedBIMILookup 

func CompletedBIMILookup(cand *LogoCandidate, warnings []string) *BIMILookup

CompletedBIMILookup wraps an already-known result as a BIMILookup whose Wait returns immediately - for callers (tests, or a caller that already has a BIMI result some other way) that don't need StartBIMILookup's goroutine.

func NoBIMILookup 

func NoBIMILookup() *BIMILookup

NoBIMILookup is CompletedBIMILookup(nil, nil): shorthand for "there is no BIMI candidate," the common case (most domains have no BIMI record at all) and the standard stand-in in tests that aren't exercising BIMI specifically.

func StartBIMILookup 

func StartBIMILookup(ctx context.Context, client Fetcher, domain string, lookupTXT TXTLookupFunc, now time.Time) *BIMILookup

StartBIMILookup immediately spawns a goroutine running the full BIMI step and returns a handle to it; call Wait to block for the result. client must be safe for concurrent use alongside whatever else the caller is doing with it at the same time (e.g. the homepage fetch) - see fetchsec.Client's budget accounting, made atomic under concurrency for exactly this purpose.

func (*BIMILookup) Wait 

func (b *BIMILookup) Wait(pageURL string) (*LogoCandidate, []string)

Wait blocks until the lookup finishes and returns its candidate (nil if none) and any warnings. pageURL is stamped onto a non-nil candidate's SourcePageURL here rather than at StartBIMILookup time, since the caller may not know the final homepage URL (post-redirect) until after StartBIMILookup was called - that's the whole point of starting it concurrently with the homepage fetch.

type BIMIResult 

type BIMIResult struct {
	Found   bool
	LogoURL string
	VMCURL  string // "a=" authority evidence location; recorded, never fetched by this prototype
}

BIMIResult is the outcome of a default._bimi.<domain> TXT lookup.

func LookupBIMI 

func LookupBIMI(ctx context.Context, domain string, lookupTXT TXTLookupFunc) *BIMIResult

LookupBIMI performs the Section 5 priority (1) DNS TXT lookup. A lookup error (including NXDOMAIN, the overwhelmingly common case since most domains have no BIMI record) is treated as "not found", not a hard error - BIMI is optional evidence.

type Classifier 

type Classifier interface {
	// ResolveText picks a single winner from competing name or
	// description candidates. Returns nil if candidates is empty.
	ResolveText(candidates []TextCandidate) *TextCandidate
}

Classifier resolves conflicts among competing candidate values for a single brand field into one winner. Spec Section 5 reserves LLM use for exactly this kind of "classification and conflict resolution" (never asset recovery/generation, schema-validated output, injection- resistant prompting, counted in cost metrics) - this interface is where that would slot in later. This prototype ships only HeuristicClassifier: deterministic, priority-rank-then-first-seen, no network/LLM calls.

type ColorCandidate 

type ColorCandidate struct {
	Hex           string
	Role          model.ColorRole
	Signal        model.SignalType
	SourcePageURL string
	ObservedAt    time.Time
}

ColorCandidate is a not-yet-resolved declared color (theme-color, manifest background/theme_color, or a CSS custom property). Quantized colors derived from the chosen logo are added later, by rank, with origin=derived; every ColorCandidate here is origin=declared.

type Fetcher 

type Fetcher interface {
	Fetch(ctx context.Context, rawURL string) (*fetchsec.FetchResult, error)
}

Fetcher is the subset of *fetchsec.Client ExtractAll depends on. See normalize.Fetcher for why this is a narrow interface rather than the concrete type.

type HeuristicClassifier 

type HeuristicClassifier struct{}

HeuristicClassifier is the only Classifier implementation in this prototype. It picks the lowest PriorityRank (highest spec-Section-5 priority); ties break on first-seen (stable candidate order, which callers should populate in extraction order) so results are deterministic and reproducible run-to-run.

func (HeuristicClassifier) ResolveText 

func (HeuristicClassifier) ResolveText(candidates []TextCandidate) *TextCandidate

type LinkCandidate 

type LinkCandidate struct {
	Category      string // "careers"|"privacy"|"terms"|"contact"|"blog"|"pricing"
	URL           string
	Signal        model.SignalType
	SourcePageURL string
	ObservedAt    time.Time
}

LinkCandidate is a discovered careers/privacy/terms/contact/blog/pricing link. Spec Section 5 does not define a dedicated signal for these (see the extractor's ambiguity notes); this implementation scans anchors site-wide on the homepage and tags them header_dom. 

func ExtractCommonLinks(doc *html.Node, pageURL string, now time.Time) []LinkCandidate

ExtractCommonLinks scans every anchor on the page for careers/privacy/ terms/contact/blog/pricing hrefs, first match per category wins.

type LogoCandidate 

type LogoCandidate struct {
	URL           string
	Signal        model.SignalType
	Rank          PriorityRank
	Bytes         []byte // fetched (and, for SVG, sanitized) bytes; nil if fetch failed/was skipped under budget
	SniffedFormat model.LogoFormat
	Width         int // declared or sniffed pixel width, 0 if unknown
	Height        int
	DeclaredSizes string // raw HTML "sizes" attribute, kept for diagnostics
	SourcePageURL string // page the reference to this asset was found on
	ObservedAt    time.Time
	FetchErr      error // set when the candidate was identified but fetching it failed
	SVGWarnings   []string
}

LogoCandidate is a not-yet-ranked, not-yet-deduped logo found by a signal extractor. rank.Rank turns a slice of these into the final []model.LogoAsset.

func FaviconFallback 

func FaviconFallback(ctx context.Context, client Fetcher, homepageURL string, now time.Time) (*LogoCandidate, []string)

FaviconFallback implements signal (8b): GET /favicon.ico by convention and, if it sniffs as a usable image, return it as a LogoCandidate ranked (via RankFaviconFallback) above only og:image.

Tuning round 2: ExtractAll now starts this concurrently with every other fetch (via startFaviconFallback below) rather than waiting to see whether signals 1-6 came up empty first - that "wait, then maybe fetch one more thing" was exactly the kind of serial tail latency this round targets. The "only when signals 1-6 produced nothing usable" condition from spec Section 5 (8b) still holds; it's now enforced by ExtractAll discarding this candidate (rather than not requesting it at all) when something better also finished in time.

Per spec Section 6 ("image bytes sniffed, not trusted"), the response is never trusted by its .ico extension: many hosts serve a PNG/WebP/JPEG/ SVG under that conventional path, which is handled the same way any other image signal is. A genuine ICO container is parsed for its real embedded-frame dimensions rather than left at 0x0 (see decodeICO) - standard library/x/image decoders have no ICO support at all, so without this the fallback would find bytes but never a usable width/height or a format any downstream code understands.

The FieldMeta signal tag is "link_icon": favicon.ico is exactly what a declared <link rel="icon"> would have pointed at, and Section 2's signal enum is frozen (this ruling did not add a new enum value) - the same convention already used for the apple-touch-icon.png and manifest.json fallbacks, which keep their own signals' tags. RankFaviconFallback (not the signal tag) is what actually encodes the "above only og:image" ordering.

func FindAppleTouchIcons 

func FindAppleTouchIcons(doc *html.Node, pageURL string, now time.Time) []LogoCandidate

FindAppleTouchIcons returns every <link rel="apple-touch-icon"> (and the "apple-touch-icon-precomposed" variant), priority (4).

func FindHeaderLogos 

func FindHeaderLogos(doc *html.Node, pageURL string, now time.Time) []LogoCandidate

FindHeaderLogos implements Section 5 priority (6): "header/nav DOM heuristics on raw HTML (img/svg in header, class/id ~ logo|brand, wrapped in href="/", alt ~ name)". A candidate is kept if it matches at least one of those signals, to avoid pulling in unrelated header icons (hamburger menus, search glyphs, social icons).

func FindLinkIcons 

func FindLinkIcons(doc *html.Node, pageURL string, now time.Time) []LogoCandidate

FindLinkIcons returns every <link rel="icon"> / "shortcut icon" variant, priority (5), ranked by declared size (largest first) by the caller (rank package); this just collects them with whatever size metadata was declared.

type Manifest 

type Manifest struct {
	Name            string
	ShortName       string
	ThemeColor      string
	BackgroundColor string
	Icons           []ManifestIcon
}

Manifest is the subset of a Web App Manifest Section 5 priority (3) cares about.

func ParseManifest 

func ParseManifest(body []byte) (*Manifest, error)

ParseManifest decodes a fetched manifest.json body. It tolerates trailing commas/BOM-less minor deviations the way real fetched manifests sometimes have by simply failing soft (returns an error the caller can log and move on from; manifest.json is optional evidence).

type ManifestIcon 

type ManifestIcon struct {
	Src   string
	Sizes string // e.g. "192x192", "512x512", "any"
	Type  string
}

ManifestIcon mirrors one entry of a web manifest's "icons" array.

type MetaTags 

type MetaTags struct {
	SiteName    string
	Description string
	ThemeColor  string
	OGImage     string
}

MetaTags collects the handful of <meta> values Section 5 priority (7) cares about: og:site_name, description (og:description falling back to the plain "description" meta), theme-color, and og:image (kept separately as the last-resort logo signal, priority 9).

func ExtractMetaTags 

func ExtractMetaTags(doc *html.Node, pageURL string) MetaTags

type OrgInfo 

type OrgInfo struct {
	Name        string
	Description string
	LogoURL     string
	SameAs      []string
}

OrgInfo is what we pull out of a schema.org Organization JSON-LD block: name/logo/description/sameAs, per spec Section 5 priority (2).

func ExtractJSONLDOrganization 

func ExtractJSONLDOrganization(doc *html.Node, pageURL string) *OrgInfo

ExtractJSONLDOrganization scans every <script type="application/ld+json"> block on the page and returns the first Organization-typed node found (depth-first, document order). It tolerates a bare object, an array of objects, or an object using "@graph".

type PriorityRank 

type PriorityRank int

PriorityRank mirrors the Section 5 signal priority order (lower = higher priority). Used by rank/classifier logic to prefer earlier signals when multiple candidates conflict. 

const (
	RankBIMI PriorityRank = iota + 1
	RankJSONLD
	RankManifest
	RankAppleTouchIcon
	RankLinkIcon
	RankHeaderDOM
	RankMetaOG
	RankSocialAnchor
	// RankFaviconFallback is Section 5 signal (8b), added 2026-08-08: the
	// /favicon.ico convention fallback used only when signals 1-6 yield
	// nothing. Ranked above only og:image, per that ruling.
	RankFaviconFallback
	RankOGImageLastResort
)

type Result 

type Result struct {
	NameCandidates        []TextCandidate
	DescriptionCandidates []TextCandidate
	LogoCandidates        []LogoCandidate
	ColorCandidates       []ColorCandidate
	SocialCandidates      []SocialCandidate
	LinkCandidates        []LinkCandidate

	// OrgNameFromJSONLD is the raw JSON-LD Organization name, if any,
	// exposed separately so the pipeline layer can compare it against a
	// subdomain's own JSON-LD name for the multi_brand heuristic in spec
	// Section 4 (that comparison needs two separate fetches/extractions
	// and so is orchestrated above this package, not inside it).
	OrgNameFromJSONLD string

	Warnings []string

	// SignalTimings is per-stage wall-clock time within ExtractAll, for
	// the optional debug block (spec Section 8c ruling 4).
	SignalTimings []SignalTiming
}

Result aggregates every candidate produced for one homepage, plus diagnostics.

func ExtractAll 

func ExtractAll(ctx context.Context, client Fetcher, homepageBody []byte, homepageURL, domain string, bimi *BIMILookup) (*Result, error)

ExtractAll runs every Tier 1 signal in spec Section 5's priority order against an already-fetched homepage, fetching subresources (BIMI SVG, manifest.json, and a priority-ordered pool of icon/logo candidates) through client, which shares its fetch budget with whatever the caller already spent resolving the homepage.

Tuning round 2 (spec Section 8c, latency): manifest.json, every pending icon/logo candidate, and the favicon.ico fallback are now fetched CONCURRENTLY (bounded worker pool, per-host politeness cap) in place of round 1's fully sequential pipeline - cost data showed BIMI (~120ms) and manifest (~245ms) alone stacking to ~713ms before icon fetches even started, which was the dominant cause of 44/84 no-logo domains hitting the 2500ms wall clock mid-progress. bimi is a *BIMILookup already started by the caller (envelope.build) concurrently with the homepage fetch itself - see StartBIMILookup - since BIMI's DNS lookup depends only on the domain, not on the homepage body.

type SignalTiming 

type SignalTiming struct {
	Signal   model.SignalType
	Duration time.Duration
}

SignalTiming records how long one named Tier 1 stage took inside ExtractAll, for the optional debug block (spec Section 8c ruling 4). Always collected (the overhead is a handful of time.Now() calls); it is the caller's choice whether to surface it in the envelope.

type SocialCandidate 

type SocialCandidate struct {
	Type          model.SocialType
	URL           string
	Signal        model.SignalType
	SourcePageURL string
	ObservedAt    time.Time
}

SocialCandidate is a discovered social profile link.

func ExtractSocialAnchors 

func ExtractSocialAnchors(doc *html.Node, pageURL string, now time.Time) []SocialCandidate

ExtractSocialAnchors implements Section 5 priority (8): "social link extraction from anchors". Scans every <a href> on the page (not scoped to header/nav, since social icons are as often in the footer).

func SocialCandidatesFromSameAs 

func SocialCandidatesFromSameAs(sameAs []string, pageURL string, now time.Time) []SocialCandidate

SocialCandidatesFromSameAs converts JSON-LD sameAs URLs into social candidates (signal json_ld, higher priority than anchor scraping).

type TXTLookupFunc 

type TXTLookupFunc func(ctx context.Context, name string) ([]string, error)

TXTLookupFunc abstracts DNS TXT lookup so tests can inject a fake resolver instead of touching real DNS. *net.Resolver's LookupTXT method already matches this signature.

type TextCandidate 

type TextCandidate struct {
	Value         string
	Signal        model.SignalType
	Rank          PriorityRank
	SourcePageURL string
	ObservedAt    time.Time
}

TextCandidate is a not-yet-resolved name/description value.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.