dataset

Documentation

Overview

Package dataset builds the committed pilot and benchmark domain lists from a pinned Tranco ranking.

Rebuild:

content-parity dataset -list W36Q9 -out internal/dataset

The selection is deterministic for a given list, pool, and seed. Tests use committed fixtures and do not call the live network.

Index

• Constants
• Variables
• func Classify(host string, status int, hdr http.Header, body []byte, sig pageSignals) (sector, signal, exclude, detail string)
• func CountryFromDomain(domain string) string
• func CountryFromIP(ip net.IP) string
• func CountryFromLang(tag string) string
• func ParseTSV(r io.Reader) (Meta, []Row, error)
• func RankTier(rank int) string
• func ResolveCountry(domain string, ip net.IP, langs ...string) string
• func WriteLists(dir string, meta Meta, benchmark, pilot []Row, excl []Exclusion) error
• type Exclusion
• type Meta
  • func LookupList(client *http.Client, listID string) (Meta, error)
• type Options
• type ProbeResult
• type Prober
  • func (p *Prober) Probe(ctx context.Context, t TrancoRow) ProbeResult
• type Result
  • func Build(ctx context.Context, opt Options) (Result, error)
• type Row
  • func SelectBenchmark(rows []Row, seed int64) []Row
  • func SelectPilot(benchmark []Row, seed int64) []Row
• type TrancoRow
  • func DownloadTranco(client *http.Client, downloadURL string) ([]TrancoRow, error)
  • func ParseTranco(r io.Reader) ([]TrancoRow, error)
  • func SelectPool(all []TrancoRow, n int) []TrancoRow

Constants

const (
	DefaultListID = "W36Q9"
	DefaultSeed   = 1
	DefaultPool   = 4500

	BenchmarkSize = 300
	PilotSize     = 30

	SectorPublisher = "publisher"
	SectorEcommerce = "ecommerce"
	SectorCorporate = "corporate"

	TierTop1k   = "1-1000"
	Tier1k10k   = "1000-10000"
	Tier10k100k = "10000-100000"

	ReasonLoginWall     = "login_wall"
	ReasonAdult         = "adult"
	ReasonHealthFinance = "health_finance_account"
	ReasonDocumentation = "documentation"
	ReasonRobots        = "robots_disallowed"
	ReasonUnreachable   = "unreachable"
	ReasonUnclassified  = "unclassified"
)

const DefaultMaxPool = 12000

Variables

var FocusCountries = []string{"TR", "DE", "FR", "JP", "BR"}

FocusCountries are required in the benchmark with meaningful counts.

Functions

func Classify

func Classify(host string, status int, hdr http.Header, body []byte, sig pageSignals) (sector, signal, exclude, detail string)

Classify inspects one homepage. exclude is set for login walls, adult content, documentation, and health/finance account areas.

func CountryFromDomain

func CountryFromDomain(domain string) string

CountryFromDomain is the ccTLD country, or "".

func CountryFromIP

func CountryFromIP(ip net.IP) string

CountryFromIP uses the committed IPv4 range table.

func CountryFromLang

func CountryFromLang(tag string) string

CountryFromLang maps Content-Language or html lang to ISO 3166-1. Bare "en" is not a country.

func ParseTSV

func ParseTSV(r io.Reader) (Meta, []Row, error)

ParseTSV reads a list file produced by WriteLists.

func RankTier

func RankTier(rank int) string

RankTier assigns a stratum from the Tranco rank.

func ResolveCountry

func ResolveCountry(domain string, ip net.IP, langs ...string) string

ResolveCountry: suffix, then the server address, then page language.

func WriteLists

func WriteLists(dir string, meta Meta, benchmark, pilot []Row, excl []Exclusion) error

WriteLists writes the annotated TSV files, domain-only lists, SOURCE, and exclusions.

Types

type Exclusion

type Exclusion struct {
	Domain string
	Rank   int
	Reason string
	Detail string
}

Exclusion is a domain dropped from the lists with a recorded reason.

type Meta

type Meta struct {
	ListID      string
	CreatedOn   string
	DownloadURL string
	Pool        int
	Seed        int64
	Probed      int
}

Meta is recorded in SOURCE so a rebuild can pin the same list.

func LookupList

func LookupList(client *http.Client, listID string) (Meta, error)

LookupList returns metadata for a pinned Tranco list identifier.

type Options

type Options struct {
	ListID  string
	Pool    int
	MaxPool int
	Seed    int64
	Workers int
	Delay   time.Duration
	Cache   string
	Tranco  []TrancoRow
	Meta    Meta
	Prober  Prober
	Log     func(string)
}

Options control a rebuild. Tests pass Tranco and Prober.Client.

type ProbeResult

type ProbeResult struct {
	Domain   string
	Rank     int
	RankTier string
	OK       bool
	Row      Row
	Exclude  Exclusion
}

ProbeResult is one host after robots and a single page GET.

type Prober

type Prober struct {
	Client  *http.Client
	Timeout time.Duration
	// BaseURL, if set, is used instead of https://{domain}. Tests use
	// this to point at a fixture server. The Host header stays the domain.
	BaseURL string
}

Prober fetches with this tool's identity. One page request per host.

func (*Prober) Probe

func (p *Prober) Probe(ctx context.Context, t TrancoRow) ProbeResult

Probe runs robots.txt then at most one homepage GET.

type Result

type Result struct {
	Meta       Meta
	Benchmark  []Row
	Pilot      []Row
	Exclusions []Exclusion
	Classified []Row
}

Result is the selected lists plus every recorded exclusion.

func Build

func Build(ctx context.Context, opt Options) (Result, error)

Build downloads (or uses) a pinned Tranco list, probes a spread pool, and selects the benchmark and pilot.

type Row

type Row struct {
	Domain       string
	Sector       string
	SectorSignal string
	Rank         int
	RankTier     string
	Country      string
	Platform     string
}

Row is one list entry. The TSV column order is the field order.

func SelectBenchmark

func SelectBenchmark(rows []Row, seed int64) []Row

SelectBenchmark fills the 300-domain strata from classified rows.

func SelectPilot

func SelectPilot(benchmark []Row, seed int64) []Row

SelectPilot is a proportional 30-domain sample of the benchmark.

type TrancoRow

type TrancoRow struct {
	Rank   int
	Domain string
}

TrancoRow is one ranking line.

func DownloadTranco

func DownloadTranco(client *http.Client, downloadURL string) ([]TrancoRow, error)

DownloadTranco GETs the pinned list CSV.

func ParseTranco

func ParseTranco(r io.Reader) ([]TrancoRow, error)

ParseTranco reads rank,domain CSV. A header line is skipped.

func SelectPool

func SelectPool(all []TrancoRow, n int) []TrancoRow

SelectPool takes a spread of ranks across the three tiers. n is the target size, on the order of a few thousand.