session

Documentation

Overview

Package session keeps a pool of identities — a proxy (or a direct connection) and a cookie jar each, all sending the same honest header profile, language and timezone — and tracks how well each one is doing.

Every identity has a health score between 0 and 1. A good response raises it; a block or captcha lowers it, adds a strike and puts the identity on a cooldown that doubles with each strike; at MaxStrikes the identity is retired for good. A success clears the strikes, so it is blocks in a row that retire an identity, not blocks over its lifetime. Acquire hands out the healthiest identity not cooling down, preferring the least recently used among equals.

A pool with no proxies is valid: its identities connect directly.

Identities are not a way around blocks. Every identity sends the same honest profile, so none of them poses as a different browser, and forage.Client stops sending requests to a host after the first block or challenge from it, whichever identity received it.

Index

• Variables
• type Config
• type Health
• type Identity
  • func (id *Identity) Apply(req fetch.Request) fetch.Request
• type Pool
  • func NewPool(cfg Config) (*Pool, error)
  • func (p *Pool) Acquire(ctx context.Context) (*Identity, error)
  • func (p *Pool) Health() []Health
  • func (p *Pool) Len() int
  • func (p *Pool) Report(id *Identity, o detect.Outcome)
  • func (p *Pool) ReportError(id *Identity)

Variables

var ErrExhausted = errors.New("session: every identity is retired")

ErrExhausted is returned by Acquire when every identity is retired.

Types

type Config

type Config struct {
	// Proxies to spread identities over. Empty means direct connections.
	Proxies []*url.URL
	// Profile is the header set every identity sends. A zero Profile means
	// fetch.DefaultProfile. Identities never get different profiles, so
	// one client never poses as several kinds of browser.
	Profile fetch.Profile
	// Language, such as "en-US". Empty means "en-US".
	Language string
	// Timezone, such as "America/New_York", used by the browser fetcher.
	// Empty means none is set.
	Timezone string
	// Size is the number of identities. Zero means one per proxy, or one
	// if there are no proxies. Identity i takes proxy i modulo the number
	// of proxies.
	Size int

	// Cooldown is the pause after a first strike, doubling with each
	// further one. Default one minute.
	Cooldown time.Duration
	// MaxCooldown caps the doubling. Default 30 minutes.
	MaxCooldown time.Duration
	// MaxStrikes is how many blocks in a row retire an identity. Default 3.
	MaxStrikes int
}

Config sizes and tunes a Pool. The zero value is a pool of one direct identity using fetch.DefaultProfile.

type Health

type Health struct {
	ID           string
	Score        float64
	Strikes      int
	CoolingUntil time.Time // zero when not cooling
	Retired      bool
	Uses         int
}

Health is a snapshot of one identity's standing.

type Identity

type Identity struct {
	ID       string
	Proxy    *url.URL      // nil connects directly
	Profile  fetch.Profile // the pool's profile, the same for every identity
	Jar      http.CookieJar
	Language string
	Timezone string
	// contains filtered or unexported fields
}

Identity is one route and cookie jar a scraper uses.

func (*Identity) Apply

func (id *Identity) Apply(req fetch.Request) fetch.Request

Apply fills in the identity's proxy, profile, cookie jar, language and timezone on req.

type Pool

type Pool struct {
	// contains filtered or unexported fields
}

Pool hands out identities. It is safe for concurrent use.

func NewPool

func NewPool(cfg Config) (*Pool, error)

NewPool builds the identities described by cfg.

func (*Pool) Acquire

func (p *Pool) Acquire(ctx context.Context) (*Identity, error)

Acquire returns the healthiest identity that is not cooling down. If all of them are cooling it waits for the first to come back, or for ctx. If all of them are retired it returns ErrExhausted.

func (*Pool) Health

func (p *Pool) Health() []Health

Health returns every identity's standing, in creation order.

func (*Pool) Len

func (p *Pool) Len() int

Len returns the number of identities, retired ones included.

func (*Pool) Report

func (p *Pool) Report(id *Identity, o detect.Outcome)

Report records how a request made with id turned out.

OK raises the score and clears strikes. Empty lowers the score a little without a strike. Blocked and Captcha lower it a lot, add a strike, and start a cooldown — or retire the identity at MaxStrikes. Identities from another pool are ignored.

func (*Pool) ReportError

func (p *Pool) ReportError(id *Identity)

ReportError records a transport failure: the score drops a little, with no strike, since a dead connection says more about the network than about the identity.