extract

Documentation

Overview

Package extract pulls named fields out of HTML with CSS selectors.

Each field lists selectors in order of preference; the first one that yields a non-empty value wins, so a site's old and new markup can both be listed while it migrates. A field marked Required that no selector finds is an error naming the field — markup changes fail loudly instead of quietly producing records full of empty strings.

Index

• Variables
• func Parse(r io.Reader) (*goquery.Document, error)
• type Field
• type MissingFieldError
  • func (e *MissingFieldError) Error() string
  • func (e *MissingFieldError) Is(target error) bool
• type NoItemsError
  • func (e *NoItemsError) Error() string
  • func (e *NoItemsError) Is(target error) bool
• type Record
• type Selector
  • func Attr(css, attr string) Selector
  • func Text(css string) Selector
  • func (s Selector) String() string
• type Set
  • func (s Set) Extract(root *goquery.Selection) (Record, error)
  • func (s Set) ExtractAll(root *goquery.Selection, itemCSS string) ([]Record, error)
  • func (s Set) ExtractAllHTML(r io.Reader, itemCSS string) ([]Record, error)
  • func (s Set) ExtractHTML(r io.Reader) (Record, error)
  • func (s Set) Validate() error

Variables

var ErrMissingField = errors.New("extract: required field missing")

ErrMissingField matches any *MissingFieldError with errors.Is.

var ErrNoItems = errors.New("extract: no items matched")

ErrNoItems matches any *NoItemsError with errors.Is.

Functions

func Parse

func Parse(r io.Reader) (*goquery.Document, error)

Parse reads an HTML document.

Types

type Field

type Field struct {
	Name      string
	Selectors []Selector
	Required  bool
}

Field is one named value and the selectors that may hold it, best first.

type MissingFieldError

type MissingFieldError struct {
	Fields []string
	// Item is the index of the item, for ExtractAll, or -1.
	Item int
}

MissingFieldError names the required fields no selector found.

func (*MissingFieldError) Error

func (e *MissingFieldError) Error() string

func (*MissingFieldError) Is

func (e *MissingFieldError) Is(target error) bool

Is makes errors.Is(err, ErrMissingField) true.

type NoItemsError

type NoItemsError struct {
	Selector string
}

NoItemsError is returned by ExtractAll when the item selector matches nothing, which usually means the list's markup changed.

func (*NoItemsError) Error

func (e *NoItemsError) Error() string

func (*NoItemsError) Is

func (e *NoItemsError) Is(target error) bool

Is makes errors.Is(err, ErrNoItems) true.

type Record

type Record map[string]string

Record is one extraction: field name to value. Optional fields that matched nothing are absent.

type Selector

type Selector struct {
	CSS  string
	Attr string
}

Selector finds a value: the text of the first element matching CSS, or the named attribute of it when Attr is set.

func Attr

func Attr(css, attr string) Selector

Attr selects an attribute of the first element matching css.

func Text

func Text(css string) Selector

Text selects the text of the first element matching css.

func (Selector) String

func (s Selector) String() string

String renders a selector for error messages: "css" or "css@attr".

type Set

type Set struct {
	Fields []Field
}

Set is the fields extracted together, such as the parts of one product.

func (Set) Extract

func (s Set) Extract(root *goquery.Selection) (Record, error)

Extract pulls one record from root. On a missing required field it returns the fields it did find along with a *MissingFieldError.

func (Set) ExtractAll

func (s Set) ExtractAll(root *goquery.Selection, itemCSS string) ([]Record, error)

ExtractAll pulls one record from each element matching itemCSS, such as each product card on a listing page. It fails if nothing matches itemCSS, or on the first item missing a required field.

func (Set) ExtractAllHTML

func (s Set) ExtractAllHTML(r io.Reader, itemCSS string) ([]Record, error)

ExtractAllHTML parses the HTML in r and runs ExtractAll over the whole document.

func (Set) ExtractHTML

func (s Set) ExtractHTML(r io.Reader) (Record, error)

ExtractHTML validates the set, parses the HTML in r and extracts one record from the whole document.

func (Set) Validate

func (s Set) Validate() error

Validate checks that every field has a unique, non-empty name and at least one selector, and that every selector compiles.