spider

Documentation

Index

• Constants
• type Options
  • func (o *Options) Default()
• type Spider
  • func New(opts *Options) *Spider
  • func (s *Spider) Run(URL string) error

Constants

const UserAgent = "wospi - https://github.com/vlad-s/wospi"

UserAgent is the default user agent of the Spider.

Types

type Options

type Options struct {
	MinLength int    // minimum length of words to be stored by the Spider
	MaxDepth  int    // maximum depth for the crawling
	UserAgent string // user agent used in HTTP requests

	// strict domain matching; if set, www sub-domain won't be added on non-www URL,
	// and non-www domain won't be added on www URL
	StrictDomain bool

	StripResult  bool
	StripCharset []rune
}

Options provides configuration params for the Spider.

func (*Options) Default

func (o *Options) Default()

Default applies default values to the Options.

type Spider

type Spider struct {
	*Options
	// contains filtered or unexported fields
}

Spider provides the options and engines for the scraper.

func New

func New(opts *Options) *Spider

New returns a new Spider.

func (*Spider) Run

func (s *Spider) Run(URL string) error

Run starts the crawling & scraping on the provided URL.