processors

Documentation

Index

• type Output
  • func (o Output) Process(url, content, title string, urls []string, err error)
• type Saver
  • func NewSaver(base string) Saver
  • func (s Saver) Process(url, content, title string, urls []string, previousErr error)

Types

type Output

type Output struct {
}

A simple processor that just outputs basic information to the standard output

func (Output) Process

func (o Output) Process(url, content, title string, urls []string, err error)

Process a fetched resource with its content, title, its children URLs and an optional error

type Saver

type Saver struct {
	// contains filtered or unexported fields
}

A saver processor handles web pages by saving their content to the disk after getting rid of HTML tags, multiple whitespaces and so on.

func NewSaver

func NewSaver(base string) Saver

Create a new saver processor.

func (Saver) Process

func (s Saver) Process(url, content, title string, urls []string, previousErr error)

Process a fetched page with its content, title, its children URLs and an optional error