excluders

Documentation

Index

• type RobotsTxtExcluder
  • func NewRobotsTxtExcluder(agent string, enable bool, url *urlpkg.URL) (*RobotsTxtExcluder, error)
  • func (t *RobotsTxtExcluder) Exclude(path string) bool

Types

type RobotsTxtExcluder

type RobotsTxtExcluder struct {
	// contains filtered or unexported fields
}

RobotsTxtExcluder bases its exclusion rules on a given robots.txt file

func NewRobotsTxtExcluder

func NewRobotsTxtExcluder(agent string, enable bool, url *urlpkg.URL) (*RobotsTxtExcluder, error)

func (*RobotsTxtExcluder) Exclude

func (t *RobotsTxtExcluder) Exclude(path string) bool

Exclude checks against the rules found in the robots.txt file to see if a given path should be excluded. It allows all paths if disabled.