robots

Documentation

Overview

Package robots provides robots.txt parsing and compliance checking.

Index

• type Parser
  • func New(userAgent string) *Parser
  • func (p *Parser) FetchAndParse(ctx context.Context, baseURL string) error
  • func (p *Parser) GetCrawlDelay() time.Duration
  • func (p *Parser) IsAllowed(urlPath string) bool
  • func (p *Parser) Parse(reader io.Reader) error
  • func (p *Parser) RobotsTxtFound() bool

Types

type Parser

type Parser struct {
	// contains filtered or unexported fields
}

Parser handles robots.txt parsing and URL compliance checking

func New

func New(userAgent string) *Parser

New creates a new robots.txt parser

func (*Parser) FetchAndParse

func (p *Parser) FetchAndParse(ctx context.Context, baseURL string) error

FetchAndParse fetches and parses robots.txt from the given base URL

func (*Parser) GetCrawlDelay

func (p *Parser) GetCrawlDelay() time.Duration

GetCrawlDelay returns the crawl delay specified in robots.txt

func (*Parser) IsAllowed

func (p *Parser) IsAllowed(urlPath string) bool

IsAllowed checks if the given URL path is allowed by robots.txt

func (*Parser) Parse

func (p *Parser) Parse(reader io.Reader) error

Parse parses robots.txt content from a reader

func (*Parser) RobotsTxtFound

func (p *Parser) RobotsTxtFound() bool

RobotsTxtFound returns whether robots.txt was found and parsed