robots

Documentation

Overview

Package robots is a small, pure robots.txt parser and matcher. kage obeys it by default so a clone stays polite; --no-robots bypasses the matcher and puts the consequences on the user.

Index

• type Matcher
  • func AllowAll() *Matcher
  • func Parse(data string, agent string) *Matcher
  • func (m *Matcher) Allowed(path string) bool

Types

type Matcher

type Matcher struct {
	Sitemaps   []string
	CrawlDelay time.Duration
	// contains filtered or unexported fields
}

Matcher answers whether a path may be crawled for one user-agent group, and carries the sitemaps and crawl-delay declared in the file.

func AllowAll

func AllowAll() *Matcher

AllowAll returns a Matcher that permits every path (used when robots is disabled or the file is absent).

func Parse

func Parse(data string, agent string) *Matcher

Parse reads robots.txt content and returns a Matcher for the given agent token (e.g. "kage"). The most specific matching group is used, falling back to the "*" group. Sitemaps are collected globally.

func (*Matcher) Allowed

func (m *Matcher) Allowed(path string) bool

Allowed reports whether path may be crawled. The longest matching rule wins; on a tie, Allow beats Disallow. An empty Disallow means "allow everything".