robots

README

robots.txt

robots is a library written in golang for parsing the robots.txt file format.

Usage

First install the library with

go get xojoc.pw/crawl/robots

...then run...

go test

if something fails open an issue.

See godoc for the complete documentation.

Who?

robots was written by Alexandru Cojocaru (https://xojoc.pw).

Donate!

Documentation

Overview

Package robots parses a robots.txt file as specified by Wikipedia https://en.wikipedia.org/wiki/Robots.txt

Index

• type Txt
  • func Parse(r io.Reader) (*Txt, error)
  • func (t *Txt) Allowed(ua string, path string) bool
  • func (t *Txt) Delay(ua string) int

Types

type Txt

type Txt struct {
	CrawlDelay map[string]int
	Allow      map[string][]string
	Disallow   map[string][]string

	Sitemaps []string
}

Txt contains the robots.txt rules.

func Parse

func Parse(r io.Reader) (*Txt, error)

Parse parses a robots.txt file.

func (*Txt) Allowed

func (t *Txt) Allowed(ua string, path string) bool

Allowed returns true if user agent ua can access path. False otherwise.

func (*Txt) Delay

func (t *Txt) Delay(ua string) int

Delay returns the number of seconds to wait between successive accesses to the same host. Returns 0 if no delay is specified.