Documentation ¶
Overview ¶
Package robots handles caching robots.txt files
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Cacher ¶
type Cacher interface { IndexExists() (bool, error) Setup() error Put(*Robots) Get(host string) (*Robots, error) }
Cacher handles the caching backend for robots.txt files
type ElasticSearch ¶
type ElasticSearch struct { Client *elastic.Client Index string Type string Bulk *elastic.BulkProcessor }
ElasticSearch holds our Elasticsearch connection and index information
func (*ElasticSearch) Get ¶
func (e *ElasticSearch) Get(sh string) (*Robots, error)
Get retrieves a single cached robots.txt file
func (*ElasticSearch) IndexExists ¶
func (e *ElasticSearch) IndexExists() (bool, error)
IndexExists returns true if the index exists
func (*ElasticSearch) Mapping ¶
func (e *ElasticSearch) Mapping() string
Mapping is the mapping of our robots Index
func (*ElasticSearch) Setup ¶
func (e *ElasticSearch) Setup() error
Setup creates an index for caching robots.txt files
type Robots ¶
type Robots struct { SchemeHost string `json:"-"` StatusCode int `json:"status"` Body string `json:"body"` Expires string `json:"expires"` Cached bool `json:"-"` }
Robots is a single robots.txt response
func New ¶
New creates a new Robot & sets the ID to the host Robots are handled on a per-host basis https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt
func (*Robots) SetBody ¶
SetBody saves the body of a robots.txt file Since Group has unexported fields we must save the entire body, not just the group
func (*Robots) SetExpires ¶
SetExpires marks the date the robots.txt file should expire from cache
func (*Robots) SetStatusCode ¶
SetStatusCode sets the HTTP response code