crawlerdetector

README

About Crawler Detector

Crawler Detector is a Golang package for detecting bots/crawlers/spiders via the user agent.

Installation

Run go get github.com/dxas90/crawlerdetector.

Usage

detector := crawlerdetector.New()
isCrawler := detector.IsCrawler(r.Header.Get("User-Agent"))

if isCrawler {
    // Do something
}

Contributing

If you find a bot/spider/crawler user agent that Crawler Detector not detect, please submit a pull request with the regex pattern.

Documentation

Index

• func CrawlersList() string
• func ExclusionsList() string
• func MobilesList() string
• func PiwikCrawlersList() string
• func PiwikMobilesList() string
• func ShortCrawlersList() string
• func ShortExclusionsList() string
• func ShortMobilesList() string
• type CrawlerDetector
  • func New() *CrawlerDetector
  • func NewPiwik() *CrawlerDetector
  • func NewShort() *CrawlerDetector
  • func (cd *CrawlerDetector) GetMatched() []string
  • func (cd *CrawlerDetector) IsCrawler(userAgent string) bool
  • func (cd *CrawlerDetector) IsExclusion(userAgent string) bool
  • func (cd *CrawlerDetector) IsMobile(userAgent string) bool
  • func (cd *CrawlerDetector) Parse(userAgent string) *CrawlerDetector
  • func (cd *CrawlerDetector) ParseUnsafe(userAgent string) *CrawlerDetector
• type Piwik
  • func (m *Piwik) String() string

Functions

func CrawlersList

func CrawlersList() string

CrawlersList is list of crawlers/spiders/bots

func ExclusionsList

func ExclusionsList() string

ExclusionsList is list of exclusions

func MobilesList

func MobilesList() string

MobilesList is list of most used mobiles

func PiwikCrawlersList

func PiwikCrawlersList() string

PiwikCrawlersList is list of crawlers/spiders/bots by Piwik

func PiwikMobilesList

func PiwikMobilesList() string

PiwikMobilesList is list of most used mobiles

func ShortCrawlersList

func ShortCrawlersList() string

ShortCrawlersList is list of crawlers/spiders/bots

func ShortExclusionsList

func ShortExclusionsList() string

ShortExclusionsList is list of exclusions

func ShortMobilesList

func ShortMobilesList() string

ShortMobilesList is a small list of most used mobiles

Types

type CrawlerDetector

type CrawlerDetector struct {
	Crawlers   *regexp.Regexp
	Mobiles    *regexp.Regexp
	Exclusions *regexp.Regexp
	Matched    []string
	Browser    bool
	Crawler    bool
	Mobile     bool
}

CrawlerDetector is crawler detector structure

func New

func New() *CrawlerDetector

New returns a new initialized CrawlerDetector

func NewPiwik

func NewPiwik() *CrawlerDetector

NewPiwik returns a new initialized CrawlerDetector from Piwik

func NewShort

func NewShort() *CrawlerDetector

NewShort returns a new basic initialized CrawlerDetector

func (*CrawlerDetector) GetMatched

func (cd *CrawlerDetector) GetMatched() []string

GetMatched is getter of matched result

func (*CrawlerDetector) IsCrawler

func (cd *CrawlerDetector) IsCrawler(userAgent string) bool

IsCrawler is detect crawlers/spiders/bots by user agent

func (*CrawlerDetector) IsExclusion

func (cd *CrawlerDetector) IsExclusion(userAgent string) bool

IsExclusion is detect exclusion from user agent

func (*CrawlerDetector) IsMobile

func (cd *CrawlerDetector) IsMobile(userAgent string) bool

IsMobile is detect mobile device by user agent

func (*CrawlerDetector) Parse

func (cd *CrawlerDetector) Parse(userAgent string) *CrawlerDetector

Parse is to perform all operations by user agent

func (*CrawlerDetector) ParseUnsafe

func (cd *CrawlerDetector) ParseUnsafe(userAgent string) *CrawlerDetector

ParseUnsafe is to perform all browser and mobile operations by user agent but if not is a browser we asume that is a crawler

type Piwik

type Piwik struct {
	Name  string `json:"name"`
	Regex string `json:"regex"`
}

Piwik struct to parse the yml

func (*Piwik) String

func (m *Piwik) String() string

String function to dump the Regex