crawlerdetector

package module
Version: v0.0.8 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 17, 2019 License: MIT Imports: 5 Imported by: 0

README

About Crawler Detector

Crawler Detector is a Golang package for detecting bots/crawlers/spiders via the user agent.

Installation

Run go get github.com/dxas90/crawlerdetector.

Usage

detector := crawlerdetector.New()
isCrawler := detector.IsCrawler(r.Header.Get("User-Agent"))

if isCrawler {
    // Do something
}

Contributing

If you find a bot/spider/crawler user agent that Crawler Detector not detect, please submit a pull request with the regex pattern.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CrawlersList

func CrawlersList() string

CrawlersList is list of crawlers/spiders/bots

func ExclusionsList

func ExclusionsList() string

ExclusionsList is list of exclusions

func MobilesList

func MobilesList() string

MobilesList is list of most used mobiles

func PiwikCrawlersList added in v0.0.2

func PiwikCrawlersList() string

PiwikCrawlersList is list of crawlers/spiders/bots by Piwik

func PiwikMobilesList added in v0.0.2

func PiwikMobilesList() string

PiwikMobilesList is list of most used mobiles

func ShortCrawlersList added in v0.0.6

func ShortCrawlersList() string

ShortCrawlersList is list of crawlers/spiders/bots

func ShortExclusionsList added in v0.0.6

func ShortExclusionsList() string

ShortExclusionsList is list of exclusions

func ShortMobilesList added in v0.0.6

func ShortMobilesList() string

ShortMobilesList is a small list of most used mobiles

Types

type CrawlerDetector

type CrawlerDetector struct {
	Crawlers   *regexp.Regexp
	Mobiles    *regexp.Regexp
	Exclusions *regexp.Regexp
	Matched    []string
	Browser    bool
	Crawler    bool
	Mobile     bool
}

CrawlerDetector is crawler detector structure

func New

func New() *CrawlerDetector

New returns a new initialized CrawlerDetector

func NewPiwik added in v0.0.6

func NewPiwik() *CrawlerDetector

NewPiwik returns a new initialized CrawlerDetector from Piwik

func NewShort added in v0.0.6

func NewShort() *CrawlerDetector

NewShort returns a new basic initialized CrawlerDetector

func (*CrawlerDetector) GetMatched

func (cd *CrawlerDetector) GetMatched() []string

GetMatched is getter of matched result

func (*CrawlerDetector) IsCrawler

func (cd *CrawlerDetector) IsCrawler(userAgent string) bool

IsCrawler is detect crawlers/spiders/bots by user agent

func (*CrawlerDetector) IsExclusion

func (cd *CrawlerDetector) IsExclusion(userAgent string) bool

IsExclusion is detect exclusion from user agent

func (*CrawlerDetector) IsMobile

func (cd *CrawlerDetector) IsMobile(userAgent string) bool

IsMobile is detect mobile device by user agent

func (*CrawlerDetector) Parse added in v0.0.3

func (cd *CrawlerDetector) Parse(userAgent string) *CrawlerDetector

Parse is to perform all operations by user agent

func (*CrawlerDetector) ParseUnsafe added in v0.0.4

func (cd *CrawlerDetector) ParseUnsafe(userAgent string) *CrawlerDetector

ParseUnsafe is to perform all browser and mobile operations by user agent but if not is a browser we asume that is a crawler

type Piwik added in v0.0.2

type Piwik struct {
	Name  string `json:"name"`
	Regex string `json:"regex"`
}

Piwik struct to parse the yml

func (*Piwik) String added in v0.0.2

func (m *Piwik) String() string

String function to dump the Regex

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
t or T : Toggle theme light dark auto
y or Y : Canonical URL