crawlerdetect

README

crawlerdetect

About

crawlerdetect is a Go version of PHP class @CrawlerDetect.

It helps to detect bots/crawlers/spiders via the user agent and other HTTP-headers. Currently able to detect 1,000's of bots/spiders/crawlers.

Installation

go get github.com/x-way/crawlerdetect

Basic Usage

import "fmt"
import "github.com/x-way/crawlerdetect"

func main() {
    uastring := "curl/7.54.0"
    if crawlerdetect.IsCrawler(uastring) {
        fmt.Println("Found a crawler")
    }
}

Contributing

The patterns and testcases are synced from the PHP repo. If you find a bot/spider/crawler user agent that crawlerdetect fails to detect, please submit a pull request with the regex pattern and a testcase to the upstream PHP repo.

Documentation

Overview

Package crawlerdetect provides a Go version of the https://github.com/JayBizzle/Crawler-Detect PHP library It can be used to detect crawlers based on their HTTP User Agent header

Here is a simple example

uastring := "curl/7.54.0"
if crawlerdetect.IsCrawler(uastring) {
        fmt.Println("Found a crawler")
}

To use custom patterns a dedicated instance can be created:

uastring := "curl/7.54.0"
crawlerDetect := crawlerDetect.New()
crawlerDetect.SetCrawlers([]string{`curl`})
if crawlerDetect.IsCrawler(uastring) {
        fmt.Println("Found a crawler")
}

Index

• func IsCrawler(input string) bool
• type CrawlerDetect
  • func New() *CrawlerDetect
  • func (c *CrawlerDetect) IsCrawler(input string) bool
  • func (c *CrawlerDetect) SetCrawlers(crawlers []string)
  • func (c *CrawlerDetect) SetExclusions(exclusions []string)

Functions

func IsCrawler

func IsCrawler(input string) bool

IsCrawler checks for a user agent string if it is a crawler

Types

type CrawlerDetect

type CrawlerDetect struct {
	// contains filtered or unexported fields
}

CrawlerDetect contains the patterns and exclusions for detecting crawlers

func New

func New() *CrawlerDetect

New creates a new CrawlerDetect instance with the default patterns

func (*CrawlerDetect) IsCrawler

func (c *CrawlerDetect) IsCrawler(input string) bool

IsCrawler checks for a user agent string if it is a crawler

func (*CrawlerDetect) SetCrawlers

func (c *CrawlerDetect) SetCrawlers(crawlers []string)

SetCrawlers sets a custom list of crawler patterns

func (*CrawlerDetect) SetExclusions

func (c *CrawlerDetect) SetExclusions(exclusions []string)

SetExclusions sets a custom list of exclusion patterns