crawler

Documentation

Index

• type Crawler
  • func New(id int) Crawler
• type Pool
  • func NewCrawlerPool() Pool
• type SpiderQueue
  • func NewSpiderQueue() SpiderQueue

Types

type Crawler

type Crawler interface {
	Init(*spider.Spider) Crawler // Initialize the collection engine
	Run()                        // Run the task
	Stop()                       // Take the initiative to terminate
	CanStop() bool               // Can you stop
	GetID() int                  // Get the engine ID
}

Crawler is a acquisition engine

func New

func New(id int) Crawler

New is a function for create new crawler

type Pool

type Pool interface {
	Reset(spiderNum int) int
	Use() Crawler
	Free(Crawler)
	Stop()
}

Pool is a engine pool collection

func NewCrawlerPool

func NewCrawlerPool() Pool

NewCrawlerPool is a function for create new crawler pool

type SpiderQueue

type SpiderQueue interface {
	Reset() // Reset the queue
	Add(*Spider)
	AddAll([]*Spider)
	AddKeyIns(string) // Add the KeyIn property to the queue member traversal, but the prerequisite must be that the queue member has not been added to the keyin
	GetByIndex(int) *Spider
	GetByName(string) *Spider
	GetAll() []*Spider
	Len() int // Return queue length
}

SpiderQueue is a interface for collect the rule queue in the engine

func NewSpiderQueue

func NewSpiderQueue() SpiderQueue

NewSpiderQueue is a function for create new spider queue