crawler

Documentation

Index

• func DoCrawler()
• type CollyCrawler
  • func NewCollyCrawler(d time.Duration) *CollyCrawler
  • func (c *CollyCrawler) ExportAndSave(date model.Date)
  • func (c *CollyCrawler) Work()
• type Pool
  • func NewPool(maxGoroutines int) *Pool
  • func (p *Pool) Run(w Worker)
  • func (p *Pool) Shutdown()
• type Worker

Functions

func DoCrawler

func DoCrawler()

DoCrawler 开始爬取

Types

type CollyCrawler

type CollyCrawler struct {
	// contains filtered or unexported fields
}

func NewCollyCrawler

func NewCollyCrawler(d time.Duration) *CollyCrawler

func (*CollyCrawler) ExportAndSave

func (c *CollyCrawler) ExportAndSave(date model.Date)

ExportAndSave 导出excel文件并储存至数据库

func (*CollyCrawler) Work

func (c *CollyCrawler) Work()

type Pool

type Pool struct {
	// contains filtered or unexported fields
}

Pool 提供一个 goroutine 池，这个池可以完成 任何已提交的 Worker 任务

func NewPool

func NewPool(maxGoroutines int) *Pool

NewPool 创建一个新工作池

func (*Pool) Run

func (p *Pool) Run(w Worker)

Run 提交工作到工作池

func (*Pool) Shutdown

func (p *Pool) Shutdown()

Shutdown 等待所有 goroutine 停止工作

type Worker

type Worker interface {
	Work()
}

Worker 必须满足接口类型， 才能使用工作池 用来管理一个 goroutine 池来完成工作