cache

Documentation

Overview

Package cache provides common configuration and cache for task runtime.

Index

• Variables
• func ExecInit(order int)
• func GetPageCount(i int) uint64
• func PageFailCount()
• func PageSuccCount()
• func ResetPageCount()
• func WaitInit(order int)
• type AppConf
• type Report

Variables

var (
	StartTime  time.Time    // timestamp when start button was clicked
	ReportChan chan *Report // text data summary report channel

)

var Task = new(AppConf)

Task holds the default runtime configuration.

Functions

func ExecInit

func ExecInit(order int)

ExecInit marks the init at the given order as completed.

func GetPageCount

func GetPageCount(i int) uint64

GetPageCount returns page counts: i>0 returns success count, i<0 returns failure count, i==0 returns total.

func PageFailCount

func PageFailCount()

PageFailCount increments the failure page count.

func PageSuccCount

func PageSuccCount()

PageSuccCount increments the success page count.

func ResetPageCount

func ResetPageCount()

ResetPageCount resets the page counters.

func WaitInit

func WaitInit(order int)

WaitInit blocks until the init at the given order has completed. Must be called from a goroutine.

Types

type AppConf

type AppConf struct {
	Mode           int    // node role
	Port           int    // master node port
	Master         string // master node address (without port)
	ThreadNum      int    // global max concurrency
	Pausetime      int64  // pause duration reference in ms (random: Pausetime/2 ~ Pausetime*2)
	OutType        string // output method
	BatchCap       int    // batch output capacity per flush
	Limit          int64  // crawl limit; 0 means unlimited; if set to LIMIT in rules, uses custom limit; otherwise defaults to request count limit
	ProxyMinute    int64  // proxy IP rotation interval in minutes
	SuccessInherit bool   // inherit historical success records
	FailureInherit bool   // inherit historical failure records
	Keyins         string // custom input; later split into Keyin config for multiple tasks
}

AppConf holds the common configuration for task runtime.

type Report

type Report struct {
	SpiderName string
	Keyin      string
	DataNum    uint64
	FileNum    uint64
	// DataSize   uint64
	// FileSize uint64
	Time time.Duration
}

Report summarizes task execution results.