Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Distributer ¶
type Distributer interface {
// The master node sends a task from the repository
Send(clientNum int) Task
// Receive a task from the node to the repository
Receive(task *Task)
// Returns the number of nodes connected to it
CountNodes() int
}
Distributer : Distributed interface
type Task ¶
type Task struct {
ID int
Spiders []map[string]string // spider rule name field with keyin field, specify format map [string] string {"name": "baidu", "keyIn": "henry"}
ThreadNum int // global maximum concurrency
Pausetime int64 // pause duration reference / ms (random: Pausetime / 2 ~ Pausetime * 2)
OutType string // output method
DockerCap int // Subdivide container capacity
DockerQueueCap int // Segment output pool capacity, not less than 2
SuccessInherit bool // inherit history success record
FailureInherit bool // inherit history failure record
Limit int64 // set the upper limit, 0 is not limited, if the rule is set to the initial value of Limit is a custom limit, otherwise the default limit request number
ProxyMinute int64 // The number of minutes of proxy IP replacement
// Optional
KeyIns string // Custom input, late cut into multiple tasks KeyIn custom configuration
}
Task : for distributed distribution tasks
type TaskJar ¶
type TaskJar struct {
Tasks chan *Task
}
TaskJar : task repository
Source Files
Click to show internal directories.
Click to hide internal directories.