workqueue

Documentation

Overview

Package workqueue provides a mechanism for keeping a queue of work to be done.

Index

• type QueueAddCount
• type QueueAddFunc
• type QueueDoneFunc
• type WorkCounter
  • func (ctr *WorkCounter) Add(todo int64)
  • func (ctr *WorkCounter) Done(done int64)
  • func (ctr *WorkCounter) SetStatusCallback(f func(int64, int64))
  • func (ctr *WorkCounter) Stats()
• type WorkQueue
  • func NewWorkQueue(queueSize int, scope []*url.URL, allowUpgrades bool) *WorkQueue
  • func (q *WorkQueue) AddTasks(tasks ...*task.Task)
  • func (q *WorkQueue) GetAddCount() QueueAddCount
  • func (q *WorkQueue) GetAddFunc() QueueAddFunc
  • func (q *WorkQueue) GetCounter() *WorkCounter
  • func (q *WorkQueue) GetDoneFunc() QueueDoneFunc
  • func (q *WorkQueue) GetWorkChan() <-chan *task.Task
  • func (q *WorkQueue) InputFinished()
  • func (q *WorkQueue) Run()
  • func (q *WorkQueue) RunInBackground()
  • func (q *WorkQueue) SeedFromRobots(scope []*url.URL, clientFactory client.ClientFactory)
  • func (q *WorkQueue) WaitPipe()

Types

type QueueAddCount

type QueueAddCount func(int)

type QueueAddFunc

type QueueAddFunc func(...*task.Task)

type QueueDoneFunc

type QueueDoneFunc func(int)

type WorkCounter

type WorkCounter struct {
	sync.Mutex
	sync.Cond
	// contains filtered or unexported fields
}

Count work to do and work done

func (*WorkCounter) Add

func (ctr *WorkCounter) Add(todo int64)

Increment the count to be done (input)

func (*WorkCounter) Done

func (ctr *WorkCounter) Done(done int64)

Increment the count that is done (output)

func (*WorkCounter) SetStatusCallback

func (ctr *WorkCounter) SetStatusCallback(f func(int64, int64))

Set the status callback for this workcounter

func (*WorkCounter) Stats

func (ctr *WorkCounter) Stats()

Update the stats of the counter

type WorkQueue

type WorkQueue struct {
	// contains filtered or unexported fields
}

WorkQueue is a singleton that maintains the queue of work to be done. It reads from one input channel, verifies that the URL is in scope, queues it, then writes it to the work channel to be done. Internally, it implements a singly-linked list.

func NewWorkQueue

func NewWorkQueue(queueSize int, scope []*url.URL, allowUpgrades bool) *WorkQueue

func (*WorkQueue) AddTasks

func (q *WorkQueue) AddTasks(tasks ...*task.Task)

func (*WorkQueue) GetAddCount

func (q *WorkQueue) GetAddCount() QueueAddCount

func (*WorkQueue) GetAddFunc

func (q *WorkQueue) GetAddFunc() QueueAddFunc

func (*WorkQueue) GetCounter

func (q *WorkQueue) GetCounter() *WorkCounter

Get the counter

func (*WorkQueue) GetDoneFunc

func (q *WorkQueue) GetDoneFunc() QueueDoneFunc

func (*WorkQueue) GetWorkChan

func (q *WorkQueue) GetWorkChan() <-chan *task.Task

func (*WorkQueue) InputFinished

func (q *WorkQueue) InputFinished()

func (*WorkQueue) Run

func (q *WorkQueue) Run()

func (*WorkQueue) RunInBackground

func (q *WorkQueue) RunInBackground()

func (*WorkQueue) SeedFromRobots

func (q *WorkQueue) SeedFromRobots(scope []*url.URL, clientFactory client.ClientFactory)

func (*WorkQueue) WaitPipe

func (q *WorkQueue) WaitPipe()