nsqcrawl

Documentation

Index

• type Queue
  • func NewProducer(topic string) *Queue
  • func NewQueue(topic, channel string, maxInFlight int) *Queue
  • func (queue *Queue) Close() (err error)
  • func (queue *Queue) Get() (crawl.Job, error)
  • func (queue *Queue) Schedule(ctx context.Context, req *crawl.Request) (err error)
• type Request

Types

type Queue

type Queue struct {
	*consumer.Consumer
	*producer.Producer
	// contains filtered or unexported fields
}

Queue - NSQ Queue.

func NewProducer

func NewProducer(topic string) *Queue

NewProducer - Creates queue producer.

func NewQueue

func NewQueue(topic, channel string, maxInFlight int) *Queue

NewQueue - Creates nsq consumer and producer.

func (*Queue) Close

func (queue *Queue) Close() (err error)

Close - Closes consumer and producer.

func (*Queue) Get

func (queue *Queue) Get() (crawl.Job, error)

Get - Gets job from channel.

func (*Queue) Schedule

func (queue *Queue) Schedule(ctx context.Context, req *crawl.Request) (err error)

Schedule - Schedules job in nsq. It will not call job.Done ever.

type Request

type Request struct {
	Request  *crawl.Request `json:"request,omitempty"`
	Deadline time.Time      `json:"deadline,omitempty"`
	Metadata metadata.MD    `json:"metadata,omitempty"`
}

Request - Request as it is in NSQ.