archiver

Documentation

Index

• type Archiver
  • func NewArchiver(parallelism uint) *Archiver
  • func (archiver *Archiver) Archive(urls ...*url.URL) (*warc.File, error)
  • func (archiver *Archiver) OnJobCompleted(job *pipeline.Job, records ...*warc.Record)
  • func (archiver *Archiver) OnJobFailed(job *pipeline.Job, err error)
  • func (archiver *Archiver) OnPerformJob(job *pipeline.Job)
  • func (archiver *Archiver) Schedule(jobs ...*pipeline.Job)

Types

type Archiver

type Archiver struct {
	MaxDepth      uint32
	Render        bool
	RenderQuality int

	ResolverAddress net.IP
	ResolverPort    uint16
	UserAgent       string
	// contains filtered or unexported fields
}

Archiver contains options for an archiver.

func NewArchiver

func NewArchiver(parallelism uint) *Archiver

NewArchiver creates a new archiver following best practices.

func (*Archiver) Archive

func (archiver *Archiver) Archive(urls ...*url.URL) (*warc.File, error)

Archive archives a URL as a WARC archive.

func (*Archiver) OnJobCompleted

func (archiver *Archiver) OnJobCompleted(job *pipeline.Job, records ...*warc.Record)

OnJobCompleted handles a completed job.

func (*Archiver) OnJobFailed

func (archiver *Archiver) OnJobFailed(job *pipeline.Job, err error)

OnJobFailed handles a failed job, potentially rescheduling it.

func (*Archiver) OnPerformJob

func (archiver *Archiver) OnPerformJob(job *pipeline.Job)

OnPerformJob handles jobs before being performed.

func (*Archiver) Schedule

func (archiver *Archiver) Schedule(jobs ...*pipeline.Job)

Schedule schedules a job for processing.